MongoDB 3.0
Old joke:
- Question: Why do policemen work in pairs?
- Answer: One to read and one to write.
A lot has happened in MongoDB technology over the past year. For starters:
- The big news in MongoDB 3.0* is the WiredTiger storage engine. The top-level claims for that are that one should “typically” expect (individual cases can of course vary greatly):
- 7-10X improvement in write performance.
- No change in read performance (which however was boosted in MongoDB 2.6).
- ~70% reduction in data size due to compression (disk only).
- ~50% reduction in index size due to compression (disk and memory both).
- MongoDB has been adding administration modules.
- A remote/cloud version came out with, if I understand correctly, MongoDB 2.6.
- An on-premise version came out with 3.0.
- They have similar features, but are expected to grow apart from each other over time. They have different names.
*Newly-released MongoDB 3.0 is what was previously going to be MongoDB 2.8. My clients at MongoDB finally decided to give a “bigger” release a new first-digit version number.
To forestall confusion, let me quickly add:
- MongoDB acquired the WiredTiger product and company, and continues to sell the product on a standalone basis, as well as bundling a version into MongoDB. This could cause confusion because …
- … the standalone version of WiredTiger has numerous capabilities that are not in the bundled MongoDB storage engine.
- There’s some ambiguity as to when MongoDB first “ships” a feature, in that …
- … code goes to open source with an earlier version number than it goes into the packaged product.
I should also clarify that the addition of WiredTiger is really two different events:
- MongoDB added the ability to have multiple plug-compatible storage engines. Depending on how one counts, MongoDB now ships two or three engines:
- Its legacy engine, now called MMAP v1 (for “Memory Map”). MMAP continues to be enhanced.
- The WiredTiger engine.
- A “please don’t put this immature thing into production yet” memory-only engine.
- WiredTiger is now the particular storage engine MongoDB recommends for most use cases.
I’m not aware of any other storage engines using this architecture at this time. In particular, last I heard TokuMX was not an example. (Edit: Actually, see Tim Callaghan’s comment below.)
Most of the issues in MongoDB write performance have revolved around locking, the story on which is approximately:
- Until MongoDB 2.2, locks were held at the process level. (One MongoDB process can control multiple databases.)
- As of MongoDB 2.2, locks were held at the database level, and some sanity was added as to how long they would last.
- As of MongoDB 3.0, MMAP locks are held at the collection level.
- WiredTiger locks are held at the document level. Thus MongoDB 3.0 with WiredTiger breaks what was previously a huge write performance bottleneck.
In understanding that, I found it helpful to do a partial review of what “documents” and so on in MongoDB really are.
- A MongoDB document is somewhat like a record, except that it can be more like what in a relational database would be all the records that define a business object, across dozens or hundreds of tables.*
- A MongoDB collection is somewhat like a table, although the documents that comprise it do not need to each have the same structure.
- MongoDB documents want to be capped at 16 MB in size. If you need one bigger, there’s a special capability called GridFS to break it into lots of little pieces (default = 1KB) while treating it as a single document logically.
*One consequence — MongoDB’s single-document ACID guarantees aren’t quite as lame as single-record ACID guarantees would be in an RDBMS.
By the way:
- Row-level locking was a hugely important feature in RDBMS about 20 years ago. Sybase’s lack of it is a big part of what doomed them to second-tier status.
- Going forward, MongoDB has made the unsurprising marketing decision to talk about “locks” as little as possible, relying instead on alternate terms such as “concurrency control”.
Since its replication mechanism is transparent to the storage engine, MongoDB allows one to use different storage engines for different replicas of data. Reasons one might want to do this include:
- Fastest persistent writes (WiredTiger engine).
- Fastest reads (wholly in-memory engine).
- Migration from one engine to another.
- Integration with some other data store. (Imagine, for example, a future storage engine that works over HDFS. It probably wouldn’t have top performance, but it might make Hadoop integration easier.)
In theory one can even do a bit of information lifecycle management (ILM), by using different storage engines for different subsets of the database, by:
- Pinning specific shards of data to specific servers.
- Using different storage engines on those different servers.
That said, similar stories have long been told about MySQL, and I’m not aware of many users who run multiple storage engines side by side.
The MongoDB WiredTiger option is shipping with a couple of options for block-level compression (plus prefix compression that is being used for indexes only). The full WiredTiger product also has some forms of columnar compression for data.
One other feature in MongoDB 3.0 is the ability to have 50 replicas of data (the previous figure was 12). MongoDB can’t think of a great reason to have more than 3 replicas per data center or more than 2 replicas per metropolitan area, but some customers want to replicate data to numerous locations around the world.
Related link
- I occasionally post a few notes about MongoDB use cases, e.g. last May.
Comments
9 Responses to “MongoDB 3.0”
Leave a Reply
3.0’s ability to have multiple plug-compatible storage engines. Sounds like AWS Lamdba architecture minus elastic cloud. Paves the way to build your own modern ecommerce ERP on-premises / in-cloud?
-Migrate data from 30 y/o Cullinet / Basic 4, S/2.
-Bring your own ERP code or build from Node.js
-Build your own SAP HANA appliance.
Coda. Priceless granularity in your review.
Tokutek is indeed offering a MongoDB storage engine, http://www.tokutek.com/2015/01/announcing-tokumxse-v1-0-0-rc-0/
I expect the engine API to be more successful for MongoDB than it was for MySQL. The MongoDB API requires less from a storage engine than MySQL, so it is easier to implement as more work is done above the engine by generic MongoDB code. The outcome will be more robust implementations to choose from — WiredTiger, Tokutek, RocksDB and others I have yet to discover.
Mark et al.,
Cool would be something that was a joint storage engine for MySQL and MongoDB, with good performance for both. Akiban could perhaps have been a basis for that if it had survived long enough. A little extra concurrency control would be needed to serve two top-layer masters at once, but I doubt that would be the biggest of problems.
Tokutek uses the same underlying storage technology in it’s MySQL and MongoDB offerings. As Mark said, the complexity of the MySQL storage engine API is so much more complicated than for MongoDB, so the cost of entry is likely higher than most are willing to pay.
Only Tokutek has managed to implement both MongoDB and MySQL engine APIs. RocksDB is attempting to repeat that. The MySQL engine API has always been hard and it is getting harder with more functionality pushed into it. InnoDB has gotten much better since Oracle became the owner of MySQL, but that also means we might end up in an InnoDB-only world — https://twitter.com/morgo/status/565207495035990017
Maybe MySQL will target engine diversity when innovation and interest moves over to MongoDB. MyISAM could have been a lot better many years ago just by making it crash-safe for single-writer, multi-reader workloads.
Is it possible for MySQL to both innovate/improve InnoDB while retaining a storage engine API for others to participate? I’m starting to believe that an InnoDB-only world is inevitable.
[…] two-policemen joke seems ever more […]
[…] which was the biggest deal in MongoDB 3.0, will become the default in 3.2. I continue to think analogies to InnoDB are reasonably […]