The Continuent Tungsten MySQL replication story
To the consternation of its then-CEO, I wrote very little about my then-client Continuent. However, when I knew Schooner’s recent announcement was coming, I reached out to other MySQL scale-out vendors too. I’ve already posted accordingly about CodeFutures (the dbShards guys) and ScaleBase. Now it’s late-responding Continuent’s turn.
Actually, what I’m mainly going to do is quote a very long email that Continuent’s current CEO/former CTO Robert Hodges sent me, and which I lightly edited.
As you know we use replication to build clusters of database replicas for open source DBMS, primarily MySQL. We have two products:
Tungsten Replicator – Tungsten Replicator is the basic data movement part of Tungsten. We are going to release it as a separate product this year and push most features into open source to increase usage.
Tungsten Enterprise – Cluster management and connectivity based on master/slave replicas. Tungsten Enterprise is commercially licensed on a subscription basis.
I guess that means Continuent’s essentially failed legacy multi-master replication product has — unsurprisingly — fallen by the wayside.
Here are the major use cases we solve.
Note that operational ease is much higher on the list than performance. Note also that Robert naturally compares Continuent Tungsten to his worst competitor, rather than any of the other interesting start-ups’ products.
1. Better MySQL Replication. Tungsten Replicator significantly extends the built-in MySQL replication to handle the following sub-use cases related to master/slave replication. Tungsten is basically a configurable message-processing engine that does database replication.
1.1 Failover with multiple slaves. Tungsten has global transaction IDs, so if you have a failure you can promote the most advanced slave from a pool of slaves to master and point all others to it. MySQL replication by contrast tracks location using pointers into the master file system, so when the master disappears the slaves lose track of their position. This use case comes up both locally as well in building DR sites.
1.2 Replication between MySQL versions. Tungsten has minimal dependencies on MySQL versions and can replicate from versions 5.0/5.1/5.5 to versions 4.1/5.0/5.1/5.5. MySQL replication by contrast at best replicates up to the next version and either cannot replicate down or has limitations on replicating down version. This comes up commonly in migrations, for example going from MySQL 4.1 to 5.1.
1.3 Heterogeneous replication. Similarly, Tungsten can replicate out of MySQL into other databases such as PostgreSQL (we are currently implementing this for a large SaaS) as well as Oracle and drizzle (currently very basic and not regularly tested but available for polishing when engagements show up). This comes up commonly when users need to move reporting data out of MySQL into something more civilized for analytics. Greenplum is popular, for example; we are working on that now.
1.4 Multi-master replication. Tungsten can consolidate from multiple masters into a single slave. We also support topologies involving bi-directional replication with slaves on each master, including ability to promote slaves to take over from its master. MySQL replication cannot do this. These topologies are required for reporting as well as systems that have configuration or process merchant data across sites.
1.5 Parallel replication. Tungsten has shard-based parallel replication from the MySQL log, which is essential to get around slave lag problems. In addition, we can spread replication processing load flexibly across CPUs and even different hosts. MySQL replication is single-threaded with a hard-coded processing path and cannot spread work across CPUs, processes, or hosts to optimize resource usage or deal with delays due to i/o.
And more operational stuff.
2. Zero-downtime upgrade. Tungsten Enterprise enables applications to implement “daylight maintenance” by taking slave databases transparently offline, upgrading/maintaining them, and bringing them back online again. This is critical for 24×7 systems that cannot afford to go down for a day in order to upgrade from MySQL 5.1 to 5.5.
Tungsten provides the management commands to make this possible, for example a simple “switch” command that flips a master/slave pair without stopping applications. Tungsten also provides the necessary connectivity, namely transparent, high-performance proxies that hide the location of master and slave databases. Finally, Tungsten Replicator provides underlying flush transactions that allow us to confirm that all user transactions have safely reached slaves before switching the master.
Now we get to performance, but read-only, and even then only for limited use cases.
3. Performance scaling. Tungsten Enterprise can load balance reads across slaves which unloads the master and utilizes otherwise idle slaves. The performance increases can be quite impressive–for reads that are highly I/O intensive you get linear scaling as you add slaves. It is less beneficial for simple key lookups, especially when databases are fully memory resident.
We have a couple of ways to do this but they basically use transparent proxies or a JDBC library to split reads and writes automatically by examining incoming transactions. For this to work most effectively applications have to follow certain conventions for their transactions. However, they are very simple and many applications do them already. For example, under our “SmartScale” load balancing only auto-commit reads are eligible for automatic load balancing. We don’t try to load-balance reads within transactions as this would lead to errors for many applications. Our performance scaling capabilities will grow better as our parallel replication capabilities improve–a big problem with MySQL replication is slaves tend to lag the masters in busy systems, hence cannot be used as easily for reads that require current information.
Yet more operations.
4. Automated administration and failure recovery. Tungsten Enterprise allows users to do complex operations like switch master/slave or take database nodes on and offline. Also many operations like failover or recognizing new slave nodes occur automatically. This reduces labor enormously both on the part of administrators as well as the effort to implement such operations in the first place.
Tungsten enables this use case in two ways. It has built-in procedures for management tasks (switch, backup, restore, online, offline, etc.) Tungsten also has built-in monitoring that feeds a rules engine containing rules to handle master failure, slave failure, new slave appearing, etc. These execute automatically, though users can turn off features like automatic failover at their discretion.
5. Disaster Recovery Sites. Tungsten Enterprise allows users to set up and and manage DR sites containing up-to-date database copies. The key features that enable this are automatic VIP management (we use a virtual IP to allow the DR site to continue receiving updates even as the master moves on the main site) as well as global transaction IDs in the replicator, which ensure local and remote slaves do not become confused about their position regardless of how failure occurs. I have a full write-up of the architecture in the following article:
http://scale-out-blog.blogspot.com/2010/10/mysql-disaster-recovery-with-tungsten.html
Finally it’s time to raise questions about Continuent competitors. 🙂
I hope this answers your questions and is not too late. I read with interest your post on DbShards. One interesting problem I have never quite figured out with their architecture is how to ensure requests are serialized. They intercept requests at the library level and apply them both locally as well as in a log that can be applied to other databases. The big problem with this approach is transaction ordering. You either have to accept extremely slow performance (e.g., by single-threading all updates) or applications have to generate order independent updates, which means a significant rewrite to follow these conventions. Otherwise you end up with slaves that look different from the master. Either way it greatly limits the market.
Schooner has always impressed me as being the most interesting play of those you are covering. It’s interesting they bailed out of hardware. Appliances for MySQL strike me as a non-starter. SSD integration has been very rapidly explored and commoditized by members of the MySQL community–people like Mark Callaghan and Peter Zaitsev are more hardware driven than any database people I have ever worked with. The fact that the code that accesses storage devices is completely open helps drive this kind of behavior. Schooner still is going to have a problem that MySQL users are very resistant to proprietary code, at least the big ones like Facebook, Yahoo, Rackspace, and Google.
A bit about future plans:
Meanwhile I expect we will take a page from Schooner and start patching the MySQL server over time to make our software run more effectively and provide better solutions to customers. Large web properties in particular are very comfortable with this approach.
And finally some mercifully short rah-rah, still on the theme “We only acknowledge one competitor and we can clobber them.” Actually, I moved this from further up, where it seemed a bit jarring.
I anticipate that over the course of 2011 Tungsten will be recognized as the best replication engine for MySQL. This is a big statement considering the entrenched position of MySQL built-in replication but our feature set is very broad and we will also have most of it available in open source by mid-year, which means that a lot more people will start using it. The list of stable improvements over MySQL is so long I cannot fit them on a single slide any more.
Comments
9 Responses to “The Continuent Tungsten MySQL replication story”
Leave a Reply
Hi Curt!
Thanks for posting the write-up as well as your insights.
I would like to respond on one point. Given the huge number of actual deployments on MySQL master/slave replication for major businesses ranging from Yahoo, Facebook, and Google down to small SaaS businesses, it seems hard to argue that Oracle’s MySQL is not the dominant solution in this space. Our main competition, in fact almost only competition, has always been open source components plus consulting and/or in-house expertise.
We are quite interested in what companies like Schooner or DBShards are up to and wish them well–we are all trying to solve a very hard problem. To be frank, though, I have never met any of them in a competitive setting. If we start to see them we would begin to pay a lot more attention. Meanwhile, our focus is to plug a number of significant gaps in the base MySQL offering used in hundreds of thousands of deployments worldwide. We will have more performance focus over the next year as we plug those gaps.
Like a lot of other people I enjoy reading your blog and it’s a pleasure to be the subject of one of your posts. Thanks again, Robert
Seems they failed not only in multi-master replication but also in trying to support Oracle, based on post from 2009. I guess this is non-trivial even in commercial implementation
@George
We still have support for replication *into* Oracle from MySQL and intend to maintain that as we see customers asking for it. The biggest issue in supporting Oracle is that the market is so different from open source that we found there was little synergy with our open source efforts. Win some, lose some…
Robert
Curt,
Thanks for the post, your insights are always invaluable.
As far as dbShards replication goes, we have definitely implemented it such that the primary/secondary servers match (otherwise it would be useless), with write speeds that are very close to (and sometimes faster than) standalone MySQL. There are many architectural and other optimizations we have implemented to accomplish this.
Given that, however, our strength is that we provide full support for Relational Sharding, so that databases scale to very high volumes without losing the relational nature of the application model (no easy task). For us replication was key because you cannot really implement a shared-nothing sharding scheme without ensuring each physical shard (a shared-nothing server) stays up. Accordingly, a big focus of our replication technology is to ensure that it is fully scalable as we add shards, with minimal overhead compared to native database performance. It was certainly a tall order to accomplish, but it is working very well in production for customers, so we’re very happy about that.
We too will look at Schooner, I’m interested to see how customers do adopting this approach, but it looks very promising.
Cory
Robert, Cory,
You might be interested in my student’s (Alex Thomson) work on high performance, scalable database replication: http://cs-www.cs.yale.edu/homes/dna/papers/determinism-vldb10.pdf
It provides an interesting way around the quandary that Robert thought DbShards might have, that he mentioned above.
Hi Daniel, thanks for the reference. I cheated and skipped to the end where you summarized that making lock behavior deterministic simplifies solving replication performance problems, a point on which many reasonable persons would agree. I will read the paper more carefully but one practical point that immediately pops up for me is that implementing completely deterministic locks is very hard. For example, any DBMS that allows either page locks or lock promotion will introduce behavior that varies depending on the exact distribution of data on storage pages as well as things like the exact number of locks held by currently executing threads.
Hello,
This has been an interesting read, especially as I am looking to use a tool that provides essential services for replication such as automatic failover. A search on MySQL forums for replication best practices yielded Tungsten, and I looked out for it.
Though I am not an expert, or even a novice for that matter, on Tungsten Enterprise product, I think it is unfair to compare Tungsten with Schooner or other such product, as the latter provides synchronous replication management, where Tungsten only provides support for, and builds over, asynchronous replication. It does not support Cluster management with respect to synchronous replication.
An area where I think Tungsten is lacking is providing a full fledged GUI for management, as opposed to their command suite.
I am looking forward to evaluate their product and will have more experiences to share after that.
Meanwhile, I’d be interested in hearing your and others’ feedback (positive or negative) who have used Tungsten so that it helps newbies like me reach a decision.
Kind Regards,
Trupti
Hi
I am kindly asking that you please send me a step by step tutorial on how to set up fan in replication with two separate databases on two master servers being replicated onto one shared database on a slave machine using tungsten replicator. It is my first time to use tungsten so I need a detailed approach right from installation to configuration and lastly testing in linux ubuntu. Currently I am using mysql.
I have searched the internet and all I can find is the fact that it is possible but not how exactly to do so.
Your assistance will be greatly appreciated.
Rumbi
mukrue@yahoo.com
Hi Rumbi,
You might want to contact Continuent directly.