DATallegro’s technical strategy
Few areas of technology boast more architectural diversity than data warehousing. Mainframe DB2 is different from Teradata, which is different from the leading full-spectrum RDBMS, which are different from disk-based appliances, which are different from memory-centric solutions, which are different from disk-based MOLAP systems, and so on. What’s more, no two members of the same group are architected the same way; even the market-leading general purpose DBMS have important differences in their data warehousing features.
The hot new vendor on the block is DATallegro, which is stealing much of the limelight formerly enjoyed by data warehouse appliance pioneer Netezza. (After some good early discussions, Netezza abruptly reneged on a promise a year ago to explain more about its technology workings to me, and I’ve hardly heard from them since. Yes, they’re still much bigger than DATallegro, but I suspect they’ve hit some technical roadblocks, and their star is fading.)
Like Netezza, DATallegro’s basic strategy is to stream data on and off disk very quickly, sequentially rather than randomly, resolving queries with full table scans. Unlike Netezza, which actually has chip-level optimizations, DATallegro’s most direct I/O acceleration lies in protocol software for talking to Infiniband storage systems.
The “on” part of “on and off disk” is particularly important in DATallegro’s case. V2, released this week, has optimizations for moving data around quickly. This is useful for rebalancing range partitions, to maximize performance. (In general, range partitioning is a big part of DATallegro’s story.) CEO Stuart Frost also claims a much more dramatic benefit when a query creates large intermediate tables, such as the >1 terabyte tables a Microstrategy query can create along the way to a final result. Supposedly, a query that DATallegro can do in a few minutes takes all night on competitive systems.
Other new V2 goodness includes a very simple way to prioritize short queries over long ones, and hence give them consistent response time (which is good for “operational BI” and the like). Namely, if a query thread has been around for along time, it automatically gets lowered in priority. This is a simple matter of operating systems and clocks; the query optimizer is in no way involved.
Ultimately, DATallegro’s technical success story seems a bit paradoxical. It relies on the richer data warehousing functionality found in Ingres vs., say, the PostgreSQL that Netezza relies on. But absent DATallegro, one wouldn’t tend to think of Ingres as a particularly strong data warehousing system at all. Go figure.
In line with the praise I gave earlier for one of their white papers, an unusually clear brief technology overview may be found on DATallegro’s website here.
Comments
17 Responses to “DATallegro’s technical strategy”
Leave a Reply
Thanks, this is great. My only comment is on the partitioning. You’re right that true range partitioning is important in our system – especially for dates etc. However, the key is that we can use it in conjunction with hash partitioning both across and within the nodes.
For example, to get full advantage of the parallel power of the system, I start by looking for a hash key that will spread the data as evenly as possible and be useful for a significant number of queries (e.g. account number, phone number or a suitable surrogate key). This is used to spread rows across the nodes.
Within the nodes, I’d usually start by date ranging, but then also look at hashing on further keys that are involved in joins or perhaps that I use a lot in where clauses. Most DBAs know this level of info about their data, so it’s not such a big deal and is usually less than a day’s work to setup.
The part about intermediate tables also involves hash partitioning. If a series of queries (e.g. from MicroStrategy) involves creating a few tables (e.g. aggregates) and then joining them at the end, the problem is that the tables might be created in a way that makes the join very slow (i.e the relevant partitions of the various tables are on different nodes). In competitive systems, the data movement to resolve this is very slow, but in ours, we’ve come up with a mechanism to make it very fast. However, it still involves hash partitioning, not range.
Sorry, just though of another comment – this time on Ingres:
You’re correct in saying that Ingres isn’t known as a particularly strong data warehousing platform. However, we’ve spent the last 2 years working with Ingres to add a lot of features so that it works well within our platform.
Also, our own software drives Ingres in a very specific way so that it doesn’t need ‘traditional’ DW features such as bit-mapped indexes etc. In fact, our focus is very much on avoiding indexes, since they add overhead, are difficult to manage and generally force the system to use random I/O rather than sequential (the latter being much faster and allowing the use of large, low cost disks).
In effect, Ingres is embedded in our appliance, with each instance being used as a self-contained database. Our own software takes care of the parallel aspects of running queries across these instances.
We also provide all customer support (although on-site hardware support is outsourced to IBM).
The end result is that end users and (generally speaking) DBAs work with a DATAllegro database and aren’t really aware that Ingres is inside.
Thanks, Stuart.
I just looked around, and couldn’t find any evidence that PostgreSQL, the underpinning of Netezza, supports hash partitioning either. Who knew?
Given that, I’ll skip any further questions about the relative importance of hash and range partitioning to you.
As for the Ingres comment, please let me confirm something to make it crystal clear. As I understand it, a query is first parsed and optimized by your own proprietary engine. The “data access method” for that engine is to dispatch the appropriate SQL queries to individual nodes, where they are executed by individual instances of Ingres.
Correct?
And have you done anything inside Ingres itself to change it?
Curt Monash
Your assumptions on how we use Ingres are correct. We’ve worked very closely with Ingres Corp on bug fixes and some additional features to improve performance in our context.
All of these changes to Ingres have been published in the GPL version. Note that our parallization code only uses APIs or other standard interfaces to Ingres (e.g. SQL) and therefore remains proprietary.
The lack of hash partitioning in PostgreSQL was a major reason for our switch to Ingres as soon as it became available in open source. We’ve found the partitioning in Ingres to be extremely powerful and easy to use. When combined with our own software, we think it gives us a huge advantage over competing systems that lack true partitioning at the node level.
Stuart
Thanks. I guess I just misremembered and thought you’d put more emphasis on range than hash partitioning when we spoke.
By the way, you and I are a pair of right geeks for posting back-and-forth like this late on a Friday night. But I outgeek even you — I’m on the EAST Coast. 🙂
Yes, you won that one Curt – I went to bed after my last post!
A clarification – we use Infiniband for communication between the various nodes in a system, not for storage. The benefits are very high speed (10Gbps node-to-node with no blocking at the switch so all nodes can communicate at the same time) and minimal CPU loading on the nodes so other queries can still run well while data is being moved. This is MUCH better than the GigE connections used by our competitors.
Beyond just using standard Infiniband, we’ve developed our own protocols to allow reliable bulk movements of data at very high speed. We’ve also integrated this capability into Ingres so tables can be loaded and moved very quickly.
For disk I/O, we use dedicated RAID arrays in each chassis so we can completely control what’s going on on each disk. This allows us to optimize the entire system stack from a SQL query coming in to what happens on each disk. The end result is that we can maintain high sequential read rates, even when there are many different queries running in parallel.
We feel this is one of the main benefits of the appliance approach. Traditional ‘software’ DBMS vendors can’t predict or control the hardware platform to anything like the same extent. The end result is a series of compromises that lead to massively increased complexity and greatly reduced performance.
Stuart
Stuart,
Thanks for the further correction on the Infiniband point — and I hope you slept well! 🙂
Now, since you stressed the appliance approach, a natural follow-up question occurs to me. Suppose you were to port your software to a large hardware vendor’s system — what, if anything, would be lost in the translation?
CAM
[…] The data warehousing appliance vendors integrate DBMS and OS. Stuart Frost of DATallegro made some excellent, detailed comments in this thread laying out that case. […]
Now that’s an interesting question!
I believe we’re unique among the DW appliance vendors wrt using 100% commodity hardware, so it is possible for us to ‘port’ our software to another platform.
If we were to specify a specific combination of hardware from a third-party vendor, we could maintain the benefits of the appliance approach, while at the same time allowing customers to use hardware from their preferred vendors. However, the hardware cost would be significantly higher, so there would be some trade-offs in that area. We’d also need to assess the performance of the available storage systems and dedicate a set of disks to each CPU. Even so, I think we would still be extremely competitive on price and performance.
Stuart
[…] A good illustration of these points may be found in an exchange I had with Stuart Frost, CEO of DATallegro. DATallegro is a data warehouse appliance maker heavily optimized for streaming data on and off disk, moving it between mulitiple processors, and making the best use of onboard cache, memory bus, etc. Stuart thinks it’s likely that he could get much of the same benefit from shipping on a known configuration of name-brand equipment. But I can’t imagine any way that DATallegro’s software would work well on any kind of virtualized multi-purpose grid kind of setup. Blech. The vast majority of their engineering would be simply wasted. […]
You’re absolutely right about ‘virtualized multi-purpose grids’ or anything else where we don’t have full control of every aspect of the software and hardware platform. Data warehousing is just too extreme in terms of both I/O and CPU power to work through so many layers of infrastructure. The whole idea of the appliance concept is that the vendor takes responsibility for the entire stack. The end result is greatly reduced setup and maintenance efforts on the part of the customer and a lot less pain for the DBA.
Stuart
[…] Virtualization is in many ways pure goodness, just as proponents say. But even so, I think it’s being overhyped. As software, hardware, and processor vendors all get larger, economies of scale are allowing very tight development coupling so as to optimize performance, power usage, etc. For example, I’m running into Intel engineers at almost every large company I follow. If you buy software — and who builds their own if they don’t have to? — you’re now likely to get something that’s been carefully optimized for very specific operating environments. And then there are appliances, which are still trending up, not down. (See also what Stuart Frost of DATallegro has to say on that point.) Or check out this ostensibly pro-virtualization article that really is in agreement for me. […]
[…] Data warehouse appliances (see especially this discussion of DATallegro’s architecture) […]
[…] Over the past year, Netezza has exhibited the squirreliest question-dodging behavior I’ve seen from a DBMS vendor since – actually, since Sybase tried to conceal the System 10 fiasco in 1993-5. To its credit, however, Netezza finally decided to open the kimono. Specifically, they invited me to their user conference, which I attended today, and indeed were quite helpful in FINALLY getting my questions addressed, and in offering more access as needed. […]
[…] DATallegro has almost no indices, and also has preconfigured hardware. But it has some partitioning, optionally. […]
[…] A while ago – for example, in a comment dated July 9, 2006 — CEO Stuart Frost of DATAllegro hinted that the company might port its software to commodity hardware before long. If this user story is to be believed, that has now happened. (Specific quote: “the Datallegro system is based on Dell and EMC hardware …”) Officially, the company is pulling a Sgt. Schultz on the subject. But the evidence is pretty clear. MORE And so it would seem that Netezza and Teradata are the only remaining data warehouse appliance vendors committed to custom hardware strategies. […]
[…] Общая ссылка на освещение DATAllegro […]