DATAllegro on compression
DATAllegro CEO Stuart Frost has been blogging quite a bit recently (and not before time!). A couple of his posts have touched on compression. In one he gave actual numbers for compression, namely:
DATAllegro compresses between 2:1 and 6:1 depending on the content of the rows, whereas column-oriented systems claim 4:1 to 10:1.
In another recent post, Stuart touched on architecture, saying:
Due to the way our compression code works, DATAllegro’s current products are optimized for performance under heavy concurrency. The end result is that we don’t use the full power of the platform when running one query at a time.
Not immediately seeing the connection, I emailed Stuart for clarification. His answer boiled down to:
- Compression adds to I/O latency.
- This only causes a performance hit when there’s an unrealistically small number of users/processes.
- So they didn’t bother fixing it.
More precisely, he wrote:
This gets a little complex, but decompression effectively adds latency to I/O. Our platform runs a certain number of threads on each server for a given query (currently six) so that each CPU has a dedicated disk platter. With a single query, the (serial) latency for decomp after the I/O effectively creates some I/O wait time and therefore slows down overall performance. With multiple queries, this I/O wait time is used for other useful work on different queries, so the platform is fully utilized.
Also, we have eight cores, so six threads doesn’t always use all of them for a single query. Again, this goes away when you have multiple queries.
Since most customers have multiple queries running at the same time, we optimized for that workload, rather than trying to run more threads for single queries, because the latter starts to create problems when you have LOTS of things going on.
Comments
Leave a Reply