The “baseball bat” test for analytic DBMS and data warehouse appliances
More and more, I’m hearing about reliability, resilience, and uptime as criteria for choosing among data warehouse appliances and analytic DBMS. Possible reasons include:
- More data warehouses are mission-critical now, with strong requirements for uptime.
- Maybe reliability is a bit of a luxury, but the products are otherwise good enough now that users can afford to be a bit pickier.
- Vendor marketing departments are blowing the whole subject out of proportion.
The truth probably lies in a combination of all these factors.
Making the most fuss on the subject is probably Aster Data, who like to talk at length both about mission-critical data warehouse applications and Aster’s approach to making them robust. But I’m also hearing from multiple vendors that proofs-of-concept now regularly include stress tests against failure, in what can be – and indeed has been – called the “baseball bat” test. Prospects are encouraged to go on a rampage, pulling out boards, disk drives, switches, power cables, and almost anything else their devious minds can come up with to cause computer carnage. The goal is to see whether systems keep running, how well they perform when impaired, or how fast they recover if they indeed do go down.
Teradata benchmark honcho Gene Erickson definitely encourages this kind of behavior. Kognitio – which sees “resilience” as a competitive advantage and reports that prospects increasingly care about same — is friendly to such behavior as well. And that’s surely not a complete list.
Of course, some data warehouse users have cared about robustness for years, even up to the point of replication and hot standbys. Even so, I think there’s recently been a change in the market. For example, when Vertica and before that its research predecessor C-Store were rolled out, Mike Stonebraker repeatedly called attention to the potential for implementing them with high redundancy, but I and most other observers basically yawned.
Selecting analytic DBMS products is even more fun than it used to be. 🙂
Comments
6 Responses to “The “baseball bat” test for analytic DBMS and data warehouse appliances”
Leave a Reply
I love the baseball bat analogy, Curt. Makes me wish it was summer.
Another important aspect to consider is the elimination of “planned” downtime (e.g., loading data, upgrading the system, etc.). Aster Data is also focused on eliminating/reducing this aspect as well. More here: http://www.asterdata.com/product/alwayson.php
There is also a funny video about using a baseball bat on servers here (we used too small of a sledgehammer, but the effect was similar): http://www.asterdata.com/blog/index.php/2008/10/12/a-childhood-dream/
We see baseball bat testing by Vertica evaluators a lot. Comcast talked about the “swings” they took in a webcast they did last year – http://www.vertica.com/comcast
Curt, is there a reason high availability (baseball bat) testing wasn’t included in your list of analytic database assessment criteria in http://www.dbms2.com/2008/12/15/how-to-buy-an-analytic-dbms-overview/ ?
Andy,
It was kind of implicit in the references to POCs and SLAs.
Mainly, however, I was talking about universals, and not so much stressing things that are important to some users but not others.
Are companies putting their DW’s in Iraqi data centers or what?
In the large enterprises that I’ve worked with the S.O.P. is to have a completely duplicate “disaster recovery” version of the DW. Expensive but reassuring (a bit like Teradata…).
I really don’t see them moving away from this approach for a number of years, even though it is clearly wasteful and unnecessary.
I think the reality is that many CIOs are not technically proficient enough to be convinced that a “highly resilient” and/or “distributed” data warehouse can remove the need for a DR standby.
[…] And vendor-site POCs can be done quite respectably. For example, the guy who first focused me on baseball-bat testing is Teradata’s chief of in-house […]
[…] The Netezza guys at “Data Liberators” are being a bit too cute in talking about FULL DISCLOSURE yet not actually saying they’re from Netezza — but only a bit, in that their identity is pretty clear even so. That said, they’ve proposed a not-terrible checklist of how to conduct POCs. Of course, vendor-provided as it is, it’s incomplete; e.g., there’s no real mention of a baseball-bat test. […]