The Netezza guys propose a POC checklist
The Netezza guys at “Data Liberators” are being a bit too cute in talking about FULL DISCLOSURE yet not actually saying they’re from Netezza — but only a bit, in that their identity is pretty clear even so. That said, they’ve proposed a not-terrible checklist of how to conduct POCs. Of course, vendor-provided as it is, it’s incomplete; e.g., there’s no real mention of a baseball-bat test.
Here’s the first part of the Netezza list, with my comments interspersed.
1. Decide on a timeline and constrain that to two to three weeks from the time the appliance arrives. That’s a bit biased toward systems which require and indeed allow for relatively little tuning. Still, less tuning is better, all else being equal.
2. Pay very close attention to what it takes to install from delivery to being load-ready. Good advice. But again, for many customers ease of installation is a nice-to-have, not a must-have.
3. Require that all testing be done on-site with no external access allowed. A less biased way of saying that is that one should be very clear about what’s happening off-site. More seriously, I think there are times when testing at a third-party consultant’s site is the best way to go (e.g., when you don’t want your own data center overrun with tests).
4. Define the frequency of loads and data volumes not just for today but for future analytics needs:
– Is more data important to decision making?
– Has the volume been constrained by performance and cost?
I agree completely, as per Slide 28 and its immediate sequels from my upcoming TDWI presentation.
5. Choose the queries from hell and pull out the ones you are afraid to run. Yes, but everybody knows that, perhaps to the point that it’s over-emphasized in POCs.
6. Decide on which tools you need to have tested with the appliance and put them through their paces. Sound advice.
7. Test your backup and recovery process. Also good. But why stop there? How about testing how easy it is to add nodes, an area where some of the software-only vendors may be ahead of the appliance guys?
8. Monitor and participate in all activity to ensure fairness. Or hire consultants to do that — if you’re sure that they aren’t particularly vendor-biased.
9. Demand that testing be performed exactly as you plan to structure your own production environment. Nice ideal. Not always 100% practical. But to the extent you do deviate, be very wary.
I won’t repeat the last three points, because they aren’t really about POCs. Basically, they say “Make sure you’re not buying from a sleazy, fast-talking sales organization.”
Comments
One Response to “The Netezza guys propose a POC checklist”
Leave a Reply
[…] Tips for benchmarking an analytic DBMS […]