SANs vs. DAS in MPP data warehousing
Generally speaking:
- SANs (Storage Area Networks) are pulling ahead of DAS (Direct Attached Storage).
- Much of the growth in storage is due to data warehousing.
- MPP (Massively Parallel Processing) is pulling ahead of SMP (Symmetric MultiProcessing) for high-end data warehousing.
- MPP architectures are commonly shared-nothing.
- Shared-nothing entails DAS.
But if you think about it, those facts don’t exactly add up.
The freshest take I have on the subject right comes from Vertica, with whom I met on Wednesday. It turns out that while Vertica initially thought DAS would be the only viable way to go, quite a few Vertica customers actually run SANs. A big Vertica installation these days is 10 nodes and 10-20 TB of user data, while a small one might be 1/10 that size. Within that range, SANs do just fine, as long as they have sufficient bandwidth, which commonly equates to 4 gigabit HBAs (Host Bus Adapters). One point Vertica noted is that SANs commonly have lots of cache and 15K RPM disks, which sounds higher-end than the storage hardware I usually hear about in DAS configurations.
Also interesting is CEO Jeff Vogel’s view over at Calpont. Jeff’s roots are in storage, and in response to my recent blog post mentioning storage issues he sent over a note I got permission to publish here. It says:
The storage blog underscores the increasing importance of efficient storage capacity and the impact that mixed workloads have on DW performance. It’s also front and center on the single biggest issue facing the growth prospects of the DW industry – Concurrency/User scalability. Storage efficiency and mixed workload variation have implications to cost . The single biggest cost issue in the data center is storage. It’s also a huge management cost issue. Energy consumption has only made the issue worse. Notwithstanding the issues associated with data access and availability, the proliferation of DW storage throughout the enterprise will bring business users and storage administrators together around a common storage services strategy. What emerges is a DW infrastructure strategy that incorporates storage management disciplines that have existed for years as part of a larger effort centered on Information Lifecycle Management (ILM) principles. The true benefits of ILM were only possible after companies went through the painful process of consolidating and centralizing their storage. Consolidation was made possible with the advent of the storage area network (SAN). There’s a reason why an overwhelming majority of all raw storage shipped into the enterprise is connected to a SAN.
From my perspective, the storage blog raises some important lessons for DW vendors and offers some interesting parallels. SANs made it possible for Unix and midrange OS servers to become part of the consolidated storage effort. As a result, we saw spectacular growth in applications attached to those servers. As a consequence, DAS retreated and is less than 20% of the market for warehouse and analytics. According to IDC over 50% of warehouse and analytics application storage in 2007 was midrange and is expected to top 60% within 5 years. The ‘Servers’ parallel here is Users. For DW applications to take off in a material way, we’ll need to solve the User scalability issues associated with cost, access and workload flexibility. Back to the “storage” future.
While the new DWA players have done a good job of addressing DBMS performance, scalability and costs, they don’t go far enough. Asset utilization, infrastructure flexibility and storage policies that govern data are fundamental concerns that DBMS players will need to contemplate at the architecture level. For example, while compression should be part of any solution, it’s only one storage cost dimension of a multi-faceted puzzle to asset utilization and, conversely, doesn’t fully address the broader issues mentioned in the blog. Up to now, these issues haven’t slowed DW sales, but you can hear the train in the distance.
We’re down a path on a crucial subject and we’ll be hearing a lot more from the storage players as to how the benefits of storage management will play out with DBMS solutions. As an industry, we’ll need to move beyond quid-pro-quo reference architectures and begin to think about how database behavior can help storage vendors lower costs, increase asset utilization, improve flexibility and in return, bring more Users into the game. A rising tide lifts all boats.
Comments
24 Responses to “SANs vs. DAS in MPP data warehousing”
Leave a Reply
On SANs:
Facts:
– Sequential IO is the core of what data warehousing requires
– Data Warehousing workloads commonly do not benefit from IO cache because too much data is scanned to fit in cache
– 4 gigabit FC-AL interface runs at a maximum 400 MB/s
– the latest 8 gigabit FC-AL interface runs at a maximum 800 MB/s
– SAN management nodes limit sustained bandwidth by routing data through their CPUs. Commonly this is in the range of 1,000 MB/s to 3,000 MB/s for the entire SAN, regardless of how many disks are ‘inside’
– A SAN management node, disks, SAN switch and softare typically cost in the neighborhood of $500,000 – $2,000,000
– A Dell 2950 server with two MD1000 DAS arrays sustain a sequential IO performance of approximately 2,000 MB/s, now typical of DAS configurations using SAS – total cost $30,000 list
Derived facts:
– The price performance of DAS is commonly >20 times better than SAN for sequential IO
– The primary advantages of SAN are storage management and the ability to virtualize storage
– SAN performance is designed for OLTP, random IO and caching
Conclusions:
– Data warehouses need dedicated disk resources and SANs aren’t designed for this use-case where oversubscription and virtualization aren’t helpful
– Provided that the database can deliver storage management, DAS is a much better fit for the workload
When we add compression to the equation, it can mitigate some of the harmful impact of storage virtualization by reducing pressure on the IO bandwidth requirements, but this comes at the expense of CPU cycles spent compressing and decompressing.
A hybrid strategy may make sense for circumstances where customers have deployed SAN already and need to justify their expenditures, but DAS will always provide better price performance and overall TCO.
From a DBMS product perspective some of the performance advantages noted are indeed obvious, but the inescapable facts when looking at from an enterprise C-level perspective are:
– A majority of the over 6,500 petabytes of DW/Analytics storage in 2011 will need to be tied into a central IT management strategy. And that’s up from over one million terabytes in 2007. Continued tradeoffs in performance, in exchange for the enterprise benefits of centralized storage management, will be likely. The magnitude of infrastructure asset costs and data compliance issues are too great have decentralized.
– To stay competitive, enterprises will enable larger populations of internal Users and external partners to get access to the data. Limiting access to the privileged few is old mainframe era thinking that desktop computing changed long ago. Highly variable concurrency loads and changing query dynamics will create workload flexibility issues that will put enormous pressure on asset utilization rates. Expect customers to more frequently reconfigure or add to those initial DW deployments as performance requirements fluctuate with changing users and data usage patterns.
– When it comes to storage, the most trusted players in the data center are the storage vendors. They’ve spent enormous sums of money and brand reputation on centralized storage management strategies. According to Gartner, storage vendors sell, on average, over $7 of software for every $1 of raw storage. Expect this trend to influence future DBMS strategies.
– The real issue for large volume User markets is the cost-per-User-access expense which includes everything from hardware to software to management to services of the entire solution. Again, there’s a reason why the overwhelming majority of storage shipped into the enterprise is connected to a SAN. The benefit of centralizing storage drives down the cost as more users gain access to the data.
The IDC 2007 numbers are already cast in stone. DW SAN terabytes outsold DAS by a factor of 3.5x and the gap is widening through 2011. That doesn’t mean that there isn’t a market for DAS. However, based on the facts, it would appear that customers are trading pure performance for the benefits that centralized storage management has to offer. The implications to future DBMS architecture are already here.
Jeff
Since SANs can read from disk so much faster than they can push data out, there’s an opportunity for SANs to reduce the data volume by checking a flag, applying a where clause, or implementing a distributed framework like MapReduce.
This requires a close relationship with a DW vendor and careful performance monitoring, just the kind of exclusive and high-margin services SAN vendors already love to sell.
@Jay,
Your conclusions are right on target, which is to say the analysis leads one to move the analysis to the data instead of the data to the analysis. Ironically this is the value proposition behind a lot of the DWA vendors, especially Netezza. In my experience EMC and the other storage vendors have looked closely into providing data warehouse appliances (something like smarter storage), but that would invariably create channel conflict with some of their biggest partners, the RDBMS vendors themselves.
I’d venture to guess that some enterprising software company will come up with a DAS MPP data warehouse, that allows enterprise storage management and they will reverse the trend towards SAN.
Hi Jay,
Predicate pushdown isn’t very interesting IMO in that scalable data warehousing involves a lot more than increasing storage bandwidth by the small factor that predicate pushdown typically provides.
What is really needed is what Derek describes, which is pushing the analysis to the data and not v.v. To do this requires the pushdown of the complete query execution stack, which is what proper MPP databases do.
Note that Oracle is about to unveil a secret project that uses HP DL185 servers as storage devices with some predicate pushdowns to implement a data warehouse “appliance”. In the end, they are still left with lethal performance problems involving joins and aggregations that aren’t solved by their predicate pushdown approach. The most they do with this approach is remove some of the bottleneck associated with EMC storage, the most typical deployment partner for Oracle data warehouses.
Given that Oracle is only solving a sliver of their scaling problem with this new “HP appliance”, I’d guess that most Oracle customers are going to be reluctant to jump on this new form of DAS with storage management by Oracle. It also represents yet another set of HW vendor relationships that Oracle will burn (bye bye EMC, HDS + Oracle)…
– Luke
I’m pleased to have found this discussion, since it’s a topic that I and my company (EMC) think about a lot.
Lots of interesting perspectives here, not all of them 100% spot on IMHO.
As far as the price/performance discussion by one commenter, the underlying point is correct (e.g. DAS is pretty fast), but the rest of the metrics (costs, controller performance limits, etc.) are somewhat off.
Yes, sequential read I/O is important, but — based on practical experience, don’t forget random read I/O (when your query isn’t optimized), random write I/O (when your DW is being updated), and sequential write I/O (e.g. if you ever have to recover the DW).
Unless those things aren’t important, of course …
Which brings up a key point in the DAS vs. SAN debate for DW: backing up and recovering your DW. If it’s important (and it’s big) this may end up tipping the scales for you in favor of SAN.
DAS approaches offer primitive-at-best data protection as compared to a modern array. Many customers back up the DW directly from the array to minimize downtime. And, if the DW is *really* important, you’ll be interested in remote replication.
There are also some use cases where it’s pretty clear SAN is faster than DAS, period. But it’s an interesting exercise exploring the different corners of this discussion.
The “appliance” discussion is most interesting. Everyone likes the idea of the perfect DW appliance, but — in reality — that image of perfection appears quite different for everyone, largely defeating the purpose.
In larger shops, there are well-defined server, network and storage strategies, and there is strong incentive to have DW solutions conform to existing standards. IT folks don’t like it when a vendor in one category tries to dictate approaches in other categories.
For that reason, and several others, I remain highly skeptical of the new Oracle/HP DW “appliance”. As a company, we’re not taking it too seriously.
The HP hardware is quite impressive on paper, but I can’t see too many serious Oracle users getting too excited, as most people realize the problem is an Oracle architectural issue that is only modestly amenable to faster hardware.
Oracle users have had access to the industry’s fastest hardware (storage and server) for decades. Won’t change the game much, I’d offer.
Regarding the discussion of pushing logic directly to the array, we’ve looked at it on and off over the last 10 years, and have never convinced ourselves it’d be worth the trouble to us — and our customers. But hope springs eternal.
There’s probably 3 or 4 good blog posts here, so I’ll save my material for when I get around to coding it up 🙂
Hope to be part of this discussion going forward.
Curt’s original post states that “shared nothing entail DAS.” This is a totally incorrect generalization. DATAllegro attaches mini-SANs (CX3) to each compute node (see http://tinyurl.com/4dhqra )and IBM would love to sell you a DB2 deployment attached to a TotalStorage SAN array.
…and…
Chuck Hollis wrote:
“…as most people realize the problem is an Oracle architectural issue that is only modestly amenable to faster hardware.”
Thanks for throwing Oracle under the bus, Chuck. Are you insinuating that Oracle therefore is not amenable to even faster monolithic SAN arrays?
Also, I can understand why EMC only got as far as “looking” at pushing database logic directly to the array over the last 10 years. It would be impossible for EMC to implement database offload processing unilaterally.
Even if high end SAN arrays did support offload processing there would still not be enough head bandwidth to scan all the disks at full disk bandwidth to start with (see http://tinyurl.com/67s8nw ). And, not all queries benefit from offload processing so you’d be right back to the same old head-saturation problems for those queries.
[…] 6.3 – SANs vs. DAS in MPP data warehousing Sat, Sep 6, 2008 5:45 AM Generally speaking: SANs (Storage Area Networks) are pulling ahead of DAS (Direct Attached Storage. Much of the growth in storage is due to data warehousing. MPP (Massively Parallel Processing) is pulling ahead of SMP (Symmetric MultiProcessing) for high-end data warehousing. MPP architectures are commonly shared-nothing. Shared-nothing entails DAS. But if you think about it, those facts don’ t exactly add up. The […] […]
Kevin,
If something is shared, then the set-up isn’t “shared-nothing.” QED
That said, I’m as guilty as anybody of referring to DATAllegro’s shared-not-very-much architecture as “shared nothing”.
CAM
Here’s what I suspect is going on. The arguments for not sharing RAM and for not sharing disk are quite different. They’ve tended to be conflated, for historical reasons, in that the most archetypical vendors and products (Oracle, Teradata, Netezza, et al.) tended to be either ones that share both of those resource or ones that share neither of them.
Not sharing RAM saves you from all sorts of thread synchronization issues. Unless there’s something about multi-core evolution I’m not understanding, the case for not sharing RAM seems solid for the foreseeable future.
Not sharing disk, however, stems largely from bandwidth concerns in current SAN configurations and/or manufacturing cost concerns in custom appliance designs. Those arguments are lot less robust or future-proofed than the ones against sharing RAM, as is evidenced by the number of unshared-RAM MPP data warehouse DBMS providers who DO support SANs in theory and practice alike.
Sound reasonable?
Thanks for the great discussion,
CAM
The sequential I/O issue, like most of the others here, is not trivial. If you’re mainly doing queries that access a lot of data, it’s possible for a large fraction of your query I/O to be sequential. If you’re doing pinpoint data lookup in some operational BI type of scenario, however, then you’re apt to be pretty random.
Even for analytical queries, the more complex your partitioning and/or schema, the harder it is to be purely sequential. And by the way, Luke has been giving me an earful recently about the breadth of Greenplum’s schema agnosticism and support for various index types.
As for updates — in many warehouse scenarios, even low-latency updates can be a lot more sequential than Chuck suggests. MVCC approaches lend themselves to sequential updating. Or if you’re on a once-every-few-minutes microbatch updating schedule, the whole thing is apt to be either pretty sequential or else so low-volume as not to be a major performance concern.
CAM
[…] visible Information Week blog post. After the great recent (and still ongoing!) discussion in the SAN vs. DAS comment thread, I’d like to throw some questions out for discussion, […]
Chuck,
Good to see you taking an interest in the DBMS subject of SANs. As to your comment regarding “pushing logic to the array”, EMC should be careful not to think about this in one dimensional terms of DBMS logic and it’s clearly not just about performance benefits. Given EMC’s early market leadership in creating SANs (Mike Ruettgers comes to mind), you wouldn’t want to be caught sleeping at the wheel of a multi-billions dollar market where storage management is concerned (-: Sorry for being somewhat elusive, but confidentiality is important.
[…] SANs vs. DAS in MPP data warehousing […]
Day 4 – Grumpy Old Man…
Well, I was starting to worry that I was completely out of step with the blogosphere, because finally a keynote presentation worth writing about and no-one seems keen, but I’ve only just noticed this quote in Pete Scott’s blog posting."For str…
[…] that follows this optimal storage-facing parallelization strategy. Actually, opinions differ as to whether rigid coupling of processors to specific disks is actually necessary. But after supporting one extreme (the disk part of shared-everything), Oracle with Exadata has […]
[…] SANs vs. DAS in MPP data warehousing […]
There is a good Cost Comparison at the site link above.
The most important thing IMHO is cost per MB/s of throughput. When you factor in how much it’ll cost you, SAN doesn’t get a look-in. I currently work on a system with total sequential throughput of 3 GigaBYTES per second which cost around USD 35,000. We costed out the same throughput on SANs – it would have worked out as 8 times more expensive.
You also can’t say that you’ll use enterprise infrastructure to deliver the performance without compromising performance service levels — after all, as soon as you use the Enterprise SAN, you’ve now got contention on fabric switches and any other shared components.
After about 2 years, when we revisit this topic, I can’t helping thinking that DAS (SAS/SATA) is still making more sense in DW environment.
SAN is highly virtualized and easier to expand & manage. But DW storage is rarely shared with other applications once it grows beyond 40TB. To provide the same 8 GigaBytes/Sec or 40 GigaBytes/Sec to the database server layer, SAN will have to cost a lot more than those database-aware / database-optimized DAS storage solution.
Exadata V2 X2-8 is a compelling sample. The main vendor finally acknowledges the value of DAS solution for DW. Its storage server is very similar to Netezza’s FPGA I/O boards. SAN still manages 512-byte sectors, while Exadata deals with 1-MegaBytes blocks, and Netezza handles 3-Megabytes blocks respectively as the smallest storage unit. Imagine how much overhead the SAN’s ASIC chipset have to assemble the 0.5KB blocks into 16KB or 32KB db blocks and return them via the fiber fabric. Bying saving the $$$ on expensive hardware RAID controller, fiber switch, fiber HBA… we can actually invest some of the savings to SSD cache, Infiniband, and big-block-optimized compression modules; therefore the overall I/O can be greatly improved for less. The only disadvantage is that such optimization has not no industry standard, so the storage hardware is tightly coupled with vendor’s database engine, the only way to share these storage is via application or network layer API.
As the gap between DW demands and SAN cost is growing wider, the software-based and even hardware-based storage server for database engine will gain more ground. Netezza broke the ground, Oracle finally followed; Teradata and IBM are still SAN-based, but I won’t be surprised if Teradata pushes AMP down to dedicated I/O hardware with DAS.
And then there’s the EMC/Greenplum approach, in which you put your primary copy of the data on DAS, and the mirror on the SAN.
Curt,
Any update from the SAN vendors EMC/NetApp/HP/IBM regarding specialized hardware/firmware for database?
You mentioned Greenplum, which is a recent acquisition. Even ORACLE praised Netezza’s innovation during yesterday’s EXADATA CUSTOMER FORUM. It will have better adoption if such storage server module is coming from SAN vendor. Is there any progress in your radar?
Regards.
Eric,
As you can see at the top of the blog, I haven’t been keeping up with my briefings for the past few weeks due to some personal issues.
[…] vehement, multi-party debate on SAN versus DAS […]