Mega-trends driving data warehousing and business intelligence
Philip Russom opines (emphasis mine):
What’s driving change in data warehousing (DW) and business intelligence (BI)? There are obvious scalability issues, due to burgeoning data, reports, and user communities. Plus, end-users need more real-time and on-demand BI. For many organizations, integrating existing systems into DW/BI is a higher priority than putting in new ones. And the “do more with less” economy demands more BI at lower costs. Hence, most drivers of change in BI and DW concern four Mega-Trends: size, speed, interoperability, and economics.
Depending on which universe of enterprises and vendors you’re looking at, Philip’s claim of “most” may be technically true. But from where I sit, Philip omitted two other crucial trends: new kinds of data and increased analytic sophistication.
A year ago, I divided data into three kinds:
- Human/tabular, which is what Philip’s comments seem to be focused on.
- Human/nontabular, e. g. what is best handled via text analytics.
- Machine-generated, such as web log or sensor data.
Most organizations on the planet could benefit from better understanding or exploiting their human-generated tabular data. But even so, many of the best opportunities to add analytic value come from capturing and analyzing fundamentally newer kinds of information.
I further would suggest that analytic sophistication is going up, for at least two reasons:
- New kinds of data call for or at least allow new kinds of analytics.
- Better price-performance (on bigger data sets) allows for more sophisticated analytic techniques.
Some of the best examples of these trends, especially the second one, may be found in what I recently called analytic profiling.
Comments
4 Responses to “Mega-trends driving data warehousing and business intelligence”
Leave a Reply
After having previously spent time working on products for nontabular and machine-generated data, I’ve got to say that I agree 100% with your statement regarding the value that organizations can derive from these kinds of data. I’ve worked with large organizations who are leveraging such data for new types of analysis (both automated and ad hoc) and the success has been quite startling. It really has provided those organizations with significant competitive differentiation within their fields. However, I wonder if this is a “trend”. For me, it still feels a little early in the adoption cycle to be a trend.
Conor,
That’s a fair challenge. I’d say that the list of industry sectors where there’s no need to doubt whether these are trends yet is reasonably impressive — the whole online sector, the whole defense/intelligence sector, the telecommunications sector, the pharma research sector, scientific research in general (the fairness of Mike Stonebraker’s “zero billion dollar market” label notwithstanding, soon financial trading as well.
In addition, they’re already trends wherever somebody is closely engaged with the internet, in sufficiently high volumes to justify the use of sophisticated analytic tools.
OUTSIDE of those sectors and perhaps a couple more I’m forgetting at the moment, I would agree text and sensor data have been a bit slow to take off.
I agree Curt. There’s a set of industries/verticals where nontabular data is of a high potential business value, and it’s with those organizations where the trend has been apparent for many years.
For example, I’ve worked with both a US city police department and a major European based international criminal investigation group who both have used nontabular data mining technologies integrated with a traditional data warehouse for advanced crime analysis – in production for 4 or 5 years now.
[…] Mega-trends driving data warehousing and business intelligence (dbms2.com) […]