BI and quasi-DBMS
I’m on two overlapping posting kicks, namely “lessons from the past” and “stuff I keep saying so might as well also write down”. My recent piece on Oracle as the new IBM is an example of both themes. In this post, another example, I’d like to memorialize some points I keep making about business intelligence and other analytics. In particular:
- BI relies on strong data access capabilities. This is always true. Duh.
- Therefore, BI and other analytics vendors commonly reinvent the data management wheel. This trend ebbs and flows with technology cycles.
Similarly, BI has often been tied to data integration/ETL (Extract/Transform/Load) functionality.* But I won’t address that subject further at this time.
*In the Hadoop/Spark era, that’s even truer of other analytics than it is of BI.
My top historical examples include:
- The 1970s analytic fourth-generation languages (RAMIS, NOMAD, FOCUS, et al.) commonly combined reporting and data management.
- The best BI visualization technology of the 1980s, Executive Information Systems (EIS), was generally unsuccessful. The core reason was a lack of what we’d now call drilldown. Not coincidentally, EIS vendors — notably leader Comshare — didn’t do well at DBMS-like technology.
- Business Objects, one of the pioneers of the modern BI product category, rose in large part on the strength of its “semantic layer” technology. (If you don’t know what that is, you can imagine it as a kind of virtual data warehouse modest enough in its ambitions to actually be workable.)
- Cognos, the other pioneer of modern BI, depending on capabilities for which it needed a bundled MOLAP (Multidimensional OnLine Analytic Processing) engine.
- But Cognos later stopped needing that engine, which underscores my point about technology ebbing and flowing.
I’m not as familiar with the details for MicroStrategy, but I do know that it generates famously complex SQL so as to compensate for the inadequacies of some DBMS, which had the paradoxical effect of creating performance challenges for MicroStrategy used over more capable analytic DBMS, which in turn led at least Teradata to do special work to optimize MicroStrategy processing. Again, ebbs and flows.
More recent examples of serious DBMS-like processing in BI offerings may be found in QlikView, Zoomdata, Platfora, ClearStory, Metamarkets and others. That some of those are SaaS (Software as a Service) doesn’t undermine the general point, because in each case they have significant data processing technology that lies strictly between the visualization and data store layers.
Related link
- Context for this post may be found in my piece on The two sides of BI. (August, 2013)
Comments
5 Responses to “BI and quasi-DBMS”
Leave a Reply
For me this story underscore two things:
– database engines are not good enough so descent BI can be built on top of them. Frequently BI vendors are forced to build their own “query layer” which do part of the job and the rest delegate to the underlying DBMS.
– BI suite is better way to commercialize good query capability, probably for two reasons:
– It can be sold to management instead of engineers.
– Unless you expose free SQL a lot of engine problems can be hidden behind UI, which just will not generate “inconvenient” queries.
I am the opposite. Db engines are typically good enough but that semantic definitions (as defined logically into physical objects) usually aren’t for various reasons (time, experience, constant change in the app stack, etc.). Hard to keep as workable model going with ANY technology with that sort of thing going. Hence the push to more tech that is highly flexable but also agnostic. Self-defeating I think.
A useful perspective is to look at the dynamic between the spectrums:
a – cost vs. time to market
b – data purity and certification vs. ungoverned
c – performance/scalability vs. not
A lot of BI gets sold as all-in-one solutions, so that pushes towards no admin, and thus no dbms. This is starter edition sales ploy (to business in a hurry, like traders or sales), and as you say – over time this market goes away and products drop that. You can *always* point to new BI products that act like a dbms, and the survivors over time will get coopted and no longer need that capability.
There is a lot of dynamic now in the cleansed:uncleansed data spectrum. Either you’re interested in data fast or you want semantically sound data that foots to what the rest of the company reports. (Semantics are hard, not technically – but because they apply agreement across the business, and over time these lose sponsorship and *politically* blow up.)
Hadoop and such (Spark, D3, AWS, etc.) have blown up the cost structure of a lot of these, as well as the performance.
MicroStrategy and Business Objects both build cubes to cache data. BO called them microcubes. MicroStrategy calls them Intelligent Cubes and says they are “cache on steroids”. Cognos, which has never been the performance leader, commonly uses TM1 cubes as a data source to speed things up.
Cognos left its PowerPlay cube technology stranded, but then bought three companies — Frango, Adaytum and Applix — in fairly quick succession, so they never really did without cubes.
The real innovation is that Business Objects and Microstrategy have the ability to load these cubes on demand.
The companies Frango, Adaytum and Applix each had their own cube technology, as I failed to write.