April 9, 2015
Which analytic technology problems are important to solve for whom?
I hear much discussion of shortfalls in analytic technology, especially from companies that want to fill in the gaps. But how much do these gaps actually matter? In many cases, that depends on what the analytic technology is being used for. So let’s think about some different kinds of analytic task, and where they each might most stress today’s available technology.
In separating out the task areas, I’ll focus first on the spectrum “To what extent is this supposed to produce novel insights?” and second on the dimension “To what extent is this supposed to be integrated into a production/operational system?” Issues of latency, algorithmic novelty, etc. can follow after those. In particular, let’s consider the tasks:
- Reporting for regulatory compliance (financial or otherwise). The purpose of this is to follow rules.
- This is non-innovative almost by design.
- Somebody probably originally issued the regulations for a reason, so the reports may be useful for monitoring purposes. Failing that, they probably are supported by the same infrastructure that also tries to do useful monitoring.
- Data governance is crucial. Submitting incorrect data to regulators can have dire consequences. That said, when we hear about poor governance of poly-structured data, I question whether that data is being used in the applications where strong governance is actually needed.
- Other routine, monitoring-oriented business intelligence. The purpose can be general monitoring or general communication. Sometimes the purpose is lost to history entirely. 🙂 This is generally lame, at least technically, unless interesting requirements are added.
- Displaying it on mobile devices makes it snazzier, and in some cases more convenient. Whoop-de-do.
- Usually what makes it interesting these days is the desire to actually explore the data and gain new insights. More on that below.
- BI for inherently non-tabular data is definitely an unsolved problem.
- Integration of BI with enterprise apps continues to be an interesting subject, but one I haven’t learned anything new about recently.
- All that said, this is an area for some of the most demanding classical data warehouse installations, specifically ones that are demanding along dimensions such as concurrency or schema complexity. (Recall that the most complicated data warehouses are often not the largest ones.) Data governance can be important here as well.
- Investigation by business analysts or line-of-business executives. Much of the action is here, not least because …
- … it’s something of a catch-all category.
- “Business analyst” is a flexible job description, and business analysts can have a variety of goals.
- Alleged line-of-business executives doing business-analyst work are commonly delegating it to fuller-time business analysts.
- These folks can probably manage departmental analytic RDBMS if they need to (that was one of Netezza’s early value propositions), but a Hadoop cluster stretches them. So easy deployment and administration stories — e.g. “Hadoop with less strain”/”Spark with less strain” — can have merit. This could be true even if there’s a separate team of data wranglers pre-processing data that the analysts will then work with.
- Further, when it comes to business intelligence:
- Tableau and its predecessors have set a high bar for quality of user interface.
- The non-tabular BI challenges are present in spades.
- ETL reduction/elimination (Extract/Transform/Load) is a major need.
- Predictive modeling by business analysts is problematic from beginning to end; much progress needs to be made here.
- … it’s something of a catch-all category.
- Investigation by data scientists. The “data scientist”/”business analyst” distinction is hardly precise. But for the purpose of this post, a business analyst may be presumed to excel at elementary mathematics — even stock analysts just use math at a high school level — and at using tabular databases, while data scientists (individuals or teams) have broader skill sets and address harder technical or mathematical problems.
- The technology for “data science” is generally on the newish side. Management and performance at scale are still improving.
- There’s a need and/or desire for more sophisticated analytic tools, in predictive modeling and graph.
- Rapid-response trouble-shooting. There are some folks — for example network operators — whose job includes monitoring things moment to moment and, when there’s a problem, reacting quickly.
- Splunk and/or Flume commonly suffice to collect that data, but of course that’s a moving target as data sources proliferate.
- I expect a lot of innovation relevant to the analytic side, in areas such as streaming, low-latency BI, event series analytics, and BI/predictive modeling integration.
- “Operationalization” of investigative results. This is a hot area, because doing something with insights — “insights” being a hot analytic buzzword these days — is more valuable than merely having them.
- This is where short-request kinds of data stores — NoSQL or otherwise — are often stressed, especially in the low-latency analytics they need to support.
- This is the big area for any kind of “closed loop” predictive modeling story, e.g. in experimentation.
- At least in theory, this is another big area for streaming.
And finally — across multiple kinds of user group and use case, there are some applications that will only be possible when sensors or other data sources improve.
Bottom line: Almost every interesting analytic technology problem is worth solving for some market, but please be careful about finding the right match.
Related links
- Where the innovation is (January, 2015)
- Various notes (November, 2014)
- “Freeing business analysts from IT” (August, 2014)
- Data integration as a business opportunity (July, 2014)
- Differentiation in BI usability (March, 2014)
- Analytic database distinctions (February, 2013)
- Juggling analytic databases (March, 2012)
- Applications of an analytic kind (February, 2012)
- Agile predictive analytics (November, 2011)
- Eight kinds of analytic database (July, 2011)
- Use cases for low-latency analytics (April, 2011)
- The three principle kinds of analytic business benefit (March, 2011)
Categories: Business intelligence, Data warehousing, Databricks, Spark and BDAS, Hadoop, Netezza, NoSQL, Predictive modeling and advanced analytics, Tableau Software
Subscribe to our complete feed!
Comments
One Response to “Which analytic technology problems are important to solve for whom?”
Leave a Reply
Curt,
I just want to say I really enjoy and appreciate everything you write and share. It’s fascinating and very helpful.
Sincerely,
Chris