False-positive alerts, non-collaborative BI, inaccurate metrics, and what to do about them
I’ve been hinting at some points for quite a long time, without really spelling them out in written form. So let’s fix that. I believe:
- “Push” alerting technology could be much more granular and useful, but is being held back by the problem of false positives.
- Metrics passed down from on high didn’t work too well in Stalin’s USSR, and haven’t improved sufficiently since.
- A large, necessary piece of the solution to both problems is a great engine for setting and modifying metrics definitions.
I shall explain.
False positives on alarm systems are a huge problem. Alarms were ignored (indeed, turned off) on the Deepwater Horizon due to too many being false. When I had a fire in my house, the smoke alarm wasn’t on because of its penchant for earsplitting false positives. And false positives held back the software category of intrusion detection systems until it was eventually subsumed by related kinds of security appliance.
Similarly, false positives are an under-appreciated problem that restrains the advance of business intelligence software. It’s no big inconvenience when, out of your 3 biggest problems this week, your dashboard calls attention to 17 of them. But suppose you want to make alerts more granular, take them up in quantity a couple of orders of magnitude, and have your smartphone pinged each time the alert condition is met. Whoa. Now those bogus warnings start to be more of a consideration.
Why would you want some kind of “push” alerting? Use case circumstances include but are not limited to anything that might herald:
- A machine malfunctioning or reaching capacity limits (many kinds of machine).
- A critical shortage (inventory or supply).
- A sudden market trend change (if you’re in the kind of consumer market where significant developments an occur in hours or less)
- An investment opportunity or threat, or a financial risk threshold being breached (if you’re an investor, if you buy or sell commodities, or if you’re a corporate treasurer).
- An attitude change on the part of a specific important customer (sales contact, service contact, news development, whatever).
- The advisability of rerouting or rescheduling something.
Why would these alerts ideally be granular and high-volume? Well, you might have lots of machines, lots of trucks, lots of customers, lots of potential investments, and so on. I could list many more examples, but these probably suffice to at least start the discussion.
I’m agnostic about what the UI would be. Sounds going off on your mobile phone? Email on your mobile device? An app more like a chat window? Something more like a mobile dashboard? Whatever it is, false positives would really screw up your work day.
Classical enterprise dashboards suffer from a similarly fundamental problem — dashboard technology is optimized for a screwed-up enterprise analytic methodology. That is:
- Somebody tells you what you’re supposed to care about.
- Somebody tells you how to measure what you’re supposed to care about.
- The simpler and more top-down all this is, the better it creates enterprise-wide unity of vision/focus/purpose/blahblahblah.
Yes, you can do drilldown or even data exploration, QlikView-style or otherwise. You can communicate around the BI tools, whether via general portal technology or something built-in. Vendors keep trying to make it easier for you to build your own reports, charts, and dashboard elements. But you can’t very easily define and track your own metrics, and I don’t see a lot of effort toward making that better.
Perhaps vendors don’t try to provide strong metrics management because doing so might lead to – gasp! — more than “one version of the truth.” If so, such thinking is regrettably misguided. Examples of formulas – i.e., metrics — that should and can not be cast in stone* include:
- Profitability of a customer
- Expected lifetime value of a customer
- Anything else which boils down to “importance of a customer”
- Profitability, value, or importance of a lead or prospect action
- Significance of a social media flare-up
- Cost of a schedule slip
- Risk of a portfolio
*At least as a general rule – we surely can think of some simple cases or exceptions where extreme cookie-cutterness is just fine.
It’s not that there shouldn’t be preferred or default metrics for these quantities. Rather, my point is that individuals should be allowed to:
- Question those metrics
- Consider and test alternatives to those metrics
- Make decisions based on those alternatives (as long as they also keep the results and implications of the approved/default versions of the metrics in mind)
Why? Because the person closest to the situation may be the one best able to assess it, and also because recognizing that fact is a really good idea from the standpoints of employee morale, engagement, and even development. (Or you might say – shiny buzzword! – employee “empowerment”.) In particular, when a boss and her subordinate disagree about the ideal way to calculate a metric, they should be able to track both versions side-by-side. When there’s a significant difference between the two, that could be a fine time for an out-of-the-norm decision.* If that ability were available, portal-style BI collaboration would almost automatically come into its own.
*What kind of decision that is of course depends on the specifics of the case. If only one of two competing metrics says you should bother giving extra care to a specific customer, the answer is easy – you give it. In other cases, an actual meeting or other conversation may be in order to decide what to do.
A metric is just a function. To offer BI users the flexibility I want, it should be very easy to use these functions as inputs into other, custom functions. I.e., I’m talking about something with the flexibility of a stripped down Excel, although the UI would be more like that for a rules engine. And if you think about it, that’s exactly the same functionality needed to personalize your alerts and weed out false positives.
One last point – what I’m calling for could greatly increase the BI performance burden, perhaps even by three orders of magnitude. Well, that’s what we have all this great new super price-performance analytic database technology for. And anyhow the throughput hit won’t come overnight. It will be interesting to see where the boundaries end up between DBMS and inside-the-BI-tool analytic processing. But overall, the analytic DBMS industry – and the hardware vendors backing them up – should be able to handle anything the BI vendors throw at them.
Related links
-
Gooddata may be a little more oriented toward what I’m talking about here than some bigger vendors are.
- Use cases for mobile business intelligence
Comments
10 Responses to “False-positive alerts, non-collaborative BI, inaccurate metrics, and what to do about them”
Leave a Reply
Good article. I’d suggest that the metrics must be dimensional and might be thought of as services, i.e. Can be subscribed to to create derived metrics. This metric engine / designer would require that the 3 bi tools: etl, query, report, are merged into a single tool as the usage of the metric might dictate that it needed to be precalculated during the equivalent of the etl run.
Justin,
I’d say the server side model is probably the BI tool knows how to do everything itself, but can also delegate to DBMS and/or ETL when that makes sense. This could mean a whole new level of cooperation between BI and DBMS technology.
Naturally, they’d be more prone to do this with DBMS vendors that:
A. They own
B. Have high market ahare
all else being equal – not that all else would be equal …
The best way to control false positives is to allow end-users to set the alert thresholds themselves. That’s a better adaptive system than any artificial intelligence-based system can deliver.
In terms of metrics, you need standard metrics so everyone knows what’s going on (e.g. customer profitability) and you don’t have a tower of Babel. But around the metrics (and even within them), power users should be able to explore the business activity behind the metrics to see what new information and insights they can glean.
So metrics are for casual users and running the business; explorative analytics is for power users to seek insights or clarification around the metrics. These are wholly compatible.
One thing you don’t want to do is create multiple versions of metrics for enterprise consumption…. chaos.
Wayne,
But it’s not necessarily as simple as a threshold. A compound expression/condition/test may be what’s needed to avert false positives.
As for preventing people from ever discussing alternate forms of metrics with each other — that’s way too Big Brotherish for my tastes …
The plurality of definitions makes me think of Chorus’ support for multiple marts. Is this sort of functionality part of what Greenplum is targeting? Each group getting to define its metrics differently in its own virtual mart and still access the official definition as well?
John,
I’m thinking of something much more granular than that. Chorus can be restricted to the model Wayne was suggesting — let your specialists do a better job of considering alternate possibilities. I want line managers to be allowed to think for themselves as well.
Monitoring…
Goals P1 Collect status and alerts from multiple source systems (tickers, data quality, automation,etc.) in one place and know whether a part of the system is broken sooner. Institute a better way to communicate such issues,……
[…] moved slowly in accuracy-intensive areas such as alerting or predictive […]
[…] neglected, on the other hand, are opportunities for flexible, personalized analytics. (Note: There’s a lot of discussion in that link.) The best-acknowledged example may be better […]
[…] neglected, on the other hand, are opportunities for flexible, personalized analytics. (Note: There’s a lot of discussion in that link.) The best-acknowledged example may be better […]