The CEP guys are getting a bit chippy
In a thread responding to my post Independent CEP vendors continue to flounder, Paul Vincent wrote:
I’m not aware of anyone claiming “CEP is an alternative to a relational RDBMS” – except maybe as an application platform for processing events (where RDBMS could be seen as a square hole for an event round peg).
Huh?
Actually, it’s hard to think of an application for off-the-shelf CEP where the alternative technologies aren’t:
- Custom CEP
- Just write it to an RDBMS and query it
What’s more, except where super-low latency is needed, #2 is apt to be the primary alternative.
True, CEP-based data reduction is a nice-to-have. But in a world where data warehouses can be loaded at >1 gigabyte/second, it’s rarely a must-have. And guess what? For the most part, CEP vendors are retreating to the markets where their technology really is a must-have.
(Of course, my views on this could change if Aleri or Truviso persuades me that they’re successfully pursuing a significantly broader range of applications than now appears to be the case.)
Anyhow, it could have been worse — they might have lit into me the way they went after Philip Howard.
Comments
18 Responses to “The CEP guys are getting a bit chippy”
Leave a Reply
“Just write it to a RDBMS and then query it…”
Of course that approach is probably the first one that comes into mind the first time you face an event processing problem…but then, almost immediately you realize that you have no easy answer to the following question: when should I query the RDBMS? An event might happen at any time so…
The natural next step is to quickly think of a workaround based on database triggers…But this is not where a RDBMS really shines…A RDBMS really shines at queries…So, wouldn’t it be nice if I had an specialized engine for triggers rather than using an RDBMS to do that? Yes, that would probably make sense, ok: should I write such an engine myself (custom CEP) or is there one out there that I can buy and use out-of-the-box?(CEP engines). But,mmhmhm…An engine of the kind seems not easy to write and I want to focus on solving my business problem and not on writing middleware, therefore I will rather buy one CEP engine…But then, how do I manage to get all my relevant data pass thorugh the engine? Would not that create a bottleneck in the system? Woldn’t it be too costly, even impossible, to modify my existing applications to have them send the relevant data to the engine and with the format(s) that the engine expects/supports? After all my actual RDBMS already receives all relevant data, so after all it might make sense to use triggers…
Conclusion: there’s no satifactory solution to the problem with actual technology and the decision boils down to: what is cheaper? To buy a CEP engine, modify my applications to integrate with it, develop my event processing logic with the engine’s event processing proprietary language (that locks me into the CEP engine’s vendor) or to use triggers in my database? Most of the time the later, and is that more cheap and easy compared to the other approach that other important factors such as scalability, performance, expressive power, etc. are not able to counterweight.
That’s why most CEP engine vendors today are focusing in providing adapters out of the box to make it much easier to integrate with their engines and that’s why some database vendors are focusing in increasing the performance, scalability and expressive power of their triggers.
But, in the end, neither of the two approaches above are good enough: what we need are event processing platforms, that is, distributed runtime engines that are able to host milions of concurrent, long running stateful event processing agents communicating on top of a content-based pub/sub system. But this exists today only in the head of some reserachers and there’s a long way from there until production…But we will get there…eventually.
Regards,
PatternStorm
PatternStorm,
I agree with most or all of that.
It just depends on which application categories one is talking about.
Best,
CAM
I would guess that the most common alternative to a CEP engine is #1.
People who succeed with #2 shouldn’t and probably don’t want to bother with either #1 or a CEP engine. Because processing through a database is easier for many reasons.
CEP vendors don’t get most of their sales from people succeeding with a solution like #2. They don’t go into many sales calls up against a solution like that.
They sell to people that have tried both #1 and #2 solutions, have chosen #1 in the past, and are now looking for something that can do better in the future.
Performance is possibly the main reason for choosing solution #1. But super low latency is definitely not the only performance requirement. There are many types of application that run into issues (from hardware usage to application performance) with even moderate information flow and a traditional RDBMS/cache combination. They adopt a #1-style solution on top of the DB or as a replacement to it. These are the applications that CEP vendors usually target with their sales and marketing.
And that is possibly where Paul’s comment came from.
Also, there are many, many applications that process their events through a rules engine or even just loop over record sets (or constantly query for the latest records) and get relatively little benefit from their DBMS.
In cases like this, it is true that the database is just a sort of queue in between messages and processing. Sometimes you want that type of permanent and queryable queue, but sometimes you don’t.
Enter Paul and Tibco, EMS and a CEP engine. Again, not exactly competing against proper database usage, but providing an alternative to the many cases where the database isn’t adding value.
Hans,
Fair enough. But Coral8’s growth story before it threw in the towel was, in effect, “better BI”. The same seems true of Truviso, although the plan there does seem to change quickly. In general, a lot of CEP growth stories that have been suggested are in the vein of “Now use your data so fast that it is NO LONGER appropriate to go through a DBMS.”
CAM
Hmm, I think I was thrown off by your previous CEP post. You discussed the fact that some vendors focus on sectors that either have many non-BI CEP use cases, or genuinely have performance concerns with their data feeds. Those are exactly the vendors that are *not* playing the failing game of unnecessarily replacing DBMS’s. But to you, that seems to be floundering. Or maybe I am getting the wrong message from that post.
On another note, it’s worth mentioning that there are some BI use cases that are genuinely a pain to do in the database, but are dead simple with some CEP products. For example, processing moving windows of data (trend elimination, prediction, windowed aggregation). Or using stateful filters (Kalman, etc) or pattern matching. And there are more.
Of course, one can question the size of the market for BI such as this, or point out that database vendors will integrate these capabilities into their products soon enough. But at the moment, there are certainly BI use cases that benefit from a CEP product without being high performance or wanting to eliminate the database.
Hi Curt: thanks for the comment 😉 but I note you didn’t quote any independent CEP vendor making claims they were a substitute for a generic DBMS…
Perhaps what you are saying is that “RDBMS can store events (as DB inserts) from which I can run aggregate queries, triggers etc – therefore CEP technology is competing with RDBMS”.
An interesting counterpoint is that RDBMS vendors are themselves investing in CEP – presumably so they can compete in the CEP market, not vice versa.
FYI for some notes on a large RDBMS vendor’s views on the differences between “CEP” and “RDBMS” check out Note [2] in http://tibcoblogs.com/cep/2008/07/03/debs081-active-dbs-contribution-snoop/
Thanks for the lively discussion!
Cheers
In many cases, the reason for preferring “custom CEP” on just query databases is the fact that SQL is inadequate to express the functions required by CEP (i.e. it is very complicated and unnatural to use SQL, including SQL variations).
See: http://epthinking.blogspot.com/2009/03/on-event-processing-as-part-of-dbms.html
cheers,
Opher
Hans et al.,
I think CEP is one of those markets that can be usefully viewed through the old Geoffrey Moore lenses — early adoption, chasm, “bowling alley”, “tornado”, “Main Street”, etc. The bowling alley stage appeared to be getting started — but additional pins, while wobbling, aren’t so far as I can tell falling over in droves.
That’s part of what I mean by “floundering”.
The other part is that it seems, if and when CEP regains momentum outside its niche markets, the beneficiaries will be large, diversified, usually DBMS-selling software companies. I used the word “independent” in the title of the post for a reason.
Hi Curt,
You are exactly right and I am really pleased to read your posts. In capital markets, where there does not appear to have been a generic event processsing platform available, the term “CEP” has been adopted because the platforms marketing themselves as “CEP” have made progress. The adoption is not related to “CEP” per se but software marketing their “wares” as “CEP”.
Outside of processing marketdata, there have been folks willing to test these platforms, and a “few bowling pins” have wobbled (as you say); but a few wobbling bowling pins are not “main street” by any leap in imagination.
Recently we evaluated so called “CEP engines” to monitor events on a high performance web site with a real eye for using a so called “CEP product”. After spending much time and effort, we picked Zabbix for processing network-centric events. Zabbix works great and is free, open source, extensible, and does all the basic time window, rule-based processing the site needed. The “CEP offerings” took too much work because there have been optimized for processing market data, not network “event soup” data.
The piece we are current missing is future event prediction (outages, server load problems, etc.) and none of the self-described CEP engines are close to this stage of event processing (at this stage of the game). We are considering integrating the R statistical package; but have not made a decision yet.
Cheers and Best Regards, Tim
PS: I was a bit surprised to read some negative comments about your posts over at David Luckham’s forum. Hang in there!
Opher,
I question how many application areas there are where known patterns have enough complexity that standard database languages (SQL, MDX, text search., etc.) are inappropriate to express them. (Assuming some standard temporal extensions if needed.)
FWIW, many people wonder: how big is the market for event-driven BI? Will businesses outside of certain core markets ever want to trade the stability and simplicity of databases and batch operations for an event-driven BI model?
For that matter, will the event-driven computing market ever grow beyond the industries that use it now? Is the entire event-driven computing market big enough to support a large company in the long term?
It’s not just CEP that faces these questions.
Were I a potential CEP investor, I would wonder how you came to believe that big players will dominate this field. Historically speaking, niche industries, especially deep niches, have been exactly where medium sized technology players can survive and produce good returns on a relatively modest investment.
I’m not saying you’re wrong. Just pointing out that there are several valid sides to the story.
Hans,
For years, the BI and data warehousing industry have been responding to customer interest in getting ever-faster access to data. But so far, the speed of enhancement in relational data warehousing technology has been enough to meet this need.
I’m of the opinion customers shouldn’t be satisfied, and should demand a revolution in BI. But there are a lot of needed revolutions in BI, and my preferred one in better alerting and metrics management is by no means necessarily one that will soon occur.
As for my views on the independent companies — there are few enough that matter that we can just look at them one by one. Aleri — apparently focused on financial services (but I have a briefing scheduled with them soon for more detail). Coral8 — sold to Aleri. StreamBase — retreated into a couple of niches. And so on.
FWIW 🙂
Event processing and complex event processing are not dependent on “event driven architecture”. Events can be “fired and forgotten” or they can be requested by the CEP/EP server.
In other words, there is nothing about CEP/EP that makes it dependent on EDA. Event processing is as “old as the hills”, and there is nothing new from a technical perspective in any of the products on the market that use CEP as a marketing term.
What is new?
Please tell us !!
Curt, as a final thought: I am skeptical about short-term, broad demand for faster alerting or event driven BI. If there were broad demand for such a thing, we would see more hand-coded solutions filling the need now. Coding event driven BI may be harder than coding a report, but it’s not so hard as to be unattainable by a medium sized company. But we (or I) don’t see very many hand coded, event driven BI solutions out there.
In certain fields, though, we do see a lot of hand coded, event driven BI. I question why, if there is really demand beyond those fields, we do not see more custom solutions cropping up to fill the need now.
In the longer term, I imagine that as various products make developing event-driven software easier, demand will arise in areas where we don’t see it now. But that is long term and certainly not something on which a vendor can make money today.
Tim, if you don’t see anything new in CEP/EP, then I am surprised to find you still talking about it. One would think that a better use of your time would be to find something that you think is new, and talk about that. Perhaps in monitoring your Apache logs and learning about statistics and R, you will find this new thing that you seek.
[…] skeptical remarks on the Aleri/Coral8 merger generated some pushback. Today I actually got around to talking with John Morell, who was marketing chief at Coral8 and has […]
Hi Hans,
Perhaps you can share with us exactly what you are doing at Credit Suisse.
My experience has been that folks in financial services are all excited about a stream processing rules engine.
What do you actually “detect” in real time?
Thanks!
OBTW @Hans,
Monitoring the Apache2 log file in the current event processing applications we are working on provides only 3 precursor events of a total set of nearly 350 event and precursor events (none of which are a part of the same event stream).
Our team has not decided (yet) to go with R, BTW. We don’t need to “learn statistics” as much as we need to find the right architecture, especially since none of the current products calling themselves “CEP engines” do the trick.
Actually, I find a lot new in what we have been doing, because I am not focused solely on streaming rule-based processing and systems (as you seem to be). Processing event streams with rules is an area which I was starting working on over 20 years ago in the event processing (network and security) space.
It is, however, encouraging to read your posts recently as you “catch up” and learn about detection theory and the limitations of rule-based stream processing to detecting complex (and changing) situations and event.
I am encouraged by how much your have learned about event processing over the past two to three years. Keep up the good work!
Yours sincerely, Tim