The games of Watson
IBM excels at game technology, most famously in Deep Blue (chess) and Watson (Jeopardy!). But except at the chip level — PowerPC — IBM hasn’t accomplished much at game/real world crossover. And so I suspect the Watson hype is far overblown.
I believe that for two main reasons. First, whenever IBM talks about big initiatives like Watson, it winds up bundling a bunch of dissimilar things together and claiming they’re a seamless whole. Second, some core Watson claims are eerily similar to artificial intelligence (AI) over-hype three or more decades past. For example, the leukemia treatment advisor that is being hopefully built in Watson now sounds a lot like MYCIN from the early 1970s, and the idea of collecting a lot of tidbits of information sounds a lot like the Cyc project. And by the way:
- MYCIN led to E-MYCIN, which led to the company Teknowledge, which raised a lot of money* but now has almost faded from memory.
- Cyc is connected to the computer science community’s standard unit of bogosity.
*Much of it, I’m ashamed to say, with my help, back in my stock analyst days.
AI is something of an umbrella category, often just meaning “Computerized stuff that we don’t know how to do yet”, or ” … only recently figured out how to do.” Automated decision-making is an aspect of AI, for example, but so also is natural language recognition. It used to be believed that most AI should be approached in the same way:
- Come up with a clever way to represent knowledge.
- Match the actual situation against the knowledge.
- Produce a smart result.
But that template unfortunately proved disappointing time after time. The problem was typically that not enough knowledge could in practice be represented, and thus well-informed automated decisions could not be made. In particular, there was a “first step fallacy,” in which a demo system would solve a “toy problem”, but robust real-life systems never emerged.
Of course, there are exceptions to this general rule of disappointment; for example, Teknowledge and its fellow over-hyped expert system technology vendors of the 1980s (Intellicorp, Inference, et al.) did get a few solid production references. But the ones I remember best (e.g. American Express credit, United Airlines seat pricing, some equipment maintenance scheduling) were often for use cases that we’d now address in more straightforwardly mathematical ways.
Watson is generally promoted as helping with decision-making, but that message has to be scrutinized carefully. So far as I’ve been able to guess, the true core technology of IBM Watson is extracting knowledge from text — or primarily from text — and representing it in some way that is reasonably useful in answering natural language queries. The hope would then be to eventually achieve a rich enough knowledge base to support the Star Trek computer. But automated decision-making doesn’t just require knowledge; it also requires decision-making rules. And if Watson is significantly ahead of the 1980s decisioning state of the art (Rete, backward chaining, etc.), I’m not aware of how.
So if Watson is going to accomplish anything soon, it will probably be in areas where serious decision-making chops aren’t needed. Indeed, the application areas that I’ve seen mentioned for the past or near term are mainly:
- Playing Jeopardy! That’s pretty simple from a decision-making standpoint.
- Advising on treatments for a specific disease (not actually built yet). As noted above, that’s 1970s-level decisioning.
- Knowledge extraction from medical research articles. That has very little to do with decisioning, and incidentally sounds a lot like what SPSS (before it was acquired by IBM) and Temis were already doing years ago.
- Natural-language customer interaction. That may not involve any decisioning at all.
Returning to the point that Watson’s core technology is probably natural language, it seems fair to say that IBM these days is probably better at the text mining side than at speech understanding. Evidence I’m thinking of includes:
- That seems to be what IBM itself is saying on its speech recognition page.
- I also recall IBM’s natural language recognition projects being regarded as not going well in the late 1990s. (Project Penelope, I believe, although I can’t confirm that via googling.)
- IBM’s LanguageWare sounded more oriented to text mining in 2008.
- IBM bought SPSS, which had decent text mining technology.
And while this is too old to really count as evidence, IBM had a famously unsuccessful language recognition deal with Artificial Intelligence Corporation way back in 1983-4.*
*Yeah, I helped raise money for AICorp too, and also for Symbolics. As you might imagine, my investment banking trophies do not have pride of place on my desk.
One last observation — text mining has a very mixed track record. Watson will have to go far beyond predecessor text technologies to become nearly the big deal IBM is suggesting it will be.
Related links
- Some language recognition humor
- Jack Vaughan reports on Watson
Comments
3 Responses to “The games of Watson”
Leave a Reply
Winning Jeopardy with a computer is not that easy given many of the questions are cryptic. I put the Jeopardy questions into a search engine and had about a 55% success rate at getting the answer back in the first couple listings. I thought in particular Watson blitzing the category where you had to name the medical condition was remarkable and it’s easy to see how it could be used to double check human decisions in health care.
Watson does not have a rules engine or decision trees, it works by mimicking a human in understanding a question and a probabilistic engine to rank answers. A rules based approach means you need a set of rules for any situation and you cannot handle unexpected exceptions, they have built Watson to be adaptable.
IBM have turned Watson into a new software brand, put a $1b team around it, committed a third of all research to cognitive computing and estimated annual revenues of $10b within a few years. You’ve asked whether it goes far beyond predecessors, IBM have already bet big that this is the case.
Interesting link! You showed that while Watson’s advantage over plain Google search is small in most cases, it’s large when there’s wordplay involved.
[…] An earlier version of some of this material was in my January, 2014 post on The games of Watson. […]