Brittleness, Murphy’s Law, and single-impetus failures
In my initial post on brittleness I suggested that a typical process is:
- Build something brittle.
- Strengthen it over time.
In many engineering scenarios, a fuller description could be:
- Design something that works in the base cases.
- Anticipate edge cases and sources of error, and design for them too.
- Implement the design.
- Discover which edge cases and error sources you failed to consider.
- Improve your product to handle them too.
- Repeat as needed.
So it’s necesseary to understand what is or isn’t likely to go wrong. Unfortunately, that need isn’t always met.
Murphy’s Law and exaggerated fears
We should always bear in mind Murphy’s Law, which in its simplest form states: Anything that can go wrong, will. But also remember that Murphy’s Law is a joke; and even if it were serious, nothing concise is ever precise.
People who tend to over-believe in Murphy’s Law include but are hardly limited to:
- Bureaucrats.
- Worried parents, especially of only children. (Later kids tend to have it easier, as their parents have more experience.)
- Any buyer or voter you believe has been over-persuaded toward fear, uncertainty and doubt.
- Relational bigots who view the Ted Codd guarantee as an absolute requirement for data management.
Adversaries
The strongest scenarios for Murphy’s Law should be adversarial ones, in which somebody is actively trying to cause problems. But even there it doesn’t always apply. For example:
- Information security commonly fits the Murphy model. Hackers keep outwitting defenders.
- Email spam, however, does not. It’s pretty much of a solved problem; the few spam emails that still get through hardly matter.
- Web search is somewhere in between. Both sides are partially successful in the combat over adversarial information retrieval, as “good” and “bad” sites alike are both well-represented in search results.
Single-impetus failures
Since bad or scary things will happen — Murphy’s Law isn’t entirely wrong — a standard design practice is to avoid single points of failure. Brittleness has a lot to do with which single points of failure have been overlooked; improvement has a lot to do with belatedly cleaning them up. In adversarial scenarios, avoiding single points of failure relates closely to defense in depth.
Some of the nastiest surprises occur when failures have no obvious single point, yet wind up being possible from a single impetus.* This happens when multiple points or moments of failure are somehow correlated, or when they actually cascade. Examples vary widely, including:
- The collapse of the World Trade Center buildings.
- An authoritarian leader who manages to destroy a whole democratic system of government.
IT examples that are relatively big deals include:
- Security breaches in which an attacker becomes able to fully impersonate a well-credentialed user.
- Power outages or other whole-building breakdowns that bring down all parts of a (locally) redundant cluster.
- Software bugs that bring down all parts of a supposedly redundant system at once.
- Analytic failures that stem from misleading data sets. (Garbage in, garbage out.)
*I chose the phrase “single impetus” rather than “single cause” because NOTHING has a truly single cause; things only can happen when all kinds of conditions are satisfied for them to succeed. But there can indeed be an identifiable force, plan or occurrence that sets a chain of events in motion, and that’s what I’m calling the “impetus”.
Related link
- A lot of analytics turns out to be adversarial.
Comments
5 Responses to “Brittleness, Murphy’s Law, and single-impetus failures”
Leave a Reply
A useful of looking at this problem is the deep misunderstanding engineers (especially software) have of priorities of consumers. Brittle designs and feature gaps are generally reflections of immature product and missed understanding of need.
Brittle political or software systems may reflect manipulated requirements definition.
I’d add that if there’s a missing feature, product designers usually know that. What they get wrong is how important the lack will be competitively.
“Product designers usually know [about missing features]”
I think that’s attributing way too much insight to product designers. The more successful a product, the more it *creates* new uses, and then new needs, that were not foreseen – and *could not have been foreseen* by anyone. Sometimes this leads to “missing features” that completely supplant – and may even be in conflict with – the original designs.
Consider HTML. The original goal was to share scientific content across a wide variety of contexts. As part of this, a “missing feature” was the ability of the creator of the content to tightly control how it was rendered: The creator specified semantics (within limits), renderers figured out how to present it. As the Web rapidly grew to sell things rather than share content, this was, indeeed, seen to be a “missing feature”, and now the formatting is tightly controlled by creators.
One of Codd’s arguments for relational databases is exactly that the designers of the database could not foresee the uses to which their data would be put – so it was necessary to design databases with maximum flexibility in the face of future use changes. (One can debate the degree to which relational databases succeed, but the *goal* was there.)
I think there’s a hierarchy here.
At the bottom are “small extension” products. These can be designed by asking the intended users what they want. This gives you better saddles, not cars.
At the next level are “leap” products. These are typically the result of some visionary individual, or small group of individuals, realizing that there’s a better way to solve a problem – sometimes a problem that no one else even realizes they have. Often this is based on understanding that one or more technological advances have come to the point that they can be applied in a product. A visionary often knows that there are things in his vision that can’t be done right now, so they get left out – to be recognized by the rest of the world, after the fact, as “missing features”. The iPhone is a great example here – but there are many small-scale ones (and the line between good product designers and visionaries is by no means sharp).
Finally, there are the “big leap” products. These are the ones that start a sequence that changes how the world works. Looking back after the fact, it’s obvious that various early steps were “missing features”. But at the time, each of those features was itself something new that not so long before could not have been foreseen. The telephone. The automobile. The Internet. The Web. Web search.
BTW, it’s worth mentioning that people will go back and dig out the paper that foresaw some advance way ahead of time. Vannever Bush’s classic on the Memex is one example. These are fascinating bits of history but they tell us little: There were many other papers “looking forward” written at the same time, now forgotten because history didn’t happen to go that way.
— Jerry
Western Europe also formed
(palimpsests). In the XIII-XV centuries in