System accident

A system accident (or normal accident) is an "unanticipated interaction of multiple failures" in a complex system.^[1] This complexity can either be of technology or of human organizations and is frequently both. A system accident can be easy to see in hindsight, but extremely difficult in foresight because there are simply too many action pathways to seriously consider all of them. Charles Perrow first developed these ideas in the mid-1980s.^[2] Safety systems themselves are sometimes the added complexity which leads to this type of accident.^[3]

Pilot and author William Langewiesche used Perrow's concept in his analysis of the factors at play in a 1996 aviation disaster. He wrote in The Atlantic in 1998: "the control and operation of some of the riskiest technologies require organizations so complex that serious failures are virtually guaranteed to occur."^[4]^[a]

Characteristics and overview

In 2012 Charles Perrow wrote, "A normal accident [system accident] is where everyone tries very hard to play safe, but unexpected interaction of two or more failures (because of interactive complexity), causes a cascade of failures (because of tight coupling)." Perrow uses the term normal accident to emphasize that, given the current level of technology, such accidents are highly likely over a number of years or decades.^[5] James Reason extended this approach with human reliability^[6] and the Swiss cheese model, now widely accepted in aviation safety and healthcare.

These accidents often resemble Rube Goldberg devices in the way that small errors of judgment, flaws in technology, and insignificant damages combine to form an emergent disaster. Langewiesche writes about, "an entire pretend reality that includes unworkable chains of command, unlearnable training programs, unreadable manuals, and the fiction of regulations, checks, and controls."^[4] The more formality and effort to get it exactly right, at times can actually make failure more likely.^[4]^[b] For example, employees are more likely to delay reporting any changes, problems, and unexpected conditions, wherever organizational procedures involved in adjusting to changing conditions are complex, difficult, or laborious.

A contrasting idea is that of the high reliability organization.^[7] In his assessment of the vulnerabilities of complex systems, Scott Sagan, for example, discusses in multiple publications their robust reliability, especially regarding nuclear weapons. The Limits of Safety (1993) provided an extensive review of close calls during the Cold War that could have resulted in a nuclear war by accident.^[8]

System accident examples

Apollo 13

The Apollo 13 Review Board stated in the introduction to chapter five of their report: [emphasis added]^[9]

... It was found that the accident was not the result of a chance malfunction in a statistical sense, but rather resulted from an unusual combination of mistakes, coupled with a somewhat deficient and unforgiving design...

(g): In reviewing these procedures before the flight, officials of NASA, ER, and Beech did not recognize the possibility of damage due to overheating. Many of these officials were not aware of the extended heater operation. In any event, adequate thermostatic switches might have been expected to protect the tank.

Three Mile Island accident

Perrow considered the Three Mile Island accident normal:^[10]

It resembled other accidents in nuclear plants and in other high risk, complex and highly interdependent operator-machine systems; none of the accidents were caused by management or operator ineptness or by poor government regulation, though these characteristics existed and should have been expected. I maintained that the accident was normal, because in complex systems there are bound to be multiple faults that cannot be avoided by planning and that operators cannot immediately comprehend.

ValuJet Flight 592

On May 11, 1996, Valujet Flight 592, a regularly scheduled ValuJet Airlines flight from Miami International to Hartsfield–Jackson Atlanta, crashed about 10 minutes after taking off as a result of a fire in the cargo compartment caused by improperly stored and labeled hazardous cargo. All 110 people on board died. The airline had a poor safety record before the crash. The accident brought widespread attention to the airline's management problems, including inadequate training of employees in proper handling of hazardous materials. The maintenance manual for the MD-80 aircraft documented the necessary procedures and was "correct" in a sense. However, it was so huge that it was neither helpful nor informative.^[4]

Financial crises and investment losses

In a 2014 monograph, economist Alan Blinder stated that complicated financial instruments made it hard for potential investors to judge whether the price was reasonable. In a section entitled "Lesson # 6: Excessive complexity is not just anti-competitive, it's dangerous", he further stated, "But the greater hazard may come from opacity. When investors don't understand the risks that inhere in the securities they buy (examples: the mezzanine tranche of a CDO-Squared; a CDS on a synthetic CDO ...), big mistakes can be made–especially if rating agencies tell you they are triple-A, to wit, safe enough for grandma. When the crash comes, losses may therefore be much larger than investors dreamed imaginable. Markets may dry up as no one knows what these securities are really worth. Panic may set in. Thus complexity per se is a source of risk."^[11]

Continuing challenges

Air transport safety

Despite a significant increase in airplane safety since 1980s, there is concern that automated flight systems have become so complex that they both add to the risks that arise from overcomplication and are incomprehensible to the crews who must work with them. As an example, professionals in the aviation industry note that such systems sometimes switch or engage on their own; crew in the cockpit are not necessarily privy to the rationale for their auto-engagement, causing perplexity. Langewiesche cites industrial engineer Nadine Sarter who writes about "automation surprises," often related to system modes the pilot does not fully understand or that the system switches to on its own. In fact, one of the more common questions asked in cockpits today is, "What's it doing now?" In response to this, Langewiesche points to the fivefold increase in aviation safety and writes, "No one can rationally advocate a return to the glamour of the past."^[12]

In an article entitled "The Human Factor", Langewiesche discusses the 2009 crash of Air France Flight 447 over the mid-Atlantic. He points out that, since the 1980s when the transition to automated cockpit systems began, safety has improved fivefold. Langwiesche writes, "In the privacy of the cockpit and beyond public view, pilots have been relegated to mundane roles as system managers." He quotes engineer Earl Wiener who takes the humorous statement attributed to the Duchess of Windsor that one can never be too rich or too thin, and adds "or too careful about what you put into a digital flight-guidance system." Wiener says that the effect of automation is typically to reduce the workload when it is light, but to increase it when it's heavy.

Boeing Engineer Delmar Fadden said that once capacities are added to flight management systems, they become impossibly expensive to remove because of certification requirements. But if unused, may in a sense lurk in the depths unseen.^[12]

Theory and practice interplay

Human factors in the implementation of safety procedures play a role in overall effectiveness of safety systems. Maintenance problems are common with redundant systems. Maintenance crews can fail to restore a redundant system to active status. They may be overworked, or maintenance deferred due to budget cuts, because managers know that they system will continue to operate without fixing the backup system.^[3] Steps in procedures may be changed and adapted in practice, from the formal safety rules, often in ways that seem appropriate and rational, and may be essential in meeting time constraints and work demands. In a 2004 Safety Science article, reporting on research partially supported by National Science Foundation and NASA, Nancy Leveson writes:^[13]

However, instructions and written procedures are almost never followed exactly as operators strive to become more efficient and productive and to deal with time pressures ... even in such highly constrained and high-risk environments as nuclear power plants, modification of instructions is repeatedly found and the violation of rules appears to be quite rational, given the actual workload and timing constraints under which the operators must do their job. In these situations, a basic conflict exists between error as seen as a deviation from the normative procedure and error as seen as a deviation from the rational and normally used effective procedure.

Notes

^ In the same article, Langewiesche continued: [emphasis added]^[4]
Charles Perrow's thinking is more difficult for pilots like me to accept. Perrow came unintentionally to his theory about normal accidents after studying the failings of large organizations. His point is not that some technologies are riskier than others, which is obvious, but that the control and operation of some of the riskiest technologies require organizations so complex that serious failures are virtually guaranteed to occur. Those failures will occasionally combine in unforeseeable ways, and if they induce further failures in an operating environment of tightly interrelated processes, the failures will spin out of control, defeating all interventions.
— William Langewiesche (March 1998), "The Lessons of Valujet 592", p. 23 [Section: "A 'Normal Accident'"], The Atlantic
^ See especially the last three paragraphs of this 30-plus-page Atlantic article: "... Understanding why might keep us from making the system even more complex, and therefore perhaps more dangerous, too."^[4]

Sources

Charles Perrow (1984). Normal accidents : living with high-risk technologies. ISBN 0-465-05143-X. Wikidata Q114963622.
Charles Perrow (1999). Normal accidents : living with high-risk technologies : with a new afterword and a postscript on the Y2K problem. Princeton University Press. ISBN 0-691-00412-9. Wikidata Q114963670.

References

^ Perrow 1999, p. 70.
^ Perrow 1984.
^ ^a ^b Perrow 1999.
^ ^a ^b ^c ^d ^e ^f Langewiesche, William (1 March 1998). "The Lessons of ValuJet 592". The Atlantic.
^ Perrow, Charles (December 2012). "Getting to Catastrophe: Concentrations, Complexity and Coupling". The Montréal Review.
^ Reason, James (1990-10-26). Human Error. Cambridge University Press. ISBN 0-521-31419-4.
^ Christianson, Marlys K; Sutcliffe, Kathleen M; Miller, Melissa A; Iwashyna, Theodore J (2011). "Becoming a high reliability organization". Critical Care. 15 (6): 314. doi:10.1186/cc10360. PMC 3388695. PMID 22188677.
^ Sagan, Scott D. (1993). The Limits of Safety: Organizations, Accidents, and Nuclear Weapons. Princeton University Press. ISBN 0-691-02101-5.
^ Chair Edgar M. Cortright. "Chapter 5. Findings, Determinations, and Recommendations". REPORT OF APOLLO 13 REVIEW BOARD ("Cortright Report") (Report).
^ Perrow, Charles (1982). "16. The President's Commission and the Normal Accident". In David L. Sills; C. P. Wolf; Vivien B. Shelanski (eds.). Accident at Three Mile Island : The human dimensions. Boulder, Colorado, U.S: Westview Press. pp. 173–184. ISBN 978-0-86531-165-7.
^ Blinder, Alan S. (November 2014). "What Did We Learn from the Financial Crisis, the Great Recession, and the Pathetic Recovery?" (PDF). Griswold Center for Economic Policy Studies Working Papers. Princeton University. No. 243.
^ ^a ^b Langewiesche, William (September 17, 2014). "The Human Factor - Should Airplanes Be Flying Themselves?". Vanity Fair. ... pilots have been relegated to mundane roles as system managers ... Since the 1980s, when the shift began, the safety record has improved fivefold, to the current one fatal accident for every five million departures. No one can rationally advocate a return to the glamour of the past.
^
Leveson, Nancy (April 2004). "A New Accident Model for Engineering Safer Systems" (PDF). Safety Science. 42 (4): 237–270. doi:10.1016/S0925-7535(03)00047-X. ... In fact, a common way for workers to apply pressure to management without actually going out on strike is to 'work to rule,' which can lead to a breakdown in productivity and even chaos ...
- Citing: Rasmussen, Jens; Pejtersen, Annelise Mark; Goodstein, L. P. (1994). Cognitive systems engineering. New York: Wiley. ISBN 978-0-471-01198-9.