Living with High Risk Technologies

1. Living with High Risk Technologies Charles Perrow, “Normal Accidents”

2. Technology • First Picture of Water on Mars!

4. What is a Normal Accident?

5. Definitions Complexity and catastrophe Looking at Systems Risk outweigh benefits Conclusions. Outline

6. Normal Accident • synonym for "inevitable accidents."

7. Normal Accidents • Normal accidents in a particular system may be common or rare ("It is normal for us to die, but we only do it once."), but the system's characteristics make it inherently vulnerable to such accidents, hence their description as "normal."

9. Failures • Discrete Failures • A single, specific, isolated failure is referred to as a "discrete" failure. X

10. Redundant Systems • Redundant sub-systems provide a backup, an alternate way to control a process or accomplish a task, that will work in the event that the primary method fails. This avoids the "single-point" failure modes.

11. Interactive Complexity • A system in which two or more discrete failures can interact in unexpected ways is described as "interactively complex." In many cases, these unexpected interactions can affect supposedly redundant sub-systems. A sufficiently complex system can be expected to have many such unanticipated failure mode interactions, making it vulnerable to normal accidents.

12. Tight Coupling • The sub-components of a tightly coupled system have prompt and major impacts on each other. If what happens in one part has little impact on another part, or if everything happens slowly (in particular, slowly on the scale of human thinking times), the system is not described as "tightly coupled." Tight coupling also raises the odds that operator intervention will make things worse, since the true nature of the problem may well not be understood correctly.

13. The interactive complexity and tight coupling Unexpected interactions will occur An accident will be reduced. Premise: characteristics of system – Not based on frequency. Normal Accident

14. NASA View • NASA nominally works with the theory that accidents can be • prevented through good organizational design and • management. • Normal accident theory suggests that in complex, tightly coupled • systems, accidents are inevitable. • There are many activities underway to strengthen our safety • posture. • NASA’s new thrust in the analysis of close-calls provides insight • into the unplanned and unimaginable. • To defend against normal accidents, we must understand the • complex interactions of our programs, analyze close-calls and

15. Complexity – levels of system and organization. Coupling - how closely the systems interact. Redundant pathway – Backup system that would prevent accidents. High Risk – Event with catastrophic potential. Definitions

16. Discrete Failures – failures of isolated single systems Interactive Complexity Definitions

17. Systems Individual Components Interactions Feedback systems Definitions

18. Human error results in most accidents Mechanical failure is the highest cause of accidents. The environment impacts the accident. Design of the system is the most important prevention. Procedures are most important. Questionnaire…

19. Answers • Eighty percent of Accidents are caused by human error.

20. Creating Systems Organizations Sub-Organizations Understanding how they interact? Understand the risk? High Risk Systems

21. Systems • Human Interface – complexity/saturation

22. Four Distinct Failures Cooling system Valves closed Pilot Operated Relief Valve sticks open False indicators These Four occurred in 13 seconds Three Mile Island

23. Diagram: Three Mile Island

24. The Hydrogen Bubble: Hydrogen produced from zirconium Accident Took 33 hours into the accident Overpressure was ½ the design strength Three Mile Island

25. Errors • Familiar with System • System Design flaws

26. Benefit of Understanding Reduce Dangers – could TMI happen again? Remove the dangers Better operator Training (Three E’s) More Quality Control Effective Regulation. Risk & Benefits

27. Operating Experience – Not sufficient Construction – pressure to build Safer Designs = less vulnerability? Defense in Depth (nuclear term) High Risk Systems

28. Characteristics • High-Risk Technologies Characteristics (Beyond the toxic, explosive dangers) • Complexity • Coupling

29. Definitions • Complexity A system in which two or more discrete failures can interact in unexpected ways is described as "interactively complex." In many cases, these unexpected interactions can affect supposedly redundant sub-systems. A sufficiently complex system can be expected to have many such unanticipated failure mode interactions, making it vulnerable to normal accidents.

30. Coupling • Coupling The sub-components of a tightly coupled system have prompt and major impacts on each other. If what happens in one part has little impact on another part, or if everything happens slowly (in particular, slowly on the scale of human thinking times), the system is not described as "tightly coupled." Tight coupling also raises the odds that operator intervention will make things worse, since the true nature of the problem may well not be understood correctly.

31. High Complexity Unexpected outcome • X Fails, Y was out of order • Interaction Piper Alpha

32. High Complexity • X Fails, Y Fails, Z was out of order • Interaction Unexpected outcome Bhopal

33. Learning from Mistakes? • Numerous examples given. • High Risk systems still in use • Still at risk? • How do we evaluate this?

34. Complexity • Low Complexity – (Linear systems, near linear) • Result: Accident will not spread or be as serious.

35. High Complexity Systems • Not all Interactions known • Some failure points not identified

36. Normal Accidents • Why haven’t we had more?

37. Low Complex Characteristics • Low Complexity (Organization) • Additional Resources available • Time to Spare • Other ways to accomplish task

38. High Complexity - Organizations • Large organization • Slow for action • Complex Systems • Interconnection • Contradictions

39. CMM • Definition – Complexity Maturity Model • Reference • Handout

40. CMM Scoring • One Method

41. High Complexity - CMM

42. Coupling Definition: Example:

43. Coupling • Coupling. (High) • Processes happen fast • Can’t be turned off • Failed parts can’t be isolated • No other way to keep production going safety

44. High Coupling - decisions • Reluctant to shut down • $ is driver?? • Politics? • Production? • Unable to shut down process • Cost to shut down • Pressure to shut down • Damage to shut down

45. Cost of Shut Down $300 Million to shut down a Nuclear Power Plant License good for 40 years only

46. Coupling • Coupling Results: • Recovery is not possible • Disturbance spreads quickly • Irretrievable Results • Operator Action may make it worse

47. How it Happens? • Normal Accident: Interactive Complexity and Tight Coupling

48. High Complexity and Coupling • Examples: • Nuclear Power Plants • Laboratories • Industrial Processes

49. Complex and Linear Interactions Event disrupts Both systems invisible Sub-system System 1 Sub-system Visible simultaneous

50. Example • Chernobyl • Hot spot was not visible • Graphite rod affects