The Role of Complexity in System Safety and How to Manage It

1. The Role of Complexity in System Safety and How to Manage It Nancy Leveson

2. You’ve carefully thought out all the angles • You’ve done it a thousand times • It comes naturally to you • You know what you’re doing, it’s what you’ve been trained to do your whole life. • Nothing could possibly go wrong, right?

4. What is the Problem? • Traditional safety engineering approaches developed for relatively simple electro-mechanical systems • New technology (especially software) is allowing almost unlimited complexity in the systems we are building • Complexity is creating new causes of accidents • Should build simplest systems possible, but usually unwilling to make the compromises necessary • Complexity related to the problem itself • Complexity introduced in the design of solution of problem • Need new, more powerful safety engineering approaches to dealing with complexity and new causes of accidents

5. What is Complexity? • Complexity is subjective • Not in system, but in minds of observers or users • What is complex to one person or at one point in time may not be to another • Relative • Changes with time • Many aspects of complexity: Will focus on aspects most relevant to safety

6. Relation of Complexity to Safety • In complex systems, behavior cannot be thoroughly • Planned • Understood • Anticipated • Guarded against • Critical factor is intellectual manageability • Leads to “unknowns” in system behavior • Need tools to • Stretch our intellectual limits • Deal with new causes of accidents

7. Types of Complexity Relevant to Safety • Interactive Complexity: arises in interactions among system components • Non-linear complexity: cause and effect not related in an obvious way • Dynamic complexity: related to changes over time • Decompositional complexity: related to how decompose or modularize our systems • Others ??

8. Interactive Complexity • Level of interactions has reached point where can no longer be thoroughly anticipated or tested • Coupling causes interdependence • Increases number of interfaces and potential interactions • Software allows us to build highly coupled and interactively complex systems • How affects safety engineering? • Component failure vs. component interaction accidents • Reliability vs. safety

9. Accident with No Component Failures

10. Software-Related Accidents Are usually caused by flawed requirements Incomplete or wrong assumptions about operation of controlled system or required operation of computer Unhandled controlled-system states and environmental conditions Merely trying to get the software “correct” or to make it reliable will not make it safer under these conditions.

11. Types of Accidents Component Failure Accidents Single or multiple component failures Usually assume random failure Component Interaction Accidents Arise in interactions among components Related to interactive complexity and tight coupling Exacerbated by introduction of computers and software

12. Safety = Reliability Safety and reliability are NOT the same Sometimes increasing one can even decrease the other. Making all the components highly reliable will not prevent component interaction accidents. For relatively simple, electro-mechanical systems with primarily component failure accidents, reliability engineering can increase safety. But this is untrue for complex, software-intensive socio-technical systems Our current safety engineering techniques assume accidents are caused by component failures

13. (From Rasmussen)

14. Accident Causality Models • Underlie all our efforts to engineer for safety • Explain why accidents occur • Determine the way we prevent and investigate accidents • May not be aware you are using one, but you are • Imposes patterns on accidents “All models are wrong, some models are useful” George Box

15. Chain-of-Events Model Explains accidents in terms of multiple events, sequenced as a forward chain over time. Simple, direct relationship between events in chain Events almost always involve component failure, human error, or energy-related event Forms the basis for most safety-engineering and reliability engineering analysis: e,g, FTA, PRA, FMECA, Event Trees, etc. and design: e.g., redundancy, overdesign, safety margins, ….

16. Reason’s Swiss Cheese Model

18. Swiss Cheese Model Limitations • Focus on “barriers” (from the process industry approach to safety) and omit other ways to design for safety • Ignores common cause failures of barriers (systemic accident factors) • Does not include migration to states of high risk: “Mickey Mouse Model” • Assumes randomness in “lining up holes” • Assumes some (linear) causality or precedence in the cheese slices • Human error better modeled as a feedback loop than a “failure” in a chain of events

19. Non-Linear Complexity • Definition: Cause and effect not related in an obvious way • Systemic factors in accidents, e.g., safety culture • Our accident models assume linearity (chain of events, Swiss cheese) • Systemic factors affect events in non-linear ways • John Stuart Mill (1806-1873): “Cause” is a set of necessary and sufficient conditions • What about factors (conditions) that are not necessary or sufficient? e.g., Smoking “causes” lung cancer • Contrapositive: A → B then ~ B→ ~ A

20. Implications of Non-Linear Complexity for Operator Error Role of operators in our systems is changing Supervising rather than directly controlling Not simply following procedures Non-linear complexity makes it harder for operators to make real-time decisions Operator errors are not random failures All behavior affected by context (system) in which occurs Human error a symptom, not a cause Human error better modeled as feedback loops

21. Dynamic Complexity • Related to changes over time • Systems are not static, but we assume they are • Systems migrate toward states of high risk under competitive and financial pressures [Rasmussen] • Want flexibility but need to design ways to • Prevent or control dangerous changes • Detect when they occur during operations

22. Decompositional Complexity • Definition: Structural decomposition not consistent with functional decomposition • Harder for humans to understand and find functional design errors • For safety, makes it difficult to determine whether system will be safe • Safety is related to functional behavior of system and its components • Not a function of the system structure • No effective way to verify safety of object-oriented system designs

23. Human Error, Safety, and Complexity Role of operators in our systems is changing Supervising rather than directly controlling Complexity is stretching limits of comprehensibility Designing systems in which operator error inevitable and then blame accidents on operators rather than designers Designers are unable to anticipate and prevent accidents Greatest need in safety engineering is to Limit complexity in our systems Practice restraint in requirements definition Do not add extra complexity in design Provide tools to stretch our intellectual limits

24. It’s still hungry … and I’ve been stuffing worms into it all day.

25. So What Do We Need to Do?“Engineering a Safer World” • Expand our accident causation models • Create new hazard analysis techniques • Use new system design techniques • Safety-driven design • Integrate safety analysis into system engineering • Improve accident analysis and learning from events • Improve control of safety during operations • Improve management decision-making and safety culture

26. STAMP(System-Theoretic Accident Model and Processes) • A new, more powerful accident causation model • Based on systems theory, not reliability theory • Treats accidents as a control problem (vs. a failure problem) “prevent failures” ↓ “enforce safety constraints on system behavior”

27. STAMP (2) Safety is an emergent property that arises when system components interact with each other within a larger environment A set of constraints related to behavior of system components (physical, human, social) enforces that property Accidents occur when interactions violate those constraints (a lack of appropriate constraints on the interactions) Accidents are not simply an event or chain of events but involve a complex, dynamic process Most major accidents arise from a slow migration of the entire system toward a state of high-risk Need to control and detect this migration

28. STAMP (3) • Treats safety as a dynamic control problem rather than a component failure problem. • O-ring did not control propellant gas release by sealing gap in field joint of Challenger Space Shuttle • Software did not adequately control descent speed of Mars Polar Lander • Temperature in batch reactor not adequately controlled in system design • Public health system did not adequately control contamination of the milk supply with melamine • Financial system did not adequately control the use of financial instruments

29. Example Safety Control Structure

30. Safety Control in Physical Process

31. Safety Constraints • Each component in the control structure has • Assigned responsibilities, authority, accountability • Controls that can be used to enforce safety constraints • Each component’s behavior is influenced by • Context (environment) in which operating • Knowledge about current state of process

32. Control processes operate between levels of control Controller Accidents occur when model of process is inconsistent with real state of process and controller provides inadequate control actions Model of Process Control Actions Feedback Feedback channels are critical -- Design -- Operation Controlled Process

33. Relationship Between Safety and Process Models (2) • Accidents occur when models do not match process and • Required control commands are not given • Incorrect (unsafe) ones are given • Correct commands given at wrong time (too early, too late) • Control stops too soon Explains software errors, human errors, component interaction accidents …

34. Accident Causality Using STAMP

35. Uses for STAMP More comprehensive accident/incident investigation and root cause analysis Basis for new, more powerful hazard analysis techniques (STPA) Supports safety-driven design (physical, operational, organizational)) Can integrate safety into the system engineering process Assists in design of human-system interaction and interfaces

36. Uses for STAMP (2) Organizational and cultural risk analysis Identifying physical and project risks Defining safety metrics and performance audits Designing and evaluating potential policy and structural improvements Identifying leading indicators of increasing risk (“canary in the coal mine”) Improve operations and management control of safety

37. STPA (System-Theoretic Process Analysis) • Identifies safety constraints (system and component safety requirements) • Identifies scenarios leading to violation of safety constraints • Includes scenarios (cut sets) found by Fault Tree Analysis • Finds additional scenarios not found by FTA and other failure-oriented analyses • Can be used on technical design and organizational design • Evaluated and compared to traditional HA methods • Found many more potential safety problems

38. 5 Missing or wrong communication with another controller

39. Technical Safety analysis of new missile defense system (MDA) Safety-driven design of new JPL outer planets explorer Safety analysis of the JAXA HTV (unmanned cargo spacecraft to ISS) Incorporating risk into early trade studies (NASA Constellation) Orion (Space Shuttle replacement) NextGen (planned changes to air traffic control) Accident/incident analysis (aircraft, petrochemical plants, air traffic control, railroad, UAVs …) Proton Therapy Machine (medical device) Adaptive cruise control (automobiles) Does it work? Is it practical?

40. Analysis of the management structure of the space shuttle program (post-Columbia) Risk management in the development of NASA’s new manned space program (Constellation) NASA Mission control ─ re-planning and changing mission control procedures safely Food safety Safety in pharmaceutical drug development Risk analysis of outpatient GI surgery at Beth Israel Deaconess Hospital UAVs in civilian airspace Analysis and prevention of corporate fraud Does it work? Is it practical? Social and Managerial

41. Integrating Safety into System Engineering • Hazard analysis must be integrated into design and decision-making environment. Needs to be available when decisions are made. • Lots of implications for specifications: • Relevant information must be easy to find • Design rationale must be specified • Must be able to trace from high-level requirements to system design to component requirements to component design and vice versa. • Must include specification of what NOT to do • Must be easy to review and find errors

42. Intent Specifications • Based on systems theory principles • Designed to support • System Engineering (including maintainance and evolution) • Human problem solving • Management of complexity (adds intent abstraction to standard refinement and decomposition) • Model-Based development • Specification principles from preceding slide Leveson,Intent Specifications: An Approach to Building Human Centered Specification, IEEE Trans. on Software Engineering, Jan. 2000

44. Level 3 Modeling Language: Spectrm-RL • Combined requirements specification and modeling language. Supports model-based development. • A state machine with a domain-specific notation on top of it • Reviewers can learn to read it in 10 minutes • Executable • Formally analyzable • Automated tools for creation and analysis (e.g., incompleteness, inconsistency, simulation) • Black-box requirements only (no component design)

45. SpecTRM-RL • Black-box requirements only (no component design) • Separates design from requirements • Specify only black box, transfer function across component • Reduces complexity by omitting information not needed at requirements evaluation time • Separation of concerns is an important way for humans to deal with complexity • Almost all software-related accidents caused by incomplete or inadequate requirements (not software design errors)

48. Conclusions • Traditional safety engineering techniques do not adequately handle complexity • Interactive, non-linear, dynamic, and design (especially decompositional) • Need to take a system engineering view of safety rather than the current component reliability view when building complex systems • Include entire socio-technical system including safety culture and organizational structure • Support top-down and safety-driven design • Support specification and human review of requirements

49. Conclusions • Need a more realistic handling of human errors and human decision-making • Need to include behavioral dynamics and changes over time • Consider processes behind events and not just events • Understand why controls drift into ineffectiveness over time and manage this drift

50. Nancy Leveson “Engineering a Safer World” (Systems Thinking Applied to Safety) MIT Press, December 2011 Available for free download from: http://sunnyday.mit.edu/safer-world