Inventing the Operational Safety Assessment ATN’99

1. Inventing the Operational Safety Assessment ATN’99 Steve Paasch Federal Aviation Administration Aircraft Engineering Division Avionics Branch AIR-130 c/o ANM-100S 1601 Lind Avenue SW Renton, WA 98055-4056 phone: 425-227-1186 fax: 425-227-1181 email: steve.paasch@faa.gov

2. Operational Safety Assessments - How they became to be The purpose of conducting formal safety assessments The processes and methods involved in performing safety assessments Controller Pilot Data Link Communications (CPDLC) program Operational Safety Assessment (OSA) Contents

3. Operational Safety Assessments - How they became to be Technology Aging & Evolution Aviation Expansion & Globalization Need for New Operational Capabilities Safety Airspace Modernization RTCA TASK FORCE IV Need For Certification Efficiencies End-to-End “Certification” OSA Digital Communications Requirements (SC-189/WG-53)

4. RTCA SC-189 / EUROCAE WG-53 • Chartered to develop safety, performance, & interoperability requirements for air traffic services supported by communications • Subgroup 2 is working on methods and examples for developing operational environment descriptions and performing operational safety assessments • air-ground end-to-end safety assessment from an operational viewpoint

5. Website for SC-189 http://www.mews.org/atssir//

6. RTCA Task Force IV • Opportunities to Reduce the Time, to Reduce the Cost, and to Provide Better Certification Service • Achieving Operational Benefits • Human Performance • End-to-End Aviation Systems Considerations • Regulation, Policy, and Guidance Development • Authority Organization, Processes, and Industry Interface

7. End-to-End Aviation Systems Considerations • The Task Force heard many concerns that systems were not being properly considered overall, or "from end to end." The introduction of new elements into the ground or airborne parts of the system are not generally preceded by appropriate systems engineering practices, including definition of operations concepts and requirements. It is clear that overall system performance is rarely specified and that authorities often do not take a structured approach to establishing the requirements for International Airspace System (INAS) systems and components. It is common for new ground or airborne components to have specifications or performance that are not matched to the other elements of the system with which they work to perform their function. One consequence may be that the new system element is over-specified, and therefore more expensive than it should be to achieve the incremental improvement in performance. Another possible consequence is that the new system element is not properly specified in light of the performance of other system elements, and the expected improvement in efficiency from the new system element is not attained.

8. Task Force IV Recommendations • Recommendation 2: The authorities should establish and maintain a systems engineering capability. This function should be used to establish overall performance requirements for all advanced systems and their subsystems, in conjunction with the user community. As part of this effort, the authorities should consider developing clear approval standards and processes for ground system elements that are integrated, to the degree necessary, with airborne system element certification. (Section 3.4) • Recommendation 5: The authorities should broadly implement a process where the regulators and applicants come to an early and clear agreement on their respective roles, responsibilities, expectations, schedules, and standards to be used in certification projects. The process should apply broadly across airborne and ground systems, allow non-applicant equipment suppliers to engage in certification programs, and provide greater opportunity to approve components or processes independent of the airplane. (Section 3.6)

9. The purpose of conducting formal safety assessments

10. Starting from what we have and going to what we need • We have a traditional aircraft-related system safety assessment process • What is it? • What is for?

11. What is the traditional, aircraft related system safety assessment process? • It is a systems engineering activity to assure that safety objectives are met......by identifying where systems requirements are needed to eliminate or mitigate potential safety problems • Systems engineering is a two sided coin - optimistic vs pessimistic • SSA turns the systems engineering perspective of performance, functionality, form, etc., around • “Do this” vs “what if it doesn’t do this?”

12. What is the traditional, aircraft related system safety assessment process for? • In a nutshell-- To have a systematic way to analyze aircraft and aircraft systems function-related failure conditions, as well as failure condition contributors and mitigators, in order to: • Set safety objectives for failure conditions • Identify systems safety requirements to meet safety objectives • Assure systems safety requirements (and thus safety objectives) are met

13. A Systems Engineering Discipline • The System Safety Assessment side of the systems engineering coin: • has its own methods for discovering requirements • has its own processes to organize the methods • has its own vocabulary to facilitate the processes • has its own guidance materials for passing knowledge on

14. System safety assessments are tied to aircraft of a type, and the installed systems and equipment, or engines and engine systems.

15. ...But the aircraft isn’t the only player in the airspace game...

16. Broadening our horizons beyondan aircraft.....to the airspace system * multiple aircraft * multiple capabilities * ground systems * signal networks * operational procedures * ad hoc evolution * modernization program

17. What is the Operational Safety Assessment Process for? • In a nutshell-- A systematic way to analyze airspace and air traffic management service-related operational hazards, and operational hazard contributors and mitigators, in order to: • Set safety objectives for operational hazards • Identify systems and procedural safety requirements to meet safety objectives • Assure systems and procedural safety requirements (and thus safety objectives) are met

18. An Airspace Planning Discipline • The Operational Safety Assessment side of the airspace planning coin: • should have its own methods for discovering requirements • should have its own processes to organize the methods • should have its own vocabulary to facilitate the processes • should have its own guidance material for passing knowledge on

19. The processes and methods involved in performing safety assessments

20. Inventing a vocabulary • Starting with the system safety assessment vocabulary

21. What to say when good systems go bad • Failure Condition • Failure • Failure Mode • Fault • Error

22. What can we do with our specialized vocabulary? • We can organize our concepts into relationships

23. Aircraft designer’s view Aircraft FAILURE CONDITION ERROR FAILURE FAILURE FAILURE MODES FAULT FAULT FAULT FAULT ERROR PHYSICS PHYSICS ERROR PHYSICS ERROR PHYSICS ERROR

24. Functions Failure conditions Failures Failure Modes Faults Errors Air Traffic Services Operational hazards Failures Failure Modes Faults Errors Terminology comparison OSA: SSA:

25. Airspace designer’s view

26. Inventing a process • Starting with the system safety assessment process

27. A metaphor for systems engineering?

28. System Safety Assessment Process • Identify aircraft or systems functions • Identify failure conditions • Determine failure condition severity • Set safety objectives based on failure condition severity • Determine system safety requirements to meet safety objectives • Allocate safety requirements across systems and components • Assure safety requirements are met

29. What can we do with our specialized process? • We can organize our activities to be systematic and thorough

30. System Safety Assessment Process - discovering safety requirements as-built System Safety Assessments strategies & refinement Preliminary System Safety Assessments objectives Functional Hazard Assessment Common Cause Analyses Aircraft or System Function Definition

31. Identify aircraft or systems functions Identify failure conditions Determine failure condition severity Set safety objectives based on failure condition severity Identify air traffic services Identify operational hazards Determine oper. hazard severity Set safety objectives based on operational hazard severity Process comparison SSA: OSA:

32. Determine systems safety requirements to meet safety objectives Allocate safety requirements across systems and components Assure safety requirements are met Determine operational safety reqts to meet safety objectives Allocate safety reqts across institutions and airspace components Assure safety requirements are met Process comparison (continued) SSA: OSA:

33. Operational Safety Assessment Process -discovering safety requirements as-built Ground System Safety Assessments Aircraft System Safety Assessments Institutional Safety Assessments strategies & refinement Allocation of Safety Objectives and Requirements objectives Operational Hazard Assessment Operational Environment Definition (OED -- Services and airspace characteristics that may affect hazard severity) Common Cause Analyses

34. How do operational safety assessments and system safety assessments relate?

36. Inventing methods? • Starting with system safety assessment methods?

37. System Safety Assessment Methods • Inverse relationship for classifying failure conditions and setting assurance levels • Fail Safe Principles • Fault Tree Analysis • Failure Modes and Effects Analysis • Markov Analysis • Dependence Diagramming • Mathematics of failure rates, probability, and Boolean algebra

38. What can we do with our specialized methods? • We can discover cause and measure effect in a relatively precise fashion with tabular, graphical, mathematical, logical means

39. System Safety Assessment hazard classification scheme

41. Inverse relationship for classifying failure conditions and setting assurance levels Fail Safe Principles Fault Tree Analysis Failure Modes and Effects Analysis Markov Analysis Dependence Diagramming Mathematics of failure rates, probability, and Boolean algebra Inverse relationship for classifying operational hazards and setting assurance levels Otherwise, we’re working on it Matrix and templates Institutional methods at institutional levels CPDLC OED Hazard table FTA Reqts & allocation Methods comparison SSA: OSA:

42. Operational Safety Assessment hazard classification scheme

43. Inverse relationship

44. Inventing guidance material • Adding to system safety assessment guidance

45. Safety assessment guidance material • AC 23.1309-1C for Normal, Utility, Acrobatic, Commuter Airplanes • AC/AMJ 25.1309-1B for Transport Airplanes • AC 27-1A for Normal Rotorcraft • AC 29-2B for Transport Rotorcraft • SAE ARP 4754 for all • SAE ARP 4761 for all

46. What can we do with our specialized guidance? • We can pass the vocabulary, processes, and methods on to the community of airspace planners, developers, service providers, and users

47. System Safety Assessments AC 23.1309-1C AC 25.1309-1B AC 27-1A AC 29-2B SAE ARP 4754 SAE ARP 4761 Operational Safety Assessment RTCA SC-189 ED-DO docs Guidance Methodology SPR FANS Guidance comparison SSA: OSA:

48. SC-189/WG-53 Summary

49. Objs/Rqmts Coordination Publication overview ED/DO-GUID Cross-regional/area planning • Implementation • Aircraft certification • ATS system commissioning • ATS operational approval • User operational approval • Airspace approval ED/DO-SPR CNS/ATM System, Procedures, & Airspace Development CNS/ATM Service Operation Homogeneous ATM Area Planning Objs/rqmts ED/DO-METH • ED/DO-INTEROP • ARINC 622 • ARINC 623 • ATN • MIX

50. ED/DO-GUID Planning Requirements Determination Operational capability Air traffic services Functions Objective: To agree on approach To establish requirements Activities: Definition, Assessment, Allocation Validation Evidence: Approval plan(s) Assessments Requirements Traceability OED TechChoice OSA RCP Allocated requirements Interop Coordination Aircraft Ops App Aircraft Cert ATS Prov Sys App ATS Prov Ops App Airspace App