Towards Feasible Verification of Safety Properties in Safety-Critical Systems

1. Towards Feasible Verification of Safety Properties in Safety-Critical Systems Min Young Nam Mu Sun

2. Designer Intuitive Description Front End Model Simplification Formal Verification Back End Our Plan – Getting to the Point • Safety-Critical Systems • Verify Safety Properties during Design • Make more intuitive • Make more feasible

3. Outline • Research Planning • Background, Motivation, and Definitions • Finding the Research Issues • Current Technology and Limitations • Approaching the Problems • Research Timeline

4. Outline • Writing Planning • Pre-research Writing (Abstract) • Introduction Content • Paper Outline • Our Improvements

5. Background, Motivation, and Definitions Outline • Research Planning • Background, Motivation, and Definitions • Finding the Research Issues • Current Technology and Limitations • Approaching the Problems • Research Timeline

6. Background, Motivation, and Definitions Safety-Critical Systems • Systems whose failures are disastrous • Loss of human lives • Serious damage to property or environment • Major loss in revenue • Examples: Avionics Systems, Medical Devices, Nuclear Power Plants

7. Background, Motivation, and Definitions Motivation from the Real World • “A 32-year-old woman was having a laparoscopic cholecystectomy performed under general anesthesia. During that procedure and at the surgeon’s request, a plain film x-ray was shot during a cholangiogram. The anesthesiologist stopped the ventilator for the x-ray . • The x-ray technician was unable to remove the film because of its position beneath the table. The anesthesiologist attempted to help the technician, but found it difficult because the gears on the table had jammed. Finally, the x-ray was removed, and the surgical procedure recommenced. At some point, the anesthesiologist glanced at the EKG and noticed severe bradycardia. He realized he had never restarted the ventilator. • This patient ultimately died.” • APSF Newsletter, Winter 2005.

8. Background, Motivation, and Definitions Motivation from the Real World

9. Background, Motivation, and Definitions Motivation from the Real World Blurry Image

10. Background, Motivation, and Definitions Motivation from the Real World

11. Background, Motivation, and Definitions Motivation from the Real World

12. Background, Motivation, and Definitions Medical Plug and Play (MD PnP) • Most medical devices are designed to operate independently • MD PnP provides patient-centric systems and improves patient safety

13. Background, Motivation, and Definitions An Expert Opinion Current Problem Most medical devices are designed to operate independently and cannot interface easily with other devices MD PnP provides patient-centric systems and improves patient safety Benefits MD PnP White Paper. MD PnP Program, February 17, 2007

14. Finding the Research Issues Outline • Research Planning • Background, Motivation, and Definitions • Finding the Research Issues • Current Technology and Limitations • Approaching the Problems • Research Timeline

15. Finding the Research Issues Where are the Research Issues? • All the technology is already available • Isn’t Medical Plug and Play just an engineering problem? Perhaps looking at an example will help

16. Finding the Research Issues Take X-Ray Request Automated System Medical Personnel Done Signal Controller Photo Turn on command Turn off request Request response X-Ray Ventilator Breath Ventilator X-Ray System Take picture Patient

17. Finding the Research Issues Take X-Ray Request Automated System Medical Personnel Controller Turn off request X-Ray Ventilator Breath Ventilator X-Ray System Patient

18. Finding the Research Issues Automated System Medical Personnel Controller Turn on command Request response X-Ray Ventilator Ventilator X-Ray System Patient

19. Finding the Research Issues Automated System Medical Personnel Controller Photo X-Ray Ventilator Ventilator X-Ray System Take picture Patient

20. Finding the Research Issues Automated System Medical Personnel Done Signal Controller BAD DESIGN Turn on request X-Ray Ventilator Breath Ventilator X-Ray System Patient

21. Finding the Research Issues Automated System Medical Personnel Controller GOOD DESIGN X-Ray Ventilator Ventilator X-Ray System Patient

22. Finding the Research Issues The Research Issues • Conceptual shift (old idea; new situation) • Integrating components developed by different manufacturers can lead to unforeseen interactions • Safety-critical systems requires high reliability and robustness • Something must be changed in conventional design paradigms to address safety properties

23. Finding the Research Issues Analysis • Properties • Plug and Play • Modularization of design • Abstraction of details • Many different designers • Interoperability • Complex interactions • Issues • Increased complexity requires automated verification of certain properties • Abstraction of details hides properties inside the design

24. Finding the Research Issues Questions to Answer • How can we verify the safety properties of the system? • How can we make the specification easier for developers? • How do we obtain feasible verification?

25. Current Technology and Limitations Outline • Research Planning • Background, Motivation, and Definitions • Finding the Research Issues • Current Technology and Limitations • Approaching the Problems • Research Timeline

26. Current Technology and Limitations Safety Verification Tools • Architectural Description Languages • Formal Verification

27. Current Technology and Limitations Current Technology - I • Architectural Description Languages thread xm_thread features contr_input: in event data port… timeout_input: in event port; … properties Dispatch_Protocol => Sporadic; end xm_thread; thread implementation xm_thread.impl annex behavior_specification {** states idle: initial complete state; … state variables xray_done: Behavior::boolean; … initial xray_done := false; … transitions idle -[contr_input?(in_message) when in_message = 1]-> pending_xray {timeout_req!(pending_timeout);}; … Behavior Specification of the X-Ray Machine

28. Current Technology and Limitations Current Technology - I • Architectural Description Languages • Allows developers to specify details of an architecture (used for later verification) • AADL • Current industrial standard • Flexible and extensible • Many people contribute to the set of verifiable properties in AADL

29. Current Technology and Limitations Current Technology - I • Architectural Description Languages • Deficiencies • Does not provide all the details needed for verification • Many constructs are not standardized • Companies may not provide information on how they specify their designs

30. Current Technology and Limitations Current Technology - II • Formal Verification [‘xm_th : thread | --- Features in-port(‘contr_input, nil) ; in-port(‘timeout_input, nil) ; … --- Properties active ; --- State state(‘idle) ; --- State variables with initial values var(‘xray_done, false) ; … --- Transitions ‘idle –[? ‘contr_input(‘in_message) when ‘in_message = 1]-> ‘pending_xray {! ‘timeout_req(‘pending_timeout);}; … ] Formal Description of the X-Ray Machine

31. Current Technology and Limitations Current Technology - II • Formal Verification • Exhaustively search for properties / conditions in a system • Check if the system enters a certain error state • Can verify anything as long as a model and method exists

32. Current Technology and Limitations Current Technology - II • Formal Verification • Deficiencies • All required information about a system must be provided by the system designer • The method for verifying a property must be developed and proven • Complex system designs can easily lead to state explosions and infeasible verification time

33. Current Technology and Limitations Current Problems and Challenges • Lack of expressiveness in Architectural Description Languages • Feasible use of Formal Verification

34. Approaching the Problems Outline • Research Planning • Background, Motivation, and Definitions • Finding the Research Issues • Current Technology and Limitations • Approaching the Problems • Research Timeline

35. Approaching the Problems Our Strengths • Min Young • AADL and its limitations • Mu • High level verification tools • Group Collaborators • SEI (AADL) • Professor Meseguer (Formal Verification) • Lockheed Martin (real industrial applications)

36. Approaching the Problems Our Approach Designer Current Paradigm Formal Verification

37. Approaching the Problems Our Approach Designer Intuitive Input for Verification Architecture Description Front End Formal Verification Back End

38. Approaching the Problems Our Approach Designer Increase feasibility of verification Architecture Description Front End High Level Verification (Model Prunning) Formal Verification Back End

39. Approaching the Problems Our Approach • Extend the Architecture Description Language expressiveness • Generate formal verification specifications from the Architecture Description Language • Use partial verification tools to reduce the verification complexity beforehand

40. Research Timeline Outline • Research Planning • Background, Motivation, and Definitions • Finding the Research Issues • Current Technology and Limitations • Approaching the Problems • Research Timeline

41. Research Timeline Research Plan - I • Analyze a very simple example • Ventilator X-Ray interoperation system • Safety can be isolated in one component • The ventilator must work! • The X-Ray is not that important Ventilator X-Ray

42. Research Timeline Research Plan - II • Analyze a more complex example • Ventilator Laser Surgery interoperation system • Safety cannot be isolated in one component • The laser can burn the patient if the ventilator is on • Cooperation is required for safety Ventilator Laser

43. Research Timeline Research Plan - III • Analyze a real system • Autonomous Car Project • Very complex interactions with many safety critical components

44. Research Timeline Concrete First Steps • Use dependency analysis to reduce the verification complexity • Add dependency functionality into AADL

45. Research Timeline Concrete First Steps • Use dependency analysis to reduce the verification complexity • The Ventilator function cannot be affected by the X-Ray machine • Ignore the X-Ray state when performing analysis • Save a multiplicative factor in verification time USE

46. Pre-research Writing (Abstract) Outline • Writing Planning • Pre-research Writing (Abstract) • Introduction Content • Paper Outline • Our Improvements

47. Pre-research Writing (Abstract) A CORE Summary of the Details • Mary Shaw’s Questions • What specific questions does your research seek to answer? • Why are these questions important? • Is there a connection between this question and larger questions or issues? • Who will be the audience for your research?

48. Pre-research Writing (Abstract) A CORE Summary of the Details • What specific questions does your research seek to answer? • Why are these questions important? • Is there a connection between this question and larger questions or issues? • Who will be the audience for your research?

49. Pre-research Writing (Abstract) A CORE Summary of the Details • How can we ensure safety when designing systems? • Formal verification tools can verify safety properties of the system, but they can sometimes be infeasible in systems with complex interactions. What specific questions does your research seek to answer? Why are these questions important? Is there a connection between this question and larger questions or issues? Who will be the audience for your research?

50. Pre-research Writing (Abstract) A CORE Summary of the Details • How can we make formal verification more feasible? • We plan to use dependency analysis on a system to reduce the number of device states considered in a systems and thus expand the domain of systems that can be feasibly verified. What specific questions does your research seek to answer? Why are these questions important? Is there a connection between this question and larger questions or issues? Who will be the audience for your research?