1 / 26

Example of a Complementary use of Model Checking and Agent-based Simulation

Example of a Complementary use of Model Checking and Agent-based Simulation. Gabriel Gelman & Karen Feigh Georgia Institute of Technology & John Rushby Stanford Research Institute. Introduction. Increasing Complexity. …. Leads to. Automation Surprises. Challenges in HMI.

hovan
Download Presentation

Example of a Complementary use of Model Checking and Agent-based Simulation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Example of a Complementary use of ModelChecking and Agent-based Simulation Gabriel Gelman& Karen Feigh Georgia Institute of Technology & John Rushby Stanford Research Institute

  2. Introduction Increasing Complexity … Leads to Automation Surprises Challenges in HMI Model Checking Such as Tackled by Agents Simulation Combine to leverage benefits of both Pilots Automation Potential Issues To examine System Behavior HMI = Human-Machine Interaction

  3. Comparison: Model Checking/ Simulation

  4. Method: Connecting the Frameworks Extending the Counterexample Guided Abstraction Refinement (CEGAR) method Create Model & Specifications for Model Checking (SAL) Analyze Using Model Checking (SAL) Scenario Narrative Create Models & Metric Specifications for Simulation (WMC) Analyze Using Simulation (WMC) • Verify that the action sequence predicted by MC to be problematic continues to be problematic • Refine MC prediction to include specific temporal relationships between events

  5. Automation Surprise Aviation Case Study

  6. Automation Surprise “An Automation Surprise occurs when the automation behaves in a manner that is different from what the operator is expecting”, Palmer (1995) • Result of implementation of badly designed automation or lack of pilots’ training on system • Introduction of highly automated aircraft (glass cockpits) • Starting with aircraft like B-757, B-737 and A320 Sarter and Woods A320 study (80% surprised; n = 167) Failure to activate Approach Automatic Mode Changes

  7. Case Study: Airbus Automatic Speed Protection Sequence on approach • Flight Path Angle mode engaged • Airspeed too fast • Overspeed Protection • Open mode engaged OPEN CLIMB • Note: During descent FCU altitude is usually set to Missed Approach altitude if Go Around required Higher FCU altitude with respect to current altitude FCU: Flight Control Unit V/S: Vertical Speed FPA: Flight Path Angle Lower OPEN DESCENT

  8. Sequence Automation Surprise FPA = 3° e.g. 5000ft FCU Altitude = Go Around Altitude 1 5 2 3 Instrument Landing System (ILS) Glideslope 10° > FPA > 3° 4 Altitude Ground Runway Step 1: Aircraft is on ILS Glideslope and in FPA V/S mode Step 2: Air Traffic Control tells aircraft to level off Step 3: Aircraft tries to recapture ILS Glideslope with higher FPA Step 4: Because of steeper approach the speed exceeds Vmax Step 5: Mode change to OP CLB because FCU alt higher than current alt FCU: Flight Control Unit FPA: Flight Path Angle

  9. Modeling Platforms

  10. Model Checking: SAL (Symbolic Analysis Laboratory) • Simple models are checked for a given property • Reachable state space of a specification is explored • Exhaustive exploration of action space • Symbolic Model Checking does not require to explore full space Initial Conditions Abstract System Model Start State OK List<Actions> Actionx State 1 Trace of Actions State 3 Actionj Actioni Actionk State 2 Action1,…, Actioni,…Actionj,…Actionk State NOT OK (singe action or combination of actions) List<Actions>

  11. Case Study Modeled in SAL Note: Each step is a state transition, time is not modeled Airplane: Flies with Flaps (descending) (exceeds Vmax) Automation: Reverses Mode Pilot: Does nothing Airplane: Flies with Flaps (descending) Automation: OP CLB mode Pilot: Does nothing Airplane: Flies (descending) Automation: Track Mode Pilot: Dials Descend Initial State (FCU Alt = 3201 feet) 1 3 5 Airplane: Flies with Flaps (descending) Automation: OP CLB mode Pilot: Does nothing Airplane: Flies (descending) Automation: VS/FPA mode Pilot: Extends Flaps AUTOMATION SURPRISE 6 2 4 State State Transition FCU: Flight Control Unit • Alt increase from 2990 to 3291 • Mental Model still in descend • Positive Pitch

  12. Simulation: WMC (Work Models that Compute) Mental Model Altitude, Heading, • WMC Work Model Expectations Aircraft Work Model Actions Resources Agents Speed, Vertical Speed Auto Surprise Scripted Events Initial Conditions Pulls Stores SIM Core Mental Model Human Agent Updateable World Representation Traces of Key Metrics

  13. Simulation Runs Based on MC Output • Verify that the action sequence predicted by SAL to be problematic continues to be problematic • Refine SAL's prediction to include specific temporal relationships between events Step 1: Arm Approach t = 2: Arm Approach Becomes Step 2: Extend Flaps t = 5: Extend Flaps Step 3: Monitor Speed t = 9: Monitor Speed

  14. Simulation States that Varied Cruise FPA = 3° Go Around Altitude Level Off Duration Flaps Extension Speed STAR approach Level Off Altitude Altitude ILS Glideslope Ground Runway STAR: Standard Terminal Arrival Route ILS: Instrument Landing System FPA: Flight Path Angle

  15. Results

  16. Meaningful Scenarios from Simulation Traces Simulation Traces Leads to OPEN DES Automation Surprise OPEN CLB No Auto Surprise No Change

  17. Overview of Scenarios in Simulation Output SC: Scenario AS: Automation Surprise (*) Possibly due to artifact (**) SAL Scenario

  18. Model Checking Matching Case SAL Unknown time step WMC

  19. Scenario 4: OPEN CLB • Level off • Return to glideslope (dive) • Flaps Extension • Sets max speed below current speed (former max speed = 220 knots, max speed with flaps = 205 knots) • OPEN CLB engages • Aircraft climbs Zoom

  20. Scenario 6: OPEN CLB • Level off • Return to glideslope (dive) • Overspeed from dive • OPEN CLB engages • Aircraft climbs Zoom

  21. Preconditions for Scenarios • Go Around (GA) altitude fixed at 3291 feet (as in SAL) • Flaps Extension speed fixed at 226 knots (as in SAL) • Level Off altitude and duration varied SC: Scenario AS: Automation Surprise

  22. Preconditions for Scenarios • Go Around (GA) altitude fixed at 6000 feet • Level Off altitude fixed at 7000 feet • Level Off duration and Flaps Extension speed varied SC: Scenario AS: Automation Surprise

  23. Conclusion

  24. Next Step: Simulation  Model Checking • Implement capability for new scenarios into model checking • Make model checking model more detailed Create Model & Specifications for Model Checking (SAL) Analyze Using Model Checking (SAL) Scenario Narrative Create Models & Metric Specifications for Simulation (WMC) Analyze Using Simulation (WMC)

  25. Conclusion • Examined same scenario using both model checking and simulation • Simulation results show expansion of Model Checking results (more scenarios & comprises aircraft dynamics and time) • Method was shown how to use the two frameworks in conjunction to examine system behavior Model Checking Simulation

  26. Questions & Comments Welcome Now

More Related