Beyond Human Factors: An Approach to Human/Automation Teams

1. Beyond Human Factors: An Approach to Human/Automation Teams Haomiao Huang Jerry Ding Wei Zhang Claire J. Tomlin Hybrid Systems Lab Action Webs Meeting 11/17/2010

2. Advances in complex multi-agent systems require smart integration of human elements. [nasa.gov, businessweek.com, tgdaily.com, techeasy.co.za, deere.com, aurore-sciences.org]

3. This requires new approaches to analyze humans as part of the system! [foxnews.com] [wikipedia] [adriandayton.com] Let’s think about humans as part of the solution, not the problem. [media.weirdworm.com] [knowyourmeme.com]

4. Two related problems 1) Modeling- Properly representing humans as components of the overall system 2) Control - generating useful directives and controls for human agents

5. Outline • Motivation • Scenario for Research on Human/Automation Teams • Adversarial Game Problem • Reachability Based Approach • Results • Conclusions & Future Work

6. Choosing a Research Scenario Games are representative of hard, real-world problems, yet provide relatively benign “sandbox” environments for development Roboflag Robocup Chess Starcraft What is a good game to capture the aspects of human-automation teams that we want to explore?

7. Capture-the-Flag Capture-the-flag embodies the basic research challenges we are trying to address Human players Adversarial Limited Information Multiple Agents Competing Objectives Time tested and fun http://www.goforyourlife.vic.gov.au/hav/articles.nsf/pages/Capture_the_Flag

8. Automation-Assisted Human Capture-the-Flag Using mobile phones, computers, and UAVs, we have turned capture-the-flag into a testbed for advanced automation concepts involving human team members STARMAC QuadrotorUAVs Game software on Android phones Server-side Management Software

9. Narrowing the problem Human players Adversarial Limited Information Multiple Agents Competing Objectives Time tested and fun http://www.goforyourlife.vic.gov.au/hav/articles.nsf/pages/Capture_the_Flag

10. Outline • Motivation • Scenario for Research on Human/Automation Teams • Adversarial Game Problem • Problem statement • Related Work • Solution Insights • Reachability Based Approach • Results • Conclusions & Future Work

11. Our Problem Characterize and solve a 1-sided capture-the-flag game with a single attacker and defender Game Domain CaptureRegion Defender Flag Attacker Flag Region Return Region

12. Related Work on Adversarial Games • Multi-agent games on discrete state spaces Greedy search Hespanha, Kim, and Sastry 1999 Approximate DP/Reinforcement Learning Lagoudakis and Parr 2002 Discrete Play Matching Browning, Bruce, and Veloso 2005 • Pursuit-evasion games with continuous states Receding-Horizon Control Mcgrew, How, Bush, Williams and Roy 2008 Sprinkle, Eklund, Kim, and Sastry 2004 Optimal Trajectory Planning Earl and D’Andrea 2001 Chasparis and Shamma 2005 Analytical game theory approaches Basar 1989, Lewin 1994, Stipanovic, Melikyan, Hovakimyan 2010 Hamilton-Jacobi Reachability Mitchell, Bayen, and Tomlin, 2005 Ding, Sprinkle, and Tomlin 2008 Assumed, learned, or randomized opponent model

13. Reachability Approach, derived from pursuit-evasion games: CTF game can be posed as a reachability problem. Assume system dynamics Where is the input for Player Iand is the input for Player II Define as the reach-avoid set where a player can arrive in a goal region in at most time while avoiding region , no matter what the other player does

14. Capture-the-Flag as Reachability Victory conditions for each player can be encoded as reach-avoid sets in the joint state-space Joint Capture Set Joint Return Set Attacker Game Domain Flag Return Set (For Attacker) Defender

15. 1-D Game

16. Geometric insights Geometric analysis allows some insight into the 2-D capture-the-flag problem

17. Geometric insights Geometric analysis allows some insight into the 2-D capture-the-flag problem

18. Utility of Reachability Analysis Reachability analysis gives complete characterization of game, and are a natural display tool for guiding human decision-making and allowing least-restrictive control Teo and Tomlin, 2003 Geometric analysis is not terribly general, though…

19. Outline • Motivation • Scenario for Research on Human/Automation Teams • Adversarial Game Problem • Reachability Based Approach • Hamilton-Jacobi Reachability • Computation • Results • Conclusions & Future Work

20. Hamilton-Jacobi Reachability Reachability in continuous state-spaces can analyzed as a terminal cost-only optimization problem, solved backward in time Classic Optimal Control Cost Function Reachability Cost Function Tomlin 2009

21. Level-Set Representation Sets can be represented using sub-level sets of signed distance functions as terminal cost functions Set operations using point-wise minimum and maximums can be used to create arbitrary sets Tomlin 2009, Mitchell 2003

22. Solution Based on HJBI Equation The cost-to-go function is the unique viscosity solution to the Hamilton-Jacobi-Bellman-Isaacs equation Classic Optimal Control Cost Function Hamilton-Jacobi-Bellman-Isaacs Equation Optimal Hamiltonian

23. Reachability Via Modified HJBI Equation The backward reachable set is the zero sub-level set of the viscosity solution to a modified HJBI equation Reachability Cost Function Optimal Hamiltonian Modified HJBI Equation Mitchell, Bayen, Tomlin 2005

24. Numerical Solution to the Modified HJBI Equation The viscosity solution to the modified HJBI Equation can be computed on a grid using the Level Set Toolbox from UBC http://www.cs.ubc.ca/~mitchell/ToolboxLS/index.html

25. Outline • Motivation • Scenario for Research on Human/Automation Teams • Adversarial Game Problem • Reachability Based Approach • Results • HJBI Reachability applied to capture-the-flag • Simulation results • Experimental setup • Conclusions & Future Work

26. Problem Formulation for 1v1 Capture-the-Flag HJBI reachability analysis allows us to fully characterize the game Dynamics Optimal Hamiltonian Optimal Inputs

27. Flag Return & Flag Capture Winning regions for each portion of the game can be calculated directly from reach-avoid conditions

28. Sequenced Capture and Return Winning regions for the full sequence (flag capture and subsequent return) can be computed by using the intersection of the flag return set and flag zone as the initial condition for flag capture

29. Simulation Results Simulation results demonstrate the use of the reachability solutions

30. Field Experiments in Progress Reachability-based control and input directives are being implemented on Droid Incredible phones Player Positions and State Game software on Android phones Reachable sets & optimal control inputs Server-side Management Software

31. Outline • Motivation • Scenario for Research on Human/Automation Teams • Adversarial Game Problem • Reachability Based Approach • Results • Conclusions & Future Work

32. Conclusions • Capture-the-flag is great platform for developing human-automation systems research. • A differential game formulation using HJBI reachability solves perfect information, 1v1 CTF

33. Future Work We have the “correct” answer to the adversarial problem… now what? Human players Adversarial Limited Information Multiple Agents Competing Objectives http://www.goforyourlife.vic.gov.au/hav/articles.nsf/pages/Capture_the_Flag

34. Thank you!Questions?