Mutual Empowerment in Human-Agent-Robot Teams

1. Mutual Empowerment in Human-Agent-Robot Teams 16 December 2010 HART Workshop Jurriaan van Diggelen

2. Problem statement • Achieve more with less people • Automation can help to: • Make better use of available semi-structured information sources • Support decision makers in dealing with the complexity of problems (war amongst the people)

3. The big number cruncher • Monolithic approach, BNC replaces existing infrastructure • AI-complete Sensor data Problem solution Twitter data UAV images

4. Towards a human-machine team solution • Solution must be provided by a human machine team • Mutual empowerment seeks to improve team performance by: • Compensating weaknesses of humans and machines • Optimizing strengths of humans and machines

5. Intelligent Interfaces Distributed Collective Artificial Intelligence Intelligence User empowerment Types of Mutual Empowerment HMI human machine CCI CMI HMI human machine

6. Goal ME handbook

9. Tool support Methodology • Use cases • Claims • Cognitive requirements • Ontologies • Performance measures • Tests/benchmarks Domain Exploration Functional design • Domain • Human Factors • Technology Validation Prototyping • System requirements • Functional modules • RDF interface specifications • Prototypes • Mixed reality validation • Data collection

10. Situated Cognitive Engineering • Methodology supports • Incremental design • Reuse of earlier work (Prototypes, tests, requirements, use cases) • Collaborative development

11. Example

12. Phase 1: domain exploration • Domain • USAR • UGV, UAV • Operators in field • Human Factors • Maintaining situation awareness • Cognitive overload • Adaptive teams • Technology • Collaborative tagging, crowd sourcing • Mixed initiative systems • Adaptive/ adaptable automation

13. Phase 2: Functional design (1) • UC 23 • UAV classifies camera image as victim with certainty-level Unsure • Operator of Robot1 is notified of the potential victim and views the • camera images • Operator of Robot1 classifies the image as victim with certainty level Certain • Operator of Robot2 is notified about the victim • … • Use cases • Cognitive requirements • Claims CR 5.1 Uncertainty management Operators and agents can publish and change the certainty value of information Use cases: UC 23 • CR 5.1 • + improves situation awareness of operators and agents • - increases cognitive taskload

14. something action event item robot victim Phase 2: Functional design (2) • Ontologies • Performance measures • E.g. situation awareness measure • Tests/benchmarks • Test for evaluating performance

15. Phase 3: Prototyping • Develop system requirements that implement the cognitive requirements. • Bundle system requirements in functional modules. • Reuse existing base platform

16. Trex

17. Trex • Filter: which data do you want to see? selection of semantic tags in Sparql • Projection: How do you want to see the data? graphical object with attachment-points for semantic tags

18. Human-in-the-loop AI P Q R S T Human Machine Crowd Machine Functional modules supported by Trex • User configurable information filters • User configurable information visualization • Realtime semi-structured data exploration • Collective relevance assessment • Uncertainty management • Human-in-the-loop AI

19. DEMO

20. Future work • Develop functional modules for: • Joint conflict resolution • Adaptive Interruptiveness • Network awareness • Policy awareness • Capability awareness • Activity awareness

21. Conclusion • Mutual Empowerment library provides a flexible way to • Increase application possibilities of AI • Employ potential of collective intelligence • Reuse and structure our knowledge of human-machine collaboration tools

22. Technology Investigation Domain Analysis Human Factors Exploration Use cases Claims Metrics Tests Ontologies Cognitive Requirements Functional design Core functions System Requirements Prototyping RDF interfaces Prototype Functional modules Simulation Test participants Empirical results Testing