Human-Information Interaction

1. Human-Information Interaction TA6 – Project 4 David MottIBM UK

2. Project Overview • to be assisted in collaborative problem-solving by systems that share common understanding of the knowledge and reasoning with human users • to engineer the coalition socio-technical environment in ways that support collective/team cognition • to exploit large bodies of diverse, unstructured information content and other information assets in the execution of their tasks • understanding the interaction between cognitive, social and technological factors in collaborative problem-solving contexts, by cognitively-rich agent simulation • supporting fact extraction from plain text by using deep linguistic processing and facilitating human-machine cognitive capabilities by externalising reasoning in a Controlled Natural Language • understanding how to support human decision-makers by matching information assets to requirements and by use of meaning-rich conversational interactions • COALITION NEEDS • PROJECT FOCUS • Project 4 aims to advance our understanding of the factors that affect shared understanding and information exploitation in military coalition environments ITA Peer Review, Sept. 2014

3. Multidisciplinary Research Team • IBM UK • David Braines • David Mott • Paul Stone • Airbus Group • Gavin Powell • Matthew Roberts • Carnegie Mellon University • Katia Sycara • Yuqing Tang • University of Southampton • Paul Smart • Darren Richardson • PSU • Tom La Porta • Nan Hu • University of Cambridge • Ann Copestake • Diarmuid O Seaghdha • Boeing • Anne Kao • Stephen Poteet • Ping Xue • University of Cardiff • Alun Preece • Will Webberley • IBM US • Geeth de Mel • Dstl • Kris Challa • UCLA • Mani Srivastava • Robin Wentau Ouyang • Matthew Johnson • ARL • Cheryl Giammanco • Jon Bakdash

4. Key Technical Accomplishments • a cognitive social simulation capability to support experimental studies of team cognition in complex task environments. • an initial cognitive computational model of collaborative problem solving in a specific task context (i.e., the ELICIT task). • a human experiment using the ELICIT task, designed to gather data relating to the effect of different information sharing strategies on task performance. • integration of Controlled English (CE) components into the simulation capability to support linguistic forms of agent communication. • Task 1: Collective Cognition in Coalition Environments ITA Peer Review, Sept. 2014

5. Key Technical Accomplishments extraction of CE facts from natural language (NL) sentences, applying knowledge expressed in CE to transform linguistic semantics into domain semantics. extension of CE for more complex problem solving strategies with meta-reasoning, assumptions, rationale, as applied to realistic analytic tasks handling of NL ambiguities and uncertainties, extending linguistic mechanisms to allow the CE domain model to guide NL processing exploration of collaborative man-machine problem-solving systems as applied to “logic puzzles” used in training intel analysts. Task 2: Fact Extraction and Reasoning using Controlled Natural language ITA Peer Review, Sept. 2014 5

6. Key Technical Accomplishments the definition and validation, including initial trials with human subjects, of a conversational protocol for free-flowing human-machine, machine-machine, and machine-human exchanges a model of resource allocation in network systems as a "stochastic knapsack problem" to handle uncertain factors like unreliable wireless medium or variable quality of sensor outputs a technology integration experiment to show how key research elements can be combined to support rapid but informed decision-making capabilities at lower echelons in coalition operations Task 3: Coalition Context-Aware Assistance for Decision Makers ITA Peer Review, Sept. 2014 6

7. ACT-R Cognitive Social Simulation Capability (CSSC) Insight: Cognitive architectures can be used to model socially- distributed cognitive processes and improve our understanding of the dynamics of team cognition. • Multi-agent simulation techniques have been used to model collective behavior. • ….however, such models do not take account of the cognitive capabilities and limitations of human agents. • Cognitive architectures provide computational frameworks for the development of cognitively-plausible agents. • …however, few applications to team contexts. • We need to know how to exploit the collective cognitive potential of coalition organizations. • We need to assess the effect of human factors and socio-technical interventions without disrupting the operational environment. • ACT-R: • a mature cognitive architecture that can be used to model a broad range of cognitive processes. • extend existing single-agent ACT-R research by focusing on interactions of multiple agents, allowing study of socially-distributed cognition • extend ACT-R architecture to support multiple-agent research • Cognitive social simulation: • integration of cognitive architectures into social simulation results in cognitive social simulation. • individual machine agents share the same kind of cognitive capabilities and limitations as their human counterparts. • Experimentation: • based on collaborative simulation of the DoD CCRPELICIT identification task • simulation capability instrumented to collect and measure performance factors based upon existing ELICIT metrics • State-of-the-art • Coalition needs/benefits Paul Smart (Southampton) CMU, IBM UK, Airbus Group

8. ACT-R Cognitive Social Simulation Capability (CSSC) Matching Selection Execution ACT-R Goal Module Declarative Module Imaginal Module Vocal Module Procedural Module ACT-R/CSSC Self Module Messaging Module Language Module Web Module Messaging Website Agent Characteristics Language Processor The ACT-R/CSSC is an extension to the core ACT-R architecture to support cognitive social simulation experiments.

9. ACT-R Cognitive Social Simulation Capability (CSSC)

10. ACT-R Cognitive Social Simulation Capability Result: Extended ACT-R cognitive architecture to support studies into team cognition. Custom modules support linguistic communication (language module), interaction with shared repositories (web module), and configuration of agent behaviour (self module). The simulation capability also supports the real-time participation of human subjects. • Simulation capability supports studies into team-based problem solving. • Current experimental efforts focused on the DoD CCRP ELICIT task. • Modelling the performance of human subjects in different organizational environments. Coordinator Long-Term Goals perform human experiments, and gather data, with a variant of the ELICIT task. extend cognitive computational models with machine agents embodied in the environment evaluate cognitive models with experimental simulation studies. Understand effect of task environment features on collective cognitive performance: Team Leader Team Member Shared Repository(limited access) The ACT-R Cognitive Social Simulation capability provides a generic platform for exploring the interaction between cognitive, social and technological factors in collective cognition. ITA Peer Review, Sept. 2012

11. CE to support fact extraction and problem solving CE can integrate modelling and reasoning capabilities to assist performance of complex cognitive tasks, including transformation of linguistic information from a deep parsing system into high value domain facts Traditional NL mechanisms have difficulty extracting detailed information about complex situations Deep parsing systems, can extract more subtle and detailed information from text, but it is a complex task to convert the linguistic output into domain-specific facts Performance of complex cognitive tasks requires integration of different modeling and reasoning capabilities in a common language Military tasks require text from NL sources to analyse, fuse and infer high value information A deep parsing system, with linguistic to domain semantic transformations, could extract more domain specific facts to support these tasks Collaborating human agents need support in problem-solving tasks in a way that is understandable and externalises their domain models and problem-solving strategies. Use of DELPH-IN linguistic resources The ERG, a deep and detailed grammar of English Minimal Recursion Semantics (MRS) to represent the output of the parse in a logical form Transforming linguistic semantics to domain semantics Rules expressed in CE for application of CE domain models Exploration of linguistic theories, in a way that is more natural than typical formal languages CE reasoning and modelling system User-defined domain model for expressing concepts, facts, logical rules and problem-solving strategies Assumptions to capture uncertainty, ambiguity and sentence interpretation Rationale and proof tables to visualise reasoning and sources of uncertainty Meta-model for reasoning about the language itself, e.g. mapping words to concepts and to create new rules from sentences State-of-the-art Coalition needs/benefits David Mott (IBM UK) 11 Boeing, Cambridge, ARL ITA Peer Review, Sept. 2014

12. Linguistic reasoning to extract CE facts

13. Reasoning to support problem solving

14. CE to support fact extraction and problem solving CE linguistic reasoning can apply domain knowledge to fact extraction, bridging the gap between deep linguistic analysis and domain semantics; CE domain reasoning can externalise human problem solving to solve an analytic task in an explainable way. CE domain models, reasoning and problem solving strategies Improve linguistic processing and fact extraction • linguistic reasoning to handle a wider range of sentences, e.g. following the MRS test suite • construction of domain models from distributional semantics • more domain knowledge to resolveambiguities of sentence interpretation Long-Term Goals Extend CE expressivity More sophisticated handling of assumptions Improve visualisation of rationale Handle more complex logical problem solving Improve problem solving capability A machine agent that can participate in collective cognition and externalises human reasoning CE problem solving techniques can be applied to both fact extraction from NL sentences and reasoning from basic extracted facts to intel relevant high value information ITA Peer Review, Sept. 2012 14

15. Human-Machine Conversations Insight: Human-machine conversations using natural language (NL) and Controlled English (CE) can facilitate information exploitation and asset tasking at the tactical edge. Reducing cycle time in rapid establishment of data-to-decision pipelines (D2D) still requires significant technical skills and training Handling of uncertainties in exploiting soft and hard information is a key problem, even for technical users Low training overhead and flexible human-machine collaboration (HMC) to create D2D pipelines Exchange of information precisely and unambiguously whilst retaining human readability and flexibility, pinpointing sources of uncertainty and providing explanation A formal conversational protocol – captured as a CE model – enables HMC in the scope of a range of tactical-level use cases: “spot” reports (including uncertain QoI) information fusion (including rationale) asset tasking (including uncertain availability) Combining the expressivity of NL and the precision and reasoning capabilities of CE State-of-the-art Coalition needs/benefits Alun Preece (Cardiff) ITA Peer Review, Sept. 2014 15 IBM UK, IBM US, PSU, UCLA, ARL

16. Human-Machine Conversations NL/CE Conversational Protocol Types of interaction include: • Confirmatory dialogues to reduce ambiguity (“did you mean…?”) • Query-response, with optional rationale (“Why?”) • Machine-generated (text/graphical) “gist” forms to summarise complex CE Aim: to enable conversational interactions that flow freely between natural language and CE, specifically to support dynamic establishment of D2D pipelines Conversational protocol draws on research in agent communication languages and philosophical linguistics (speech acts) ITA Peer Review, Sept. 2014

17. Human-Machine Conversations • Main aims: • to determine the degree to which a software agent can extract CE from unrestricted NL conversations • to test the agent’s robustness with untrained users Experimentation: scenario-based Experimentation: human subjects D2D vignette from earlier ITA work Technology integration experiment including: conversational agent (steps 1+4), CE Store (2), SAM (3) Demo with CERDEC AI TECD Jan 2014 Takes account of uncertainty of asset availability (e.g. bandwidth) and QoI (e.g. due to weather) • 20 subjects viewed a series of scenes and described them in natural language via a text-based interface • The system provided feedback in CE and a score (terms recognised by system) ITA Peer Review, Sept. 2014

18. Human-Machine Conversations • Human subject experiment shows rapidly-developed conversational agent able to extract exploitable soft information “There is two policemen are riding on a horse. The horses color are white and brown! They are riding in the same direction.” Longer-Term Goals support a wider range of conversational actions (queries, commands, narratives, model and lexical updates, interjection, etc) integrate information from more sources, e.g. sensor input from the mobile device, social media improve handling of uncertainties, and show utility of this via Experimentation Framework there is a group named '#35' that   has 'policemen' as description and   has the entity concept 'person' as member …. there is a group named '#38' that   has 'The horses' as description and   has the entity concept 'horse' as member and   has the colour 'brown' as colour and   has the colour 'white' as colour. Result: a conversational approach founded on use of natural and controlled natural language (CE) allows users at the tactical edge to exploit soft and hard information – including coping with uncertainties and QoI – using mobile computing approaches with low training overhead. 18 The D2D pipeline can be viewed as a collection of conversational interactions between human & machine agents: data sources, analytic services and decision-makers ITA Peer Review, Sept. 2012 18

19. Academic & Military Impact Technical Leadership • to CERDEC in the use of CE for ontology development • to linguistic researchers, including DELPH-IN and ARL, in the use of CNL for domain modelling and reasoning, and in the consideration of rationale as a new topic in academic research • to SMEs (Prof Don Shemanski and David Alberts) in the use of CE as a means of building a collaborative man machine reasoning system to solve analytic puzzles • Alun Preece et al received 2013 ACM MiSeNet MobiCom Workshop Best Paper Award • Dave Braines gave an invited plenary speech at NS-CTA on CE for hybrid reasoning • Alun Preece, Tien Pham and Dave Braines are on the TPC for NATO IST/SET 126 Symposium Transitions • Dstl projects: • use of a CNL for intelligence analysis (MIPS) • CE for intelligence, cyber assets, (+P5) policy based service composition, (+P5) missions and assets, • staff rotation at IBM Hursley to explore a CE agent for analysing Chinese • ARL projects: • application of CE and experimental framework for NS-CTA, TerraHarvest and ARL anomaly teams • demonstration of the conversational capability to CERDEC • use of CE to orchestrate high profile NS-CTA experiment, briefed to the NS-CTA board • staff rotation of Westpoint cadets at IBM Hursley to build interactive CE database of intelligence profiles • Potential: • CE as a business language in IBM Information Visualisation product • training in CE for modelling, for future transition with ARL analysts in the Agri-Development Teams • assistance for training and simulation of emergency situations by linking ACT-R and Unity3D modelling • conversational interface in advisory capacity to a UK police force at 2014 NATO Summit

20. Closing Remarks KEY COLLABORATIONS ENABLED KEY PUBLICATIONS Smart, P. R., Sycara, K., & Tang, Y. (2014) Using Cognitive Architectures to Study Issues in Team Cognition in a Complex Task Environment. Smart, P. R., Richardson, D. P., Sycara, K., & Tang, Y. (2014) Towards a Cognitively Realistic Computational Model of Team Problem Solving Using ACT-R Agents and the ELICIT Experimentation Framework. Mott, D., Poteet, S., Xue, P, Kao, A, Copestake, A. (2014), Natural Language Fact Extraction and Domain Reasoning using Controlled English Xue, P., Poteet, S., Kao, A., Mott, D., & Giammanco, C. (2014) Representing Uncertainty in CE Preece, A., Braines, D., Pizzocaro, D., & Parizas, C. (2014) Human-Machine Conversations to Support Multi-Agency Missions Preece, A., Gwilliams, C., Parizas, C., Pizzocaro, D., Bakdash, J., & Braines, D. (2014) Conversational Sensing 3 long papers, 9 short papers, 6 demonstrations, 1 workshop for the 2014 Fall Meeting • Southampton and CMU have extended the ACT-R architecture to support cognitive social simulation experiments. • IBM UK and Southampton have integrated CE components into the ACT-R-based simulation capability. • Cambridge and IBM UK are transforming ERG linguistic semantics into CE domain facts • Boeing and IBM UKhave developed representations of NL uncertainties as CE assumptions • Cardiff and IBM UK have developed conversational interfaces, with sensor mission-matching algorithms • Cardiff and PSU have integrated measures of uncertainty into allocation algorithms Reaching In • UCLA, IBM UK and Cardiff have extended conversational interfaces to crowdsourcing (+P6) • Cardiff, IBM UK and NS-CTA have integrated CE models and reasoning into demonstrations (+NS-CTA) • Aberdeen and IBM UKhave linked argumentation and CE assumptions for hypothetical reasoning (+P6) • IBM UK, Cardiff and RPI have applied CE to model policy based service composition (+P5) • IBM UK and ARL have explored with Fraunhofer (FKIE) on CE and the Battle Management Language Reaching Out Metrics ITA Peer Review, Sept. 2014