Modeling Human Reasoning About Meta-Information

1. Modeling Human Reasoning About Meta-Information Presented By: Scott Langevin Jingsong Wang

2. Introduction • Human decision-making in real-time, dynamic environments is becoming more complex • Decision-makers must manage large amounts of incoming information and integrate it with previous knowledge to develop a “situational awareness” • Relies on domain-knowledge but also on the qualifiers (meta-information) describing the information • Problem: To replicate human reasoning or behavior, need to model both information and meta-information • Most approaches have focused on representing the information, but little discussion of the meta-information

3. What is Meta-Information? • Definitions • Data is output from a system that may or may not be useful to decision-making (radar reports storm is coming) • Information is recognized inputs that are useable to decision-making (storm is coming that may affect UAVs) • Meta-data is qualifiers of data that may or may not be useful to decision-making (radar can locate aircraft with error of +/-1.5m) • Meta-information is qualifiers of information that affect decision-making, reasoning, or behavior • Information processing • Situation awareness • Decision-making • Definitions serve to explicitly identify the critical role of meta-information in human decision-making

4. Human Behavioral Models • Attempt to replicate human cognitive processes • Attempt to model human behaviors must capture the impact of meta-information • HBM have wide variety of applications • Developing and testing theories of human cognition • Representing realistic human behavior in training • Expert and Decision Support Systems • Modelers typically do not address meta-information because of challenges acquiring, aggregating and integrating • Focus of this research is on modeling meta-information in Bayesian Belief Networks (BBNs)

5. Uncertainty and Human Decision-Making • Human decision-making under uncertainty deviates from logical decision-making and largely based on experience-based heuristic methods • Often the heuristics represent how experts reason about the meta-information • Uncertainty of information is one type of meta-information • Different methods of classifying uncertainty: • Executional uncertainty • Goal uncertainty • Environment uncertainty • Lack of information, etc • While these classifications of uncertainty and an understanding of their impacts on decision-making have been useful, they may not generalize to other types of meta-information not based on uncertainty (recency, reliability, trust)

6. Computational Approaches to Uncertainty • Probability Measures • Dempster–Shafer belief functions • Extensions to first-order logic (e.g., defeasible reasoning, argumentation) • Ranking functions • ‘plausibility” measures • Fuzzy set theory • Causal network methods (e.g., Bayesian belief networks, similarity networks, influence diagrams)

7. Types and Sources of Meta-Information • Identified the main types of meta-information that impact the decision-making process • Research from over 30 domain experts, and over 500h of interviews, observations and evaluations • From this developed a list of sources and types of meta-information that was consistently encountered across application domains • Believe this approach developed an understanding of expert reasoning and behavior sufficient to understand the impact of meta-information at a level that supports modeling

8. Types and Sources of Meta-Information

9. Modeling Human Reasoning and Behavior • Computational Representation of human reasoning and behavior • Model based on recognition-primed decision-making • Experts do not do significant amounts of reasoning and problem solving, but rather have been trained to recognize critical elements of a situation and act accordingly • Domain independent, modeling situation awareness-centered decision-making in high-stress, time-critical environments • SAMPLE is a general use HBM • Defined modules: Information Processing, Situation Assessment, Decision Making • Inputs processed by information processing module • Processed data (detected events) passed to situation assessment module • Assessed situation is passed to decision-making module • Rules, or lookup table of actions after situational assessment performed

10. SAMPLE Model

11. Bayesian Modeling about and with Meta-Information • Difficult aspect of modeling human cognition and behavioral processes is the need to reflect the known impacts of meta-information on those processes • Identified five features of reasoning that need representation within human behavior models: • Should succeed or fail to recognize relevant meta-information based on attentional and cognitive demands • Should support the representation of successful or unsuccessful human strategies to process information according to meta-information • Should represent the aggregation of meta-information • Should capture how effectively meta-information is understood relative to any prior understanding or knowledge • Should succeed and fail at incorporating meta-information-mediated situation assessments into behavior or decisions

12. Methods for Representing Human Reasoning • Bayesian belief networks • Fuzzy set theory • Rule-based production systems • Case-based reasoning • BBNs address multiple types of modeling requirements • Two types of meta-information reasoning • Deductive reasoning • Abductive reasoning • BBNs support both types of reasoning

13. Two Types of Reasoning

14. Modeling the Recognition and Aggregation of Meta-Information • In many cases, human decision-makers will have to compute meta-information from multiple factors • Data and meta-data can map to meta-information in the following ways: • One-to-one mappings • Many-to-one mappings • One-to-many mappings • Many-to-many mappings • Once meta-information is calculated, it can influence the information gathering, situation assessment, and decision-making process

15. Applying BBNs to Model Congnitive Computation of Meta Information

16. Sensor Type as Node in Network: Sensor Type 3

17. Sensor Type as Node in Network: Sensor Type 1

18. Aggregating Meta-Information to Compute Overall Confidence

19. Modeling the Recognition and Aggregation of Meta-Information • Knowing the best means to aggregate meta-information is challenging • Observation and study of human decision-making amongst subject matter experts may provide some justification, but will often unavoidably result in inclusion of biases • Using engineering data about sources may not adequately represent how a human would reason about meta-information, resulting in less reflective human behavior models

20. Modeling the Impact of Meta-Information on Situation Assessment • Three Approaches • Simply filter or prioritize information based on meta-information • Include meta-information within BBN models of information gathering, situation assessment, and decision-making processes • Use the meta-information in a specific parameter

21. Incorporating Meta-Information Explicitly into a BBN: No Confidence

22. Incorporating Meta-Information Explicitly into a BBN: Low Confidence

23. Examples of Computing the Probability of a Discrete Value for a BBN Node

24. Conclusion • We described the application of meta-information and BBNs in modeling each of the following types of cognitive tasks: • Recognition of relevant meta-information based on aggregation of available data, meta-data, information, and meta-information into types of meta-information. • Filtering and prioritization of information based on meta-information. • Aggregation of different types of meta-information to acquire their combined impact. • Understanding of the impact of meta-information on existing knowledge • Incorporation of meta-information into mediation of situation assessment and decision-making.

25. Questions?