Lecture 11: Interaction and Analysis Analytic Process and Methods

1. Lecture 11:Interaction and AnalysisAnalytic Process and Methods October 12, 2010 COMP 150-12Topics in Visual Analytics

2. Lecture Outline Decision Matrix ( • Interaction and Analysis • Definition • Interaction with data and problem • Relationship between interaction and problem-solving • Types of analysis problems • Analytical methods • Interaction with visual interfaces • Basic interaction types • Sample interaction methods • Interaction with other people • Collaborative analysis (CSCW) • Can interaction exist without visualization? • Design flow of interactive visualization tools

3. Interaction with a problem • Difference between goal-directedand open exploration • Typical problems are well-formed with known expectations and constraints • Visual analytical problems are typically the opposite. Analysts often do not know what to expect in the data given, or what they might find.

4. Examples of Analytical Tasks • Document analysis? Reading some random source code for the first time? Assessing a situation? • Others? • Detect the expected, discover the unexpected!

5. How would you solve this type of problem?

6. Analytic Discourse • People cannot reason effectively about hypotheses and scenarios that are unavailable to them. • Divergent thinking: thinking creatively to examine all plausible alternatives • Convergent thinking: assembling evidence to find an answer

7. Sense-Making Loop

8. Sense-Making Loop

9. How to Generate Hypotheses? • How do you know if you have generated all possible hypotheses? • How do you know what you don’t know?

10. How to Generate Hypotheses? • Four principle strategies: • Situational Logic • Applying Theory • Comparison with Historical Cases • “Non-strategy” – data immersion

11. Situational Logic • Most common operating mode for intelligence analysts. • Begins with consideration of concrete elements of the current situation. • The situation is regarded as “one-of-a-kind” so that it must be understood using its own unique logic

12. Situational Logic • Problems: • Difficult to see the “opposing team’s” perspective. • Does not utilize what’s already known

13. Theory • A generalization based on the study of many examples of similar phenomenon. • Advantage is that “theory economizes thought” • It helps to identify the key elements (factors) in a given situation • Allows the analyst to ignore the noise

14. Theory • Problem: • Assumes that the current situation falls into a known pattern • Are two situations ever exactly the same? • How does one generalize one into another? • What assumptions are being made because of one’s mental bias? • The case of the Shah of Iran in the late 1970s

15. Theory • Problem: • The case of the Shah of Iran in the late 1970s • Assumptions: • When an authoritarian ruler is threatened, he will defend his position with force • An authoritarian ruler with control of the military and security forces cannot be overthrown by popular opinion and agitation.

16. Situation Logic vs. Theory Situation Logic Theory

17. Comparison with Historical Cases • Differs from Situational Logic in that present situation is interpreted in the light of a more or less explicit conceptual model based on similar situations in the past • Differs from Theory in that there are not enough cases to form universally accepted set of rules.

18. Comparison with Historical Cases • Problems: • “Historical analogies often precede, rather than follow, a careful analysis of a situation” • US foreign policies tend to be one generation behind, because policy makers are determined to avoid the mistakes of the previous generation • Policies chosen would fit well with the historical situation, but not necessarily with the current one.

19. Comparison with Historical Cases • Problems: • In US foreign policy, for example: • In 1930s, policy makers adopted an isolation policy that would have worked well for preventing American involvement in WWI, but failed for WWII. • Communist aggression were seen as analogous to Nazi aggression, leading to a policy of containment that would have prevented WWII. • Vietnam was used as an argument against US preparations in the Gulf War – flawed because a difference in terrain

20. Data Immersion • Some analysts describe their work procedure as immersing in data without fitting data into any preconceived pattern. • Problem: • “Information cannot speak for itself” • Information is always considered within a context (or a person’s mental model) • A person might not be aware of the mental model!

21. Data Immersion • Data immersion is often unavoidableas the situation is often too vague, too new, and too messy. • However, keep in mind that this is “absorbing information”, not “analyzing information” • Objectivity cannot be gained by “not having any assumptions”. • It is only possible by making multipleassumptions explicitso that they can be examined and challenged

22. Creativity • Be creative. Think out of the box. See all different perspectives. • Work with colleagues who can challenge your thoughts • Expose yourself to alternative ideas and concepts • Work in an environment with creative thinking is encouraged

23. Questions?

24. Lots of Hypotheses, Now What? • Bad Strategies for Choosing a Hypothesis • “Satisficing” • Select the first identified alternative that’s good enough • Incrementalism • Focusing on a narrow range of alternatives without large deviation from existing position • Consensus • Agreement among collaborators • Reasoning by Analogy • Choosing the alternative that appears to avoid previous error (or to duplicate previous success) • Rely on principles that discriminate bad from good • Determine a set of principles and judge the hypotheses using these principles

25. Problems • Selective Perception • Biased by predispositions or mind sets • Looking for data that fit a hypotheses • Failure to Generate Appropriate Hypotheses • Ignores important information or data • Failure to Consider Diagnosticity of Evidence • A piece of evidence can be used to support different arguments (e.g. a patient with high temperature) • Failure to Reject Hypothesis

26. Experiment on Hypotheses Generation and Rejection • Experiment: • Given three numbers: 2, 4, 6 • Discover the rule behind this sequence • You are allowed to generate any 3 number sequence as many times as you’d like, and I will tell you if the sequence conforms to the rule

27. Questions?

28. Thinking Aids • Multi-attribute Utility Analysis (Decision Matrix) • Analysis of Competing Hypotheses

29. Decision Matrix

30. Decision Matrix

31. Decision Matrix

32. Decision Matrix

33. Questions?

34. Analysis of Competing Hypotheses

35. Analysis of Competing Hypotheses

36. Important Notes • Step 2: • Note the absence of evidence as well as its presence! (In a Sherlock Holmes story, “the dog did not bark” was a vital clue) • Step 4: • Need to add new evidence? Combine hypotheses? • Step 5: • All “+”s do not indicate a proven hypothesis, but the fewest “-”s are more likely to be true. • Finding all supporting evidences for a hypothesis is too easy. Finding a single evidence to disprove a hypothesis is hard (but the most significant). • Analysts often notice that their judgments are based on a few factors as opposed to the mass of information that they initially thought that they had gathered. • The matrix does not offer a solution!!

37. Important Notes • Step 6: • How flimsy is your conclusion? • If the key evidence turns out to be wrong, does that completely change your judgment? • What is that key evidence? • Step 7: • Need to report confidence level! • Discuss findings in step 6. • Step 8: • What scenario could happen in the future that will change the outcome of your analysis?

38. Questions?

39. How Much Information Do You Need? • Experiment: betting on horses

40. Problem! • Misalignment of confidence and expectation • So how many variables does an expert consider? How important is each of those variables?

41. Experiment: Stock Analysts • Stock analysts were given 50 stocks, complete with detailed information about each stock • Analysts were asked to predict long-term price appreciation • After the predictions, analysts were asked to describe how they reached the conclusions • including what variables were considered, • and how important was each variable • Results indicate that the actual predictions are more accurate than their descriptions • Also, the number of actual variables used in the prediction (mathematically shown) is less than what the analysts said they used

42. Mosaic Theory of Analysis • Proposes that analysis is like putting together a jigsaw puzzle, one piece of information at a time. Eventually a “clear picture” emerges • In reality, cognitive psychologists have shown that it’s the other way around. Analysts already have a “vague picture” in mind, and pieces of information are collected to fit into the picture

43. Questions?