Overconfidence

1. Agatha Christie: Poirot: Cards on the Table (2005) HerculePoirot: The question is, can HerculePoirot possibly by wrong? Mrs. Lorrimer: No one can always be right. HerculePoirot: But I am! Always I am right. It is so invariable it startles me. And now it looks very much as though I may be wrong, and that upsets me. But I should not be upset, because I am right. I must be right because I am never wrong. Overconfidence 1 Tuesday, 23 September 20144:23 AM

2. “The odds of a meltdown are one in 10,000 years.” Vitali Skylarov, Minister of Power and Electrification in the Ukraine, two months before the Chernobyl accident (cited in Rylsky, 1986, February). No problem in judgement, in decision making, is more prevalent and more potentially catastrophic than overconfidence. As Janis (1982) documented in his work on groupthink, American overconfidence enabled the Japanese to destroy Pearl Harbour in World War II. Overconfidence 2

3. Overconfidence also played a role in the disastrous decision to launch the U.S. space shuttle Challenger. Before the shuttle exploded on its twenty-fifth mission, NASA's official launch risk estimate was 1 catastrophic failure in 100,000 launches (Feynman, 1988, February). This risk estimate is roughly equivalent to launching the shuttle once per day and expecting to see only one accident in three centuries. Overconfidence 3

4. Challenger's Rollout From Orbiter Processing Facility To The Vehicle Assembly Building 4

5. The Crew Of The Final, Ill-fated Flight of the Challenger 5

6. The Challenger Breaks Apart 73 Seconds Into Its Final Mission 6

7. Debris Recovered From Space Shuttle Challenger 7

8. Conclusion Richard Phillips Feynman (May 11, 1918 – February 15, 1988) was an American physicist known for his work in the path integral formulation of quantum mechanics, the theory of quantum electrodynamics, and the physics of the superfluidity of supercooled liquid helium, as well as in particle physics. For his contributions to the development of quantum electrodynamics, Feynman, jointly with Julian Schwinger and Sin-Itiro Tomonaga, received the Nobel Prize in Physics in 1965. 10.8 8

9. Conclusion Feynman played an important role on the Presidential Rogers Commission, which investigated the Challenger disaster. During a televised hearing, Feynman demonstrated that the material used in the shuttle's O-rings became less resilient in cold weather by immersing a sample of the material in ice-cold water. The Commission ultimately determined that the disaster was caused by the primary O-ring not properly sealing due to extremely cold weather at Cape Canaveral. 10.9 9

10. Was NASA genuinely overconfident of success, or did it simply need to appear confident? Because true confidence is hard to measure in such situations, the most persuasive evidence of overconfidence comes from carefully controlled experiments. Stuart Oskamp published one of the earliest and best known of these studies in 1965. Overconfidence 10

11. Oskamp asked 8 clinical psychologists; 18 psychology graduate students, and 6 undergraduates to read the case study of “Joseph Kidd,” a 29-year-old man who had experienced “adolescent maladjustment.” Each participant was given excerpts from "The Case of Joseph Kidd" a chapter in Robert White's “Lives in Progress” giving a detailed account of the life and problems of a 29 year old male. The chapter is about 50 pages long. Overconfidence - The Case Of Joseph Kidd 11

12. The case study was divided into four parts. Part 1 Introduced Kidd as a war veteran who was working as a business assistant in a floral decorating studio (35 words from the first page). Part 2 Discussed Kidd's childhood through age 12 (750 words ). Part 3 Covered Kidd's high school and college years (1,000 words ). Part 4 Chronicled Kidd's army service and later activities to age 29 (600 words). Overconfidence - The Case Of Joseph Kidd 12

13. Subjects answered the same set of questions four times - once after each part of the case study. These questions were constructed from factual material in the case study, but they required subjects to form clinical judgments based on general impressions of Kidd's personality. Questions always had five forced-choice alternatives, and following each item, subjects estimated the likelihood that their answer was correct. Overconfidence - The Case Of Joseph Kidd 13

14. 5 During College, when Kidd was in a familiar and congenial social situation, he often a. Tried to direct the group and impose his wishes on it b. Stayed aloof and withdrawn from the group c. Was quite unconcerned about how people reacted to him d. Took an active part in the group but in a quite and modest way e. Acted the clown and showed off Overconfidence - The Case Of Joseph Kidd 14

15. 5 During College, when Kidd was in a familiar and congenial social situation, he often a. Tried to direct the group and impose his wishes on it b. Stayed aloof and withdrawn from the group c. Was quite unconcerned about how people reacted to him d. Took an active part in the group but in a quite and modest way e.Acted the clown and showed off Overconfidence - The Case Of Joseph Kidd 15

16. 15 Kidd's present attitude towards his mother is one of a. Love and respect for her ideals b. Affectionate tolerance for her foibles c. Combined respect and resentment d. Rejection of her and all her beliefs e. Dutiful but perfunctory affection Overconfidence - The Case Of Joseph Kidd 16

17. 15 Kidd's present attitude towards his mother is one of a. Love and respect for her ideals b. Affectionate tolerance for her foibles c. Combined respect and resentment d. Rejection of her and all her beliefs e. Dutiful but perfunctory affection Overconfidence - The Case Of Joseph Kidd 17

18. These confidence ratings ranged from 20 percent (no confidence beyond chance levels of accuracy) to 100 percent (absolute certainty). Somewhat surprisingly, there were no significant differences among the ratings from psychologists, graduate students, and undergraduates, so Oskamp combined all three groups in his analysis of the results. What he found was that confidence increased with the amount of information subjects read, but accuracy did not. Overconfidence - The Case Of Joseph Kidd 18

19. After reading the first part of the case study, subjects answered 26 percent of the questions correctly (slightly more than what would be expected by chance), and their mean confidence rating was 33 percent. These figures show fairly close agreement. As subjects read more information, though, the gap between confidence and accuracy grew (see Figure slide 9.20). Overconfidence - The Case Of Joseph Kidd 19

20. The more material subjects read, the more confident they became - even though accuracy did not increase significantly with additional information. By the time they finished reading the fourth part of the case study, more than 90 percent of Oskamp's subjects were overconfident in their answers. Overconfidence - The Case Of Joseph Kidd 20

21. Overconfidence - The Case Of Joseph Kidd 21

22. In the years since this experiment, a number of studies have found that people tend to be overconfident of their judgments, particularly when accurate judgments are difficult to make. For example, Lichtenstein and Fischhoff (1977) conducted a series of experiments in which they found that people were 65 to 70 percent confident of being right when they were actually correct about 50 percent of the time. Overconfidence Lichtenstein and Fischhoff 22

23. In the first of these experiments, Lichtenstein and Fischhoff asked people to judge whether each of 12 children's drawings came from Europe or Asia, and to estimate the probability that each judgment was correct. Even though only 53 percent of the judgments were correct (very close to chance performance), the average confidence rating was 68 percent. Overconfidence Lichtenstein and Fischhoff 23

24. In another experiment, Lichtenstein and Fischhoff gave people market reports on 12 stocks and asked them to predict whether the stocks would rise or fall in a given period. Once again, even though only 47, percent of these predictions were correct (slightly less than would be expected by chance), the mean confidence rating was 65 percent. Overconfidence Lichtenstein and Fischhoff 24

25. After several additional studies, Lichtenstein and Fischhoff drew the following conclusions about the correspondence between accuracy and confidence in two-alternative judgments: Overconfidence Lichtenstein and Fischhoff 25

26. ► Overconfidence is greatest when accuracy is near chance levels. ► Overconfidence diminishes as accuracy increases from 50 to 80 percent, and once accuracy exceeds 80 percent, people often become under confident. Overconfidence Lichtenstein and Fischhoff 26

27. In other words, the gap between accuracy and confidence is smallest when accuracy is around 80 percent, and it grows larger as accuracy departs from this level. Discrepancies between accuracy and confidence are not related to a decision maker's intelligence. Overconfidence Lichtenstein and Fischhoff 27

28. Although early critics of this work claimed that these results were largely a function of asking people questions about obscure or trivial topics, recent studies have replicated Lichtenstein and Fischhoffs findings with more commonplace judgments. For example, in a series of experiments involving more than 10,000 separate judgments, Ross and his colleagues found roughly 10 to 15 percent overconfidence when subjects were asked to make a variety of predictions about their behaviour and the behaviour of others (Dunning, Griffin, Milojkovic, and Ross, 1990; Vallone, Griffin, Lin, and Ross, 1990). Overconfidence Lichtenstein and Fischhoff 28

29. This is not to say that people are always overconfident Ronis and Yates (1987) found, for instance, that overconfidence depends partly on how confidence ratings are elicited and what type of judgments are being made (general knowledge items seem to produce relatively high degrees of overconfidence). Overconfidence Ronis and Yates 29

30. There is also some evidence that expert bridge players, professional odds makers, and National Weather Service forecasters - all of whom receive regular feedback following their judgments - exhibit little or no overconfidence (Keren, 1987; Lichtenstein et al. 1982; Murphy and Brown, 1984; Murphy and Winkler, 1984). Overconfidence - Feedback 30

31. Overconfidence - Feedback Keren (1997) discusses bridge experts calibration, with amateur players showing considerable overconfidence and expert players calibrated almost perfectly. He underlines the difference between accuracy (usually carefully studied) and resolution (also called discrimination - an ability to judge whether an event will take place or not), as there is no agreement whether these are two forms of expertise or rather two different kinds of expertise. Still, for the most part, research suggests that overconfidence is prevalent. 9.31 31

32. What if people are virtually certain that an answer is correct? How often are they right in such cases? In 1977, Fischhoff et al. conducted a series of experiments to investigate this issue. In the first experiment, subjects answered hundreds of general knowledge questions and estimated the probability that their answers were correct. For example, they answered whether absinthe is a liqueur or a precious stone, and they estimated their confidence on a scale from 0.50 to 1.00. Overconfidence - Extreme Confidence 32

33. Fischhoff et al. then examined the accuracy of only those answers about which subjects were absolutely sure. What they found was that people tended to be only 70 to 85 percent correct when they reported being 100 percent sure of their answer. The correct answer is that absinthe is a liqueur, though many people confuse it with a precious stone called amethyst. Overconfidence - Extreme Confidence 33

34. Just to be certain their results were not due to misconceptions about probability, Fischhoff et al. (1977) conducted a second experiment in which confidence was elicited in terms of the odds of being correct. Overconfidence - Extreme Confidence 34

35. Subjects in this experiment were given more than 106 items in which two causes of death were listed - for instance, leukaemia and drowning. They were asked to indicate which cause of death was more frequent in the United States and to estimate the odds that their answer was correct (i.e. 2:1, 3:1, etc.). This way, instead of having to express 75 percent confidence in terms of a probability, subjects could express their confidence as 3:1 odds of being correct. Overconfidence - Extreme Confidence 35

36. What are odds? Odds are just an alternative way of expressing the likelihood of an event such as catching the flu. Probability is the expected number of flu patients divided by the total number of patients. Odds would be the expected number of flu patients divided by the expected number of non-flu patients. Aside - Odds Skip 36

37. During the flu season, you might see ten patients in a day. One would have the flu and the other nine would have something else. So the probability of the flu in your patient pool would be one out of ten. The odds would be one to nine. Aside - Odds 37

38. More details It's easy to convert a probability into an odds. Simply take the probability and divide it by one minus the probability. Here's a formula. Odds = Probability/(1-Probability) If you know the odds in favour of an event, the probability is just the odds divided by one plus the odds. Here's a formula. Probability = Odds/(1+Odds) Aside - Odds 38

39. Example If both of your parents have an Aa genotype, the probability that you will have an AA genotype is 0.25. The odds would be Odds = 0.25/(1-0.25) = 0.333 which can also be expressed as one to three. Aside - Odds 39

40. If both of your parents are Aa, then the probability that you will be Aa is 0.50. In this case, the odds would be Odds = 0.5/(1-0.5) = 1 We will sometimes refer to this as even odds or one to one odds. Aside - Odds 40

41. When the probability of an event is larger than 50%, then the odds will be larger than 1. When both of your parents are Aa, the probability that you will have at least one A gene is 0.75. This means that the odds are. Odds = 0.75/(1-0.75) = 3 which we can also express as 3 to 1 in favour of inheriting that gene. Let's convert that odds back into a probability. An odds of 3 would imply that Probability = 3/(1+3) = 0.75 Aside - Odds 41

42. Suppose the odds against winning a contest were eight to one. We need to re-express as odds in favour of the event, and then apply the formula. The odds in favour would be one to eight or 0.125. Then we would compute the probability as Probability = 0.125/(1+0.125) = 0.111 Notice that in this example, the probability (0.125) and the odds (0.111) did not differ too much. This pattern tends to hold for rare events. In other words, if a probability is small, then the odds will be close to the probability. On the other hand, when the probability is large, the odds will be quite different. Aside - Odds 42

43. Aside - Odds When Spell-Check Can’t Help – NY Times - 13 May 2014 Confusion about odds By Philip B. Corbett, “After Deadline” blog The present article includes warnings about vague statements involving odds. Consider these two examples: The odds of Mr. Gandhi’s becoming the next prime minister have dropped so low that Mumbai bookies have stopped taking bets on him. [Headline] Iraq Unrest Narrows Odds for Maliki to Keep Seat Take care to be clear in referring to “odds.” “Higher” odds could suggest that something is more likely (higher probability) or less likely (1,000 to 1, say, compared with 10 to 1). It was difficult to tell whether “narrows odds” in the second headline meant he had more chance or less. Consider “probability,” “likelihood” or “chance” as alternatives if “odds” might be ambiguous. 9.43 43

44. Aside - Odds On a related note the following quotations from What the Numbers Say: A Field Guide to Mastering Our Numerical World by Derrick Niederman and David Boyum (p. 174) are relevant: “If Congress ever decided to act in the public interest, it could do no worse than to pass a law banning the use of odds as a method for stating probabilities.” “If you're confused [about odds], don't worry, for even if you understand how odds work, you can never be sure if the person you're talking to does.” 9.44 44

45. What Fischhoff et al. (1977) found was that confidence and accuracy were aligned fairly well up to confidence estimates of about 3:1, but as confidence increased from 3:1 to 100:1, accuracy did not increase appreciably. When people set the odds of being correct at 100:1, they were actually correct 73 percent of the time. Even when, people set the odds between 10,000:1 and 1,000,000:1 indicating virtual certainty they were correct only 85 to 90 percent of the time (and should have given a confidence rating between 6:1 and 9:1). Overconfidence - Extreme Confidence 45

46. Although these results may seem to contradict Lichtenstein and Fischhoffs earlier claim that overconfidence is minimal when subjects are 80 percent accurate, there is really no contradiction. The fact that subjects average only 70 to 90 percent accuracy when they are highly confident does not mean that they are always highly confident when 70 to 90 percent accurate. Overconfidence - Extreme Confidence 46

47. Finally, as an added check to make sure that subjects understood the task and were taking it seriously, Fischhoff et al. (1977) conducted three replications. In one replication, the relation between odds and probability was carefully explained in a twenty minute lecture. Subjects were given a chart showing the correspondence between various odds estimates and probabilities, and they were told about the subtleties of expressing uncertainty as an odds rating (with a special emphasis on how to use odds between 1:1 and 2:1 to express uncertainty). Overconfidence - Extreme Confidence 47

48. Even with these instructions, subjects showed unwarranted confidence in their answers. They assigned odds of at least 50:1 when, the odds were actually about 4:1, and they gave odds of 1000:1 when they should have given odds of 5:1. Overconfidence - Extreme Confidence 48

49. In another replication, subjects were asked whether they would accept a monetary bet based on the accuracy of answers that they rated as having 50:1 or better odds of being correct. Of 42 subjects, 39 were willing to gamble-even though their overconfidence would have led to a total of more than $140 in losses. Overconfidence - Extreme Confidence 49

50. And in a final replication, Fischhoff et al. (1977) actually played subjects' bets. In this study, 13 of 19 subjects agreed to gamble on the accuracy of their answers, even though they were incorrect on 12 percent of the questions to which they had assigned odds of 50:1 or greater (and all would have lost from $1 to $11, had the experimenters not waived the loss). Overconfidence - Extreme Confidence 50