Judgment in forecasting

1. Judgment in forecasting Nigel Harvey

2. Studying judgment in forecasting • Forecasting using judgment alone (judgmental forecasting). • Using judgment to make adjustments to forecasts obtained by formal methods. • Using judgment to combine forecasts made by judgment and/or formal methods. • Using judgment to choose a forecasting method. • Using judgment to develop a formal method for forecasting (eg selecting terms to include in an econometric model). Harvey Evidence 2/11/4

3. Heuristics in judgmental forecasting (Hogarth & Makridakis, 1981) • We can forecast by extracting and using pattern information (time series analysis, Fourier analysis) or by using heuristics that have broad but not universal applicability (eg smoothing techniques). • Unlike perception (Marr, 1982), judgment appears to rely on mental heuristics (Tversky & Kahneman, 1974; Gigerenzer et al, 1999). • People can learn to replicate patterns in series of numbers but this ability does not correlate with forecasting ability (Harvey, 1988). Harvey Evidence 2/11/4

4. Kahneman & Tversky (1974) Three heuristics • Availability: Assess the frequency of a class or the probability of an event by the ease with which instances or occurrences can be brought to mind (eg K,L,N,R,V). • Representativeness: Judge likelihood by the degree to which something is representative of or resembles something else (eg BBBGGG vs BGGBGB). • Anchoring and adjustment: Make an estimateby starting from an initial value and adjusting it to yield a final answer. Initial values are often suggested in the problem. Adjustments are typically insufficient (eg UN). Harvey Evidence 2/11/4

5. Forecasts are made using one or more of the following: • Information held in memory. Which party will win the next election? Which team will win the match? Availability seems important here. • Information about another variable. What will sales be next year if we double our promotional spend? Representativeness seems important here. • Past information about the forecast variable: Given QJEP submission rates since its inception, what submission rate do we expect next year? Anchoring and adjustment seems important here. Harvey Evidence 2/11/4

6. Forecasting from information held in memory: Summary • Availability • Supporting evidence • The recognition heuristic as a variant • Its possible use for financial forecasting • Availability vs frequency records Harvey Evidence 2/11/4

7. Forecasting using information held in memory: Availability • Frequent events are experienced more often and more recently. Memory for something is better if it has been experienced more often and more recently. Thus memory trace strength for an event should be a reasonable guide to its frequency. Availability is a sensible basis for forecasting the likelihood of events. • It will be less effective when the media distorts the frequencies of events and when salience of memory for events is affected by factors unrelated to their frequency. Harvey Evidence 2/11/4

8. Forecasting using information held in memory: Availability • Well-reported events (deaths from electrocutions or tornadoes) are over-forecast; poorly reported events (deaths from asthma) are under-forecast (Lichtenstein et al, 1978). • People imagined a well-defined scenario of the Democrat or Republican candidate winning the 1976 US presidential election. Their later forecasts were consistent with what they had imagined. Similar effects were found after imagining sports teams winning or losing games (Carroll, 1978). Harvey Evidence 2/11/4

9. Forecasting using information held in memory: Availability • Not only does asking people to imagine events that are easy to imagine increase estimates of their future likelihood but asking them to imagine events that are difficult to imagine decreases estimates of future likelihood (Sherman et al, 1985). • Manipulating the availability of one unpleasant event affects the availability of other unpleasant events. So reading a report of a violent death increases people’s forecasts for the number of people who will die from leukaemia next year (Johnson & Tversky, 1983). Harvey Evidence 2/11/4

10. A variant: The recognition heuristic (Goldstein & Gigerenzer, 1999) • If only one of two objects is recognised, assume it has the higher value. Despite G & G’s protestations, this is not fundamentally different from the availability heuristic. Models of recognition are based on reaching a criterion level of memory trace strength (availability). • Ayton & Önkal (1997) found Turkish students were as good as English students at forecasting the outcomes of English Soccer matches. Teams from large cities tend to win and be recognised. In 662 matches where Turks recognised one team, they chose it 627 times. Harvey Evidence 2/11/4

11. The recognition heuristic in financial forecasting (Borges et al, 1999) • For higher returns, choose the stocks you recognise or that a given percentage of a group recognises. This requires some ignorance of stocks. • 360 lay people (pedestrians in Chicago or Munich) and 120 US and German finance/economics graduate students were surveyed on their knowledge of 798 companies in German and US stock markets. • For 6 months from 13/12/96, returns from 90% recognised stocks were compared with the market index, mutual funds, and a chance portfolio. Harvey Evidence 2/11/4

12. Intra-national Recognition Harvey Evidence 2/11/4

13. International Recognition Harvey Evidence 2/11/4

14. The recognition heuristic in individual investment choices • German lay people and experts identified 10 stocks for investment. Portfolios were formed from the 10 most often selected in each group. • Mean recognition rate for lay choices was .80 but for expert choices was only .48. • But… Harvey Evidence 2/11/4

15. The recognition heuristic in financialforecasting: Replication failures • Boyd (2001) tested 184 students’ recognition of 111 companies selected randomly from Standard & Poor’s 500. For 6 months return (6/00 - 12/00: Bear Market) Market index Recognition >90%Recognition <10% -4.5% -14.75% 16.75% • Rakow (2002) tested 54 UK students’ recognition of 30 companies in the Italian Mib 30. For an 8 week return in a strong market (10/02 – 12/02): Market index ExpertRecognitionAnti-recognition 21% 21% 13.3% 26.2% Harvey Evidence 2/11/4

16. Availability versus frequency records • Hasher & Zachs (1984) argue that memory experiments have shown we automatically register occurrences of events in our environment and use the resulting accurate records of their frequencies in our judgments. • Ayton & Wright (1994) point out that this is not consistent with use of the availability heuristic. • Lopes & Oden (1991) argued that the letters (KLNRV) used in the original letter study are not representative of most English consonants. 12/20 are more frequent in the first position. People may have accurately estimated the first position frequency of the letter class ‘consonant’: they gave a mean ratio of .67 and the correct value is .60. Harvey Evidence 2/11/4

17. Distinguishing availability from recording event frequencies • Sedlmeier et al (1998) asked people whether each of 13 letters occurred more in the first or second position of words and asked for the ratio of these two events. • First and second position frequencies for letters were obtained from a word corpus. Relative availability was estimated by asking people to recall as many words as possible with the target letters in the two positions. • Lopes & Oden’s letter class account was excluded. • Results fell between the two predictions. People may be able to retrieve frequency sometimes but use availability if they can’t. Or they may combine estimates. Or judged frequencies may be regressed versions of actual ones. Harvey Evidence 2/11/4

18. Sedlmeier et al (1998): Data and predictions from three models Harvey Evidence 2/11/4

19. Forecasting using information about another variable: Summary • Representativeness with single forecasts • Evaluation vs forecasting • Translation vs forecasting • Representativeness with multiple forecasts • Probability matching • Forecasting via intuitive regression Harvey Evidence 2/11/4

20. Forecasting using information about another variable: Representativeness • Kahneman & Tversky (1973) argue that this heuristic is used in different ways to make a single forecast and to make a set of forecasts. • When making a single forecast, people represent the deviation of the predictor variable from its mean in their forecast by ensuring the forecast variable has a similar deviation from its mean. • This eliminates the regression to the mean that should be present when there is error in the input information. So people’s forecasts appear to ignore this error. Harvey Evidence 2/11/4

21. Evaluation versus forecasting: Kahneman & Tversky (1973) • Current evaluation of information should be less subject to error than use of that information to make forecasts (cf forecast horizons). Regression to the mean should be greater in the latter case. But it will not be if representativeness is used to make both judgments. • Both groups receive descriptions of students. One group uses them to estimate ‘the % of students in the class whose descriptions indicate higher ability’. The other group forecasts the student’s GPA and class standing in percentiles at the end of the freshman year (ie percentile GPA). Harvey Evidence 2/11/4

22. Evaluation versus forecasting: Kahneman & Tversky (1973) • The two groups produced equally extreme judgments. Standard deviations of judgments did not differ between them. • Replicated by Beyth for current evaluations vs forecasts five years hence. • Pro representativeness. Harvey Evidence 2/11/4

23. Translation versus forecasting: Kahneman & Tversky (1973) • There should be no regression when one variable is merely translated from one scale to another. Group 1 was given percentile GPAs for 10 students and then estimated their GPAs. (A percentile GPA of 65% means that the GPA is better than 65% of other students.) • Group 2 forecast GPAs from scores on a test of mental concentration they were told was unreliable and affected by other factors (eg mood, sleep loss, etc). • Group 3 forecast GPAs from scores on a test of sense of humour that was only approximately related to GPA. Harvey Evidence 2/11/4

24. Translation versus forecasting:Kahneman & Tversky (1973) • Normatively Groups 2 & 3 should be regressive but Group 1 should not be. • Representativeness would cause Groups 1 & 2 to be non-regressive. If SOH is not seen as representative of academic ability, Group 3 would show regression. • Note leniency in forecasts. Harvey Evidence 2/11/4

25. Forecasting many outcomes from values of another variable • In this case, the set of forecasts represents distributional information about the outcomes. This information may be given as feedback, presented as a set of past outcomes, or assumed. • So noise as well as pattern is included in forecasts. • In probability matching experiments, cards (80% red, 20% black) are shuffled and turned over one at a time. Forecasts for the next colour are 80% red and 20% black rather than 100% red. Harvey Evidence 2/11/4

26. Representativeness in forecasting many outcomes from another variable • Gray et al (1965) showed people many paired X and Y values (numbers or line lengths) that correlated .96, .75 or .44. Next they predicted nine Y values for each of nine X values. Correlations between predictions and X should have been unity but were .94, .78 and .53 in one experiment and .94, .70, and .45 in another. • Gray (1968) showed people’s predictions even replicate the distribution (unimodal vs rectangular) of Y values across each X value. Also spread betting on predictions during training sessions had no effect during testing. Harvey Evidence 2/11/4

27. Representativeness: Forecasts for many outcomes (Harvey, 1995) People are presented with data (red) and make several forecasts (black). These are not in the most likely positions for future points (the regression line); they represent a sequence that is as noisy as the one that will occur. Noisier data give noisier forecast sequences. Harvey Evidence 2/11/4

28. Forecasting using past history of the variable: Summary • Anchoring and adjustment • Series with linear and exponential trends • Series with independent data • Natural series • Overconfidence: An anchoring effect? • Under-use of decision aids and advice: Another anchoring effect? Harvey Evidence 2/11/4

29. Forecasting using past history of the variable: Anchoring and adjustment • The anchor may be the last data point in the time series or an earlier forecast. Underadjustment can produce biases. • Because forecasts are too close to the last data point, linear and exponential trends appear damped and independent data autocorrelated (eg Bolger & Harvey, 1993, Eggleton, 1982; Lawrence & Makridakis, 1989; Harvey & Bolger, 1996; Sanders, 1992, Wagenaar & Sagaria, 1975). Harvey Evidence 2/11/4

30. Anchoring leading to damping of linear trends People are presented with data (red) and make a forecast (black). On average, the last datum is on the trend line. People use this as an anchor and make an insufficient adjustment for trend. Thus damping is observed. Harvey Evidence 2/11/4

31. Anchoring leading to non-independent forecasts from independent data • People are presented with independent data points (red) and make a forecast (black). They use the last data point as an anchor and make an insufficient adjustment towards the mean. Thus they appear to perceive sequential dependence in the data when there is none. Harvey Evidence 2/11/4

32. Lack of underadjustment with natural series (Lawrence & O’Connor (1995) • Natural series tend to have decelerating trends and positive autocorrelation. Thus, on average, adjustment is likely to be appropriate with these series. • Lawrence and O’Connor (1995) found that, on average, underadjustment did not occur with the 111 time-series in the 1982 M-competition. People have evolved to fit the statistical properties of their natural environment. • We still expect natural series with less than the average autocorrelation to show underadjustment and greater than the average autocorrelation to show overadjustment. Harvey Evidence 2/11/4

33. ‘Overconfidence’ in forecasts: Anchoring again? • People are presented with data (red), make a forecast (black), and then set an (eg 80%) prediction interval. They use their forecast as an anchor. Underadjustment from this point results in the interval being too narrow. Thus they appear overconfident. Harvey Evidence 2/11/4

34. Underestimation of prediction interval widths: Problems for anchoring • Block & Harper (1991) found less not more bias when forecasts were explicit rather than implicit. • Fiedler & Juslin (in Press) report greater overconfidence when an interval is produced than when one is evaluated. In the latter case, proportion in the sample is used to estimate proportion in the population (OK). In the former case, dispersion in the sample is used naïvely to estimate dispersion in the population (not OK). • Their naïve sampling model can be interpreted as an account of overconfidence in terms of representativeness. Harvey Evidence 2/11/4

35. Using statistical forecasts to modify judgmental ones: Underadjustment again? People are presented with data (red), make an initial forecast (black), are given a good statistical forecast that has proved better than their own in the past (blue), and make a final forecast (green). Underadjustment away from their initial forecast depresses their performance. (Their own forecast is weighted twice as much as the statistical one.) Harvey Evidence 2/11/4

36. Insufficient use of advice: Another case of underadjustment? • When people make an initial forecast and then take advice from a more expert source (decision aid, expert system, superior judge), they do not take sufficient account of that advice (Harvey & Fischer, 1997; Lim & O’Connor, 1995, 1996). • This may be a case of underadjustment from an anchor. • For Krueger (2003), it demonstrates egotism (cf Svenson, 1981). (Harvey & Harries (2004) try to disentangle anchoring and conservatism (egotism) effects in forecasting and find some evidence favouring Krueger.) Harvey Evidence 2/11/4

37. Adaptiveness of anchoring and adjustment and representativeness in forecasting • Use of heuristics need not produce biases. We have seen how availability and anchoring & adjustment can exploit natural features of our environment and ourselves to allow us to make effective judgmental forecasts. Representativeness can also be appropriate (eg probability matching allows effective sharing of resources across consumers - cf Minkowski’s fish). • Heuristics may be less appropriate when applied to atypical series or ‘unnatural’ (eg financial) series characteristic of modern technological societies. Harvey Evidence 2/11/4

38. So much more to discuss… • Overforecasting/optimism (Eggleton, 1982: Lawrence & Makridakis, 1989; Harvey & Bolger, 1996). Related to ‘leniency ‘ effects? • Data format effects (Angus-Leppan & Fatseas, 1986; Dickson et al, 1986; Lawrence, 1983; Lawrence et al, 1985; Harvey & Bolger, 1986). • Series length effects (Lawrence & O’Connor, 1992). • Forecasting with more than one type of information (Goodwin & Fildes, 1999; Harvey et al, 1994). Harvey Evidence 2/11/4

39. FIN