Who Nose What Tomorrow Brings?

1. Who Nose What Tomorrow Brings? David J. Weiss

2. Some Predictions from Swami Weiss • There will be a severe hurricane threat to Florida during the second week of October 2014 • The LA Lakers will fail to make the playoffs in the 2013-2014 season • The married folks in the audience will buy an expensive gift for their spouse during December 2014 • During 2015, Michael Birnbaum will publish an excellent paper presenting results that cannot be accounted for by SEU or prospect theory

3. The Expert Forecaster • Weatherperson, sports bookmaker, investment advisor, intelligence analyst, safety engineer, personnel analyst, admissions director, marriage counselor, parole board member, custody judge • The first two professions make predictions about short-term, unitary events. • The others make predictions about events that will play out over a relatively long term. They also usually recommend actions based on their probability estimates.

4. Everyone Forecasts • Amateurs also make predictions. • Are the professionals really expert? • How can we tell?

5. Three General Approaches • Credentials • Experience • Performance-Based Assessment • Scoring outcomes (prediction accuracy) • This talk examines some of the challenges in scoring outcomes (Hammond’s “Correspondence”)

6. Technical Matters • Specificity of the prediction • Determining whether the predicted event did in fact occur • Duration of the observation period. For a prediction that unfolds over time, the ultimate result can change • These ambiguities can usually be resolved (Tetlock), but in practice are often overlooked • Announcing the prediction can affect the outcome

7. Scoring Index • Percent correct (batting average) is the easy solution, but • Scoring over the person’s predictions assumes the events are comparable • (as Moneyball highlighted, baseball batting average has a similar shortcoming) • Some of the Swami’s predictions may be more likely to come true than others

8. The Playing Field is Not Level • Base-rate differences • Weather is easy to predict in Fullerton • The road not taken • Almost every applicant accepted by Princeton graduates, not so at Cal State • Would Princeton’s rejects have graduated from Cal State?

9. Calibration Saves the Day? • When an expert repeatedly makes predictions of similar events, one can evaluate accuracy at a finer level. • Calibration imposes a different standard from batting average • “Well-calibrated” sounds like “expert” • Is calibration a clever way to discount errors?

10. Remembering Our Founder • Experts and the public resist forecasts expressed in probabilistic terms • They may be right to do so, because calibration makes sense only when both kinds of errors are equally costly. As Ward kept telling us, utilities are the proper basis for decisions. This limitation applies similarly to sophisticated “skill scores” such as those of Murphy (1988) and Stewart and Lusk (1994). • Predictions also vary in importance

11. The “Bold Prediction” • If calibration overweights the mundane, then should we judge forecasters by how well they predict the spectacular? We might care more about predicting hurricanes than partly cloudy days • Be sure to address false alarms (crying “wolf” too often leads to being ignored) • Because rare events are, well, rare, they provide little data

12. Is Forecasting Even Possible? • Taleb’s turkey highlights the danger in using past results to predict future outcomes • But what else is there to guide us but the past? • Perspective matters: When the event to be predicted is under the control of a human (such as killing a turkey or planting a bomb), someone with knowledge about that human might be able to predict it without historical information.

13. An Expert Turkey • Learns about the farmer’s (or other farmers’) plans • Observes that there are no old turkeys around, and draws an inference • These methods do not use observations of the focal event (previous turkey beheadings) to predict the future.

14. Two Kinds of Environment (Taleb) • Mediocristan, where single observations do little to change the aggregate. Processes are stationary, and statistical models are descriptive. • Regression-based prediction works in Mediocristan. Experts can potentially use better prediction models. Better can mean either a more accurate model or better parameter estimates. One can learn from results.

15. The More Challenging Environment • Extremistan, where the total can be significantly impacted by a single observation • The world of the Black Swan, where the past is not a good guide to the future • Even successful prediction may represent a case of being fooled by randomness (no re-test is available)

16. The Environment and Evaluation • Scoring outcomes is feasible in Mediocristan. Probabilistic forecasts can be compared to observed relative frequencies. • Aggregation over instances is meaningful because the utilities are comparable. • But not so in Extremistan, where a single inaccurate prediction may be much more consequential than a host of accurate ones. • Probabilistic forecasts in Extremistan are opinions, and cannot be compared to observed frequencies

17. Which Environment Do Experts Inhabit? • Both, of course. But Extremistan is where some really important events reside, and where accuracy of previous predictions need not be informative. • While weather forecasting is certainly useful, it is not typical of the kind of forecast we get excited about, namely predicting the black swan. For example, the furor over the failure of US intelligence to anticipate 9/11 was an indictment of prediction in Extremistan. • Predicting back swans that are the result of human action calls for getting an insider’s perspective. Meehl’s “broken leg” cue, insider trading, infiltration

18. Sorry, Ken Hammond • Because scoring outcomes in Extremistan is so problematic, I suggest evaluating performance by examining coherence (process) instead. • The expert turkey was able to predict catastrophe by doing what looks like what intelligence officers are supposed to do – learning about plans (spying) and drawing causal inferences from observations.

19. Evaluating Coherence • If we think we know the correct process, we could evaluate the expert on the basis of adherence to that process. • If we lack that confidence, what can be done? • We can examine the discrimination and consistency in the predictions (you may have heard this before). All this really means is that predictions should be responsive to the relevant evidence.

20. Reality Check • It is unlikely that users of predictions will be satisfied to know that their forecaster discriminates and is consistent. People like correspondence. Most will demand a track record of accurate predictions. • Unfortunately, such a record is unlikely ever to be available in Extremistan.

21. Cart Before Horse • Millions of tax dollars are currently being spent on an IARPA project whose goal is to determine how best to aggregate predictions made by individual forecasters. • IMHO, that money is going down a well. Without better understanding of how to evaluate predictions in Extremistan, one cannot say whether one aggregation method is better than another. IARPA assumes all environments are Mediocristan.

22. Summary • Batting average, while basically appropriate for evaluating expert predictors in Mediocristan, should be improved by incorporating utilities • Calibration, a sophisticated version of batting average, might be generalized to include utilities. • Probabilities cannot be compared to observed frequencies in Extremistan. Predictors need to realize when they have crossed the border.