Leonard A. Smith & Mark S. Roulston Centre for the Analysis of Time Series London School of Economics & Pembroke

2. Embracing Probability Forecasts on All Scales: Formulation, Communication, Value & Evaluation (End-to-End Forecasting) Leonard A. Smith & Mark S. Roulston Centre for the Analysis of Time Series London School of Economics & Pembroke College, Oxford www.maths.ox.ac.uk/~roulston www.lse.ac.uk/collections/cats

3. Forecast Based Decision Making: • Is there any information in the forecast? • How can I best extract that information? • Will anyone listen? • How to best communicate with rich, numerate users? • How to best communicate with the general public? • What to communicate the numerate managers of the public? • What (exactly) is the decision I am trying to make? • Time scale (short, medium, seasonal, climate). • Number of expected events given duration of interest . • Who is the user? (evacuation? insurance? or building?)

4. Main Points: Risk • The most useful flood forecasts will be probability forecasts. • Especially on longer time scales; where is there evidence of skill? • Useful probability forecasts require end-to-end forecasting. • What is the baseline for current societal and economic usage? • Is there (flood relevant) value in current (ensemble) NWP forecasts? • Standard targets need not apply (heave provides an example). • Standard methods may not be optimal (precip forecasts in a product space). • How can we follow uncertainty and inadequacy in compound models (end-to-end)? Reaction • Economic value: Cost/Loss ratios apply to rich, rational, focused users. • To have societal value requires a response as well as a warning. • Realistic response models are required, providing new morals for: • The boy who cried wolf • And Noah What is the goal of an Operational Warning System?

5. Cost Loss Evaluation: Binary Events • Assume a rich numerate user subject to: • A cost C to protect against event E; • A loss L if no protection is taken and event E occurs; • Zero loss if protection is taken. • It follows that action should be taken if the objective probability of E is greater than C/L, assuming: • The user is interested in the long run (rich); • The user faces a binary choice (focused); • The forecasts are accurate PDFs; • The cost of the forecasts is negligible. • Note an ensemble of size N is neither necessary nor sufficient for PDF resolution 1/N.

6. Beyond Cost/Loss a)There may be no natural binary alternative. b) If the cost is near the cost of ruin, it can be rational to ignore the forecast. c) In a societal application (an evacuation), the probability of action will show hysteresis. We will touch each of these in turn.

7. Weather Roulette: A Simple Example Each day you gamble your entire net worth on the temperature at Heathrow. The amount you place on each outcome is proportional to your predicted probability of that outcome (Kelly Betting). How would the ECMWF ensemble (EPS) fare against a house that set odds: - using climatology? - using Best Forecast Guide (BFG) from the ECMWF hi-resolution forecast? This provides a good analogue for statistical decision making.

9. Ideally, this calculation is done under the user’s utility function

10. Many economic users already effectively play weather roulette: • They would be happy with probability forecasts. • Can we deliver accountable PDFs? • Can we value operational PDFs? • Flood forecasting cuts across different models: • How can we track uncertainty end-to-end across modelling communities? No, but assume good PDFs. Yes, via Ignorance.

11. A forecast like this one is of great value, even of we cannot interpret it as a PDF. How do we interpret these scenarios? or pass on the information in them?

12. Interpreting Simulations New methods of ensemble interpretation may extract existing information. Skill at day 8. RMS skill scores are simply not relevant to extreme events (or events with integrated triggers) Green > 80% 80> Blue > 30% 30> Red > 0%

13. Some users already value probability forecasts; their decision is then one of skill/cost between forecasts. Bonga Floating Production Storage and Offloading vessel

14. Postage stamp forecasts can be provided in the users variables: from significant wave height near a buoy to heave at the FPSO.

15. The simulation(s) become a forecast when “dressed” to form a PDF. In this case, the ensemble forecast has little marginal value given the BFG. The relevant storms have occurred before the forecasts are made. EPS dynamical ensemble is blue, dressed ensemble is red, verification (buoy) isgreen

16. At other locations, end to end forecasts extract information from ensembles members which is unobtainable from any single BFG. Draugen

17. At Draugen local variations have impact and the dressed EPS reflect options which the BFG misses. Dressed EPS bounds truth. EPS dynamical ensemble is blue, dressed ensemble is red, verification (buoy) isgreen

18. Dressed BFG has a higher ignorance score. Large unexpected waves. BFG dynamical ensemble is blue, dressed hi-res forecast is red, verification (buoy)green

19. Realistic Societal Response 1) The boy who cried wolf: • 6 villagers for 1 hour at $10/hour • 3 sheep at $200/sheep -> C/L = 0.10 Yet the Villagers were unprepared to accept a 67% false alarm rate! Moral of the Story: If societal benefit is the aim, one must consider imperfect compliance when events are rare.

20. Realistic Response To Rare Events 2) The case of Noah: • Unusual event forecast from trusted source. • Huge cost C. • Unbounded Loss L. • One off gamble (this user will never face this event twice, esp if no protection is taken), -> C/L = ???? Yet the taking action proved worthwhile. Moral of the Story: The maths become irrelevant to rational action if the forecasts are not believed (or paid for), the stakes too large, or the costs too high.

21. Open Questions How can we increase/identify forecast value? - Better communication of end-to-end uncertainty in compound models. - Better forecast archives for every operational model. - Active (adaptive) model/ensemble response to the previous forecast. Alternatives to single model IC ensembles: - multi-model ensembles, - multi-parameterisation models, - product space interpretations, - novel approaches (especially for longer range forecasts). Different users have different needs/horizons. How to deal with model inadequacy in the climate change scenario? If model inadequacy kills an accountable probability forecast strategy in the same way that uncertainty killed the single hi-resolution forecast strategy, then how should we evaluate our models?

22. Discussion Questions: • What is the goal of an Operational Warning System? • How to propagate uncertainty across families of models? • And between families of researchers? • How to quantify the value society currently derives? • How much information do current forecasts contain? • How to transfer information to industry with maximum value? • How to transfer information to society with maximum value? For the maths, see: www.maths.ox.ac.uk/~roulston