Towards an optimized communication of probabilistic weather forecasts via the internet

1. Towards an optimized communication of probabilistic weather forecasts via the internet Liz Stephens1, Ken Mylne and David Spiegelhalter2 1. University of Bristol, 2: University of Cambridge

2. Project Brief • review existing ways of representing probabilistic information • develop online surveys to ascertain user preferences for specific designs • analyse the outputs • provide recommendations regarding which designs work best with people from different demographic profiles Intro Defining project Designing representations Developing game Running game Outcomes Summary

3. Current representations UK Met Office Intro Australia Probcast (UW) yr.no Defining project Designing representations Metcheck Developing game Running game Outcomes Summary

4. Objectives, scope and limitations • Can players make better decisions? • How does this vary with presentation type? • Demographic influences • Preference vs understanding Addressing assumptions: “the public don’t understand uncertainty” “uncertainty is too difficult to communicate” • …everyday weather not extremes • …basic representations • …no implicit representations • …temperature and rainfall only • …6 months Intro Defining project Designing representations Developing game Running game Outcomes Summary

5. Intro Defining project Designing representations Developing game Running game Outcomes Summary

6. Deterministic • Basic Rating • Percentage (rounded) • Range of values & percentage • Probability distribution & percentage Intro Defining project Designing representations Developing game Running game Outcomes Summary

7. Maximise participants Start & finish game Want to do well To share with friends Appropriate Consider theme, style and complexity Not too specific Appeal to all demographics Useful tests Correct decision (relative) Understanding of actual confidence Realistic Associate with real weather situations Hint at utility of uncertainty information Intro Defining project Designing representations Developing game Running game Outcomes Summary

8. Age, postcode, gender… Randomly assign presentation type Multiple questions: expect different responses depending on presentation type Outcomes generated on-the-fly (based on probabilities) Only look at 1st time players Quadratic scoring rule Intro Defining project Designing representations Developing game Running game Outcomes Summary

9. Playing the game How confident are you about each shift – tests understanding of presentation

10. Scoring • Each shift is allocated an outcome randomly selected from the forecast distribution • Element of luck! • Amount of ice cream bought and sold depends on advice and outcome • Total score at end of game:

11. Outcomes >8000 people – largest survey of its kind (aiming for 10,000+) Hope to be able to: • Identify the percentage of correct answers from each presentation type • Identify the skill score of confidence ratings for each presentation type • Link this with demographics Intro Defining project Designing representations Developing game Running game Outcomes Summary

12. Early Results Brad makes more money with probabilistic forecasts! Intro Defining project Deterministic Probabilistic Designing representations Developing game Running game Outcomes Summary

15. Intro Defining project Under 16 16-24 25-34 45-54 55+ 35-44 Designing representations Developing game Running game Outcomes Summary

16. Summary • Game carefully designed to answer a few simple questions • Large sample achieved across different demographics • Early results show benefit of probabilistic forecasts • Detailed analysis of different presentations is underway • Men over 55 do spectacularly badly! • You have a few days left to play: • www.metoffice.gov.uk/weather/weather-game • Not too late to win a T-shirt! • liz.stephens@bristol.ac.uk Thanks! Intro Defining project Designing representations Developing game Running game Outcomes Summary