1 / 59

Understanding and Using Uncertainty Information in Weather Forecasting

Understanding and Using Uncertainty Information in Weather Forecasting. Susan Joslyn University of Washington. Acknowledgements. Earl Hunt David Jones Limor Nadav-Greenberg John Pyles Adrian Raftery Karla Schweitzer McLean Slaughter Meng Taing Jeff Thomasson

shasta
Download Presentation

Understanding and Using Uncertainty Information in Weather Forecasting

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Understanding and Using Uncertainty Information in Weather Forecasting Susan Joslyn University of Washington

  2. Acknowledgements Earl Hunt David Jones Limor Nadav-Greenberg John Pyles Adrian Raftery Karla Schweitzer McLean Slaughter Meng Taing Jeff Thomasson This research was supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745

  3. Forecast Uncertainty • Available for some time • Rarely communicated in public forecasts • Underused by weather forecasters

  4. Forecast Uncertainty • Difficult to understand - Forecasters claim • People make mistakes when reasoning with probability • Format: Frequency (1 time in 10) is better than Probability (10% chance)

  5. Forecast Uncertainty • Useful for deterministic forecasts decision? Theoretically Practically useful? • It doesn’t matter how good the information if people can’t or won’t make use of it.

  6. Goals for Psychology Team  • Establish uncertainty information is useful Threshold forecast (forecasters & general public) - high wind advisory for boater safety •What is best presentation format to enhance Understanding? Decisions?

  7. Three Major Lines of Inquiry 1. Does probability information improve threshold forecast? Study 1 2. Does display format (visualization) matter? Study 2 3. Does the wording matter? Studies 3-4 (probability/ frequency)

  8. Study1Does Probability Information Improve Threshold Forecast? Participants: Advanced atmospheric science students Task: • Forecast wind speed and direction • Decide whether to issue high wind advisory (winds > 20 knots)

  9. Historical data Radar Imagery Satellite Imagery TAFs and current METARs Model output (AVN, MM5 & NGM) Historical data Radar Imagery Satellite Imagery TAFs and current METARs Model output (AVN, MM5 & NGM) + Chart showing probability of winds > 20 k Within Subject Design Condition 2 Condition 1

  10. Probability of Winds ≥ 20k

  11. Historical data Radar Imagery Satellite Imagery TAFs and current METARs Model output (AVN, MM5 & NGM) Historical data Radar Imagery Satellite Imagery TAFs and current METARs Model output (AVN, MM5 & NGM) + Chart showing probability of winds > 20 k Within Subjects Design Condition 2 Condition 1 • Same participants, same weather • Only difference is probability product

  12. Results Threshold Forecast: • People posted fewer wind advisories with probability product. • Similar ability to discriminate between high wind and low wind event (sensitivity).

  13. Results: Percent Advisories Y= % times forecasters posted advisory X= probability of winds > 20K

  14. Conclusion: Uncertainty Information IS Beneficial for Threshold • Increased advisories when high winds were very likely • Decreased advisories when high winds were unlikely-fewer false alarms • Increase trust in warnings!

  15. Study 2Does Display Format Matter? • 3 different visualizations of 90% predictive interval • Range of likely wind speeds • All conditions included median wind speed chart deterministic forecast

  16. 3 Visualizations: Between subjects 1. 90% Upper bound: warmer colors = higher wind speed •“ observed wind speeds will be higher only 1 time in 10” • worse case scenario: highest likely winds

  17. 3 Visualizations 1. 90% Upper bound: • wind speeds will be higher only 1 time in 10 warmer colors = higher wind speed 2. Margin of error: • range of wind speeds between UB & median • display of uncertainty in the forecast warmer colors = more uncertainty

  18. 3 Visualizations 1. 90% Upper bound: wind speeds will be higher only 1 time in 10 warmer colors = higher wind speed 2. Margin of error: range of wind speeds between upper bound and median warmer colors = more uncertainty 3. Box plot: Wind speed in knots WindSpeed in knots 90% Upper bound 90% lower bound median

  19. Method • Participants: Atmospheric Science students (replicated on NOAA Forecasters) • Practice: Learned how to read charts • Test: - Forecast wind speeds - Threshold: high wind advisory (winds >20 knots) - Rate uncertainty in forecast

  20. Results: Wind Speed Forecast BoxPlot 1.17 Upper bound 2.02 1.55 Margin of Error Knots above the Median • UB forecast significantly higher wind speeds • Display provided a high anchor (Tversky & Kahneman, 1982)

  21. Results: High Wind Advisories People in the box plot condition: • posted significantly more advisories • most in high likelihood situations

  22. Results: Uncertainty Rating • MoE best for detecting relative uncertainty • They learned: “The wider the range the greater the uncertainty” correlation • Ratings in the MoE significantly more highly correlated to range

  23. Conclusion: Format Matters Box Plot better threshold forecast wind speed: no bias (salient high and low anchors) MoE detect relative uncertainty in forecast Upper higher winds speeds: bias(anchor) Boundno benefit to threshold forecast

  24. Study 3 & 4 Does Wording Matter? • Participants: Psychology undergraduates • Frequency is easier to understand than probability(Gigerenzer, 1995, 1999, 2000) • Research on complex problems • Is that true of simple expressions of uncertainty?

  25. Does Wording Matter? There is a 10% chance that the wind speeds will be greater than 20 knots.

  26. Method Procedure: Fill out questionnaire rating expressions of uncertainty Decide whether or not to post a high wind advisory Suppose that there is a 10% chance that the wind speeds will be greater than 20 knots. “How likely are the wind speeds to be greater than 20 knots? (please fill in a bubble)” Very Unlikely Very Likely O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Would you issue a small craft advisory (winds equal or greater than 20 knots)? ___Yes ___No

  27. Method Procedure: Fill out questionnaire rating expressions of uncertainty Decide weather to post a wind advisory Suppose that there is a 10% chance that the wind speeds will be greater than 20 k. “How likely are the wind speeds to be greater than 20 knots? (please fill in a bubble)” Very Unlikely Very Likely O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Would you issue a small craft advisory (winds equal or greater than 20 knots)? ___Yes ___No

  28. Method Procedure: Filled out questionnaire rating expressions of uncertainty Decide weather to post a wind advisory Suppose that 1 time in 10 the wind speeds will be greater than 20 knots. “How likely are the wind speeds to be greater than 20 knots? (please fill in a bubble)” Very Unlikely Very Likely O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Would you issue a small craft advisory (winds equal or greater than 20 knots)? ___Yes ___No

  29. Study 32 Variables: Wording & Likelihood Probability Frequency 10% chance= 1 time in 10 90% chance= 9 times in 10

  30. 1 time in 10 wind speeds =9 times in 10 wind speeds will be greaterthan 20 knots will belessthan 20 knots Study 3:Likelihood of High Wind Held Constant

  31. Results: Reversal Error • Rate from wrong side of scale Suppose that there is a 90% chance that the wind speeds will be less than 20 knots. “How likely are the wind speeds to be less than 20 knots? (please fill in a bubble)” O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O <---very unlikely very likely ------> • They completely misunderstand the phrase • Most in “90% (9 in 10) less than” wording • Which is it? High likelihood? Less than? Reversal error

  32. Less Greater 10% chance less10 % chance greater Study 4 Manipulated Less / Greater

  33. Less Greater 10% chance less 10 % chance greater 1 in 10 less 1 in 10 greater 30% chance less30% chance greater 3 in 10 less3 in 10 greater 70% chance less70% chance greater 7 in 10 less7 in 10 greater 90% chance less 90% chance greater 9 in 10 less 9 in 10 greater Added 2 levels of likelihood

  34. Less WordingGreater Wording 10% chance less 10 % chance greater 1 in 10 less 1 in 10 greater 30% chance less 30% chance greater 3 in 10 less 3 in 10 greater 70% chance less 70% chance greater 7 in 10 less 7 in 10 greater 90% chance less 90% chance greater 9 in 10 less 9 in 10 greater Equivalent Expressions

  35. Less WordingGreater Wording 10% chance less 10 % chance greater 1 in 10 less 1 in 10 greater 30% chance less 30% chance greater 3 in 10 less 3 in 10 greater 70% chance less 70% chance greater 7 in 10 less 7 in 10 greater 90% chance less 90% chance greater 9 in 10 less 9 in 10 greater Equivalent Expressions

  36. Results: Reversal Error More often in “less than” wording (4x as likely) Mean reversal error per person High vs. low likelihood does not matter Frequency wording does not help

  37. Results: Wind Advisories 30% 70% 90% 10%

  38. Results: Wind Advisories 30% 70% 90% 10%

  39. Results: Wind Advisories 30% 70% 90% 10%

  40. Results: Probability “less” is worst 30% 70% 90% 10% 90% 10% 30% 70% Reversal error subjects eliminated from this analysis

  41. Conclusion: Wording Matters • “Less than” wording is difficult (reversal errors) • Wind speed advisories in “probability less” - too many advisories in low ranges - too few in high ranges • Frequency protects against posting errors generated by “less than” wording

  42. Conclusions • Probability information improves threshold forecasts • Many end-user weather decisions are yes/no threshold decisions • The right display format • Improves understanding • MoE communicates relative uncertainty • Improves weather decisions • Box Plot increases warnings in high likelihood • Box Plot unbiased wind speed forecast • Wording matters • “Less than” is confusing • Frequency helps sometimes • NOT in reversal errors • HELPS in posting advisories

  43. The End

  44. Results: Percent Advisories Y= % times forecasters posted advisory X= probability of winds > 20K

  45. Results: Percent Advisories • Y= % times forecasters posted advisory • X= probability of winds > 20K

  46. Results: Percent Advisories • Y= % times forecasters posted advisory • X= probability of winds > 20K

  47. Results: Percent Advisories • Y= % times forecasters posted advisory • X= probability of winds > 20K

  48. Study 1: Rating • 10% was rated significantly higher Probability condition: 10% chance (M=1.32) 90% chance (M=.99) O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O Frequency condition: 1 in ten (M=1.06) 9 out of 10 (M=.98) O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O

  49. Study 2: Rating 10 was rated higher--did not reach significance 10% (1 in 10) greater (M=1.25) 90% (9 in 10)less (M=.97) O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O 10% (1 in 10) less (M=.98) 90% (9 in 10)greater (M=.88) O-------O-------O-------O-------O-------O--------O-------O-------O-------O-------O

  50. Study 1: Reversal Error Mean reversal error per person

More Related