1 / 1

The Impact of Probabilistic Information on Deterministic & Threshold Forecasts Susan Joslyn, Earl Hunt & Karla

The Impact of Probabilistic Information on Deterministic & Threshold Forecasts Susan Joslyn, Earl Hunt & Karla Schweitzer University of Washington. EXPERIMENT

dugan
Download Presentation

The Impact of Probabilistic Information on Deterministic & Threshold Forecasts Susan Joslyn, Earl Hunt & Karla

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The Impact of Probabilistic Information on Deterministic & Threshold Forecasts Susan Joslyn, Earl Hunt & Karla Schweitzer University of Washington • EXPERIMENT • OUR QUESTION: Can weather forecasters incorporate uncertainty information expressed in probabilities into a non-probabilistic forecasting decision in a way that improves the forecast? Without bias? • Background • How do people understand and use probabilistic information? • People, in general, do not treat probability linearly • (Gonzalez & Wu, 1999) • Even experts have trouble incorporating prior probabilities • (Eddy 1982 ) • However weather forecasters are good at estimating probabilities of e.g precipitation • (Baars & Mass, 2004) • Probabilistic information is increasingly important product of ensemble forecasts. • • Yet few operational weather forecasters make use of available probability products • (Joslyn, Jones & Tewson, 2005) PROCEDURE 10 Participants: Atmospheric Science Students Task: Forecast wind speed and direction for a 48 hour period. .Decide whether to issue high wind advisory (20 + knots). Forecasts were made for 4 days for four different locations on each day. • Information Provided • Historical data for all products used in previous cognitive task analysis (Joslyn et al, submitted) • Radar Imagery • Satellite Imagery • TAFs and current METARs • Model output (AVN, MM5 & NGM) • Design: Within subjects • • Each subject made 2 forecasts with the probability product, • 2 forecasts without probability product • • All subjects saw weather data from all of the same dates • • Probability product rotated through the dates, making sure that no subject saw the same date twice • Manipulation • On 1/2 of the forecasts, participants had the • MM5 ACME Ensemble Probability of winds • greater than 20 knots • Instructions: • Free choice of products • Except: forecasts with the probability product • Required to read & record the range provided by the probability product RESULTS Wind speed forecast: Deterministic forecast R2 Predicting observed from forecasted wind speeds • We conducted regression analyses predicting observed wind speeds from forecasted wind speeds. See R2 for each condition in the table to the right. • For 3 dates people did better with the probability product • Note especially 3/21 which was the most difficult and for which the probability product made a significant difference. Posting wind advisories: Thresholdforecast • d with and without the probability product were similar • However, WITH the probability product: • the criterion was higher (less willing to post advisory) • Fewer advisories overall. • Same forecasters/same weather • Difference must be due to probability product CONCLUSIONS Probability product had a slight impact on the deterministic wind speed forecast. Wind Advisory Forecasts: • Forecasters, in general, had a liberal bias in the low to mid probability ranges: Biased to post wind advisories • Makes sense in this task: being cautious, keeping people off the water if there is any chance of danger • However, boaters may come to disregard the advisory if it often proves to be a false alarm (Roulston & Smith, 2004) • Probability product attenuated this tendency without causing them to lose sensitivity--they weren’t more likely to miss high wind situations • Forecasters, in general, had a conservative bias in the highest probability ranges • They did not post advisory as often as they should have • Probability product attenuated this tendency • more advisories in high range with probability product References Gonzalez, R., & Wu, G. (1999). On the form of the probability weighting function. Cognitive Psychology, 38, 129-166. Eddy, D.M. (1982). Probabilistic reasoning in clinical medicine: Problems and opportunities. In D. Kahneman, P. Slovic, & A. Tversky (Eds), Judgment under uncertainty: Heuristics and biases pp. 249-267) Baars, J. A. & Mass, C. F. (2004) Performance of National Weather Service Forecasts Compared to Operational, Consensus and Weighted Model Output Statistics. Joslyn, S. Jones, D.W. & Tewson, P.(2005) Designing Tools for Uncertainty Estimation in Naval Weather Forecasting. 7th International Conference on Naturalistic Decision Making Roulston, M.S. & Smith, L. A. (2004) The boy who cried wolf revisited: The impact of false alarm intolerance on cost-loss scenarios. Weather & Forecasting (19) 391-397. Probability product • Good in very low probability situation • Overforecast in the midranges • In general, improved subjects performance Subjects tended to • Over forecast in low-mid probability situations • Underforecast in high probability ranges • Attenuated with probability product This research was supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745

More Related