Verification of the Monte Carlo Tropical Cyclone Wind Speed Probabilities:

1. Verification of the Monte Carlo Tropical Cyclone Wind Speed Probabilities: A Joint Hurricane Testbed Project Update John A. Knaff and Mark DeMaria NOAA - NESDIS - Office of Research and Applications, Fort Collins, CO Chris Lauer NOAA - NWS – Tropical Prediction Center, Miami, FL 61st IHC, New Orleans, LA

2. TPC Monte Carlo Wind Probabilities Hurricane Gordon Hurricane Helene 61st IHC, New Orleans, LA

3. Motivation for Verification • Assess how good the current wind probability algorithm performs • Assess future improvements to the wind probability algorithm • Another way to track the performance of the deterministic forecasts 61st IHC, New Orleans, LA

4. QUESTIONS How does the average forecast magnitude compare to the average observed magnitude? What is the magnitude of the probability forecast errors? What is the relative skill of the probabilistic forecast over the deterministic forecasts, in terms of predicting whether or not an eventoccurred? How well do the predicted probabilities of an event correspond to their observed frequencies? What is the ability of the forecast to discriminate between events and non-events? STATISTICS THAT ANSWER … Multiplicative Bias (half) Brier Score Brier Skill Score Reliability Diagrams Relative Operating Characteristics How good are the Monte Carlo wind probabilities? 61st IHC, New Orleans, LA

5. 34-kt Probability Grids Hurricane Gordon Hurricane Helene 61st IHC, New Orleans, LA

6. 34-kt OFCL Forecast Hurricane Gordon Hurricane Helene 61st IHC, New Orleans, LA

7. 34-kt Best Track Hurricane Gordon Hurricane Helene 61st IHC, New Orleans, LA

8. QUESTIONS How does the average forecast magnitude compare to the average observed magnitude? STATISTICS THAT ANSWER … Multiplicative Bias How good are the Monte Carlo wind probabilities? 61st IHC, New Orleans, LA

9. Results: Multiplicative Biases Atlantic, July 18 – October 4, 2006; 105W – 1W, 1N – 60N Range: minus infinity to infinity. Perfect score: 1 61st IHC, New Orleans, LA

10. QUESTIONS How does the average forecast magnitude compare to the average observed magnitude? What is the magnitude of the probability forecast errors? What is the relative skill of the probabilistic forecast over the deterministic forecasts, in terms of predicting whether or not an eventoccurred? How well do the predicted probabilities of an event correspond to their observed frequencies? What is the ability of the forecast to discriminate between events and non-events? STATISTICS THAT ANSWER … Multiplicative Bias (half) Brier Score Brier Skill Score Reliability Diagrams Relative Operating Characteristics How good are the Monte Carlo wind probabilities? 61st IHC, New Orleans, LA

11. Results: Brier Skill Score Atlantic, July 18 – October 4, 2006; 105W – 1W, 1N – 60N Range: minus infinity to 1, 0 indicates no skill when compared to the reference forecast. Perfect score: 1. 61st IHC, New Orleans, LA

12. QUESTIONS How does the average forecast magnitude compare to the average observed magnitude? What is the magnitude of the probability forecast errors? What is the relative skill of the probabilistic forecast over the deterministic forecasts, in terms of predicting whether or not an eventoccurred? How well do the predicted probabilities of an event correspond to their observed frequencies? What is the ability of the forecast to discriminate between events and non-events? STATISTICS THAT ANSWER … Multiplicative Bias (half) Brier Score Brier Skill Score Reliability Diagrams Relative Operating Characteristics How good are the Monte Carlo wind probabilities? 61st IHC, New Orleans, LA

13. Reliability Diagrams Observed frequency of events Bins of Forecast Probabilities 61st IHC, New Orleans, LA

14. Results: Reliability Diagrams Atlantic, June 17 – October 4, 2006; 105W – 1W, 1N – 60N 61st IHC, New Orleans, LA

15. QUESTIONS How does the average forecast magnitude compare to the average observed magnitude? What is the magnitude of the probability forecast errors? What is the relative skill of the probabilistic forecast over the deterministic forecasts, in terms of predicting whether or not an eventoccurred? How well do the predicted probabilities of an event correspond to their observed frequencies? What is the ability of the forecast to discriminate between events and non-events? STATISTICS THAT ANSWER … Multiplicative Bias (half) Brier Score Brier Skill Score Reliability Diagrams Relative Operating Characteristics How good are the Monte Carlo wind probabilities? 61st IHC, New Orleans, LA

16. Results: ROC Diagram 2% 5% skill 10% No skill 61st IHC, New Orleans, LA

17. Results: ROC Skill Score Range: -1 to 1, 0 indicates no skill. Perfect score: 1. 61st IHC, New Orleans, LA

18. Summary Based upon a preliminary verification of Atlantic Monte Carlo Wind Probabilities • Probabilities have small negative biases • Probabilities are more skillful than the deterministic (i.e. OFCL) in determining if an event will happen • Predicted probabilities appear well related to observed frequencies • The Probabilities are skillful in their ability to discriminate between “events” and “non-events” 61st IHC, New Orleans, LA

19. Next • Complete the verification of the wind probabilities in the Atlantic, East Pacific, Central Pacific, and West Pacific • Summarize results and identify potential problems with the Monte Carlo software • Deliver the verification code to TPC 61st IHC, New Orleans, LA

20. Future Plans • Begin our new project to use the model forecast spread [i.e., the Goerss Predicted Consensus Error (GPCE)] to better estimate track error distributions used for the Monte Carlo integration • Assist, where possible, M. Mainelli (TPC) on the understanding the verification of the wind probabilities at the watch/warning break points 61st IHC, New Orleans, LA