Weather Forecast Verification Training - Recent Advancements -

1. 1 2 training modules 0. Introduction Weather ForecastVerificationTraining- Recent Advancements - Pertti Nurmi Meteorological Research & Development Finnish Meteorological Institute Laurie Wilson Meteorological Service of Canada

2. 2680 + 30 2680 + 30 = 96,8 % = 96,8 % 2800 2800 Hypothesis Never Never forecast forecast a Tornado ! a Tornado 2750 + 0 2750 + 0 = 98,2 % = 98,2 % 2800 2800 0 Verification (training) History, 1884 Tornados in USA ”The Finlay case” CALMet - Beijing

3. 0 Goals of Verification Training: • Understand the basic properties and relationships of (common) verification measures • Learn to extract useful information from (graphical) verification results • Increase interestin forecast verification and the methods • Apply verification in everyday fc practice at every fc office Generally HEAVILY neglected topic in training activities ! CALMet - Beijing

4. Harold Brooks (NSSL, USA; not shown) Barbara Brown (NCAR, USA; Chair) Barbara Casati (MSC, Canada) Ulrich Damrath (DWD, Germany) Beth Ebert (BMRC, Australia) Anna Ghelli (ECMWF) Pertti Nurmi(FMI, Finland) David Stephenson (U. Reading, UK; not shown) Clive Wilson (UKMO, UK) Laurie Wilson (MSC, Canada) 0 International Activities ”WMO WWRP/WGNE JWGV”(* *) WMO ~ World Meteorological Organization WWRP ~ World Weather Research Program WGNE ~ Working Group on Numerical Experimentation JWGV ~ Joint Working Group on Verification CALMet - Beijing

5. 0 International Activities ”WMO WWRP/WGNE JWGV” Goals of JWGV: • Cooperation between forecasters, users and verifiers • Development and application of verification methodologies • Promote awareness of the importance of verification  via training Recent activities of JWGV: • Research and training workshops on verification methods • Recommendations for verification methodologies • Test-bed activities • Beijing 2008 Olympics Real-Time Forecast Verification (RTFV) • THORPEX and TIGGE • EUMETCAL verification training modules 1 2 CALMet - Beijing

6. 0 JWGV website Google search: “Forecast > verification” gives this as the first prompt ! www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html CALMet - Beijing

7. 0 CALMet - Beijing

8. 1 www.ecmwf.int/ newsevents/ meetings/ workshops/ 2007/ jwgv/ lecture_notes/ index.html CALMet - Beijing

9. 1 31 students, One/country Argentina, Australia , Brasil, Canada, Checz Republic, China, Croatia, Denmark, Ethiopia, Finland, France, Germany, Hong Kong, India, Italy, Kenya, Lithuania, Nigeria, Norway, Romania, Senegal, Slovakia, South Africa, South Korea, Sweden, Switzerland, Ukraine, USA CALMet - Beijing

10. 1 Components: • Tutorial ~ 2 days • Scientific workshop and presentations ~ 3 days i. Tutorial contents: • Use/interpretation of “standard” verification methods • Application of new methodologies (e.g. spatial forecast verification) • Hands-on laboratory exercises • ”R” software (verification components) training ii. Workshop contents: • Invited keynotes & contributed presentations • User-oriented forecast verification • Verification of extreme/ high impact weather, EPS, uncertainty … CALMet - Beijing

11. 1 Tutorial contents: • Introduction to R (Matt Pocernich) • Basic Verification Concepts (Barb Brown) • Verification of Continuous Predictands (Barbara Casati) • Verification of Categorical Predictands (Pertti Nurmi) • Verification of Probabilistic Predictands and Ensemble Forecasts (Beth Ebert) • Confidence Intervals and Hypothesis Tests (Simon Mason) • Projects (Laurie Wilson) CALMet - Beijing

12. 1 (i) www.r-project.org CALMet - Beijing

13. 1 (ii) CALMet - Beijing

14. 1 (ii) CALMet - Beijing

15. 1 (iii) CALMet - Beijing

16. 1 (iii) CALMet - Beijing

17. 1 (iii) CALMet - Beijing

18. 1 (iv) CALMet - Beijing

19. 1 (iv) CALMet - Beijing

20. E.Ebert@bom.gov.au 1 (v) CALMet - Beijing

21. 1 (v) CALMet - Beijing

22. 1 (v) CALMet - Beijing

23. 1 (v) CALMet - Beijing

24. 1 (vi) CALMet - Beijing

25. Project No. 1: Verification of ensemble-based PoP forecasts (Dataset: Tong/ Hong Kong) Calculate the predictand…Verify the POP forecasts... Prepare and plot a reliability table... Compute the BS… Project No. 2: Temperature verification (Dataset: Diaconu/ Romania) Choose a verification problem… Look at the data… Produce and examine scatter plots…Choose verification scores… Project 3: Cloudiness forecasts in 3 categories (Dataset: Diaconu/ Romania) Choose a verification issue...Look at the data… Produce contingency tables...Choose verification measures… Project No. 4: Wind speed forecasts (Dataset: Moreau/ Canada) Choose the verification problem…Look at the data… Plot forecast-observation scatterplots...Choose verification measures...Calculate the scatter index... Project No. 5 Climatological analysis (Dataset: Korecha/ Ethiopia) Produce box and whisker plots…Calculate the annual statistics…Do the same for each month separately... Project No. 6: Precipitation forecasts (Dataset: Suaya/ Argentina) Look at the data… Prepare histogram plots…Pose the relevant verification questions…Carry out the verification and produce graphs to display your results… deterministic 1 (vii) Projects Laurie Wilson Environment Canada Lawrence.Wilson@ec.gc.ca CALMet - Beijing

26. “Probably the best training course I ever had. Thank you to all of you! “ 1 Overall evaluation by students: > 2.5 / 3 CALMet - Beijing

27. CAL training modules finished spring 2006 finished winter 2007 finished summer 2007 2 What’s included in the modules: • Introduction • Goals and general guidelines • Continuous variables • Categorical events • Probability forecasts • Glossary + References + Practical exercises ~ Quiz ! CALMet - Beijing

28. 2 www.eumetcal.org CALMet - Beijing

29. 2 www.eumetcal.org Demonstration on Sat 10:10-11:00  WELCOME TO TEST CALMet - Beijing

30. 2 www.eumetcal.org/resources/ukmeteocal/verification/www/english/courses/msgcrs/index.htm Demonstration on Sat 10:10-11:00  WELCOME TO TEST • CAL modules ! … produced by Pertti Nurmi, Laurie Wilson & Sigbritt Näsman, 2006-07

31. www.eumetcal.org/resources/ukmeteocal/verification/www/english/courses/msgcrs/index.htmwww.eumetcal.org/resources/ukmeteocal/verification/www/english/courses/msgcrs/index.htm ”Help button” ”Glossary button” ”Communication tool” ”Return to Index” 2 … open browser !

32. References: • Jolliffe and Stephenson, Eds. (2003): Forecast Verification: A Practitioner’s Guide, Wiley & Sons. • Nurmi (2003): Recommendations on the verification of local weather forecasts. ECMWF Tech. Mem. 430. • Wilks (2006): Statistical Methods in the Atmospheric Sciences, Chapter 7 (Forecast Verification). Academic Press. WMO Joint Working Group on Verification Website: • www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html ECMWF Verification Methods Training Workshop Lectures: • www.ecmwf.int/newsevents/meetings/workshops/2007/jwgv/lecture_notes/index.html EUMETCAL Forecast Verification Learning Modules: • www.eumetcal.org/resources/ukmeteocal/verification/www/english/courses/msgcrs/index.htm 0 1 2 CALMet - Beijing

33. Finishing motto There is nothing as difficult as forecasting the weather(correctly)... … except(proper)forecast verification ! - Allan Murphy, RIP Training is one key issue ! CALMet - Beijing

34. Thank You Pertti Nurmi Meteorological Research & Development Finnish Meteorological Institute