Forecast Verification Research

1. Forecast Verification Research Beth Ebert and Laurie Wilson, JWGFVR co-chairs WWRP-JSC meeting, Geneva, 21-24 Feb 2011

2. Aims Verification component of WWRP, in collaboration with WGNE, WCRP, CBS • Develop and promote new verification methods • Training on verification methodologies • Ensure forecast verification is relevantto users • Encourage sharing of observational data • Promote importance of verification as a vital part of experiments • Promote collaboration among verification scientists, model developers and forecast providers

3. Working group members • Beth Ebert (BOM, Australia) • Laurie Wilson (CMC, Canada) • Barb Brown (NCAR, USA) • Barbara Casati (Ouranos, Canada) • Caio Coelho (CPTEC, Brazil) • Anna Ghelli (ECMWF, UK) • Martin Göber (DWD, Germany) • Simon Mason (IRI, USA) • Marion Mittermaier (Met Office, UK) • Pertti Nurmi (FMI, Finland) • Joel Stein (Météo-France) • Yuejian Zhu (NCEP, USA)

4. FDPs and RDPs Sydney 2000 FDP Beijing 2008 FDP/RDP SNOW-V10 RDP Sochi 2014 MAP D-PHASE Typhoon Landfall FDP Severe Weather FDP

5. Beijing 2008 FDP Real Time Forecast Verification (RTFV) system Fast qualitative and quantitative feedback on forecast system performance in real time • Verification products generated whenever new observations arrive Ability to inter-compare forecast systems 3 levels of complexity • Visual (quick look) • Statistics (quantitative) • Diagnostic (more information)

6. Training In person Online

7. B08FDP lessons for real time verification • Real time verification considered very useful • Forecasters preferred scatterplots and quantile-quantile plots • Format and standardization of nowcasts products was critical to making a robust verification system • Difficult to compare "like" products created with different aims (e.g., QPF for warning vs hydrological applications) • Verification system improvements • User-friendly web display • More user options for exploring results

8. SNOW-V10 • Verification strategy • User-oriented verification for Olympic period of all forecasts, tuned to decision points of VANOC • Verification of parallel model forecasts for Jan to August 2010 • Nowcast and regional model verification • Rich dataset

9. Suggested categories for SNOW-V10 verification

10. Example:Visibility verification

11. Sochi 2014 Standard verification Possible verification innovations: • Road weather forecasts • Real-time verification • Timing of events – onset, duration, cessation • Verification in the presence of observation uncertainty • Neighborhood verification of high-resolution NWP, including in time-height plane • Spatial verification of ensembles • User-oriented probability forecast verification

12. Collaboration • WWRP working groups • THORPEX • GIFS-TIGGE • Subseasonal prediction • Polar prediction • CBS • Severe Wx FDPs • Coordination Group on Forecast Verification • SRNWP • COST 731 • ECMWF TAC subgroup on verification measures

13. Spatial Verification Method Intercomparison Project http://www.rap.ucar.edu/projects/icp International comparison of many new spatial verification methods Methods applied by researchers to same datasets (precipitation; perturbed cases; idealized cases) Subjective forecast evaluations Workshops: 2007, 2008, 2009 Weather and Forecasting special collection

14. Spatial Verification Method Intercomparison Project

15. Spatial Verification Method Intercomparison Project • Future variables • "Messy" precipitation • Wind • Cloud • Future datasets • MAP D-PHASE / COPS • SRNWP / European data • Nowcast dataset(s) • Verification test bed

16. Publications Publications Recommendations for verifying deterministic and probabilistic quantitative precipitation forecasts Recommendations for verifying cloud forecasts (this year) Recommendations for verifying tropical cyclone forecasts (next year) January 2008 special issue of Meteorological Applications on forecast verification 2009-2010 special collection of Weather & Forecasting on spatial verification DVD from 2009 Helsinki Verification Tutorial

17. Outreach http://www.cawcr.gov.au/projects/verification/ • Verification workshops and tutorials • On-site, travelling • EUMETCAL training modules • Verification web page • Sharing of tools

18. International Verification Methods Workshops 4th Workshop – Helsinki 2009 Tutorial • 26 students from 24 countries • 3 days • Lectures, hands-on (took tools home) • Group projects - presented at workshop Workshop • ~100 participants • Topics: • User-oriented verification • Verification tools & systems • Coping with obs uncertainty • Weather warning verification • Spatial & scale-sensitive methods • Ensembles • Evaluation of seasonal and climate predictions

19. 5th International Verification Methods Workshop • Melbourne, December 2011 • 3-day tutorial + 3-day scientific workshop • Additional tutorial foci • Verifying seasonal predictions • Brief intro to operational verification systems • Capacity building for FDPs/RDPs, SWFDP, etc.

20. New focus areas for JWGFVR research "Seamless verification" - consistent across space/time scales decadal prediction climate change seasonal prediction NWP sub- seasonal prediction global very short range • Approaches: • deterministic / categorical • probabilistic • distributional • other? regional Spatial scale nowcasts local point minutes hours days weeks months years decades Forecast lead time

21. New focus areas for JWGFVR research Spatial methods for verifying ensemble predictions • Neighborhood, scale-separation, feature-based, deformation average rain rain area rain volume maximum rain

22. New focus areas for JWGFVR research Extreme events

23. 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 0 Hit rate Success ratio (1-FAR) New focus areas for JWGFVR research Warnings, including timing

24. Thank you