Verification of multi-model ensemble forecasts using the TIGGE dataset

1. www.ec.gc.ca Lawrence.wilson@ec.gc.ca Verification of multi-model ensemble forecasts using the TIGGE dataset Laurence J. Wilson Environment Canada Anna Ghelli ECMWF With thanks to Marcel Vallée

2. Outline • Introduction – TIGGE goals and verification • Status of verification of TIGGE ensembles • Standard methods • Spatial methods • Precipitation verification project: plan and early results • Summary • SEE the extended abstract also

3. Verification and the goals of TIGGE • Goals: • Enhance collaborative research • Enable evolution towards GIFS • Develop ensemble combination methods; bias removal • Essential question: If we are going to move towards a GIFS, then we must demonstrate that the benefits of combined ensembles are worth the effort with respect to single-center ensembles. OR: Do we get a “better” pdf by merging ensembles? • Verification – Relevant, user-oriented

4. Status of Verification of TIGGE ensembles • Mostly model-oriented verification so far • Upper air data • Against analyses • Standard scoring and case studies • Studies on the TIGGE website • Park et al, 2008 • First study involving several months of data • Found modest improvement with combined ensembles, greatest benefits in tropics and lower atmosphere • “Advantage” of using one’s own analysis as truth • Pappenberger et al. • Case study of flooding event in Romania • User-oriented, Q-Q plots, RPS and RMSE main scores used • Multimodel ensemble has best average properties, ECMWF next.

5. Studies using TIGGE data (cont’d) • Johnson and Swinbank, 2008 • Study of calibration/combination methods • Used only 3 ensembles • Mslp and 2m temperature, but from analyses • Multimodel ensemble improves on individual ensembles, but not by much in general. More at 2m than 500mb • Matsueda 2008 • Comparison of 5 combined ensembles vs ECMWF alone • RMSE skill and RPSS with ECMWF as standard forecast • Multimodel eps outperforms ECMWF at medium and longer ranges.

6. Status of TIGGE – related verification • Current efforts – Verification of surface variables? • This conference ---? • Studies using spatial methods • Ebert – application of CRA technique to ensemble forecasts. So far, only ECMWF. • Application of Wilks minimum spanning tree or T. Gneiting’s multi-dimensional rank histogram for TC centers. (idea stage) • Precipitation verification project:

7. Precipitation verification project • Goal: to verify global 24h precipitation forecasts from all the ensembles in the TIGGE archive and combinations • One region at a time, using highest density observations • Canada and Europe so far • Methodology • Cherubini et al upscaling, verify only where data available • Single station, nearest gridpoint where data is sparser • Kernel density fitting following Peel and Wilson to look at extremes of distributions.

8. Precipitation verification project : methodology - Europe • Upscaling: • 1x1 gridboxes, limit of model resolution • Average obs over grid boxes, at least 9 stns per grid box (Europe data) • Verify only where enough data • Matches obs and model resolution locally • Answers questions about the quality of the forecasts within the capabilities of the model • Most likely users are modelers.

9. European Verification -Upscaled observations according to Cherubini et al (2002) -OBS from gauges in Spain, Portugal, France, Italy, Switzerland, Netherlands, Romania, Czech Republic, Croatia, Austria, Denmark, UK, Ireland, Finland and Slovenia -At least 9 stns needed per grid box to estimate average -24h precip totals, thresholds 1,3,5,10,15,20,25,30 mm -one year (oct 07 to oct 08

10. Reliability – Summer 08 – Europe – 42h

11. Reliability – Summer 08- Europe 114 h

12. Reliability – Winter 07-08 – Europe – 114h

13. ROC – Summer 08 – Europe – 42h

14. ROC – Summer 08 – Europe – 114 h

15. Precipitation verification project: methodology - Canada • Single station verification • Canadian verification over 20 widely-spaced stations, only one station per gridbox; comparison of nearest gridpoint fcst to obs • Pointwise verification, does not (we cannot) upscale properly because don’t have the data density necessary. • Valid nevertheless as absolute verification of model predictions

16. Results – Canada – ROC curves – 24h

17. Results – Canada – ROC Curves – 144h

18. RMSE of pcpn probability – Canada – Oct 07 to Oct 08 – 20 stns 2.0 mm 10 mm BOM in blue (darker blue); ECMWF in red; UKMET in green CMC in gray; NCEP in cyan (lighter blue)

19. Verification of TIGGE forecasts with respect to surface observations – next steps • Combined ensemble verification • Other regions – Southern Africa should be next. – Non-GTS data is available • Evaluation of extreme events – kernel density fitting to ensembles. • Other high-density observation datasets such as SHEF in the US • Other variables: TC tracks and related surface weather • Use of spatial verification methods • THEN maybe we will know the answer to the TIGGE question.

20. www.ec.gc.ca

21. Issues for TIGGE verification • Use of analyses as truth – advantage of one’s own model. Alternatives: • Each own analysis • Analyses as ensemble (weighted or not) • Random selection from all analyses • Use “best” analysis; eliminate the related model from comparison • Average analysis (may have different statistical characteristics) • Model-independent analysis (restricted to data – rich areas, but that is where verification might be most important for most users • Problem goes away for verification against observations (as long as they are not qc’d with respect to any model)

22. Park et al study – impact of analysis used as truth in verification

23. Issues for TIGGE Verification (cont’d) • Bias adjustment/calibration • Reason: to eliminate “artifical” spread in combined ensemble arising from systematic differences in component models • First (mean) and second (spread) moments • Several studies have/are being undertaken • Results on benefits not conclusive so far • Due to too small sample for bias estimation? • Alternative: Rather than correcting bias, eliminate inter-ensemble component of bias and spread variation.

24. ROC – Winter 07-08 – Europe – 42h

25. ROC – Winter 07-08 – Europe – 114h

26. Reliability – Winter 07-08 – Europe – 42h

27. Results – Canada – Brier Skill, Resolution and Reliability