Verification of weather parameters

1. Verification of weather parameters Anna Ghelli, ECMWF ECMWF forecast User Meeting -- Reading, June 2006

2. Overview • Deterministic forecast performance for different weather parameters • Precipitation forecast: scores and their confidence • SYNOP on the GTS • Precipitation analysis • Ensemble prediction System: its performance relative to precipitation and 10m wind ECMWF forecast User Meeting -- Reading, June 2006

3. 2m Temperature Skill (rmse) for different forecast ranges North America Top panel: North America Strong seasonality in the skill with higher values during winters. Europe Bottom panel: Europe 2mT skill have reached a plateau. The skills for different timesteps have levelled off. The two past winters were dominated by prolonged periods of cold weather. ECMWF forecast User Meeting -- Reading, June 2006

4. Performance for different forecast ranges Europe Top panel: Specific humidity Higher skill (MAE) in winter. Last four winters have consistently kept higher level of performance Europe Bottom panel: 10m wind speed RMSE: Changes in the forecasting model have improved the performance of the model in forecasting wind speed. ECMWF forecast User Meeting -- Reading, June 2006

5. Observed yes Observed no Forecast yes Forecast no 1. FREQUENCY BIAS INDEX 3. HIT RATE 2. TRUE SKILL SCORE 3. FALSE ALARM RATE ECMWF forecast User Meeting -- Reading, June 2006

6. Europe 15mm/24h t+42 24 hour accumulated precipitation verified against SYNOP on GTS The forecast is reduced to a yes/no event by selecting thresholds. Confidence intervals have been plotted for each TSS value. t+90 High thresholds have large confidence intervals, important to remember when assessing performance of the system ECMWF forecast User Meeting -- Reading, June 2006

7. Europe t+42: solid shading t+90: dotted shading 24 hour accumulated precipitation verified against SYNOP on GTS FBI measures the ratio between the frequency of the forecast events and the frequency of the observed events. FBI>1 over-estimate FBI<1 under-estimate The forecasting system over-estimate the number of events for thresholds of 1 mm/24h. A decrease of FBI was observed when in the autumn 1999, when vertical resolution was increased and a new convection scheme was implemented. Further changes in the convection during 2003 have slightly increased FBI values ECMWF forecast User Meeting -- Reading, June 2006

8. Europe 15mm/24h t+42: solid shading t+90: dotted shading 24 hour accumulated precipitation verified against SYNOP on GTS FBI decreases to values closer to 1 as we increase the threshold, but higher thresholds have larger confidence intervals! ECMWF forecast User Meeting -- Reading, June 2006

9. Stratifying the sample: selecting seasons as rule DJF MAM 15mm/24h 15mm/24h Green: t+42 Light blue: t+90 ECMWF forecast User Meeting -- Reading, June 2006

10. Precipitation analysis for Europe • High density networks in Europe (Member and Co-operating states) • Upscaling (simple box averaging to obtain a areal precipitation value) ECMWF forecast User Meeting -- Reading, June 2006

11. Each grid box will contain a certain number of stations. The number of stations will not be constant every day. The number of stations per grid box indicates how representative the analysis is for the specific grid point. ECMWF forecast User Meeting -- Reading, June 2006

12. Which reference system? t+42 Use proxy: the shortest possible forecast verifying at 6UTC. We assume the proxy to be the best forecast. Its TSS values (verification against precipitation analysis) will give us an upper limit for the verification of other forecast ranges. t+90 1 mm/24h Green : proxy ECMWF forecast User Meeting -- Reading, June 2006

13. Which reference system? 15 mm/24h Use proxy: the shortest possible forecast verifying at 6UTC. We assume the proxy to be the best forecast. Its TSS values (verification against precipitation analysis) will give us an upper limit for the verification of other forecast ranges. Green : proxy t+90 ECMWF forecast User Meeting -- Reading, June 2006

14. Europe • FBI plotted for two thresholds (1mm/24h, and 15mm/24h) • Verification against SYNOP on the GTS (greenish shading), • Verification against precipitation analysis (blue dotted) FBI values are higher (lower) in the verification against SYNOP on the GTS (analysis) for lower (higher) thresholds. Forecast range t+66 ECMWF forecast User Meeting -- Reading, June 2006

15. Europe • TSS (threshold 25mm/24h) plotted for t+66 • Verification against SYNOP on the GTS (greenish shading), • Verification against precipitation analysis (blue dotted) TSS values decrease as we increase forecast range. ECMWF forecast User Meeting -- Reading, June 2006

16. Timeseries of Brier Skill Score for Europe The BSS is written as 1- BS/BSref Sample climate is the reference system BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast Forecast vs. observations Increased resolution T255 T399 Improvements back in Autumn 1999 – High thresholds performance down at the beginning of 2005 probably linked to drier conditions over Europe. Dec-Apr 2006 performance better than previous 3 years. ECMWF forecast User Meeting -- Reading, June 2006

17. Forecast vs. observations ECMWF forecast User Meeting -- Reading, June 2006

18. Timeseries of Brier Skill Score for Europe The BSS is written as 1- BS/BSref Sample climate is the reference system BS measures the mean squared difference between forecast and observation in probability space. Equivalent to MSE for deterministic forecast C Forecast vs analysis T255 Increased resolution T399 10 m Wind speed ECMWF forecast User Meeting -- Reading, June 2006

19. Europe Rainy season (24h accum.): October to April Forecast range: t+96 Verification against SYNOP on GTS Consistent picture for the two seasons 2005-2006 1mm/24h BS=0.079 2004-2005 1mm/24h BS=0.157 ECMWF forecast User Meeting -- Reading, June 2006

20. Europe Rainy season (24h accum.): October to April Forecast range: t+96 Verification against SYNOP on GTS Consistent picture for the two seasons. Higher thresholds better reliability 2005-2006 10 mm/24h BS=0.019 2004-2005 10 mm/24h BS=0.04 ECMWF forecast User Meeting -- Reading, June 2006

21. Europe Rainy season (24h accum.): October to April Forecast range: t+96 Verification against SYNOP on GTS Consistent picture for the two seasons. Full symbol: T511/799 Shape: T255/T399 2005-2006 5 mm/24h t+96 : A=0.889 t+120: A=0.860 2004-2005 5 mm/24h t+96 : A=0.864 t+120: A=0.832 ECMWF forecast User Meeting -- Reading, June 2006

22. Europe ROC Area Verification against SYNOP on GTS for t+96 T255 Increased resolution T399 ECMWF forecast User Meeting -- Reading, June 2006

23. 2m Temperature: the skill over Europe has reached a plateau. Over N.America winters are more skilful than summers. Specific humidity shows consistent skills. Winters more skilful than summers Wind: Recent changes in the model have resulted in a decrease of the RMSE over Europe. TCC: New cloud scheme was introduced in April 2005. Forecasts of certain cloud types have improved. Conclusion • Importance of confidence intervals • Precipitation forecast improvements are slow, but evident. • FBI indicates over-estimation of small threshold events  verification against precipitation analysis shows a better picture. • Precipitation analysis can be used for verification in a delayed mode. It can be used to define the performance of a reference system. ECMWF forecast User Meeting -- Reading, June 2006

24. Conclusion • Brier skill score and ROC area: In recent year the system has maintained its good performance. The increase in horizontal resolution has had a small but positive impact on both precipitation and wind speed. • Reliability diagrams for the last two rainy seasons show a skilful system, with an evident decrease of Brier Score for the 2005-2006 season. ECMWF forecast User Meeting -- Reading, June 2006