1 / 40

COSMO-DE-EPS Ensemble: Forecast Uncertainty Analysis

Learn about setup, initial conditions, and variations in forecast systems for COSMO-DE-EPS ensemble predictions. Explore the representation of forecast uncertainty and verification methods in the pre-operational phase.

dsusie
Download Presentation

COSMO-DE-EPS Ensemble: Forecast Uncertainty Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. COSMO-DE-EPS Susanne Theis Christoph Gebhardt, Zied Ben Bouallègue, Michael Buchhold

  2. Presentation Overview • setup of COSMO-DE-EPS • first results of pre-operational phase • verification • forecasters‘ feedback • COSMO-DE-EPS plans

  3. Setup of COSMO-DE-EPS

  4. COSMO-DE-EPS status • pre-operational phase has started: Dec 9th, 2010 • pre-operational setup: • 20 members • grid size: 2.8 km convection-permitting • lead time: 0-21 hours, 8 starts per day (00, 03, 06,... UTC) • variations in physics, initial conditions, lateral boundaries model domain

  5. Generation of Ensemble Members Variations in Forecast System for the Representation of Forecast Uncertainty Initial Conditions Boundaries Model Physics

  6. Generation of Ensemble Members Variations in Forecast System for the Representation of Forecast Uncertainty Initial Conditions Boundaries Model Physics “multi-model” driven by different global models

  7. Generation of Ensemble Members Variations in Forecast System for the Representation of Forecast Uncertainty Initial Conditions Boundaries Model Physics “multi-model” COSMO-DE initial conditions modified by different global models “multi-model” driven by different global models

  8. Generation of Ensemble Members Variations in Forecast System for the Representation of Forecast Uncertainty Initial Conditions Boundaries Model Physics “multi-model” COSMO-DE initial conditions modified by different global models “multi-model” driven by different global models “multi-configuration” different configurations of COSMO-DE model

  9. Generation of Ensemble Members • plus variations of • initial conditions • model physics Ensemble Chain COSMO-DE-EPS 2.8km COSMO 7km BC-EPS GME, IFS, GFS, GSM

  10. Generation of Ensemble Members • plus variations of • initial conditions • model physics Ensemble Chain COSMO-DE-EPS 2.8km COSMO 7km BC-EPS is running as a time-critical application at ECMWF BC-EPS GME, IFS, GFS, GSM

  11. Generation of Ensemble Members 20 Members 1 2 3 4 5 IFS GME GFS BC-EPS GSM

  12. Generation of Ensemble Members Perturbation Methods Gebhardt, C., Theis, S.E., Paulat, M. and Z. Ben Bouallègue, 2011: Uncertainties in COSMO-DE precipitation forecasts introduced by model perturbations and variation of lateral boundaries. Atmospheric Research 100, 168-177. (contains status of 2009) Peralta, C. and M. Buchhold, 2011: Initial condition perturbations for the COSMO-DE-EPS, COSMO Newsletter 11, 115–123. Peralta, C., Ben Bouallègue, Z., Theis, S.E., Gebhardt, C. and M. Buchhold, 2011: Accounting for initial condition uncertainties in COSMO-DE-EPS. Submitted to Journal of Geophysical Research.

  13. First Results of Pre-operational Phase- verification - forecasters‘ feedback

  14. First Results of Pre-operational Phase- verification - forecasters‘ feedback

  15. SYNOP Verification Method RADAR • Ensemble Members • Probabilities of Precipitation

  16. PREC 1h accumulation, threshold: 0.1 mm DETERMINISTIC SCORES for Individual Members • Do the ensemble members • have different long-term statistics? • (multi-model / multi-configuration) • Are there many cases with • the same „best member“ • or „wettest member“? • look at Equitable Threat Score • - look at Frequency Bias Index • (results similar, not shown) Equitable Threat Score 0.5 0.4 0.3 0.2 0.1 0.0 JUNE 2011 IFS GME GFS GSM }20 members 0 5 10 15 20 Forecast Time [h]

  17. PREC 1h accumulation, threshold: 0.1 mm DETERMINISTIC SCORES for Individual Members • Do the ensemble members • have different long-term statistics? • (multi-model / multi-configuration) • Are there many cases with • the same „best member“ • or „wettest member“? • look at Equitable Threat Score • - look at Frequency Bias Index • (results similar, not shown) Equitable Threat Score 0.5 0.4 0.3 0.2 0.1 0.0 JUNE 2011 IFS GME GFS GSM }20 members 0 5 10 15 20 Forecast Time [h] Only small differences in long-term statistics  Members may be treated as equally probable

  18. PREC 1h accumulation • observation... • …treated as „Ensemble Member“ • …ranked according to prec amount • at each grid point and forecast hour • How frequent is each rank? • If ensemble underdispersive •  U-shaped rank histogram RANK HISTOGRAM 0.05 0.00 JUNE 2011 Frequency 1 6 11 16 21 Observation Rank

  19. PREC 1h accumulation • observation... • …treated as „Ensemble Member“ • …ranked according to prec amount • at each grid point and forecast hour • How frequent is each rank? • If ensemble underdispersive •  U-shaped rank histogram RANK HISTOGRAM 0.05 0.00 JANUARY 2011 Frequency 1 6 11 16 21 Observation Rank

  20. PREC 1h accumulation • observation... • …treated as „Ensemble Member“ • …ranked according to prec amount • at each grid point and forecast hour • How frequent is each rank? • If ensemble underdispersive •  U-shaped rank histogram RANK HISTOGRAM 0.05 0.00 JANUARY 2011 Frequency 1 6 11 16 21 Observation Rank a) Underdispersiveness relatively small b) Four groups  Many cases with large influence by global models

  21. PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE JANUARY 2011 > 0.1 mm > 1 mm > 2 mm 0 5 10 15 20 Forecast Time [h]

  22. PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE JANUARY 2011 > 0.1 mm > 1 mm > 2 mm 0 5 10 15 20 Forecast Time [h] Always positive!  Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability)

  23. PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) BRIER SKILL SCORE JUNE 2011 > 0.1 mm > 1 mm > 2 mm 0 5 10 15 20 Forecast Time [h] Always positive!  Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability)

  24. PREC 1h accumulation How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (x-axis) (probabilites of exceeding a certain threshold) - for all foreast lead times BRIER SKILL SCORE MAY - JULY 2011 0.112 5 10 20 Threshold [mm/h] For larger precipitation amounts (summer): even more additional value

  25. PREC 1h accumulation Are the probabilities already well calibrated? (without extra calibration) If we isolate all cases with a forecast probability of -say- 75-85% … did the event occur in 80% of these cases? diagonal line: optimal - for different prec thresholds (colors) (probs of exceeding a threshold) RELIABILITY DIAGRAM JUNE 2011 log (# fcst) > 0.1 mm > 1 mm > 2 mm

  26. PREC 1h accumulation Are the probabilities already well calibrated? (without extra calibration) If we isolate all cases with a forecast probability of -say- 75-85% … did the event occur in 80% of these cases? diagonal line: optimal - for different prec thresholds (colors) (probs of exceeding a threshold) RELIABILITY DIAGRAM JUNE 2011 log (# fcst) > 0.1 mm > 1 mm > 2 mm Reliability diagram shows some bias and underdispersiveness Lines are not flat  additional calibration has good potential

  27. Summary of Verification (Precipitation) • Ensemble provides additional value to COSMO-DE (for all accumulations, lead times, precipitation thresholds,…) • Ensemble underdispersiveness is relatively small • Ensemble members may be treated as equally probable • Additional calibration has good potential

  28. Summary of Verification (Precipitation) • Ensemble provides additional value to COSMO-DE (for all accumulations, lead times, precipitation thresholds,…) • Ensemble underdispersiveness is relatively small • Ensemble members may be treated as equally probable • Additional calibration has good potential Pre-operational COSMO-DE ensemble prediction system already meets fundamental quality requirements for precipitation

  29. Other Variables • T_2M and VMAX have been verified • ensemble spread is far too small • nevertheless, ensemble provides additional value to COSMO-DE

  30. Other Variables • T_2M and VMAX have been verified • ensemble spread is far too small • nevertheless, ensemble provides additional value to COSMO-DE COSMO-DE ensemble prediction system has been developed with focus on precipitation

  31. First Results of Pre-operational Phase- verification - forecasters‘ feedback

  32. Forecasters‘ Feedback • available products: see figure precipitation, snow, wind gusts, T_2m probability thresholds: warning criteria • all products on grid-scale (2.8km) • in addition: precipitation probabilities for larger areas (10x10 grid boxes) „probability that the precipitation event will occur anywhere within the region“ probabilities, quantiles, ensemble mean, spread, min, max, …

  33. Forecasters‘ Feedback • evaluate „full package“ • including the visualization tool • consistency of products • select relevant cases • consider forecasters‘ interpretation • perception as intended? • is there any value in the forecast, additional to forecasters‘ knowledge?

  34. Forecasters‘ Feedback • what they prefer to use: • 90%-quantile of precipitation • precipitation probabilities for an area (10x10 grid points)

  35. Forecasters‘ Feedback • what they prefer to use: • 90%-quantile of precipitation • precipitation probabilities for an area (10x10 grid points) • what they appreciate: • early signals for heavy precipitation • indication that deterministic run may be wrong

  36. Forecasters‘ Feedback • what they prefer to use: • 90%-quantile of precipitation • precipitation probabilities for an area (10x10 grid points) • what they appreciate: • early signals for heavy precipitation • indication that deterministic run may be wrong • what they criticize: • jumpiness between subsequent runs • lack of spread in T_2M and VMAX

  37. Forecasters‘ Feedback • what they prefer to use: • 90%-quantile of precipitation • precipitation probabilities for an area (10x10 grid points) • what they appreciate: • early signals for heavy precipitation • indication that deterministic run may be wrong • what they criticize: • jumpiness between subsequent runs • lack of spread in T_2M and VMAX • what they are learning: • dealing with low probabilities (10% probability for extreme weather  issue a warning?)

  38. COSMO-DE-EPS plans

  39. COSMO-DE-EPS plans (2011-2014) • under consideration: including past production cycles in product generation • upgrade to 40 members, redesign • statistical postprocessing • initial conditions by LETKF • lateral boundary conditions by ICON EPS 2011 2012 reach operational status 2013

More Related