570 likes | 744 Views
Basin-oriented verification of COSMO-LEPS system Andrea Montani ARPA-SIM H ydrometeorological service of Emilia-Romagna, Italy Thanks to H. Asensio, R. Buizza, F. Pappenberger, B. Ritter, J.W. Schipper. Medium-range flood forecasting and warning. X General COSMO meeting
E N D
Basin-oriented verification of COSMO-LEPS systemAndrea MontaniARPA-SIM Hydrometeorological service of Emilia-Romagna, ItalyThanks to H. Asensio, R. Buizza, F. Pappenberger, B. Ritter, J.W. Schipper Medium-range flood forecasting and warning X General COSMO meeting Cracow, 15-19 September 2008 A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Workpackage Medium-Range Plain Flood of PREVIEW project • Aims: • Set-up and validation of the probabilistic medium-range flood forecasting using the meteorological products for the Upper-Danube in the hydrological year 2002, which includes the large flood of August 2002; to achieve this, • reruns of a number of state-of-the-art of atmospheric models were performed, • “convenient” (basin-oriented) scoring techniques for probabilistic forecasts were developed. • In terms of operational applications, the goal is to demonstrate the usefulness of probabilistic medium-range flood forecasting as a sound basis for early warning and decision making. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
The region of interest A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Verification methodology MAIN FEATURES: Verification performed in terms of 24-hour precipitation (from 6UTC to 6UTC); fcst ranges: 18-42h, 42-66h, 66-90h, 90-114h, …; observations: gridded observations (about 5 km of horizontal resolution) provided by JRC; verification domain: full upper-Danube river basin and 3 (out of 19) sub-basins; verification period: 20 July - 31 August 2002. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
ID name of Basin area (km2) no. of gridded obs 1 Bratislava 131.978 5278 13 Wiblingen 2.247 80 16 Passau-Ingling 25.977 1045 20 Hofkirchen 47.534 1896 Sub-basin verification For each gridded observation, it was identified the sub-basin it belonged to. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Catchment-based measures • centre of gravity:it is a specific point at which the system's mass behaves as if it were concentrated. The centre of mass/gravity is a function only of the positions and masses of the particles of the system. Deterministic measures for verification A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Ensemble systems in PREVIEW • VarEPS by ECMWF (global, ensemble) • COSMO-LEPS by ARPA-SIM (limited-area, ensemble) A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
CoG distance CentreofGravity distance • Aim: quantify the "location error" of the precipitation forecast (by the ensemble mean); • the CoG is scaled with the "mean distance" in a catchment a value of cog=0.3 for the catchment "Passau-Ingling" (mean distance =113 km) indicates a location error of the forecasted centre of gravity of precipitation in a catchment to the observerd centre of gravity of precipitation of about 30 km. • with the scaling, we get a dimensionless number (…the lower, the better…) to compare the absolute location error with respect to the catchment size. CoG • The absolute figures are very low (distances of the order of a few tens of km). • Little dependence of the distance from the forecast range. • Slightly higher distances for the Passau-Ingling basin (possibly related to observation problems in the Alpine region). • Best results for the smallest basin. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Brier Score for VarEPS and COSMO-LEPS • BS measures the mean squared difference between forecast and observation in probability space. • equivalent to MSE for deterministic forecast. • BS between 0 and 1; the lower the better …. • the largest (Bratislava) and the smallest (Wiblingen) basins are considered (tp > 80% of obs distribution). BS • ECMWF : solid. • COSMO-LEPS: dotted. • Low values for both models: GOOD! • COSMO-LEPS performs better over the smaller basins (not well resolved by VarEPS). • Similar performance of the two systems over the largest basin. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
For the other results, come and see the poster! A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Thank you ! A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
15d VAREPS T0 +360 +240 32d MOFC T0 15d and 32d VAREPS T0 +360 +240 +768 The 32-day unified VarEPS at ECMWF Unification of the 15d VarEPS (50+1 members) and the 32d monthly forecast (MOFC) systems into the unified 32d VarEPS: old system Twice-a-day (at 00 and 12 UTC): • d 0-10: TL399L62 uncoupled • d 10-15: TL255L62 uncoupled Once a week: • d 0-32: TL159L62 coupled +768 NEW SYSTEM (since 11/3/2008) Twice-a-day (at 00 and 12 UTC): • d 0-10: TL399L62 uncoupled • d 10-15: TL255L62 coupled at 00 Once a week: • d 0-10: TL399L62 uncoupled • d 10-32: TL255L62 coupled A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Skill of ECMWF predictions for hydrological modelling • Nash_Sutcliffe efficiency coefficient: normally used to assess the predictive power of hydrological models. • … the higher, the better (ranges from – to 1) • Timing of forecasts is good, although underestimation occurs. • The ensemble spread brackets observations only partially. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008 17th of July 2007
7 Panel version A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Atmospheric models in PREVIEW • COSMO-LEPS by ARPA-SIM (limited-area, ensemble) A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
MAIN FEATURES: • initial time: 12 UTC (i.e. once a day); • bc and ic: “selected” VarEPS members; • COSMO-LEPS configuration • 10 members; • hor. res. = 10 km; 32 vertical levels; • forecast length: 132h; • archived variables: surf and plev up to +132h, every 3h; • output fields archived at ECMWF; • rerun period: 20 July – 31 August 2002. integration domain COSMO-LEPS (developed at ARPA-SIM) COSMO-LEPS: the Limited-area Ensemble Prediction System (LEPS), based on COSMO-model and implemented within COSMO (COnsortium for Small-scale MOdelling, including Germany, Greece, Italy, Poland, Romania, Switzerland). A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Brier Skill Score vs forecast range • BSS is written as 1-BS/BSref. Sample climate is the reference system. Useful forecast systems if BSS > 0. • BS measures the mean squared difference between forecast and observation in probability space. • Equivalent to MSE for deterministic forecast. BSS • For low thresholds, better performance over the smallest basin. • For higher thresholds, more difficult to assess a clear trend (possible sampling problems over small basins). • Worse than climatology only for d+5 range (10 and 20 mm threshold). A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Brier Skill Score vs increasing threshold • BSS is plotted vs increasing threshold to assess the COSMO-LEPS skill as a function of rainfall intensity. BSS +42h BSS +90h • BSS positive for all thresholds at both forecast ranges. • At low thresholds, it is confirmed the higher skill of COSMO-LEPS over the smallest basin. • Stable skill of the system for intermediate basins. • For higher thresholds, more difficult to assess a clear trend (sampling problems over the small basin). A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Ranked Probability Skill Score • A sort of BSS, but “cumulated” over all thresholds. • Useful forecast system, if RPSS > 0. RPSS • RPSS always positive. • Better (worse) performance of the system over the smallest (largest) basin. • Almost identical scores using either nearest grid-point (NGP) or bilinear interpolation (BILIN). A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
ROC area vs increasing threshold • ROC area is plotted vs increasing threshold to assess the dependence of COSMO-LEPS HIT/FAR diagram on rainfall intensity. ROC +42h ROC +90h • ROC area always above 0.7 for all thresholds at both forecast ranges. • At low thresholds, it is confirmed the higher skill of COSMO-LEPS over the smallest basin (the same as BSS). • Scores increase with thresholds for intermediate and large basins. • For higher thresholds, more difficult to assess a clear trend (sampling problems over the small basin). A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
ROC area vs forecast range • Area under the curve in the HIT rate vs FAR diagram. • Valuable forecast systems have ROC area values > 0.6. ROC • ROC area always above 0.6; • at low thresholds, better COSMO-LEPS performance over the smallest basin; • possible sampling problems at 20 mm thresholds. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Outliers • How many times the analysis is outside the forecast interval spanned by the ensemble members. • … the lower the better … OUTL • The absolute figures are quite large (about 20% at d+5 range). • Lower outliers for the smallest basin. • Similar percentages for all basins at the longest forecast range. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Main results andfuture plans • Output files are of COSMO-LEPS, for the period 20/7/ to 31/8/2002, are archived on ECFS (retrieval script disseminated) and ready to be used for hydrological purposes; for any kind of help with grib files, just ask. • Verification against gridded observations (about 5300 in the full upper-Danube basin) indicates better performance of the system over the smallest basin, especially for low thresholds. • OLD RESULT (Offenbach, 25-26-/9/2007): Verification against SYNOP stations (about 100 in the full upper-Danube basin) indicates slightly better performance of the system over larger basins, although results are not statistically robust. • Difficulty to understand and use the CoG measure for a probabilistic system. • Finish the work on the verification report. • Provide COSMO-LEPS fields for 2-month real-time testing (no “bureaucratic” problems envisaged). • Real-time testing of services will be done using the “improved” COSMO–LEPS (16 members; 40 vertical levels; physics perturbations). A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Main results • During the hydrological year 2002, reruns of a number of NWP systems, both global-scale and limited-area, both deterministic and probabilistic. • Development of new • basin-oriented scores (e.g. centre of gravity); • hydrological-oriented products (e.g. rivergrams); • All systems seem to provide useful guidance for the possible occurrence of flood events also for forecast ranges up to 7 (5) days for global (limited-area) systems. • Verification vs gridded observations (about 5300 in the full upper-Danube basin) indicates better performance of the high-res system over the smaller basin. • The verification period is probably too short to draw general (and statistically solid) conclusions about the overall skill of the different forecast systems; the “Plain-flood campaign” clearly shows the potential of state-of-the-art NWP systems in the field of weather forecasting for river flooding. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Probabilistic measures for verification • Brier Score • The BS is the mean-squared error of the probability forecasts: • where: • the observation is either ok = 1 (the event occurs) or ok = 0 (the event does not occur); • fk is the fraction of ensemble members which forecast a precipitation amount exceeding that threshold • k denotes a numbering of the N forecast/event pairs. • BS ranges from 0 to 1, the perfect forecast having BS = 0. • BS is computed for a fixed precipitation threshold. • ROC area • It is the area under the Relative Operating Characteristics curve in the HIT rate vs FAR diagram • The integral under the curve is used to indicate the skill of the forecast. • ROC area ranges from 0 to 1 … the higher the better … • Useful forecast systems have ROC area values greater than 0.6 A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Summary of measures • Deterministic measures: • Root Mean Squared Error • Mean Error • Probability of detection • Probability of false detection • True Skill Statistics • Centre of gravity • Coverage • Probabilistic measures: • Brier score • ROC area A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
continuous measures (RMSE and MAE for the ensemble mean); • catchment-based measures (CentreofGravity for the ensemble mean); • probabilistic measures (Brier Skill Score, ROC area, Percentage of Outliers, …). • Centre of Gravity: the centre of gravity of a system of raster cells is a specific point at which, for many purposes, the system's mass behaves as if it were concentrated. The centre of mass is a function only of the positions and masses of the particles that comprise the system. • CoG: Centre of Gravity measure • c: Centre of gravity • dist: Distance • f: forecast • distmean: mean distance of grid elements to catchment outlet • obs: observed • p: precipitation • c= this score indicates a location error of the forecast, a perfect score having the value of 0. Performance measures A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Catchment Information SystemRiver-grams Sylvie Lamy-Thepaut, Enrico Fucile A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Main results • During the hydrological year 2002 (1/10/2001 to 30/9/2002), reruns of a number of NWP systems, both global-scale and limited-area, both deterministic and probabilistic. • Coordinated efforts to provide state-of-the-art weather forecasts over the Danube river basin. • Development of basin-oriented scores • Development of hydrological-oriented products (e.g. rivergrams) • All systems seem to provide useful guidance for the occurrence of flood events. Time-series verification scores indicate the following trends: • increase in ROC area scores and reduction in outliers percentages; • positive impact of increasing the population from 5 to 10 members (June 2004); • some deficiency in the skill of the system can be identified after the system upgrades occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels), BUT • scores are encouraging throughout DPHASE Operations Period. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
What is left? • “Close the gap” between hydrological and meteorological communities. • Learn from MAP D-PHASE experience • Exploit the wealth of information provided by probabilistic forecasts: • assess performance over different domains (North and South of the Alps), • study individual case studies, • consider basin-by-basin performance. • “Think about” increasing horizontal resolution to 7 km. • Calibrate COSMO-LEPS fcsts using reforecasts (F. Fundel , Meteoswiss). A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
ID name of Basin area (km2) no. of “stations” approx no of grid points 1 Bratislava 131.978 5278 1320 13 Wiblingen 2.247 80 20 16 Passau-Ingling 25.977 1045 260 20 Hofkirchen 47.534 1896 470 Sub-basin verification For each gridded observation, it was identified the sub-basin it belonged to. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Ranked Probability Skill Score • A sort of BSS, but “cumulated” over all thresholds. • Useful forecast system, if RPSS > 0. RPSS • RPSS always positive. • Better (worse) performance of the system over the smallest (largest) basin. • Almost identical scores using either nearest grid-point (NGP) or bilinear interpolation (BILIN). A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
CoG distance CentreOfGravity distance • Scaled (that is, dimensionless) distance between predicted (by COSMO-LEPS ensemble mean) and observed centre of gravity for each catchment. • the "cog" is an attempt to quantify the "location error" of the precipitation forecast; we have scaled the "cog" with the "mean distance" in a catchment - so a value of cog=0.3 for the catchment "Passau-Ingling" with a mean distance distmean=113 km indicates a location error of the forecasted centre of gravity of precipitation in a catchment to the observerd centre of gravity of precipitation of about 30 km. With the scaling we get a (dimensonless) number to compare the absolute location error with respect to the catchment size. • … the lower the better … CoG • The absolute figures are very low (distances of the order of a few tens of km). • Little dependence of the distance from the forecast range. • Slightly higher distances for the Passau-Ingling basin. • Best results for the Wiblingen basin (the smallest one) • Similar distances for the other basins. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Catchment Information System River-grams • Proto type, currently implemented operational. • Database of catchments will be extended to more catchments in Europe and to include all major World catchments. • Variables are currently static (always the same for all catchments), but will be ‘dynamic’ – reflecting the usage of catchments. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Outliers • How many times the analysis is outside the forecast interval spanned by the ensemble members. • … the lower the better … OUTL • The absolute figures are quite large (about 20% at d+5 range). • Lower outliers for the smallest basin. • Similar percentages for all basins at the longest forecast range. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Catchment-based verification of GME (2) period 2002-01-01 – 2002-09-30 • Verification of GME forecasts against rain gauge data from high density observation network for the four sub-catchments of the Danube with different catchment sizes: • increasing centre of gravity score („location error“) with increasing forecast time; • a positive mean coverage error for this period. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Catchment-based verification of DWD models (2) period 2002-07-19 – 2002-08-20 • Verification of GME and COSMO-EU forecasts against rain gauge observations for the catchment Hofkirchen (upper Danube): • increasing centre of gravity score („location error“) with increasing forecast time • mostly a negative mean coverage error for this period the verification period might be too short for scores to be statistically significant. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
COSMO-1km – cont. measures A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008 A.Montani; Novelties in weather forecasting PREVIEW training workshop - Mosonmagyaróvár - 19-20 June 2008
Dim 2 Possible evolution scenarios Cluster members chosen as representative members (RMs) Initial conditions Dim 1 LAM scenario Dim 2 LAM scenario LAM integrations driven by RMs LAM scenario Dim 1 Initial conditions COSMO-LEPS methodology ensemble size reduction A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
COSMO-LEPS (developed at ARPA-SIM) What is it? It is a Limited-area Ensemble Prediction System (LEPS), based on COSMO-model and implemented within COSMO (COnsortium for Small-scale MOdelling, which includes Germany, Greece, Italy, Poland, Romania, Switzerland). Why? It was developed to combine the advantages of global-model ensembles with the high-resolution details gained by the LAMs, so as to identify the possible occurrence of severe and localised weather events (heavy rainfall, strong winds, temperature anomalies, snowfall, …) generation of COSMO-LEPS to improve the Late-Short (48hr) to Early-Medium (132hr) range forecast of severe weather events. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
ROC area vs forecast range • Area under the curve in the HIT rate vs FAR diagram. • Valuable forecast systems have ROC area values > 0.6. ROC • ROC area always above 0.6. • Similar results to those obtained in terms of BSS: at low thresholds, better COSMO-LEPS performance over the smallest basin. • Possible sampling problems at 20 mm thresholds. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
The probabilistic measures we used, are the Brier Score and the ROC area. The thresholds for their computation are based on both absolute values (>1mm) and on the percentiles of the observed cumulative precipitation distribution (> 80% of observed). Probabilistic measures for verification (1) • Brier Score • The BS is the mean-squared error of the probability forecasts; it averages the squared difference between pairs of forecast probabilities and the correspondent binary observations: • where: • the observation is either ok = 1 (the event occurs) or ok = 0 (the event does not occur); • k denotes a numbering of the N forecast/event pairs. • BS ranges from 0 to 1, the perfect forecast having BS = 0. • BS is computed for a fixed precipitation threshold and fk is the fraction of the ensemble members which forecast a precipitation amount exceeding that threshold. A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Probabilistic measures for verification (2) • ROC area • It is the area under the Relative Operating Characteristics curve in the HIT rate vs FAR diagram • Hit rate (HR) and false alarm rate (FAR) are computed for each probability class k: • where: • the verification sample is subdivided into subsamples of size Nk, according to the probability with which the event is forecasted, • ok is the frequency with which the event is observed, being forecasted with a given probability and is the sample climatology. • The cumulative HRk are plotted against the corresponding cumulative FARk, generating the ROC curve. • The integral under the curve is used to indicate the skill of the forecast. • ROC area ranges from 0 to 1, the higher the better. • Useful forecast systems have ROC area values greater than 0.6 A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
COSMO-1km – Skill Scores (IMK) A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008 A.Montani; Novelties in weather forecasting PREVIEW training workshop - Mosonmagyaróvár - 19-20 June 2008
A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008 17th of July 2007
Outline • Introduction • Measure for model performance (precipitation only) • Atmospheric models re-run for PREVIEW: • GME by DWD (global, deterministic), • COSMO-EU by DWD (limited-area, deterministic), • VarEPS by ECMWF (global, ensemble), • COSMO-LEPS by ARPA-SIM (limited-area, ensemble), • COSMO-1km by IMK (limited-area, deterministic) • (tomorrow’s talk by J.W. Schipper) • Application to the Danube sub-basins • Lesson learnt A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Continuous measures • The Mean Error (ME, bias) of the precipitation forecast against the measurement (F: Forecast; O: Observations; N: sample size): • ME ranges from –infinity to infinity; the closer to zero, the better … Deterministic measures for verification (1) • Categorical measures • Consider thresholds (e.g. tp >1mm and tp > 80% of observed mean). • Scores are generated with the help of a contingency table: A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008
Atmospheric models in PREVIEW • GME by DWD (global, deterministic) • COSMO-EU by DWD (limited-area, deterministic) A.Montani; Basin-oriented verification of COSMO-LEPS X COSMO meeting – Cracow – 15-19 September 2008