1 / 35

James Brown, Julie Demargne, OHD james.d.brown@noaa.gov

AMS pre-conference workshop 23 rd Jan. 2010. Verification of ensemble streamflow forecasts using the Ensemble Verification System (EVS). James Brown, Julie Demargne, OHD james.d.brown@noaa.gov. 1. Overview. 1. Brief review of the NWS HEFS Two approaches to generating ensembles

elita
Download Presentation

James Brown, Julie Demargne, OHD james.d.brown@noaa.gov

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AMS pre-conference workshop 23rd Jan. 2010 Verification of ensemble streamflow forecasts using the Ensemble Verification System (EVS) James Brown, Julie Demargne, OHD james.d.brown@noaa.gov 1

  2. Overview • 1. Brief review of the NWS HEFS • Two approaches to generating ensembles • “Bottom-up” (ESP) vs. “top down” (HMOS) • 2. Verification of streamflow ensembles • Techniques and metrics • Ensemble Verification System (EVS) • 3. Example: ESP-GFS from CNRFC 2

  3. Brief review of the NWS • HEFS 3

  4. Bottom-up (“ESP”) Data assimilator Raw weather and climate forecasts The “uncertainty cascade” Hydrologic observations Final hydro-meteorological ensembles Hydrologic model(s) Atmospheric pre-processor Raw hydrologic ensembles Weather and climate observations Hydrologic post-processor Final hydrologic ensembles = HEFS component = Data source

  5. Top down (HMOS) Hydrologic observations Hydrologic model(s) Raw hydrologic forecasts HMOS hydrologic post-processor Final hydrologic ensembles = HEFS component = Data source

  6. Pros and cons of “ESP” • Pros • Knowledge of uncertainty sources • Can lead to targeted improvements • Dynamical propagation of uncertainty • Cons • Complex and time-consuming • Always residual bias (need post-processing) • Manual intervention is difficult (MODs) 6

  7. Pros and cons of HMOS • Pros • Simple statistical technique • Produces reliable ensemble forecasts • Uses single-valued (e.g. MOD’ed) forecasts • Cons • Requires statistical assumptions • Benefits are often short-lived (correlation) • Lumped treatment (no source identification) 7

  8. Status of X(H)EFS testing • Pre-Processor • Post-Processor • HMOS • Data Assimilation

  9. 2. Verification of streamflow ensembles 9

  10. A “good” flow forecast is..? • Statistical aspects • Unbiased (many types of bias….) • Sharp (doesn’t say “everything” possible) • Skilful relative to baseline (e.g. climatology) • User aspects (application dependent) • Sharp • Warns correctly (bias may not matter) • Timely and cost effective 10

  11. Statistical aspects • Distribution-oriented verification • Qis streamflow, a random variable. • Consider a discrete event (e.g. flood): {Q > qv}. • Forecast (y) and observe (x) many flood events. • How good are our forecasts for {Q>qv}? • Joint distribution of forecasts and observations • “calibration-refinement” • “likelihood-base-rate” f(x,y) = a(x|y) ∙ b(y) f(x,y) = c(y|x) ∙ d(x) 11

  12. (Some) attributes of quality • Calibration-refinement: a(x|y)·b(y) • Reliable if (e.g.): • “When , should observe 20% of time” • Sharp if: • “Maximize sharpness subject to reliability” • Likelihood-base-rate: c(y|x)·d(x) • Discriminatory if (e.g.): • “Forecasts easily separate flood from no flood” 12

  13. (Some) quality metrics • Exploratory metrics (plots of pairs) • Lumped metrics or ‘scores’ • Lumps all quality attributes (i.e. overall error) • Often lumped over many discrete events • Include skill scores (performance over baseline) 3. Attribute-specific metrics • Reliability Diagram (reliability and sharpness) • ROC curve (event discrimination) 13

  14. Exploratory metric: box plot 5 4 3 2 1 0 -1 -2 -3 -4 -5 EPP precipitation ensembles (1 day ahead total) Highest member ‘Error’ for 1 forecast Zero error line “Blown forecasts” 90 percent. 80 percent. 50 percent. Error (ensemble member - observed) [mm] 20 percent. 10 percent. Lowest member Precipitation is bounded at 0 A ‘conditional bias’, i.e. a bias that depends upon the observed precipitation value. 0 10 20 30 40 50 60 70 80 Observed precipitation [mm] 14

  15. Lumped metric: Mean CRPS 1.0 0.8 0.6 0.4 0.2 0.0 Observed: FX(q)=Pr[X≤q] Forecast: FY(q)=Pr[Y ≤q] • Then average across • multiple forecasts • Small scores = better • Note quadratic form: - can decompose - extremes count less Cumulative probability 0.0 10 20 30 40 50 60 Flow (Q) [cms] 15

  16. Attribute: rel. diagram “When flooding is forecast with probability 0.5, it should occur 50% of the time.” Actually occurs 37% of time. From sample data, flooding forecast 23 times with prob. 0.4-0.6 Observed probability of flood given forecast “Sharpness plot” 50 0 Frequency 0.0 0.2 0.4 0.6 0.8 1.0 Forecast class 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Forecast probability of flood 16

  17. The Ensemble Verification System (EVS) 17

  18. The EVS • Java-based tool • GUI and command line. GUI is structured…. • Verification (at specific locations) • Add locations, data sources, metrics etc. • Aggregation (across locations) • Compute aggregate performance • 3. Output (graphical and numerical) 18

  19. Three stages (tabbed panes) Metrics Navigation Details of selected metric. Basic params. of selected metric 19

  20. 3. Example application 20

  21. N. Fork, American (NFDC1) CNRFC NFDC1: dam inflow. Lies on upslope of Sierra Nevadas. NFDC1 13 NWS River Forecast Centers 21

  22. Data available (NFDC1) • Streamflow ensemble forecasts • Ensemble Streamflow Prediction system • NWS RFS (SAC) w/ precip./temp. ensembles • Hindcasts of mean daily flow 1979-2002 • Forecast lead times 1-14 days ahead • NWS RFS (SAC)is well-calibrated at NFDC1 • Observed daily flows • USGS daily observed stage • Converted to discharge using S-D relation 22

  23. Box plot of flow errors (day 1) ‘Errors’ for one forecast 1000 800 600 400 200 0 -200 -400 -600 -800 Largest + Observed value (‘zero error’) 90% 80% Median Error (forecast - observed) [CMS] 20% 10% Largest - High bias Low bias 99th % (210 CMS) 0 200 400 600 800 1000 1200 1400 1600 Observed mean daily flow [CMS] 23

  24. Precipitation (day 1, NFDC1) 125 100 75 50 25 0 -25 -50 -75 -100 Observed value (‘zero error’) “Blown” forecasts Forecast error (forecast - observed) [mm] High bias Low bias 0 10 20 30 40 50 60 70 80 90 100 Observed daily total precipitation [mm] 24

  25. Lumped error statistics Tests of ensemble mean Lumped error in probability 25

  26. Reliability Day 1 (>50th%): sharp, but a little unreliable (contrast day 14). No initial condition uncertainty (all forcing). Day 14 (>99th%): forecasts remain reasonably reliable, but note 99% = only 210 CMS. Also note sample size. 26

  27. Next steps • To make EVS widely used (beyond NWS) • Public download available (see next slide) • Published in EM&S (others on apps.) • Ongoing research (two examples) • 1) Verification of severe/rare events • Will benefit from new GEFS hindcasts • Detailed error source analysis • Hydrograph timing vs. magnitude errors (e.g. Cross-Wavelet Transform) 27

  28. www.nws.noaa.gov/oh/evs.html www.weather.gov/oh/XEFS/ Relevant published material. Full download; user’s manual (100 pp.); source code; test data; developer documentation etc. 28 28

  29. Follow-up literature • Bradley, A. A., Schwartz, S. S. and Hashino, T., 2004: Distributions-Oriented Verification of Ensemble Streamflow Predictions. Journal of Hydrometeorology, 5(3), 532-545. • Brown, J.D., Demargne, J., Liu, Y. and Seo, D-J (submitted) The Ensemble Verification System (EVS): a software tool for verifying ensemble forecasts of hydrometeorological and hydrologic variables at discrete locations. Submitted to Environmental Modelling and Software. 52pp. • Gneiting, T., F. Balabdaoui, and Raftery, A. E., 2007: Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society Series B:Statistical Methodology, 69(2), 243 – 268. • Hsu, W.-R. and Murphy, A.H., 1986: The attributes diagram: A geometrical framework for assessing the quality of probability forecasts. International Journal of Forecasting, 2, 285-293. • Jolliffe, I.T. and Stephenson, D.B. (eds), 2003: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. Chichester: John Wiley and Sons, 240pp. • Mason, S.J. and Graham N.E., 2002: Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation, Quarterly Journal of the Royal Meteorological Society, 30, 291-303. • Murphy, A. H. and Winkler, R.L., 1987: A general framework for forecast verification. Monthly Weather Review, 115, 1330-1338. • Wilks, D.S., 2006: Statistical Methods in the Atmospheric Sciences, 2nd ed. Academic Press, 627pp. 29

  30. Additional slides 30

  31. Verification metrics 31

  32. Three stages (tabbed panes) Metrics Navigation Details of selected metric. Basic params. of selected metric 32

  33. Properties of selected location Data sources Verification parameters Locations Output data 33

  34. Common properties of discrete locations Verification units (discrete locations) Aggregation units Output data location 34

  35. Metrics for selected unit Verification / Aggregation units Output options Lead times available 35

More Related