1 / 25

Evaluation of Potential Performance Measures for the Advanced Hydrologic Prediction Service

This report discusses the evaluation and trial application of potential performance measures for the Advanced Hydrologic Prediction Service (AHPS). It reviews probabilistic forecast verification measures and provides recommendations for existing and future measures. The report also explores accuracy measures and data archival for AHPS.

sbarnhart
Download Presentation

Evaluation of Potential Performance Measures for the Advanced Hydrologic Prediction Service

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation of Potential Performance Measures for the Advanced Hydrologic Prediction Service Gary A. Wick NOAA Environmental Technology Laboratory On Rotational Assignment with the NWS Office of Hydrologic Development November 2003

  2. Overview • Performance measures for the Advanced Hydrologic Prediction Service (AHPS) • Review of probabilistic forecast verification measures • Trial application with operational forecast data • Recommendations

  3. Advanced Hydrologic Prediction Service (AHPS) AHPS Program Performance Existing Measures Future Measures Deployment Number of forecast points Coverage area Maturity, e.g. metrics addressing Usage of forecast information Probabilistic forecast effectiveness Science Projects Performance AHPS Science Activities Development Activity Effectiveness Contribution to forecast maturity Number of science tools deployed per year Contribution to information content

  4. Probabilistic Forecast Verification • Categorical forecasts • Brier Score • Rank Probability Score (RPS)

  5. Categorical Forecasts • Transforms probabilistic forecast into a categorical forecast through selection of a probability threshold • Simple but doesn’t fully address probability

  6. Brier Score • Simple extension that fully characterizes probabilistic forecasts • Limited to occurrence of a specific event

  7. Rank Probability Score • Extension characterizing full distribution of forecasts • Ideal as science measure but added complexity a concern at program level

  8. Application of Accuracy Measures • Deterministic Measures • Probabilistic Brier Score

  9. Deterministic Application • Used National Weather Service verification database • Monthly data for 177 sites starting April 2001 • Results computed “on-the-fly” • Evaluated accuracy difference between AHPS and non-AHPS points • Considered subset of points for the North Central, Ohio, and Missouri Basin River Forecast Centers • Expressed in terms of mean absolute error and root mean square error

  10. Deterministic Results

  11. Deterministic Results

  12. Implications: Deterministic • Possible to implement something rapidly • Characterization must be defined • Existing verification database and interface inadequate

  13. Brier Score Evaluation • Sample ensemble forecasts and verification provided by Kristie Franz • 43 sites from the Ohio River Forecast Center • 11 weekly mean and monthly maximum exceedance forecasts • Forecast traces, verification, and historical data • Evaluated accuracy of forecasts for exceedance of flood stage

  14. Brier Score Evaluation • Use of all forecasts suggested very high accuracy • Use of all forecasts suggests very high accuracy • Only 17% improvement over forecasts for no flooding

  15. Brier Score Evaluation • Most revealing results obtained for forecasts where flooding occurred

  16. Rank Probability Score • Could interpret as accuracy from 76 to 91%

  17. Implications: Probabilistic • Brier score can be presented simply and meaningfully • Application limited by constraint to instances of flooding • Rank probability score addresses all forecasts but meaning is harder to express • Necessary to regularly archive ensemble forecasts and verification

  18. Recommendations • Performance measures • Data collection • Additional analyses

  19. Recommended AHPS Accuracy Measures • Deterministic River Forecast Accuracy • Probabilistic River Forecast Accuracy • Flood Forecast Accuracy

  20. AHPS Deterministic River Forecast Accuracy • Percent accuracy of mean daily streamflow for days 1-3 • Evaluates short-term hydrograph forecasts

  21. AHPS Probabilistic River Forecast Accuracy • RPS derived accuracy of weekly mean streamflow exceedance • Evaluates AHPS weekly chance of exceedance forecasts • Express as percent accuracy • Apply to week 2 and week 4 forecasts

  22. AHPS Flood Forecast Accuracy • Derived from Brier score and weekly maximum stage forecasts • Simple evaluation of weekly exceedance forecasts • Express as percent accuracy for cases where flooding occurred • Apply to week 2 and week 4 forecasts

  23. Recommended Data Archival • Forecast Data • Ensemble forecast traces at selected points • Verification • Corresponding stage/streamflow observations • Historical Data • Ensure consistency with forecast quantities

  24. Further Analysis • Apply proposed measures to enhanced set of archived data • Evaluations will help illustrate where forecast skill exists and improvements are possible • Explore possible alternatives for collection of climatological data • Final metric selection best made after more comprehensive evaluation

  25. Closing Notes • Accuracy measures provide a bridge between programmatic and science activities and metrics • Important to recognize limitations of accuracy measures and continue consideration of other metrics

More Related