1 / 13

Verification Module for the DTC Ensemble Testbed

DET Ensemble Workshop 2010. Verification Module for the DTC Ensemble Testbed. Ed Tollerud , Tara Jensen, Barb Brown ( also Yuejian Zhu, Zoltan Toth , Tony Eckel , Curtis Alexander, Huiling Yuan,…)

elke
Download Presentation

Verification Module for the DTC Ensemble Testbed

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DET Ensemble Workshop 2010 Verification Module for the DTC Ensemble Testbed Ed Tollerud, Tara Jensen, Barb Brown (also Yuejian Zhu, ZoltanToth, Tony Eckel, Curtis Alexander, Huiling Yuan,…) Module 6 Objective: Provide a portable, inclusive toolset for Testing and Evaluation applications Wally Clark

  2. Module 1: Configuration Module 2: Initial Perturbations Module 3: Module Perturbations Module 4: Statistical Post-Processing External Input (HMT, HWT, HFIP, etc) Module 5: Product Generation DET MODULES Module 6: Verification

  3. DET Ensemble Workshop 2010 Motivation and Requirements • Provide ensemble and probabilistic verification utilities that are of value to: • Model development • Research applications • Operational feedback for forecasters • DET tasks and user-driven projects • Develop readily applicable routines to: • Assemble and process ensemble forecasts • Apply standard probabilistic verification algorithms • Employ newly-developed evaluation methods • Display results in thorough and intuitive fashion Questions for breakout group Ensuring applicability/acceptability to user groups?

  4. DET Ensemble Workshop 2010 Motivation and Requirements (continued) • Module should include verification utilities that are: • Objective but user-driven (‘honest broker’) • Comprehensive but portable • Flexible and extensible • User-friendly and supported (internet-enable) • State-of-the-art (Object-based; spatial ensemble verification) • Utilities should be designed for: • Retrospective diagnostic studies and intensive forecast evaluation • Real-time application during quasi-operational field exercises (HWT and HMT experience) Questions for breakout group How to insure portability to different ensemble forecast systems and applications? Critical operational or real-time capabilities? Other usage types?

  5. DET Ensemble Workshop 2010 Verification Algorithms, Options, and Related Displays • Comprehensive set of standard and probabilistic scores • Why not all that are feasible? • Examples of standard scores: MAE, ME, RMSE, ETS, POD,FAR, Bias, Odds ratio, etc. • Probabilistic scores and display types: Brier score and Brier skill scores and decomposition (reliability and resolution), Rank and Continuous Rank Probability Scores, ROC curve, Talagrand Diagram, Rank Histogram Questions for breakout group How to involve user group in contributions to displays and scoring algorithms? DET Visitor Program?

  6. Examples: Individual member standard scores from HMT/HWT

  7. DET Ensemble Workshop 2010 Examples: Brier Skill Score and Decomposition Products; Talagrand Diagram (Zhu et al.); Dispersion Diagram (Tony Eckel)

  8. DET Ensemble Workshop 2010 Examples: New Display Products (RAL and NAEFS) Questions for breakout group How to develop and solicit new display possibilities? (Ensemble Products Workshop II?)

  9. DET Ensemble Workshop 2010 Verification Dataset Choices • QPF verification example: QPE possibilities include: • Operational Gage datastreams • Other Mesonet gage sets (CoCoRahs; OK mesonet; ….) • Radar products (Stage IV; Q2;…) • Satellite estimates from many sources • Start-up default for verification: NCEP/EMC verification datasets; e.g., RTMA, PrepBufr,…. • Analysis of Record (AOR) as source of high-resolution data Questions for breakout group Offer data options? Develop standardized datasets? Supply QC applications? Incorporate MADIS? How to build in data uncertainty?

  10. DET Ensemble Workshop 2010 Verification Packages and Products to Incorporate • MET tools: Advantages include: • Strong synergy with other DTC tasks (HMT; HWT; HFIP) • In-house DTC product with good support • Continual updates, revisions, and novel additions • Evolving ensemble and probabilistic capabilities • Portable display systems with wide use (r; ncl) • ‘Metviewer’ interactive web display • Replication of EMC capabilities • Emphasis on confidence intervals and spatial methods • Other verification products and packages with valuable input: • EMC probabilistic products • OHD hydrologic verification products • RUC and HRRR ensemble display products • NEVS procedures for data access • AWIPS; ALPS; NPVU; AFWA; others….

  11. DET Ensemble Workshop 2010 Tentative Timeline and Milestones June 2010 – Initial probabilistic scoring utilities demonstrated aspart of HWT spring exercise October 2010 – Verification module plan completed and reviewed by WRF ensemble working group January 2011 – Expanded probabilistic scoring utilities demonstrated as part of HMT winter exercise June 2011 – Prototype real-time web-based verification site with basic capabilities completed September 2011 – Verification utility installed as full module in end-to-end workflow Questions for breakout group Contribute to other workshops (i.e., RAL Verification Workshop 11/2010; WMO Verification tutorials; others?)

  12. DET Ensemble Workshop 2010 Useful Websites and Information Sources NPVU: http://www.hpc.ncep.noaa.gov/npvu/ ESRL/AMB: http://ruc.noaa.gov/hcpf/hcpf_verif.cgi OHD: http://hydis8.eng.uci.edu/hepex/Workshops/postprocesswksp/ POSTERS/Demargne-33.pdf OHD: http://hydis8.eng.uci.edu/hepex/testbeds/Verification.htm EMC/NCEP: https://ams.confex.com/ams/pdfpapers/131645.pdf ‘r’ Stats Pkg: http://cran.r-project.org/web/packages/verification/verification.pdf MET Verification example: http://verif.rap.ucar.edu/eval/hmt/2010/ Others: Questions for breakout group How to promote portability between user groups?

  13. DET Ensemble Workshop 2010 Breakout Group Discussion – User Community Issues • Prioritized list of ensemble-relevant verification products • Dataset choices and quality Questions for breakout group Ensuring applicability to user groups? Offer data options? Develop standardized datasets? Supply QC applications? Incorporate MADIS? How to build in data uncertainty? How to insure portability to different ensemble forecast systems and applications? Critical operational or real-time capabilities? Contribute to other workshops (i.e., RAL Verification Workshop 11/2010; WMO Verification tutorials; others?) How to involve user group in contributions to displays and scoring algorithms? DET Visitor Program? How to promote portability between user groups? How to develop and solicit new display possibilities? (Ensemble Products Workshop II?)

More Related