1 / 49

Verification Training Operational Verification System (s) @ FMI

Verification Training Operational Verification System (s) @ FMI. Pertti Nurmi Head, Meteorological Research Applications Finnish Meteorological Institute 6 th International Verification Methods Workshop, New Delhi, 15 March 2014. Introduction - 1.

adli
Download Presentation

Verification Training Operational Verification System (s) @ FMI

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VerificationTrainingOperational Verification System(s)@ FMI Pertti Nurmi Head, Meteorological Research Applications Finnish Meteorological Institute 6th International Verification Methods Workshop, New Delhi, 15 March 2014

  2. Introduction - 1 Some basic verification measures (to include in Verification Packages) • Continuous variables • Mean, median, standard deviation of forecast and observation • Mean Error, Bias (ME) • Mean Absolute Error (MAE) • Root Mean Squared Error (RMSE) • LEPS • Categorical variables • Frequency bias (FBI) • Proportion correct (PC) • Probability of Detection (POD) • False Alarm Ratio (FAR) • Threat Score (TS or CSI) • Equitable Threat Score (ETS) • Hanssen-Kuipers Score (HSS) • Odds Ratio (OR) • Heidke Skill Score (HSS) • SEDS (for rare events) • EDI (for rare events) • … covered by Martin Göber during Tutorial

  3. Introduction - 2 More verification measures (to include in Verification Packages) • Probability & Ensemble forecasts • Brier Skill Score (BSS) • Relative Operating Characteristic (ROC) • ROC area • (Continuous) Ranked Probability Score (RPS, CRPS) • (Continuous) Ranked Probability Skill Score • (RPSS, CRPSS) • Verification basics & visualization • Mapped forecasts and observations • Time series of forecasts and observations at selected sites • Scatter plots • Quantile-quantile plots … covered by Laurie Wilson & Barb Brown

  4. Introduction - 3 • Summary of desirable specific properties for a verification measure: • Should remain useful for rare events (Most conventional scores become unusable beyond the 90-92 percentile) • Behaviour should not depend on the base rate (climatology) • Behaviour should not depend on the base value • (magnitude of verified quantity) • Dependency on spatial and temporal scales and sampling of observational data should be minimised • Should be accompanied by estimates of uncertainty - error bars • Dependency on the verification grid should be minimised • Should take both hits and false alarms into account • Should converge quickly for relatively small samples • Should not reward “hedging”

  5. Introduction - 4 Some issues… on operational verification Who … … wants to know? Goals of the verification? (in this case… it’s You) What … … kind of parameter are we evaluating ? What are its characteristics ? (e.g., continuous, probabilistic) … thresholds are we looking at? … forecast resolution (e.g. point-wise, station groups, area-averages) ? … are the characteristics of the observations (e.g. quality, uncertainty) ? How … … do we present the results (e.g., stratification/ aggregation/ visualization) ? Which … …methods and metrics to use (e.g. bias, SEDS, EDI, SEEPS etc…) ? … … …

  6. FMI Forecast Production Chain Manual products Observations Forecaster-edited Forecast database Automatically generated products from the grid database Pro p BMP2WBMP me Raw model data Fully automated products ECMWF, HIRLAM, HARMONIE models Operational Verification System @ FMI

  7. FMI Operational Real-time Verification System • Properties & Features • Web-based; Interactive; Menu-driven; Real-time; 2 languages • Daily forecast vs. observation comparison • Periodical, monthly, seasonal, annual statistics • All basic – and some novelty - verification measures • Comparison: Forecaster end product vs. model; model vs. model • Database coverage: Grid data + stations over Finland • Time coverage: 2003 until present • Even dating back to 1979 for 3 stations + Sochi Olympics version = Today’s theme ! Operational Verification System @ FMI

  8. FMI Operational Real-time Verification System • Technical Properties • Verification interface • Visualization : xhtml + css • Dynamic structure : php + javascript • php bridges databases and user interface • javascript : interaction with user interface menus • Observations : FMI Oracle database • Verification database : PostgreSQL • Graphics : JpGraph • Non-interactive • Will be replaced in the next upgrade + Sochi Olympics version = Today’s theme ! Operational Verification System @ FMI

  9. Result presentation menu • Options • Results on map • Hits on map • Daily monitoring • Model comparison • by stations • by station groups • Tabulated results • by stations • by station groups • Model vs. observations • Data matrices • Official results Zoom Verification User Interface: Web browser default page Operational Verification System @ FMI

  10. Result presentation format Options Producer Parameter Analysis hour Lead time Period type Verif. Measure Presentation Station group Zoom Master selection menu Operational Verification System @ FMI

  11. Zoom Master selection menu Operational Verification System @ FMI

  12. Zoom Master selection menu Operational Verification System @ FMI

  13. Zoom Master selection menu Operational Verification System @ FMI

  14. Zoom Master selection menu Operational Verification System @ FMI

  15. Continuous Zoom Categorical Master selection menu Operational Verification System @ FMI

  16. Zoom Master selection menu Operational Verification System @ FMI

  17. Let’s then move to Sochi 2014 Winter Olympics Operational Verification System @ FMI

  18. Operational Verification System @ FMI

  19. Result presentation format Options Producer Parameter Analysis hour Lead time Period type Verif. Measure Station group Master selection menu Operational Verification System @ FMI

  20. Master selection menu Operational Verification System @ FMI

  21. Master selection menu Operational Verification System @ FMI

  22. Master selection menu Operational Verification System @ FMI

  23. Master selection menu Operational Verification System @ FMI

  24. Continuous Categorical Precipitation Master selection menu Operational Verification System @ FMI

  25. Master selection menu Operational Verification System @ FMI

  26. Ski • Biathlon Ski Jump Valley • Alpine Ski Master selection menu Operational Verification System @ FMI

  27. Let’s (try) go online … Operational Verification System @ FMI

  28. http://frost-2014.fmi.fi/user: sochipwd: tbc Operational Verification System @ FMI

  29. Finishing motto There is nothing as difficult as forecasting the weather(correct)... … except (proper) forecast verification ! - Allan Murphy, RIP Operational (real-time) verification systems help in both ! - Pertti

  30. Extra slides... Note: Following examples are from the Finland Verification user interface, so the menu selections are somewhat different to the Sochi Version Operational Verification System @ FMI

  31. We don’t necessarily want to see this kind of output ! Operational Verification System @ FMI

  32. Operational Verification System @ FMI

  33. Operational Verification System @ FMI

  34. Operational Verification System @ FMI

  35. Operational Verification System @ FMI

  36. Operational Verification System @ FMI

  37. Operational Verification System @ FMI

  38. Operational Verification System @ FMI

  39. Operational Verification System @ FMI

  40. Operational Verification System @ FMI

  41. Operational Verification System @ FMI

  42. Operational Verification System @ FMI

  43. Operational Verification System @ FMI

  44. Operational Verification System @ FMI

  45. Operational Verification System @ FMI

  46. Operational Verification System @ FMI

  47. Use of Cumulative Distribution 1 Obs Cumulative Probability 0 Precipitation (mm) 0 TPmax No Rain “Light Rain” “Heavy Rain” 1 - SEEPS = ½ ( PSS dry/wet + PSS light/heavy ) is a 3-category score Light/heavy threshold adapts to local climate pheavy plight = 2 x pheavy plight pdry Light/heavy threshold for Balmoral in October = 5mm ‘Dry’ < 0.25 mm 0.25 mm SEEPS  Stable Equitable Error in Probability Space

More Related