1 / 26

Experiences concerning fuzzy-verification and pattern recognition methods

Experiences concerning fuzzy-verification and pattern recognition methods. Ulrich Damrath. Outlook. Results on operational verifcation for winter and summer month An approach concerning significance test of „fuzzy“-verification results

belita
Download Presentation

Experiences concerning fuzzy-verification and pattern recognition methods

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Experiences concerning fuzzy-verification and pattern recognition methods Ulrich Damrath

  2. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Outlook • Results on operational verifcation for winter and summer month • An approach concerning significance test of „fuzzy“-verification results • Estimation of consistency of forecasts using a pattern recognition method (CRA method by Beth Ebert)

  3. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 GME COSMO-EU COSMO-DE Fractions skill score for forecasts of GME, COSMO-EU and COSMO-DE for December 2008, forecast time 06-18 hours

  4. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 GME COSMO-EU COSMO-DE ETS upscaling for forecasts of GME, COSMO-EU and COSMO-DE for December 2008, forecast time 06-18 hours

  5. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Global Europe Germany Fractions skill score for forecasts of GME, COSMO-EU and COSMO-DE for August 2009, forecast time 06-18 hours

  6. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Global Europe Germany ETS upscaling for forecasts of GME, COSMO-EU and COSMO-DE for August 2009, forecast time 06-18 hours

  7. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Examination of statistical significance of „fuzzy“-verification results using bootstrapping • Basic idea of bootstrapping: • Repeat a resampling all elements of a given in a sample of forecasts and observations as often as necessary (N times) and calculate the relevant score(s) • Calculate from N scores statistical properties of the sample such as mean value standard deviation, confidence intervals and quantiles • Application to „fuzzy“-verification • Resampling is done using „blocks“. • Blocks are defined as single days. • Number of resampling cases: N=Days*100 • Calculation scores from N samples for NT thesholds and NW windows • Calculation of quantiles for each window and threshold

  8. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Values and quantiles 0.1 and 0.9 for Upscaling ETS GME, period June - August 2009 Germany

  9. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Values and quantiles 0.1 and 0.9 for Upscaling ETS COSMO-EU, period June - August 2009 Germany

  10. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Values and quantiles 0.1 and 0.9 for Upscaling ETS COSMO-DE, period June -August 2009 Germany

  11. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Next step: Evaluation of significance • First impression: Is the result of Model 1 better than the result of Model 2? • Significance hypothesis checked using a Wilcoxon-test (IDL-code RS_TEST)

  12. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Differences between GME and COSMO-EU Germany ETS(COSMO-EU) - ETS(GME) Significance test COSMO-EU better than GME COSMO-EU worse than GME

  13. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Differences between GME and COSMO-DE Germany ETS(COSMO-DE) - ETS(GME) Significance test COSMO-DE better than GME COSMO-DE worse than GME

  14. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Differences between COSMO-DE and COSMO-EU Germany Significance test ETS(COSMO-DE) - ETS(COSMO-EU) COSMO-DE better than COSMO-EU COSMO-DE worse than COSMO-EU

  15. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Differences between COSMO-DE and COSMO-EU Significance test ETS(COSMO-DE) - ETS(COSMO-EU) COSMO-DE better than COSMO-EU COSMO-DE worse than COSMO-EU

  16. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Example of good precipitation forecast of COSMO-DE Zeigten die numerischen Modelle und die statistischen Prognose- --------------------------------------------------------------- verfahren Signale für das Ereignis? ----------------------------------- Die Numerik zeigte im Vorfeld vermehrt Signale für kräftige Konvektion. Während diese bei GME und COSMO-EU recht breit gestreut und pauschal auftraten, signalisierten mehrere COSMO- DE-Läufe eine linienartige Struktur mit unwetterartigen Zellen (auf Basis der 1- bzw. 3-stündigen RR-Prognosen) im Grenzbereich von Hessen zu NRW und Niedersachsen. Diese Linie trat dann in den Mittags- und frühen Nachmittagsstunden tatsächlich auf, wenn auch nicht 100%ig kongruent, aber doch in der Nähe, so dass in diesem Fall von einer guten Prognose gesprochen werden kann (mehr dazu siehe "Zentraler UW- Sofortbericht" der VBZ).

  17. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Example of good precipitation forecast of COSMO-DE 3h-precipitation observation 10.08.2009 09-12 UTC 3h-precipitation forecast of COSMO-DE valid 10.08.2009 12 UTC, left 03 UTC +9h, right 06 UTC +6h.

  18. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Example of good precipitation forecast of COSMO-DE compared to other models

  19. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Example of good precipitation forecast of COSMO-DE compared to other models

  20. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 About consistency and inconsistency • Forecasters are interested in consistent model forecasts. • But due to growing of errors during forecast time forecasts consistency cannot be expected concerning all properties of the forecasted fields! • Inconsistency: Differences between forecasts that are valid for the same time concerning different properties of the forecasted fields (properties of the pattern, values at special points of interest, extreme values, ...) • Differences between the forecasted fields concerning • phase, • amplitude • and the remaining part

  21. Observed Forecast • Entity-based QPF verification (rain “blobs”) • by E. Ebert (BOM Melbourne) • Verify the properties of the forecast rain system against the properties of the observed rain system: • location • rain area • rain intensity (mean, maximum) CRA error decomposition The total mean squared error (MSE) can be written as: MSEtotal = MSEdisplacement + MSEvolume+ MSEpattern Configuration for the current study: - “Observations”: forecasts: 06-30 hours - Forecasts : forecasts: 30-54 hours and forecasts: 54-78 hours

  22. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Dark :forecasts 30-54 h Light:forecasts 54-78 h

  23. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Dark :forecasts 30-54 h Light:forecasts 54-78 h

  24. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Dark :forecasts 30-54 h Light:forecasts 54-78 h

  25. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Dark :forecasts 30-54 h Light:forecasts 54-78 h

  26. U. Damrath: Experiences concerning fuzzy-verification ... -COSMO GM, Offenbach 2009 Summary • Scores like Fractions skill score and ETS from upscaling show in general advantages of COSMO models compared to GME. • This is true especially for summer months. • For winter months all models have nearly the same quality for low precipitation amounts and large window sizes for averaging. • Significance test lead to the results, that: • The advantages of COSMO models compared to GME are statistically significant for most window sizes and precipitation amounts. • The differences between COSMO-EU and COSMO-DE are not significant altough there are systematical differences for different precipitation amounts and window sizes. • There are some cases with very useful precipitation forecasts of COSMO-DE compared to COSMO-EU from the view of forecasters. • A study about the consistency of precipitation forecasts showed - it could be expected , but now it is proved - that: • Forecasts of high precipitation amounts are less consistent than those for low precipitation amounts. • Pattern errors contribute most to forecast errors. • During winter months volume errors are higher than displacement errors. • During summer months displacement errors are higher than volume errors.

More Related