1 / 44

WG5: STATUS of activities

WG5: STATUS of activities. WG5, COSMO SMC meeting, 24-25 January 2018. main activities. Rfdbk VERSUS. PP INSPECT Cooperation with WG4 & WG7. COMMON PLOTS NWP Test suite Cooperation with WG3a,b. Common Plot Reports 2017-2018 Guidelines prepared and sent in Oct (wg5 web repository).

brianacook
Download Presentation

WG5: STATUS of activities

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WG5: STATUS of activities WG5, COSMO SMC meeting, 24-25 January 2018

  2. main activities Rfdbk VERSUS PP INSPECT Cooperation with WG4 & WG7 COMMON PLOTS NWP Test suite Cooperation with WG3a,b

  3. Common Plot Reports 2017-2018 Guidelines prepared and sent in Oct (wg5 web repository) • Keep the coarser resolution comparison (~5-7km) for one year (trend since 2011) • Add high res model comparison on two “semi-common” areas: different climatology • Keep 12UTC run despite for coarser resolution comparisons • Extremal dependence scores – EDI for 6h and 24h precipitation • Add LCC on top of TCC, also categorical scores with thresholds • Add wind gust categorical scores • Add wind performance rose diagrams for selected graphs WG5, COSMO SMC meeting, 24-25 January 2018

  4. Feedback from questionnaire ~7km ~1-4km EPS WG5, COSMO SMC meeting, 24-25 January 2018

  5. Common Area 1: coarse resolution DWD (ICON-EU) RHM IMGW DWD (ICON-EU) MCH COMET NMA HNMS WG5, COSMO SMC meeting, 24-25 January 2018

  6. Common Area 1: coarse resol ICON-EU (DWD), COSMO-7 (MCH), COSMO-ME4 (COMET), COSMO-GR4 (HNMS), COSMO-PL7 (IMGW), COSMO-RU7 (RHM), COSMO-5M (ARPAE) and the two driving models ECMWF-IFS, ICON. Standard Verification (seasonal): Continuous parameters - T2m, MSLP, Td 2m, WSpeed, TCC. Scores: ME, RMSE Dichotomic parameters – Precipitation. Scores: Contingency table, FBI, ETS. Cumulating: 6h and 24h Thresholds: 0.2, 0.4, 0.6, 0.8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 16, 18, 20 mm/6h and mm/24h Dichotomic parameters – TCC (30km radius method). Scores: Contingency table, FBI, ETS, CSI. Intervals: [0,25], (25, 75), [75,100] Dichotomic parameters – Windgust(3D method height optimized): Obs: WMO Code 11233, Fcs:VMAX_10M (187,201). Scores: Contingency table, FBI, ETS, CSI. Thresholds: 12.5, 15, 20 m/sec. EDI (Extremal Dependency Index) Type: Standard, Period: Seasonal, Parameter: Precipitation, Method: 15 km radius method, Index: EDI (Dichotomic). Thresholds: 1, 5, 10, 15, 20, 25, 30 mm/6h and mm/24h. WG5, COSMO SMC meeting, 24-25 January 2018

  7. Common Area 1: coarse resolution Wind rose diagrams (by Maria Stefania Tesini) During GM 2018, new representation of wind speed/dir performance was presented by MSTesini and it was agreed that it will be tested through CP activity this year A number of stations from Common Area 1 are selected and scores have to be provided for them according to the directions given. WG5, COSMO SMC meeting, 24-25 January 2018

  8. The “Performance – Rose” A novel diagram in which scores and type of errors of wind forecast are summarized according to directions WG5, COSMO SMC meeting, 24-25 January 2018

  9. For each station, 10m-wind observations (hourly or 3/6-hourly or other time aggregations) and corresponding data predicted by model are categorized in octants for wind direction and in classes for wind speed. • Light:ws<10 knots • Light-Moderate:10≤ ws < 20 Knots • Moderate:20≤ ws < 30 Knots • Strong: ≥30 Knots • For each class a separate plot is done WG5, COSMO SMC meeting, 24-25 January 2018

  10. Verification scores are plotted as symbols: • The colors represent the two types of event • Black: Correct speed class and direction • Purle: Correct speed but with a tolerance in direction (1 octants) • Perfect score 1 is in the innermost ring • Red line represents the number of forecast in the specific class • Blue line represent the number of observations in the specific class WG5, COSMO SMC meeting, 24-25 January 2018

  11. Common Area 2 WG5, COSMO SMC meeting, 24-25 January 2018

  12. Common Area 3 WG5, COSMO SMC meeting, 24-25 January 2018

  13. Common Area 2 COSMO-DE (DWD:0.025), COSMO-1 (MCH:0.01), COSMO-PL2.5 (IMGW:0.025), COSMO-IT (CoMET:0.025), COSMO-I2 (ArpaE:0.025) and the two driving models ECMWF-IFS, ICON-EU. Standard Verification: Continuous parameters- T2m ,Td,Wspeed,TCC. Scores: ME, RMSE. Forecast Step: every 3 hours Dichotomic parameters – Precipitation (8 km radius method). Scores: FBI, ETS. Cumulating: 6h and 24h Dichotomic parameters – Windgust(3D method height optimized): Obs:WMO Code 11041, Fcs:VMAX_10M (187,201). Scores: Contingency table, FBI, ETS, CSI. Thresholds: 12.5, 15, 20 m/sec. Cycle: Only 00UTC cycle will be verified for continuous and dichotomic parameters. Extremal Dependence Index Type: Standard, Period: Seasonal, Parameter: Precipitation, Method: 8 km radius method, Index: EDI (Dichotomic). Thresholds: 1, 5, 10, 15, 20, 25, 30 mm/6h and mm/24h. WG5, COSMO SMC meeting, 24-25 January 2018

  14. Common Plot Activity – Status and Future • Seasonal summary graphics available on COSMO web • Annual report prepared by D. Boucouvala available on • COSMO web - Presentation during GM • Remains as an activity to encourage all services to contribute to the minimum necessary verification practices to monitor COSMO model performance (trend since 2011) over same domain and periods • Common Verification Software (CSV) is currently VERSUS but depending on the feedback introducing MEC/Rfdbk for NWP Test suite, this could change • Transition period (2018-2019) for CP activity NOT based on CVS but with same guidelines to be followed. Introduction of FSS scores over common domains WG5, COSMO SMC meeting, 24-25 January 2018

  15. Common Verification Software • Maintenance phase for VERSUS • VERSUS installation on ecgate is used as part of the NWP test suite (currently tested against Rfdbk) • Slow process in bug correction (wind gust verif error - pending) • No additional small developments (Score extraction for Daily Cycle, EPS score calculation improvement) • Rfdbk currently used only from DWD • NMA to be trained to use MEC/Rfdbk readymade scripts on ecgate for NWP Test suite (meeting during ICCARUS) • Depending on the feedback, a discussion need to done in WG5 for initiating an effort to expand its use for CP • Several obstacles for FF preparation (BUFR2NETCDF, MEC) • PP C2I promotes the use of Rfdbk(Jan 2019). Verification Guidelines on verification practices to be provided by WG5 • Necessity to introduce a PT with the participation of all services (GM2018) for switch of CVS

  16. PP INSPECT Anastasia Bundel (1), Flora Gofa (2) (PP leaders) and Dmitry Alferov (1), Elena Astakhova (1), Petra Baumann (4), Dimitra Boukouvala (2),, Ulrich Damrath (3), Pierre Eckert (4), Alexander Kirsanov (1), Xavier Lapillonne (4), Joanna Linkowska (5), Chiara Marsigli (6), Andrea Montani (6), Anatoly Muraviev (1), Elena Oberto (7), Maria Stefania Tesini (6), Naima Vela (7), Andrzej Wyszogrodzki (5), and Mikhail Zaichenko (1), André Walser (4) • PP INSPECT was extended until the end of 2017 • Extension was necessary due to delays in Task 4 on the application of spatial verification methods to ensembles and Task 5 on the Guidelines for using spatial methods • Final deliverable is not yet ready (~ ICCARUS) WG5, COSMO SMC meeting, 24-25 January 2018

  17. Spatial methods Displacement methods Filtering methods Many√ • Neighborhood • (Ebert, 2008) • Scale Decomposition • DIST method • Features-based • Contiguous Rain Area (CRA) (Ebert and McBride, 2000) • Method for Object-based Diagnostic Evaluation (MODE) (Davis et al., 2006) • SAL technique (Wernli et al., 2008) • Field deformation √ √ √ √ √ √ √ √ √ Main benefit of INSPECT “that the wide range of spatial verification methods available will become commonly used within the COSMO community” is achieved.

  18. A number of reruns are performed for MesoVICT test cases within the project • MCH: COSMO-1 reruns for ALL MesoVICT cases are done and available at WG5 repository • ECMWF-IFS reruns (51 member) for cases 1 and 2 (8 initial dates) (Andrea Montani), • COSMO-E reruns (21 member) for cases 1 and 2 (8 initial dates) (André Walser), • COSMO-Ru2-EPS (51 member) for case 1 (1 initial date) and case 2. Reruns are interpolated to a VERA grid by Manfred Dorninger (Austria) -> Testbed for spatial methods applications, Easy to use and compare COSMO visibility increased WG5, COSMO SMC meeting, 24-25 January 2018

  19. Scripts adapting existing free software for neighborhood, CRA, SAL, and MODE methodsAvailable in WG5 repository • For the most part, the software is based on free R SpatialVx package (developed by E. Gilleland). For SAL (D.Boucoucala) and Neighborhood (J.Linkowska) comparisons are made with alternative packages -> bug fixing of SpatialVx (scripts available in WG5 repository) • VAST development by N. Vela (ARPA-PT): inclusion of time dimension and the possibility to operate with other variables besides precipitation, primarily TCC • http://www.cosmo-model.org/rep/repository/wg5 WG5, COSMO SMC meeting, 24-25 January 2018

  20. Visualization of long time series of neighborhood scores - Gain • reduces the available information by showing the most relevant part on a one-dimensional plot which resembles usual station based verification • Experience of DWD and MCH can be used as a first step to includespatial scores in Common Area plots WG5, COSMO SMC meeting, 24-25 January 2018

  21. Comparison of COSMO-EU to COSMO-DE – upscaling Equitable Threat Score (ETS) Precipitation amount W i n d o w s I z e Threshold 0.1 0.2 0.5 1 2 5 10 20 50 1.625 0.825 0.425 0.225 0.125 0.075 0.025 Mesh width of COSMO-EU WG5, COSMO SMC meeting, 24-25 January 2018

  22. Compact visualization of total precipitation FSS: to focus on the useful scale for a given lead-time and threshold (MCH) FSS: Fractions Skill Score P is the event fraction in the neighborhood. Score as a function of leadtime for a single meaningful scale

  23. Intensity-scale: filtering method (F. Gofa, HNMS) Error image is expressed as the sum of components on different spatial scales by performing a 2-dimentional discrete Haar wavelet decomposition. for various spatial scales (l =1,..L=7 that corresponds to XXkm). The spatial scales refer to the spatial scale of the error and not that of the precipitation features or their displacement as it happens in the neighborhood methods MesoVICT case 1: 20070621-15, Intensity Scale Skill Small scales 1x1 02 2x2 scales have skill close to zero, while large scales exhibit large skill. COSMO1 ISS graphs exhibit that errors arev due to displacements of small spatial scale features are more important compared to those of COSMO-2

  24. Case study2007.09.25.06, 6h precipitation, threshold>=5mm Object-based methods by IMGW-PIB • MODE, CRA, SAL VERA COSMO 2 unmatchedobject (false) Selected feature pairings based on total interest obs feature mod feature total interest 1 1 0.898 WG5, COSMO SMC meeting, 24-25 January 2018

  25. SAL: Maria Stefania Tesini and Daniele D'Alessandro A single parameter to evaluate the structure, amplitude, or location error in forecast is not enough when the precipitation field complexity is too high Precipitation intensity is overestimated in one place and underestimated in another, but the amplitude value is close to zero WG5, COSMO SMC meeting, 24-25 January 2018

  26. SAL for EPS: ways to display large amount of ensemble data (D. Boucouvala) 20/6 Median 20/6 20/6 WG5, COSMO SMC meeting, 24-25 January 2018

  27. SAL for EPS: introducing observation uncertainty (D. Boucouvala) Objects comparison for probability of precipitation >= 2mm Probability threshold =1 LEPS Observations Preci >= 2 mm Preci >= 2 mm for all 16 members 3 h Precipitation 21/6 12 UTC S=1, A=0.38, L=0.3 WG5, COSMO SMC meeting, 24-25 January 2018

  28. Object matching for EPS, MesoVICT case 1 (A.Bundel) Probability of each observed object is found and the ensemble skill can be estimated using the BSS, for example 2007062021 COSMO-E ensemble, first 6 of 21 members, precip threshold >0.5 mm/1h Probabilities of each of 5 observed objects: 1/21 20/21 10/21 19/21 14/21 Difficulty of such an approach: No merging of objects is possible as the list of observed objects must be the same for matching with all ensemble members Only to etsimatethe probability of objects paired

  29. Processing “big data”: CRA for STEPSnowcasting at RHM (Anatoly Muraviev) • Verification period May-September 2017 • 9 DMRL-radars in Central Russia), 1100x1300 km, 256х256 grid points • Forecasts for large intense precipitation areas only are analyzed • 10 min timestep until 3 h WG5, COSMO SMC meeting, 24-25 January 2018

  30. An example of STEPS SMOLENSK DMRL (RUDL) STEPS 12 frames. Rather good deterministic forecast in shape, intensity, speed, direction. There is a tendency to fragment connected areas. WG5, COSMO SMC meeting, 24-25 January 2018

  31. SpatialVx: firstиlastobjects RAKU STEPS STEPS RAKU Object recognition and matching is not straightforward

  32. R-SpatialVx, CRA STEPS forecast: Radar RAKU 20170517_1120 Such CRA tables are aggregated over the whole period May-September 2017 ! WG5, COSMO SMC meeting, 24-25 January 2018

  33. The integral longitude and latitude CRA shift characteristics: • In longitude, objects are systematically shifted to east for some radars, and to west to other radars • But in latitude, the objects are systematically forecasted more to the north for almost all radars • Critical shift value is set up to 35 km • Both in latitude and longitude, not less than 50% of forecasts don’t exceed the critical value until 90 minutes WG5, COSMO SMC meeting, 24-25 January 2018

  34. Main results, 1 • A number of reruns of high-resolution deterministic and ensemble systems are performed for MesoVICT cases -> this facilitated comparison of methods and increased COSMO visibility, provided a testbed • Scripts are written to adapt available software for COSMO data -> this facilitated implementing spatial methods in COSMO institutions • Several ways of compact visualization of neighborhood scores that can be implemented as part of Common Verification WG5, COSMO SMC meeting, 24-25 January 2018

  35. Questionnaire sent to PPINSPECT participants • Method applied (related to an INSPECT Task) and objectives • Short description of the dataset (forecast-observation data), adaptation required, software for the method application • Main findings (plots and explanation) • Characteristics of the method applied: • efficiency in calculation time • ability to deal with different density of observations • stability against observation errors • ability to assess the added value of high-resolution models • ability to address specific issues of interest (e.g. location errors, intensity errors, performance at different scales) etc. • Other • Possible comparison with other methods applied by the user on the same dataset • Other comments (optional)

  36. Main results • Intensity-Scale measure provides a tool to gain information on the spatial scale of the error and not of precipitation features • Object-based SAL (Structure-Amplitude-Location) method was found easier to implement than MODE and CRA methods as it doesn’t require pair-wise matching of observed and forecast objects, but the SAL scores must be considered carefully as they refer to average characteristics in an area • Object-based MODE and CRA methods provide more information (compared to SAL) as they estimate matched pairs of observed and forecast objects WG5, COSMO SMC meeting, 24-25 January 2018

  37. Main results, 3 • For MODE and CRA, it was found not feasible to find one optimal universal matching function, in particular for high-resolution fields with objects of complex shape. - For lower precip thresholds (-> wider features), matching based on the criterion “the size of objects less than the average size of objects” gives more reasonable results. - For higher thresholds (-> small features) other matching functions seem more promising, such as minimum boundary separation • Influence of different options, such as smoothing, was studied. Smoothing can be unnecessary for intense precipitation evaluation. Option for splitting objects is desirable sometimes. WG5, COSMO SMC meeting, 24-25 January 2018

  38. Main results • Applications of DIST, SAL and CRA methods to ensembles were made and new approaches on summarizing performance over various members and time accumulations were proposed • First results of experiments on introducing observation uncertainty into the spatial methods are given WG5, COSMO SMC meeting, 24-25 January 2018

  39. What was NOT completely fulfilled and needs further work (1) • Explicitly introducing orography factor • Wind characteristics were only explored by Maria-Stefania. It was found difficult to apply upscaling DIST method to wind direction. Applications to wind and other variables besides precipitation should be developed (promising attempt at RHM to apply CRA to areas of TCC climate trend sign). WG5, COSMO SMC meeting, 24-25 January 2018

  40. What was not completely fulfilled and needs further work (2) • Applications to ensemble systems • Introducing observation uncertainty • Processing big data (an experience of estimating STEPS nowcasting during the summer 2017 using neighborhood and CRA methods) Hence, a possibility of new PPs (in cooperation with WG4 and WG7) WG5, COSMO SMC meeting, 24-25 January 2018

  41. Priority Project Idea: Extreme events or High Impact weather verification (with collaboration with WG4 & WG7) Basic Idea: Understanding the forecast quality is critical in high impact weather. Important is also to verify such weather in a meaningful way to the end users (forecasters, emergency management, public). Weather parameters of interest in extreme ranges: precipitation, wind (+gusts), min-max temperature Forecasts: NWP products of few hours, nowcasting products, ensemble systems (account for forecast uncertainty). In general, models may not capture the intensity of high impactevents due to: Sub grid scaleprocesses, coarseresolution, difficulty Representingprocesses Observations: Hard to obtain for some parameters in useful temporal and spatial scales, importance of observation uncertainty (time, location, magnitude) Impact/prediction relation: it can be a mismatch between what models can provide and what information warnings need to be made for (lightning, hail, wind gusts, fog,…) WG5, COSMO SMC meeting, 24-25 January 2018

  42. Useful verification of high impact events • Guides users in making better decisions based onforecasts • How reliable is the forecast at capturing events? • What are typical errors in timing / location / intensity ofevents? • Are the forecastsbiased? • Informs modellers / forecast system developers on how toimprove • forecasts • Do the forecasts show the rightbehaviour? • What is the nature of theerrors? • Assists managers in monitoring forecastperformance 6 WG5, COSMO SMC meeting, 24-25 January 2018

  43. Priority Project Idea: Extreme events or High Impact weather verification • Verification methods: Usually same statistical approaches as those for everyday forecasts are used, but not always suitable. • Traditional contingency table scores (identify the time limit thatforecast ismorewrong thanright, guidance on when to switch from deterministic forecaststoprobabilisticones) • Categorical extreme dependency scores (independent of the base rate) for coarser forecasts • Probabilistic forecast verification (not suitable for single case studies, often hard to interpret from non trained users, CRPS, GDS ) • Spatial verification approaches for high resolution forecasts (tolerant to observation uncertainty, some are good also for aggregating performance, e.g. MODE provides much more information about performance than traditionalscores) • Scores that are directly connected to model climatology over a location or period (SEEPS) • New scores that are more relevant to the decision makers? Forecast/Impact relation? WG5, COSMO SMC meeting, 24-25 January 2018

  44. Thank you WG5, COSMO SMC meeting, 24-25 January 2018

More Related