WG5: STATUS of activities

1. WG5: STATUS of activities WG5, COSMO SMC, 1-2 February 2017

2. PP INSPECT status Anastasia Bundel (1), Flora Gofa (2) (PP leaders), Dmitry Alferov (1), Elena Astakhova (1), Petra Baumann (4), Dimitra Boukouvala (2), Ulrich Damrath (3), Pierre Eckert (4), Alexander Kirsanov (1), Xavier Lapillonne (4), Joanna Linkowska (5), Chiara Marsigli (6), Andrea Montani (6), Anatoly Muraviev (1), Elena Oberto (7), Maria Stefania Tesini (6), Naima Vela (7), Andrzej Wyszogrodzki (5), André Walser (4), and Mikhail Zaichenko (1) (1) RHM (a.bundel@gmail.com), (2) HNMS, (3) DWD, (4) MCH, (5) IMGW-PIB, (6) ARPA-SIMC, (7) ARPA-PT PP INSPECT status, COSMO SMC, 1-2 February 2017

3. PP INSPECT • INSPECT PP tries to summarize the COSMO experience of applying spatial verification methods to forecast systems of very-high-resolution (1-3 km) compared to high-resolution models • INSPECT runs in parallel to MesoVICT (several INSPECT tasks involve reruns of COSMO high and very-high-resolution models for MesoVICT test cases) • Same as MesoVICT, INSPECT will focus on the ensembles and variables besides precipitation • In addition to targeting the goals of MesoVICT, INSPECT is expected to provide COSMO users more choice of verification domains and reference data - newer and longer periods, two complex terrains (the Alps and the Caucasus) • Finally, INSPECT will try to provide criteria for deciding which methods are best suited to particular applications

4. Status highlights • The PP INSPECT is extended until August 2017 due to delays in some tasks, with overall FTE amount almost unchanged • Delayed tasks in question are extended with unused FTEs shifted to later periods PP INSPECT status, COSMO SMC, 1-2 February 2017

5. Tasks involving reruns of MesoVICT test cases:completedongoingpending • MCH: COSMO-1 reruns for ALL MesoVICT cases are DONE and available at WG5 repository • ARPA-SIMC: ECMWF-IFS reruns for cases 1 and 2 Update of the status is expected • COSMO-Ru2-EPS: rerun of 10 members using clusterisation procedure for the 1st MesoVICT case, rerun of complete ensemble (51 member) is under preparation • MCH: COSMO-E rerun is completed Thus, all necessary data are or will be available soon, but with some delays. Thus, the application of spatial methods to the ensembles (INSPECT priority) is also delayed for some tasks. PP INSPECT status, COSMO SMC, 1-2 February 2017

6. Tasks involving development of routines for running spatial methods • RHM, HNMS, INGW-PIB: examples of scripts for neighborhood methods, CRA, SAL, and MODE are uploaded at the WG5 repository • ARPA-PT: Works are ongoing to add two new developments in VAST: inclusion of time dimension and the possibility to operate with other variables besides precipitation, primarily TCC (to be concluded in Jan 2017) PP INSPECT status, COSMO SMC, 1-2 February 2017

7. The policy on Additional Verification Toolsand verification data and formats • HNMS, RHM, IMGW-PIB: The policy on Additional Verification Tools and verification data and formats is prepared and approved by the STC PP INSPECT status, COSMO SMC, 1-2 February 2017

8. Tasks involving the application of spatial methods to deterministic forecasts • The routines developed earlier are applied to MesoVICT cases -> this should facilitate the comparison of different approaches and summarizing the results • However, same as for MesoVICT, it was found that the setup of experiments was not completely identical (different precipitation accumulation periods, time slots, etc.), and we should bear this in mind making conclusions about different methods PP INSPECT status, COSMO SMC, 1-2 February 2017

9. Status • Wind verification using DIST for both wind speed and direction (Maria-Stefania) • HNMS, IMGW-PIB: comparisons of traditional and spatial scores for extreme events from MesoVICT cases (SAL, MODE, neighborhood) • RHM: CRA experiments using R SpatialVx for the Sochi region and MesoVICT cases PP INSPECT status, COSMO SMC, 1-2 February 2017

10. Tasks involving the application of spatial methods to ensemble forecasts • A progress is made, although some of these tasks are delayed (RHM ensemble reruns) • SAL for COSMO-LEPS • DIST method applied to the COSMO-LEPS ensemble for MesoVICT cases using two sets of observations PP INSPECT status, COSMO SMC, 1-2 February 2017

11. SAL for COSMO-LEPS 21/6 12 UTC 22/06 12UTC A<0 underestimation, More red points S> 0 too large or too flat objects are produced. PP INSPECT status, COSMO SMC, 1-2 February 2017

12. Objective verification of COSMO-LEPS Main features: variable: 6h cumulated precip (0-6, ..., 18-24 UTC); period: all mesoVICT cases (Jun – Sep 2007); region: 43-50N, 2-18E (D-PHASE area); method: NGP, BILIN, BOXES of different sizes; obs: JDC or VERA; fcst ranges: 0-6h, 6-12h, ..., 126-132h; thresholds: 1, 5, 10, 15, 25, 50 mm/6h; system: COSMO-LEPS; scores: ROC area, RPS, RPSS, Outliers, ...

13. Verification with boxes of the distributions (DIST) Grid point forecast • The verification can be made in terms of: • Average value • Maximum value • 50th percentile (Median) • 75th, 90th, 95th percentiles in a box • Operationally, we consider two measures of precipitation: • the average volume of water deployed over a specific region; • the rainfall peaks occurring within the same region. Station observation A.Montani

14. Probabilistic prediction of tp: ROC area (boxes) tp_06h > 1mm • Consider the event: average 6-hour precipitation exceeding 1 mm within boxes of increasing size 0.5 x 0.5 1.0 x 1.0 • More and more similar skill for the 2 verification networks as the box size increases. • Slightly higher skill when COSMO-LEPS is verified against VERA gridded analysis. • The skill increases with increasing box size. 1.5 x 1.5 A.Montani; The COSMO-LEPS system.

15. Follow-up of MesoVICT • The 2nd MesoVICT international meeting was held in Bologna, Italy, on 21-23 September 2016 under the umbrella of ARPA-SIMC. • During the same days/place, the INSPECT meeting also took place PP INSPECT status, COSMO SMC, 1-2 February 2017

16. Main outcomes of the MesoVICT meeting in Bologna and intersection with INSPECT • Most spatial methods applied so far indicate that the results are sensitive to accum. period, domain size, number of objects, and model resolution • Need for geometric cases for complex terrain • Include time space (Maria Stefania, VAST) • Develop method ranking idea (some ideas of Anatoly Muraviev) • Include uncertainty in observations (Andrea, others??) • inference (statistical testing) • testing SpatialVx (RHM, IMGW-PIB) • information exchange of basic settings (! Necessary for comparison of methods) • Overview results paper, special AMS collection? (Also the need of publications for INSPECT final report) PP INSPECT status, COSMO SMC, 1-2 February 2017

17. Task 5: Guidelines for relative usefulness of various spatial methods in decision-making • Goal:Using the knowledge gained from INTERP, ICP and MesoVICT projects and the long time series of spatial verification scores, to try to identify the relative usefulness of each spatial method for precipitation and other weather parameters. To propose a kind of Guidelines for using spatial methods within the COSMO community • Most of subtasks in Task 5 are delayed. • To simplify the preparation of the Task 5 deliverables, a questionnaire was sent to participants, the answers are expected in mid-March PP INSPECT status, COSMO SMC, 1-2 February 2017

18. Questionnaire sent to INSPECT participants • Method applied (related to an INSPECT Task) and objectives • Short description of the dataset (forecast-observation data), adaptation required, software for the method application • Main findings (plots and explanation) • Characteristics of the method applied: • efficiency in calculation time • ability to deal with different density of observations • stability against observation errors • ability to assess the added value of high-resolution models • ability to address specific issues of interest (e.g. location errors, intensity errors, performance at different scales) etc. • Other • Possible comparison with other methods applied by the user on the same dataset • Other comments (optional) PP INSPECT status, COSMO SMC, 1-2 February 2017

19. Common Plot Reports 2016-2017 FTE attribution (since sep 2014) STC was in favor of the proposition for FTE attribution for common plots • STC suggests to restrict the task to the plots on the common area, which are the ones bringing a benefit since really comparable. STC requests for a deeper analysis in the report • STC suggests that FTE should be required to perform some additional explanation of results which will help to detect and improve outdated installations and correct model setups. • STC decides to keep conditional verification in the task, but requests that the conditions should be decided every year new by the WG3a/b. WG5, COSMO SMC, 1-2 February 2017

20. operational coarse (-7km) res models DWD (ICON-EU) RHM IMGW DWD (ICON-EU) MCH COMET NMA HNMS WG5, COSMO SMC, 1-2 February 2017

21. Standard Verification on Common Area WG5, COSMO SMC, 1-2 February 2017

22. CP HRES scenario 2 WG5, COSMO SMC, 1-2 February 2017

23. operational COSMO EPS models IMGW (0.025) LEPS (0.0625) DWD (0.025) COMET (0.09) MCH (0.02) WG5, COSMO SMC, 1-2 February 2017

24. Common Plot Reports 2016-2017 • Responsible for CP reports and analysis A.Kirsanov (RHM) • Guidelines report for current year was prepared by WGc • Keep the coarser resolution comparison (~5-7km) for one year (2mT, 2mTd, 10mWsp, MSLP, 6h-24h Preci) • Add 12UTC run • Add high res model comparison on a common area with restricted model participation • Motivation that models can predict extreme values associated with dangerous weather (rare binary events). Extremal dependence scores were added on seasonal reports (EDI – SEDI for preci) • Operational EPS verification is postponed to be discussed next year • Conditional verification not included • Webpage is updated with first two seasons : JJA2016, SON2016 • http://www.cosmo-model.org/content/tasks/verification.priv/common/analytics/2016-2017/default.htm

25. Issues to be considered • Purpose of CP statistics and reports; Current use • Maintain VERSUS as common software for the score preparation (Verification tools strategy – agenda item) • Apply other more suitable scores or methods in CP activity (spatial, upper air, ECMWF headline scores) • Conditional verification: Can this be transferred to individual testing; Definition of a PT with WG3 support ? (agenda item) • Communication of CP-derived information to COSMO management and other WGs (Conditional Verification). Perform several individual CV tests and organize meeting with WG3 to be presented and discussed WG5 COSMO SMC, 1-2 February 2017

26. VERSUS: Common Verification Software • EPS score discrepancy (CRPS) testing is pending • Problems with bugs revealed during Common Plots activity and NWP test suite – slow response is creating problems • Maintenance phase since Sep. 2016 - VERSUS SCA : Antonio Vocino • “The Long-Term Maintenance services are planned to be assured by C.O.Met (the new Operational Meteorological Center Of the Italian Air Force) for 2016/2017 COSMO year , conditionally to the available resources” WG5, COSMO SMC, 1-2 February 2017

27. WG5 repository New WG5 repository was established ion new COSMO server (T. Andreadis). It is aimed to host all relevant material to Common Plots presentation (guidelines, BUFR meansurements), PP INSPECT material, AVT (additional verification software) as described in Policy for verification input format, etc. http://www.cosmo-model.org/rep/repository/wg5 WG5, COSMO SMC, 1-2 February 2017

28. ADDITIONAL SLIDES

29. Common Verification Plots Conditional Verification experiments

30. Common Verification Plots for Common Area Conditional Verification • 2mT verification with the following criteria (one condition): • Soil water content ≥4 (condition based on forecasts) • Soil water content <2 (condition based on forecasts) • Wind Speed verification with the following criteria (one condition): • Roughness length <0.2 m(condition based on forecasts) • Roughness length >1m (condition based on forecasts) • Need to report the sample size or percentage of cases in each category • Easier to draw conclusions when difference from unconditioned errors on parameters are plotted

31. 17th COSMO General Meeting, Wroclaw - WG5 parallel session, 7.9.2015: Common Plot reports • Cooperation with WG3a (2014-2015): Processing of verification feedback on model development • Scope: Direct and Indirect effects of the option of COSMO model that considers the interaction between the turbulence scheme and the wind tendency due to the presence of subgrid scale variability (LTKESSO) will be evaluated. • 2mT, Wind speed and MSLP verification with the following criteria (one condition): • subgrid scale orography variance (SSO_STDH) ≤25m (condition based on forecasts) • subgrid scale orography variance (SSO_STDH) ≥100m (condition based on forecasts)

32. 17th COSMO General Meeting, Wroclaw - WG5 parallel session, 7.9.2015: Common Plot reports • Conditional Verification experiments: 2015-2016 • With Communication with WG3b Coordinator • Study: Impact of soil type and vegetation height on the performance of various weather parameters in the lower atmosphere • Reasoning: The soil-vegetation representation in the model involves the fluxes of energy and water at the surface and determines the exchange of heat, moisture and momentum between the surface and the atmosphere. This has consequently an impact on near surface weather parameters (temperature, dewpoint, wind) • Scope: Evaluate the relevant effect in bias of modeled weather elements due to the variability on terrain characteristics

33. Task description Score Production Preparation of input data and calculation of seasonal statistics over a common area according to the guidelines derived on an annual basis from WG5 (http://www.cosmo-model.org/content/tasks /verification.priv/common/guidelines.pdf) for each participating model. This Task includes conditional verification tests performed over this area. IFS and ICON driving model statistics have also been added. Reporting Processing of data from all models for each parameter and conditional verification test in appropriate format R scripting for production of graphs (cross model representation) Preparation of final annual report Commenting of significant errors or discrepancies between models Preparation of web graphics Long term trend calculations WG5, COSMO SMC, 1-2 February 2017

34. Main Issues to be considered • Purpose of CP statistics and reports, current use • Main focus of operational verification in each service • Maintain VERSUS are common software for the score preparation • Conditional verification: Can this be transferred to individual testing; Definition of a PT with WG3 support • Apply other more suitable scores or methods in CP activity (spatial, upper air, ECMWF headline scores) • Availability and number of SYNOP reports for each sub-area (scenario): necessary to perform analysis (volunteer?) • Communication of CP-derived information to COSMO management and other WGs (Conditional Verification) – Preparation of short report to be distributed to WG3a,b • Currently only 00UTC run is tested WG5, COSMO SMC, 1-2 February 2017

35. Decisions from GM2016: • A. Keep present common coarse domain for an additional year either with fewer models and/or adding 4-5km (COMET,HNMS) to the list • Pros: keep long term trend • Cons: restrict participation and usability of feedback • B. Substitute with HRES verification over multiple domains and varying resoltutin • Pros: test model performance over a variety of geographical characteristics (how many and which?), participation of more services • Cons: increase work load for score extraction, increase work load of CP analysis, loss of long term trend • C. Add HRES (one domain) to current test (A) • Pros: test model performance over a variety of geographical characteristics (how many???), participation of more services, keep long term trend • Cons: increase further work load for score extraction, increase work load of CP analysis, loss long term trend • D. Expand operational verification for CP to EPS WG5, COSMO SMC, 1-2 February 2017

36. OBSERVATION MASK Verification grid (e.g. 1.0x1.0)

37. Probabilistic prediction of tp: ROC area (ngp vs bilin) • Area under the curve in the HIT rate vs FAR diagram; the higher, the better … • Valuable forecast systems have ROC area values > 0.6. • Consider two events: 6-hour precipitation exceeding 1 mm and 10 mm. tp_06h > 1mm tp_06h > 10mm • 1mm: similar performance of the system with respect to the 2 verification networks. • 10 mm: higher skill when COSMO-LEPS is verified against VERA gridded analysis. • Almost no impact of the verification technique (ngp ~ bilin) for both thresholds. A.Montani; The COSMO-LEPS system.

38. COSMO-LEPS was verified against the following networks/methodologies for all mesoVICT cases: Verification networks and methodologies A.Montani; The COSMO-LEPS system.