CBS/OPAG-IOS Workshop on Radar Data Exchange Exeter, UK, 24-26 April 2013

1. 4.1 A critique of the existing data models for weather radar data exchange as presented in item 3.1 docs CBS/OPAG-IOS Workshop on Radar Data Exchange Exeter, UK, 24-26 April 2013 Daniel Michelson, SMHI, Sweden

2. Data exchange model has four components: File format – container for storing data in physical files. Data/information model – the way in which information is organized/represented, either in computer memory or files, independently of file format. Envelope – extra header announcing/describing the contents of the information being exchanged. Protocol – mechanisms for communication over a network. WMO

3. E.g. an early BALTRAD HTML exchange message Content-Disposition: form-data; name="<baltrad_frame_xml/>" Content-Type: multipart/form-data; charset=UTF-8 Content-Transfer-Encoding: 8bit <?xml version="1.0" encoding="UTF-8"?> <baltrad_frame> <header mimetype="multipart/form-data" sender="local_production_system"/> <content channel="test_channel" name="pas1.h5" type="file"/> </baltrad_frame> --s7Wj4EEwDrDPwOdPM30KMQzaEpfnyl Content-Disposition: form-data; name="<baltrad_frame_file/>"; filename="pas1.h5" Content-Type: multipart/form-data Content-Transfer-Encoding: binary ‰HDF ÿ(… binary payload ODIM_H5 file …) --s7Wj4EEwDrDPwOdPM30KMQzaEpfnyl-- test footer

4. Different categories of file formats Transmission – to facilitate data transfer from site to central facility. Can be optimized for network load balancing, e.g. ray-by-ray. Production – contains all data and metadata required to derive higher-order outputs (e.g. QC:ed data, products) from input data. Exchange – representation of data or product for sharing but not necessarily further processing, e.g. a radar composite or vertical profile. Archive – e.g. to represent data from many different observation systems for storage. WMO

5. Basic data exchange concepts “Passive” Simple availability, e.g. “pull” Sender/recipient are aware of each other but transact indirectly, e.g. “push” (GTS) Directory polling or file-system event monitoring “Active” Sender/recipient are aware of each other and interact/transact directly E.g. BALTRAD WMO

6. GTS for radar data exchange? Advantages Already exists Active networking Disadvantage Hierarchical organization causes delays For example: WMO faster? Exeter  Offenbach  Norrköping  Riga

7. BALTRAD for radar data exchange? Positives Already exists Active networking WIS integration already demonstrated (2010) Challenges Still being developed Requires dedicated software Not yet officially recognized as part of WIS WMO

8. File format: GRIB • WMO standard • Already in use with national composite products in RA IV. • Demonstrated use for exchange. (GRIB1) • No single official, centrally maintained, software. • No mature information model representation.

9. File format: TITAN • Already in use in two regions (I and III) • Storm tracks also available, e.g. XML • Open software, centrally maintained • Efficient for transmission, production, exchange • Documentation • No DQ preparedness • Prepared for dual-pol moments? • Built-in compression? • Unknown outside meteorology

10. File format: BUFR • WMO standard. • Suitable for point and profile data. • Very flexible, “can encode anything”. • Demonstrated use in Region VI, ODIM_BUFR • “Unwieldy” for non-trivial data such as radar. • Many mutually incompatible software and data model implementations. • No single official, centrally maintained, software. • Bottleneck to academia. • Not a marketable skill. • Largely unknown outside meteorology.

11. File format: HDF5 • Central, open, modern software developed/maintained by HDF Group. • Bundled for various OSes. • Established in Earth Sciences. • ODIM_H5 prepared for dual-pol and DQ. • ODIM_H5 is “policed”. • Efficient for production and exchange. • ODIM_H5 compatibility between regions V and VI • Steep learning curve. • Currently unsuitable as transmission format. • ODIM_H5 not an orthodox information model. • Ambiguities in ODIM_H5 risk incompatibilities.

12. File format: netCDF, version 4 • Central, open, modern software developed/maintained by NCAR. • Bundled for various OSes. • Established in Earth Sciences through “CF Conventions”. • GIS-compliant. • “CF Radial” prepared for dual pol? • Requires HDF5, added abstraction layer. • No mature CF Convention yet for operational radar, although “CF Radial” for R&D. • Unprepared for DQ?

13. ODIM in OPERA, February 2013 Green = ODIM_H5 Brown = ODIM_BUFR Yellow = Both WMO Input data Output products

14. Summary Terminology needs clarifying. “Data model”, “exchange”, “file format”, etc. Several file formats in use. No orthodox information model as such in use.Not necessarily a disadvantage. Tight connections between file formats and information models. Good solutions exist already. WMO

15. Thank you for your attention • Daniel Michelson • Swedish Meteorological and Hydrological Institute • Norrköping, Sweden • daniel.michelson@smhi.se