Implementing WMS interface on Imagery Database (BDI)

1. Implementing WMS interface on Imagery Database (BDI) Frédéric GUILLAUD DSI/DEV EGOWS Meeting KNMI, June 2009

2. Overview • Context • Towards SOA for production and workstations (Marie-Françoise talk) • DWG Meteorology and WMS IE • Soprano imagery database (BDI) • WMS implementation • Issues • Still to do • Conclusion

3. Context • IT SOA pilot project • Goals • Improve our skill with SOA architecture and tools (ESB) • Improve our skill with OGC standards • Early identify issues • Make technical decisions • DWG meteorology IE on WMS • Calendars compatibles (Over 6 month by the end of 2009) • WMS BDI may be a service involved in IE experiments • To do this, WMS should be deployed in DMZ • This lead to security issues

4. Soprano Imagery database (BDI) : Before • RDBMS Oracle • 3 days real-time data online (“blob” data type). Data is not archived. • About 250 datasets • Legacy catalogue content not always consistent … • Native formats • BUFR, TIFF-MeteoFrance (TIFF with - open - specific MF tags…) • Native extent depends on dataset origin (MSG, METOP, other geostationary platforms, radar data ..) • Native CRS • “Space View” (One-Point Perspective Projection ?) and “Polar Stereographic” • Data access services • Low level (application binary executable) • No API available • No remote access possible

5. Soprano Imagery database : product overview • Basic radar reflectivity measurements (for each radar) • Mosaics • Precipitation flux (through reflectivity conversion) • Precipitation accumulation (5’, 15’, …) • Forecasted intensities (2PIR algorithm)

6. Soprano Imagery database : product overview • Satellite data • From MSG, METOP, GOES platforms … • Composite CMS products, …

7. Soprano Imagery database : products overview • Composite products • Radar precipitation intensity calibrated with ground measurements (ANTILOPE) • Satellite-Based fog diagnostics (CARIBOU)

8. WMS BDI features : Now • About 250 Layers • GetMap returns only one LAYER at a time • LAYER name is an internal identifier (tokens delimited by “_”, in order to make easier mapping WMS – RDBMS data access • Pre-defined STYLE : Each LAYER has a default colour map, which is the official colour map on operational workstations at Meteo-France. • User defined styles are not yet supported (SLD=) • If TIME not specified, GetMap returns the last image • Outputs formats : image/gif; image/png • Output CRS : only EPSG:4326 and Polar Stereographic at the moment • No support for metadata layers (e.g. pixel timestamp, pixel quality, ...) but may be WCS scope ?

9. WMS Implementation : Reuse of legacy components • Data access component : “RDImage” • Application Binary Interface (executable) • Performs data extraction from BDI on criteria : • Data type (radar, satellite …) • Production type (observation, forecast …) • Data origin (sensor platform, system, algorithm …) • Native domain covered by the dataset • Parameter (rain intensity, accumulation, radiance/wave length …) • Image processing component : “TransImage” • Application Binary Interface (executable) • Performs : • Sampling a sub-domain • Format translation • Coordinate reference system operations (re-projection)

10. WMS Implementation • Meteo-France WMS framework • Implements WMS interface in PHP • Check on GetMap – GetCapabilities consistency • WMS 1.1.1 or 1.3.0 • WMS BDI driver • Links GetMap parameters to RdImage & TransImage ones • Depending on GetMap parameters, invoke TransImage if necessary • Generate GetCapabilities response (XML)

11. WMS implementation overview GetMap GetCapabilities WMS Meteo-France WMS Framework (PHP) BDI Driver (PHP) RDImage (.exe C/C++) TransImage (.exe C/C++) Capabilities (C ) OCI BDI (Oracle)

12. Issue : GetCapabilities layering • GetCapabilities response • Mainly product metadata • Describe product types (characteristics common to product instances) • GetCapabilities tree : Observations & Measurements approach • 3 Levels of layers : • Process • Product type, Platform, Algorithm e.g Satellite MSG • Feature Of Interest • Domain e.g Europe-Atlantic … • Observed property • Parameter e.g IR 108 • Leaf layer is a named product type (queryable)

13. Issue : GetCapabilities layering • Layer granularity • Balance LAYER / Multi-dimensional LAYER (LAYER plus DIM) • E.g : LAYER=SA__OBS__MSGC__V_EURATL__WV-62 Or LAYER= SA__OBS__MSGC__V_EURATLDIM_Wavelength=62 Or LAYER=SA_OBSDIM_Platform=MSGC DIM_NativeDomain=V_EURATL DIM_Wavelength=62 • Parameter names, units, … for DIMENSION usage • We need controlled vocabularies E.g : DIM_Wavelength=62, 73, 108 , DIM_AccumulationTime=5, 15, 60 … DIM_BaseTime , DIM_Quality, DIM_PixelTimeStamp • The granularity of the layers will impact : • The efficiency of catalogue searching • The size of GetCapabilities response and performances

14. Issue : Keywords & Titles • Issue : Finding the layer name to perform GetMap request … • Layer names are quite complicated (internal identifiers), but unique : • Ex : “RD_CPO_P__OBS__RD_ETRANGER__V_PIEMONT__REFLECTIVITE” • About 250 layers in GetCapabilities • At the moment “Title” == “Name” (that is bad !) • difficult to be interpreted by a human… Title should be explicit • No abstracts, no keywords • To help catalogue searching, we are looking for keywords, “Controlled vocabularies”, “Ontology” … • E.g “METEOSAT” “MSG”, “METEOSAT-8” “METEOSAT-9”, … • Keywords should be hierarchic • E.g “TEMPERATURE -- CLOUD -- TOP” • Catalogue searching engine tested : GeoNetwork • Possible keywords references : • Global Change Master Directory (GCMD) • Common WMO code tables (Table C13, …) • AMS Glossary of Meteorology (GMET) • EUMETSAT , HMA (for satellite data) ? • Others ? To be defined ?

15. GeoNetwork evaluation as BDI catalogue

16. ISO 19128 – ISO 19115 relationship

17. Issue : ISO 19128 – ISO 19115 relationship • How to keep consistency between WMS GetCapabilities (ISO 19128) and ISO19115 Metadata common fields ? • Where is the metadata reference or entry point ? • SOPRANO ISO 19115 metadata database • To be created … • Partially harvested from VGISC metadata (probably not sufficient for BDI because focussed on GTS products) • BDImage (or other databases) descriptive schema content • Incomplete (keywords, abstracts are missing …) • Not always consistent (because of the database history…) • Difficult to change (because of backward compatibility)

18. Issue : Time • Time information • It would be useful to include product instance availability information (ISO 19108) in GetCapabilities response : • Potential availability (time range plus step) • Real time availability (real time list : may be complex and CPU time – consuming • There is a TIME parameter in GetMap, but no dedicated tags for TIME information in GetCapabilities response WMS 1.1.1 and 1.3 • GetMap : no support for multi-dimensional time (forecasted imagery) e.g. : Forecasted precipitation intensity = f ( Base time, Validity Time ) • TIME semantics may be ambiguous e.g : Sampling Time, Validity Time, Result Time, Base Time … • Possible approach : • Using “Dimension” / “ExtendedCapabilities” ?

19. Issue : CRS identifiers • There are three methods to identify and describe CRS • Namespaces & codes • We need CRS identifiers not yet define in EPSG database and / or AUTO namespace, others … • EPSG (not parametric) • Lot of codes to be defined ( all CRS in use at Meteo-France ? ) • AUTO2 (parametric) : • Universal Transverse Mercator AUTO2:42001 • Transverse Mercator AUTO2:42002 • Ortographic AUTO2:40003 • Equirectangular AUTO2:40004 • Mollweide AUTO2:40005 • Is it possible to define other codes ? • Ex Polar Stereographic for each prim meridian, “Space View”, .. ? • Reference (URL) to ISO 19111 documents • Proj4 “input parameters – like” ?

20. Issue : Performances • It takes about 1 seconds to process full disk MSGC image “on the fly” • Reprojection appears to be very expensive • ”On the fly” processing probably not possible if many concurrent requests. • We probably need pre-defined CRS • This would allow pre-calculation of the projection matrix for each CRS transformation (in TransImage) • WMTS implementation ? But we’ll have to pre-compute tiles for each layer, CRS, Zoom Level, Format, Time … • WMTS implementation is obviously a challenge for real-time Met data.

21. Left to do … • Lot of work on Metadata • Keywords, abstracts, …. • Link GetCapabilities response – ISO 19115 metadata • New output CRS • New output formats (does GeoTiff make sense in a WMS ?) • Support for multi dimensional times • Support for metadata layers (pixel time stamp, pixel quality, … ?) • Support for client style (SLD=) • Improve performances Limited set of CRS, projection matrix in TransImage component , WMTS • Proj4 – GDAL migration of TransImage ( GEDAL is already in use on SYNERGIE) • TransImage as service • Security topics to be investigated

22. Conclusion : BDI WMS works but … To get really interoperable WMS, we need to agree on : • Metadata implementation rules (at least keywords) • Colour Maps / Units of Measure • E.g : Radar Data : Reflectivity (DBZ) – Intensity ( Kg m-2 / h) • CRS policies (EPSG, AUTO2, …) • How to manage multiple time dimensions • Layer granularity for each product type ( balance LAYER – layer DIMENSION ) • Parameter names, units for DIMENSION usage

23. Thank you for your attention questions ?