1 / 22

DEWS: Delivering CF-NetCDF marine data through OGC Map and Coverage Services

DEWS: Delivering CF-NetCDF marine data through OGC Map and Coverage Services. Jon Blower (on behalf of DEWS partners) Technical Director Reading e-Science Centre Environmental Systems Science Centre University of Reading. Delivering Environmental Web Service (DEWS).

birch
Download Presentation

DEWS: Delivering CF-NetCDF marine data through OGC Map and Coverage Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DEWS: Delivering CF-NetCDF marine data through OGC Map and Coverage Services Jon Blower (on behalf of DEWS partners) Technical Director Reading e-Science Centre Environmental Systems Science Centre University of Reading

  2. Delivering Environmental Web Service (DEWS) • £2.2M DTI inter-enterprise computing • Serve time-critical, high-volume environmental data to end users • Health and Marine test cases • Based on OGC-compliant Web Services • Secured using NERC Data Grid mechanism • Information tailored for customers at Web Portal • Nearing completion DEWS Web Portal Other clients BMT SeaInfo Security layer (NDG) Marine Map Service (WMS) Health Data Web Service Marine Data Service (WCS) Met Office data

  3. DEWS Marine Stream tasks • Create an OGC Web Coverage Service for data downloads • Data in CF-NetCDF (WCS aggregates) • Mainly numerical forecast model data • Serve large amounts of data, securely • Create an OGC Web Map Service for data visualization • Do research on suitability of these standards

  4. DEWS modifications to OGC WCS 1.0.0 • CF-NetCDF as output format • Security • Based on WS-Security, hence SOAP messaging • (we've had fun with forcing WS-Sec implementations to interoperate!) • Asynchronous requests for "large" datasets • Enables queueing of requests • Based on Web Processing Server spec (use of STORE and STATUS request parameters) • Client gets “ticket” for data request that is used to poll the server for progress • (N.B. WCS 1.1.0 now out and addresses some of these issues) • Experimented with Geoserver, ended up with custom implementation • Mainly due to 4-D limitations in Geoserver at the time

  5. DEWS Web Map Service (“ncWMS”) • DEWS WMS is the basis of a 4-D online visualization system • WMS spec supports TIME and ELEVATION dimensions • Expose CF-NetCDF data via a WMS interface • Minimal configuration required (CF gives most of the metadata we need) • (actually reads other file formats too) • Standard Java web app (WAR) • (first version was CDAT/Python) • Based on Java NetCDF libs, so can read OPeNDAP too • Emphasis on fast generation of images for interactive viz. • Aim for < 0.5s per image • Cache previously-generated data arrays so new images can be generated quickly (web interface is tile-based) • Data reading done in contiguous chunks as far as possible • Generates animations (animated GIF) • Generates KML as “image format” for Google Earth

  6. ncWMS architecture Data sources NetCDF NetCDF Non-standard file format OPeNDAP ncWMS Web interface (Godiva2) GIS client Virtual Globe

  7. Modifications to OGC WMS 1.3.0 • Serve Capabilities (metadata) piecemeal • For efficiency: Capabilities docs can get very large • Modify STYLES to allow greater flexibility without complexity of SLD • Modify colour scale extremes, choose palette, set opacity • cf MIME types: e.g. “boxfill;palette:rainbow;scale:-5:35” • WMS Capabilities doc doesn't have a place to encode units or valid_range • We added these as a piece of linked metadata • Non-map outputs • Value at a point (using GetFeatureInfo) • Timeseries plot at a point (ditto) • To come: transects (x-z), Hofmuller (x-t), etc (not in WMS domain, but useful nonetheless)

  8. Issues of translating CF <-> OGC • Mapping of Dataset/Variable to OGC nested Layers and Coverages • In ncWMS, Parent Layer (=Dataset) contains child Layers (=Variables) • Unique id (<Name>): DATASET_NAME/internal_name • E.g. “FOAM_NINTH/tmp” • Human readable title (<Title>): CF standard_name • E.g. sea_water_potential_temperature • For visualization, would be very useful to store the extrema of each variable’s values as attributes • Perhaps one min-max pair for each depth level? • Requires more thought! (discussion on CF mailing list ongoing) • Have to be careful with BBOX in non-lat-lon space (e.g. subsetting a Polar stereographic dataset) • Relates closely to EPSG CRS definitions • OGC Styling of WMS is really designed for feature data (e.g. obs), not raster (e.g. model output) • ColorMap exists, but not very flexible • Mainly CF <-> OGC mapping is fairly straightforward • But perhaps community would benefit from a “best practice” study?

  9. Vector quantities • CF does not explicitly identify the components of a vector • Implicitly represented in standard_name, e.g. [northward,eastward]_sea_water_velocity • Very useful to be able to serve the velocity components simultaneously • E.g. client requests “sea_water_velocity” • Server prepares all components as data (WCS), or a visual representation of the data as arrows (WMS) • Current DEWS WMS parses the standard names, looking for: • [northward,eastward]_* • [x,y]_* • *_[x,y]_* • This matches most cases but misses: • [wind_speed, wind_from_direction] • Plotting vectors is hard if source data is not on a lat-lon grid! • Looking to Gridspec for help here! • (Similarly, one could take same approach for other "derived" quantities such as density)

  10. Reprojection of coordinate reference systems • Probably the hardest challenge! • Needs to be done efficiently, on the fly in ncWMS • Numerical grids specified in CF by axis specifications: • So you can find lat,lon for given i,j • Reprojection (e.g. generation of map image in lat-lon coordinates) requires the inverse: • You want to find i,j for given lat,lon • This inverse does not always exist! • E.g. Murray tripolar (NEMO model), ROMS curvilinear ... • DEWS WMS uses a look-up table for this translation • LUT usually oversampled • (would our LUT generation software be useful to the community?) • We can get away with this for visualization, but can we be more precise? • Unlikely to be good enough for calculations on the data • Gridspec will surely be very useful!

  11. Result of using LUT • NEMO tripolar grid • Reveals curvilinear grid structure but “cells” appear blocky at high zooms • Regridding done on the fly

  12. Demos DEWS portal: http://lovejoy.nerc-essc.ac.uk:8080/dews-portal Godiva2 dynamic WMS client: http://lovejoy.nerc-essc.ac.uk:8080/ncWMS/godiva2.html

  13. Selection of depth Select from all the depth levels of the model

  14. Selection of time (range) Select from all the timesteps in the model Selection of a time range leads to an animation

  15. Finding the data value at a point Click on the data layer, data value and precise position is shown Lon: -64.08 Lat: 36.21 Value: 19.27

  16. Timeseries plots If a time range is selected, can create a timeseries plot at a point

  17. WMS: Standards give interoperability Cadcorp SIS Google Earth Web GIS NASA World Wind

  18. Virtual Globes • Much interest in using VGs (e.g. Google Earth) in environmental sciences • Easy-to-use way of sharing data and visualizations • Held workshop with NIEeS in Apr 07 for scientific community • 80 participants, inc international • Identified many uses for VGs in science and requirements for future development • We are building a community of scientists and developers to take this forward • www.scispace.net/geobrowsers • Important to note limitations of VG platforms – not a substitute for more sophisticated systems!

  19. Scientific use case for Virtual Globes: Diagnosis of models and observations • Picture left shows comparison of NEMO model and observations for Nov 2004 • Red dots show bad model-obs fits, green dots are good fits • Google Earth allows very efficient browsing of these large datasets • Could read obs and model data from different sources (OPeNDAP) and bring together in Google Earth or another client

  20. Suggestions for CF • Profiles (i.e. specific subsets of CF) • E.g. "rectangular lat-lon projections only", no 2-D coordinate variables • data_min, data_max attributes • Redundant but very useful for tools to choose a colour scale • Maybe one min-max pair per depth level • If not provided in file, how about "recommended" min-max for each standard_name? • Linking of vector components • So that tools can find the components of a vector quantity • Or is this adequately encoded in standard_names? • Redundant metadata can be very useful to give performance and usability gains! • (c.f. normal forms for relational databases)

  21. Conclusions • DEWS created software for secure access to CF-NetCDF data through OGC services • Outputs of DEWS that could be of interest to GO-ESSP members: • WMS implementation (+ web interface) • WCS implementation (based on GADS software) • look-up-table generator for non-invertible grids • Intellectual outputs • Styling of WMS layers for scientific CF-NetCDF data is a big conceptual issue • Fast, on-the-fly reprojection is a big practical issue

  22. Future work • Enhancements driven by current projects (MERSEA, ECOOP, ...) • Simple statistics, anomalies (how encode as WMS request?) • THREDDS integration of WMS (Pauline Mak) • Resurrection of original Python/CDAT WMS, possible inclusion in future CDAT (Charles Doutriaux, Dean Williams) • Work with Google to incorporate some scientific data layers in standard Google Earth distribution (in progress) • "Son of DEWS" – in discussion

More Related