Using python client libraries to access remote servers via Web Coverage Services

1. Using python client libraries to access remote serversvia Web Coverage Services Dominic Lowe, BADCBen Domenico, Unidata/UCAR* and the GALEON Team *Work partially supported by US NSF Atmospheric Science Division

2. Working Together on A Mosaic for Atmospheric Data This presentation describes and draws on the work* of many collaborating individuals and institutions * Unidata’s contribution supported by the U.S. National Science Foundation Ostia Antica circa 7 BC

3. Acronym Glossary • GALEON (Geo-interface for Air, Land, Environment, Oceans NetCDF) • FES (Fluid Earth Systems, mainly the data systems of the atmospheric and ocean sciences) • http://www.unidata.ucar.edu/content/publications/acronyms/glossary.html

4. Outline • GALEON Background • OWSLib – OGC Web Services Library • Simple Exercises for WCS Using OWSLib Clients • Participating Institutions and Servers • Future Directions • References

5. Reconciling Disparate Data Models:Different Ways of Thinking about Data • To the GIS (solid earth and societal impacts) community, the world is: • A collection of static features (e.g., roads, lakes, plots of land) with geographic footprints on the Earth (surface). • The features are discrete objects with attributes which can be stored and manipulated conveniently in a database. • To the fluids (atmosphere and oceans) communities, the world is: • A set of parameters (e.g., pressure, temperature, wind speed) which vary as continuous functions in 3-dimensional space and time. • The behavior of the parameters in space and time is governed by a set of equations. • Data are simply discrete points in the mathematical function space.

6. GIS Servers GIS Server GIS Server Hydrologic, demographic, infrastructure, societal impacts, … datasets Taking Advantage of Web Services for Data System Interoperability FES Client Applications GIS Client Applications OGC or THREDDS, OPeNDAP, ADDE. FTP… protocols OpenGIS Protocols: WMS, WFS, WCS, CSW OGC or proprietary GISprotocols FES Servers THREDDS Server THREDDS Server Satellite, radar, forecast model output, … datasets

7. Traditional GIS view Features as points, lines, polygons Attributes in DBMS tables

8. Typical NetCDF Visualization

9. OGC GALEON • Phase 1: • Many successful experiments with WCS 1.0 • Gridded data • Phase 2: • WCS 1.1 (and later) plus CS-W for catalogs • Non-gridded data collections via other Services (WFS, SOS) • Dearth of client implementations slows progress • OGC Network Rather than Interoperability Experiment

10. Apply GIS Tools ToAtmospheric Science Data

11. OWSLib – OGC Web Service utility library • is a lightweight package for working with OGC map, feature, and coverage services using Python  • provides a common API for accessing service metadata and wrappers for GetCapabilities, GetMap, GetCoverage and GetFeature requests • facilitates implementation of simple WCS clients as python scripts

12. OWSLib –Design Goals • Instantiate an OGC web service proxy (ala xmlrpclib)  • Make a GetCapabilities request and marshal the response into a Python structure that models OWS Common metadata  • Proxy GetMap/GetFeature/GetCoverage/Get* requests to the service with Pythonic wrappers for those operations  • Note that OWS common is an initiative to separate out common elements of OGC web services -- likely important as GALEON attempts to serve non-gridded data

13. OWSLib History • Originally developed by Sean Gillies as part of the Python Cartographic Library. • Version 0.2 contained support for accessing WMS version 1.1.1 and WFS 1.0.0 servers.  • Version 0.3 developed by Sean Gillies & Dominic Lowe introduced support for WCS versions 1.0.0 and 1.1.0. • Version 0.3 also introduced a harmonized OWS Common-like API to service metadata that is common across WCS, WMS and WFS. This API is a 'pythonic' interpretation of OWS-Common which provides easy access to the OWS-common metadata. 

14. OWSLib Design Decisions • Lightweight • Easy to use • Modelled on OWS-common but not verbatim; “pythonic beats pedantic as long as significant meaning is not lost” • Use python idioms: e.g. convenient dictionary like access to coverage metadata: wcs['temperature'] • Harmonize metadata between services where not harmonized (and where possible) in line with intentions of OWS common. 

15. Rudimentary WCS Exercisesvia OWSLib Python Scripts • Use getCapabilities for list of coverages • Extract name of first coverage in list • If multiple times are available, use the initial time • Use getCoverage to request first coverage • Use defaults wherever possible • Focus on whether protocol works • Worry about “correctness” of the returned coverages later

16. WCS Sites • US Pacific Fisheries Environmental Lab • US Unidata THREDDS Data Server • US USGS Coast Environmental • US NCDC NOMADS • Netherlands KNMI Geoservices • Netherlands ARGOSS • UK BADC • US NSIDC • US George Mason U. • Italy U. of Florence CNR • US NNEW Weather.aero

17. General Results • 3 servers returned a coverage on the first try • 2 required minor changes to the default parameter list • The rest required interactions between Dominic, Ben, and the WCS site administrator • 2 cases of missing required parameters • 2 cases uncovered bugs in python client library which were subsequently fixed • All cases working in the end

18. Future Enhancements • Include more servers in the exercise • Augment exercise in the direction of a true test • Retrieve representative coverages • Retrieve representative times • Use bounding box other than default • Examine returned coverage • Track evolution of WCS specification

19. More Inforamtion • OWSLib wiki: http://trac.gispython.org/lab/wiki/OwsLib • Subversion Repository: svn co http://svn.gispython.org/svn/gispy/OWSLib/trunk • Python Cheeseshop: http://pypi.python.org/pypi/OWSLib/0.3 • GALEON Wiki page:https://sites.google.com/site/galeonteam/Home/GALEON%20WCS%20"Tests"