The Unified Access Framework for Gridded Data

1. The Unified Access Frameworkfor Gridded Data … the 1styear focus of NOAA’s Global Earth Observation Integrated Data Environment(GEO-IDE) Steve Hankin, NOAA/PMEL and the NOAA UAF team (including Rich Signell, USGS!)

2. Enterprise-wide integration of data is a very hard problem! NOAA UAF-grid

3. Heterogeneous requirements • Data Providers with different outlooks(different "corporate cultures”) • Data Users with diverse needs and skills Corporate cultures and people’s habits are notoriously difficult to change … and by the way, getting funds for integrationis often a hard sell NOAA UAF-grid

4. Integrating NOAA’s datais such a challenge • weather forecast (time critical) • fisheries management (regulatory aspects) • nautical charting • climate, ocean, atmosphere monitoring • … • modeling & observations • research & operations NOAA UAF-grid

5. The accepted approach:build a “system of systems” • Connect existing data systems using aService Oriented Architecture (“wrap” existing systems with loosely coupled, standardized services) NOAA UAF-grid

6. Global Earth ObservationIntegrated Data Environment • GEO-IDE Con-ops outlines such a plan • 2005-06, 66 pages NOAA UAF-grid

7. finally Seed funding٨available this year • How do you approach a very big problem with a very small amount of money? NOAA UAF-grid

8. Define requirements • Create enterprise architecture documents • Assemble volunteer teams to implement NOT ! NOAA UAF-grid

9. An alternative approach “Don't Solve Problems-- Copy Success” From the book: “Switch: How to Change Things When Change Is Hard", by Chip and Dan Heath http://www.fastcompany.com/magazine/142/switch-how-to-change-things-when-change-is-hard.html NOAA UAF-grid

10. Servicestack: netCDF-CF-DAP-THREDDS-WMS -WCS Applications: Matlab ArcGIS Ferret GrADS IDV Google Earth LAS ERDDAP … Dataformats: netCDF GRIB HDF … Projects: (too many to name) Users: (too many to name) NOAA UAF-grid

11. Unified Access Framework (UAF)(initial focus: gridded data) • netCDF • CF • OPeNDAP • THREDDS • WMS • WCS -- an abstract data model -- metadata conventions -- network transport -- network topology -- rendering -- gridded data service NOAA UAF-grid

12. A Simple NetCDF File

13. Need NetCDF Conventions • One of the greatest things about NetCDF is that it places few demands on the data provider - they are free to specify whatever attributes they want, or none at all • This is also one of the worst things, making it hard to develop flexible software

14. Climate and Forecast (CF)Metadata Conventions Groups adopting CF: GO-ESSP: Global Organization for Earth System Science Portal IOOS-DMAC: Integrated Ocean Observing System ESMF: Earth System Modeling Framework OGC: Open Geospatial Consortium

15. OPeNDAPhttp://www.opendap.org • Open-source Project for a Network Data Access Protocol: delivery of multidimensional scientific data via http • Allows efficient slicing from large 4D data via the web, just as NetCDF works for local files. • http://stellwagen.er.usgs.gov/cgi-bin/nph-dods/ocean.nc?h[0:5:200][0:10:300] • There are many different OPeNDAP servers (TDS, Pydap, Hyrax, GDS), and many different OPeNDAP clients

16. Who is using this approach? • Climate, ocean, atmosphere modelers • IPCC, GFDL, NCAR, … • Satellite programs • GHRSST, PathFinder, CoastWatch, … • NCEP weather and ocean forecasts(GRIB via NOMADS) • Coastal (“HF”) radar • A growing list of observations programs • Argo, OceanSites, SAMOS&GOSUD (soon), … NOAA UAF-grid

17. Some rough spots • Need better compliance with CF conventions • “Files" need to be aggregated into logical datasets • Need richer discovery-level metadata NOAA UAF-grid

18. NcMLa tool for fixing compatibility problems Example 1: Making GRIB data CF compliant, adding a standard_name” attribute and the “coordinates” attribute <netcdf location=“sst_20100824.grib2” <variable name=“temp2">  <attribute name="standard_name" value=“sea_surface_temperature” /> <attribute name=“coordinates" value=“lat lon” /> </variable> The existing file is untouched. But access to the file through the services is made to conform to standards. NOAA UAF-grid

19. NcMLa tool for fixing compatibility problems Example 2: e.g. aggregating 3 years of data into 1 <aggregation type="joinExisting“dimName="TimeAxis"> <netcdf location=“year1.nc" ncoords=“365"/> <netcdf location=“year2.nc" ncoords=“365"/> <netcdf location=“year3.nc" ncoords=“365"/> </aggregation> The files are untouched. But the services see a single logical file. NOAA UAF-grid

20. NcMLa tool for fixing compatibility problems Example 3: Adding metadata <netcdf location=“sst_20100809.grib2”><attribute name="creator_email" value=“rsignell@usgs.gov"/> <attribute name="creator_name" value=“Richard P. Signell"/> <attribute name="institution" value=“USGS"/> </netcdf> The files are untouched. But the services see additional metadata. NOAA UAF-grid

21. How users will find UAF datasets • UAF web crawler extracts THREDDS metadata • Ingests into a database • other metadata also ingested as available • ISO 19115-2 records generated • Datasets and services registered • GCMD, Geospatial One-Stop, GEOSS, data.gov NOAA UAF-grid

22. The Emerging UAF The UAF Team • 8-10 active participants • self-selected 9 months into the project … The UAF THREDDS catalog: • ~1090 “clean” datasets • .. growing as the team aggregates and configures using ncML • team members become coaches  lasting solutions must be implemented at the data provider sites NOAA UAF-grid

23. Scripts Take differences The Emerging UAF Users access data through their client applications • Without downloading files Configuration of clients is underway Some examples … Live Access Server (LAS) NOAA UAF-grid

24. Desktop access in Matlab Model 1: UMASS-ECOM Model 2: UMAINE-POM Data: SST 2008-Sep-08 07:32 Matlab NetCDF-Java Toolkit http://njtbx.sourceforge.net NOAA UAF-grid

25. Ocean models used for forecasting BP Spill

26. My own Oil Spill simulation • Grab ocean current data from various models via OpenDAP • Run NOAA GNOME (Generalized NOAA Oil Modeling Environment) • ..\..\models\gnome\bp\SpillMovie_surf.mov

27. Desktop access in Ferret NOAA UAF-grid

28. Desktop access in GrADS NOAA UAF-grid

29. Desktop access in IDV NOAA UAF-grid

30. Access in ArcGISusing the Environmental Data Connector (EDC) NOAA UAF-grid

31. Access in Google Earth NOAA UAF-grid

32. WMS provides access to many GIS applications NOAA UAF-grid

33. Browser access using ERDDAP. And from there … NOAA UAF-grid

34. .kml REST URL access to data subsetsin several simple formats(accessible through home-grown scripting of many types) .mat .nc .mat NOAA UAF-grid

35. UAF home page http://geo-ide.noaa.gov/ Next year’s target: observational data Collections of time series & profiles arealready on-line NOAA UAF-grid

36. “No standard should be approved without having been used to implement a few projects of realistic complexity.” The UAF strategy buildson historical lessons The Rise and Fall of CORBA(Henning, 2006) NOAA UAF-grid

37. Thank you NOAA UAF-grid