1 / 34

The Unified Access Framework (UAF) for Global Earth Observation

Building NOAA’s data integration step by step with innovative approaches for better data interoperability across various fields.

aliciaparker
Download Presentation

The Unified Access Framework (UAF) for Global Earth Observation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Unified Access Framework (UAF) Kenneth.Casey@noaa.gov Julie.Bosch@noaa.gov Tina.Chang@noaa.gov Scott.Cross@noaa.gov Roy.Mendelssohn@noaa.gov Steven.C.Hankin@noaa.gov Jeremy.Malczyk@noaa.gov Jordan.Alpert@noaa.gov Jim.Sargent@noaa.gov Jeff.deLaBeaujardiere@noaa.gov Ted.Habermann@noaa.gov John.Relph@noaa.gov Bob.Simons@noaa.gov David.Neufeld@noaa.gov Upendra.Dadi@noaa.gov rsignell@usgs.gov Phil.Cogbill@noaa.gov Glenn.Rutledge@noaa.gov Mike.Grogan@noaa.gov Jeff.Budai@noaa.gov Steve Hankin (PMEL), Kevin O’Brien (PMEL/JISAO), Lewis McCulloch (HQ/TPIO) and the NOAA UAF team Building NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) one step at a time

  2. Enterprise-wide integration of datais a very hard problem! (i.e. adopting shared approachesto achieve data interoperability) NOAA/UAF-grid

  3. Why? Different fields have different concepts of ‘data’ … and have developed solutions that make sense to them. Getting people (and organizations) to change habits is difficult! NOAA/UAF-grid

  4. NOAA-world weather forecast (time critical) fisheries management (regulatory concerns) nautical charting climate, ocean, atmosphere research … the list goes on … NOAA management has been reluctant to invest funds in Agency-wide data integration NOAA/UAF-grid

  5. The accepted approach:build a “system of systems” “Wrap” existing systems with loosely coupled, standardized services  a Service Oriented Architecture NOAA/UAF-grid

  6. Global Earth ObservationIntegrated Data Environment • GEO-IDE Con-ops outlines such a plan • 2005-06, 66 pages(by “DMIT”) NOAA/UAF-grid

  7. Seed funding٨available last year finally Question:How do you approach a very big problem with a very small amount of money? NOAA/UAF-grid

  8. Tried and true approach … • Generate use cases • Define requirements • Write a Concept of Operations • … and an Implementation Plan • Assemble volunteer teams to implement NOT ! NOAA/UAF-grid

  9. An alternative (‘agile’) approach Don't Solve Problems-- Copy Success "Switch: How to Change Things When Change Is Hard", Chip and Dan Heath (psychologists), 2010 NOAA/UAF-grid

  10. Servicestack: netCDF-CF-DAP-THREDDS-WMS Applications: Matlab ArcGIS Ferret GrADS IDV Google Earth LAS ERDDAP … Dataformats: netCDF GRIB HDF … Projects: (too many to name) What “success” did UAF chose to copy? Year 1 focused on gridded datasets. Users: (too many to name)

  11. UAF(*)-grid(initial focus: gridded data) • netCDF • CF • OPeNDAP • THREDDS • WMS & WCS -- file format/API/data model -- metadata conventions -- network transport -- network topology -- OGC rendering and transport (*) Unified Access Framework NOAA/UAF-grid

  12. THREDDS “network topology”a tree defined as distributed XML NOAA/UAF-grid GEO-IDE/UAF NOAA NOAA Affiliated OAR NMFS NWS NESDIS IOOS National Partners IOOS Regional Partners ESRL OCO PFEG GFDL PMEL NDBC AOML NGDC NODC NAVO AOOS NOMADS GCOOS SCCOOS Coastwatch PACIOOS SECOORA NERACOOS GLOS NANOOS CENCOOS CARICOOS MACOORA

  13. Who is using this approach? • Modelers • IPCC, GFDL, NCAR, … • Satellite programs • GHRSST, PathFinder, CoastWatch, … • NCEP weather and ocean forecasts • GRIB files served via NOMADS • Coastal (“HF”) radar • A growing list of observations programs • Argo, OceanSites, … • Pending adoption by OGC NOAA/UAF-grid

  14. Some rough spots • compliance with CF conventions is inconsistent • individual files often not aggregated into logical datasets • metadata are often in need of enhancement NOAA/UAF-grid

  15. … but excellent repair tools exist:NcML and “virtual files” e.g. Improve CF compliance by adding “standard_name” attribute to GRIB data <variable name="vorticity">  <attribute name="standard_name" value="atmosphere_absolute_vorticity“ /></variable> The file, itself, is untouched. But access to the file through the services is made to conform to standards. NOAA/UAF-grid

  16. NcML for aggregation e.g. Aggregate three 1-year files of the same (say) model run <aggregation type="joinExisting“dimName="TimeAxis"> <netcdf location=“year1.nc" ncoords=“365"/> <netcdf location=“year2.nc" ncoords=“365"/> <netcdf location=“year3.nc" ncoords=“365"/> </aggregation> The files are untouched, but services see a single logical file. NOAA/UAF-grid

  17. The emerging UAF The UAF Team • 8-10 active participants • self-selected (volunteers) 1 year into the project … The UAF THREDDS catalog: • ~500 “clean” datasets • .. growing as the team aggregates and configures using ncML • team members become coaches  lasting solutions must be implemented at the data provider sites NOAA/UAF-grid

  18. How to reach users?Through their preferred tools (without downloading files) NOAA/UAF-grid

  19. Desktop access in Matlab Model 1: UMASS-ECOM Model 2: UMAINE-POM Data: SST 2008-Sep-08 07:32 NOAA/UAF-grid

  20. Access in ArcGISusing the Environmental Data Connector (EDC) NOAA/UAF-grid

  21. Desktop access in Ferret NOAA/UAF-grid

  22. Desktop access in GrADS NOAA/UAF-grid

  23. Desktop access in IDV NOAA/UAF-grid

  24. Google Earth Scripts Take differences Web browser examples … Live Access Server (LAS) NOAA/UAF-grid

  25. WMS can be used by many GIS applications NOAA/UAF-grid

  26. Browser access using ERDDAP …and from there …

  27. .kml REST URL access to data subsetsin several simple formats(accessible through home-grown scripting of many types) .mat .nc .mat NOAA/UAF-grid

  28. How can users find the datasets?(less mature) • Web crawlers • extract metadata from files (and elsewhere) • ingest into a data bases and from there: • formal metadata records generated (ISO 19115) • data discovery portals harnessed(e.g. RAMADDA, GeoNetwork, GI-Cat, …) • metadata made available for harvesting • GCMD, Geospatial One-Stop, GEOSS, data.gov NOAA/UAF-grid

  29. UAF home page Instructions for end users: how to access data through their favorite applications UAF is experimenting to see how documentation may be shared by other projects… NOAA/UAF-grid

  30. From the THREDDS catalog “viewers” may also be called up • netCDF Java Tools • LAS • ERDDAP • GODIVA NOAA/UAF-grid

  31. File-internal metadata accessible directly from THREDDS catalogs NOAA/UAF-grid

  32. The coming year’s target:Integrating “1-d grids” (observations) Collections of time series & profiles in netCDF-CF format are already on-line NOAA/UAF-grid

  33. UAF team met just 3 weeks ago (Seattle) • Preliminary plans: • Some initial target collections of in situ obs: • Argo, OceanSites, ICOADS • stream gauge data from USGS • atmospheric profilers (NCDC) • … • Common denominators for accessing the collections: • UAF year-1 techniques allow 1-at-a-time access • ERDDAP tables (ASCII and OPeNDAP Sequence) • a gateway for IOOS/SOS, RDBMS, DAPPER, … • Year 1 metadata techniques largely applicable to obs  Still largely volunteer-powered NOAA/UAF-grid

  34. Thank you NOAA/UAF-grid

More Related