1 / 63

The Unified Access Framework (UAF)

The Unified Access Framework (UAF).

fnancy
Download Presentation

The Unified Access Framework (UAF)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Unified Access Framework (UAF) Kenneth.Casey@noaa.gov Julie.Bosch@noaa.gov Tina.Chang@noaa.gov Scott.Cross@noaa.gov Roy.Mendelssohn@noaa.gov Steven.C.Hankin@noaa.gov Jordan.Alpert@noaa.gov Jim.Sargent@noaa.gov Ted.Habermann@noaa.gov John.Relph@noaa.gov Bob.Simons@noaa.gov David.Neufeld@noaa.gov Upendra.Dadi@noaa.gov Rich Signell (rsignell@usgs.gov )Phil.Cogbill@noaa.gov Glenn.Rutledge@noaa.gov Mike.Grogan@noaa.gov Jeff.Budai@noaa.gov Steve Hankin (PMEL), Kevin O’Brien (PMEL/JISAO), and the NOAA UAF team Philosophy, progress, and plans DAARWG Meeting, Seattle, Nov. 2011

  2. Review… continuing into new material NOAA/UAF

  3. GEO-IDEa strategy for enterprise-wide integration of NOAA data (i.e. adopting standards and practicesto achieve data interoperability) 2005-06 66 pgs(by “DMIT”) Why is this a hard problem? NOAA/UAF

  4. NOAA-world weather forecast (time critical) fisheries management (regulatory concerns) nautical charting climate, ocean, atmosphere research response and restoration … the list goes on … NOAA/UAF

  5. Different disciplines have different concepts of ‘data’. Each develops solutions that make sense to them. Getting people (and organizations) to change habits is difficult! (and data management has often been an after thought.) NOAA/UAF

  6. The accepted approach:build a “system of systems” Wrap existing systems with loosely coupled, standardized services  a Service Oriented Architecture GEO-IDE Con-ops outlines such a plan. How to build it with a largely volunteer team? NOAA/UAF

  7. Tried and true approach … • Generate use cases • Define requirements • Write a Concept of Operations • … and an Implementation Plan • Assemble (volunteer) teams to implement NOT ! NOAA/UAF

  8. An alternative (‘agile’) approach Don't Solve Problems-- Copy Success -- NOAA/UAF

  9. Why ‘agile’ is attractive Because inevitably: • funding is much smaller than needed; • collaborations are more difficult than anticipated; • infrastructure is being built on a background of rapidly evolving technology NOAA/UAF

  10. Why ‘agile’ is attractive Change equals risk. Mitigate risk by following a strategy of incremental change that serves users (increasingly) well at every stage of evolution. infrastructure is being built on a background of rapidly evolving technology “Visualize a decade away. Build things that work today.” NOAA/UAF

  11. Agile Principles(condensed from the ‘Agile Manifesto’) • Working software is the meaningful measure of progress: ensure early, continuous, frequent releases • Emphasize simplicity - maximize the work not done • Build projects around motivated individuals. Give them the support they need and entrust them • The best architectures, requirements, and designs emerge from self-organizing teams • Welcome changing requirements • Scientists (‘business people’) and developers must work together on a daily basis NOAA/UAF

  12. Lemma “Don’t let perfect be the enemy of good” NOAA/UAF

  13. Servicestack: netCDF-CF-DAP-THREDDS (WMS) Applications: Matlab ArcGIS Ferret GrADS IDV Google Earth LAS ERDDAP … Dataformats: netCDF GRIB HDF … Projects: (too many to name) What “success” did UAF chose to copy? Year 1 focused on gridded datasets. Users: (too many to name) NOAA/UAF

  14. Who is providing data this way? • Modelers • AR4&5, GFDL, NCAR, … • Satellite programs • GHRSST, PathFinder, CoastWatch, … • NCEP weather and ocean forecasts • GRIB files served via NOMADS • Coastal (“HF”) radar • A growing list of observations programs • Argo, OceanSites, tide gauges, … • Adoption by OGC is well underway NOAA/UAF

  15. How to reach users? (without downloading files) Through their preferred tools NOAA/UAF

  16. Desktop access in Matlab Model 1: UMASS-ECOM Model 2: UMAINE-POM Data: SST 2008-Sep-08 07:32 NOAA/UAF

  17. Access in ArcGISusing the Environmental Data Connector (EDC) NOAA/UAF

  18. Desktop access in Ferret NOAA/UAF

  19. Desktop access in GrADS NOAA/UAF

  20. UAF home page Instructions for end users: how to access data through their favorite applications UAF is experimenting to see how documentation may be shared by other projects… NOAA/UAF

  21. Under the hood Projects with data to provide • make their data available as netCDF-CF(or other TDS-compatible format) • host a THREDDS / OPeNDAP server Lets look at the tools we have to link projects together … NOAA/UAF

  22. UAF “network topology”a tree defined in THREDDS (XML) NOAA/UAF GEO-IDE/UAF NOAA NOAA Affiliated OAR NMFS NWS NESDIS IOOS National Partners IOOS Regional Partners ESRL OCO PFEG GFDL PMEL NDBC AOML NGDC NODC NAVO AOOS NOMADS GCOOS SCCOOS Coastwatch PACIOOS SECOORA NERACOOS GLOS NANOOS CENCOOS CARICOOS MACOORA

  23. Not so fast, kiddo! • compliance with CF conventions is inconsistent • files commonly are not aggregated into logical datasets • metadata are often in need of enhancement Perspective – this is not an unusual situation: Standards compliance problems are *the norm* Divergent dialects often pile up (e.g. GRIB, BUFR) UAF tools offer a solution … NOAA/UAF

  24. ‘NcML’ can be used to repair problems (*) e.g. Improve CF compliance by adding “standard_name” attribute to GRIB data <variable name="vorticity">  <attribute name="standard_name" value="atmosphere_absolute_vorticity“ /></variable> The file, itself, is untouched. The ‘virtual file’ seen through the services conforms to standards. (*) IOServiceprovider modules also important … not discussed here NOAA/UAF

  25. NcML for aggregation e.g. Aggregate three 1-year files of the same (say) model run <aggregation type="joinExisting“dimName="TimeAxis"> <netcdf location=“year1.nc" ncoords=“365"/> <netcdf location=“year2.nc" ncoords=“365"/> <netcdf location=“year3.nc" ncoords=“365"/> </aggregation> A long time series ‘virtual file’ is seen through the services. NOAA/UAF

  26. UAF Team members are helping data managers, person-to-person, to improve the data services from their projects (a gradual, but important process) • in parallel … NOAA/UAF

  27. Developing the UAF Catalog Cleaner(a ‘web crawler’) ‘RAW’ UAF ‘RAW’ catalog UAF ‘CLEAN’ catalog NOAA NOAA NOAA Affiliated NOAA Affiliated IOOS Regional Partners IOOS Regional Partners OAR OAR NMFS NMFS NWS NWS NESDIS NESDIS IOOS National Partners IOOS National Partners ESRL ESRL OCO OCO PFEG PFEG GFDL GFDL PMEL PMEL NDBC NDBC AOML AOML NGDC NGDC NODC NODC NAVO NAVO AOOS AOOS NOMADS NOMADS ‘CLEAN’ GCOOS GCOOS SCCOOS SCCOOS PACIOOS PACIOOS Coastwatch Coastwatch SECOORA SECOORA NERACOOS NERACOOS GLOS GLOS NANOOS NANOOS CENCOOS CENCOOS CARICOOS CARICOOS MACOORA MACOORA NOAA/UAF

  28. The Catalog Cleaner • Crawl the raw catalog • Extracts metadata from the files, themselves, and from THREDDS into a relational database • Process the data base to detect aggregations, etc. • Create new THREDDS XML that is aggregated and metadata-cleaned NOAA/UAF

  29. Services: Usually OPeNDAP. Often WMS. Sometimes more. Optional documentation (beyond what is inside the file) Optional viewers ‘09 Carbon Tracker files from ESRL ‘raw catalog’ snippit NOAA/UAF

  30. ‘clean catalog’ • -- same data, but … • augmented with • Uniform services • Uniform viewers • improved metadata uniform viewers uniform services NOAA/UAF

  31. Uniform services Simple interface to get a subset metadata quality assessment ISO-standard metadata NOAA/UAF

  32. NOAA/UAF

  33. NOAA/UAF

  34. Godiva2 uses the WMS map service (as do other GIS apps) NOAA/UAF

  35. Segue to desktop tools Differencing Property-property plots Sections and Hofmullers Google Earth Vector plots Animations Line plots Analyses NOAA/UAF

  36. and ERDDAP provides … NOAA/UAF

  37. .kml REST URL access to data subsetsin several formats(accessible through home-grown scripting of many types) .mat .nc .mat NOAA/UAF

  38. Data Discovery Crawl the clean catalog. Create ISO metadata ISO-standard metadata NOAA/UAF

  39. Agile principle: “Maximize the work not done” Who has already built reasonably mature discovery portals (preferably free) ? • Unidata’s RAMADDA • GI-CAT (able to crawl THREDDS catalog) • Geoportal (able to harvest ISO metadata) NOAA/UAF

  40. Data Discovery using ESRI Geoportal NODC is leading the UAF Geoportal investigations

  41. Data Discovery using ESRI Geoportal Enter Search terms

  42. Data Discovery using ESRI Geoportal Search Results

  43. Data Discovery using ESRI Geoportal Refined Search New Results

  44. Data Discovery using ESRI Geoportal Expand Selection

  45. Data Discovery using ESRI Geoportal Available Services

  46. Data Discovery using ESRI Geoportal Available Services

  47. Data Discovery using ESRI Geoportal View metadata details about the dataset

  48. Data Discovery using ESRI Geoportal Available Services

  49. Data Discovery using ESRI Geoportal Direct link into the THREDDS Data Server

  50. Data Discovery using ESRI Geoportal Available Services

More Related