1 / 43

Integrated data management in the ESMF (ESME)

Focus on data integration for modelers, partnership for better management, use of metadata standards, and enhancing data accessibility. Explore the importance and challenges of data integration in environmental science modeling.<br>

rhart
Download Presentation

Integrated data management in the ESMF (ESME)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Integrated data management in the ESMF (ESME) Steve Hankin(NOAA/PMEL & IOOS/DMAC) ESMF Team meeting July 2004

  2. The growing importance of data integration to modelers: We no longer fund “modeling”. Today we fund “climate prediction” or “coastal processes” … the science topics. Modeling is just a component. a program manager (anonymous)

  3. Model outputs need to be made useful to many classes of users education research community modeling community project model run

  4. Data and products need to be made more usable for modelers Real time and delayed-mode observations assimilation data products modeling boundary & initial conditions validation comparison

  5. An ESME must include a plan for data management. But, how ? Funds are limited …

  6. Partnership: A community of data managers has formed … GO-ESSP – Global Organization of Earth System Science Portals(http://esportal.gfdl.noaa.gov) • Unidata • ESG (NCAR, LLNL) • OPeNDAP (a.k.a. ‘DODS’) • COLA • NOMADS (GFDL, PMEL, NCDC, NCEP) • NASA/GCMD • BADC, BODC • WMO • …

  7. Ocean data systems following similar approaches … • National Virtual Ocean Data System (NVODS) • US Integrated Ocean Observing System • GODAE (US and International) • OCMIP, AOMIP, …

  8. Workflow Data portal components: • Data discovery • Data access/transport • On-line browse and comparison • (Segue to analysis)

  9. Data discovery == Metadata + search  Mature standards do not exist today. A task for ESMF: define and utilize metadata standards for modelers

  10. Metadata for modelers • Reviewed & “published” (a “standard”) • Structured (XML) • Generated automatically in conjunction with setting up model runs • Standardized parameter names (“controlled vocabularies”) • Hierarchical • components, grids, fields and attributes

  11. Data discovery == Metadata + search Others are working hard on search: • Traditional metadata partners(e.g. GCMD) • Semantic Web (Google on steroids)(3 years off?)

  12. Workflow Data portal components: • Data discovery • Data access/transport • On-line browse and comparison • (Segue to analysis)

  13. Data transport OPeNDAP (a.k.a. “DODS”) • Network data access • Format-independence • Subsetting • Aggregation (GDS, Unidata) • Compression • Security: Grid-enabled OPeNDAPg

  14. OPeNDAP: distributed access to data and semantic metadata

  15. CF (“climate and forecast”) CF 1.0 is now a “standard” • “use” metadata – e.g. units, coords. • curvilinear, hybrid-Z, time-dependent • great applicability beyond modeling, too Discussion question: As the use of the CF standard widens how should the community support it?Not enough to “endorse” it. Need a partnership.

  16. Workflow Data portal components: • Data discovery • Data access/transport • On-line browse and comparison • (Segue to analysis)

  17. Live Access Server

  18. netCDF

  19. LAS -- an “Information Product Server” Ferret, CDAT or other • Metadata (XML) contains the “intelligence” • Back end applications do the real work • OPeNDAP provides remote data access

  20. Uniform data access Informationaccess

  21. The UI talks to LAS through an XML “web service”

  22. For example …

  23. On-line comparison

  24. Example:Average over a lat-long box

  25. Workflow Data portal components: • Data discovery • Data access/transport • On-line browse and comparison • (Segue to analysis)

  26. discover  browse  access Metadata Standards NASA Global Change Master Directory (GCMD)

  27. Collaborating groups of modelers LAS “sisters” share metadata to form a unified (virtual) site.OPeNDAP allows LAS to difference distributed fields.

  28. A wealth of data products are available through the National Virtual Ocean Data System (NVODS)

  29. Example: Zebiac model output“live” from IRI/LDEO

  30. Access to observations (WODB -- 9 million ocean profiles)

  31. configurable constraints

  32. “Batch” access to products Query available data sets: Query variables in data set “model_1”: Query space-time domain: Request a subset of data as a file: (“asc” for ASCII format) >lasls http://cpu/LAS >lasls http://cpu/LAS  model_1 >lasls http://cpu/LAS  model_1  sst >lasget  -x 20:60  -y 20:60 -t 11-Dec-2000  -f asc        http://cpu/LAS  model_1 sst

  33. IOOSUS Integrated Ocean Observing System • Detect and Predict Change • Mitigate natural hazards • Improve safety and efficiency of marine ops • Ensure national security • Reduce public health risks • Protect and restore marine ecosystems • Sustain marine resources

  34. RegionalData ManagementSystems Products InternationalData ManagementSystems Users Maps Forecasts Terrestrial and AtmosphericData ManagementSystems On-line Browse Archive Centers Modeling Ships Hand Measurements Satellites Floats Primary DataAssembly & QC Moorings IOOS Data Management andCommunications Subsystem Metadata, Data Discoveryand Data TransportStandards and Protocols

  35. Ships Hand Measurements Satellites Floats Primary DataAssembly & QC Moorings IOOS Data Management andCommunications Subsystem RegionalData ManagementSystems Products Users InternationalData ManagementSystems Maps Forecasts Terrestrial and AtmosphericData ManagementSystems Metadata, Data Discoveryand Data TransportStandards and Protocols On-line Browse Archive Centers Modeling

  36. Recommendations for ESMF • Define ESMF metadata standard and use it • Consider a GCMD modelers’ portal, too • Endorse partnership with GO-ESSP and the emerging tools • Data available through OPeNDAP(g) • Live Access Servers for on-line collaborations

  37. Questions? NVODS LAS:http://www.ferret.noaa.gov/nvods GCMD “DODS” Portal:http://gcmd.gsfc.nasa.gov/Data/portals/dods

More Related