1 / 25

DMAC The Data Management and Communications Subsystem of the U.S. Integrated Ocean Observing System (IOOS)

DMAC The Data Management and Communications Subsystem of the U.S. Integrated Ocean Observing System (IOOS). Steve Hankin NOAA/Pacific Marine Environmental Laboratory DMAC Steering Team DMAC Modelers’ Caucus. GOOS - Global Ocean Observations GTSPP, ARGO, Satellites, VOS, others.

wood
Download Presentation

DMAC The Data Management and Communications Subsystem of the U.S. Integrated Ocean Observing System (IOOS)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DMACThe Data Management and Communications Subsystemof theU.S. Integrated Ocean Observing System (IOOS) Steve Hankin NOAA/Pacific Marine Environmental Laboratory DMAC Steering TeamDMACModelers’ Caucus

  2. GOOS - Global Ocean ObservationsGTSPP, ARGO, Satellites, VOS, others Why not do something comparablefor the coastal ocean? TECO-WIS

  3. Regional Associations US IOOS Coastal System: “National Backbone” TECO-WIS

  4. DMAC Subsystem Modeling & Analysis Subsystem Societal Goals Safe & Efficient Maritime Operations Metadata standards Data discovery Data transport Online browse Data archival Homeland Security Natural Hazards Climate Change Public Health Ecosystem Health Living Marine Resources Integrated Ocean Observing System Observing Subsystem Satellites Aircraft Fixed Platforms Ships Drifters & Floats AUVs TECO-WIS

  5. Many Variables – the IOOS Core Set • Physical • Salinity, temperature, air-sea heat flux • Sea and water level, water depth (Shore line) • Surface waves (height, direction, freq), currents & sea ice • Chemical • Contaminants & dissolved nutrients • Dissolved oxygen • Biological • Fish species & abundance • Chlorophyll, ocean color & phytoplankton abundance/species • Zooplankton abundance/species • Pathogens • Interdisciplinary • Optical properties (Atmospheric & in-water) • Bottom character and benthic habitats TECO-WIS

  6. The Governance Challenge • Many independent observing system activities • federal, state, county, municipal, academic & private players • Independent funding paths • Widely varying program missions • A planning office -- Ocean.US -- established to coordinate the efforts Effective governance depends upon a workable data management and communications (DMAC) plan TECO-WIS

  7. …competingcharacteristics ruled out the most straightforward solutions for IOOS TECO-WIS

  8. No simple data standard can be designed that all ocean data providers will utilize TECO-WIS

  9. No management structure exists to enforce adoption of a complex, data management solution TECO-WIS

  10. Data cannot be centrally managed at a single location TECO-WIS

  11. The DMAC Plan – a community effort Steering Comm. convened June 2002 • Brought together … • 6 Federal agencies • 6 Universities • 4 Regional/State agencies • 3 Private sector orgs. • Steering Committee guided2 Outreach Teams,4 Expert Teams • Data Facilities Outreach • User Outreach • Data Discovery & Metadata • Data Transport • Data Archive & Access • Applications & Products Public drafts circulated Dec. 2003 TECO-WIS

  12. The DMAC Plan – a community effort • Four levels of review • Solicited expert review • Public workshop • Public email-based review • Federal Registry comments • Drafts broadly circulated • ExCom, NORLC, IWG, USGST, Ocean.US Workshops, Regional Summits, Professional Society newsletters, email lists and websites • 150 reviewers • 6 Federal Agencies • 22 Universities & Institutes • 13 Private Sector • 13 Regional/State agencies • 5 International Published March 2005 • An implementation plan(a roadmap to detailed specification)

  13. The DMAC Plan is a framework for interoperability among independent, heterogeneous systems, large and small • Independence: Organizations will manage their own data • Standards: enabling innovation! • “Web services”: the common language to be spoken by pre-existing systems TECO-WIS

  14. How DMAC selects standards “Adapt, adoptand only as a last resortdevelop” Strong preference for open standards Classified by IOOS maturity levels • R&D • Pilot • Pre-operational • operational TECO-WIS

  15. DMAC planning process continues … Steering Team Expert Teams Transport and Access Metadata and Data Discovery Archive Caucuses Private sector International Modeling Regional Working Group Systems Engineering Interagency Oversight Working Group TECO-WIS

  16. Data Transport “Adapt, adoptand only as a last resortdevelop” • Expert Team -- 10 members • academia (4), federal (4), private sector (2) Harmonize developments from 2 communities • GIS – utilizing OGC W*F standards • FES* – utilizing OPeNDAP * FES = fluid earth systems TECO-WIS

  17. Digression on OPeNDAP …”DAP” == Data Access Protocol Since 1992 - a “web service” before the term existed • “Files” vanish  “datasets”, instead • On the server • netCDF, GRIB, HDF, IEEE, … or relational databases • aggregations (multi-terabyte datasets) • On the client • “open” and read subsets • any netCDF application is a web client (may need to relink) • Matlab, IDL, IDV, GrADS, Ferret, CDAT, … • Widely used in research modeling communities TECO-WIS

  18. Data Transport IOOS maturity levels • R&D • Pilot • Pre-operational • operational TECO-WIS

  19. Metadata andData Discovery “Adapt, adoptand only as a last resortdevelop” Team approach • Coordination with organizations/programs such as MMI*, ISO, FGDC*, GCMD*, ORION* and QARTOD* • Engage data providers in the process Determine marine metadata content requirements • data lifecycle: identify, assess, access, utilize and archive • adaptive to data type and method of data collection Compare metadata elements across multiple standards • FGDC, ISO, DIF, EML, ESML, SensorML, MarineXML, etc. • MMI = Marine Metadata Interoperability project • FGDC = US Federal Geographic Data Committee • GCMD = NASA Global Change Master Directory • ORION = US NSF-funded ocean research initiative • QARTOD = Quality Assurance Real Time Ocean Data TECO-WIS

  20. Metadata andData Discovery Starting point: “Guide for IOOS Data Providers, v1.0” Initial guidance contains: • Required vocabularies • IOOS keywords, core variables, national backbone programs • Recommended vocabularies • ISO 19115 topic categories, GCMD keywords, IHO codes, OBIS species, CF standard names • Metadata standards • ISO 19115, FGDC CSDGM, GCMD DIF, OBIS • Minimal list of elements • Discovery, access, data transport, archive, consumer use, data management • Data discovery systems • NSDI Clearinghouse, Geospatial OneStop, GCMD, OBIS TECO-WIS

  21. DMAC Modelers’ Caucus – Challenges • Advancement and use of CF (netCDF & OPeNDAP)Further work: • unstructured grids • Geospatial datums • Nested grids • Downscaling and splicing of neighboring domains • Rich model metadata • OSSEs – couple modeling and observing system design • Model-data validation and model-model comparison Modeling and Analysis Steering Team (MAST) >> Convened Oct. 25, 2006 << Community Modeling Networks (CMNs) to follow

  22. “Community Standards forUnstructured Grids”UCAR/Unidata, Boulder CO -- >> October 16-18 <<NOAA-IOOS funded • Standards for unstructured coastal model outputs • A community process • Standardize file contents, formats • Standardize network access protocols • A demonstration testbed will be created • Technical discussions will be open • on a moderated web site http://cf-pcmdi.llnl.gov/ TECO-WIS

  23. DMAC and WIS – Closely related challenges Both seek to • Link organizationally-independent data suppliers • Address a broad range of user needs • Integrate heterogeneous data types • Merge real-time (push) and delayed mode (pull) • Build upon standards WORLD METEOROLOGICALORGANIZATIONWeather  Climate  Water TECO-WIS

  24. DMAC and WIS – Closely coupled solutions DMAC areas in need of particular attention(to what degree can WIS solutions be adopted?) • Real time (push) ( IDD/LDM? ) • Access control • Service registries • Data assembly (regional and national) WORLD METEOROLOGICALORGANIZATIONWeather  Climate  Water TECO-WIS

  25. kahm-sa-hahm-ni-da(Thank you) TECO-WIS

More Related