1 / 37

MBARI’s SSDS

MBARI’s SSDS. Data Management for Ocean Observatories. Brian Schlining ブライアン シュリニング. Overview. About MBARI Overview of MOOS Functional Requirements of SSDS Metadata use by SSDS Dataflow through SSDS. M onterey B ay A quarium R esearch I nstitute. Established in 1987.

katen
Download Presentation

MBARI’s SSDS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング

  2. Overview • About MBARI • Overview of MOOS • Functional Requirements of SSDS • Metadata use by SSDS • Dataflow through SSDS

  3. Monterey Bay Aquarium Research Institute Established in 1987

  4. Santa Cruz MBARI Monterey Monterey Canyon

  5. MOOS History Monterey Ocean Observing System • 1989: Began mooring-based observations. • 1999: Adopted the term “MOOS” (Monterey Ocean Observing System) • 1999: Began development of an integrated ocean observatory.

  6. MOOS – The Present Concept MBARI Mooring Autonomous Underwater Vehicle (AUV) Benthic Node

  7. MOOS – The Present Concept ‘Plug and Work’ Instruments Mooring Mooring AUV Benthic Node

  8. Future of Ocean Observing Variety of platforms

  9. Data Management Challenges: • Large number of data sources • Large variety of data sources • Dynamic system • Data sources may appear and disappear • No standard data format • Data can be instrument ‘native’

  10. SSDS Shore-side Data System • Data management system for MOOS

  11. SSDS – Functional Requirements • Capture and store data from MOOS data sources • Includes files and ‘streams’ • Capture information (i.e. metadata) about the stored data • Location, instrument, platform, data format, etc. • Capture and store data products • Derived products, quality controlled data, plots, etc.

  12. SSDS – Functional Requirements • Provide access to the original ‘raw’ data • Convert data to common formats for user application tools (Excel , Matlab, ARCView, …) • Present simple plots of any well-described data. • Provide access to data through an application program interface (API) and a web interface.

  13. Metadata –What we have learned • Metadata must accompany the data • A system’s power relies on good knowledge of its data • Without metadata, the data quickly becomes unusable • Metadata must accompany the instrument • Every connector between the two increases error rates • Once data and metadata detached, reattaching is painful • Metadata must be flexible and structured • Flexible: you’ll need to define new kinds of data sources • Structured: consistency => automation => value

  14. SSDS – Metadata (Object View)

  15. SSDS –Metadata (Object View) • ‘Deployment’ information. • SSDS tracks: • Where the data was collected. • When it was collected. • What other data was used. • Relation to other deployments

  16. SSDS –Metadata (Object View) • The data source. • SSDS tracks: • Software or hardware source • Unique identifier • Manufacturer information • References to documentation

  17. SSDS –Metadata (Object View) • References to the data. • SSDS tracks: • The data storage location. • The deployment that produced this data.

  18. SSDS –Metadata (Object View) • Format and contents of a DataContainer. • SSDS tracks: • The contents of a data-set. • The data format (To allow parsing by software).

  19. SSDS – Metadata Standards SSDS does not currently use metadata standards such as ESML, ESRI’s Marine Data Model, FGDC, Dublin Core or Marine XML. Why not? • Multiple overlapping standards. Which to choose? • None have achieved broad community support • Some are not yet complete (Marine XML) • Excessive documentation requirements (FGDC) • May use with SSDS later (Dublin Core)

  20. Example SIAM to SSDS Data Flow Mooring Portal SSDS

  21. Example SIAM to SSDS Data Flow Device Mooring Portal SSDS A device is connected to a platform, such as a Mooring.

  22. Example SIAM to SSDS Data Flow Device Mooring Portal SSDS <?xml version="1.0" encoding="UTF-8"?> <Metadata> <Deployment name="2003.30.01" role="platform“> <Device id="9873"/> <Deployment role="instrument" > <Device id="101"/> <output> <DataStream> <RecordDescription bufferStyle="binary" bufferLengthType="fixed" endian="little"> <RecordVariable name="time" columnIndex="1" format="double" longName="Time(GMT)" units="milliseconds since Jan 01, 1970"/> <RecordVariable name="temperature" columnIndex="3" format="double" longName="Seawater temperature" units="degrees C"/> </RecordDescription> </DataStream> </output> </Deployment> </Deployment> </Metadata> The mooring retrieves the metadata from the device.

  23. Example SIAM to SSDS Data Flow Device Mooring Portal SSDS Metadata Packet <?xml …> <Metadata> … </Metadata> The metadata is packaged and sent to a portal on shore before any data is sent to shore.

  24. Example SIAM to SSDS Data Flow Device Mooring Portal SSDS Metadata Packet The portal forwards the metadata to SSDS. <?xml …> <Metadata> … </Metadata>

  25. Example SIAM to SSDS Data Flow Device Mooring Portal SSDS <?xml …> <Metadata> … </Metadata> DB SSDS stores the metadata in a database. This allows applications to query for and use data.

  26. Example SIAM to SSDS Data Flow Device Mooring Portal SSDS DB

  27. Example SIAM to SSDS Data Flow Device Mooring Portal SSDS 34,56.234,0.0023,... DB The device produces a data record.

  28. Example SIAM to SSDS Data Flow Device Mooring Portal SSDS Data Packet DB 34,56.234,0.0023,... The data is packaged and sent to SSDS.

  29. Serialized Example SIAM to SSDS Data Flow Device Mooring Portal SSDS SSDS uses information in the packet to sort and store the data in a ‘raw’ format. DB VersionID, DeviceID, MetadataID, RecordType, PlatformID, SystemTime, SequenceNumber, DataBuffer(34,56.234,0.0023,…)

  30. Serialized Example SIAM to SSDS Data Flow Device Mooring Portal SSDS The ‘raw’ data is parsed and stored as netCDF for easier access. DB VersionID, DeviceID, MetadataID, RecordType, PlatformID, SystemTime, SequenceNumber, DataBuffer(34,56.234,0.0023,…) netCDF

  31. Serialized Example SIAM to SSDS Data Flow Device Mooring Portal SSDS MBARI Software Software applications allow users to discover and obtain data in formats useful to the typical MBARI user. (netCDF, text, etc.) DB netcdf parosci { dimensions: time = UNLIMITED ; // (17761 currently) variables: double time(time) ; time:long_name = "Time (GMT)" ; time:units = "seconds since 1970-01-01 00:00:00" ; double depth(time) ; depth:long_name = "depth" ; depth:units = "UNKNOWN" ; // global attributes: :title = "AUV data" ; :created = "2003-06-12T23:34:58Z" ; :history0 = ": Deployment information for parosci.log" ; :deploymentName = "2003.099.10" ; :instrumentId = "3699" ; } netCDF

  32. Serialized Example SIAM to SSDS Data Flow Device Mooring Portal SSDS MBARI Software Software applications also provide simple visual representations of data DB netCDF

  33. Serialized Example SIAM to SSDS Data Flow Web Pages Device Mooring Portal SSDS MBARI Software DB Provide internet access netCDF

  34. Serialized Example SIAM to SSDS Data Flow Existing netCDF Software Web Pages Device Mooring Portal SSDS MBARI Software DB Save development time by using existing software applications netCDF

  35. 2002 2003 2004 SSDS Schedule Requirements Development Relational DB Schema Metadata Metadata Tools Training Infrastructure & Capabilities Extreme Week Access & Visualization MTM Deployed MTM AUV Science Ops Start Projects AUV CTD MTM II MOOS ‘Lite’

  36. Development Team • John Graybeal • Kevin Gomes • Michael McCann • Brian Schlining (武頼庵　洲美守) • Rich Schramm • Daniel Wilkin

  37. Science and Technical Advisors • John Ryan • Francisco Chavez • Ken Johnson • Drew Gashler • Mark Chaffee • Tom O’Reilly • Mark Chaffee Funding provided by the David and Lucile Packard Foundation

More Related