1 / 27

Kevin Gomes, MBARI MBARI Data Architecture

Kevin Gomes, MBARI MBARI Data Architecture. OOI Cyberinfrastructure: Data Product Generation Workshop San Diego May 20-21, 2008. MBARI’s Data Architecture (Observatory-related). Sensing and Acquisition Instruments, Moorings, AUV, FOCE, ALOHA, Benthic Rover Infrastructure

emilie
Download Presentation

Kevin Gomes, MBARI MBARI Data Architecture

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Kevin Gomes, MBARIMBARI Data Architecture OOI Cyberinfrastructure: Data Product Generation Workshop San DiegoMay 20-21, 2008

  2. MBARI’s Data Architecture (Observatory-related) • Sensing and Acquisition • Instruments, Moorings, AUV, FOCE, ALOHA, Benthic Rover • Infrastructure • OASIS, SIAM, SSDS, Data Processing • Data • Formats, Transformations, Standards • Interfaces • Infrastructure, User, API, SOA • Other Information from Questionnaire

  3. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  4. 12.2, 10.1 OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver <Metadata> HTTP Inst 1001001 001110 JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  5. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service OASIS Notes: Drivers are part of controller software (I.e. a priori knowledge) Instrument lifecycles are controlled by operator in comms session Very little metadata in system, added later Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  6. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst <Metadata> <Metadata> JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp ID: X Parent: Y Data: [0110..] ID: X Parent: Y Data: [0110..] Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  7. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service SIAM Notes: Drivers travel with instrument in PUCK (Plug and work) Some aspects of instrument lifecycles can now be doneautomatically by node (better management) Higher level commands still done by human interventionover comms session. *Metadata attached to instrument. Sent when instrumentservice is started. Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  8. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK <Metadata> <Metadata> Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE ID: X Parent: Y Data: [0110..] ID: X Parent: Y Data: [0110..] SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  9. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Cabled Notes: Middleware doesn’t change, only execution location Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  10. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter <Metadata> <Metadata> SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst ID: X Parent: Y Data: [0110..] ID: X Parent: Y Data: [0110..] Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  11. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK [1010011101] Or 12.3, 432.0 Adapter Inst Adapter <Metadata> Strategy Rover Rover Portal Proxy Adapter Connector ID: X Parent: Y Data: [0110..] Inst AUV AUV Portal

  12. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Files: 010101010111 Files: 010101010111 Adapter Strategy Rover Rover Portal Proxy <Metadata> Adapter Connector Inst AUV AUV Portal

  13. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK AUV Notes: Data itself not located in SSDS, distributed Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  14. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  15. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  16. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  17. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  18. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS Why NetCDF? Well suited for large data sets Modelers use it = easier data integration Large community of support over many years Developed in support of earth sciences Leveraging atmospheric community (actively supportedby UNIDATA) Already works with many tools science users arefamiliar with (Matlab, Ocean Data View, Ferret, etc.) Community built server side tools around it (LAS, OPeNDAP) THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  19. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  20. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  21. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  22. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client • COARDS, Climate and Forecast (CF), Ocean Sites, THREDDS: • COARDS & CF community (atmospheric) driven = experience • Ocean Sites developed by oceanographic science community • Ease of use through standards (Joe Q. Public/Student) • Aggregation becomes easier • * Integration of Earth Sciences SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  23. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS Interfaces THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy User Cyberinfrastructure Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  24. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service “Give me your tired, your poor, Your huddled masses yearning to breathe free, The wretched refuse of your teeming shore. Send these, the homeless, tempest-tost to me, I lift my lamp beside the golden door!” Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB Huddled Masses = Data, User or Developer? PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  25. OASIS Telemetry OASIS 2 SSDS OASIS Web App Driver HTTP Inst JMS HTTP SSDS NetCDF Ocean Sites OPeNDAP SIAM Portal MOOS THREDDS Client SIAM Service Proxy API Strategy Raw Data Metadata NetApp Web Services PUCK Inst OGC Services SIAM ALOHA (MARS/Seahurst) Inst Service Proxy PUCK Strategy Adapter ESB Adapter SIAM Portal FOCE SIAM Adapter Service Proxy Strategy ESB PUCK Adapter Inst Adapter Strategy Rover Rover Portal Proxy Adapter Connector Inst AUV AUV Portal

  26. Other Questionnaire Information • Security and Policy • Data embargo is about all we could agree on • Data in SSDS is open • Security requirements were difficult to gather • Hooks there in SSDS (J2EE, container-managed, LDAP, etc.) • Event Detection • What is an Event • Several rounds, MASE is latest • QC • We leave mostly to experts due to domain knowledge • SSDS has metadata for basic QC (outliers, etc.) • Users have access to external QC through SSDS • Data gaps, long latencies • Handled by OASIS and SIAM, not by SSDS • Education and Outreach • Remember interfaces? We haven’t done enough

  27. Conclusion (= Soapbox?) • Interfaces! (User mostly) • User requirements • Do the “easy” things well to pull in users • Integrate quickly, then innovate • http://www.perceptivepixel.com Questions?

More Related