OOI CI Overview and Status

1. OOI CI Overview and Status • Matthew Arrott • DMAC-STWashington DC, Jan 18-19, 2012

2. Agenda • Review of the OOI CI Construction Objectives • Introduction of Release 2 Functional Objectives • Computational Infrastructure Objectives • External Observatory Integration • What will OOI-CI mean to me

3. OOI CI Construction Objectives

4. OOI Project Scope • Multi Dimensional Engineering Concern • Social • Organizational • Functional • Procedural • Technical Building/deploying science infrastructure for understanding a changing water planet Cyberinfrastructure fulfills the integrative mechanism across these dimensions

5. The Hubble Telescope for the Oceans

6. The Hubble Telescope for the Oceans

7. Core Science & Education Capabilities • Interactive Ocean Observing (R3 - R4) • Interactive Ocean Modeling & Data Assimilation (R3 - R4) • Discipline-Driven Semantic Organization of Data (R3 - R4) • Automated Data Product Generation (R1 - R3) • Interactive Instrument Network (R1 - R3) • Integrated Observatory Management (R1 - R2) • User-Driven Integration of Resource (R1 - R3) Observatory Activity Model

8. OOI Integrated Observatory Product • The fully operational research observatory will meet the following • Goals: • Continuous observations at time scales of seconds to decades • Spatial measurements from millimeters to kilometers • Sustained operation during storms and other severe conditions • Real-time or near-real-time data as appropriate • Two-way transmission of data and remote instrument control • Power delivery to sensors between the sea surface and the seafloor • Standard plug-n-play sensor interface protocol • Autonomous underwater vehicle dock for data download/battery recharge • Access to deployment and maintenance vehicles that satisfy the needs of specific observatories • Facilities for instrument maintenance and calibration • A management system that makes data publicly available • An effective education and outreach program

9. Extending the Traditional Data Access Model

10. Multi-Scale Research Specific Ocean Laboratories

11. Release Schedule Inception Phase Elaboration Phase Construction Phase Transition Phase

12. Release Schedule by Subsystem Inception Phase Elaboration Phase Construction Phase Transition Phase

13. CI Components Developed for R1 • Python Capability Container • Java Capability Container Access Library • Web UI Platform • Exchange Messaging System • Distributed State Infrastructure • Resource Registry Framework • Science Data Persistence and Transport Format • Data Publish-Subscribe Framework • Event Notification Framework • Instrument Agent Framework • Elastic Processing Unit • Virtualized Cloud Management Tools

14. Capabilities of the Managed Instrument Network • Operate Marine Observatories • Operate Platforms and Instruments • Manage Instrument Lifecycle • Test and Troubleshoot Instruments • Acquire Data and Generate Data Products • Search Data • Visualize Data • Manage the Integrated Observatory Network

15. R1 Integrated Observatory Network Web UI https://confluence.oceanobservatories.org/display/CIPUB/OOI+Review+2011+Demos

16. Release 2 Functional Objectives

17. Observatory Operations & Administration

18. Scientific Collaboration General Science Users ScienceExperts Resource Providers User Experience Incremental Development Plan • CI Release-1: Data Distribution Network • Provide framework for discovery and interaction based on data-intensive users’ needs • CI Release-2: Managed Instrument Network • Provide support for operations and maintenance of marine networks based on RSN and CGSN needs • CI Release-3: On Demand Measurement Processing • Provide dynamic analysis and visualization tools based on science communities’ user needs • CI Release-4: Interactive Ocean Observatory • Provide mission control and collaboration support based on science communities’ needs

19. UC.R2.55 Manage Help TicketUC.R2.56 Monitor ION NetworkUC.R2.57 Configure Start Page UC.R2.01 Acquire External Data SourceUC.R2.02 Derive Data ProductUC.R2.03 Produce Real-Time QC DataUC.R2.04 Browse to Get Data ProductUC.R2.05 Register and Connect InstrumentUC.R2.06 Command InstrumentsUC.R2.07 Direct Instrument Access IIUC.R2.08 Manage Instrument LifecycleUC.R2.09 Activate Instrument DriverUC.R2.10 Manage Marine PlatformUC.R2.11 Operate Marine ObservatoryUC.R2.12 Deploy Agents On Remote PlatformUC.R2.13 Acquire Data From InstrumentUC.R2.14 Monitor an InstrumentUC.R2.15 Qualify Instrument InterfaceUC.R2.16 Install Instrument Automatically UC.R2.17 Define Visualization MethodUC.R2.18 Visualize Data ProductUC.R2.19 Produce Matlab Visualization UC.R2.20 Annotate ResourcesUC.R2.21 Transform Data in WorkflowUC.R2.22 Version ResourceUC.R2.23 Ingest Dataset SupplementUC.R2.24 Search for ResourceUC.R2.25 Advanced Resource SearchUC.R2.26 Navigate Resources and MetadataUC.R2.27 Manage Replicated ArchiveUC.R2.28 Manage Resource MetadataUC.R2.29 Integrate External Data Source UC.R2.30 Define InteractionUC.R2.31 Define New ServiceUC.R2.32 Conduct NegotiationUC.R2.33 Enroll in an OrgUC.R2.34 Share an Org ResourceUC.R2.35 Share Affiliated Orgs' ResourcesUC.R2.36 Create an OrgUC.R2.37 Control Service InteractionsUC.R2.38 Define Resource Life CycleUC.R2.39 Manage ION UsersUC.R2.40 Monitor ION ResourcesUC.R2.41 Recover Failed ProcessUC.R2.42 Define Resource PolicyUC.R2.43 Operate Message Brokers UC.R2.44 Put Services Anywhere EasilyUC.R2.45 Replicate Activated ServiceUC.R2.46 Operate Integrated SystemUC.R2.47 Deploy Versioned User ProcessUC.R2.48 Schedule User-Defined ProcessUC.R2.49 Deploy Distributed ProcessesUC.R2.50 Define Scaling PolicyUC.R2.51 Define Execution EngineUC.R2.52 Manage ION Processes UC.R2.53 View Modeler-Submitted ProductsUC.R2.54 Access NEPTUNE CA Data UC.R2.58 Display Arbitrary ResourceUC.R2.59 Generate New Screen R2 Product Description: List of Use Cases EOI Ops COI DM CEI S&A A&S UX

20. Managed Instrument Network

21. Sensor Set #1 Instrument Agents

22. Sensor Set #2 Instrument Agents

23. Sensor Set #3 Instrument Agents

24. CI Architecture for Instrument Integration

25. Data Processing and Product Generation

26. COL RSN CGSN CGSN CGSN Sensor Working Group COL RSN RSN CI COL RSN CGSN RSN CI CI CI CGSN CI CI CI OOI's Sensor Life Cycle OOI Final System Design Core Sensor List Core Sensor Specs Core Sensor RFPs Common Sensor Selections Procure Common Sensors Deploy at Sea Develop QC/Xform Algorithms Encode Algorithms System Integration Test CI Integration Test Test w/Instrument Test Kit Integrate w/ Sensor Agent Develop Sensor Drivers

27. Instrument Development Kit • Scheduled for Release-2 • Includes: • Logical Test Facility Workbench (dry testing) • Marine Specific System Test Facility (wet testing) • Configurations for RSN and CGSN observatories • Access and Management Portals for Interactive access

28. Computational Infrastructure Objectives

29. National Network Deployment (Year 3)

30. CyberPoPs and Network Infrastructure • Acquisition Point CyberPoPs: • Portland, OR • Woods Hole, MA • Distribution Point CyberPoPs: • Seattle, WA • Chicago, IL • San Diego, CA • McLean, VA (optional) • Engineering Center: • San Diego, CA • Network Infrastructure • Dedicated 10GE loop San Diego, Portland, Seattle, Chicago; branch to McLean (optional) • Dedicated 1GE connection to Woods Hole

31. San Diego Engineering Center • Purchased and deployed San Diego CyberPoP and network equipment (see rack drawing) • Located in UCSD’s Atkinson Hall server room 1101 (secure and protected) • In use for • Release-1 production and QA • Management tools (Confluence, Jira, etc)

32. San Diego CyberPoP and Network Equipment

33. Acquisition Point CyberPoP Deployments (Y3) • Sites • Portland • Woods Hole • Function • Data acquisition from marine observatories • Real-time data processing and OOI data product generation • Data preservation • High availability

34. CyberPoP Physical Layout (Year 3 Deployments) • Sites • Seattle • Chicago • Function • Content distribution (web servers, data servers, messaging) • Interconnects with major national and international network providers (layer 2 peering) • Links to compute clouds

35. Network Peering Integration Seattle Chicago

36. Multi-Tier Messaging Federation

37. Multi-Site Service Network Deployment

38. External Observatory Integration

39. IOOS Integration Model

40. External Observatory Integration Progress • Identification of datasets and data sources needed to support early adopter group (ESPRESSO/Rutgers, HiOOS/UHawaii) numerical model workflows • Classification of datasets by type and representation • Development of Dataset Agents for automatic data and metadata ingestion • Translation of data/metadata into the Common Science Data Format • Streaming of data packets and metadata update notifications • Integration of real-time data sources in early-adopter on-site numerical workflow generation processes, using the Integrated Observatory Network Release-1

41. External Observatory Integration • Exemplar: MARACOOS (Rutgers University) • Experimentally substitute part of the scientists’ data assimilation workflow preceding numerical model execution

42. External Observatory Integration: Master Dataset List • Datasets • NAVY : NRLSSC (DAP) : 1 • NOAA : NDBC SOS (HTML) : 34 • NOAA : NDBC SDF (DAP) : 1 • NOAA : PFEG (DAP) : 1 • Rutgers : tashtego (DAP) : 2 • UCSD : HFRNET (DAP) : 1 • UH : SOEST (DAP) : 1 • USGS : WaterService (HTML) : 26

43. “What will OOI-CI mean to me”

44. Regional Data Provider & Manager • Effective mechanism to reliably at scale distribute your data • to your user community and data management/preservation facility • Independently publish and receive data • in a wide variety of syntactic formats and community vocabularies • Publish and receive as near-real-time data streams and as historical data sets • Effective mechanism to reliably at scale aggregate and disseminate data from your publishers and to your end user communities

45. Participating Agency • Participate as providers, managers and consumers • Anticipated some requirements for publishing on the OOI as a separate Agency will require coordination with NSF • Requirements are being addressed as a part of the IOOS OOI Integration effort

46. How does one “join” – approval / certification? • Open data policy network for consumers of data • Anticipate publishers required to register and certified • Data policies and rights management being established • Open data policies and content limitations to science and education data content • Publishing requires acceptance of some form of open use policy • Consuming requires some form of “as is”; “at your on risk”; “with attribution of source if republished” policies

47. Relationship with IOOS, NEPTUNE and others observatories • External Observatory Integration subsystem within the OOI • Specifically focused on providing bi-directional integration interfaces to IOOS, NEPTUNE-Canada and WMO (aka GTS) • Integration continues on into the Operations and Maintenance phase of the OOI as a user/observatory “on boarding” capability • Provide the training and support to communities that wish to integrate with OOI

48. OOI define standards for them to “plug in”?, e.g SensorML • OOI provides a wide variety of syntactic formats and community vocabularies by which to interface with the OOI • OOI and the user community will be able to extend these interfaces • Capability facilitated by architectural choice of using controlled set of canonical data models into and out of which all data are transformed • New transformations are added, “plug ins”, to the network for a new format and/or vocabulary to one of the controlled data models

49. Will a region be able to host an instance of the OOI “stack”? • The OOI “Integrated Observatory Network” is a federation architecture • Comprising participants from multiple domains of authority operating from their local domain of authority • The OOI “stack” exists at a couple of levels: • It is a messaging protocol • It is a component software implementation that implements the messaging protocol and process management containers (think modern web server) that publish and consume services • “Capability Containers”, can be and are written in multiple languages • achieve their interoperability through use of the common OOI messaging protocol • OOI message protocol and any of the OOI supplied Capability Containers will be provided under one of the standard open source licenses

50. Thanks!