OOI Operations and Maintenance Functions related to Data Management

1. OOI Operations and MaintenanceFunctions related to Data Management 4/28/2014

2. OOI Data Management O&M • Layers of O&M • Topmost layers are exposed to the users • Lower layers are used/managed by administrators and operators • Each layer imposes O&M activities • Data Management aspects will be discussed today Application Layer Infrastructure Layer Management and Monitoring Tools Data Management OS Layer Hardware Network

3. Data Management in O&M • Key responsibilities: • Keep the system running- fix problems • Perform updates • Don’t lose data • Deliver data • Monitor Performance • Availability • Reliability

4. Problem Management in O&M • Same tool used by all OOI during O&M • Problem Reporting • Use existing JIRA tool • Defect Tracking & Resolution • Lifecycle state • Triaging • Correction • Integration • Deployment • Type specific attributes

5. Management and Monitoring Process • Toolset used to monitor and manage system health: • Network performance • System response time • Storage status • Security monitoring • Maintained by CI Operations Team • O&M Activities: • Technology Refresh of tools • Provide monitoring, analysis, and troubleshooting

6. Data Storage • Uncabled Instrument and Platform Data are first stored at Operations and Management Centers (OMCs) • Instrument and Platform Data are ingested into OOINet via Agent Drivers • Cabled- directly- in near real-time • Uncabled- transferred from OMCs • OOINet stores: • Data in the form received from instrument • Added QC flags • Human in the Loop QC results

7. Backup and Recovery • Science and Engineering Data Files • Hierarchical flat files on RAID • Planned schedule and procedures for backups • Recovery tools and procedures to replace lost or damaged files • OOINet metadata database (PostgreSQL) • Planned schedule and procedures for backups • Recovery tools and procedures to replace lost or damaged records

8. Data Transfer to Archive • Where: National Oceanographic Data Center (NODC) • What: OOI Science Data Products and associated metadata • How: OOI-NODC telecons upcoming to determine: • Frequency • Volume • Metadata format • Data format • Transmission method (ftp, email, etc.)

9. Algorithms, Calibration Tables, QC Lookup Tables • Available through oceanobservatories.org • Available through OOINet User Interface • Name and version numbers are maintained for algorithms (Data product, Calibration, QC) • Calibration Tables are associated with an instrument device (serial #)/instrument site/deployment • QC Lookup Tables are associated with a version of a QC Algorithm and specific site

10. Documents • Available through oceanobservatories.org • Examples • Cruise Data (e.g., documents, photographs) • OOI Documents (e.g., the Technical Data Package) • Calibration Tables

11. Asset Tracking • Provided in OOINet • Functionality provided: • Track location of physical assets (including marine and land assets) • Assign assets to a deployment • Maintain history of asset location • Maintain history of asset repair and calibration

12. Property Management • OL is the responsible organization • OL flows down Property Management to the Marine IO institutions using their institutional systems.

13. Back up Slides • These slides are not directly related to data management

14. Outline • Generic Operations and Maintenance (O&M) • CyberPop O&M • Support Layers • Management and Monitoring processes • Specific Management activities (Data, HW, OS, Network) • OOINET O&M • Ongoing tasking • User Interface • Technology Refresh

15. Generic O&M • Problem Management • Will be used by all aspects of OOI during O&M • Problem Reporting • Use existing JIRA tool • Defect Tracking • Lifecycle state • Type specific attributes • Defect Resolution • Triaging • Correction • Integration

16. Network Management • Network Layer is rarely “seen” by the user • Maintained by System Administrators • O&M Activities: • Technology Refresh • Monitoring data volumes

17. Hardware Management • Hardware Layer is rarely “seen” by the user • Maintained by System Administrators • O&M Activities: • Replacing failed components • Spares control • Plan for “on-site” sparing • Monitoring status of hardware • Obsolescence Management • Technology Refresh • 5 year cycle

18. Operating System (OS) Management • OS Layer is rarely “seen” by the user • Maintained by System Administrators • O&M Activities: • Technology Refresh • Monitoring status of Operating System

19. Management and Monitoring Process • Toolset used to monitor and manage system health: • Network performance • System response time • Storage status • Maintained by System Administrators • O&M Activities: • Technology Refresh of tools • Provide monitoring, analysis, and troubleshooting

20. Application Layer • Comprised of OOINET (including User Interface) • Directly used by users • Maintained by System Administrators

21. Application Layer O&M • OOINET Core Services • Rolling out new deployments • Defect correction • Technology Refresh • User Interface • Eliciting user feedback • Providing Data Access and Retrieval • Technology refresh • Technology Refresh • Modernizing Core and UI components • Migrating off of deprecated support software

22. Marine IO O&M • Typically used by operators and system administration personnel • Calibration and configuration data saved within SAF • Used by OOINET O&M team to roll out new device drivers and/or fixes to existing drivers • Cruise Data transfer? • Asset (and property) tracking through the OOINET Asset Tracking tool