1 / 23

NOAA OneStop and the Cloud

NOAA OneStop and the Cloud. Kenneth Casey, PhD Deputy Director, Data Stewardship Division NOAA/NESDIS National Centers for Environmental Information (NCEI) Tuesday, 08 October 2019 WGISS-48 Hanoi, Vietnam. Cloud is a Key NESDIS Strategic Objective. The NESDIS Cloud Project!.

nbanks
Download Presentation

NOAA OneStop and the Cloud

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NOAA OneStop and the Cloud Kenneth Casey, PhD Deputy Director, Data Stewardship Division NOAA/NESDIS National Centers for Environmental Information (NCEI) Tuesday, 08 October 2019 WGISS-48 Hanoi, Vietnam

  2. Cloud is a Key NESDIS Strategic Objective The NESDIS Cloud Project!

  3. Business Readiness: People, Process, and Technology

  4. NESDIS Cloud Roadmap Initiation & Assessment Cloud Migration Leverage Cloud Cloud Optimization Proof of Concept Staging & Implementation Experimentation Migration Transformation • Develop NESDIS Cloud Strategy • Organize IPT(s) • Kick-off Business Readiness Assessment • Develop ConOps • Design and build Pilot Projects • Update the organization • Organize and consolidate • Migrate data to the cloud • Migrate services to the cloud • Develop phasing management plan • Prepare cloud environment • Orchestrate cloud implementation • Automate cloud services • Monitor and manage cloud environment • Optimize services, architecture, and cost components • Evolve architecture based on new cloud service offerings WE ARE HERE TODAY

  5. No More Stovepipes! Link to NESDIS Cloud Strategy: https://docs.google.com/document/d/1yv8E1jsXH4hcLLL5MO2pp1IfwS4AA5oXW_tD9XQd1ug/edit

  6. NESDIS Enterprise Cloud Vision Software and Release Management Continuous Integration Continuous Deployment Consolidated Ingest Product Generation Science and Development Sandbox Developers Data Metadata Query Data Metadata Metadata Data Data Distribution and Access Users Storage Metadata Catalog Data Metadata Query Data Configuration and Program Office

  7. Functional Focus Areas

  8. Functional Focus Areas

  9. Progress to Date Multiple Products Generated Multiple Data Streams being Ingested Monitoring Dashboards All Available for Discovery and Access

  10. Ongoing Work Cost Modeling Training Communications

  11. Focus in on OneStop

  12. OneStop in a Nutshell To improve the discovery of, access to, and usability of NOAA’s vast and diverse collection of big data

  13. What Have We Achieved? • > 378 collections made OneStop-ready in 15 groups • Tiers of OneStop readiness (Bronze-Silver-Gold) established • OneStop metadata and data format best practices documented • Data Stewardship Maturity Model assessments conducted at scale, incorporated into ISO metadata, and used as basis for streamlined Data Stewardship Maturity Questionnaire (DSMQ) • ISO-lite granule metadata spec with millions generated

  14. What Have We Achieved?

  15. What Have We Achieved? https://www.ncei.noaa.gov/onestop NOAA OneStop

  16. What Have We Achieved?

  17. What Have We Achieved? • Initial Common Metadata Editor Tool (CoMET) deployed and available for stakeholder testing • Incorporated DSMQ into CoMET • Collection Manager supporting tools (Docucomp, CEdit, Metaview, Metaserver, Pipelines, WAFs) operationalized • Training and user feedback collected at Sept 2019 NOAA Environmental Data Management Workshop

  18. What Have We Achieved? • User’s Guide and DSMQ glossary published to NOAA EDM Wiki • MediaCurrent Third-Party Usability Testing • Completed a second round of UI testing in late 2018 • OneStop 2.1.2 user interface incorporates much of the professional feedback and is live • CoMET testing completed as well, with feedback being incorporated into latest release (focus on “golden paths”!) Established initial framework for the Open Data Framework we have been calling the Mission Data Management System….

  19. NCEI’s Open Data Framework30,000 ft Data Producers and Operators Data Managers and Data Scientists Data Consumers and the Public Data Ingest Services Metadata Management Services (Inventory Manager) Search andDiscovery Services (OneStop) Archival Storage and Access Services

  20. NCEI’s Open Data Framework10,000 ft Data Producers and Operators Data Managers and Data Scientists Data Consumers and the Public Data Ingest Services Metadata Management Services Search andDiscovery Services S2N ATRAC Collection Manager OneStop UI and Thin Portals Common Ingest Inventory Manager OneStop API Archival Storage Services Tape Order Fulfillment + Access Services Cloud Disk (via Mission Science Network)

  21. NCEI’s Open Data Framework2,000 ft OneStop API/ Search Engine External Systems Collection Manager Index CEdit CoMET Metaserver OneStop UI Data Managers EMMA CoMET Collection M/D Rubrics DocuComp Public Thin Portals C G Common Ingest Data Streams Inventory Manager workflows Order Fulfillment + Access Services Storage Services M/D Event Log Cloud Disk Tape S2N + ATRAC Storage/Access Services Ingest Services MD Services Search Services Data Producers and Operators

  22. Summary and Next Steps OneStop’s Open Data Framework components at different stages of maturity, some fully operational other just getting started… • Integrate the components to improve operational robustness • Next UI release will include login.gov capability • Improve CoMET tools through user testing and iterative feedback • Continue Inventory Manager development • Iteratively improve the OneStop discovery UI • Working with the NESDIS Cloud IPT to stay in sync where possible and prepare for eventual cloud migration

  23. Questions?

More Related