1 / 24

Archive Architecture Way Forward

Archive Architecture Way Forward. DAARWG Meeting. Overview. Developing an Archive Architecture for the NOAA Enterprise Current Archive Activities CLASS Update. Developing an Archive Architecture for the NOAA Enterprise . Need for an Enterprise Approach.

sheryl
Download Presentation

Archive Architecture Way Forward

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Archive Architecture Way Forward DAARWG Meeting

  2. Overview • Developing an Archive Architecture for the NOAA Enterprise • Current Archive Activities • CLASS Update DAARWG Meeting

  3. Developing an Archive Architecture for the NOAA Enterprise

  4. Need for an Enterprise Approach Growing Volume & Complexity of Data Small volume of “other” data sets, but highly complex (i.e., formats, metadata, quality) Much more of this data is not in the Data Centers, but is distributed across NOAA Model data is fastest growing by volume, but more ephemeral DAARWG Meeting

  5. Need for an Enterprise Approach NOAA Admin Order 212-15, Section 3. Policy (4 Nov 10) • Management of NOAA environmental data will be based upon an end-to-end data management lifecycle that includes: • Determining what environmental data are required to be preserved for the long term and how preservation will be accomplished • Developing and maintaining metadata throughout the environmental data lifecycle that comply with standards • Obtaining user requirements and feedback • Developing and following data management plans that are coordinated with the appropriate NOAA archive for all observing and data management systems • Conducting scientific data stewardship to address data content, access, and user understanding • Providing for delivery to the archive and secure storage • Providing for data access and dissemination • Enabling integration and/or interoperability with other information and products What to Archive How to Archive DAARWG Meeting

  6. CLASS as Enterprise IT Solution Preliminary Level 1 Requirements Document (L1RD), 6 Nov 08 Scope. Enterprise-wide IT systemsupporting long-term, secure storage of and common access to environmental datasets and information stewarded by NOAA’s Archives. • Large Data “Campaigns”: Satellites (NPP/JPSS, GOES, POES, DMSP, MetOp, EOS), Radar (NEXRAD), and NCEPModels (including reanalysis) • Enterprise Approach • Providing common services for development and operation of IT systems supporting NOAA Archives • Consolidating legacy archival storage systems to reduce acquisition costs • Relieving data producers of responsibility for archival Yet there is no NOAA Data Management Framework or Archive Architecture to define the enterprise… DAARWG Meeting

  7. Need for Archive ConOps Common Theme from Independent Review Teams • Independent review by Dolly Perkins, NASA; NESDIS/OSD commissioned; Final Report, 11/25/09 • ”There does not appear to be any strategic plan for CLASS that lays out its vision as an enterprise system.” • Independent review by Tim Hall, Aerospace; CLASS Operations and Planning Board (COPB) commissioned; Final Report, 4/1/11 • “There is no government CONOPS for CLASS as an enterprise system” • “The lack of a CONOPS will continue to frustrate efforts to establish a valid, executable set of joint requirements, and CLASS is unlikely to meet its goals and requirements” • “The lack of a government CONOPS for CLASS is a root cause of the inadequacy of the architecture as well as of gaps in joint requirements” • “There is no enterprise organization including staff and supporting policy to operate an enterprise system across all data centers” DAARWG Meeting

  8. Clarify Guidance and Directives Need for Archive ConOps • “The DAARWG found…that because the scope of CLASS has evolved over time, there is widespread misunderstanding within NOAA of the CLASS role and purpose…a lack of understanding of the roles and responsibilities in creating and using a NOAA archive is handicapping development.” • NAO 212-15 updated • OCIO Data Architecture • EDMC/AAT accelerating development of • Archive Architecture • Data Management Directives • NESDIS DAA requested COPB/CLASS Project finalize L1RD NOAA Administrative Order 212-15 OCIO Data Architecture NOSC/EDMC Archive Architecture Data Management Directives NESDIS/COPB Archive ConOps CLASS Level 1 Requirements DAARWG Meeting

  9. Enterprise ConOps and Architecture Aerospace Contract ($625K, Sep 11 – May 12) • Support to NOAA Data Center Enterprise Concept of Operations (CONOPS), Architecture, and Data Management Strategy Development • Joint EDMC/Data Center project • Objectives and Scope: • Assist the Government in developing a Concept of Operations (CONOPS) for the NOAA Data Center archival and stewardship architecture target state • Capture the baseline (“as is”) architecture, generate a target architecture, and develop a transition roadmap and deployment/communication strategy • Assist the Government in developing an enterprise data management strategy based on government and industry best practices, and including policy and governance recommendations • Kick-off Meeting: 17 November DAARWG Meeting

  10. Enterprise ConOps and Architecture Expected Outcomes of the Study • Data Center ConOps • Flow ConOps into required Federal Enterprise Architecture documents for KDP2 • KDP2 - Solution Alternatives Identification (for CLASS Enterprise release) • Following KDP2, adopt Final L1 Requirements Documents • Move rapidly to KDP3 and remaining steps for CLASS Enterprise DAARWG Meeting

  11. Current Archive Activities

  12. Archive Data Management Framework Inter-Agency (e.g., NWS-CS MOA, in draft) Memorandum of Agreement (MOA) Producer-Archive (e.g., NCEP-NCDC SLA, in draft) Service Level Agreement (SLA) Preliminary Phase • NAO 212-15, Management of Environmental Data and Information • Procedure for Scientific Records Appraisal and Archive Approval (What to Archive) • Open Archival Information System Reference Model (OAIS-RM) • Producer-Archive Interface Methodology Abstract Standard (PAIMAS) Initiate Request to Archive Initial Decision Requirements Documents What to Archive Process Recommendation Package Final Decision Formal Definition, Transfer and Validation Phases How to Archive Process Design Documents Preliminary Agreement Design Approval Submission Agreement (SA) Test & Implement Archive DAARWG Meeting

  13. Maturity Matrix & Data Quality Example for a Satellite Data Set DAARWG Meeting

  14. Tiers of Service Enterprise Development DAARWG Meeting

  15. Federation of Systems Prototype Capability • NCDC partnership with RENCI and DataNet Consortium • Prototype “system of systems” framework; federation of NOAA data systems with NOAA archive using iRODS (Integrated Rule-Oriented Data System) • Connectivity to data systems such as RENCI, ORNL, OOI, and Earth System Grid (ESG) • Pilot Project • Federate with RENCI to share 70TB of NEXRAD data • Utilize highly distributed computing to derive climate-quality precipitation re-analysis, push data products to NCDC archive system (CLASS) • Future Plans to Support Climate Assessments • Federate with NOS systems via RENCI to integrate data from OOI with climate data at NCDC • Federate with GFDL and ESG to integrate climate model data with in situ and satellite data at NCDC • Special Interest Group Climate (SIGClimate): Collaborative effort with Goddard DAARWG Meeting

  16. NCDC Archive Architecture (S0) DAARWG Meeting

  17. CLASS Status Update

  18. Need for Data Center Ownership Common Theme from Independent Review Teams • Independent review by Dolly Perkins, NASA; NESDIS/OSD commissioned; Final Report, 11/25/09 • ”The COPB is not yet fully responsible for or held accountable for CLASS operations. Responsibility and Accountability requires having the Authority which has not been conveyed to the COPB.” • ”Communication between the Data Centers (individually and via the COPB) and the CLASS project is inadequate and ineffective. No one other than the CLASS project seems to have current understanding of where CLASS development is heading.” • Independent review by Tim Hall, Aerospace; CLASS Operations and Planning Board (COPB) commissioned; Final Report, 4/1/11 • “At the core of data center concerns is lack of ownership and control of CLASS. CLASS has evolved as a system viewed and treated as outside the architecture of the Data Centers” DAARWG Meeting

  19. Program Management Transition to NCDC, Jun 11 NCDC NGDC NODC COPB COWG Support Services Division Remote Sensing Applications Division Budget Execution - 5% (T. Leary) Government Task Monitor – 30% (N. Ritchey) COR / Proj Engineer + CWIP – 100% (Jim Goudouros ) CLASS Program Manager – 100% (Kern Witcher) Alt COR – 5% (J. Niemiec ) IT Specialist/Systems Engineer - 100% (Jay Morris ) Legend Budget Analyst– 5% (L. Cholid) Operations Manager – 100% (D. Carter) NCDC Inventory Mgr – NCDC – 1% (K. Tarver ) Other NESDIS OMB Exhibit 300 – 30% (K. Tarver, acting ) Contractor Project Specialist – 100% (K. Tarver , acting) Supervisory Reporting Inventory Spec – NCDC – 10% (A. Annis, acting ) ISSO – 100% (Scott Koger) DAARWG Meeting

  20. Program Management Transition to NCDC, Jun 11 Change in Approach • First and foremost – Deliver on our Commitments to our Customers • NPP and GOES-R preparations • Day to day ingest, archive, and access • Adopt ‘Major’ Program Management Best Practices • Portions of NASA Procedural Requirements NPR 7120.5 and NPR 7123.1 • NOAA Draft 2007 Major Program Acquisition • Realign work to be more product/project oriented • Adopt Federal Enterprise Architecture with all components documented leading to a KDP-2 for CLASS enterprise software build DAARWG Meeting

  21. Recent Accomplishments • Completed all test in preparation for NPP launch • Archived • Climate Forecast System Reanalysis and Reforecast (CFSRR) • Continuously Operating Reference Stations (CORS) • NESDIS approved plan to migrate from legacy systems by FY15 DAARWG Meeting

  22. Next Steps • Release of 5.4.1; implement final version of NEAAT • Complete Cloud Computing Study • Migrate data from legacy systems (NEXRAD) DAARWG Meeting

  23. Summary • EDMC and Data Centers are contracting with Aerospace to develop an Archive Architecture • DAARWG should be engaged in this effort • CLASS has turned a corner and is becoming fully integrated into the Data Centers DAARWG Meeting

More Related