NASA/GSFC Independent Review Process

1. NASA/GSFCIndependent Review Process Bob Savage, Deputy Chief Systems Review Office (SRO) Tel: 301-286-8840 Email: Robert.M.Savage@nasa.gov

2. Systems Review Office (SRO) Scope • The Systems Review Office (SRO) supports Center and Agency leadership in the independent review and assessment of projects. • Independent reviews are performed at critical milestones during the life cycle of flight and flight support systems. • The SRO conducts and/or participates in the review of projects implemented by the GSFC.

3. Key Milestone and Gateway Reviews Key Decision Point (KDP) Gateway Reviews KDP B KDP C KDP D KDP E KDP F KDP A GSFC Gateway Reviews SMSR EOPM Senior Review CRR MRR Mission-level Milestone Reviews ORR SIR MCR SRR/MDR (PNAR) PDR (NAR) CDR PLAR CERR DR FOR MOR PSR* PER *Note: PSR may be conducted by SRB in conjunction with the ORR/FRR with non-SRB members from the IIRT in attendance. Launch Vehicle Milestone Reviews Pre-VOS LVRR LRR/FRR Element-level Milestone Reviews(spacecraft, instruments, ground systems and other key systems) SRRs PDRs CDRs PERs PSRs Engineering Peer Reviews(assemblies, components, and functional areas) Phase A Concept & Technology Development Pre-Phase A Concept Studies Phase C Final Design & Fabrication Phase D System Assembly, Integration & Test, Launch Phase B Preliminary Design & Technology Completion Phase E Operations & Sustaining Engineering Phase F Close out NASA/HQ: CMC: Launch Provider: SRB: GSRT: Project: Review Authority:

4. SRO Functions The Systems Review Office (SRO) is the implementation arm of the GSFC independent review program. • Planning – Making the reviews happen • Facilitating – Making sure the right criteria are established • Management – Making sure the right people are there and review goes smoothly. • Participation – As a chair or a SME

5. Scope of Independent Reviews • Review Board does not have any authority over any project content. • Reviews provide expert assessment of the technical and programmatic approach, risk posture, and progress against the project baseline. • Review board outputs are briefed to the project for which the review is being conducted prior to being reported to the next higher level of management. • Findings from individual panel members are compiled by the review Chair and documented along with his/her own findings and recommendations prior to the CMC reviews.

6. Reporting of Results • Provided in the form of formal reports, electronic summaries, and briefings. • Project teams leverage upon the background and experiences of the independent review board membership, adding value through confirmation of engineering solutions and risk mitigation approaches. • Findings and recommendations are generated independently amongst the Board membership in a consensus or non-consensus manner and are oriented toward assessing risks, primarily the probability of occurrence and/or the impact of potential problems being encountered and confronted during the development lifecycle process.

7. Independent Review Program • Primary objective is to provide an independent assessment of a project’s maturity and risk status to the appropriate decision and technical authorities enabling them to make informed decisions at key development/operations milestones and gateways. • Key Decision Point (KDP) gateway briefing conducted by the Agency or Mission Directorate PMC.1 • GSFC gateway briefings conducted by the CMC 1 • Supported by: • Mission-level milestone reviews conducted by Agency chartered Standing Review Board (SRB) and CMC chartered GSFC System Review Team (GSRT).2 • Element-level Milestone Reviews conducted by an GSRT. • Engineering Peer Reviews conducted by a project chartered independent peer review team.3 • 1 Reference: NPR 7120.5D, NASA Space Flight Program and Project Management Requirements 2 Reference: Reference 1 and GPR 8700.4, Integrated Independent Reviews 3 Reference: GPR 8700.6, Engineering Peer Reviews

8. Center Engineering & Management Policies & Practices Mission Directorate Programmatic Requirements Program Plans Project Plans NASA Requirements Flow-down NPD 1000.0 NPD 1000.3 Strategic Management & Governance Handbook The NASA Organization w/Change 26 (06/19/2008) Engineering NPD NPD 7120.4 NPD 8700.1 NPD 8900.5A Mission Support Office NPDs NASA Policy for Safety & Mission Success NASA Health & Medical Policy for Human Space Exploration Program/Project Management NPR 7123.1 & other Engineering NPRs NPD 7120.5D OSMA NPRs NID 124—41 & OCHMO NPRs Support Organization NPRs Health & Medical Requirements MSO Functional Requirements Program/Project Mgmt Requirements SMA Requirements Engineering Requirements Criteria for KDP Reviews GPR 8700.4, Independent Reviews GPR 8700.6, Engineering Peer Reviews GSFC-STD-1000, GSFC Engineering Standards GSFC-STD-1001, GSFC Milestone Review Criteria Program Plan Project Plan Systems Review Plan (SRP) Engineering Peer Review Plan (EPRP)

9. Program/Project Management Governance

10. GSFC Standards and Guidelines Review ProcessRequirements Flow-down NPD 7120.5D NPR 7123.1 Science Mission Directorate NPRs OSMA NPRs Other GSFC GPRs Standing Review Board (SRB) Handbook Program Plan GPR 8700.4, Independent Reviews GPR 8700.6, Engineering Peer Reviews STD-1000, GOLD Rules STD-1001, Milestone Review Criteria Mission Assurance Guidelines (MAG) Project Plan Terms of Reference (ToR) Systems Review Plan (SRP) Engineering Peer Review Plan (EPRP) Mission Assurance Requirements (MAR)

11. Review Team Interactions MULTI-DISCIPLINE MANAGEMENT AND SYSTEMS ORIENTED Special Reviews Standing Review Boards Mission and KDP Reviews GSFC System Review Teams Other Mission Reviews Flight and Ground Elements and Systems Engineering Peer Review Teams Flight and Ground Subsystems and Components DISCIPLINE-SPECIFIC FUNCTION ORIENTED

12. Types of Review Teams • Standing Review Board (SRB) • Agency Review Team through IPAO at Langley • Goddard System Review Team (GSRT) a.k.a IIRT • Goddard independent review team through SRO • Engineering Peer Reviews (EPR) • Center independent review team through project

13. Sample Discipline Matrix

14. Code 301 Organization Carolyn Dent/ Office Chief Robert Savage/ Deputy Office Chief Linda Millsap/Secretary • Systems Review Managers • Carolyn Dent/Code 301 • TDRS-K, LDCM, SGSS • Doug Fineberg/Code 301 GOES-R, TIRS • Ted Hammer/Code 301 • JWST, GPM • Linda Pacini/Code 301 • NPP, GEMS, MMS, MOMA • Susanna Petro/Code 301 • MAVEN, SO, SPP Magnetometers • Kirk Rhee/Code 301 • RBSP, ICESat-2, JPSS-Free Flyer • Bob Savage/Code 301 • SMAP Radiometer, IRIS, Astro-H/SXS, JPSS-Flight Business/Schedule Assessments Charles (Chuck) Athas/ Sigma Space Corporation • GRMS and • Review Management Support • Joe Polesel/ASRC • Anna Montoro/ASRC • Andre Young/ASRC • External GSRT Chairs • James Barrowman/SGT • ICESat-2 • Jesse Leitner/Code 300 LADEE

15. Code 301 Organization Backup

16. Systems Review Office (SRO) Scope The Systems Review Office (SRO) supports Center and Agency leadership in the independent review and assessment of projects per NASA and GSFC directives and standards. This is accomplished in the form of life cycle reviews and represent essential elements of conducting, managing, evaluating, and approving space flight programs/projects. Independent reviews are performed at critical milestones during the life cycle of flight and flight support systems developed by; (a) the Center, (b) other NASA Centers, (c) other Agencies, (d) private industry or universities under contract to the Center, and (e) foreign governments under cooperative international programs. The SRO conducts and/or participates in the review of the overall mission design and operations, integrated payloads, spacecraft, instruments, flight support-ground based systems, and launch vehicle systems for projects implemented by the GSFC.

17. SRO Functions The Systems Review Office (SRO) is the implementation arm of the GSFC independent review program. Planning entails participating in, or taking the lead in, the development of the Systems Review Plan, standing up review board’s, developing review team charter’s, and developing and maintaining procedural requirements and standards documentation. Facilitating a successful review involves working with the Project to establish the Success Criteria and Key Evaluation Factors used in establishing the level of success of a particular review; as well as supporting the generation and management of Requests for Action. Management functions include budgeting the manpower and cost requirements of the proposed review program, organizing an effective review process tailored to the project , and coordinating the reviews through official review team appointments and agenda review. Participation in a review by the SRO involves serving as a Subject Matter Expert on selected systems and peer review teams; as well as chairing key mission and systems-level development lifecycle reviews.

18. Scope of Independent Reviews • Review Board does not have any authority over any project content. Board members can only offer advise and suggestions to the project and the convening authority. • Provide advanced technical advice and risk assessments. Review Board members cannot provide direction or participate directly in the decision-making process of the project. • Identify major technical issues and formulates an independent position based on consensus or non-consensus building approach, depending on controlling requirements for conduct of the review. • Reviews provide expert assessment of the technical and programmatic approach, risk posture, and progress against the project baseline. • Review board outputs are briefed to the project for which the review is being conducted prior to being reported to the next higher level of management. • Findings from individual panel members are compiled by the review Chair and documented along with his/her own findings and recommendations prior to the CMC reviews.

19. Reporting of Results • Provided in the form of formal reports, electronic summaries, and briefings to Project Management, Program Management, Center Management Council (CMC), and the Center Directors Office. Additional briefings to the Mission Directorate Program Management Council (PMC) and the Agency PMC are also supported as required. • Project teams leverage upon the background and experiences of the independent review board membership, adding value through confirmation of engineering solutions and risk mitigation approaches. • Findings and recommendations are generated independently amongst the Board membership in a consensus or non-consensus manner and are oriented toward assessing risks, primarily the probability of occurrence and/or the impact of potential problems being encountered and confronted during the development lifecycle process.

20. Standing Review Board • Agency established panel of experts responsible for the conduct of combined technical and management assessments associated with Agency defined Key Decision Points (KDP’s) progressively located throughout the project development lifecycle. • Assessments encompass the selected mission-level reviews defined within the project System Review Plan (SRP). Requirements for these reviews are specified in the SRP directly and by referencing the listed “applicable” documents, including GPR 8700.4 and NPR 7120.5D. • A Terms of Reference (ToR) is created during the initial phase of the project lifecycle and provides specific definition of the SRB charter including the roles and responsibilities of all of the participants and defines the anticipated lifecycle independent review program. • Developed by the SRB Chair and IPAO Review Manager in consultation with the HQ Program Executive (PE) and GSFC Systems Review Manager (SRM). • SRB membership is comprised of independent multi-discipline experts recruited from the GSFC, the GSFC System Review Team (GSRT), and sources external to the Center.

21. GSFC Systems Review Team • Center established panel of experts responsible for the conduct of combined technical and management assessments at key lifecycle milestones. • Assessments encompass selected mission-level reviews, the complete set of element and system-level lifecycle reviews, and selected subsystem reviews as defined by the project SRP. • The requirements for the GSRT reviews are specified in the SRP directly and by referencing the listed “applicable” documents, including GPR 8700.4 and NPR 7120.5D. • Separate review teams with differing Chairs and/or SRM’s may be established for specific elements of a mission (i.e., Ground Systems and Mission Operations, Instrument Systems, etc.) to enhance coverage by the subject matter experts and consequently the rigor of the associated reviews. • Membership is comprised of independent multi-discipline and discipline-specific experts internal and external to the GSFC. Includes selected Engineering Peer Review and SRB member participation as requested by, and/or negotiated with, the GSRT Chair(s).

22. Engineering Peer Review Panel • Project established panel of experts responsible for the conduct of assessments specific to components, subsystems, systems, and discipline-related functions as defined in the projects Engineering Peer Review Plan (EPRP) per GPR 8700.6. • Review scope and panel participants are established by the project lead systems engineer as the project Technical Authority in coordination with the GSFC Chief Engineer and AETD senior management. • Focused and technically in-depth reviews supporting the evolving design and development of a product subsystem or discipline area. • Purpose is to add value and reduce risk through expert knowledge infusion, confirmation of approach, and specific recommendations. • Provides a penetrating examination of design, analysis, manufacturing, integration, test and operational details, drawings, processes and data. • Panel membership draws upon independent resources from inside and outside the implementing organizations with knowledge and experience targeted to the product or function under review. Includes participation of selected GSRT and SRB members as requested by, and/or negotiated with, the project.