Technical Reviews Module Space Systems Engineering, version 1.0

1. Technical Reviews ModuleSpaceSystems Engineering, version 1.0

2. Module Purpose: Technical Reviews • To understand the purpose and value of conducting technical reviews. • To discuss the timing of technical reviews over the course of a project’s life cycle. • To consider the entrance criteria and success criteria for the standard project technical reviews. • To understand when a technical review is complete.

3. Definition of Technical Reviews • Technical reviews are scheduled within a project or program to communicate an approach, demonstrate an ability to meet requirements, or establish status. • Technical reviews often serve as control gates for management to make a go/no-go decision on a project. • Control gates involve formal examination of a project’s status in order to obtain approval to proceed. • Technical reviews may be held for subsystems too; depending on their size and complexity. Subsystem reviews precede their corresponding system review. • When do technical reviews occur? • Throughout the entire life cycle of a project, as shown on the following life cycle chart.

4. Reviews in a NASA Project’s Life Cycle

5. The Value of Technical Reviews • Technical reviews are key development milestones used to measure progress, assess project maturity and to infuse lessons from the past. They… • Provide confirmation of completed effort and readiness to commit additional resources for the next phase. • Encourage and establish project discipline with an integrated project team perspective. • Identify risks and review mitigation options. • Describe plans and priorities for the next phase. • Technical reviews give everyone (design team, non-advocate discipline experts, customer and consumer) the opportunity to agree on the current project baseline (requirements, interfaces, allocations, margins, schedules, risks, budgets, etc.). • Functional, resource and performance allocations and assumptions are described and confirmed. • System development constraints and interfaces are described and agreed to by both sides. • Risks and development problems are identified and mitigation options discussed.

6. More Values of Technical Reviews • A significant, and perhaps unexpected, value to the development team is the preparation for a review. Usually everyone is busy in their own domain - preparing for a review forces the team to describe what they have done and understand what others are doing. While there is great value in the preparation, there is also unique value in the execution. One Air Force officer once took Dwight D. Eisenhower’s famous quote: “Plans are nothing. Planning is everything.” too seriously and on the day before it was scheduled cancelled the PDR of a $100 million space mission. After a minor revolt, the PDR was reinstated and the team benefited from the preparation and the review. • Periodic project reviews are held to demonstrate that the appropriate products, accomplishments and plans have been completed before proceeding to the next phase. Appropriate products, accomplishments and plans are based on the lessons of hundreds of past projects (best practices). NASA, for example, describes standard entrance and success criteria for all standard milestone reviews.

7. NASA’s Minimum Set of Technical Reviews • Mission Concept Review (MCR), • System Requirements Review (SRR) and/or Mission Definition Review (MDR), • Preliminary Design Review (PDR), • Critical Design Review (CDR), • System Integration Review (SIR), • Test Readiness Review (TRR), • Operational Readiness Review (ORR), • Flight Readiness Review (FRR), • Post-Launch Assessment Review (PLAR), • Critical Event Readiness Review (CERR), and • Decommissioning Review (DR). While this may seem like a large number of technical reviews, each has its own focus and proven value. In addition, since mission lifetimes are several years, typically there is 6 months or more between major reviews.

8. The Primary Questions of Each Review • Mission Concept Review - Does the proposed concept meet the mission need and objectives? • System Requirements Review and/or Mission Definition Review - Do the functional and performance requirements and the selected concept satisfy the mission? • Preliminary Design Review - Does the preliminary design meet all the system requirements within acceptable cost, schedule, and risk? • Critical Design Review - Is the system design mature enough to proceed with full-scale fabrication, assembly, integration and test? • System Integration Review - Are the system, facilities, personnel, plans and procedures ready for system integration? • Test Readiness Review - Is the project ready to commence with verification testing? • Operational Readiness Review - Are all systems hardware, software, personnel, and procedures in place to support operations? • Flight Readiness Review - Is the system ready for launch? Are the ground facilities and personnel ready to support launch? • Post-Launch Assessment Review - After launch and deployment, are the spacecraft systems ready to proceed with routine operations? • Critical Event Readiness Review -Is the project ready to execute the mission’s critical activities during flight operation? • Decommissioning Review - Is the system ready to be terminated?

9. Pre-Phase A Reviews Mission Concept Review (MCR) • A validation that the mission has clearly established needs, objectives, and top-level functional/performance requirements, and that at least one way of conducting the proposed mission is realistic and attainable within existing or projected technology and ROM cost. • An internal Center review(s) of all Pre-Phase A activities and products should be conducted prior to forwarding the Pre-Phase A report to Headquarters. Technical, management, resources, and scientific personnel should conduct the review. Peer Reviews • Used informally during Pre-Phase A and Phase A. • The peer group is composed of individuals selected from outside the project according to their expertise in the applicable disciplines. • Throughout Pre-Phase A and Phase A, peer reviews should informally check the evolving mission concept against objectives, requirements, and constraints. • Peer reviews help you take advantage of other engineering experience from colleagues who have worked on different missions. They can point out issues they confronted that may be similar for your mission concept.

10. Phase A Reviews System Requirements Review (SRR) • The primary focus of the SRR is to verify the realism of the functional and performance requirements, ensure their congruence with the mission and system configuration, and ensure the mission objectives can be satisfied. • The SRR encompasses all major participants (NASA and contractors), and is chaired by the Project Manager. • A product from the SRR is the project system specification that is formally baselined and placed under configuration management control. Mission Definition Review (MDR) • A validation that the mission objectives can be satisfied, the partitioning of the functionality to each of the systems is adequate, the top-level performance requirements for each system have been defined, and the technology required to develop the systems and implement the mission is attainable. • The MDR is keyed to the end of Phase A and evaluates the mission definition, system design, operational concepts, schedule, and cost estimates.

11. Example Entrance Criteria — System Requirements Review 1. Successful completion of the Mission Concept Review (MCR) and responses made to all MCR Requests for Actions (RFAs). 2. A preliminary SRR and/or MDR agenda, success criteria, and charge to the board have been agreed to by the technical team, project manager, and review chair prior to the SRR and/or MDR. 3. The following technical products for hardware and software system elements are available to the cognizant participants prior to the review: a. System Architecture. b. System requirements document. c. System software functionality description. d. Updated concept of operations. e. Updated mission requirements, if applicable. f. Baselined SEMP. g. Preliminary system requirements allocation to the next lower level system. h. Updated cost estimate. i. Technology Development Maturity Assessment Plan. j. Preferred system solution definition including major trades and options. k. Updated risk assessment and mitigations. l. Updated cost and schedule data. m. Logistics documentation (preliminary maintenance plan, etc.). n. Preliminary human rating plan, if applicable. o. Software Development Plan (SDP). p. System safety and mission assurance plan. q. Configuration management plan. r. Project management plan. s. Initial document tree. t. Verification and validation approach. u. Preliminary hazard analysis (PHA).

12. Example Success Criteria — System Requirements Review • The resulting overall concept is reasonable, feasible, complete, responsive to the mission requirements, and is consistent with system requirements and available resources (cost, schedule, mass power, etc.). • The project utilizes a sound process for the allocation and control of requirements throughout all levels, and a plan has been defined to complete the definition activity within schedule constraints. • Requirements definition is complete with respect to top level mission and science requirements, and interfaces with external entities and between major internal elements have been defined. • Requirements allocation and flow down of key driving requirements have been defined down to subsystems. • System and subsystem design approaches and operational concepts exist and are consistent with the requirements set. • The requirements, design approaches, and conceptual design will fulfill the mission needs within the estimated costs. • Preliminary approaches have been determined for how requirements will be verified and validated down to the subsystem level • Major risks have been identified, and viable mitigation strategies have been defined.

13. PBS 3/26 5/4 4/9 5/10 5/10 5/14 5/10 5/14 5/14 5/14 CLV SRR Issue ID, Form Entry Develop PBS Presentation Develop PBS Presentation 4/9 3/26 Issue ID, Form Entry 5/9 5/5 4/13 4/6 Rev. Issue Table, POC, & Sched. Rev. Issue Table, POC, & Sched. Rev. Issue Table, POC, & Sched. Rev. Issue Table, POC, & Sched. Issue ID, Form Entry Issue ID, Form Entry 5/8 Constellation Program Office Schedule of SRRs You are Here! 2007 5/23 March April May Provide Issue list w/ Resolution Plan & Impacts 3/29 Rev. Issue Table, POC, & Sched. 5/14 4/6 4/9 Develop PBS Presentation 4/16 4/23 4/30 Provide Status on Issue Resolution (every Mon. @ Cx SE&I 8:30 telecon) (send updates to Cx SE&I COB every other day in May) 5/10 Provide Issue list w/ Resolution Plan & Impacts 4/16 4/23 Provide Status on Issue Resolution (every Mon. @ Cx SE&I 8:30 telecon) 4/30 (send updates to Cx SE&I COB every other day in May) CEV SRR 5/10 Provide Issue list w/ Resolution Plan & Impacts 4/23 4/30 4/16 Provide Status on Issue Resolution (every Mon. @ Cx SE&I 8:30 telecon) (send updates to Cx SE&I COB every other day in May) MO SRR Provide Issue list w/ Resolution Plan & Impacts Develop PBS Presentation GO SRR Provide Status on Issue Resolution (send updates to Cx SE&I COB every other day in May) Provide Issue list w/ Resolution Plan & Impacts Issue ID, Form Entry Develop PBS Presentation EVA SRR Provide Status on Issue Resolution (send updates to Cx SE&I COB every other day in May)

14. Board AI Total AI Count: 48 Total 9 Closed 39 Open 30 are “Past Due” 9 are in work, “Not Due Yet” RIDs Total RID Count: 6,283 6225 Closed 58 Open Majority closed prior to PBS All RIDs should be closed prior to CxP SDR Constellation Program Office Status - Closure of SRR TBD/TBR • Total Count: 2,532 • Received plans for 2,064 TBRs/TBDs in 72 of 95 documents (76% of docs.) • Need burn-down plans for 196 TBRs/TBDs in 16 documents (17% of docs.) • 7 documents (274 TBRs/TBDs) will not be updated/baselined until post CxP PBS. These TBRs/TBDs will not be closed prior to CxP PBS

15. Preliminary Design Review (PDR) • The PDR is not a single review but a number of reviews starting with the specific component PDRs, followed by the system-level review. • The PDR establishes the “design-to” baseline and ensures that it meets the program, project , system, subsystem, or specific component baseline requirements. • The PDR process should: • Establish the ability of the selected design approach to meet the technical requirements (i.e., Verifiability/ Traceability); • Establish the compatibility of the interface relationships of the specific end item with other interfacing items; • Establish producibility of the selected design; • Establish the operability of the selected design; • Assess compliance with reliability and system safety requirements; • Establish the feasibility of the approach; • Address status, schedule and cost relationships.

16. When is a Review Complete? Reviews are considered complete when the following is accomplished: • Agreement exists for the disposition of all Review Item Discrepancies (RID) and Request for Actions (RFA). • The review board report and minutes are complete and distributed. • Agreement exists on a plan to address the issues and concerns in the review board’s report. • Agreement exists on a plan for addressing the actions identified out of the review. • Liens against the review results are closed, or an adequate and timely plan exists for their closure. • Differences of opinion between the project under review and the review board(s) have been resolved, or a timely plan exists to resolve the issues. • A report is given by the review board chairperson to the appropriate management and governing program management committees charged with oversight of the project. • Appropriate procedures and controls are instituted to ensure that all actions from reviews are followed and verified through implementation to closure.

17. Example Agenda for a Project System Design Review by a Standing Review Board (SRB) • Purpose of Review & Charge to SRB SRB Chair • Project Overview & Status Project Manager • System Engineering & Status Project SE • Requirements & V&V plans • Trade studies • Technical margins • WBS-level 2 Design State & Status for each area WBS Managers • System Design • Key Requirements • Trade Studies • Technology Readiness • Acquisition Strategy & Long Lead • Logistics & Facilities • Challenges & Risks • Integrated System (e.g., power) State & Status for each area Discipline Leads • Integration & Test Integration Manager • Safety & Mission Assurance (S&MA) S&MA Manager • Human Rating Project HR Rep • Risk Risk Manager • Schedule Project Planner • Cost Cost Manager

18. Pause and Learn Opportunity Student role-play: You are the chief systems engineer for a New Frontiers-class mission to Europa. Your PDR is scheduled in 3 months. What do you do? What do you ask the development team to do? What benefits would you expect from having a PDR? How might it waste time? Or Read and discuss the Crosslink article: The Role of Independent Assessments for Mission Readiness (Crosslink_Independent Review.pdf)

19. Module Summary: Technical Reviews • Technical reviews are key development milestones used to measure progress, assess project maturity and to infuse lessons from the past. They… • Provide confirmation of completed effort and readiness to commit additional resources for the next phase. • Encourage and establish project discipline with an integrated project team perspective. • Identify risks and review mitigation options. • Describe plans and priorities for the next phase. • There are 11 reviews in the minimum set of technical reviews for a NASA robotic mission. These reviews cover the entire mission life — assessing the concepts and designs early; readiness for test, flight and operations in mid-life and plans for disposal at the mission’s end. • These reviews are held to demonstrate that the appropriate products, accomplishments and plans have been completed before proceeding to the next phase. • Appropriate products, accomplishments and plans are based on the lessons of hundreds of past projects (best practices).

20. Backup Slidesfor Technical Review Module

21. Major Project Reviews Precede Each Key Decision Point FORMULATION IMPLEMENTATION A B C D Pre-A E F Concept & Technology Development System Assembly, Test, & Launch Concept Studies Preliminary Design & Technology Completion Final Design & Fabrication Operations & Sustainment Closeout Project Phases C D E B A F Key Decision Points Mission Concept Review Systems Requirements Review Major Reviews Mission/System Definition Review Preliminary Design Review Critical Design Review Systems Integration Review Independent Cost Estimates Operational Readiness Review Flight Readiness Review Post Launch Assessment Review Decommissioning Review

22. Purpose of Technical Reviews A technical review is an evaluation of the project, or element thereof, by a knowledgeable group for the purposes of: • Assessing the status of and progress toward accomplishing the planned activities. • Validating the technical tradeoffs explored and design solutions proposed. • Identifying technical weaknesses or marginal design and potential problems (risks) and recommending improvements and corrective actions. • Making judgments on the activities’ readiness for the follow-on events, including additional future evaluation milestones to improve the likelihood of a successful outcome. • Making assessments and recommendations to the project team, Center, and Agency management. • Providing a historical record that can be referenced of decisions that were made during these formal reviews. • Assessing the technical risk status and current risk profile.

23. Need for Management Reviews The progress between life-cycle phases is marked by key decision points (KDPs). At each KDP, management examines the maturity of the technical aspects of the project. For example, management examines whether the resources (staffing and funding) are sufficient for the planned technical effort, whether the technical maturity has evolved, what the technical and non-technical internal issues and risks are, or whether the stakeholder expectations have changed. If the technical and management aspects of the project are satisfactory, including the implementation of corrective actions, then the project can be approved to proceed to the next phase.

24. Blue text indicates changes from last update