GENERAL GUIDANCE ON CONDUCT OF TECHNICAL REVIEWS

1. GENERAL GUIDANCE ON CONDUCT OF TECHNICAL REVIEWS Type and Number Selection should be based on the complexity of the program and the phase of development (modification). Participants Include tasking and performing activity personnel responsible for the area or item being reviewed, key representatives for lower level items, and other personnel who have a stake in the specific objectives of the review. Objectives 1. Demonstrate progress 3. Verify the expected maturity 2. Ensure issues have been resolved 4. Confirm that risks are acceptable

2. FOCUS IN MAJOR REVIEWS In General • Confirm that issues addressed during previous reviews have been satisfactorily resolved. • Only areas requiring close scrutiny should be addressed in detail. Proper integration and management of interim, subsystem, and functional reviews (as planned in SEMP) should make a detailed total evaluation unnecessary. • Opportunity to modify program emphasis for the next phase based on risk levels.

3. TYPES OF REVIEWS Major: Demonstrate that risk levels are acceptable and provide an opportunity to modify program emphasis for next phase/effort. (Examples: ASR, SRR, SFR, PDR, CDR, SVR, PCA) Functional: System wide review conducted by involved disciplines to address progress and issues associated with one functional area. (Examples: Development, Training, Support, Verification, Manufacturing, Disposal, etc.) Subsystem: Multidisciplinary reviews to assess progress in defining and satisfying subsystem requirements. Interim: Reviews held between major reviews that cover spectrum of system, segments, and configuration items.

4. Integrated Technical Reviews (Cont.) Major Technical Reviews include: - Alternate System Review (ASR): The ASR forms the basis for determining which system concepts should be continued as candidate development programs. It provides the data for recommendations to eliminate potential systems concepts. - System Requirements Review (SRR): The SRR forms the basis early in System Development and Demonstration whether customer requirements are understood. It should confirm the progress and direction of technology verifications and demonstrations and convergence on viable system requirements

5. Integrated Technical Reviews (Cont.) Major Technical Reviews include: - System Functional Review (SFR): The SFR (formally called the SDR) forms the basis to establish and verify an appropriate set of functional and performance requirements for the complete system. - Preliminary Design Review (PDR): The PDR forms the basis for determining whether a preliminary physical architecture and design approach for configuration item or aggregation of configurations (including software) is ready to start detailed design. A configuration item development spec must be complete and ready for use in detailed design

6. Integrated Technical Reviews (Cont.) Major Technical Reviews include: - Critical Design Review (CDR): The CDR confirms the detailed design for each configuration item (including software), each aggregation of configuration items, and that requirements are satisfied. - System Verification Review (SVR): The SVR demonstrates the total system (people, products, and processes) satisfies functional and allocated configuration documentation requirements and confirms readiness for production, support, training, deployment, operations, an disposal. The Production Readiness Review (PRR) is a part of the SVR.

7. Customary System Level Review Accomplishments SRR - Review system functional and performance requirements. SDR/SFR - Review allocation of system requirements to configuration item (CI) aggregates. Assess maturity of system spec. and CI aggregates. PDR - Review design progress for each CI and aggregate of CIs: 1. Review predicted performance 2. Review physical and functional interfaces 3. Verify integration of specialty eng. req. 4. Evaluate risk resolution

8. Customary System Level Review Accomplishments (Cont.) CDR - Review maturity of design for each CI and aggregate CIs: 1. Verify that functional and performance requirements of CI specs are satisfied. 2. Verify that engineering specialty requirements have been satisfied. 3. Verify compatibility between CIs and hardware, software, facilities and personnel. 4. Review preliminary product specs. 5. Assess producibility risks.

9. Customary System Level Review Accomplishments (Cont.) • SVR - Confirms readiness to enter full rate production: • 1. Verify work on all CIs – all functions & performance levels met. • FCA conducted for CIs. See page 107 of Sys. Eng. Fundamentals for additional accomplishments.

10. Systems Engineering Master Schedule (SEMS) • Purpose: • Contractual Agreement Between Contractor and Customer on Critical Tasks Required for Completing Each Major Program Milestone and Defining the Criteria for Successful Completion of Those Tasks.

11. Role of SEMS in Acquisition Planning The SEMS Outlines the Integration of All Major Tasks Necessary to Design, Develop, Test, Produce, Deploy and Support the System. Reliability Producibility Schedule Performance Cost Support Training Risk Management System Balanced Process/Product to Satisfy User Needs Prime Mission Equipment

12. SEMS Key Tasks • Typical Tasks • Develop Test Plans • Develop Software Plans • Tasks Define Interim Steps to Developing and Producing System • Tasks Should Include Verification Criteria

13. Typical Accomplishment for Sample Tasks Test Plan Complete Software Development Plans Complete Accomplishment Should Be Easily Measurable Example of Difficult to Measure Accomplishment Test Plan 85% Complete SEMS Accomplishment Guidelines

14. SEMS Sample Accomplishment • Milestone 3 - Test Readiness Review (TRR) • Accomplishments • 3.1 Complete Software Test Procedures • 3.2 Complete System Integration Test Procedures • 3.3 Complete All Major Subsystem TRRs • 3.4 Complete All Action Items from Milestone 2

17. Section L - Instructions, Conditions, and Notices to Offerors Systems Engineering Master Schedule (SEMS) volume 3 attachment 1

18. Section L - Instructions, Conditions, and Notices to Offerors (Cont.) 3.0 SEMS PROGRAM EVENTS Listed below are the Major Program Events and their scheduled time of achievement. Each event shall be considered achieved upon successful demonstration of criteria accomplishments. Criteria for each event is defined in Section 4.0. Event Number Event Title Event Date 1. System Hardware Preliminary 5 MACA Design Review(PDR) 2. System Software PDR 7 MACA 3. ILS Management Team (ILSMT) 9 MACA Meeting 4. System Hardware Critical Design 11 MACA Review (CDR) 5. System Software CDR 13 MACA 6. System Test Readiness Review 22 MACA 7. Ground Durability/Qualification 24 MACA Test Release

19. Section L - Instructions, Conditions, and Notices to Offerors (Cont.) Event Number Event Title Event Date 8. (Reserved) 9. RF-4C #1 Development Flight 29 MACA Test Release 10. RF-4C DT&E/IOT&E start 29 MACA 11. UARS #1 Development Flight 29 MACA Test Release 12. UARS DT&E/IOT&E start 29 MACA 13. F/A-18 DT&E/IOT&E Start 29 MACA 14. FSD Maintainability Demonstration 34 MACA 15. Production Readiness Review 35 MACA 16. System Function Configuration 36 MACA Audit (FCA) 17. RF-4C DT&E/IOT&E Complete 38 MACA 18. UARS DT&E/IOT&E Complete 38 MACA

20. Section L - Instructions, Conditions, and Notices to Offerors (Cont.) Event Number Event Title Event Date 19. F/A-18 DT&E/IOT&E Complete 38 MACA 20. Formal Qualification Review (FQR) 38 MACA 21. Milestone IIIA 39 MACA 22. System Physical Configuration 57 MACA Audit (PCA) 23. LRIF Maintainability Evaluation 57 MACA 24. Milestone IIIB 66 MACA 25. F-14D/TARPS DT&E/IOT&E 72 MACA Start 26. F-14D/TARPS DT&E/IOT&E 81 MACA Complete 27. Initial Operational Capability See (IOC) Appendix 28. Delivery Per Table 4-2 As Required *MACA: Months After Contract Award

29. EXERCISE #2 Exercise: Develop the portion of a Systems Engineering Master Schedule (SEMS) for the AJS design and development tasks. Assume the system level schedule for Peace Whey includes a SRR, SFR, PDR, CDR, and PRR. Your SEMS should identify a list of AJS program reviews with accomplishment criteria. • Part 1: Referring to the system level Peace Whey reviews, identify and list the major reviews which need to be supported by inputs from the AJS design and development tasks. Based upon this list, determine a set or reviews for the AJS development effort required to support the Peace Whey program. • Part 2: Identify the accomplishment criteria for one AJS review. • You will have 20 minutes to complete this exercise.

30. IMP/SEMS & IMS/SEDS Relationships Integrated Master Plan (IMP) Integrated Master Schdule (IMS) System Engineering Detailed Schedule (SEDS) System Engineering Master Schedule (SEMS) Program Office Contracts Security Program Office Contracts Security

31. A NN NN NN NN Product Event No. Accomplishment Criteria IMS Task IMP/IMS Development Steps 1. Define Events, Accomplishments (RFP “Shalls”), and Criteria SYSTEM PRODUCT HIERARCHY SPEC TREE 2. Define a Product Oriented Management Structure (SPEC Tree, WBS, and IPT Organization) A WBS IPT SPEC TREE B C D E WBS IPT SPEC TREE WBS J K L M IPT SPEC TREE WBS F G H I IPT 3. Build a Top Down, Layered IMP/IMS with Activity Numbers as Shown Below

32. Integrated Master Plan (IMP) and Integrated Master Schedule (IMS) Terms • Product - Hardware, Software, Facilities, Data, or Materials • Event - Decision Point at End of Major Project Activity (EX. - CDR) • Accomplishment - Desired Result at Specified Events • Criteria - Measure of Meeting Accomplishment • Task - Specific Activity to Complete a Criteria

33. Top Level Task Relationships Product Requirements Suppliers Development • Primary Production • Derived Product Definition • Inventory Control Delivered • Parts Production Product • Assembly Hardware Design Parts Assemblies Requirements Software For Design Design Component & System Weapon Testing Software System Implementation Integration Requirements FEEDBACK & Testing For • Requirements Integrated Test • Design

34. Network Chart Example (D) 6 Weeks (A) 4 Weeks (E) 3 Weeks (H) 2 Weeks (B) 2 Weeks (F) 8 Weeks (I) 4 Weeks (C) 1 Week (G) 4 Weeks (K) 8 Weeks (J) 5 Weeks Network Layout C S A D E H B F I C G J K 14 2 4 6 8 12 10 Conversion to Milestone Chart

35. Integrated Master Schedule(IMS) and Network Schedules • IMS Show Dates for Major Tasks and Milestones • - Usually Presented as Gantt (Bar) Chart • - Shows Tasks Corresponding to Entry/Exit Criteria • - Must Include All IMP Milestones • Network Schedules Are Essential to the IMS • - Helps Determine if IMS Dates and IMP Events Are Achievable • - Critical-Path Method (CPM) Schedule Software is Essential • - Essentials: Tasks, Relationships, Durations, Resources

36. Negotiated IMP Will Be Place on Contract, Changes Will Require a Contract Change Proposal IMS Shall Support the IMP. IMS Updates Shall Be Approved Prior to Implementation of Changes. IMS is Time-Based Customer May Provide Government IMP in RFP. Government IMP Should Provide Detailed Information for Next Acquisition Phase to Identify Specific Events, Accomplishments and Criteria Necessary to Satisfy Planned and Required Technical Exit Criteria IMP Is a Key Control Element of SE Process. IMP Can Be Used as the Basis for Quantitative Requirements for Award Fees IMP/IMS Summary

37. Exercise # 3 • Exercise: • Referring to the SEMS from Exercise #2, develop a System Engineering Detailed Schedule (SEDS) for the AJS design and development task. The AJS Safety of Flight unit must be available in month 26 and first production aircraft in month 36. Be prepared to report your results to the class. You will have 20 minutes for this exercise. Hint – Use building block tasks on next page.

38. Electronic Warfare

39. Backup

40. Progressive Subsystem Informal Subsystem Formal Subsystem Limited Formality Formal Issues Roll-Up

41. Integrated Technical Reviews (Cont.) Technical reviews are a series of systems engineering activities by which the technical progress of a program is demonstrated relative to its technical or contractual requirements. They are conducted at logical transition points in the development effort to reduce risk by assessing progress against SEMS requirements and identifying and correcting problems/issues resulting from the work completed before the program is disrupted or delayed. They provide a method for the contractor and government to determine if the development of a system and/or configuration item and its documentation have met contract requirements.

42. Planning Guidance from DoDD 5000.1(Defense Acquisition) Part 2.B.3 Acquisition Strategies, Exit Criteria, and Risk Management “Event driven acquisition strategies and program plans must be based on rigorous, objective assessments of a program’s status and the plans for managing risk during the next phase and the remainder of the program. The acquisition strategy and associated contracting activities must explicitly link milestone decision reviews to events and demonstrated accomplishments in development, testing and initial production. The acquisition strategy must reflect the interrelationships and schedule of acquisition phases and events based on logical sequence of demonstrated accomplishments not on fiscal or calendar expediency.”

43. Integrated Technical Reviews • The government assesses the status, maturity, risk, and verification status by conducting reviews that will: • Be adequately planned prior to the initiation technical work • Have event-driven entry and exit criteria with defined accomplishments and success metrics via the SEMS. • Be conducted incrementally (as appropriately) to measure progress towards achieving major accomplishments and obviate unexpected issues resulting from incomplete or inconclusive development activities. • Be conducted by multidisciplinary teams to ensure that all the system's functions are addressed and all system elements are integrated and balanced across the system.