Lecture 3.2: Technical Reviews and Audits (SEF Ch 11)

1. Lecture 3.2: Technical Reviews and Audits (SEF Ch 11) Dr. John MacCarthy UMBC CMSC 615 Fall, 2006

2. Agenda • Definitions • Progress Measurement (11.1) • Planning • Technical Reviews and Audits (11.2) • Tailoring (11.3) • Summary (11.4)

3. Technical Review (TR): A review of the key technical artifacts developed (or updated) in a given design phase/activity Verifies artifacts have been completed with adequate quality Technical Review Meeting: The management meeting held at the end of the TR that verifies that the entry and exit criteria for the TR have been accomplished Technical Manager verifies that the technical maturity of the system/item is sufficient to move to the next development phase/activity Closes current development phase/activity (for EV credit) Major SDD Technical Reviews and Audits: System Requirements Review (SRR) System Design/ Definition/ Functional Review (SDR/SFR) Preliminary Design Review (PDR) Software Specification Review (SSR) [basically a PDR for SW] Critical Design Review (CDR) Test Readiness Review (TRR) Functional Configuration Audit/ System Verification Review (FCA/SVR) Production Readiness/Approval Review (PRR/PAR) Physical Configuration Audit (PCA) Major Types of Reviews Requirements Reviews Design Reviews Verification Reviews Generally there are four Phases to a Technical Review: Planning (for the TR) Review (of the technical artifacts) Technical Review Meeting (Closure) Follow-on (Actions) The Technical Review Meeting is NOT the same as the Technical Review Often they are confused Technically the TR begins weeks prior to the TR Meeting It often has a “Kick-off meeting” It often has one or more associated TIMs or IPT WG Meetings The TR Meeting closes the TR. Definitions

4. Technical Reviews and Progress Measurement (11.1) [1] • Technical Reviews (and Audits) provide an objective measure of progress toward completing the design and development of the product (are used as EV criteria) • Technical Reviews are held at the end of each major technical activity/development stage (see “V”) to: • review progress on design • review risk • determine if design is mature enough to proceed to the next phase • Technical Reviews have a set of documented (objective) entry and exit criteria • Technical Reviews reduce program risk and ease transition to production • Technical Review Meetings are customer/management level meetings focused on decisions: • They should focus on providing evidence that key artifacts/products are completed and of adequate quality • Getting decisions on issues (from IPTs & stakeholders) • They should NOT be a detailed review of each of the artifacts • Technical Review Meetings should be preceded by a series of Technical Interchange Meetings (TIMs) where the artifacts/ products are reviewed in detail by the contractor, customer, & other stakeholders) and issues are identified and worked to the extent possible by the

5. Technical Reviews Reduce Risk (11.1) [2] • Technical Reviews reduce program risk and ease transition to production by: • Assessing the maturity of the design/development effort • Clarifying design requirements (at various stages) • Challenging the design and related processes (at various stages of completeness) • Checking the proposed design configuration against technical requirements, customer needs and system requirements • Evaluating the system configuration at different stages • Providing a forum for communications, coordination, and integration across all disciplines and IPTs • Establish a common configuration baseline from which to proceed to the next level of design • Recording design decision rational in the decision database

6. Successful Technical Reviews require extensive up-front planning: Visibility into Review preparation activities Tailoring to be consistent with program risk levels Establishing event-driven entry and exit criteria Identification & allocation of resources for TOTAL Review Effort Scheduling Conduct of incremental reviews (TIMs & IPT WG Meetings) Implementation by IPTs Review of all critical artifacts Confirmation of quality of all critical artifacts Develop Check Lists: Pre-review, review and post review activities Entry Criteria: To be Baselined Artifacts Source Artifacts Other Artifacts Exit Criteria Review (quality) criteria for each artifact (key questions) Key Databases: Baseline Document Database Decision Database Action Item Database Issues Database Risk Database Review Participants & Roles Planning for Technical Reviews (11.1) [3]

7. Planning for Technical Reviews (11.1) [4] • Key TR Activities: • Pre-Review: • Identify Participants • Establish Entry/ Exit Criteria • Detailed review of artifacts • Identification of Risks and Issues • Review: • Document Entry/Exit criteria met (or not) • Document Decisions • Document Actions (responsibility & due date) • Post-Review: • Track Actions Maintain a Technical Review Decision Database and Action Item Database

8. Generally Technical Reviews are conducted at the conclusion of each “V” level of development stage Both contractor and government must have common expectations regarding content and outcomes (hence need to document in SEP) Conducting Reviews: Reviews should be Event-Driven, not Schedule Driven* Conduct when product under development merits review Review should be a confirmation of completed effort Data required for Entry/Exit Criteria should be distributed, reviewed, analyzed, and coordinated prior to the Review Meeting Participants should include representation from all appropriate stakeholders: government, contractor, subcontractor, etc. Only designated participants should attend the Technical Review Meeting: Managers, IPT Leads (that developed the artifacts or are affected by them), and critical stakeholder representatives are participants Those that developed the artifacts under review (IPT members) and critical technical/subject matter experts generally are also participants Note: Not everyone involved in the review process is necessarily a participant Participants should complete coordinated review of artifacts prior to Meeting and have identified issues requiring management decision that could not be resolved No new issues should arise in the Meeting Decisions and Action Items should be documented and tracked Technical Reviews (11.2) [1] * Reality Check: Reviews must be scheduled, but EV credit for completion should not given until Exit Criteria are achieved (as actions if necessary)

9. Phasing Technical Reviews (11.2) [2] • Types of Technical Reviews reflect the “V” level of development stages: • Requirements • Design • Verification • Note correlation: • System Requirements = System Performance Spec = Functional Baseline • (Configuration) Item Design Requirements = Item Performance Spec = Allocated Baseline • Detailed Design (Requirements) = Item Detail/TDP = Product Specification • Note that the products are initially baselined but continue to be modified (under Configuration Control)

10. Technical Reviews (11.2) [3] • Types of Technical Reviews reflect the “V” level of development stages: • Requirements • Design • Verification • Note that these overlap at SFR

11. Technical Reviews and the Acquisition Life Cycle (11.2) [4] • Note Figure is OLD • FCA, SVR and PCA are now part of SDD Phase • MNS & ORD are replaced by ICD and CDD

12. Alternative Systems/Concept Review (ASR/ACR) Select preferred system concept Approve/Kick off acquisition System Requirements Review (SRR) Approve/Kick off start of project PMP/SEMP/TEMP Customer Requirements/Capabilities List Draft TPMs Draft Top-Level Integrated Architecture (Function & Physical) Draft System Spec System Design/ Definition/ Functional Review (SDR/SFR) Approve/Baseline System Spec/Functional Baseline Approve/Baseline High-level Integrated Architecture (Conceptual Design) End of Concept/Architecture Phase Software Specification Review (SSR): See PDR Preliminary Design Review (PDR) Approve/Baseline Performance Item Spec/Allocated Baseline Approve/Baseline Integrated Architecture (Preliminary Design) End of Requirements Phase Critical Design Review (CDR) Approve/Baseline Final Design (DI Specs) Note Design is >85% complete End of Design Phase Test Readiness Review (TRR) Approve/Baseline start of [integration] testing End of Development Phase Functional Configuration Audit/ System Verification Review (FCA/SVR) [1st unit acceptance] Verifies that PI Spec meets Customer Requirements Verifies that DI Spec meets PI Spec End of Testing Production Readiness/Approval Review (PRR/PAR) Approve start of unit production Physical Configuration Audit (PCA) Formalizes (corrected) Product Baseline for Production Follows PRR/PAR Summary of Technical Reviews and Audits (11.2) [5] Note: Names depend on what standard one is using (MS-1521B, DODI 5000.2, EIA/IS-632, IEEE P1220). Note: See SEF & DAG for more detailed description of each Review/Audit. Your SEPs should reflect this

13. Tailoring (11.3) • Process and Phases may be tailored • The more complex the system the less it should be skipped • Care should be taken, systems are often more complex than they appear • Names of technical reviews and audits may vary • Essential Techncial Reviews: • SFR or equivalent (Functional Baseline/ System Spec review) • PDR or equivalent (Allocated Baseline/ Item Spec review) • CDR or equivalent (Product Baseline/ Design review) • SVR or equivalent (product meets requirements)

14. Summary Points [11.4] • Each level of product development is evaluated and progress is controlled by Specification Development (System, Item Performance, Item Detail, Process and Material) and Technical Reviews (SRR, SFR (SDR/SSR), PDR, CDR, TRR, PRR, FCA, SVR, PCA) • Technical Reviews assess development maturity, risk, and cost/schedule to determine readiness to proceed • Technical Reviews must be planned, managed, and followed up to be effective • Technical Reviews and Audits follow the development process: • Early Technical Reviews focus on baselining the technical baseline documents (specifications, architecture, design, etc.) • Later Technical Reviews and Audits focus on verifying that the system meets those requirements and is ready for production