Larry Szutenbach OASN(RD&A) PP&R SZUTENBACH.LARRY@HQ.NAVY.MIL

1. The Application Of Risk In Metrics Used For Department of Navy Weapon Systems Acquisition(May 9, 2002) Larry Szutenbach OASN(RD&A) PP&RSZUTENBACH.LARRY@HQ.NAVY.MIL

2. This Presentation Will Briefly Touch on the Use of Risk in Our: • Program Summary Documents (PSDs) • Program Performance Metrics • Defense Acquisition Executive Summary (DAES) • Integrated Baseline Reviews • Consideration of Scorecards

3. Risk as used in Defense: • The potential for a • Two characteristics: • Probability of occurrence • Consequences • Use of risk in our metrics: • Often intentionally mix performance with risk, but it can be important to recognize the difference. • Sometimes risk is indirectly addressed, such as in the EVM “Bullseye” chart negative future reality.

4. Program Assessment Indicators Performance Characteristics Green Schedule Green Test & Evaluation Green Contracts Green Logistics Reqts & Readiness Obj Green Production Green Cost Green Management Structure Green Funding Green Interoperability Green Managers’ Assessment (for each Assessment Indicator rated Green Advisory, Yellow, Yellow Advisory, or Red). Program Assessments and PSDs Program Assessment Indicators/Program Manager’s Assessment(indicate Green/Yellow/Red, Program managers comments, an assessment based on current funding profile and current approved Acquisition Program Baseline) DAES Report • Schedule Performance. • Contracts. • Production. • Management Structure. • Performance Characteristics. •  Test and Evaluation. •  Logistics Requirements and Readiness Objectives.   • Cost Performance. • Funding.

5. ACAT III and IV Program Status Mr. Randel H. Stone Date of Review: 16 Nov 2000 PEO (W) ACAT III & I V ACAT PM CODE PROGRAM TITLE APB DATE COST SCH PERF OVERALL IIIPMA201 GBU-24 E/B OCT 99 III PMA208 IMPROVED TACTICAL AIR LAUNCHED DECOY DEC 99 III PMA242 HARM BLOCK VI (IHUP) JUL 99 III PMA281 AFLOAT PLANNING SYS (APS) SEP 99 III PMA281 JOINT SERV IMAGING PROC SYS NAVY (JSIPS) SEP 98 III PMA282 TACTICAL TOMAHAWK WEAPONS CONT SYS - III PMA282 ADV TOMAHAWK WPN CONTROL SYS (ATWCS) DEC 99 IVM PMA208 SUPERSONIC SEA SKIMMING TRGT (SSST) JUN 99 IVM PMA208 21ST CENTURY AERIAL TARGET (TARGET 21) DEC 99 IVM PMA208 BQM-74E MOBILE SUBSONIC AERIAL TGT DEC 99 IVM PMA208 VANDAL EXTENDED RANGE (EER) MAY 97 IVM PMA208 AQM-37C SUPERSONIC TARGET SYSTEM DEC 99 IVM PMA208 QF-4S FULL SCALE A/C TRGT SYS DEC 99 IVT PMA258 AN / AWW-13 ADVANCED DATA LINK POD AUG 91

6. Color Coded Rating System For DAES • And DoN PSDs and Program Performance Metrics • The DAESrating system [adapted for use in PSDs and the summary charts in program performance metrics] is intended to be used as an early warning report of both potential and actual problems. • In this regard, the system depends on the Program Manager exercising sound judgment in assessing the program’s status. • Rating an indicator as "on-track," solely because the APB might be "on-track" is counterproductive and leads to downstream problems. • If the problem is a potential one, the PM should clearly note this fact, so there is no doubt that this is an advisory and that the situation is being properly managed. • Early reporting of potential problems and that corrective action plans are underway is essential.

7. Color Coded Rating System For DAES, and • DoN PSDs and Program Performance Metrics Summary Charts • GREEN: • On-Track: All aspects of the program are progressing satisfactorily. Some minor problem(s) may exist, but solutions are available. Costs are expected not to exceed approved funding levels and not to exceed contract target costs by more than 5%. • YELLOW: • Potential or Actual Problem: Some event, action or delay has occurred that impairs progress against major objectives in one or more segments of the program. • In the case of a potential risk to a major program objective or acquisition program baseline, the Program Manager should state this distinction in Section 3 (Program Manager’s Comments). Early reporting is encouraged. • RED: • Major Weakness (Red): Some event, action, or delay has occurred that seriously impedes successful accomplishment of one or more major program objectives. • ADVISORY (GREEN, YELLOW OR RED): • Advisory indicates the program is either assessed to be between ratings, or is moving from one rating to another, thus this provides advance notification of shifting status. Mixing performance and risk Mixing performance and risk

8. A brief description of Issue # 2 and rationale for its rating. • Approach to remedy/mitigation • A brief description of Issue # 3 and rationale for its rating. • Approach to remedy/mitigation • A brief description of Issue # 1 and rationale for its rating. • Approach to remedy/mitigation • A brief description of Issue # 5 and rationale for its rating. • Approach to remedy/mitigation • A brief description of Issue # 4 and rationale for its rating. • Approach to remedy/mitigation • A brief description of Issue # 6 and rationale for its rating. • Approach to remedy/mitigation 5 High 4 Medium 3 Likelihood 2 Risk Advisory Board Mr. X, PMA xxx Ms. Y, XYZ Corp Ms. Z, DASN YY etc 1 Low 1 3 4 5 2 Consequence This is one of the most commonly used charts for depicting risk. Program Risk Assessment Shows the plan to manage risk Shows the risks

9. PEOXXX Low Risk Medium Risk High Risk Logistics Areas (examples) 5 5 4 : Overall Assessment 1: Training 2: Support Equipment 3: Publications 4: Facilities 5: Maintenance Concept 6: Supply Support 7: MTBF 4 7 Likelihood 3 6 2 3 1 1 2 1 3 4 5 2 Consequence RISK #5 Brief description of Issue and rationale for its rating. Approach to remedy/mitigation. Risk mitigation funding. RISK # 4 Brief description of Issue and rationale for its rating. Approach to remedy/mitigation. Risk mitigation funding. RISK # 6 Brief description of Issue and rationale for its rating. Approach to remedy/mitigation. Risk mitigation funding. ProgramAcronymACAT XX Logistics Risk Assessment CAPT J. Doe, PMS-499 Date of Review: dd mmm yy

10. Low Risk Medium Risk High Risk 5 5 4 4 7 3 6 Likelihood  2 3 1 1 2 1 3 4 5 2 Consequence  Definitions Of “Likelihood” And “Consequence” When Constructing This Chart Risk Assessment: Address each issue which might affect the success of the program. Likelihood • Negligible - One can reasonably assume no occurrence (<10%) • Unlikely - Occurrence possible but less than likely (10-40%) • Likely - significant chance of occurrence (40-65%) • Highly Probable - Very high chances of occurrence (65-90%) • Near Certainty - Assume and anticipate occurrence (>90%) Consequence • Marginal - Remedy will cause disruption to the program • Significant - Shorts a significant mission need • Serious - Shorts a critical mission need but expect no breech • Very Serious - Potentially fails a KPP or OPEVAL. • Catastrophic - Jeopardizes an exit criterion of current

11. G Solvency Programmed (%Obligated) Activity FY99 (%) FY00 (%) FY 01 FY 02 FY 03 FY 04 FY05 FY06 Display each significant task Budget Program Manager/Office______________ Date of Review_________ This addresses whether there enough money budgeted/planned over the course of the life of the program to accomplish the program goals and objectives

12. G Program Solvency This “Solvency” addresses how well the PM is able to cover the cost of the PROGRAM within funds already allocated to the program. The color ratings used generally follow these themes: GREEN-can handle within existing resources without descoping or affecting KPPs YELLOW-can resolve within existing budget by out-year transfer of funds between appropriations or descoping, and will request changing KPPs RED-shortfall in the current execution or next fiscal year with no available offsets or workarounds and will request Congressional approval be sought for a current year reprogramming; in short, program requires outside help.

13. ProgramAcronymACAT XX Y October 2000 Guidance – Example Master ChartContract Performance for [give short contract title] PEO and Program Manager/PMS-XXX Briefed: FEB 02 Solvency Nxxxxx-YY-Cxxxx Contractor Name [Prime or Significant Sub] [TCPIEAC = 0.76] CV = $2.0 M SV = $2.9 M 04/98 04/00 01/00 04/02 08/02 Total Calendar Schedule PM’s EAC $M 100% 108% 0 % 50% 42% 122% $110 EAC 1.18 Ahead of Schedule and Underspent $100 TAB 111% Behind Schedule and Underspent 1.14 100% $90 BAC 04/97 07/97 1.10 10/97 04/98 1.06 PM’s Projected Performance at Completion for CPI and Duration. 01/98 07/98 10/98 1/99 1.02 01/00 04/99 CPI 56% $50 02/00 07/99 0.98 04/00 ACWP 03/00 0.960 0.94 EV % Spent 10/99 01/00 0.90 Total Spent 0.86 0.940 Behind Schedule and Overspent Ahead of Schedule and Overspent 0% 0 0.82 KTR’s EAC:104M 0.82 0.86 0.90 0.94 0.98 1.02 1.06 1.10 1.14 1.18 1.18 SPI Date of Last Rebaselining: JAN00Number of Rebaselinings: 1Date of Next Rebaselining: MMM YY Date of Last Award Fee: MMM YYDate of Next Award Fee: MMM YY YYMMDD

14. Y Contract Solvency This “Solvency” addresses how well the PM is able to cover the cost of the CONTRACT within funds already allocated to the program. The color ratings used generally follow these themes: GREEN-can handle within existing resources without descoping or affecting KPPs YELLOW-can resolve within existing budget by out-year transfer of funds between appropriations or descoping, and changing KPPS will be considered. RED-shortfall in the current execution or next fiscal year with no available offsets or workarounds. the program office has no way to resolve the funding shortfall and needs assistance from the sponsor, DASN, ASN, PEO or others.

15. This Is Not a Metrics Chart, It Is an Illustration of Aggressive Bidding and A Changing Contract G Solvency OCT 00 Rebaselined/Renegotiated Contract - JAN 01 PM’s Est Cost $115 Contractor’s Est Cost $100 Y Contract Target Cost Solvency $112 PM Allocation for Contract $100 Y Solvency Original Contract - APR 00 [Same status at OCT 00] PM’s Est Cost $100 Contractor’s Est Cost $85 Contract Target Cost PM Allocation for Contract $68 $112 Contract performance is poor, but yields little budget risk Contract performance is poor, but yields little budget risk Contract performance is poor, but yields little budget risk Current Status APR 01 PM’S Est Cost Contractor’s Est Cost $109 $130 Contract Target Cost PM Allocation for Contract $112 $100

16. PEOXXX ProgramAcronymACAT XX CPARS/IPARS/Award Fee Matrix CAPT J. Doe, PMS-499 Date of Review: ddmmmyy At times, an interesting phenomenon occurs: Contract past performance cost ratings closely follow the “Solvency” rather than actual performance. At times, an interesting phenomenon occurs: Contract past performance cost ratings closely follow the “Solvency” rather than actual performance. At times, an interesting phenomenon occurs: Contract past performance cost ratings closely follow the “Solvency” rather than actual performance.

17. For Reference: Comparison of Color Ratings (We have attempted to normalize these as far as we can) Color Rating * PDUSD(A&T) CPARS/IPARS 24 AUG 1999 In DoN CPARS/IPARS Guidance Applies to Numerous Indicators DAES Not for release to contractors. Brings up issues within Defense. Applies to Numerous Indicators * ACAT I & II Program Contracts (EVM) 5 OCT 2000 Applies to Cost and Schedule Exceptional Exceed many requirements; resolve all problems. Blue NOT USED CPI and SPI > 1.15 and exceed many requirements, resolve all problems. Purple Very Good - Exceeds some requirements; some minor problems, which were corrected. NOT USED CPI and SPI > 1.1 Green Satisfactory Meets requirements; some minor problems, which were corrected “On-Track.” [CPI >0.95] Program progressing satisfactorily. Contract costs not expected to exceed contract target by > 5%, thus not an issue. CPI and SPI > 1.0 Yellow Marginal Does not meet some requirements; serious problem with no corrective action identified. CPI and SPI > 0.90 Potential or Actual Problem Some event has occurred that impairs progress against major objectives of the program. Red Unsatisfactory Does not meet most requirements and timely recovery unlikely. Major Weakness Some event seriously delaying major program objective; breach of program baseline or unit cost. CPI and SPI < 0.90

18. Risk Assessments In • Integrated Baseline Reviews • For Contracts • Managed At The Program Manager Level • No Consistent Form And Format For The IBRs. • Defense/Industry Developing Joint IBR Guide • Joint Strike Fighter IBR Offers A Good Example to Emulate

19. Extracts of Templates Used In The Integrated Baseline Review For The Joint Strike Fighter

20. WBS 1300 (Sample)WBS 1346 Fuel Sys Interview Summary Overall Assessment • The Team believes that this cost account can be successfully accomplished within cost and schedule at acceptable risk Dialogue Highlights • Concerns • Staffing availability • Hardware fabrication cycle with new vendors (different than Phase ll) • Communication between IPT’s - late data / data updates could cause redos • Risks • Resource availability/timing • Design maturity / hardware schedule • Strengths • good discussion; CAM well-prepared • comprehensive plan • linkages well defined • Weakness • Resource availability

21. IBR Discussion Evaluation Summary (Level 3 IPT)

22. Our Guidance in Defense for Risk • Systems Engineering risk management calls for risk: • Avoidance • Control • Assumption22 • Transfer • None of these imply an opportunity. • All of these focus on possible negative future occurrences. • The basic objective of risk management in systems engineering is to reduce all risk to an acceptable level.

23. Industry Takes Risk a Step Farther • Industry sees the potential • Compensaton of Industry managers is often tied to the risk of success, which is tracked in scorecards. • In DASN(PP&R) we are reviewing industry use of scorecards and seeing if we can reasonably attach our metrics to a scorecard appropriate for the government and reaching for the risk of success. success. of risk