Cost-Risk Connections Between CCRM CRM

1. Cost-Risk Connections Between CCRM & CRM David R. Graham OCFO/Cost Analysis Division HQs NASA

2. Purpose Describe and illustrate how CRM and CCRM are complementary Describe to cost community how CRM procedures and resource specialists can assist CCRM in determining cost-risk Describe to risk management community how CCRM cost-risk quantification can assist in assessing risk mitigation costs

3. Focus of Briefing How CCRM�s cost-risk quantification can augment CRM�s risk mitigation costing Not going to focus on: Early cost/performance trades (e.g., CAIV) even though both CRM and CCRM emphasize early assessment of risk in early Analysis of Alternatives trades Is really a sub-set of general cost-risk quantification Nor on cost-risk data collection/cost-risk database Will concentrate on risk identification, prioritization, assessment, quantification and management commonalities between CRM and CCRM

4. Continuous Cost-Risk ManagementA System of Cost Systems linked together in sequence by the same risks  In order to meet the space project cost challenges for the next decade and beyond, NASA cost management processes must evolve from traditional methods to modes that are truly transformational. The new focus for cost management at NASA will be Continuous Cost-Risk Management (CCRM). Additionally, the days of implementing cost management as a set of �stovepipe� activities are over. In reality, cost management is a series of related cost activities and involves three main Stages that are linked together through a shared set of project risks. Cost management, in effect, IS the management of cost-risk and can be characterized as continuous feedback on cost-risk. Feedback is essential to the transformation of cost management into a dynamic, continually reacting system where focused reporting of metrics on medium and high-risk drivers alert the project manager that a negative cost trend has been identified and requires action. The three Stages of Continuous Cost-Risk Management: Preparing for cost-risk feedback; Developing cost-risk feedback; and, Applying cost-risk feedback, occur at different points in time during an acquisition phase and involve the collaboration among cost estimators, project engineers, project managers, procurement analysts and Earned Value Management (EVM) specialists in managing the challenges presented by the risks. Cost management is NOT a grouping of unrelated stove-piped cost activities but is a "system of cost systems" based on viewing 12 cost activities normally treated as stovepipes as a continuum of activities interconnected through risk. This CCRM repeats in most acquisition phases. � The first Stage in Continuous Cost-Risk Management, Preparing for cost-risk feedback, involves NASA project teams performing three main activities: cost/performance trades (e.g., Cost as an Independent Variable (CAIV)); developing a definition of the program (e.g., part of the Cost Analysis Data Requirements (CADRe)) and, producing a range of possible costs (e.g., probability density function (PDF) and cumulative distribution function (CDF) or �S�-curve). CAIV trades flow out of a well-defined Concept of Operations (CONOPS) and demonstrate a commitment to evolutionary acquisition by being initiated in pre-Phase A for earliest implementation. These cost/performance trades are the first opportunity for representing the potential cost impacts due to risk. The CADRe will contain the definition of the project (analogous to the DoD Cost Analysis Requirements Description or CARD) for use by cost estimators where the traceability from Work Breakdown Structure (WBS) element, through functions, to initial requirements will be clearly identifiable. The cost range exemplified in a PDF and CDF involves developing a reference point cost estimate from a cost model (e.g., NAFCOM, PRICE, SEER, etc.), and incorporating cost model estimation, technical and correlation risk. Participants in the Preparing for cost-risk feedback Stage of the CCRM are mainly cost estimators, project engineers and project managers. This represents the starting point for cost-risk management. From this point forward the challenge will be in managing to the cost level chosen, no matter what cost-risk margin has been included. � The second Stage in Continuous Cost-Risk Management is Developing the feedback to manage the cost-risks. Since hardware contractors are selected to develop NASA systems, they must be informed about the potential cost-risk impacts identified by the NASA cost teams for their attention, monitoring, management, and reporting to the NASA project office. They must be informed in the Request for Proposal (RFP) Data Requests (DR) to produce multiple products that reflect the status and trends of these potential cost-risks. For example, as part of the CADRe, they will be required to produce a Life Cycle Cost Estimate (LCCE) for the proposal and LCCE updates at significant contract milestones (at least annually) as part of the contractual effort. The �S�-curve products of these requirements will enable the measurement of variance changes in the cost-risk distribution over time reflecting the management of risk and cost-risk. The CADRe will also require that initial key technical parameters, and changes to them over time, be documented along with actual costs associated with all WBS elements. These data will eventually populate the One NASA Cost Engineering (ONCE) database, keeping a record of project cost behavior for updating NASA cost models and available for cost analysis. Requirements to monitor, manage, and report monthly on the top medium and high-risk WBS elements identified during the Preparing Stage will be included. When required, a monthly earned value report will also be in the RFP, requiring performance measurement, variance analysis and corrective action reporting on all WBS elements, with a special focus on medium and high-risk WBS elements. These reports will require monthly Estimates at Completion (EAC) on all elements with a special focus on medium and high-risk WBS elements. Electronic access to these data will also be required. Other sources for monitoring and managing the top medium and high-risk WBS elements can come from Technical Performance Measure (TPM), Risk Management, Technical Interchange Meeting (TIM), Integrated Product Team (IPT) and Probability Risk Assessment reports. Based in part on the way bidders address these RFP requirements in the cost proposals submitted, NASA will select a winning bidder and set up a post-award meeting with the selected contractor to verify the proposed cost-risk management methods. If EVM is required on the effort, the meeting at which this discussion takes place (along with baseline validation) is called an Integrated Baseline Review (IBR). Participants in Developing cost-risk feedback are the cost estimators, project engineers, project managers, procurement analysts and EVM specialists. � The third Stage in cost-risk management is Applying the cost-risk feedback for managing costs. If the first two Stages in cost-risk management, Preparing for cost-risk feedback and Developing cost-risk feedback, have been properly accomplished, the cost-risk feedback from the EVM (or similar) system, supported by TPM reports, Risk Management Reports, TIM minutes, IPT meeting minutes, etc., will give the project manager the highest quality information possible for managing those WBS elements most likely to drive cost growth. The focus for reporting, analysis and action will be on medium and high-risk WBS elements since they were identified for specific reporting in the RFP and addressed by the winning contractor in his proposal. If cost and schedule performance analysis indicates problems, a decision to reiterate a cost/performance trade (part of the Preparing for cost-risk feedback Stage above) may have to be made, for a possible adjustment to a system requirement. EVM performance analysis, focused on risk impacts to cost and schedule, will enable development of monthly EACs providing the project manager crucial feedback on the potential cost effects of the risks. This information provides the project manager with focused insight into the cost-risk in order to better manage his/her costs. CADRe-required updates to the initial LCCE estimate at significant milestones (at least annually) can be analyzed for changes (hopefully reductions) in �S�-curve variances, indicating progress in managing risks and cost-risks. At the end of the effort a volume of high-quality cost, risk, and cost-risk information will have been collected that can be added to the ONCE database for follow-on contractor performance analysis, cost-risk methodology calibration and updating cost models in order to better cost estimate future projects. Participants in Applying cost-risk feedback are primarily project engineers, project managers and EVM specialists with cost estimator involvement during cost/performance trades (if required), and updating �S�-curves, databases and cost models. In order to meet the space project cost challenges for the next decade and beyond, NASA cost management processes must evolve from traditional methods to modes that are truly transformational. The new focus for cost management at NASA will be Continuous Cost-Risk Management (CCRM). Additionally, the days of implementing cost management as a set of �stovepipe� activities are over. In reality, cost management is a series of related cost activities and involves three main Stages that are linked together through a shared set of project risks. Cost management, in effect, IS the management of cost-risk and can be characterized as continuous feedback on cost-risk. Feedback is essential to the transformation of cost management into a dynamic, continually reacting system where focused reporting of metrics on medium and high-risk drivers alert the project manager that a negative cost trend has been identified and requires action. The three Stages of Continuous Cost-Risk Management: Preparing for cost-risk feedback; Developing cost-risk feedback; and, Applying cost-risk feedback, occur at different points in time during an acquisition phase and involve the collaboration among cost estimators, project engineers, project managers, procurement analysts and Earned Value Management (EVM) specialists in managing the challenges presented by the risks. Cost management is NOT a grouping of unrelated stove-piped cost activities but is a "system of cost systems" based on viewing 12 cost activities normally treated as stovepipes as a continuum of activities interconnected through risk. This CCRM repeats in most acquisition phases. � The first Stage in Continuous Cost-Risk Management, Preparing for cost-risk feedback, involves NASA project teams performing three main activities: cost/performance trades (e.g., Cost as an Independent Variable (CAIV)); developing a definition of the program (e.g., part of the Cost Analysis Data Requirements (CADRe)) and, producing a range of possible costs (e.g., probability density function (PDF) and cumulative distribution function (CDF) or �S�-curve). CAIV trades flow out of a well-defined Concept of Operations (CONOPS) and demonstrate a commitment to evolutionary acquisition by being initiated in pre-Phase A for earliest implementation. These cost/performance trades are the first opportunity for representing the potential cost impacts due to risk. The CADRe will contain the definition of the project (analogous to the DoD Cost Analysis Requirements Description or CARD) for use by cost estimators where the traceability from Work Breakdown Structure (WBS) element, through functions, to initial requirements will be clearly identifiable. The cost range exemplified in a PDF and CDF involves developing a reference point cost estimate from a cost model (e.g., NAFCOM, PRICE, SEER, etc.), and incorporating cost model estimation, technical and correlation risk. Participants in the Preparing for cost-risk feedback Stage of the CCRM are mainly cost estimators, project engineers and project managers. This represents the starting point for cost-risk management. From this point forward the challenge will be in managing to the cost level chosen, no matter what cost-risk margin has been included. � The second Stage in Continuous Cost-Risk Management is Developing the feedback to manage the cost-risks. Since hardware contractors are selected to develop NASA systems, they must be informed about the potential cost-risk impacts identified by the NASA cost teams for their attention, monitoring, management, and reporting to the NASA project office. They must be informed in the Request for Proposal (RFP) Data Requests (DR) to produce multiple products that reflect the status and trends of these potential cost-risks. For example, as part of the CADRe, they will be required to produce a Life Cycle Cost Estimate (LCCE) for the proposal and LCCE updates at significant contract milestones (at least annually) as part of the contractual effort. The �S�-curve products of these requirements will enable the measurement of variance changes in the cost-risk distribution over time reflecting the management of risk and cost-risk. The CADRe will also require that initial key technical parameters, and changes to them over time, be documented along with actual costs associated with all WBS elements. These data will eventually populate the One NASA Cost Engineering (ONCE) database, keeping a record of project cost behavior for updating NASA cost models and available for cost analysis. Requirements to monitor, manage, and report monthly on the top medium and high-risk WBS elements identified during the Preparing Stage will be included. When required, a monthly earned value report will also be in the RFP, requiring performance measurement, variance analysis and corrective action reporting on all WBS elements, with a special focus on medium and high-risk WBS elements. These reports will require monthly Estimates at Completion (EAC) on all elements with a special focus on medium and high-risk WBS elements. Electronic access to these data will also be required. Other sources for monitoring and managing the top medium and high-risk WBS elements can come from Technical Performance Measure (TPM), Risk Management, Technical Interchange Meeting (TIM), Integrated Product Team (IPT) and Probability Risk Assessment reports. Based in part on the way bidders address these RFP requirements in the cost proposals submitted, NASA will select a winning bidder and set up a post-award meeting with the selected contractor to verify the proposed cost-risk management methods. If EVM is required on the effort, the meeting at which this discussion takes place (along with baseline validation) is called an Integrated Baseline Review (IBR). Participants in Developing cost-risk feedback are the cost estimators, project engineers, project managers, procurement analysts and EVM specialists. � The third Stage in cost-risk management is Applying the cost-risk feedback for managing costs. If the first two Stages in cost-risk management, Preparing for cost-risk feedback and Developing cost-risk feedback, have been properly accomplished, the cost-risk feedback from the EVM (or similar) system, supported by TPM reports, Risk Management Reports, TIM minutes, IPT meeting minutes, etc., will give the project manager the highest quality information possible for managing those WBS elements most likely to drive cost growth. The focus for reporting, analysis and action will be on medium and high-risk WBS elements since they were identified for specific reporting in the RFP and addressed by the winning contractor in his proposal. If cost and schedule performance analysis indicates problems, a decision to reiterate a cost/performance trade (part of the Preparing for cost-risk feedback Stage above) may have to be made, for a possible adjustment to a system requirement. EVM performance analysis, focused on risk impacts to cost and schedule, will enable development of monthly EACs providing the project manager crucial feedback on the potential cost effects of the risks. This information provides the project manager with focused insight into the cost-risk in order to better manage his/her costs. CADRe-required updates to the initial LCCE estimate at significant milestones (at least annually) can be analyzed for changes (hopefully reductions) in �S�-curve variances, indicating progress in managing risks and cost-risks. At the end of the effort a volume of high-quality cost, risk, and cost-risk information will have been collected that can be added to the ONCE database for follow-on contractor performance analysis, cost-risk methodology calibration and updating cost models in order to better cost estimate future projects. Participants in Applying cost-risk feedback are primarily project engineers, project managers and EVM specialists with cost estimator involvement during cost/performance trades (if required), and updating �S�-curves, databases and cost models.

5. Continuous Cost-Risk Management (CCRM) A cost management architecture providing: Identification of medium and high risk WBS elements, their assessment & translation of risk into cost-risk in LCCEs Supports adequate budget for project Communication of identified medium and high risk WBS elements to project managers (contractor or NASA) Post-cost estimate tracking of medium and high risk WBS element cost and schedule performance Application of EVM system Produces early warning of potential cost and schedule problems Enables actionable intelligence for timely mitigation/management Updates of technical and cost data (including annual LCCEs) History of cost and technical data for use in updating cost models

7. CRM Definition (7120.5C) Continuous Risk Management (CRM). The process that identifies risk; analyzes their impact and prioritizes them; develops and carries out plans for risk mitigation or acceptance; tracks risk and the implementation of plans; supports informed, timely, and effective decisions to control risks and mitigation plans; and assures that risk information is communicated and documented.

8. Cost-Risk Identification & Assessment CRM

9. CRM Risk Identification

10. Analyze

11. Likelihood of Occurrence

12. Categorizing Risks Primary Purpose of Slide: Key Points: Primary Purpose of Slide: Key Points:

13. Cost-Risk Identification & Assessment CCRM

15. Cost-Risk Assessment in CCRM Assessment areas Cost model uncertainty Cost estimators handle this Input parameter uncertainty Engineering and CRM assessment needed Indigenous/Programmatic uncertainty Engineering and CRM assessment needed for application of Relative Risk Weighting (RRW) 3 WBS element risk profiles (pessimistic, optimistic & reference) in terms of their Key Engineering Performance Parameters (KEPPs) Correlation uncertainty Engineering/CRM/cost estimator assessment needed

16. KEPPs as Discrete Risks A KEPP is a technical or operational parameter that can be described as a requirement An advantage to defining any risk profile in terms of KEPPs is that a more discrete picture of a WBS risk emerges e.g., The radiation resistance for a power subsystem�s ASIC has never been designed for this level of tolerance so there�s likely to be engineering challenges This discrete picture is more intuitively attractive to a project manager than vague statistical notions of risk These profiles document the statistical representations of the risks behind the �S�-curve

17. Key Engineering Performance Parameters1 (KEPP) Examples KEPPs for new electronic component for a S/C Dynamic load resistance Operating voltage Power regulation Radiation resistance Emissivity Component mass Operating temperature range Operating efficiency KEPPs for a Laser/Amplifier Transmitter Wave front sensing Wave generation Mirror coatings and gratings Autonomous resonator alignment Bore sighting Electrical power generation

18. CORRELATION What is Correlation?2 A measure of association between two variables It measures how strongly the variables are related, or change, with each other Engineers and CRM specialists can assist cost estimators in identifying and quantifying correlation between WBS elements Correlation should be understood by all cost analysts performing quantitative cost risk analysis. So what is correlation? First, it is a measure of association between two variables. It measures how strongly these variables are related, or change, with each other. If two variables tend to move up or down together, they are said to be positively correlated. If they tend to move in opposite directions, they are said to be negatively correlated. The most common statistic for measuring Association is the Pearson correlation coefficient. Correlation should be understood by all cost analysts performing quantitative cost risk analysis. So what is correlation? First, it is a measure of association between two variables. It measures how strongly these variables are related, or change, with each other. If two variables tend to move up or down together, they are said to be positively correlated. If they tend to move in opposite directions, they are said to be negatively correlated. The most common statistic for measuring Association is the Pearson correlation coefficient.

19. Cost-Risk Quantification CRM

20. CRM Cost-Risk Quantification Limited to grassroots/bottoms-up valuation of specific, discrete risk mitigation costs Advantage is that defining discrete risks and identifying probabilities for their occurrence is intuitively attractive to the project manager Can understand in more concrete terms what she will be getting for her risk dollars What her risk dollars are being spent on Similar to risk profiles in RRW

21. Cost-Risk Quantification CCRM

22. Cost Model and Input Parameter Uncertainty Cost Quantification

23. Indigenous/Programmatic Uncertainty Quantification Using RRW Since, in most cases, the best information we have is low, most likely and high estimates (or can credibly develop), a triangular distribution is used (bottom of chart). The development of the low and high ends of the triangle are the result of factors applied to the government point cost estimate (GPE). A risk category matrix (top of chart) is developed utilizing risk categories and weighted Applying the Analytical Hierarchy Process (AHP), a purely mathematical technique for generating valid, ratio-level (vice ordinal-level) weights. The AHP is also used to weight the Very Low to Very High scales. Three profiles of the WBS element are rated against the weighted risk categories Applying the weighted scales and three risk �scores� are generated representing how risky each is perceived to be by the raters, mostly engineers. Two ratios are developed from these three risk scores and used as factors on the point cost estimate (assumed to be the most likely in the triangle) to generate the low and high ends of the triangular distribution. Since the CARD specifications are rated for the Reference Profile and result in the Reference Profile risk score and the CARD specifications result in the Reference Point cost estimate for that WBS element, there is an implied equivalency between the Reference Profile risk score and the Reference Point cost estimate. In other words, the Reference Profile risk score represents the WBS in qualitative, technical risk form. The Reference Point cost estimate represents the WBS in cost form. It is this equivalency that justifies the application of the ratios to the point cost estimate that generates the upper and lower bounds of the cost-risk triangle. Since, in most cases, the best information we have is low, most likely and high estimates (or can credibly develop), a triangular distribution is used (bottom of chart). The development of the low and high ends of the triangle are the result of factors applied to the government point cost estimate (GPE). A risk category matrix (top of chart) is developed utilizing risk categories and weighted Applying the Analytical Hierarchy Process (AHP), a purely mathematical technique for generating valid, ratio-level (vice ordinal-level) weights. The AHP is also used to weight the Very Low to Very High scales. Three profiles of the WBS element are rated against the weighted risk categories Applying the weighted scales and three risk �scores� are generated representing how risky each is perceived to be by the raters, mostly engineers. Two ratios are developed from these three risk scores and used as factors on the point cost estimate (assumed to be the most likely in the triangle) to generate the low and high ends of the triangular distribution. Since the CARD specifications are rated for the Reference Profile and result in the Reference Profile risk score and the CARD specifications result in the Reference Point cost estimate for that WBS element, there is an implied equivalency between the Reference Profile risk score and the Reference Point cost estimate. In other words, the Reference Profile risk score represents the WBS in qualitative, technical risk form. The Reference Point cost estimate represents the WBS in cost form. It is this equivalency that justifies the application of the ratios to the point cost estimate that generates the upper and lower bounds of the cost-risk triangle.

24. Correlation Dr. Stephen Book (MCR) plotted the theoretical underestimation of percent total cost standard deviation (y-axis) when correlation (x-axis) is assumed to be zero rather than its true value, r. In cost estimates we would underestimate % SD ~60%-80% @ 0.2 actual correlation This chart shows the percent underestimation of total cost sigma when correlation is ignored, rather than Applying its true value. Note that there is a knee in the curve for each line. Each line represents a different number of WBS elements being summed. As this number increases, the actual correlation value where the knee in the curve appears becomes smaller.This chart shows the percent underestimation of total cost sigma when correlation is ignored, rather than Applying its true value. Note that there is a knee in the curve for each line. Each line represents a different number of WBS elements being summed. As this number increases, the actual correlation value where the knee in the curve appears becomes smaller.

25. CONVOLVE ALL COST-RISK DISTRIBUTIONS IN MONTE CARLO The monte carlo simulation process results in a summary Probability Density Function (PDF) and a summary Cumulative Distribution Function (CDF) from which costs are easily associated with confidence levels visually. It is these �S�-curves that are generated at the proposal, mid-phase and end-of-phase by the contractors to indicate progress (hopefully) in increasing confidence around the �Aggressive Cost Objective�. If the government has done an analysis itself, then during source selection its �S�-curve can be used to compare with the offerors�.The monte carlo simulation process results in a summary Probability Density Function (PDF) and a summary Cumulative Distribution Function (CDF) from which costs are easily associated with confidence levels visually. It is these �S�-curves that are generated at the proposal, mid-phase and end-of-phase by the contractors to indicate progress (hopefully) in increasing confidence around the �Aggressive Cost Objective�. If the government has done an analysis itself, then during source selection its �S�-curve can be used to compare with the offerors�.

26. Cost-Risk Management in CRM

27. Track Acquire Data Determine what information is required Obtain the data Compile Data Organize data into understandable information Develop trends Format data in a form that is consistent with what is being used on other projects, and that the reviewers are use to seeing Report Data Communicate the compiled data to the Risk Management Board Make recommendations on the status of each risk and any modifications that may be required

28. Control

29. Cost-Risk Management CCRM

30. CPR Data Requirements Description For cost-risk feedback, the contractor or performing organization needs to be informed in the RFP/Project Plan about: Medium and high-risk systems, subsystems and/or WBS elements identified initially in the cost estimate EVM performance measurement requirements against these specific risky WBS elements e.g., WBS element reporting levels (NPR 7120.5C) An EVM CPR DRD template is available on the Cost Estimating Handbook website www.ceh.nasa.gov

31. ACWP Actual Cost of Work Performed BCWS Budgeted Cost of Work Scheduled BCWP Budgeted Cost of Work Performed CDRL Contract Data Requirements List SPO System Program Office WBS Work Breakdown Structure ACWP Actual Cost of Work Performed BCWS Budgeted Cost of Work Scheduled BCWP Budgeted Cost of Work Performed CDRL Contract Data Requirements List SPO System Program Office WBS Work Breakdown Structure

32. Traditional Level 3 Reporting10% Variance Reporting ACWP Actual Cost of Work Performed ADACS Attitude Determination and Control System AKM Apogee Kick Motor BAC Budget at Completion BCWP Budgeted Cost for Work Performed BCWS Budgeted Cost for Work Scheduled EAC Estimate at Completion EPS Electrical Power System OBS Organization Breakdown Structure RWA Reaction Wheel Assembly TT&C Telemetry, Tracking and Control VP/GM Vice President/General Manager WBS Work Breakdown Structure Control Account is the logical intersection of the organizational breakdown stricture (who can do the job) with the work breakdown structure (what needs to be done). ACWP Actual Cost of Work Performed ADACS Attitude Determination and Control System AKM Apogee Kick Motor BAC Budget at Completion BCWP Budgeted Cost for Work Performed BCWS Budgeted Cost for Work Scheduled EAC Estimate at Completion EPS Electrical Power System OBS Organization Breakdown Structure RWA Reaction Wheel Assembly TT&C Telemetry, Tracking and Control VP/GM Vice President/General Manager WBS Work Breakdown Structure Control Account is the logical intersection of the organizational breakdown stricture (who can do the job) with the work breakdown structure (what needs to be done).

33. CCRM Med/High Risk ReportingHigh-Risk No-Threshold Variance Reporting3 ACWP Actual Cost of Work Performed ADACS Attitude Determination and Control System AKM Apogee Kick Motor BAC Budget at Completion BCWP Budgeted Cost for Work Performed BCWS Budgeted Cost for Work Scheduled EAC Estimate at Completion EPS Electrical Power System OBS Organization Breakdown Structure RWA Reaction Wheel Assembly TT&C Telemetry, Tracking and Control VP/GM Vice President/General Manager WBS Work Breakdown Structure Control Account is the logical intersection of the organizational breakdown stricture (who can do the job) with the work breakdown structure (what needs to be done). ACWP Actual Cost of Work Performed ADACS Attitude Determination and Control System AKM Apogee Kick Motor BAC Budget at Completion BCWP Budgeted Cost for Work Performed BCWS Budgeted Cost for Work Scheduled EAC Estimate at Completion EPS Electrical Power System OBS Organization Breakdown Structure RWA Reaction Wheel Assembly TT&C Telemetry, Tracking and Control VP/GM Vice President/General Manager WBS Work Breakdown Structure Control Account is the logical intersection of the organizational breakdown stricture (who can do the job) with the work breakdown structure (what needs to be done).

34. CCRM Cost-Risk Management CPR will deliver key performance measurement data on medium and high risk WBS elements monthly This will enable the project managers to determine actions to take to mitigate potential problems The NASA project manager works closely with the in-house NASA control account managers (CAMs) and contractor CAMs to determine what mitigation actions to take The NASA project manager works with both in-house and contactor CAMs to determine if performance measurement can be dropped on previously risky WBS elements due to risk retirement

35. Conclusion Focused only on cost-risk between CRM and CCRM There are other areas CRM and CCRM have in common for future discussions Cost/performance trades; requirements allocation to functions and WBS elements; risk data collection; risk databases Bottom line: CRM and CCRM are complementary processes CCRM needs CRM input CCRM can provide CRM with valuable cost-risk quantification

36. BACKUPS

37. CRM Risk Identification Clearly define objectives (everyone understands) Brainstorm issues/concerns to meeting objectives (what can go wrong) Decide which issues/concerns are real Develop risk statement from each issue/concern Assign attributes to the risk Brainstorm potential mitigations Be sure to follow rules for brainstorming

38. CRM Risk Identification Tools Some tools to help identify and manage Risk: Brainstorming (identifying issues) Decision Analysis (mapping decision paths) Lessons Learned Personal knowledge and experience (gut feeling) Expert Interviews and Independent Assessments Questionnaires. Topic or taxonomy based WBS, EVM, budgets and schedules FTA, FMEA�s, PRA, Monte Carlo ( reliability tools) Safety / Hazard analysis These tools also help identify problems Tools help identify risks as well as problems For further information on these and other risk identification tools, please refer to the Continuous Risk Management Guidebook, NASA System Engineering handbook (sp-6105 section 4), and the S.E tool box ****** Tools help identify risks as well as problems For further information on these and other risk identification tools, please refer to the Continuous Risk Management Guidebook, NASA System Engineering handbook (sp-6105 section 4), and the S.E tool box ******

39. Evaluate Risks at the Appropriate Level

41. CORRELATION What is Correlation?2 A measure of association between two variables. It measures how strongly the variables are related, or change, with each other. If two variables tend to move up or down together, they are said to be positively correlated. If they tend to move in opposite directions, they are said to be negatively correlated. The most common statistic for measuring association is the Pearson correlation coefficient, rP. Engineers and CRM specialists can assist cost estimators in identifying and quantifying correlation between WBS elements Correlation should be understood by all cost analysts performing quantitative cost risk analysis. So what is correlation? First, it is a measure of association between two variables. It measures how strongly these variables are related, or change, with each other. If two variables tend to move up or down together, they are said to be positively correlated. If they tend to move in opposite directions, they are said to be negatively correlated. The most common statistic for measuring Association is the Pearson correlation coefficient. Correlation should be understood by all cost analysts performing quantitative cost risk analysis. So what is correlation? First, it is a measure of association between two variables. It measures how strongly these variables are related, or change, with each other. If two variables tend to move up or down together, they are said to be positively correlated. If they tend to move in opposite directions, they are said to be negatively correlated. The most common statistic for measuring Association is the Pearson correlation coefficient.

42. Indigenous/Programmatic Uncertainty QuantificationRelative Risk Weighting Task 1: Identify and prioritize cost-risk drivers and intensity scales Task 2: Define pessimistic, reference, and optimistic risk profiles for each WBS element and �score� them against the prioritized cost-risk drivers Applying the intensity scales Task 3: Create two ratios {pessimistic/reference} and {optimistic/reference} from profile score ratios to scale the reference point cost estimate to create pessimistic and optimistic cost estimates Task 4: Use results as cost-risk triangular distribution endpoints RRW Relative Risk Weighting WBS Work Breakdown StructureRRW Relative Risk Weighting WBS Work Breakdown Structure

43. How the RRW Works CARD Cost Analysis Requirements Description RPE Reference Point Estimate RRW Relative Risk WeightingCARD Cost Analysis Requirements Description RPE Reference Point Estimate RRW Relative Risk Weighting

44. Track

45. Communicating Cost & Schedule RequirementsCCRM Step 6 CARD Cost Analysis Requirements Description CPR Cost Performance Report EVM Earned Value Management IBR Integrated Baseline Review LCCE Life Cycle Cost Estimate RFP Request for Proposal SPO System Program Office V&V Verification and Validation WBS Work Breakdown Structure �CARD Cost Analysis Requirements Description CPR Cost Performance Report EVM Earned Value Management IBR Integrated Baseline Review LCCE Life Cycle Cost Estimate RFP Request for Proposal SPO System Program Office V&V Verification and Validation WBS Work Breakdown Structure �

46. Continuous Cost-Risk ManagementSystem of Cost Systems linked together in sequence by the same risks  In order to meet the space project cost challenges for the next decade and beyond, NASA cost management processes must evolve from traditional methods to modes that are truly transformational. The new focus for cost management at NASA will be Continuous Cost-Risk Management (CCRM). Additionally, the days of implementing cost management as a set of �stovepipe� activities are over. In reality, cost management is a series of related cost activities and involves three main Stages that are linked together through a shared set of project risks. Cost management, in effect, IS the management of cost-risk and can be characterized as continuous feedback on cost-risk. Feedback is essential to the transformation of cost management into a dynamic, continually reacting system where focused reporting of metrics on medium and high-risk drivers alert the project manager that a negative cost trend has been identified and requires action. The three Stages of Continuous Cost-Risk Management: Preparing for cost-risk feedback; Developing cost-risk feedback; and, Applying cost-risk feedback, occur at different points in time during an acquisition phase and involve the collaboration among cost estimators, project engineers, project managers, procurement analysts and Earned Value Management (EVM) specialists in managing the challenges presented by the risks. Cost management is NOT a grouping of unrelated stove-piped cost activities but is a "system of cost systems" based on viewing 12 cost activities normally treated as stovepipes as a continuum of activities interconnected through risk. This CCRM repeats in most acquisition phases. � The first Stage in Continuous Cost-Risk Management, Preparing for cost-risk feedback, involves NASA project teams performing three main activities: cost/performance trades (e.g., Cost as an Independent Variable (CAIV)); developing a definition of the program (e.g., part of the Cost Analysis Data Requirements (CADRe)) and, producing a range of possible costs (e.g., probability density function (PDF) and cumulative distribution function (CDF) or �S�-curve). CAIV trades flow out of a well-defined Concept of Operations (CONOPS) and demonstrate a commitment to evolutionary acquisition by being initiated in pre-Phase A for earliest implementation. These cost/performance trades are the first opportunity for representing the potential cost impacts due to risk. The CADRe will contain the definition of the project (analogous to the DoD Cost Analysis Requirements Description or CARD) for use by cost estimators where the traceability from Work Breakdown Structure (WBS) element, through functions, to initial requirements will be clearly identifiable. The cost range exemplified in a PDF and CDF involves developing a reference point cost estimate from a cost model (e.g., NAFCOM, PRICE, SEER, etc.), and incorporating cost model estimation, technical and correlation risk. Participants in the Preparing for cost-risk feedback Stage of the CCRM are mainly cost estimators, project engineers and project managers. This represents the starting point for cost-risk management. From this point forward the challenge will be in managing to the cost level chosen, no matter what cost-risk margin has been included. � The second Stage in Continuous Cost-Risk Management is Developing the feedback to manage the cost-risks. Since hardware contractors are selected to develop NASA systems, they must be informed about the potential cost-risk impacts identified by the NASA cost teams for their attention, monitoring, management, and reporting to the NASA project office. They must be informed in the Request for Proposal (RFP) Data Requests (DR) to produce multiple products that reflect the status and trends of these potential cost-risks. For example, as part of the CADRe, they will be required to produce a Life Cycle Cost Estimate (LCCE) for the proposal and LCCE updates at significant contract milestones (at least annually) as part of the contractual effort. The �S�-curve products of these requirements will enable the measurement of variance changes in the cost-risk distribution over time reflecting the management of risk and cost-risk. The CADRe will also require that initial key technical parameters, and changes to them over time, be documented along with actual costs associated with all WBS elements. These data will eventually populate the One NASA Cost Engineering (ONCE) database, keeping a record of project cost behavior for updating NASA cost models and available for cost analysis. Requirements to monitor, manage, and report monthly on the top medium and high-risk WBS elements identified during the Preparing Stage will be included. When required, a monthly earned value report will also be in the RFP, requiring performance measurement, variance analysis and corrective action reporting on all WBS elements, with a special focus on medium and high-risk WBS elements. These reports will require monthly Estimates at Completion (EAC) on all elements with a special focus on medium and high-risk WBS elements. Electronic access to these data will also be required. Other sources for monitoring and managing the top medium and high-risk WBS elements can come from Technical Performance Measure (TPM), Risk Management, Technical Interchange Meeting (TIM), Integrated Product Team (IPT) and Probability Risk Assessment reports. Based in part on the way bidders address these RFP requirements in the cost proposals submitted, NASA will select a winning bidder and set up a post-award meeting with the selected contractor to verify the proposed cost-risk management methods. If EVM is required on the effort, the meeting at which this discussion takes place (along with baseline validation) is called an Integrated Baseline Review (IBR). Participants in Developing cost-risk feedback are the cost estimators, project engineers, project managers, procurement analysts and EVM specialists. � The third Stage in cost-risk management is Applying the cost-risk feedback for managing costs. If the first two Stages in cost-risk management, Preparing for cost-risk feedback and Developing cost-risk feedback, have been properly accomplished, the cost-risk feedback from the EVM (or similar) system, supported by TPM reports, Risk Management Reports, TIM minutes, IPT meeting minutes, etc., will give the project manager the highest quality information possible for managing those WBS elements most likely to drive cost growth. The focus for reporting, analysis and action will be on medium and high-risk WBS elements since they were identified for specific reporting in the RFP and addressed by the winning contractor in his proposal. If cost and schedule performance analysis indicates problems, a decision to reiterate a cost/performance trade (part of the Preparing for cost-risk feedback Stage above) may have to be made, for a possible adjustment to a system requirement. EVM performance analysis, focused on risk impacts to cost and schedule, will enable development of monthly EACs providing the project manager crucial feedback on the potential cost effects of the risks. This information provides the project manager with focused insight into the cost-risk in order to better manage his/her costs. CADRe-required updates to the initial LCCE estimate at significant milestones (at least annually) can be analyzed for changes (hopefully reductions) in �S�-curve variances, indicating progress in managing risks and cost-risks. At the end of the effort a volume of high-quality cost, risk, and cost-risk information will have been collected that can be added to the ONCE database for follow-on contractor performance analysis, cost-risk methodology calibration and updating cost models in order to better cost estimate future projects. Participants in Applying cost-risk feedback are primarily project engineers, project managers and EVM specialists with cost estimator involvement during cost/performance trades (if required), and updating �S�-curves, databases and cost models. In order to meet the space project cost challenges for the next decade and beyond, NASA cost management processes must evolve from traditional methods to modes that are truly transformational. The new focus for cost management at NASA will be Continuous Cost-Risk Management (CCRM). Additionally, the days of implementing cost management as a set of �stovepipe� activities are over. In reality, cost management is a series of related cost activities and involves three main Stages that are linked together through a shared set of project risks. Cost management, in effect, IS the management of cost-risk and can be characterized as continuous feedback on cost-risk. Feedback is essential to the transformation of cost management into a dynamic, continually reacting system where focused reporting of metrics on medium and high-risk drivers alert the project manager that a negative cost trend has been identified and requires action. The three Stages of Continuous Cost-Risk Management: Preparing for cost-risk feedback; Developing cost-risk feedback; and, Applying cost-risk feedback, occur at different points in time during an acquisition phase and involve the collaboration among cost estimators, project engineers, project managers, procurement analysts and Earned Value Management (EVM) specialists in managing the challenges presented by the risks. Cost management is NOT a grouping of unrelated stove-piped cost activities but is a "system of cost systems" based on viewing 12 cost activities normally treated as stovepipes as a continuum of activities interconnected through risk. This CCRM repeats in most acquisition phases. � The first Stage in Continuous Cost-Risk Management, Preparing for cost-risk feedback, involves NASA project teams performing three main activities: cost/performance trades (e.g., Cost as an Independent Variable (CAIV)); developing a definition of the program (e.g., part of the Cost Analysis Data Requirements (CADRe)) and, producing a range of possible costs (e.g., probability density function (PDF) and cumulative distribution function (CDF) or �S�-curve). CAIV trades flow out of a well-defined Concept of Operations (CONOPS) and demonstrate a commitment to evolutionary acquisition by being initiated in pre-Phase A for earliest implementation. These cost/performance trades are the first opportunity for representing the potential cost impacts due to risk. The CADRe will contain the definition of the project (analogous to the DoD Cost Analysis Requirements Description or CARD) for use by cost estimators where the traceability from Work Breakdown Structure (WBS) element, through functions, to initial requirements will be clearly identifiable. The cost range exemplified in a PDF and CDF involves developing a reference point cost estimate from a cost model (e.g., NAFCOM, PRICE, SEER, etc.), and incorporating cost model estimation, technical and correlation risk. Participants in the Preparing for cost-risk feedback Stage of the CCRM are mainly cost estimators, project engineers and project managers. This represents the starting point for cost-risk management. From this point forward the challenge will be in managing to the cost level chosen, no matter what cost-risk margin has been included. � The second Stage in Continuous Cost-Risk Management is Developing the feedback to manage the cost-risks. Since hardware contractors are selected to develop NASA systems, they must be informed about the potential cost-risk impacts identified by the NASA cost teams for their attention, monitoring, management, and reporting to the NASA project office. They must be informed in the Request for Proposal (RFP) Data Requests (DR) to produce multiple products that reflect the status and trends of these potential cost-risks. For example, as part of the CADRe, they will be required to produce a Life Cycle Cost Estimate (LCCE) for the proposal and LCCE updates at significant contract milestones (at least annually) as part of the contractual effort. The �S�-curve products of these requirements will enable the measurement of variance changes in the cost-risk distribution over time reflecting the management of risk and cost-risk. The CADRe will also require that initial key technical parameters, and changes to them over time, be documented along with actual costs associated with all WBS elements. These data will eventually populate the One NASA Cost Engineering (ONCE) database, keeping a record of project cost behavior for updating NASA cost models and available for cost analysis. Requirements to monitor, manage, and report monthly on the top medium and high-risk WBS elements identified during the Preparing Stage will be included. When required, a monthly earned value report will also be in the RFP, requiring performance measurement, variance analysis and corrective action reporting on all WBS elements, with a special focus on medium and high-risk WBS elements. These reports will require monthly Estimates at Completion (EAC) on all elements with a special focus on medium and high-risk WBS elements. Electronic access to these data will also be required. Other sources for monitoring and managing the top medium and high-risk WBS elements can come from Technical Performance Measure (TPM), Risk Management, Technical Interchange Meeting (TIM), Integrated Product Team (IPT) and Probability Risk Assessment reports. Based in part on the way bidders address these RFP requirements in the cost proposals submitted, NASA will select a winning bidder and set up a post-award meeting with the selected contractor to verify the proposed cost-risk management methods. If EVM is required on the effort, the meeting at which this discussion takes place (along with baseline validation) is called an Integrated Baseline Review (IBR). Participants in Developing cost-risk feedback are the cost estimators, project engineers, project managers, procurement analysts and EVM specialists. � The third Stage in cost-risk management is Applying the cost-risk feedback for managing costs. If the first two Stages in cost-risk management, Preparing for cost-risk feedback and Developing cost-risk feedback, have been properly accomplished, the cost-risk feedback from the EVM (or similar) system, supported by TPM reports, Risk Management Reports, TIM minutes, IPT meeting minutes, etc., will give the project manager the highest quality information possible for managing those WBS elements most likely to drive cost growth. The focus for reporting, analysis and action will be on medium and high-risk WBS elements since they were identified for specific reporting in the RFP and addressed by the winning contractor in his proposal. If cost and schedule performance analysis indicates problems, a decision to reiterate a cost/performance trade (part of the Preparing for cost-risk feedback Stage above) may have to be made, for a possible adjustment to a system requirement. EVM performance analysis, focused on risk impacts to cost and schedule, will enable development of monthly EACs providing the project manager crucial feedback on the potential cost effects of the risks. This information provides the project manager with focused insight into the cost-risk in order to better manage his/her costs. CADRe-required updates to the initial LCCE estimate at significant milestones (at least annually) can be analyzed for changes (hopefully reductions) in �S�-curve variances, indicating progress in managing risks and cost-risks. At the end of the effort a volume of high-quality cost, risk, and cost-risk information will have been collected that can be added to the ONCE database for follow-on contractor performance analysis, cost-risk methodology calibration and updating cost models in order to better cost estimate future projects. Participants in Applying cost-risk feedback are primarily project engineers, project managers and EVM specialists with cost estimator involvement during cost/performance trades (if required), and updating �S�-curves, databases and cost models.

47. Cost-Risk Assessment CRM

48. Cost-Risk Assessment CCRM

49. Cost Model Uncertainty Due to the �spread� of the underlying CER or analogy data, estimating error must be accounted for: Type of distribution is dependent on statistical data available

50. Input Parameter Uncertainty Due to uncertainty about the deterministic nature of an estimated CER-driving parameter value, its indeterminate nature must be accounted for: e.g., mass, power, volume, S/W productivity, etc. Type of distribution is dependent on statistical data available

51. Define WBS Element Risk Profiles(In Writing) �Pessimistic� A situation surrounding the development of the WBS element�s Key Engineering Performance Parameters (KEPPs) that assumes the realization of the worst conditions under each category of risk affecting the element in meeting the WBS performance expectations documented in Parts 1 & 2 of the CADRe �Optimistic� A situation surrounding the development of the WBS element�s KEPPs that assumes the realization of the best conditions under each category of risk affecting the element in meeting the WBS performance expectations documented in Parts 1 & 2 of the CADRe �Reference� A situation surrounding the development of the WBS element�s KEPPs that assumes the realization of the most likely conditions under each category of risk in meeting the WBS performance expectations as documented in Parts 1 & 2 of the CADRe CARD Cost Analysis Requirements Description RRW Relative Risk Weighting WBS Work Breakdown StructureCARD Cost Analysis Requirements Description RRW Relative Risk Weighting WBS Work Breakdown Structure

52. Indigenous/Programmatic Uncertainty May need a unique technique for processing this type of uncertainty e.g., Relative Risk Weighting A process whereby WBS element-appropriate cost-impacting influences are taken into account in a subjective yet quantitative manner Examples of influences; TRL; Design/Engineering; Schedule; Integration; Requirements Stability; Complexity; Manufacturing, Supportability, etc. RRW Relative Risk Weighting WBS Work Breakdown StructureRRW Relative Risk Weighting WBS Work Breakdown Structure

53. Indigenous/Programmatic Risk TRL DES/ENG COMPLEXITY SCHEDULE TOTAL RISK (0.35) (0.25) (0.2) (0.2) SCORE Pessimistic High VH VH HIGH Profile (5) (7.5) (6) (5.3) 5.9 Reference MOD MOD MOD MOD Profile (3) (3.5) (2.7) (2.2) 2.9 Optimistic LOW ML MOD MOD Profile (1) (2.5) (2.7) (2.2) 2.0 Ref Profile Calc: (0.35)(3) + (0.25)(3.5) + (0.2)(2.7) + (0.2)(2.2) = 2.9

54. CCRM Risk Assessment Estimators know the uncertainty in cost methodology Talk to those involved in performing Continuous Risk Management (CRM) Talk to engineers on the project They understand the risks in: CER input parameters values (e.g., weight) Correlations between input parameters and between WBS elements Technology state of the art (TRL) Designs that use the technologies Engineering necessary to implement the technologies used in the designs Adequacy of the schedule to design and implement the technologies Integration involved at the box, component, subsystem and system levels