Breakout Session 1506 Mr. Jeff Veselenak Senior Acquisition Manager Directorate of Contracting U.S. Air Force Flight Tes

2. The Value of Risk Management in Acquisition Planning Breakout Session 1506 Mr. Jeff Veselenak Senior Acquisition Manager Directorate of Contracting U.S. Air Force Flight Test Center April 15, 2008 3:20-4:20 PM

3. What is risk management? • Risk: A measure of the inability to achieve program objectives within defined safety, performance, schedule, and cost expectations • Two components of risk: • Probability – Likelihood of failing to achieve particular safety, performance, schedule, or cost objectives • Consequence – Impact of failing to achieve those objectives

4. Why do risk management? Larger workforce: Shrinking workforce: Observe Observe Track Work Reduction Manage Manage Risk Aversion Risk Management

5. Why do risk management? (cont.) • Federal Acquisition Regulation • Part 7, Acquisition Planning, 7.105 (a) (7) & (8) • Part 16, Types of Contracts (Fixed Price vs Cost Reimbursement decision) • Strongly implicit or integrated into Parts 10 (market research), 15 (negotiation/trade-off), 28 (bonds & insurance), 32 (contract costing) • DoD Directive 5000.01, 12 May 2003, paragraph E1.14: …reduce technology risk, …reduce integration risk … reduce manufacturing risk

6. Why do risk management? (cont.) • SAF/AQ Policy Memo 03A-002, 4 Feb 03: • Accurately assessing our confidence level requires a disciplined risk assessment program. • … tradeoff non-critical elements within programs to buy down risk. • Establish a methodology to assess program risk…

7. Risk Management Process RiskHandling Risk Monitoring Risk Planning Risk Assessment Risk Identification Risk Analysis Risk Prioritization

8. Planning (cont.) • Select a cross-functional team • User • Program Management • Safety • Security • Contracting • Finance/Resource Advisor • Quality Assurance • Engineering • Maintenance/Logistics • Industry (potential offerors)

9. Risk Management - Assessment • Process of identifying, analyzing, and prioritizing programmatic and critical technical risks; includes quantifying risks in terms of performance, schedule, and cost • Three components • Risk identification • Risk analysis • Risk prioritization • Also must first understand requirements

10. Risk Assessment - Requirements • List program requirements • Hit key requirements only, i.e., funding constraints, schedule milestones, major performance requirements • For best results: risk statements should be written as negative or “if/then” statements that delineate the causes and not merely the symptoms of impact (i.e., need to demonstrate cause and effect relationships) • Sources of requirements • User requirements/capability documents • Program Management Directives (PMDs) • Test & Evaluation Master Plans (TEMPs) • Preliminary technical requirements document, statement of objectives, statement of work • Other contracts

11. RISK STATEMENTS • Risk: A measure of the inability to achieve program objectives within expected safety, performance, schedule, and cost constraints • Future event that could result in negative consequences • “IF network uptime threshold is not met, • THENtest ops tempo will slow, delaying scheduled tests and adding to customer costs” • Refine statements by determining the root cause(s) • Five Whys...”Why would the network uptime threshold not be met? …& so forth…

12. Operational or Technical Customer (User) Uncertainty Changing Requirements Complexity/Dependency # of processes (operations) Lack of Knowledge or Familiarity with Technology Areas Unfamiliarity of Data Rights Schedule Unrealistic Requirements Incomplete ID of Tasks Cost Uncertain Number of Products or Types of Services Uncompetitive Environment Data Management Poor Communication Lack of Succession Planning Lack of Leadership Support (Internal Advocacy) Personal Agendas Personality Conflicts Lack of Functional Representation Lack of Timely, Effective Training Poor Resource Planning Poor Resource Utilization Risk Assessment - Identification(general internal risks)

13. Political actions Funding instability Company instability Lack of competition Available personnel Occupational restraints Labor actions Physical damage Poor information assurance Criminal potential Terrorism potential Supplier viability Regulations Climate / Environment Low priority Transportation dependencies Maintenance dependencies Risk Assessment - Identification (general external risks)

14. Probability Scale DefinitionsExample Probability is the likelihood of failing to achieve a desired program outcome.

15. Consequence Scale DefinitionsExample Consequence is the damage to the program associated with failing to achieve a desired program outcome.

16. Consequence Scale DefinitionsAnother Example

17. Examples of RisksService Contracts • Management Capabilities • Inability to attract, hire, and retain highly qualified personnel • Inability to effectively manage subcontracts • Inability to contract or expand to meet mission changes • Organizational staffing lacks proper skill mix to effectively perform work • Key management positions lack necessary qualifications or experience • Submittals for government approvals lack understanding of critical processes • Cost/Price • Inability to perform to proposed labor rates • Inability to actually cost projects/tasks • Unreasonably low labor rates, non-competitive benefits • Taxing of subcontract work results in unacceptable costs for customers

18. Examples of RisksService Contracts • Phase-In or Contract Transition • Ill-defined interfaces to successful phase-in (government/other on-site contractors, etc.) • Inability to hire sufficient qualified personnel to meet phase-in schedule, contract start date • Poorly designed phase-in schedule resulting in disruption to mission’ • No designated on-site phase-in team to ensure timely actions • Schedule • Continually misses project completion milestones • Poorly defined communication, interfaces, and approval processes • Technical or Operational • Depends on acquisition, usually look at lack of knowledge, understanding, or experience with subject matter (e.g. design, interoperability, systems engineering, etc)

19. Risk Assessment - Prioritization • Why is prioritization important? • Limited time • Limited resources • Oversight to insight • Determine priority of each risk – two common methods • Group risks (high, moderate, low) by plotting on a scatter diagram/matrix of probability vs. consequence • Rank/order individual risks by a variety of voting techniques

20. Probability/Consequence Screening Construct 6 8 10 91- 99 7 9 5 4 6 7 8 9 61- 90 5 Probability [%] 3 5 6 7 8 41- 60 4 11-40 2 4 5 6 7 3 1- 10 1 3 4 5 6 2 1 2 3 4 5 Low Moderate High Consequence

21. IDENTIFY & LIST ALL RISKS FOR EACH KEY REQUIREMENT STEP TWO PRIORITIZE RISKS STEP FOUR Risk Assessment Process STEP ONE STEP THREE UNDERSTAND KEY REQUIREMENTS ANALYZE RISKS RISK HANDLING

22. Risk Handling • Process that identifies, evaluates, selects, and implements risk handling strategies to set risk at acceptable levels • Four strategies • Avoid risk by changing requirements • Transfer the risk (another party/stage) • Control the risk through active steps • Assume the risk without special efforts • Identify specific actions or steps to be taken to handle risks • Reduce probability • Reduce impact • Or reduce both • Can include contingency plans, source selection discriminators (Section M), increase communication among interested parties, etc.

23. Risk Handling (cont.) • Incentives • Award Fee Plans • Award Term Plans • Options • Technical Requirements • Commercial vs. Non-commercial Performance Plans • Past Performance Evaluation • CPARS • Questionnaires Metrics • Acquisition Strategy • Contract type • CLIN structure Instruction to Offerors (Section L of RFP) Risks Source Selection Discriminators (Section M of RFP) Funding Constraints Market Research • Program Objectives • Statement of Work • Statement of Objectives • Performance Work Statement • Technical Requirements Document • Early Industry Involvement • Site visits • Industry days • Joint participation in risk management planning

24. Risk Monitoring • Continuous process that systematically evaluates risks • Input to risk re-assessments • Are there other risks previously unidentified? • Have low or moderate identified risks become high risks? • Executes the risk handling strategies • Determine the effectiveness of the handling action • What is the cost of the risk handling action? • What is the impact on the risk probability? • What is the impact on the risk consequence? • Develops further risk handling strategies • Input for management decisions • One way to monitor risk is to establish metrics against one or more individual risks

25. Risk Monitoring – Data Collection • Frequency • Who collects and how • Contractor shares in responsibility • Sample sources of data • Performance Plan • Service Summary (SS) metrics (Performance-Based Services Acquisitions)

26. Risk Monitoring (cont.) CAN’T MANAGE WHAT YOU DON’T SEE!

27. Documentation • Stand-alone Risk Management Plan and Re-Assessments recommended for large or complex programs • Acquisition Strategy documentation or related formats (e.g., slides or papers) at minimum recommended for smaller or less complex programs • Documentation of risks and associated handling plans recommended for all programs

28. Documentation (cont.) • Sample format for process documentation • Introduction • Program summary • Definitions • Risk management strategy and approach • Risk management team / organization • Risk management process and procedures • Risk planning

29. Example Format:Risk Handling Matrix * Probability; ** Consequence; *** Rating (Low, Moderate, High)

30. Acquisition Planning Considerations – Issues Observed • Teams bring programs forward for Acquisition Strategy Approval without well defined requirements or sound acquisition strategy • Acquisition strategy not well thought out • Briefer(s) can’t answer questions when asked • Technology maturity level – strategy doesn’t address • Schedule charts– teams don’t seem to understand schedule fully; can’t answer basic questions on realism or achievability • Future competition not addressed • Next increment and future increments poorly addressed

31. Acquisition Planning Considerations – Issues Observed (cont’d) • Source selection evaluation criteria NOT tied to significant risks discovered • Identifying significant risk via risk assessment activities, but then NOT evaluating them during source selection to ensure offeror(s) has an adequate strategy to mitigate the risk during contract performance • Source selection evaluation criteria established that are NOT key discriminators in best-value decision • Source Selection– strategy does not adequately explain what team plans to do

32. Acquisition Planning Considerations – Issues Observed (cont’d) • Risks poorly addressed; teams don’t understand risk mitigation • Data management and technical data rights not addressed • Contract type not adequately addressed • Should fully address why the contract type is appropriate for the given strategy/risks • Incentives– strategy does not address what they are, or why they will work Risk Management  Identification of Risk  Mitigation of Risk  Flow of biggest risks shape acquisition / business strategy to be adopted  these get conveyed in Section L, Instructions to Offerors  Section M, Evaluation Criteria

33. Acquisition Planning Considerations – Issues Observed (cont’d) • Acquisition Strategy documentation weak / missing • Roles of Players • Small Business Strategy • Bundling/Consolidation Decision Justification • Performance based (SOO vs SOW or PWS) • Consideration of resources provided to the contractor • Reliability, maintainability, quality assurance • Baseline management – technical & product • Use of warranties • Specifications and configuration approach

34. Acquisition Planning Considerations – Issues Observed (cont’d) • Government vs Contractor Support Analysis • Need business case to support determination • Competition considerations: repairs and supply • Open Technology (Interoperability) Considerations often Lacking • Organizational Conflict of Interest (OCI) Concerns

35. Acquisition Strategy Best Practices • Ideal entrance criteria for acquisition strategy plan approval: • Firm requirements well documented • Official direction/leadership advocacy • Budget approved • Proposed acquisition strategy defined • Program schedule/program office estimate complete • Coordinated acquisition strategy with cognizant small business specialist (Government) • Life-cycle management plans (or equivalent) drafted

36. Lessons Learned • Have a structured risk management process in place • Don’t allow process to focus on ‘risks’ associated with events that have already happened (problem tracking) - use process to look forward • Ensure everyone understands the process and the benefits - training up front • Use agreed-to standard, detailed, tailored definitions • Tailor process to your program - be flexible • Depth of assessment depends on time available • Ensure all team members participate in all steps of risk assessment - understand risks interrelationships

37. Lessons Learned (cont.) • Have a workshop to learn and follow the process • Use a non-team member facilitator - get a more objective perspective • Use subject matter experts from outside program • Look at all sources of risks areas - use guides • Internal & external - tangible & intangible risks • Thoroughly document results • Avoids misunderstandings & ambiguities • Use results to manage - resource allocation • Money is not the only way to handle a risk • Continue risk management process in execution

38. Lessons Learned (cont.) • Timely resolution of “small’ problems avoids downstream disasters. • Ignoring a problem by denying its existence or “hoping it will go away” is often the root cause of a future out-of-control, high-risk situation. • Be knowledgeable of risks early and work to manage their impact • Successful programs have the attitude: risk management is my job

39. Summary • Risk management is a continuous effort to more effectively manage an acquisition program • Risk management is an integral part of decision-making • Risk management creates the opportunity for program success • More efficient and effective program • Focus resources on risk areas • Better understand requirements • Develop acquisition strategy and resultant documentation

40. Summary (cont.)Risk-Driven Acquisition Strategy High RISK MATRIX High X X X X X High X RISK IMPACT PROB RATING P R O B Requirement X X X Risk 1 X X X X Risk 3 X X X X Low Risk 5 IMPACT High Low • Acquisition Strategy (Output-Risk Handling) • RFP Content (SOO/SOW/PWS) • Evaluation Criteria (Section M) • Proposal Preparation (Section L) • Incentives (Contract type) • Post Award Management Concept MFT

41. References • Air Force Institute of Technology SYS 208 Applied Risk Management Course Materials, 2003 • Air Force Material Command Source Selection Training Module: Risk Management in Source Selections, April 2004 • Air Force Material Command Top Ten Training Module: Acquisition Strategy Planning, February 2008 • Warner-Robbins Air Logistics Center Acquisition Center of Excellence Risk Workshop Briefing, November 2007 • Documented Results of Risk Assessments and Acquisition Planning Experiences at the U.S. Air Force Flight Test Center, 2003-2008

42. Teaming and CommunicatingMitigates Risk Risk Management is a contact sport!