Probability of Program Success Component Standardization : Status Brief

1. Probability of Program Success Component Standardization : Status Brief Ms. Jane Rathbun Special Assistant , Program Management Improvement OSD(AT&L)(A&T)(PSA) May 5, 2010

2. Status • PoPS Component Working Group has reached agreement on a common framework • PARCA in conjunction with PSA, SE and ARA will work to incorporate PoPS into their governance efforts.

3. PoPS Framework Components As defined by Navy version • Program Health: The current state of an acquisition program’s requirements, resources, planning and execution activities, and external influencers, and how those factors are impacting the program’s ability to deliver a capability within specific constraints. • Factors: These are Program Health organizational categories. The four Factors are: Program Requirements, Program Resources, Program Planning and Execution, and External Influencers. • Metrics: Major sub-categories that collectively define the scope of a particular Factor. There are 18 Metrics in the Naval PoPS 2.0 Program Health framework. Metrics are the basic building blocks of Naval PoPS. • Criteria: Parameters (qualitative and quantitative) used to evaluate a particular Metric. Each Criteria is associated with a unique identification number to enable traceability between Naval PoPS documents and tools.

4. Proposed Enterprise Framework

5. Agreement on Enterprise Framework What does it Mean? • The same framework will be used by all Military Departments—Titles and Definitions • The same sets of criteria will be aligned to and used for each metric • The same metric weighting system by phase will be used by all components • The criteria will change according to the agreed upon number of phases to be evaluated • A target Enterprise IOC date of Third quarter 2011 What has to occur on the PoPS side? • Working Group will work to align the criteria for the planning phase first, and then the criteria for the remainder of the phases will be aligned • Final agreement on the phases to be used at the enterprise level will have to be reached • Weighting process has to be worked out • An enterprise PoPS governance and management process will need to be established to manage changes to framework—factors, metrics and criteria • Need to determine how this will be incorporated into other components. • Identification of implementation resources Question to be answered: How will OSD use this tool?

6. How will we use it? Possible uses of PoPS in the Defense Enterprise • Goal: At the enterprise level, leverage (to the greatest extent possible) program assessment tools in use by the components, minimize additional workload on the program offices • Possible Enterprise Uses • Processes: DAES, Program Support Reviews, Performance Assessments • Organizations: PARCA, SE, T&E • Pending Legislation: HR 5013 proposes a new chapter to Part IV of title 10— “Performance Management of the Defense Acquisition System” • “ …all elements of the defense acquisition system are subject to regular performance assessments” • “ the SECDEF shall establish categories of metrics for the defense acquisition system, including at a minimum , categories related to cost, quality, delivery, workforce and policy implementation…” • Action: PARCA, PSA, SE and ARA will form a working group to determine how best to leverage the Standard PoPS into governance efforts

7. Approach • Phase One, Part Two A MilDep Working Group : May 2010-September 2010 • Present way forward to AT&L and Component Leadership for common PoPSget approval. • Complete criteria alignment for the first phase and apply proof of concept to all other phases, align criteria and weighting • Build requirements documentation and implementation plan • Identify any additional resources needed for implementation • Build an implementation schedule • Phase One. Part Two B MilDep and OSD working group: Identification of Enterprise Uses & Governance May 2010-September 2010 • Identify enterprise reporting requirements that could be modified or replaced with a standard PoPSmodel • Share model with components other than MILDEPs • Share model with Industry counterparts and other components for evaluation as the industry/government program health assessment tool (done through ICPM) • Components and OSD establish governance and adjudication process for PoPS framework and related reporting

8. Approach • Phase Two, Military Department Implementation: FY 2011 • Components to make changes to their internal processes and PoPS models • OSD prepare to receive and utilize PoPS in identified forums • First reporting goal: Beginning of Third Quarter • Phase Three, All Component Implementation: FY 2011 • All components to make changes to their internal processes and adopt the PoPS model • First reporting goal: Beginning of First Quarter 2012

9. Back Up

10. Background & Intent • NDIA/ICPM Sub-Team working on Key Performance Indicators identified a set of metrics that might serve both government and industry ( March 2009 ICPM) • As of 2008, all Military Departments are utilizing some variant of PoPS (Probability of Program Success)—a program health assessment tool • November 2008 Memo from AT&L, Director PSA to Military Deputies established as working group to determine a way forward on a common variant • Can we get to a common variant of PoPS within Defense? • What other program health/performance indicators are needed for a complete suite of assessment tools—PoPS+? • What enterprise level information requirements could be replaced by a PoPS +? • Could a common variant of PoPS serve as the baseline measure of program health for both Government and Industry?

11. Factor/Metric Descriptions—Program Requirements • Program Requirements: Capability requirements [defined in the Initial Capabilities Document (ICD)/Capability Development Document (CDD)/Capability Production Document (CPD)] that the program must meet within approved cost and schedule constraints • Parameter Status: Progress toward defining capability requirements [ICD/CDD/CPD] and meeting those requirements through the achievement of Key Performance Parameter (KPP)/Key System Attribute (KSA)/other attribute threshold values. Also measures requirements traceability and the validity of the threat assessment. • Scope Evolution: Stability of performance parameters/other attributes/quantities from the established baseline and the impact of requirements changes on total program cost and schedule. • CONOPS: Progress toward developing and scoping the Concept of Operations (CONOPS), using it to inform program requirements, acquisition approaches, and strategies, and the validity of the CONOPS over time.

12. Factor/Metric Descriptions—Program Resources • Program Resources: Funding and manning that is allocated to the program to accomplish planning and execution activities. • Budget: Sufficiency of funding (amount and phasing) across the Future Years Defense Program (FYDP) based on last approved budget controls and degree of deviation from the current cost estimate. • Manning: Stability and adequacy of Resource Sponsor and Program Office staffing (availability, skills, experience, certification, and training).

13. Factor/Metric Descriptions—Program/Planning • Program Planning/Execution: Activities performed by the Program Office, contractors, and government performers to fulfill program requirements and deliver expected, affordable, and sustainable capability to the operating forces. • Total Ownership Cost Estimating: Measures the adequacy of the elements required to produce sound cost estimates: program description information, cost data, cost estimating process, cost estimate stability and comparisons, and cost estimate measures. Also assesses how well acquisition, systems development, and sustainment strategies are evolving in ways intended to mitigate Total Ownership Cost (TOC) growth. • Schedule: Completeness and progress against the integrated master schedule/program master schedule; also includes status of milestone documentation development. Status of procurement activities and achievement of contracting milestones against the planned schedule. • Industrial Base/Manufacturing/Production: Assesses market research activities, industrial base health, and an understanding of industrial implications for cost, schedule, and technical risks. Also measures manufacturing/production capabilities and execution. • Test and Evaluation: Progress toward defining and executing the Test and Evaluation Strategy (TES) and the Test and Evaluation Master Plan (TEMP). This includes the ability to evaluate the system's technical and operational maturity and performance through testing, the adequacy of test resource requirements to accomplish the necessary key test activities, the status of identified technological risks, system deficiencies, and the effectiveness, suitability, and survivability of the system under development. • Technical Maturity: Assessment of the maturing system and sub-systems design, as well as the technical maturity of Critical Technology Elements (CTEs) in accordance with the approved Technology Development Strategy (TDS). Evaluation of the supporting engineering processes, engineering documentation, and lessons learned to achieve an Operationally Effective and Suitable System.

14. Factor/Metric Descriptions—Program/Planning • Program Planning/Execution (cont.) • Technology Protection: Status and progress toward the safeguarding of DOD research, technology information, and applied knowledge associated with the program. Functional disciplines include threat assessments and intelligence/counterintelligence, Anti-Tamper, Supply Chain Risk Management, and physical and electronic security across government and Defense Industrial Base partners. Evaluated by the reporting of program protection strategy and plans, personnel (both internal and external to a program office), and resources. • Software: Software management and engineering (including translation and allocation of system capabilities to software, software code development, software-related risk management, etc.); applies to software activities by government agencies and/or contractors that are integral to program deliverables. Evaluated in terms of software size and stability, cost and schedule, organization, and quality. • Sustainment: Progress toward defining and executing the sustainment strategy, and the resource adequacy applied toward those life cycle sustainment activities. Sustainment is conducted as specified by an evolving Life Cycle Sustainment Plan (LCSP) and attachments. The Independent Logistics Assessment (ILA) is the milestone focus by which decision makers determine LCSP execution effectiveness and affordability. • Government Program Office Performance: Progress toward defining and executing intra-government requirements; responsiveness to deliverable submissions; delivery of facilities, funding, and Government Furnished Equipment (GFE)/Government Furnished Information (GFI) in accordance with scheduled requirements; Configuration Management/Configuration Control Board (CCB) and Risk Management Board (RMB) effectiveness. • Contractor Performance: Performance of major contractors and/or government performers as measured by the Earned Value Management System (EVMS), Contractor Performance Assessment Reports (CPARs)/Informal Performance Assessment Reports (IPARs), staffing adequacy, and work package completion. Also assesses each company’s financial health, financial systems, and manufacturing/production capabilities.

15. Factor/Metric Descriptions—External Influencers • External Influencers: Issues or actions taken by parties outside the purview of the Program Manager that may impact program planning/execution activities and the achievement of program requirements or objectives. • Fit in Vision: Program alignment with current documented Office of the Secretary of Defense (OSD) guidance and Service strategies. • Program Advocacy: Support demonstrated by key stakeholders: Congressional; Under Secretary of Defense Acquisition Technology & Logistics (USD AT&L) (or equivalent); Assistant Secretary of Defense for Networks and Information Integration (ASD NII); Cost Assessment and Program Evaluation Office (CAPE); Director of Operational Test & Evaluation (DOT&E); USD (Comptroller); Service/Component; Joint Staff/Combatant Commander (COCOM); Fleet Forces Command (FFC)/Marine Corps Forces (MARFOR); International Partners; Other Services. • Interdependencies: Interface issues affecting inter-related programs; determines whether dependent programs are on track to deliver the requisite capability or quantity on schedule.