Practical Software and Systems Measurement A foundation for objective project management

1. PSM Practical Software and Systems Measurement A foundation for objective project management COCOMO Forum October 24, 2001

2. Outline • Objectives of Measurement and Analysis Initiatives • Practical Software and Systems Measurement • ISO/IEC 15939, Software Measurement Process • CMMI Measurement and Analysis Process Area • ISO 9000-3, Guidelines for Software

3. Objectives of Measurement Initiatives • Experienced-Based Measurement to Support Management and Engineering Requirements • Consistent Measurement Process Across Measurement Initiatives • Consistent Measurement Terminology

4. Practical Software and Systems Measurement INFORMATION DRIVEN MEASUREMENT PROCESS ISO/IEC 15939 ISO 9000-3 CMMI M&A

5. Technical & Management Processes User Feedback Measurement Requirements Information Products Information Needs Core Measurement Process Establish Commitment Plan Measurement Perform Measurement Evaluate Measurement Plan Performance Information Experience Base Evaluation Results Improvement Actions Measurement Process Scope Measurement Process Model

6. Practical Software and Systems Measurement (PSM)

7. What is PSM? • A Primary Measurement and Analysis Process Used by DoD, Government, and Commercial Programs • Technology Based on Actual Experience on Real Programs • A Fully Integrated Approach Including Guidance, Tools, Training, Implementation Workshops, Support Products, and a Qualified Technical Team

8. Key Measurement Concepts • Measurement is a Process - Not a Pre-Defined List of Graphs or Reports • The Measurement Process is Flexible - Adapted To Meet Specific Project and Organizational Objectives and Information Needs • The Measurement Requirements are Integrated Into the Project Technical and Management Processes

9. Common Information Needs • Schedule and Progress • Resources and Cost • Product Size and Stability • Product Quality • Process Performance • Technology Effectiveness • Customer Satisfaction

10. Start of Build 1 I&T Apr 98 Jan 99 Apr 99 Jan 98 Oct 97 Jul 98 Oct 98 Problem Report Status Priority 1, 2, and 3 Problem Reports Build 1 1000 900 800 700 600 Discovered 500 Number of Problem Reports Closed 400 300 200 100 0 30 Aug 98

11. Estimation Defined Information Needs Feasibility Analysis Performance Analysis Types of Analysis

12. Estimation

13. Integrated Analysis Model

14. Integrated Analysis Example

15. Performance Measurement • Normative Performance Baselines • Technical and Business Policy • Investment Decisions & Analysis Enterprise Management Risk Management Process • Process Improvement • Program Planning Guidelines • Performance Based Acquisition • Organizational Norms & Benchmarks Issue-Driven Measurement Process Organizational Management • Program Estimation and Planning • Program Performance Tracking • Program Tradeoff Analysis • Resource Management Program Management Multi-Level Requirements

16. Government - Industry - Academia Team Technical Consensus of Best Practices Incremental Product Development and Improvement Empowered Transition Organizations Integrated With Other Key Initiatives Practical Software Measurement Proven Measurement Technical Guidance and Tools PSM Project Approach Comprehensive Transition Support PSM Project Strategy

17. New Technology Areas • Measurement and New Technologies • Object-oriented developments • Spiral/evolutionary approaches • Interoperability • Product lines • Initial Recommendations Were Draft Drafted in Early 2001 • Field Trials Are Underway - Additional Trials Will be Conducted in 2002

18. PSM Participating Organizations • Overseas • Defence Materiel Org. (Australia) • LiveWare (Argentina) • NRC (Canada) • Software and Systems Engineering (Israel) • Software Improvements (Australia) • Software Quality Institute (Australia) DoD and Government • US Air Force AFMC • US Air Force SSSG • US Air Force STSC • US Air Force CRSIP • US Air Force STRATCOM • US Army AMC • US Army ATEC • US Army CECOM • US Army ODISC4 • US Army SMDC • US Army TACOM - ARDEC • USMC MCTSSA • US Navy AEDC • US Navy ASN RDA • US Navy FMSO • US Navy NAVAIR • US Navy NAVSEA • US Navy NAWC • US Navy NRL • US Navy NSWC • US Navy NUWC • US Navy OPTEVFOR • ODUSD (S&T) • OSD C31 • OSD PA&E • Aerospace Corp. • DCMA • DFAS • DISA • DLA • DSMC • FAA • IDA • MITRE Corp. • NDU IRM College • NASA • National Park Service • Sandia National Lab • SEI • Soc. Sec. Admin.

19. PSM Participating Organizations • Industry • ACS GSG • American Sys. Corp. • Averstar • Bloodworth Int. Tech. • Carnegie Mellon Univ. • CSC • CTA • Distributive Software • Federal Reserve Bank • General Dynamics • General Scientific Corp. • Graeme & Garland • GTE • Hawaiian Electric • IEEE • IFPUG • IITRI • INCOSE • Independent Engineering • ITABHI • James Gregory Assoc. • Kodak Health Imaging • Litton/TASC • Lockheed Martin • L3 Communications • Northrop Grumman • OAO Corporation • PRC - Litton • PRC/TASC • QSM • Quality Plus Tech. • Raytheon - Hughes • Robbins, GIOIA • Rockwell Collins • SAIC • Sallie Mae • Sentel • Software Metrics • SPC • Sverdrup Tech. • Tecolote Research • TeraQuest Metrics • Tivoli • TRW • United Defense • Univ. Southern CA • User Trust Network • US West • VPI - State University • West Virginia High Tech. Cosortium • West Virginia University

21. ISO/IEC 15939, Software Measurement Process

22. ISO/IEC 15939 • PSM Was Base Document for This Standard • Specifies Required Activities and Tasks of a Measurement Process • FCD Ballot Passed in May 2001 With Only 1 Dissenting Vote - All Comments Resolved at ISO Plenary • Terminology Was Adjusted for Consistency With ISO/IEC 9126, 15498, and 14143 Series • DIS Ballot Will be Released in September 2001 - Ballot Will Close in November 2001 - International Standard in Late 2001

23. Basic Measurement “Architecture” ISO/IEC 15939 Measurement Information Model Measurement Process Model Information Driven Measurement Requirements Consistent, Tailorable Implementation Process

24. Information Needs Information Product Measurement Construct Measurable Concept Entities Attribute Measurement Information Model

25. Information Needs Measure that Provides an Estimate or Evaluation With Respect to Information Needs Indicator Algorithm Combining Measures and Decision Criteria Analysis Model Quantity Defined as a Function of Two or More Measures Derived Measure Derived Measure Algorithm Combining Two or More Measures Meas. Function Base Measure Base Measure Measure of a Single Attribute Defined By a Specified Measurement Method Operations Mapping an Attribute to a Scale Meas. Method Meas. Method Entities Property Relevant to Defined Information Needs Attribute Attribute

26. Information Product Information Needs Information Product Measure that Provides an Estimate or Evaluation With Respect to Information Needs Decision Criteria Indicator Algorithm Combining Measures and Decision Criteria Analysis Model Base Measure Derived Measure

27. CMMI - Measurement and Analysis (M&A)

28. CMMI M&A • ISO/IEC 15939 Was Base Document for the Measurement and Analysis Process Area • CMMI M&A Describes How to Evaluate a Measurement Process • M&A Is a Level 2 Process Area • Pilot Assessments Are Being Conducted in 2001

29. ISO 9000-3Quality management and quality assurance standards - Part 3: Guidelines for the application of ISO 9001 to the development, supply, installation, and maintenance of computer software

30. ISO 9000-3 • 9000-3 Is Being Updated by SC7 as Part of the ISO 9000:2000 Update • Part 8, “Measurement, Analysis and Improvement”, Was Updated for Consistency With 15939 • Standard Is at Working Draft Stage - Expected Publication Date Is 2002/2003

31. For More Information PSM Support Center Cheryl Jones - PSM Project Manager TACOM - ARDEC AMSTA-AR-QAT Building 62 Picatinny Arsenal, NJ 07806-5000 (973) 724-2644 (Voice) (973) 724-2382 (FAX) cljones@pica.army.mil PSM Home Page: http://www.psmsc.com