1 / 46

Success at CMM ® Level 3 but not at Level 4 Lessons Learned Al Florence MITRE

Success at CMM ® Level 3 but not at Level 4 Lessons Learned Al Florence MITRE The views expressed are those of the author and do not reflect the official policy or position of the MITRE Corporation This presentation is not based on work done at MITRE. Although.

azizi
Download Presentation

Success at CMM ® Level 3 but not at Level 4 Lessons Learned Al Florence MITRE

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Success at CMM® Level 3 but not at Level 4 Lessons Learned Al Florence MITRE The views expressed are those of the author and do not reflect the official policy or position of the MITRE Corporation This presentation is not based on work done at MITRE

  2. Although • This presentation is based on: • Work done a few years ago • On real project data • On the Software Engineering Institute (SEI) Capability Maturity Model (CMM®) • On CMM Level 3 and Level 4 • Most topics and lessons learned are applicable to: • CMM Integration (CMMI®) • Any CMMI® Level • Any change situation

  3. Presented & Published • Software Technology Conference (STC), Salt Lake City; April 2001 • Software Engineering Process Group (SEPG) Conference, New Orleans; March 2001 • CrossTalk, The Journal of Defense Software Engineering; August 2001 • The Quality Assurance Institute Journal; April 2002

  4. Overview • Introduction • Software Engineering Institute’s Software Capability Maturity Model Overview • Software Process Architecture • Corporate/Organization/Projects (Processes) • Getting to Level 3 • Not Getting to Level 4 • Reasons for Getting to Level 3 and Not to Level 4 • Conclusions • Suggested Reading • Contact Information

  5. Introduction • Getting to SEI CMM® Level 3 can be quite different than getting to Level 4 • The reasons, commitments, dynamics and resources can be quite different for success at Level 3 vs. Level 4 • This presentation is focused on those differences and provides valuable lessons learned gathered from an organization that had achieved Level 3 but failed to achieve Level 4

  6. Introduction(completed) • This author was the project development manager and Software Engineering Process Group (SEPG) lead on one of the projects at the time Level 3 was achieved for the organization • This presentation is mostly based on this project • When the word “project” is used it is only referring to this project • When the words “projects” and/or “organization” are used they are referring to all projects in the organization that were involved in the Level 3 and Level 4 efforts • This author was later the SEPG lead at the next higher organizational level and defined Level 4 and Level 5 processes, installed them and supported their execution on the project

  7. Software Engineering Institute’s Software Capability Maturity Model Overview

  8. Capability Maturity Model (CMM®) Overview Level Name Characteristics Optimizing Continuously improving process 5 Quantitative control of products and process 4 Managed Management and engineering practices defined at the organizational level Defined 3 Basic project management established 2 Repeatable Ad hoc and often chaotic 1 Initial

  9. Key Process Areas (KPAs) of the CMM® Level Name KPAs Defect Prevention, Technology Change Management, Process Change Management 5 Optimizing Quantitative Process Management, Software Quality Management 4 Managed Organization Process Focus, Organization Process Definition, Training Program, Integrated Software Management, Software Product Engineering, Inter-group Coordination, Peer Reviews Defined 3 Requirements Management, Software Project Planning, Software Project Tracking and Oversight, Software Subcontract Management, Software Quality Assurance, Software Configuration Management 2 Repeatable

  10. Software Process Architecture The following is based on the Software Engineering Institute and is useful In understanding how the Capability Maturity Models are adapted and institutionalized in organizations

  11. Software Process Architecture • Organizational Standard Process (OSP) • Process that exists at the organizational level • Generic but complete to support the most complex, critical, largest project • Project Defined Process (PDP) • Processes that exists at the project level • Tailored/adapted from the OSP to the scope of the project • Process Asset Library (PAL) • Contains OSP • Contains artifacts collected while executing process on programs • Process/project plans • Project schedules • Meeting minutes • Project documents • Etc.

  12. Software Process Architecture (continued) PAL PDP OSP Policies Process Plans Process Descriptions Process Procedures Process Tailoring/Adaptation Process Procedures Process Templates Process Templates Process Guidelines Process Guidelines Standards Standards Process Training Process Training ProcessProject Execution Process Artifacts

  13. Software Process Architecture (continued) • Policies • High-level policies that require projects to tailor/adapt the OSP into a PDP and to follow it while executing the project • Policies are directed and endorsed by the most senior executive in the organizations • Not tailored for PDP • Process Descriptions • High-level descriptions of the processes that constitute the overall process • Not tailored for PDP

  14. Software Process Architecture (continued) • Process Procedures • Detailed description of the execution of the process • Description • Entry criteria • Inputs • Roles/responsibilities • Detailed step-by-step description of activities • Measurements • Exit criteria • Outputs • Tailored for the PDP

  15. Software Process Architecture(continued) • Process Templates/Guidelines • Process plans • Process artifacts • Interview questionnaires • Product reviews • Process audits • Metrics • Peer reviews • Etc. • Used to tailor and execute PDP

  16. Software Process Architecture (Completed) • Standards • Industry or corporate standards and Data Item Descriptions (DIDs) • Requirements • Design • Test • System Engineering • Software Engineering • Process Training • Training and orientation on the various processes • Tailored for the PDP

  17. Corporation/Organization/Project Processes

  18. Corporation • Many Level 3 organizations in the Corporation • Corporation had no Level 4 organizations

  19. Organization • Organization’s projects existed at geographically dispersed locations • Projects in the organization engaged in diversified software activities • The organization had achieved Level 3 and was pursuing Level 4

  20. Organizational Standard Process Project’s Defined Process • Organizational Standard Process (OSP) exists at the Corporate level • Had process for Level 2 and Level 3 only • OSP adapted on projects as Project’s Defined Process (PDP)

  21. Software Engineering Process Groups (SEPG) • Corporate SEPG • Organizational SEPGs • Project SEPGs

  22. Process Training • Corporate had a Level 2 and Level 3 training program • All employees engaged in software development were required to take process training appropriate to their software tasks

  23. Getting to Level 3

  24. Commitment/Cooperation • Executive/Senior management committed/cooperated • Mandated that Organization become Level 3 • Projects’ management committed/cooperated • Projects’ personnel committed/cooperated • Customers committed/cooperated

  25. Resources • Sufficient funding from the corporation, the organization and the projects • Sufficient process staff • Everyone on projects involved in Level 2 and Level 3 activities • Corporate Organizational Standard Process provided process for Level 2 and Level 3 KPAs • Corporate Level 2 and Level 3 training material provided • Many Level 2 and Level 3 industry publications and many examples available as reference material

  26. Software Engineering Process Groups • Corporate SEPG had membership from Organizations SEPGs • Organization’s SEPG had membership from the Projects SEPGs • Coordinated weekly • Level 2 and Level 3 coordination quite difficult between physically separate locations • Ensured that process applied in a repeatable fashion in the projects

  27. Level 3 Assessment Preparation • Installed and executed all Level 2 and Level 3 KPA processes • Collected extensive Level 2 and 3 artifacts • Conducted extensive Level 2 and 3 training • Conducted dry run assessments (supported with many Government Software Capability Evaluations) • Everyone involved

  28. Standards on Contract MIL-STD-2167A and associated standards for the specific project which provide for: • Management Plans • System Engineering • Software Development • Risk Management • Reviews and Audits • Peer Reviews • Testing • Test Plans • Earned Value Management • Configuration Management • Quality Assurance • Metrics

  29. Processes and Artifacts • The standards required many project plans and product deliverables • The execution of project plans provided all of the Level 2 and many Level 3 processes • Product deliverables provided all of the Level 2 and most of Level 3 artifacts

  30. Achieved Level 3 • Organization achieved Level 3 • 27 months after contract award for specific project • Project skipped Level 2 assessment • Assessment was a SEI CMM Based Appraisal for Internal Process Improvement (CBA-IPI) • Lead assessor from an external vendor • Assessment team internal • Note: there were many assessors that had conducted Level 2 and Level 3 assessments

  31. Not Getting to Level 4

  32. Commitment/Cooperation • Executive/Senior management somewhat committed/cooperative • Mandated that Organization become Level 4 • Did not apply the “carrot nor the stick” • Did not “walk the talk” • Project management not committed/cooperative • Level 3 good enough • Did not sign up for Level 4 • Project personnel not committed/cooperative • Level 3 good enough • Except for Project SEPG lead and Project Software Quality Assurance Manager • Even fell back on some Level 3 processes • Project Customer may not even have been aware of Level 4 efforts

  33. Resources • Insufficient funding from the Corporation, the Organization and the Projects • Maintained at same level as for Level 3 • Should have been increased for Level 4 • Insufficient process staff • Few staff on projects involved in Level 4 processes • Corporate Organizational Standard Process did not support Level 4 or Level 5 KPAs • Corporate Process Training did not support Level 4 or Level 5 • Limited Level 4 and Level 5 industry literature and few examples to draw from

  34. Software Engineering Process Groups • Corporate SEPG not involved at Level 4 at the time • No process for Level 4 or Level 5 • The Organization SEPG had membership from the Project SEPGs • Coordinated weekly • Level 4 coordination becomes verydifficult between physically separate locations • Level 4 not applied in a repeatable fashion across the organization

  35. Level 4 Assessment Preparation • Project instituted all Level 4 and Level 5 KPA processes • Other projects instituted only Level 4 • Project executed all KPA processes • Project collected extensive Level 2, Level 3 and Level 4 artifacts • Projects conducted insufficient Level 4 training • Lack of cooperation • Lack of corporate training material at Level 4 • Projects conducted 1 dry run for Level 4 assessment

  36. Standards on Contract • Standards on contract provide for all Level 2 processes and artifacts and many for Level 3 • Standards on contract standards do not provide for processes or artifacts for Level 4 nor Level 5

  37. Level 4 Not Achieved • Although all Level 4 processes were conducted and Level 4 artifacts collected, the organization failed to achieve Level 4 • Assessment was a SEI CMM CBA-IPI • Lead assessor from an external vendor • Assessment team internal • Note: there were few assessors that had conducted Level 4 and Level 5 assessments • Interview, Interview, Interview • Be selective

  38. Reasons for Achieving Level 3 but not Level 4

  39. Reasons for Achieving Level 3 but not Level 4 Level 4 Level 2 & Level 3 Commitment, funding and cooperation inadequate Standards on contract do not provide processes and artifacts Few published examples Done for process sake Few experienced assessors Commitment, funding and cooperation existed Standards on contract provide for processes and artifacts Many published examples Based on business goals Many experienced assessors Also, the project fell back on some Level 3 processes

  40. Reasons for Achieving Level 3 but not Level 4(continued) Level 4 is a drastic paradigm shift from Level 2 &Level 3 that isn’t always recognized Activities are common sense things to do in order to develop “good” products Require existing skills (software & management) Focuses on the Organization Requires that measurements be collected and actions taken on results Goes beyond these common sense things to do and is for organizations that really want to go the “extra mile” Require new skills (quantitative & statistical) Focuses on the Projects Requires that measurements be quantitatively analyzed statistically and immediate actions taken to remedy issues Level 2 & Level 3 Level 4

  41. Reasons for Achieving Level 3 but not Level 4(concluded) Level 4 is a drastic paradigm shift from Level 2 and Level 3 (continued) Requires that process capability be institutionalized Requires that quality assurance be institutionalized Level 2 and Level 3 Level 4 Requires that process capability be understood and controlled quantitatively Requires that plans for quality goals be established and that progress towards achieving these goals be quantitatively managed

  42. Conclusions • Getting to Level 3 can be quite different than getting to Level 4 • The reasons, commitments, dynamics and resources can be quite different, meaning success at Level 3 and failure at Level 4 • Process commitment has to be maintained at all CMM Levels • Standards may provide for Level 2 and Level 3 processes and artifacts but don’t for Level 4 nor Level 5 • Very difficult to coordinate when the organization has diversified projects and is physically in separate locations • Process improvement is not the sole responsibility of the SEPG and a selected few, everyone has responsibilities

  43. Conclusions (concluded) • Process improvement will only work if: • Everyone is committed and cooperates • Proper resources are available • Improvement is based on business goals • Level 4 is a drastic paradigm shift from Level 3 • New and additional skills are required at Level 4 (quantitative & statistical) • Level 4 requires that process capability and quality goals be understood and managed quantitatively

  44. Suggested Reading • Mark C. Paulk, Bill Curtis, Mary Beth Chrissis, Charles V. Weber, February 1993, Capability Maturity Modelfor Software, V1.1, Software Engineering Institute • William A. Florac, Anita D. Carleton, 1999, Measuring for Process, Statistical Process Control for Software Process Improvement,Software Engineering Institute • Anita Carleton, Mark C. Paulk, 1997, Statistical Process Control for Software, The 1997 Software Engineering Symposium, Pittsburgh, PA • David S. Chambers & Donald J. Wheeler, 1995, Understanding Statistical Process Control, SPC Press • Donald J. Wheeler, 1993, Understanding Variation, The Key to Managing Chaos, SPC Press • Donald J. Wheeler, 1995, Advanced Topics in Statistical Process Control, SPC Press

  45. Suggested Reading • Ron Radice , March1997, Getting to Level 4 in the CMM, SEI Software Engineering Process Group (SEPG) Conference Proceedings, San Jose, CA • Jeff Perdue, November 2000, Why is Level 4 so Hard?, Washington D.C. Software Process Improvement Network • David N. Card, December 2000, Quantitative Techniques for CMM Level 4, Software Productivity Consortium • Al Florence, March 1999, CMM Level 4 and Levle 5 Approaches, 1999 SEPG Conference Proceedings, Atlanta, GA • Al Florence, May 2000 , CMM Level 4 Quantitative Analysis and Level 5 Defect Prevention, 2000 Software Technology Conference Proceedings, Salt Lake City, UT • Al Florence, February 2001, CMM Level 4 Quantitative Analysis and Defect Prevention with Project Examples, Cross Talk, The Journal of Defense Software Engineering

  46. Contact Information Alfred (Al) W. Florence florence@mitre.org (703) 983-7476

More Related