SCAMPI Class A v1.3

Sampling the SCAMPI Sampling RulesorTrying to Explain the Unexplainable Pat O’Toole, PACTwww.PACTCMMI.comMay, 2012

SCAMPI Class A v1.3 • SCAMPI • Standard CMMI Appraisal Method for Process Improvement • SCAMPI MDD • SCAMPI Method Definition Document • 250+ page process document for SCAMPI A appraisals • Specific source of angst • SCAMPI MDD v1.3 Section 1.1.4 • It’s only 7 pages, but do NOT drive or operate heavy equipment after reading it!

New Terminology with MDD v1.3 • Basic Units • Sampling Factors • Subgroups • Coverage Rules.

<of Work> Basic Unit • For CMMI-DEV, “Basic Unit” is roughly equivalent to “Project” • However, “project” doesn’t work as well for the Services or Acquisition constellations • “Basic Unit” is much more generic • Therefore, for purposes of this presentation, I will use “PROJECT!”

Sampling Factors – Why? • Project work may be performed in different ways depending on various characteristics • Small projects may be conducted differently than large ones • Project work conducted by your USA-based team may be done differently than your offshore development group • Scrum projects are run differently than Waterfall projects • Sampling factors ensure that the organizational scope provides a robust (and complete) view of the various ways work is performed by the organization.

Mandatory Candidate Sampling Factors • The Sampling Factors that MUST be considered: • Location (e.g., Minneapolis vs. Las Vegas) • Customer(e.g., DoD vs. commercial industry) • Size (e.g., small vs. medium vs. large vs. 4XL) • Organizational Structure (e.g., different divisions) • Type of Work (e.g., development vs. maintenance) • For each candidate, the question to be answered is: • Do different settings for this factor affect the way that work is performed in the organization to be appraised? • Yes – Relevant Sampling Factor • No – Irrelevant Sampling Factor.

Other Potential Sampling Factors • Other Relevant Sampling Factors may exist and, if so, must also be considered • Life Cycle (e.g., Scrum projects vs. Waterfall projects) • Funding Source (e.g., customer vs. internal funding) • Criticality (e.g., nice-to-have vs. life critical) • Etc. • Your organization may have unique characteristics that significantly affect how work is performed • These would be included as Relevant Sampling Factors in a SCAMPI A appraisal.

Example

Subgroup • “Cluster” of projects that share common relevant sampling factor values and exhibit similar process implementations • If Customer, Size, and Type of Work are relevant sampling factors, how many POSSIBLE subgroups are there? • Customer: Internal / External • Size: Small / Medium / Large • Type of Work: Development / Maintenance • 2 values for <Customer> x3 values for <Size> x2 values for <Type of Work> = 12 possible subgroups.

How Many ACTUAL Subgroups?

How Many Projects per Subgroup? Minimum # of projectsfrom a given subgroup Number of projects in the given subgroup Number of projects in the given subgroup Number of subgroups X = Total number of projects

Back to Our Example… # of IMD = (6 x 2) / 15 = 0.8 => 1 # of ISM = (6 x 1) / 15 = 0.4 => 1 # of EMD = (6 x 5) / 15 = 2.0 => 2 # of ELD = (6 x 2) / 15 = 0.8 => 1 # of ESM = (6 x 3) / 15 = 1.2 => 1 # of EMM = (6 x 2) / 15 = 0.8 => 1.

What if… # of IMD = (6 x 200)/1500 = 0.8 => 1 # of ISM = (6 x 100)/1500 = 0.4 => 1 # of EMD = (6 x 500)/1500 = 2.0 => 2 # of ELD = (6 x 200)/1500 = 0.8 => 1 # of ESM = (6 x 300)/1500 = 1.2 => 1 # of EMM = (6 x 200)/1500 = 0.8 => 1

Coverage Rules • Three flavors: • Coverage Rules for Process Areas • Coverage Rules for Basic Units (whoops! – Projects) • Coverage Rules for Support Functions.

Cover Rules for Process Areas • PA Coverage Rule 1 (in for a penny, in for a pound) • Evaluation of a given PA must cover all practices, both specific and generic • There is a wee bit of wiggle room here as the rule includes the words, “…for which they supply data” • PA Coverage Rule 2 (no need to keep pounding) • If a PA is performed in a consistent way across subgroups, evaluation of that PA may be consolidated • For example, if CM, DAR, PPQA, and RSKM are performed in the same manner across all projects, then they need not be evaluated for all 6 subgroups.

Project Coverage Rules • Project Coverage Rule 1 • For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area • This sampled project shall provide data for all process areas • Project Coverage Rule 2 • For at least 50% of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for at least one process area • Project Coverage Rule 3 • For all sampled basic units in each subgroup either artifacts or affirmations shall be provided for at least oneprocess area.

ATLAS #19a • Can you apply what you’ve learned? • Well, heard? • Let’s find out… • Work individually or together to answer the questions in ATLAS #19a • There is no need to provide comments.

Cover Rules for Process Areas • PA Coverage Rule 1 • Evaluation of a given PA must cover all practices, both specific and generic • HINT: Start with the question, “Which projects contribute to its fulfillment?”

Project Coverage Rules • Project Coverage Rule 1 • For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area • This sampled project shall provide data for all process areas • Project Coverage Rule 2 • For at least 50% of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for at least one process area • Project Coverage Rule 3 • For all sampled basic units in each subgroup either artifacts or affirmations shall be provided for at least one process area.

ATLAS #19a Results • ATLAS #19a was distributed to all SCAMPI lead appraisers and only SCAMPI lead appraisers • 61 SCAMPI lead appraisers responded • That’s about a 12% response rate • But it’s still more than “one expert’s opinion” • Let’s see how YOU did compared with those responsible for REALLY understanding this stuff…

Question 1 • Is Process Area Coverage Rule 1 fulfilled?<Evaluation of a given PA must cover all practices, both specific and generic.> 36% Yes / 64% No Which project(s) contribute to its fulfillment: P1? 82% Yes / 17% No P2? 90% Yes / 10% No P3? 74% Yes / 24% No P4? 7% Yes / 93% No.

Question 2 2. Is Project (BU) Coverage Rule 1 fulfilled?“For each subgroup, both artifacts and affirmations shall be provided for at least one project for every process area” 34% Yes / 66% No Which project(s) contribute to its fulfillment: P1? 42% Yes / 56% No P2? 14% Yes / 86% No P3? 4% Yes / 96% No P4? 4% Yes / 96% No.

Question 3 3. Is Project (BU) Coverage Rule 2 fulfilled?“For at least 50% of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for at least one process area” 51% Yes / 49% No Which project(s) contribute to its fulfillment: P1? 54% Yes / 46% No P2? 97% Yes / 3% No P3? 22% Yes / 78% No P4? 18% Yes / 82% No.

Question 4 4. Is Project (BU) Coverage Rule 3 fulfilled?“For all sampled basic units in each subgroup either artifacts or affirmations shall be provided for at least one process area.” 48% Yes / 52% No Which project(s) contribute to its fulfillment: P1? 94% Yes / 6% No P2? 98% Yes / 2% No P3? 93% Yes / 7% No P4? 25% Yes / 75% No.

Optical Illusion – 2 Images in One

Modified Coverage Rules - #1 • PA Coverage–1: Objective evidence provided for a PA must address all practices that are part of that PA, both specific practices as well as all generic practices. The Basic Unit and Support Function coverage rules indicate the types of objective evidence that must be provided for each practice. • Project Coverage–1: For each subgroup, both artifacts and affirmations shall be provided for each practice for at least one project. • Project Coverage–2: For at least 50 percent of the sampled projects in each subgroup, both artifacts and affirmations shall be provided for each practice for at least one PA • Project Coverage–3: For all sampled projects in each subgroup either artifacts or affirmations shall be provided for each practice for at least one PA.

Modified Coverage Rules - #2 • PA Coverage–1: Objective evidence provided for a PA must address all practices that are part of that PA, both specific practices as well as all generic practices. [No change] • Project Coverage–1: For each subgroup, some combination of both artifacts and affirmations shall be provided for at least one project. • Project Coverage–2: For at least 50 percent of the sampled projects in each subgroup, some combination of both artifacts and affirmations shall be provided for at least one PA • Project Coverage–3: For all sampled projects in each subgroup either artifacts and/or affirmations shall be provided for at least one PA.

Conclusion • The new sampling and coverage rules are intended to provide a truly representative set of projects that reflect how work is performed in the organization • There are some issues, however: • They only solve the “proportional problem,” not the “how much is enough” problem • There are multiple ways to interpret the rules as written, arriving at significantly different results • Based on ATLAS #19a, the SEI has committed to clarify, or even re-write the coverage rules • With respect to brain function, covering this topic is equivalent to drinking 6 beers; please drive carefully!

Questions??? Pat O’TooleProcess Assessment, Consulting & Training (PACT) PACT.otoole@att.net www.PACTCMMI.com • If anyone is interested in receiving my monthly “Do’s and Don’ts” articles, either: • Give me a business card • Send me an email.

SCAMPI Class A v1.3