Has the SEI Gone Too Far or Not Far Enough? a Panel Discussion

1. Has the SEI Gone Too FarorNot Far Enough?aPanel Discussion

2. Panelists • Madhumita Sen – IBM India • Bill Deibler – Software Systems Quality Consulting • John Maher – Organization & Process Improvement, Inc. • Ron Radice – Software Technology Transition

3. Approach and Intent • The panelists will discuss the Pros and Cons on a number of questions, which were supplied to each panelist in advance. The questions were offered to the full LA, including HMLA, community to rank preference. • The intent of the discussion is not to criticize, but to offer an open forum on issues and concerns within the LA, HMLA, and appraisal sponsor community, such that the SEI might benefit in how it will address their needs. • The intent is to collaborate with the SEI on addressing pressing issues and concerns.

4. How Many Responded?Limited to Lead Appraiser Community • 437 – LA/HMLAs in population set • 13 – Fatal/Blocked/Failed • 166 – Scored Responses = 95% CL and 6% CI • 56% HMLAs responded, 36% LAs, 25% of SEI LAs • 18 of the 36 questions rated highest become candidates for the panel

5. Most Wanted Discussion Topics 1. Does the current process for HMLA certification address the concern regarding the validity of high maturity appraisals? 2. Should the SEI announce when LAs, including HMLAs, have their authorizations removed? What criteria are used to determine the necessity for removal and how often has it been necessary? 3. How do you feel about the lack of visibly defined criteria for when the authorizations should be removed, and the impact on appraisals that were done by a person/persons removed? 4. Should sample rates have guidance beyond the current MDD “3 project” statements? Should guidance for larger OUs consider sampling theory for defined confidence levels? 5. Will the SCAMPI BOK meet its intended objective? How do we avoid falling into the “Take a test to be certified” trap similar to PMI and testing certifications that test memorization but not reasoning and judgment ability? 6. Should the entry criteria for Lead Appraisers be strengthened (proactive rather than reactive), particularly in the area of industry experience?

6. Most Wanted Discussion Topics 7. Are guidelines for the ADS Level 4/5 “Additional Information” sufficient, considering the wide variation in current submissions? 8. Should the SEI support delta appraisals, similar to certification schemes like ISO 9001 appraisals? What are the potential cost savings associated with such a strategy? 9. The quality guru of the 1980s, Dr. Edward Deming, warned against applying SPC to sub-processes by themselves; he believed these actions could lead to optimizing the sub-process, possibly at the expense of the system. What might this suggest when SEI insists that sub-processes are to identified and managed? 10. What QA should the SEI be doing on Level 5 activities? Does it really look at how Level 5 activities build on Level 4 outputs? 11. How much of advanced statistical techniques should be needed for an organization to be Level 4/5 using; e.g., ANOVA, Monte Carlo Simulation, DOE? 12. Should all projects in an organization be required to be using the same advanced statistical techniques and models? How much is enough for any project to use or need?

7. Most Wanted Discussion Topics 13. How many sub-processes should be required for an organization to be a Level 4? 14. Is SCAMPI really for “small” organizations? Why is not in SEI’s view for organizations that are large enterprises? Is SPC strictly being applied to software projects as was intended for example with manufacturing? What are the limitations of SPC in software projects using advanced statistical techniques? 15. What evidence exists that SPC can be used for software processes that are aperiodic in nature? What research exists that shows it is applicable and appropriate? 16. Has the SEI modified its interpretation of Level 4 without sufficient transfer of expectations to LAs and appraised organizations? 17. Why has SEI recently chosen to elevate subpractices in level 4 from informative to normative components? Should the SEI rewrite the model to take into account their latest position on subpractices in Levels 4 and 5? 18. Who must the SEI serve: its sponsor; the worldwide constituency who has embraced the CMMI, both? How well is the SEI doing in any regard?

8. What Panelists Have Been Given • List of topics rated highest by respondents • Textual remarks (attribution removed) from those respondents who gave such input (28 gave textual remarks also or exclusively)

9. Some Curious Remarks • From a non-respondent HMLA at SEI: “You’re titling at windmills.” • “What do you mean by "on a scale of 1 through 5, where 5 is high, just like in appraisals". Where is a 5 point scale used in an appraisal?”

10. General Reactions From Respondents • “Great Idea”, “Really Needed”, “Great Questions”, and “Best of Luck” • So, we have 4 Don Quixote’s with you today on the panel • How many in the community want to ride this horse? • Any SanchoPanza’s? • Hopefully, we’ll all have some fun, learn, and give some needed help for SEI focus areas

11. Rules • One of the top rated discussion topics is selected by the panel chair • One panelist will statydiscussio; others will respond • Starting panelist has 2 minutes MAX to reply • Each of the other there has one minute each MAX • We have time at the end to take 15 minutes of questions from the audience

12. You, the Audience, Will Have Access To • Full set of topics provided to respondents • Inputs given to panelists • This introduction • Summary of responses, etc. Go to www.stt.com Papers & Presentations after March 30th