CBS Process WG

1. CBS Process WG

2. David Carney (chair) M. Al-Said (scribe) Tony Jordano Kyung Whan Lee Jeffrey Poulin David Klappholz Glenn Berg Jongmoon Baik Rami Razouk George Huling Tim Spinney Steve Cross Mike Moore Co-chairs: Betsy Clark Dan Port Tricia Oberndorf Participants

3. Working Approach • Homework: come in with 3 issues • Brainstormed, collapsed, clustered • Prioritized by importance (practitioner & researcher) & difficulty • Formulated process-oriented statements • Discussed prospective CeBASE Top 10 List

4. Issue 1: CBS Lifecycle Models and Processes - 1 Is there a new lifecycle model for CBSs? • Can one (or more) of the existing lifecycle models be tailored to CBSs? How do we define a CBS life-cycle process?

5. Issue 1: CBS Lifecycle Models and Processes - 2 Both of these questions in light of: • A requirements-driven (COTS-supported) vs. COTS-driven paradigm • Strategies for handling different kinds of market segments • CBS time to market pressures • Level of service quality needs • …. How do you identify [-> metrics group] and factor in multiple CBS cost drivers?

6. Issue 2: Post-Deployment Process In what ways does the CBS post-deployment process differ from: • the CBS development process • the custom post-deployment process? In what ways do the differences affect cost estimation for CBSs?

7. Issue 3: Release Planning What are the process implications of planning for system releases where COTS product: • releases are not synchronized with each other • releases are not synchronized with your system • end-of-life occurs during your system life Planning includes cost estimation, scheduling, determining system release content, etc.

8. Issue 4: Assessment, Evaluation & Testing Process How and when do you assess, evaluate, and test: • COTS products • COTS-based systems “When” includes timing (e.g., multiple points in the process) and system-independent certification of COTS products. “How” includes process, techniques, tools, roles, and metrics. Related to determination of CBS requirements.

9. Issue 5: COTS Market and Identification What are the processes for • Acquiring and maintaining market knowledge • Analyzing the forces in a market segment • Forecasting trends in both market segments and specific products • Obtaining & disseminating COTS product information and product-specific experiences • Establishing and maintaining vendor relationships Both the active (e.g., influencing) and passive aspects of these should be considered.

10. Issue 6: Surprise! How does a CBS process accommodate/address large discontinuities stemming from COTS product surprises? Surprises result from releases or patches: • that are not backward compatible • that fail to perform • whose fundamental properties change Exacerbated by inadequate documentation. • [-> architecture issue concerning product characterization]

11. Issue 7: Organizational Assessment Capability How does an organization assess its capability to be successful with CBSs? How does an organization improve its CBS capability? This includes such issues as: • relationship between CMMI and CBS processes • metrics for CBS process maturity Applies to capabilities of CBS acquirers, developers, and maintainers.

12. Other Issues • Dealing with extra features • Managing customer expectations (post-selection) • World-class SE organizations becoming COTS integrators • Licensing • Systematic way of deriving system challenges • In-place transition of baselines for uninterruptible systems • Metrics for degree of product & system risk

13. COTS Process Issue Priorities –1a

14. COTS Process Issue Priorities –1b

15. COTS Process Issue Priorities –2a

16. COTS Process Issue Priorities –2b

17. COTS Process Issue Priorities –3a

18. COTS Process Issue Priorities –3b

19. Release, Surprise Votes

20. Post-deployment, AE&T Votes

21. Assessment, COTS Id. Votes

22. COTS-supp/–driven Votes

23. Priorities (Diff. = Imp.) 8

24. Priorities (Diff/Imp = 1) 8

25. Prospective CeBASE Top 10 General reactions: • Use of list as guidance • Good (only?) as top 10 list of things CeBASE will research • Use of unfamiliar and imprecise terminology • Is it valid to have hypotheses on this list?

26. Specifics -1 1. Valid within a limited scope only. 2. What else is new? So does everything else! • Add: “…. Just like traditional” • Or “Cost & schedule overruns are as common with COTS as with custom development and they (CBSs) can cost as much.”

27. Specifics -2 3. What is “CBS type”? • Domain? • COTS-driven vs COTS-supported? • COTS product type? • …..? 4. Should be studied, but we don’t believe the hypothesis as posed: • How can a parametric model account for the situation in which the selected products can’t be integrated?

28. Specifics -3 5. False, this is one of many parameters that influence the estimate. • Prefer: ” Architectural mismatch will affect CBS cost.” 6. “ …just as with traditional systems; exaggerated with COTS products.” • Change to “CBS post-deployment costs dominate …”

29. Specifics -4 7. Intuitively agree; defects will be in mismatches between products, not something visible by inspection of line of code. A (ADDITION): Writing glue code requires greater skill than traditional coding. Often need to discover lots of things.

30. Specifics -5 8. Agree with “frequently far worse than linear”, question use of square; heavily dependent on other factors (e.g.; use of standards, quality of products, degree of integration) 9. Question “twice”. • Replace with“Risk specific to CBSs must be managed; CBS risks (and their mitigations) are different; risk doesn’t disappear.” • Current statement is not a useful hypothesis. 10. Disagree with percentage; agree vaporware is one of many factors.

31. Conclusions Process is foundational for CBSs. Risk management is still key - and will be different for CBSs. It was cold, but we had fun anyway!