COSYSMO: COnstructive SYStems Engineering Cost MOdel

1. COSYSMO: COnstructive SYStems Engineering Cost MOdel Ricardo Valerdi USC Annual Research Review March 11, 2002

2. Outline • Background on COSYSMO • EIA632 • Approach • Delphi Survey • Delphi Round 1 Results • Analysis/Conclusions • Lessons Learned/Improvements • The Next Step • Q & A

3. “All models are wrong, but some of them are useful” - W. E. Deming Source: www.deming.org

4. What is it? The purpose of the COSYSMO project is to enhance the current capability of the COCOMO II model by accounting for costs that are outside the realm of software engineering by introducing system engineering drivers.

5. The Challenge To develop a preliminary model for estimating the cost impact of front-end System Engineering tasks in the design of software intensive systems. These include system definition, integration, and test activities as defined in standard EIA632.

6. Approach • Begin with front-end costs of information systems engineering • Follow 7-step modeling methodology • Steps 5,6, & 7 (gather data and refine loop) • Use model parameters compiled by TRW, SAIC, Raytheon, and USC/CSE • System size (requirements, TPMs, I/F) • Effort drivers (maturity, cohesion, stability)

7. COCOMO Suite COPROMO COQUALMO COPSEMO COCOMOII CORADMO COCOTS COSYSMO

8. COSYSMO Operational Concept # Requirements # Interfaces # TPM’s # Scenarios # Modes # Platforms # Algorithms COCOMO II-based model Size Drivers Effort COSYSMO Cost Drivers Duration WBS guided By EIA 632 • 7 Application factors • 8 Team factors Calibration

9. EIA632 33 activities organized into 5 groups: • Acquisition and supply • Technical management • Planning process • Assessment process • Control process • System design • Requirements definition process • Solution definition process • Product realization • Technical evaluation

10. Delphi Survey • 3 Sections: • Scope, Size, Cost • Used to determine the range for size driver and effort multiplier ratings • Identify the cost drivers to which effort is most sensitive to • Reach consensus from systems engineering experts

11. 7 Size Drivers • Number of System Requirements • Number of Major Interfaces • Number of Technical Performance Measures • Number of Operational Scenarios • Number of Modes of Operation • Number of Different Platforms • Number of Unique Algorithms

12. 15 Cost Drivers Application Factors (7) • Requirements understanding • Architecture understanding • Level of service requirements, criticality, difficulty • Legacy transition complexity • COTS assessment complexity • Platform difficulty • Required business process reengineering

13. 15 Cost Drivers (cont…) Team Factors (8) • Number and diversity of stakeholder communities • Stakeholder team cohesion • Personnel capability • Personal experience/continuity • Process maturity • Multisite coordination • Formality of deliverables • Tool support

14. Delphi Round 1 23 Surveys returned

15. System Engineering Effort Per EIA Stage Std. Dev. Suggested 5% 15% 15% 20% 20% 15% 5% 5% Stage Supplier Performance Technical Management Requirements Definition Solution Definition Systems Analysis Requirements Validation Design Solution Verification End Products Validation Delphi 5.2% 13.1% 16.6% 18.1% 19.2% 11.3% 10.5% 6.6% 3.05 4.25 4.54 4.28 5.97 4.58 6.07 3.58

16. Delphi Round 1 Highlights (cont.) Range of sensitivity for Size Drivers 6.48 5.57 6 Relative Effort 4 2.54 2.10 2.21 2.23 2 1 # TPM’s # Modes # Scenarios # Algorithms # Platforms # Interfaces # Requirements

17. Two Most Sensitive Size Drivers

18. Delphi Round 1 Highlights (cont.) Range of sensitivity for Cost Drivers (Application Factors) 4 EMR 2.81 2.43 2.24 2.13 2 1.13 1.74 1.93 COTS Legacy transition Architecture und. Platform difficulty Requirements und. Bus. process reeng. Level of service reqs.

19. Delphi Round 1 Highlights (cont.) Range of sensitivity for Cost Drivers (Team Factors) 4 EMR 2.46 2.16 1.91 1.94 1.78 1.84 2 1.25 1.28 Tool support Multisite coord. Process maturity Formality of deliv. Stakeholder comm. Personnel capability Stakeholder cohesion Personal experience

20. Four Most Sensitive Cost Drivers

21. Conclusions “Control the controllables” Not only do we need to better manage requirements, we also need to manage: 1) # of Interfaces 2) # of Algorithms 3) Personnel Capability 4) Level of service requirements, criticality, difficulty 5) Level of understanding

22. Lessons Learned/Improvements • Lesson 1 – There are lots of people and groups • Interested in more precisely estimating system costs • - And they are willing to help do it free. • Lesson 2 – Currently, system engineering effort is • estimated using activity-based costing heuristics • Lesson 3 – When mounting a Delphi, clearly identify • what you are trying to do. • Else, system engineers will attack you with a • shotgun • Lesson 4 – We could use help in developing a betterdesigned template for Delphi instruments

23. The Next Steps • Incorporate suggestions from Delphi 1 • Write report • Get Masters degree …in the future • Data from completed systems will then be • used to statistically confirm or deny initial • ratings • Round 2 of Delphi

24. Special Thanks to: • Advisors Dr. Boehm, Dr. Axelband, Don Reifer • Affiliates Gary Hafen, Tony Jordano, Chris Miller, Karen Owens, Don Reifer, Garry Roedler, Evin Stump, Gary Thomas, Marilee Wheaton

25. Where can I get more info? • COSYSMO Working Group meeting Thursday March 14th 12:00 - 5:00 • valerdi.com/cosysmo

27. Delphi Round 1 Participants