Toward an Ontology for Measuring Systems Engineering Return on Investment

1. Toward an Ontology for Measuring Systems Engineering Return on Investment Advancing the knowledge of systems engineering Eric Honour +1 (850) 479-1985 ehonour@hcode.com Dr. Ricardo Valerdi +1 (617) 253-8583 rvalerdi@mit.edu Toward Ontology for SE-ROI Version

2. Topics • SE-ROI Project • “Ontology” concept • COSYSMO work toward ontology • Categorization from current standards • Future directions Toward Ontology for SE-ROI

3. Systems Engineering Return on Investment Summary of the SE-ROI Project Toward Ontology for SE-ROI

4. Traditional Design Risk SYSTEM DESIGN DETAIL DESIGN PRODUCTION INTEGRATION Time TEST Risk Saved Time/ Cost “System Thinking” Design Time Heuristic Claim of SE • Better systems engineering leads to • Better system quality/value • Lower cost • Shorter schedule Toward Ontology for SE-ROI

5. Greater SE Effort led to better cost/schedule compliance and better predictability Value of SE – 2004 Results Problems/challenges: • Quantitative data on SE not available in program databases • All data points were subjective • Detailed structure not available Value = 1.0 if program met cost/schedule goals Each dot is one program, with sizes between $1M and $6.5B Source: SECOE 01-03INCOSE 2003 Toward Ontology for SE-ROI

6. SE-ROI Project • Interviews • Just-completed programs • Key PM/SE/Admin • Translate program data into project structure • Desired Results • Statistical correlation of SE methods with program success. • Leading SE indicators that can be used during a program. • Identification of good SE practices under different conditions. • Program characterization • Program success data • SE data (hours, quality, methods) Statistical correlation Toward Ontology for SE-ROI

7. “Ontology” Concept What is this word and how does it relate to systems engineering? Toward Ontology for SE-ROI

8. Current State of SE Definition • Fragmented by domain opinions • Military – DOD/MOD • Space - NASA/ESA • Commercial products • Aircraft • Automobiles • Nuclear waste • Process engineering • Tool vendors • Etc. Etc. Etc. • Fragmented by discipline opinions • Technical leaders • System architects • System analysts • Requirements engineers • Operations analysts • Design engineers • Fragmented by standards • ANSI/EIA-632 • IEEE-1220 • ISO-15288 • CMMI • MIL-STD-499C Toward Ontology for SE-ROI

9. Ontology “…a branch of metaphysics concerned with the nature and relations of being” aesthetics interfaces functions structure inputs components methods outputs categories understanding • POSIWID – The Purpose of a Systems Is What It Does • Jack Ring • The purpose of systems engineering is different in the eyes of different people, because they perceive different actions/results from SE Toward Ontology for SE-ROI

10. Purpose of this Paper • Explore the variety of what people see in SE • Formulate some general categories • Interpret historical SE effort data • Provide a structure for the data-gathering in the SE-ROI project. Toward Ontology for SE-ROI

11. COSYSMO work toward ontology An exploration of ontology as performed in the COSYSMO project Toward Ontology for SE-ROI

12. COSYSMO Effort Profile • How is Systems Engineering effort distributed over time? Toward Ontology for SE-ROI

13. Effort Distribution Across ANSI/EIA 632 Fundamental Processes Toward Ontology for SE-ROI

14. Systems Engineering Effort Profile Toward Ontology for SE-ROI

15. Categorization from Current Standards The start of an ontology, by identifying the widely-accepted categories. Toward Ontology for SE-ROI

16. Categories in the Standards • Mission/Purpose Definition • Requirements Management • System Architecting • System Implementation • Technical Analysis • Technical Management/Leadership • Verification & Validation CMMI ANSI/EIA-632 MIL-STD-499C IEEE-1220 ISO-15288 Colored boxes on following slides show the terminology used by each standard Toward Ontology for SE-ROI

17. Mission/Purpose Definition • Define the mission or purpose of the new/changed system. • Typically described in the language of the system users rather than in technical language • CMMI • Develop customer requirements • ANSI/EIA-632 • Not included • MIL-STD-499C • Not included • IEEE-1220 • Define customer expectations • ISO-15288 • Stakeholder needs definition Toward Ontology for SE-ROI

18. Requirements Management • Creation and management of requirements • Efforts to define, analyze, validate, and manage the requirements • CMMI • Requirements development • Requirements mgmt • ANSI/EIA-632 • System design • Requirements definition • MIL-STD-499C • System requirements analysis and validation • IEEE-1220 • Requirements analysis • ISO-15288 • Requirements analysis Toward Ontology for SE-ROI

19. System Architecting • Define the system in terms of its component elements and their relationships • Diagrams that depict the system, its environment, components, and relationships • CMMI • Technical solution • ANSI/EIA-632 • System design • Solution definition • MIL-STD-499C • System product technical requirements analysis and validation • Design or physical solution representation • IEEE-1220 • Synthesis • ISO-15288 • Architectural design • System life cycle mgmt Toward Ontology for SE-ROI

20. System Implementation • Development/completion of the system • Specific system-level efforts in the standards are system integration and transition to use • ANSI/EIA-632 • Product realization • Implementation • Transition to use • CMMI • Product integration • MIL-STD-499C • Not included • IEEE-1220 • Not included • ISO-15288 • Implementation • Integration • Transition Toward Ontology for SE-ROI

21. Technical Analysis • System-level technical analysis • Assessment of system performance • Trade-off analysis of alternatives • CMMI • Measurement and analysis • ANSI/EIA-632 • Technical evaluation • System analysis • MIL-STD-499C • Functional analysis, allocations and validation • Assessments of system effectiveness, cost, schedule, and risk • Tradeoff analyses • IEEE-1220 • Functional analysis • Requirements trade studies and assessments • Functional trade studies and assessments • Design trade studies and assessments • ISO-15288 • Requirements analysis Toward Ontology for SE-ROI

22. Technical Management/Leadership • Guiding the engineering teams involved in system design programs • Size/complexity of teams demands leadership • CMMI • Project planning • Project monitoring & control • Supplier agreement mgmt • Process/product quality assur. • Configuration mgmt • Integrated project mgmt • Decision analysis/resolution • Quantitative project mgmt • Risk mgmt • ANSI/EIA-632 • Technical Mgmt • Planning • Assessment • Control • MIL-STD-499C • Planning • Monitoring • Decision making, control, and baseline maintenance • Risk mgmt • Baseline change control • Interface mgmt • Data mgmt • Subcontract mgmt • Technical reviews/audits • IEEE-1220 • Technical mgmt • Track analysis data • Track requirements and design changes • Track performance • Track product metrics • Update specifications • Update architectures • Update plans • Maintain database • ISO-15288 • Planning, Assessment, Control • Decision mgmt • Config mgmt • Acquisition, Supply • Resource mgmt • Risk mgmt Toward Ontology for SE-ROI

23. Verification & Validation • Verification: comparison of the system with its requirements through objective evidence. • Validation: comparison of the system or requirements with the intended mission • ANSI/EIA-632 • Technical Evaluation • Requirements validation • System verification • End products validation • CMMI • Verification • Validation • MIL-STD-499C • Design or physical solution verification and validation • IEEE-1220 • Requirement verification • Functional verification • Design verification • ISO-15288 • Verification • Validation • Quality mgmt Toward Ontology for SE-ROI

24. Future Directions Where is SE-ROI going? Toward Ontology for SE-ROI

25. Project Advisory Group • Group of interested people/organizations • Communicate via web, telecon, meetings • Help define the data organization • Build public interest in the project • Provide access to real programs • View interim (protected) data as it develops Current members come from: AF Institute of Technology Northrop Grumman DOD Office of Secy of Defense DRS Johns Hopkins Univ MIT The Mitre Corp NAVAIR Raytheon Rand Corporation Rafael Systems & Software Consortium Univ of South Australia USN Chief Engineer For information, see http://www.hcode.com/seroi/ Toward Ontology for SE-ROI

26. Summary • Systems engineering current state of knowledge is fragmented • Broadly-accepted ontology is needed • SE-ROI project needs categorization now • Structure the data to be correlated • Discover leading SE indicators • Identify SE best practices and methods • Categorization across standards helps develop the needed ontology Toward Ontology for SE-ROI

27. Questions? Eric Honour +1 (850) 479-1985 ehonour@hcode.com Dr. Ricardo Valerdi +1 (617) 253-8583 rvalerdi@mit.edu For information, see http://www.hcode.com/seroi/ Toward Ontology for SE-ROI

28. Survey of SE-ROI Knowledge Toward Ontology for SE-ROI

29. Next Steps • Define interview data sheets • Use this categorization • Identify and interview trial projects • Obtain initial data • Evaluate the interview data sheets • Identify and interview projects • Ongoing effort for 2-3 years • Perform statistical correlation work • Ongoing effort for duration of project • Interim reports to participating organizations • Final report expected 2009 Toward Ontology for SE-ROI