SoS Pain Points & Implications for MBSE

1. SoS Pain Points & Implications for MBSE Dr. Judith Dahmann The MITRE Corporation Co-Chair INCOSE SoSWG January 2012

2. Topics INCOSE SoS WG SoS ‘Pain Points’ Implementer’s View of SoS SE M&S Opportunities, Challenges and Exemplars Summary and Next Steps Objective is to share information and open a dialog

3. Systems of Systems Working Group The SoS Charter is to SoS Chairs Alan Harding (co-chair) Judith Dahmann (co-chair) INCOSE Connect address: https://connect.incose.org/tb/soswg/ INCOSE Web page: TBD • Promote application of SE to SoS through • Understand and share what we mean by SoS in our various contexts • Develop guidance and advice • Inform and up-skill practitioners • Exert influence on BKCASE, Standards, SE Vision 2025 etc. • Develop the practice of SE for SoS. • Work in partnership with other groups addressing aspects of SoS (e.g. INCOSE WGs, IEEE, NDIA). • Understanding and applying insights from relevant research Published Products New WG Draft SoS Pain Points Paper Planned Work • BKCASE: Contribute to development and support of SoS Section • SoS Bibliography: Develop and share • SoS ‘Pain Points’: Identify key SoS challenges to guide WG activities • Related organization: identify and develop relationships (inside and outside INCOSE) • Webinars: Program to share SoS activities and experiences 1

4. SoS Pain Point Survey Purpose To collect information on major issues or 'pain points' in the area of Systems of Systems operation, management and systems engineering To support planning for activities of the WG • Survey Logistics • Developed during February and March 2012, with several drafts and pretests • Released to the community in April with a cutoff of respondents in Mid-May. • Administered over the internet using KWIK Surveys (www.kwiksurverys.com) • Respondents • 38 survey respondents • 65 SoS ‘pain points’ reported • Respondent location • US (86%). • UK (8%) • Australia (6%) • Respondent SoS experience • Extensive (60%) • Some (37%) • Almost all (94%) are from defense sector • Questions & Analysis • Asked respondents to identify and describe their priority SoS areas of concern: describe up to three 'pain points' including a short name, a description and an example • Results were analyzed, a paper on the results was drafted and circulated for comment

5. SoS Pain Points Survey identified seven ‘pain points’ raising a set of SoS SE questions

6. Implementers View of SoS SE External Environment Presented at IEEE Systems Conference April2011 [1] More familiar and intuitive time-sequenced “wave” model representation Information is thus rendered in a form more readily usable by SoS SE practitioners in the field Representation that corresponds with incremental development approaches that are the norm for SoS capability evolution [1] “An Implementers View of Systems of Systems” Dahmann, Baldwin, Rebovich, Lane and Lowry Concept of Wave Planning was developed by Dr. David Dombkins See “Complex Project Management” Booksurge Publishing, South Carolina: 2007.

7. Implementers View of SoS SE Pain points affect SE throughout the process

8. Initiate SoS Decision has been made to establish an SoS SE organization An entity is responsible for the SoS with SE support to the SoS As an acknowledged SoS, the systems which constitute the SoS maintain operational and management independence At the initiation of an SoS, the information typically available includes initial or first order Statement of top-level objectives for the SoS (SoS capability objectives) Description of how systems in the SoS will be employed in an operational setting (SoS CONOPS) and Programmatic and technical information about systems that affect SoS capability objectives (systems information) Risks are identified when an SoS is launched and mitigation actions are tracked and updated throughout each cycle, along with new risks (Risks and Mitigations)

9. Conduct SoS Analysis Provides analysis of the ‘as is’ and basis for SoS evolution Results provide basis for architecture development and planning for SoS updates

10. Conduct SoS AnalysisM & S Opportunities Wide variety of opportunities Understanding CONOPs………… implications for functional baseline …… performance needs ……… current performance • M&S could be used to explore many more use cases than would be otherwise possible • Potential for uncovering “emergent behavior” and impacts of interdependencies that would not be predictable by other means • Enable more complete exploration of • SoS CONOPs and functionality • SoS requirements space and more potential constituent systems that could contribute to the SoS

11. Conduct SoS AnalysisM & S Challenges Wide variety of opportunities Understanding CONOPs………… implications for functional baseline …… performance needs ……… current performance … and challenges • Impact of governance issues on cross system access to data and validation • Difficulties in clearly identifying SoS capabilities and successfully mapping these to constituent systems • Available constituent system models/data may not lend themselves to reuse • SoS complexity and uncertainty magnify challenges for model validity and predictability

12. Conduct SoS Analysis Exemplars (1 of 2) • Man-in-the-Loop Simulation Experimentation • Assess the vulnerability of current architecture • Assess CONOPS and systems interoperability

13. Conduct SoS Analysis Exemplars (2 of 3) Sandia - SoS Analysis Toolset (SoSAT) “provides logistics analysts with the ability to define operational and support environments of a SoS, ascertain measures of platform and SoS level performance effectiveness, and determine logistics support issues.” http://reliability.sandia.gov/tools_sosat.html

14. Conduct SoS Analysis Exemplars (3 of 3) Mission Level Modeling (MLM)

15. Develop/Evolve SoS Architecture Develops and evolves the persistent technical framework for addressing SoS evolution

16. Develop/Evolve SoS ArchitectureM & S Opportunities Representing E2E flow………,… architecture options …….…… systems implementations Opportunities • Explore many more architectural alternatives than would be otherwise possible • Potential for uncovering “emergent behavior” of potential architectures enable more complete exploration of the: • SoS architecture alternative space; to include its evolution over time • More potential constituent systems that could contribute to the architecture • More detailed exploration of potential constituent systems that could contribute to the SoS architecture, to include • Ease or difficulty with which a constituent could evolve to the SoS architecture over time • 2nd order changes to the constituents caused by their participation in different SoS

17. Develop/Evolve SoS ArchitectureM & S Challenges Opportunities Representing E2E flow………,… architecture options …….…… systems implementations … and challenges • Impact of governance issues on cross system access to architecture data and validation of cross system architecture alternatives • Available constituent system architecture representations may not lend themselves to reuse • SoS complexity and uncertainty magnify challenges for model/architecture validity and predictability

18. Develop/Evolve SoS ArchitectureExemplars (1 of 2) 1 • SERC Research - SoS Interdependency Analysis Analysis of Architecture Alternatives Research: SoS Analysts Workbench Han, Marais & DeLaurentis,2012 Assessing the Impact of Developmental Disruptions and Dependencies in Analysis of Alternatives of Systems of Systems Final Technical Report SERC-2012-TR035, Dec 31,2012 1: Systems Engineering Research Center, DoD University Affiliated Research Center for SE

19. Conduct SoS Analysis Exemplars SERC Research – SoS Architecture Analysis Using Agent-Based Modeling An Advanced Computational Approach to Systems of Systems Analysis and Architecting Using Agent-Based Behavioral Model Final Technical Report SERC-2012-TR021, Sept 30,2012 Agent-based SoS Acquisition and Architecting Model Three dimensional representation of the surface of SoS evaluation for all values of Performance and Affordability

20. Plan SoS Update Evaluates the SoS priorities, options and backlogs to define the plan for the next SoS upgrade cycle

21. Plan SoS UpdateM & S Opportunities Analysis of needs in SoS context…. Assessment of trades……Analysis of interdependencies Opportunities • Understand capability shortfalls in the broader SoS mission context • Identify and explore many more alternative options • Better scrubbing/exploration of alternative space • Enable more complete analysis of impacts of change • Could include simulation based experimentation, to identify areas of uncertainty or risk

22. Plan SoS UpdateM & S Challenges Opportunities Analysis of needs in SoS context…. Assessment of trades……Analysis of interdependencies … and challenges • Depends upon a solid representation of architecture and SoS baseline including dependencies …with the all the challenges discussed above • Understanding dependencies is critical here; hard to model something you don’t understand • May depend on system level data which could be difficult to obtain or incompatible with the SoS context

23. Plan SoS UpdateExemplars

24. Implement SoS SE Update Monitors implementations at the system level and plans and conducts SoS level testing, resulting in a new SoS product baseline

25. Implement SoS SE UpdateM & S Opportunities Opportunities • Address risk when testing is impossible or too costly • Address asynchronous development issues • Employ system simulations as surrogates in testing • Enable more complete testing • Conduct many more test threads than would be otherwise possible in live testing • Make better use of live test time • Help select tests that the SoS SE and PM are willing to invest in live testing because of their importance to the SoS mission when M&S “testing” shows marginal performance in a critical capability • One “opportunity” flows primarily from “SoS SE Update” to “M&S of the SoS” • Feedback from field ops and exercises is used to refine the M&S of the SoS, with the result that M&S better serves “Conduct SoS Analysis” and “Develop/Evolve Architecture “in future waves. Improve testing … and feedback to SE

26. Implement SoS SE UpdateM & S Challenges Challenges Opportunities • Depends upon a solid representation of architecture and SoS baseline including dependencies …with the all the challenges discussed above • Independence of constituent systems may make access to surrogate systems simulations as difficult as coordinated testing • Systems simulations may not be technically suitable for use as surrogates Improve testing … and feedback to SE

27. Implement SoS SE UpdateExemplars • Simulations used to augment defense systems testing One-SAF C0Developer Technical Exchange Meeting http://www.onesaf.net/community/static/web/Vienna/123_OneSAF_CoDeveloper_TEM_InterTEC_v2.pdf PM Magazine, Sept-Oct 1997, Defense Acquisition University

28. Implications of Pain Points for SE and M&S of SoS (1 of 3) • Lack of SoS Authorities & Funding and Leadership • Poses serious governance and management issues for SoS • Recognized impact on ability to implement SE as traditionally applied to products • Similar impacts on M&S based on ability to implement effective crosscutting M&S activities • Constituent Systems • Beyond this, when the systems in an SoS are designed for different purposes and contexts which persist, there are technical issues with aligning systems to SoS architecture and context • Likewise, models, simulations and data for these systems will naturally be attuned to the specific needs of the systems, and may not lend themselves to supporting SoS analysis or engineering

29. Implications of Pain Points for SE and M & S of SoS (2 of 3) • Autonomy, Interdependencies & Emergence  • Complexity characterizes SoS derived from the autonomy (and often complexity) of the constituents, their interdependencies, and the potential for emergent behavior which is often unpredictable • While M & S can aid in representing and assessing these complexities to the degree they are understood, these issues pose challenges for M & S particularly when the understanding is limited and can magnify issues of validation • Capabilities & Requirements • Top level SoS capability needs may be high level and need definition in order to align them with requirements which are supported by constituent systems, which may not be able (or willing) to address them • M & S can support tools to assist in this process, but these too are dependent on a articulation of capabilities in order to effective represent them

30. Implications of Pain Points for SE and M & S of SoS (3 of 3) • Testing, Validation & Learning • The independence and asynchronous evolution of the constituent systems in an SoS, poses challenges for SoS testing, as do the costs and practicalities of creating test environments for large dynamic SoS • M & S provides options clear options for augmenting live test and addressing risk in SoS when testing is not feasible; however, the requires a solid representation of the SoS which can be difficult as discussed in earlier points • SoS Principles • The lack of well understood SoS principles and examples of success hinders progress in SoS SE • Similarly, the absence of a well established base of SoS principles to drive development of M & S and knowledge on how to best apply M & S in this context is an obstacle to effective use of potentially powerful tools

31. Summary and Next Steps Challenges to application of SE to SoS as reflected in the SoSWG pain points Similar challenges when considering SoS context in engineering an individual system Modeling and Simulation can help address these challenges when supporting SoS SE …however, same challenges can also modeling simulation Collaboration between SoSWG and MBSE teams provides an opportunity for progress with increased likelihood of success for both sides