Measures, Metrics, and Systems-of-Systems

Measures, Metrics, and Systems-of-Systems Bridging a Gap Between Academic and DoD Systems Engineering Terminology Dr. Thomas Holland, Ph.D. Naval Surface Warfare Center Dahlgren, VA NSWCDD-PN-14-00163 Public Release Statement A: Approved for Public Release

Overview Measure or Metric? Systems Perspective and the DoD MOEs & MOPs Systems vs. Systems-of-Systems Military Definitions of SoS In Conclusion Public Release Statement A: Approved for Public Release

Measure or Metric? “By measuring only the amount of merchant tonnage sunk, they did not measure the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have determined a method to measure not only British public opinion, but also grain supply levels and the availability of bread. Another metric worthy of measure might have been American war intentions. These additional metrics assess other aspects of translating their strategy into the desired end state.” Public Release Statement A: Approved for Public Release

Measure or Metric? Measure: a dimension, quantity, or capacity (degree), ascertained by comparison to an accepted standard. • length in inches • weight in pounds • temperature in degrees Celsius (Oxford Dictionary, 2013) Public Release Statement A: Approved for Public Release

Measure or Metric? Metric: a technical system or standard of measurement; a set of figures or statistics that measure results. • Productivity: number of sales calls an employee makes per day. • Efficiency: probability of correct data entry. (Oxford Dictionary, 2013) Public Release Statement A: Approved for Public Release

Systems Perspective & the DoD • In SE, “metric” refers to performance. • “The SoS metrics are collected and analyzed as part of analyses to assess whether the SoS is making progress towards objectives.” (OAD, 2008) • Despite these positive indications, the United States has not declared victory against terrorism nor has the U.S. government satisfied world or domestic opinion with explanations of forward progress in the war. Metrics must be developed to give meaning to our military actions.” (Walker, 2005) Public Release Statement A: Approved for Public Release

Systems Perspective & the DoD We can measure many things. But there are certain measures and collections of measures that are important within the context of our interest. The right metric helps us answer our specific questions. Public Release Statement A: Approved for Public Release

Systems Perspective & the DoD Metric: a set of measures or methods that ascertain the progress a system is making toward achieving its goal. Did you know, “Metric” is not defined in the DAU Glossary? Public Release Statement A: Approved for Public Release

MOEs and MOPs Measure of Effectiveness (MOE): “The data used to measure the military effect (mission accomplishment) that comes from the use of the system in its expected environment. That environment includes the system under test and all interrelated systems, that is, the planned or expected environment in terms of weapons, sensors, command and control, and platforms, as appropriate, needed to accomplish an end-to-end mission in combat.” (Defense Acquisition University, 2013) Public Release Statement A: Approved for Public Release

MOEs and MOPs Measure of Performance (MOP): “System-particular performance parameters such as speed, payload, range, time-on-station, frequency, or other distinctly quantifiable performance features. Several MOPs may be related to the achievement of a particular Measure of Effectiveness (MOE).” (Defense Acquisition University, 2013) Public Release Statement A: Approved for Public Release

MOEs and MOPs Public Release Statement A: Approved for Public Release

Systems vs. Systems-of-Systems System: any interacting or interdependent group of entities that forms a unified and purposeful whole. System-of-systems: any adaptive, emergent, scalable, and autonomous system where the behavior changes continuously in response to influence of stakeholders and is composed of systems that are themselves autonomous. (Fisher, 2006) Public Release Statement A: Approved for Public Release

Systems vs. Systems-of-Systems Public Release Statement A: Approved for Public Release

Military Definitions of SoS System: A functionally, physically, and/or behaviorally related group of regularly interacting or interdependent elements; that group of elements forming a unified whole. Public Release Statement A: Approved for Public Release

Military Definitions of SoS System-of-systems: a set or arrangement of systems that results when independent and useful systems are integrated into a larger system that delivers unique capabilities. Both individual systems and SoS conform to the accepted definition of a system in that each consists of parts, relationships, and a whole that is greater than the sum of the parts; however, although an SoS is a system, not all systems are SoS. Public Release Statement A: Approved for Public Release

Systems vs. Systems-of-Systems SoS: Context not entirely in original systems’ contexts Context for B SoS makes some of the original context internal FOS: Shared Context of systems operating independently SoS may exclude some of the original context Context for A Public Release Statement A: Approved for Public Release

Military Definitions of SoS Academic DoD System: a functionally, physically, and/or behaviorally related group of regularly interacting or interdependent elements; that group of elements forming a unified whole. System: any interacting or interdependent group of entities that forms a unified and purposeful whole. Public Release Statement A: Approved for Public Release

Military Definitions of SoS Academic DoD System-of-systems: a set or arrangement of systems that results when independent and useful systems are integrated into a larger system that delivers unique capabilities. Both individual systems and SoS conform to the accepted definition of a system in that each consists of parts, relationships, and a whole that is greater than the sum of the parts; however, although an SoS is a system, not all systems are SoS. System-of-systems: any adaptive, emergent, scalable, and autonomous system where the behavior changes continuously in response to influence of stakeholders and is composed of systems that are themselves autonomous. Public Release Statement A: Approved for Public Release

In Conclusion Academic definitions of Systems, Systems-of-Systems, Measures, and Metrics have Military Domain Counterparts. A working knowledge of the terms are critical to bridge the audiences when addressing different communities or domains. Do not introduce new terminology, or redefine existing terms, if it is already authoritatively defined. Introduce new terms in light of new concepts only if necessary. Public Release Statement A: Approved for Public Release

In Conclusion Original Revised “By using only the MOP of shipping sunk, they were not able to ascertain the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have developed an MOE representative of the relationship between British public opinion, grain supply levels, and the availability of bread. Another useful MOE might have assessed American war intentions. These MOEs would have assessed the extent to which their capability achieved the desired end state.” “By measuring only the amount of merchant tonnage sunk, they did not measure the progress of their strategy in a holistic manner and lost sight of their ultimate strategic goal. In addition to shipping sunk, the planners should have determined a method to measure not only British public opinion, but also grain supply levels and the availability of bread. Another metric worthy of measure might have been American war intentions. These additional metrics assess other aspects of translating their strategy into the desired end state.” (Walker, 2005) Public Release Statement A: Approved for Public Release

Thank You! Thomas Holland, Ph.D. Naval Surface Warfare Center Dahlgren, VA Public Release Statement A: Approved for Public Release

References ASN(RDA) Chief Systems Engineer, 2006. Naval “Systems of Systems” Systems Engineering Guidebook, Vol. II. Defense Acquisition University, 2013. Glossary of Defense Acquisition Acronyms and Terms. [Online] Available at: https://dap.dau.mil/glossary/Pages/GlossarySearch.aspx?rec=MOE Fisher, D. A., 2006. An Emergent Perspective on Interoperation in Systems of Systems, Pittsburgh: Carnegie-Mellon Software Engineering Institute. Maier, M. W., 1998. Architecting Principles for Systems-of-Systems. In: Systems Engineering. pp. 267-284. Meyers, B. C., Monarch, I. A., Levine, L. & Smith, J. D., 2005. Including Interoperability in the Acquisition Process, Pittsburgh: Carnegie Mellon University, Software Engineering Institute. Moffat, J., 2003. Complexity Theory and Network Centric Warfare, Washington, DC: DoD Command and Control Research Program. Public Release Statement A: Approved for Public Release

References continued Office of the Deputy Under Secretary of Defense for Acquisition and Technology, Systems and Software Engineering, 2008. Systems Engineering Guide for Systems of Systems.Version 1.0 ed. Washington, DC: ODUSD(A&T)SSE. Office of the Deputy Under Secretary of Defense for Acquisition and Technology, 2008. Systems and Software Engineering. Office of the Deputy Under Secretary of Defense for Acquisition and Technology. Oxford, 2013. Oxford Dictionaries. Available at: http://oxforddictionaries.com/ Shah, N. B., Rhodes, D. H. & Hastings, D. E., 2007. Systems of Systems and Emergent System Context. Hoboken, NJ Smith, J. D., 2006. Topics in Interoperability: Structural Programmatics in a System of Systems, CMU/SEI-2006-TN-037, Pittsburgh: Carnegie Mellon Software Engineering Institute . Walker, D. H., 2005. Developing Metrics in the Global War on Terrorism, Newport, RI: Naval War College. Public Release Statement A: Approved for Public Release

Measures, Metrics, and Systems-of-Systems