Foundations for Measuring Effectiveness

1. Foundations forMeasuring Effectiveness Major Richard ‘Kelly’ Bullock Dr Richard F. Deckro Department of Operational Sciences Air Force Institute of Technology Wright-Patterson AFB, OH USA 30 August 2005

2. Research Motivation • Accurately predicting outcomes and assessing progress has challenged leaders since the earliest use of national power to achieve political aims • Especially true of military power • Many theories on how to employ national power to achieve desired end-states • Douhet, Jomini, Trenchard, Mitchell, Schelling,… • Effects-based Operations • Not new, but efforts to ‘institutionalize’ EBO concepts are • US Joint doctrine and service doctrine, particularly USAF doctrine, has undergone change to reflect EBO concepts • History has shown theory is of little value if not supported by an empirically feasible measurement method One accurate measurement is worth a thousand expert opinions. – ADMIRAL GRACE HOPPER, 1906 – 1992

3. Patton on MOEs… Discussion between General George Patton and General Orlando Ward during WWII: “How many officers did you lose today?” asked Patton. “We were fortunate,” Ward replied. “We didn’t lose any officers.” “Goddamit, Ward, that’s not fortunate! That’s bad for the morale of the enlisted men. I want you to get more officers killed.” A brief pause followed before Ward said, “You’re not serious, are you?” “Yes, goddamit, I’m serious! I want you to put some officers out as observers,” said Patton. “Keep them well up front until a couple get killed. It’s good for enlisted morale.”

4. Overview • Background • Measurement Concepts • Measurement Theory • Application of Measurement • Effects-based Operations • Effects • Effectiveness Measurement Foundations • Definitions • Concepts • Framework • Takeaways

5. BackgroundMeasurement Concepts • Measurement is the objective representation of real-world objects, processes, and phenomenon • Measurement captures information through attributes • Attributes may not be directly observable • A real-world system is defined by the attributes chosen to represent it • Measurement is an abstraction • Measurement is the process of assigning symbols to a system attribute such that the assigned symbols reflect the underlying nature of the attributes

6. = source for potential error BackgroundMeasurement Concepts validity – how well a measure reflects what it was supposed to represent amplitude – how well a measure represents higher order constructs reliability – how consistent or repeatable the measurement process is System Measure Assessment Reasoning & Mathematics Attributes Numbers Abstract Object, Process, or Phenomenon Insights & Information Empirical Numerals random – ‘noise’ variation from any source impacting the system observational – oversight of a key system attribute or using the wrong measure for a system attribute systemic – (bias) derives from construction of the measure or definition of the measurement process Statistical Theory – concerns making inferences from data Measurement Theory – link between data and real-world To measure is to know. – LORD KELVIN, 1824 – 1907

7. BackgroundMeasurement Theory • Measurement Theory is a branch of applied mathematics that attempts to: • Describe, categorize, and evaluate the quality of measurements • Improve the usefulness, accuracy, and meaningfulness of measurements • Representational view of measurement built upon 3 theorems Set Y Set X • Representation – Proves the measure f preserves the relationships in X after mapping into Y (i.e. model validation) • Uniqueness – Characterizes the family of measures that preserve the real-world relationships (i.e. scale type) • Meaningfulness – Shows resultant is invariant to scale change for measures meeting the uniqueness condition E 1.2 A 3.1 2.5 8.9 B 5.4 3.7 C 6.2 D measurement Nominal Ordinal Interval Ratio Absolute

8. BackgroundApplication of Measurement • Application of measurement is… • an art • the mechanism for extracting information from empirical observation • within a specified context • the translation of behavior into a set of ‘vital signs’ indicating variations in behavior • Application guidelines • Frameworks • Vertical • Horizontal • Effectiveness vs. Efficiency • Measure Types & Characteristics ...things are to you such as they appear to you and to me such as they appear to me... – PROTAGORAS, 485 – 421 B.C.

9. AttributeZ Attribute2 Attribute1 BackgroundFrameworks Measure of Effectiveness (MOE) Objective31 Effectiveness Objective32 Objective21 Vertical (or Hierarchical) Structure Objective33 Objective22 Objective11 What? Fundamental Objective Objective23 Objective34 Objective12 Objective24 Objective35 Objective1M Objective36 Objective2N Sub- Objective O1 Sub- Objective O2 Sub- Objective OX Objective3P Why? Value V1 Value V2 Value VY How? Horizontal Structure Efficiency Measure of Performance (MOP) We must make the important measurable, not the measurable important. – ROBERT MCNAMARA, 1916 –

10. BackgroundMeasure Types & Characteristics Strategically-linked – traceable to fundamental objective; responsive to change and provides indication of how much change can be attributed to an input Timely – able to be collected and processed in a timeframe needed to be relevant within the context Objective – 1) Collection: easy to understand, same regardless of the assessor, same under similar circumstances; ‘face-value’ or whether the measure logically represents what it is supposed to represent 2) Interpretation: unambiguous interpretation; distinguish between desired and undesired consequences Economical – Collection and processing should provide benefits that off-set the burden of measurement activities Complete – Measures should address all areas of concern in enough detail to discern reasons for differences in actual and expected system results Measurable – hold for Measurement Theory conditions; within a given context if the measure can be feasibly obtained with available resources (i.e. operational )

11. BackgroundEBO Overview • Effects-based Operations (EBO) • Theory for the employment of capabilities in dynamic and uncertain environments in a manner to best attain objectives • Provides a conceptual framework for determining the integration and application of capabilities to achieve specific effects to influence the environment of interest yielding desired outcomes • Key tenets • Focus on end outcomes • Reduced emphasis on weapon systems • De-emphasis on destruction as a sole means of achieving effects “You begin with the objectives that indicate what you really care about. Then you follow simple logical reasoning processes to identify the mechanisms by which the objectives can be achieved. Finally, for each mechanism, you create alternatives by asking what control you have over that mechanism.” Keeney, Ralph, Value Focused Thinking: A Path to Creative Decision Making, Harvard University Press, Cambridge, 1992. Not everything that can be counted counts, and not everything that counts can be counted. – ALBERT EINSTEIN, 1879 – 1955

12. Effects-based Operations measures measures measures measures measures measures measures measures measures measures measures measures Planning Employment Assessment BackgroundEBO Overview • Planning – focus on desired end-state; develop strategy (who, what, where, why, when, and how) • Measures of Effectiveness – tie pillars together; provide feedback on strategy progress towards desired end-state • Assessment – determining if the intended effects were achieved and if they are shaping the desired outcomes • Employment – efficient employment of capabilities; increased emphasis on non-lethal means

13. Indirect Sequential Cascading Intended Positive Functional Indirect Parallel Intended Positive Functional Indirect Sequential Cumulative Intended Positive Systemic Indirect Parallel Cumulative Intended Positive Systemic Indirect Sequential Cascading Unintended Positive Functional Indirect Parallel Intended Positive Functional Indirect Sequential Cascading Intended Positive Functional Direct Parallel Intended Positive Physical Direct Parallel Intended Positive Functional Direct Parallel Intended Negative Psychological Direct Parallel Intended Positive Psychological Direct Sequential Intended Positive Functional Indirect Parallel Intended Positive Psychological Effect Effect Effect Effect Effect Effect Effect Effect Effect Effect Effect Effect Effect Indirect Unintended Negative Psychological Indirect Unintended Negative Psychological Effect Effect Indirect Unintended Positive Functional Indirect Cascading Unintended Negative Psychological Indirect Unintended Negative Psychological Effect Effect Effect Mechanism BackgroundEffects Overview • Effect • System state change • Effect • Consequence or result of a particular action Tactical Strategic Operational Input Input Input Input Input Input Input System Input Behavior Input Input Input Input Input Input System Boundary

14. ApproachDefinitions • DEFINITION 1: A SYSTEM is a set of elements where relationships exist between the elements and the SYSTEM has a purpose or normative behavior. • DEFINITION 2: A system ELEMENT, or SUBSYSTEM, is a system providing functionality or support to a parent system. • DEFINITION 3: A MODEL is a formal image of an empirical structure. • DEFINITION 4: An ATTRIBUTE, or NODE, is a characteristic, feature, or property of a system that is directly or indirectly observable. • DEFINITION 5: A MEASURE is a model of an attribute. • DEFINITION 6: A MEASUREMENT, or observation, is a particular manifestation, or instantiation, of an attribute. • DEFINITION 7: A system STATE is a particular instantiation of all system attributes, or state variables, at a particular point in time. • DEFINITION 8: An EFFECT is a system state change. • DEFINITION 9: EFFECTIVENESS gauges the magnitude of a system state change. • DEFINITION 10:An END-STATE characterizes the desired measurements for all system attributes, or state variables. The beginning of wisdom is calling things by their right names. –CONFUCIUS, 551 – 479 B.C.

15. ApproachProduct Structure Product Structure Development Process Empirical System • System Identification • Determines system boundary • Covers all pertinent aspects of desired end-state • Sub-system Identification • Decomposes into smaller parts • Only relevant sub-systems are needed • Sub-system Importance • Discerns relative importance among selected sub-systems • Amounts to weighting each sub-system Formal System (model or product structure) • Attribute (Node) Identification • Characterizes each sub-system by its salient features • Only attributes relevant to context are needed sub-system weight = 1 System • Attribute (Node) Importance • Discerns relative importance among selected attributes (nodes) • Amounts to weighting each attribute (node) Sub-system1 Sub-systemn attribute weight = 1 Attribute1 Attributem • Measure Development • Quantification of attributes (nodes) • Constructed measures may be needed • Measures should hold for metric properties Measure1 Measureq Measurem = 1 measure

16. ApproachConcepts Time t t = T t = TEnd-state t = 0 Element (Subsystem) ai Effect (of inputt = 0 at t = T) Attribute (Node) Empirical System A = < a1,…, an > Empirical System A Link Input Mechanism System xA xA xA xA Elements (Subsystems) Formal System (or Model) xA = < x1, x2,…, xn-1, xn > Formal System (or Model) xA x1 x2 xn-1 xn x1 x2 xn-1 xn x1 x2 xn-1 xn Attributes (Nodes) 1 2 n-1 m 1 2 n-1 m 1 2 n-1 m Measures ά1 ά2 άn-1 άm ά1 ά2 άn-1 άm ά1 ά2 άn-1 άm Measurements (Observations) Measurements (Observations) Measurements (Observations) System Measurement xA = < .2, .7,…, .3, .5 > xA = < 0, 0,…, 0, 0 > xA = < 1, 1,…, 1, 1 > Effectiveness (of inputt = 0 at t = T) Count what is countable, measure what is measurable, and what is not measurable, make measurable... – GALILEO GALILEI, 1564 – 1642

17. Takeaways • Measurement turns real-world system behavior into a set of ‘vital signs’ for the purpose of monitoring the system • Measurement Theory provides a robust basis for turning qualitative observations into quantitative data • Effectiveness vs. Efficiency • Measure of Effectiveness (MOE) • External measure • Invariant to means of achievement • Are we doing the right things? • Measure of Performance (MOP) • Internal measure • Coupled to means of achievement • Are we doing things right? • EBO is an instantiation of an existing, structured, decision-theoretic framework (VFT) for approaching problems with abstract or ill-defined objectives • An effect is a system state change • Effectiveness gauges the magnitude of the change

18. Questions &Discussion A prudent question is one-half of wisdom. –SIR FRANCIS BACON, 1561 – 1626