Value Focused Metrics 26 ISMOR D.F. Davis A.D. Caldwell ddavis@gmu andrewldwell.uk@osd.mil

1. Value Focused Metrics 26 ISMOR D.F. Davis A.D. Caldwell ddavis@gmu.edu andrew.caldwell.uk@osd.mil

2. VFM Value Focused Metrics is a term stolen from Keeney’s Value Focused Thinking. This use is not unique and a simple Google search brings out 99 other uses. Most of these are in the context of “Agile Human Resources,” “Agile Systems Design.” and “Project Management.” Similar uses have been around since the early 1900s, but mostly since the maturing of Decision Science in the 1960s . Value Focused Thinking by Ralph Keeney came out in 1992.

3. Insight “The purpose of computing is insight, not numbers.” Richard W. Hamming while at IBM labs “The purpose of analysis is insight, not numbers.” Wilbur Payne while at TRADOC Analysis Command “Our purpose is insight, not metrics.” Dave Davis while at ISMOR

4. NATO Code of Best Practices For Command and Control Assessment Metrics should flow from the process, not be separated from it. RTO-TR-081 Page 5-1

5. COBP • The Code of Best Practice has adopted the following five levels of MoM (Measures of Merit): • Measures of Policy Effectiveness (MoPE), which focus on policy and societal outcomes; • Measures of Force Effectiveness (MoFE), which focus on how a force performs its mission or the degree to which it meets its objectives; • Measures of C2 Effectiveness (MoCE), which focus on the impact of C2 systems within the operational context; • Measures of Performance (MoP), which focus on internal system structure, characteristics and behaviour; and • Dimensional Parameters (DP), which focus on the properties or characteristics inherent in the physical C2 systems. RTO-TR-081 Page 5-2

6. COBP This figure implies that metrics are nested. I’m not sure this is always the case. RTO-TR-081 Page 5-3

7. MPICE Sectors Goals Drivers/Capacities Indicators Measures

10. Richardson, H, 2004, MOE and EBO in the Headquarters ARRC, in The Cornwallis Croup IX: Analysis for Stabilization and Counter-Terrorist Operations, The Canadian PeaceKeeping Press (http://www.TheCornwallisGroup.org), P83

11. Richardson, H, 2004, MOE and EBO in the Headquarters ARRC, in The Cornwallis Croup IX: Analysis for Stabilization and Counter-Terrorist Operations, The Canadian PeaceKeeping Press (http://www.TheCornwallisGroup.org), P85

12. Richardson, H, 2004, MOE and EBO in the Headquarters ARRC, in The Cornwallis Croup IX: Analysis for Stabilization and Counter-Terrorist Operations, The Canadian PeaceKeeping Press (http://www.TheCornwallisGroup.org), P87

13. Chernoff Faces

14. Several Issues • Confusion between objectives, activities, inputs, outputs, outcomes, ways, means, ends, ... • What influences what, • Are metrics dependent on the objectives or are objectives dependent on the metrics? • Roll up the metrics or display for mental integration • Whose toy should we use?

15. Bumper Sticker Metrics should be Organic to the Plan

16. Value Focused Metrics Metrics should be organic to the objectives and flow from the plan, not come from separate lists or be tacked on at the end. Value Focused Metrics (VFM) says: First identify your objectives, then determine how each Fundamental Objective can be served by a network or Means Objectives. This is the ‘Model’ of the overall intervention/plan or mission. Once the structure and content of the Model is agreed, then identify what the interveners can do or control – what activities, projects and programs need to be put in place, and what Means Objectives are these expected to influence. Finally, when the Model is enhanced by the activities, then and only then determine what Metrics should be associate with both Fundamental and Means Objectives. Keep track of resources, but they are controls on the activities, not metrics.

17. Definitions • Some definitions from Keeney (and some from Davis): • Objective (Keeney, p34): “..statement of something that one desires to achieve.” • Context • Object • Preference • Goal (K60) (An objective with values, standards or thresholds.) • Fundamental Objectives: “Just Because” • Means Objectives: “In order to accomplish X” • Attribute (K100): “The degree to which an objective is achieved..” • Metric (Davis): The definition of an attribute • Measurement (Davis): The collected value of a metric.

18. What the METRICS will mean Measurements associated with Metrics on the Fundamental Objectives indicate overall Outcomes, or mission achievement. Are we accomplishing the overall effects that we are there to create? Measurements associated with Metrics on the Means Objectives indicate how well the plan is being accomplished. Are we using the right ways and means to accomplish our goals? Metrics on Fundamental Objectives are for Mission Achievement. Metrics on Means Objectives are for Managerial Efficiency. BOTH ARE VALID AND NECESSARY BUT they ARE NOT THE SAME THING!!!!

19. The Process • The modeling process starts with developing a Fundamental Objectives Hierarchy (FOH) and a Means Ends Objectives Network (MEON). A MEON is a model of causality, it is meant to show how a Fundamental Objective (FO) can be met. To do so, there are several requirements: • A list of domain elements must be collected – what are the things in this domain that impact or may impact the FO? For the intervention domain we are using the ETL/MPICE/CMPO. This yields CONTENT. • These domain elements must be arrayed in a network showing both causal and correlation linkages. We are doing this using Microsoft Belief Networks (a freeware from MS that implements Bayesian Networks). This process provides us with STRUCTURE. • Once the CONTENT has been arrayed in the STRUCTURE then the theories of change must be added. In the case of Bayes Nets, this theory is one of probabilities. This then gives us VALUE. • Drivers/Activities/Programs/Projects can then be added to represent what the interveners can actually CONTROL. Activities may require RESOURCES. • Many, but not necessarily all, objectives can have attributes, or METRICS attached.

20. Sources of the Elements Catalogs – MPICE Workshops and expert opinion Affinity Lists (Parnell) Old models “The Plan”

21. What's a what • For each element in your list, ask first: • Why is this important? • “In order to do this or that” – Add to list of Means Objectives • “Just because, or That is the reason we are here” – Add to list of Fundamental Objectives • “Its something that we do” – add to list of potential activities • If a Means objective, what does it influence? (potential link) • For every objective, fundamental or means: How would we know that this objective has been accomplished? Candidate attributes or metrics. • Then, of course, ask: What have we forgotten?

22. The System Metrics Fundamental Objective Hierarchy Means Ends Objective Network Metrics Resources Controls, Activities, Programs, Projects

23. Structure - Basic The Yellow node is the Strategic (Fundamental) Objective. The Gray nodes are the Means Objectives linked to represent how we think they support each other.

24. Structure - Influences The Green nodes are activities, things we can actively control

25. Structure - Metrics The Blue nodes are the metrics – the metrics depend on the objectivesand do not form their own, independent system.

26. Value Value depends on classifying the objective’s possible states. In this case Needs can be in either of the states Provided, Marginal, or Significant need persist. Value is completed by assessing the local theories (probabilities) at each node. For example, the probability that Needs will be Provide For GIVEN that Shelter is Sufficient AND Education is Available AND Access to Water is Sufficient is .897. Whereas the probability that Needs will be Provided For GIVEN that Shelter is Available Below Standards AND Education is Marginal AND Access to Water is Marginal is only .526. These theories are based on input assessments by experts.

27. Conclusion Metrics should be Organic to the Plan

28. Questions?