The Human Side of Metrics

1. The Human Side of Metrics Metricon 1.0 Vancouver, Canada August 1, 2006 Dennis Opacki, CISSP QDSP dopacki@covestic.com Covestic, Inc.

2. How do organizations use metrics? Most purposes cited reduce to: (Hauser & Katz 1998) • Metrics enable firms to take stock of where they are, and help them plan for the future. • Metrics provide estimates of future performance. • Managers use metrics to allocate assets and select strategies. • Metrics form the basis for bonuses and promotions. Even measures derived from automated systems reflect and influence human behavior. slide 2

3. Common metrics pitfalls Goal displacement (Kerr, 1975) • Means becomes the ends in themselves. • People seek rewards to the exclusion of all non-rewarded behavior. Unintended consequences • Metrics affect decisions even if those decisions inadvertently sacrifice long-term benefits. (Hauser & Katz, 1998) • Broken reward systems encourage behavior organizations are trying to discourage. (Kerr, 1975) Passive or active resistance • Staff will find ways to avoid metrics they oppose. (Neal, 2006) • Zone of indifference – “a person can and will accept a communication as authoritative only when… at the time of his decision, he believes it to be compatible with his personal interests as a whole.” (Kerr, 1975) slide 3

4. Factors contributing to failure Evolutionary psychology (Economist, 2005) • Humans are hard-wired not for logic, but for detecting injustice. • People will even forego their own self-interests to punish those behaving unfairly. Social psychology (Neal, 2006) • Locus of control • Internal – Sense of self-reliance – “master of own destiny” • External –Sense of helplessness – external parties are in control • Fundamental attribution errors – incorrectly assuming bad behavior is irrational or malicious • Diffusion of responsibility – bystander effect • Everyone believes someone else is better suited to act. • This can occur if responsible parties lacks authority to affect change. • Reciprocity norm – What’s in it for me? slide 4

5. Factors contributing to failure (cont.) Behavioral Economics (Kahneman, 2003) • Intuition vs. reason – people don’t think very hard • Intuition uses heuristics, prototypes and predictions – fast and parallel. • Reason is slow and single-threaded. • Natural attributes enhance intuition – size, color, valence. • Good/bad can be substituted for any attribute. • Default options have a natural advantage. • Prospect theory • People anticipate emotions associated with changes in state. • People are willing to ignore scope when presented with definite solutions to emotional problems. • Framing • How we present situations greatly influences outcomes. • Multiple frames increase the chance of specific outcomes. slide 5

6. Improving chances for success Select a few good metrics • Focus on attributes and scales that people gauge intuitively - good/bad, like/dislike, percentile. • Use a small set of metrics to keep time pressure from driving cognitive processes from reason to intuition. (Kahneman, 2003) • Survey employees to understand which behaviors are actually being rewarded. (Hauser & Katz, 1998) • Express metrics in dollars; research shows this improves peoples’ ability to assess probability. (Kahneman, 2003) • Use clear metric names and descriptions to increase accessibility of metrics’ traits. (Kahneman, 2003) Present them well • Don’t ignore entertainment value. (Pijpers & Montfort, 2006) • Drill-down is key to getting middle-management buy-in. (Gartner, 2005) • Slicing and dicing – bells and whistles • Use common visualizations to improve mental prototyping. • Give bad news first - peak/end rule (Kahneman, 2003) slide 6

7. Works Cited [1] Buytendijk, F. and Gassman, B. (2005). Management Update: Just Give Me a CPM Dashboard. Gartner whitepaper. [2] Hauser, J. R. and Katz, G. M. (1998) Metrics: You are what you measure! European Management Journal, 16(5). [3] Kerr, Steven (1975).On the Folly of Rewarding A, While Hoping for B. Academy of Management Journal. 18(4). [4] Kahneman, Daniel (2003). Maps of Bounded Rationality: Psychology for Behavioral Economics. The American Economic Review. 93(5). [5] Neal, Russ. (2006). Social Psychology Variables that Contribute to Resistance to Security Assessment Findings. Information Systems Security, 15(1). [6] No Author Cited, (2005). Survey: The concrete savannah. The Economist, 377(8458). [7] Pjipers, G.G.M. and van Montfort, K. (2006). An Investigation of Factors that Influence Senior Executives to Accept Innovations in Information Technology. International Journal of Management. 23(1). slide 7