1 / 17

Adding Richness to Measurement

Adding Richness to Measurement. A Case for Developing and Using Complex Measures. Data is Not Information The Search for Meaning in Measures. Meaning and Methodology - the Medium is the Message Multiple Users/Stakeholders Reporting Versus Quantitative Analysis Measuring Complex Outcomes

vaughn
Download Presentation

Adding Richness to Measurement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Adding Richness to Measurement A Case for Developing and Using Complex Measures

  2. Data is Not InformationThe Search for Meaning in Measures • Meaning and Methodology - the Medium is the Message • Multiple Users/Stakeholders • Reporting Versus Quantitative Analysis • Measuring Complex Outcomes • Enterprise-Level Activities

  3. Complexity - Multiple Stakeholders • The Public • Other Agencies - Entities • Budgeting • Program Funding Outcomes • Policy Decision Outcomes • Evaluating Agency - Vendor Performance

  4. Complex Outcomes • Multiple Players (in a Stovepipe System) – Enterprise Level Activities • Significant Number/Scope of Independent Variables (Limited Control & Influence over Many Primary Outcomes) • Non-linear Processes (starts, stops, shifts, drops, etc.) • Hypothetical Nature of Many Public Sector Activities

  5. What does a typical KPM data chart really communicate? • Standard format is a column chart with a target Line overlay • Expressions are most often a yearly raw Mean • The format often implies variation in “performance” when differences are just normal process variation

  6. What Lies Beneath …

  7. And … in case you think I am making this up … real data from a real agency

  8. And you find out things about your process you didn’t know before …

  9. Aggregate Measures – Selling Points • Primary expression is a single expression “dashboard” indicator (Easy to understand – Easy to track) • Statistically based (mathematically verifiable – easy to audit) – immediately useful for process improvement purposes • Properly constructed indexes can be “de-aggregated” to provide increasingly granular detail back to the original raw datasets • Can combine different types of data into the same measure

  10. Aggregate Measures – Selling Points • Provides a powerful analytic – process improvement tool • Provides more complete, compelling and valid data for budget support • Organizations can use a combination of related operational measures to create a single outcome index (fewer measures, and little need for multiple part measures in the system)Common Indices (Organizational Health, Timeliness of Process, Process Improvement, Customer Service, etc.) • Allows for updating and adjusting measure components without the need for a formal delete/replace (?)

  11. Constructing Aggregate Measures • What is the Outcome? • What are the Primary Components of the Outcome? • What are the Critical Measures of the Components? • Normalizing Data – (removing outliers and translating data into a common unit of expression) • Weighting Components

  12. Outcomes in the Public Sector • Change in Status • Change in Capability • Client/Customer Satisfaction • Process Outcomes – Efficiency/Effectiveness 1. Timeliness 2. Defects (errors, rework) 3. Cost Reduction (savings, avoidance) • DEFINED Outcomes

  13. Normalizing Data • Distribution AnalysisData “shape” (distribution)Removing “outliers” – Special Causes of Variation = (Mean +/- 2 Standard Deviations) Upward and Downward Process Control Limits Baseline-ing • Combining Unlike Data Converting to a common expression - % of target

  14. Weighting Criteria • Contribution to Outcome (High, Moderate, Low) • Criticality (Death, Dismemberment, Skin Rash) • Frequency (Constantly, Sometimes, Rarely) • Data Reliability (.99999, OK, Flip a Coin)

  15. Examples • BOLI (Bureau of Labor and Industries) Composite Timeliness Measure (Wage and Hour, Civil Rights) • Department of Revenue “Taxpayer Assistance” • DHS-Courts-CCF Shared “Permanency of Placement”

  16. Putting it all Together “Effective Discovery – Disclosure of Legal Records” Example Index Components

More Related