Developing Performance Measures at UCEDD: Moving Forward with The Logic Model

1. Developing Performance Measures at UCEDD: Moving Forward withThe Logic Model

2. Contact Information Jon Desenberg Director Consulting Services, Performance Management The Performance Institute 1515 North Courthouse Road, Suite 600 Arlington, VA 22201 Phone: 703-894-0481 - Fax: 703-894-0482 www.performanceweb.org Desenberg@performanceweb.org

3. Measuring Performance for Results If you don’t know where you are going, any road will get you there. –Lewis Carroll

4. Why measure? To Plan? To Comply? To Manage? To Optimize? To Innovate? What gets measured gets done. –Peter Drucker

5. Performance is not about mandates, it’s about management • PERFORMANCE MEASURES ALLOW YOU TO: • STRATEGIZE • COMMUNICATE • MOTIVATE • MANAGE

6. Long-Term Move Towards Performance Performance management is not a new phenomenon—50 years of work in the making to link resources with results • Budget Accounting Procedures Act (BAPA) of 1950 • Planning-Programming-Budgeting System (PPBS), 1965-1971 • Management by Objectives (MBO) 1973-1974 • Zero-Based Budgeting (ZBB), 1977-1981

7. Finally a Move in the Right Direction • GPRA—(1993) Government Performance Results Act • PMA—(2001) President’s Management Agenda • PART—(2002) Program Assessment Rating Tool • Mandates focusing on performance and accountability—bottom line results

8. Program Assessment Rating Tool (PART) • OMB Tool to Evaluate Effectiveness of Program Performance in Budget Submissions for Individual Programs • Designed to evaluate 20% of programs per fiscal year • Methodological • Standardized • Evidence-based…and Transparent • Consistent with Government Performance Results Act (GPRA), 1993

9. Four Sections of PART • Program Relevance Mission and purpose of program • Program Planning Focus on program’s strategic objectives • Program Management Stewardship by front-line managers • Program Results Program accountability to strategic objectives

10. PART Categories • Competitive Grant Programs • Block/Formula Grant Programs • Regulatory-Based Programs • Capital Assets and Service Acquisition Programs • Credit Programs • Research and Development Programs • Direct Federal Programs

11. FY06 Budget: Lessons Learned

12. Lessons Learned • Root Cause of Failure: Poor Plans, Bad Measures, Weak Management • Lack of meaningful, results-oriented performance measures • Failure to address management deficiencies identified by GAO, IG’s, et al. • Ill-defined, conflicting, duplicative program purpose

13. Lessons Learned • Causes for Success • Good alignment towards Performance Plans • Effective management of strategic plans • Strong linkage between successful implementation of strategic goals

14. Making the Grade:OMB’s weighting for each section • Section One: 20% • Section Two: 10% • Section Three: 20% • Section Four: 50%

15. Section One: Relevance • Clear agency mission • Unique contribution • Specific interest, problem or need • Optimally Designed to address specifics

16. Section Two: Planning • Long-Term Performance Goals • Annual Performance Goals • Stakeholder Dialogue • Budget-Performance Integration

17. Section Three: Management • Collection of timely, credible performance information • Manager Accountability • Strict allocation of funds • Budget-Performance Integration

18. Section Four: Program Results • Demonstrable progress towards goals • Achievement of annual goals • Improvement in efficiencies and cost effectiveness • Favorable performance to comprable programs

19. Performance Measurement Issues: Manageable by PART • Outcomes are extremely difficult to measure • Are among many contributors to a desired outcome • Have results that will not be achieved for many years • Relate to deterrence or prevention of specific behaviors • Have multiple purposes and funding that can be used for a range of activities • Are administrative or process oriented

20. Performance Management in Non-profits translates a mission into reality and evaluates the results to all stakeholders • Strategic Level • Measure Progress on Issues • Define & Validate Policy Strategies • Enhance Stakeholder Satisfaction and Support • Operational Level • Drive Change to Implement Organizational Strategies • Ensure Compliance • Achieve Efficiencies • Improve Cycle Time • Individual Level • Improved Morale/Retention • Achieve Clarity of Responsibilities Transparency of Performance to: • Donors • Elected Leaders • Senior Management • Oversight Entities • Employees • Customers • Partners

21. Effective Performance Measures are SMART S PECIFIC MEASUREABLE ACCOUNTABLE RESULTS-ORIENTED (#1) TIME-BOUND

22. Applying SMART End-Outcome: Reduce smoking-related deaths, illness and costs Intermediate Outcome: “Reduce the number of new youth smokers (10-18) by 2% each year” • Results-oriented: Youth smoking is where you can stop the habit before it takes hold and has a lasting health impact • Specific: “number of new youth smokers (10-18)” • Accountable: You have the ability to make it happen • Measurable: “reduce by 2%” • Time-Bound: “per year”

23. Selecting Performance Measures: The Doctor Analogy Outcome Goal (End Outcome): Achieve and Maintain Proper Health and Quality of Life • Measures: Dozens…Pulse, Years Lived; Satisfaction Survey, etc. Objective Goals (Intermediate Outcomes): -Achieve Appropriate Fitness for Age, Gender -Stop Pain in Y -Restore Function of X • Measures: Thousands: Weight, Blood Analyses, Scans, etc.

24. Meaningful Reports tangible and significant accomplishments against objectives; useful for decision makers and other stakeholders Clear Easily understand by managers, partners, other stakeholders; tells clear story Legitimate Accepted or legitimated by those who must use the data Consistent Clear definition and data collection methodology across populations Reliable Captures what it purports to measure in an unbiased fashion Granular Able to detect performance movement Relevant Does not become obsolete too early: sets a pattern or baseline of performance Performance Measure Selection Criteria (Step 4)

25. Technically Available data meets technical adequacy standards such as accuracy, validity, timeliness Adequate Responsible Does not have unintended and undesirable consequences Actionable Indicates what is good or bad, driving desired behavior and the timing of action Accountable Related to direct action or influence of an accountable and attributable entity Balanced One of set of measures providing a clear picture of the full range of performance Vital Measures are mutually exclusive and collectively exhaustive Feasible Reasonable cost and accessibility of data that is not already collected Performance Measure Selection Criteria (Step 4)

26. Performance measurement is a culture shift From • “These measures are draining valuable resources and are a data burden” • “I can’t measure my outcomes I can only measure activities” • “I need these measures because my employees feel it is important” • “You can’t measure my program.” • To • We are committed to tracking measures that matter most. • We are accountable for delivering our outputs and our intermediate outcomes. • We are responsible for our end outcomes.

27. Identifying Characteristics of Effective Performance Management Systems

28. 8 Critical Success Factors for Effective Performance Management Systems • Defining and Aligning to Enterprise Strategy • Developing Meaningful Performance Measures • Increasing Data Availability • Maximizing Data Integrity • Enhancing Performance Reporting • Improving Evaluation and Analysis • Achieving Performance Integration • Driving Decision-Making

29. #1 Defining and Aligning to Enterprise Strategy 1.1 Has clearly defined its mission, vision and values 1.2 Has specific strategies in place to achieve organizational results (based on a SWOT or other strategic landscape analysis) 1.3 All structures (divisions, support functions) are fully aligned with enterprise-wide strategies 1.4 A formal strategic plan is clearly communicated to all employees at all levels of the organization Survey Questions: 1, 2, 3, 4

30. #2 Developing Meaningful Performance Measures 2.1 Reliable measurement and reporting on Outcomes 2.2 Reliable measurement and reporting on Strategies 2.3 Organizational process metrics (Quality, Cycle Time, Efficiency) 2.4 Goals and measures enjoy support and buy-in from internal and external stakeholders Survey Questions: 5, 6, 7, 8 Module Two: Characteristics of Performance Management System

31. #3 Increasing Data Availability 3.1 Data sources are identified and readily accessible 3.2 Data burden is worth the information gleaned Survey Questions: 9, 10 Module Two: Characteristics of Performance Management System

32. #4 Maximizing Data Integrity 4.1 Data is collected, managed, and analyzed in a uniform and consistent manner 4.2 Data is validated or verified through sampling or independent means PI Management Survey Question: 11, 12 Module Two: Characteristics of Performance Management System

33. #5 Enhancing Performance Reporting 5.1 Internal reporting produces information for frontline managers and senior decision-makers on a “real time” basis. 5.2 Has a reporting system that produces comprehensive performance reports that include measures, analysis, trends, suggestions for improvement PI Management Survey Questions: 13, 14

34. #6 Improving Evaluation and Analysis 6.1 For process measures, benchmarks and service levels are evaluated (1-2 year cycles) 6.2 For outcome and strategy measures, program performance is evaluated for “cause-effect” (2-5 year cycles) PI Management Survey Questions: 15, 16

35. #7 Achieving Performance Integration 7.1 INTERNAL Integration: Support services’ contributions (HR, IT, Finance, etc.) to program performance is documented and managed 7.2 EXTERNAL Integration: Performance contributions of multiple contributors in same measurement area are tracked and compared PI Management Survey Questions: 17

36. #8 Driving Decision-Making 8.1 Budgets and investments are made based on clear contributions to performance 8.2 Supply chain partners are held accountable for products and services 8.3 Employee bonuses and pay increases are linked to individual performance evaluations. PI Management Survey Questions: 18, 19, 20 Module Two: Characteristics of Performance Management System

37. Top Five/Bottom Five by “EXECUTION” BOTTOM FIVE INITIATIVES TOP FIVE INITIATIVES • All structures are fully aligned with enterprise-wide strategies (1.3) • Reliable measurement and reporting on strategies (2.2) • Internal integration of support service alignment to performance (7.1) • Data is collected, managed, and analyzed in a uniform manner (4.1) • Goals and measures enjoy support from internal/external stakeholders (2.4) • Publishing a strategic plan (1.4) • Has specific strategies in place to achieve organizational results (1.2) • Organizational process metrics (2.3) • Evaluation of process measures, benchmarks (6.1) • Budgets & investments are made based on contributions to performance (8.1) Module Two: Characteristics of Performance Management System

38. Top Five/Bottom Five by “IMPACT” BOTTOM FIVE INITIATIVES TOP FIVE INITIATIVES • Has clearly defined its mission, vision and values (1.1) • Data burden is worth the information gleaned (3.2) • Publishing a Strategic Plan (1.4) • Comprehensive performance reports (5.1) • Data is validated through sampling or independent means (4.2) • All structures are fully aligned with enterprise-wide strategies (1.3) • Reliable measurement and reporting on strategies (2.2) • Employee bonuses and pay increases are linked to individual performance (8.3) • Budgets & investments are made based on contributions to performance (8.1) • Internal reporting produces real-time data for decision-making (5.1)

39. Module Three Understanding Logic Models

40. What is a logic model? • Logical chain of events providing blueprint for mission achievement • Graphic representation that illustrates the rationale behind a program or organization • Depicts causal relationships between activities, strategies, and end results • Contains goals and performance measures • Integrates various program activities into a cohesive whole • Vehicle for dialogue, planning, program management and evaluation

41. What does a logic model look like? • Graphic display of boxes and arrows; vertical or horizontal • Relationships, linkages • Any shape • Circular, dynamic • Cultural adaptations, storyboards • Level of detail • Simple • Complex • Multiple models

42. Logic modeling is based on mapping and defining linkages between what we do and why we do it . Series of If-Then Relationships IF THEN IF THEN IF THEN IF THEN Have Better Image, Feel Better & Live Longer I Work Out for One Hour Each Day I Will Burn More Calories Than I Consume Lose Fat and Build Muscle Improve My Looks and Health INPUTS OUTPUTS OUTCOMES Assumptions: improving looks = better self image Factors: Health History

43. Clarifying the terms Inputs People and resources required to achieve outcomes Activities/Outputs What the inputs produce End Outcome End goal or ultimate benefit Immediate and Intermediate Outcomes Changes required to achieve end outcome Assumptions: beliefs or evidence that supports your IF-THEN logic Factors: external influences beyond control that effect IF-THEN relationships

44. What is your goal at your annual dental check-up? IF THEN IF THEN IF THEN • Toothpaste • Floss • Tooth brush • Brush twice a day • Floss once a day Have fewer (ideally zero) cavities Remove plaque (decrease plaque in my mouth) INPUTS OUTPUTS OUTCOMES Assumptions: Plaque causes tooth decay Factors: Genetics

45. Logic Model “V” Top-Line Return Alignment Measurement Linkage Bottom-Line Investment

46. Logic Model “V” Performance Dimensions

47. Value Chain Diagram Distribute program grants “So That” Ultimate Program Intent Child violence and abuse can be prevented and detected Output “So That” Child health and development can be protected and maintained Intermediate Outcome “So That” Children can grow into productive citizens and attain their intended impacts on society Degree of Influence by Department Intermediate Outcome High-Level Outcome

48. Global Logic Model: Childhood Lead Poisoning Program Early Activities Later Activities Early Outcomes Later Outcomes If we do… Outreach Screening ID of elevated kids Then…. EBLL kids get medical treatment Family performs in-home techniques Lead source identified Environment gets cleaned up Lead source removed And then… EBLL reduced Developmental slide stopped Quality of life improves And we do… Case mgmt of EBLL kids Refer EBLL kids for medical treatment Train family in in-home detection techniques Assess environment of EBLL child Refer environment for clean-up Definition: EBLL – Elevated Blood Lead Levels Module Three: Using Logic Models

49. Most logic models incorporate the following elements. EFFECT CONTROL Inputs Activities Outputs Intermediate Outcomes Attitudes Behaviors Conditions End Outcomes WHY? HOW

50. Resources used to support activities • Not performance measures • System integration: Link inputs through process to goals and outcomes in: • Budget requests • Human resource plans • Information technology plans Drivers Inputs Activities Intermediate Outcomes End Outcomes Outputs