Overview

1. Measuring the Impact of Nonprofit ProgramsNonprofit and Public ManagementCenter University of MichiganDecember 1, 2010Brian DatesDirector of Evaluation and ResearchSouthwest Counseling Solutionsbdates@swsol.org

2. Overview • History • Originally guided by stakeholder requirements • Fulfilled their requirements to funders and boards of directors • Built principally on outputs as related to funding • Funders were happy to get information that justified giving money • Programs were happy to get continuation funding • Concentrated on “numbers” • Funded • Served

3. Overview Compliance rather than a performance management model Organizations did not use the system for management purposes Systems used for management remained obscure Formed an “unofficial” management system Generally devoid of organizational outputs and outcomes Government Performance and Results Act of 1993 (GPRA) Sought to shift the focus of government decision- making and accountability away from a preoccupation with the activities that are undertaken to results of those activities

4. Overview “To provide for the establishment of strategic planning and performance measurement in the Federal Government, and for other purposes “ Required to be submitted to the Office of Management and Budget (OMB) for approval National Partnership for Reinventing Government of 1993 (NPR) “… to make the entire federal government less expensive and more efficient.” National Performance Review

5. Overview Program Assessment and Rating Tool (PART) Introduced in 2002 by OMB as a way of systematizing GPRA outcomes “… to establish a meaningful systematic link between GPRA and the budget process.” Four sections Program Purpose & Design - Weight: 20% Strategic Planning - Weight: 10% Program Management - Weight: 20% Program Results/Accountability - Weight: 50% Four outcomes Effective: 85 – 100 Moderately Effective: 70 – 84 Adequate: 50 – 69 Ineffective: 0 – 49

6. Why Bother? Focuses activity across the organization Aligns strategy with goals with performance Reduces emotionalism as a basis of decision-making Decisions are not made from an emotional perspective Formalizes “gut-level” decision-making Makes improvement possible “If you don’t know where you’re going, you’ll probably wind up somewhere else.”

7. Why Bother? Eliminates the natural tendency to over-accentuate the positive Produces a consistently defined vocabulary for communication Strategy Objectives Indicators Goals

8. Getting Started Organizational Mission Overall statement of purpose ReHabilitation Center: The ReHabilitation Center builds brighter futures for people with disabilities. Strategic plan Document specifying the goals necessary to achieve in order to realize the mission Objectives: Steps to goal attainment Indicators: Metrics tied to objectives Goals: Targets for performance

9. Performance Management Models Provide template for examining performance Outcome Sequence Model Generic term for Logic Model Identifies the algorithm necessary to produce positive outcomes Four components Resources Inputs that drive the process of attaining strategic goals Activities Components of workload that connect strategic goals to resources

10. Performance Management Models Outputs Measures of activities Outcomes Measures of goal attainment

11. Development of the Outcome Sequence Model Procedures Purposes Context Resources Activities Outputs Outcomes - Money - Staff - Volunteers - Equipment - Supplies - Facilities - Laws - Regulations - Funder(s)’ Requirements - Target Population • - Assessment • - Planning • - Intervention • - Distribution • Support • Training • - Number of • Persons • Served • - Number of • Service • Hours • - Number of • Sessions/ • Classes • Number of • Voters • Registered - Change in Knowledge - Change in Skills - Change in Attitudes - Change in Behavior - Change in Condition - Change in Status • Organization • - Community • - System of • Services • Politics • Priorities • Mission • Based • Objectives Revisions Learning Feedback

12. The ReHabilitation Center Outcome Sequence Model Resources Activities Outputs Outcomes • - $25,584,329 • 282 Employees • 7 sites • Persons with • developmental • disabilities • Transportation • services • 509(a)(2) • corporation • Provide • opportunities for • community • participation • Provide • opportunities for • work • Provide • counseling • Improve access to • services • Provide family • supports • Provide access to • services • % of persons • participating in • community • activities • % of eligible • persons with • employment • % developing own • service plan • % of eligible • persons served • # of families • served • # of people with • adequate • transportation • % of persons • feeling lonely • % of persons • employed • continuously • % of persons • achieving 50% of • goals • % of participants • involved in • needed services • % satisfaction • with supports • received • % decrease in • cancelled • appointments Revisions Learning Feedback

13. Performance Management Models Challenges Implied causality rather than correlation or synchronous goal attainment Other organizational factors get lost/left out Finding a place for other types of strategic goals, e.g., organizational process Part of strategic plan Not related directly to outcomes for persons served Determining resources for various types of outcome Solutions Include as part of overall model Include as separate sections

14. The ReHabilitation Center Outcome Sequence Model (Organizational Processes) Resources Activities Outputs Outcomes • (?)$ • -(?) Sites • - (?) Employees • - Persons with • developmental • disabilities • Direct care • staff • - Management • staff • IT system • Identify new • service • opportunities • Improve • environmental • safety • Serve persons in • a timely manner • Develop • information • systems • Establish • measures to • analyze data • # of opportunities • identified • # of identified • hazards reduced • Average time from • referral to intake • % of employees • with access to • EMR • # of trend reports • produced • # of new services • implemented • # of reportable • incidents • # of persons • engaged (entering • services) • % reduction in • time for record • keeping • # of new • performance • improvement • plans Revisions Learning Feedback

15. The ReHabilitation Center Outcome Sequence Model (Staff Development) Resources Activities Outputs Outcomes • (?)$ • -(?) Sites • - (?) Employees • - Management • staff • Service delivery • staff • Recruit and retain • quality personnel • Improve staff • professional • competencies • Increase • employee • satisfaction • Increase • utilization of • technology • Align • organizational and • individual KPI’s • % of interviewed • candidates • meeting standards • % of staff • completing • training • Average • employee • satisfaction score • % of employees • trained in • technology • utilization • % of reviews with • aligned KPI’s • % employee • turnover • % of staff • demonstrating • competency • - % employee • retention • % displaying • competency • % attained on • reviews with • aligned KPI’s Revisions Learning Feedback

16. The ReHabilitation Center Outcome Sequence Model (Finance) Resources Activities Outputs Outcomes • (?)$ • -(?) Sites • Finance • employees • - Development • employees • IS employees • - IT system • Maintain financial • stability • Increase revenue • sources • Improve efficiency • Report of Days • Cash on Hand • Report of • Operating Margin • Report of Current • Ratio • # of additional • revenue sources • Create operating • surplus • Cash on Hand • >60 Days • Operating Margin • >3.0 • Current Ratio • >2.1 • Increase in net • revenue from • additional sources • Increase in net • income Revisions Learning Feedback

17. Performance Management Models Comments on the outcome sequence model Built to emphasize outcomes to service recipients Consumers Constituency Little literature for reference regarding goals not directed at consumer outcomes Implies causality

18. Performance Management Models Balanced Scorecard Based on for-profit system Emphasizes four interconnected business perspectives Financial Customer Internal business process Learning and growth

19. Performance Management Models Financial What do our financial stakeholders expect or demand? Customer Who are our target customers, what are their expectations, and what is our value proposition in serving them? Internal Process At what business processes must we excel to drive value for our customers? Vision and Strategy Employee Learning and Growth How do we align our intangible assets to improve our ability to support our strategy?

20. Performance Management Models Mission Customer Whom do we define as our customer? How do we create value for our customer? Internal Process To satisfy customers while meeting budgetary constraints, at what business processes must we excel? Financial How do we add value for customers while controlling costs? Vision and Strategy Employee Learning and Growth How do we align our intangible assets – people, systems, and culture – to execute our strategy?

21. Performance Management Models • Customer perspective

22. Performance Management Models • Business process

23. Performance Management Models • Learning and growth

24. Performance Management Models • Financial

25. Performance Management Models Comments on the balanced scorecard Indicators include both outputs and outcomes Avoids need to distinguish strategic approach Outcomes may get lost in the process Does not imply causation Considers role of confounding variables Requires in-depth analysis Translates well into indices Weighted or unweighted indicators

26. Performance Management Models

27. Challenges Identifying indicators Derive from strategic goals and objectives S.M.A.R.T. Model Specific Must answer “who, what, when, where, and/or which” elements are to be included Operationalize the definition General goal: Recruit and retain quality personnel Specific goal: % employee turnover Measurable Must answer “how much” and/or “how many” Achievable Consider “changeability” with regard to reporting periods Monthly to quarterly

28. Challenges Consider extent to which the organization has control over the goal Consider the degree to which data are available Relevant Ask whether the indicator is indeed a reflection of the strategic objective it is intended to measure Time-Bound Must consider the extent to which data are available for the time period in question Easy or imposed indicators Do not use if they are not employed to manage the organization Separate reporting from performance management functions

29. Challenges Incorporate reporting function into performance management Examine the goal of “Manage reporting of information to [stakeholder(s)].” Add leading indicators Most indicators are lag indicators Indicate accomplishments in the past Need to identify indicators that tell the organization what the likely future is Generally developed over time Invaluable in making organizational decisions actively rather than reactively

30. Challenges Setting targets for performance Start with existing benchmarks Results of studies National trade organizations Stakeholder requirements (Be careful!) Consider forming a benchmarking collaborative Examine internal archival data Conduct a one-year internal benchmarking study

31. Challenges Leadership commitment Sustainability of effort Failing to make decisions on data Using “slam dunk” indicators for window dressing Using information as a “gotcha” Commitment to organizational teaching All levels of the organization should have some training in performance management and in particular, the specific system being used All levels should be included in designing the system

32. Challenges All levels should receive the results Managing the data collection burden Focus on information which is collected in the course of doing business Create organizational procedures to institutionalize data collection which in not part of the ongoing procedures Avoid measuring too much Adds a burden which inhibits Measurement Review Improvement processes

33. Challenges Measuring too little Information collected does not fulfill management needs Breaking the “dichotomy” of business versus non-profit “If I wanted to deal with numbers, I’d have been an accountant.” “We’re a service agency, not a business.”

34. Thinking Ahead Expansion of the financial perspective Efficiency Average cost of serving a consumer Productivity Average number of consumers served per monetary unit Cost effectiveness Average cost of producing a specified outcome How much does it cost to develop one consumer eligible for work? How much does it cost to achieve meaningful change in one consumer?

35. Thinking Ahead Product or function earning power What is the net gain from products or functions of the organization? Does good performance in a program area attract money? What is the added likelihood of attracting additional funds to sustain or expand? What is the added monetary value? Does a high performing organizational function attract money? Development department Performance management system

36. Thinking Ahead Linking strategic planning and performance management to individual performance Completes organizational alignment Caution: Must use individual indicators germane to the position and related to the organizational indicator, not the organizational indicator

37. Next…? Performance Improvement Systems

38. Resources References Hatry, Harry. (2007). Performance measurement: Getting results (2nd ed). Washington, DC: Urban Institute Press. Kaplan, R. S., & Norton, D. P. (1992). The balanced scorecard: Measures that drive performance. Harvard Business Review, 70 (1), 71-80. Kaplan, R. S., & Norton, D. P. (1993). Putting the balanced scorecard to work. Harvard Business Review, 70 (5), 2–16. Niven, Paul R. (2008). Balanced scorecard step-by-step for government and non-profit agencies (2nd ed). Hoboken, NJ: John Wiley & Sons, Inc.