DHHS and Outcome Measurement

1. DHHS and Outcome Measurement Wes Albinger, Contract Services Coordinator Milwaukee County Department of Health and Human Services Contract Administration March 16th, 2006 Presentation will be available at: http://county.milwaukeecounty.org/ContractMgt15483.htm Starting Monday, March 20, 2006

2. Intended “Outcomes” of OM Initiative • Initial: Agencies understand DHHS definition and application of OM concepts. Agencies understand the direction DHHS is headed with Outcome Measurement and their role in that process. • Intermediate: DHHS articulates long term outcomes for each program area. Agency program activities and initial outcomes are aligned with DHHS long term Outcomes identified for each program. • Long Term: Indicators and data collection protocols are established for each program area. OM is part of program and contract requirements.

3. DHHS and Outcomes: WHY? • To clearly communicate expectations (goals and objectives) to provider agencies. • To establish an objective standard which is consistently applied to measure and report program impact. • To provide DHHS and Agencies with a meaningful way to report success to stakeholders (e.g. client, provider, funding agencies, etc.).

4. DHHS and Outcomes: WHY? • To promote client choice. • To increase benefits for clients from contract dollars. • To improve program performancevia feedback and program improvement.

5. Purpose of Today’s presentation • This is a multi-year project that will begin simply with establishment of outcomes and a requirement that agencies report to us how they think they are achieving those outcomes. At this point, DHHS will not be holding programs to a specific set of indicators. In subsequent years, contracts may actually include not only the outcomes, but also the indicators and performance goals. • None of this is etched in stone yet -- we have called you together at this juncture specifically to gather your input.

6. FAQ Q: Is DHHS telling us how to run our program? A: No. Outcome Measurement focuses more on the impact of the program than the program itself. While the outcome is affected by the program, we are more concerned with the “end” of the program than the “means” to that end. Basically, we are interested in defining where we want programs to go, but not how to get there. In some instances, external factors prescribe program outcomes or processes. An example would be the new Federal Birth to Three Outcomes which will be required for all BT3 programs. Additionally, the growth in popularity of evidence based program designis putting increasing pressure on programs to adopt proven approaches to service delivery. These will likely play an increasing role in programs.

7. FAQ Q: What role will Agencies play in this process? • A: DHHS will articulate long term outcomes for each program, distilled from estimation of client need, agency practice and capacity, current program requirements, federal, state and other mandates, internal staff, and research. DHHS will solicit information from agencies relating to outcomes, indicators, and data collection methods to help inform this process. Keep in mind that on a program by program basis, the amount of discretionary “slack” in setting outcomes varies greatly. • Throughout the process, DHHS will release information for review and comment.

8. FAQ • Q: What are the implications of these changes for my Agency in the short term? • A: For 2007, we will not be prescribing indicators and performance goals in association with the long-term outcomes. Long term outcomes will be identified for several pilot programs for the FY 2007 RFP. For these programs, you will be asked to describe in detail how your program design and outcomes relate to the long term outcome identified for your program, using successful past performance or a research based justification. Additionally, you will be asked to identify what indicators you are or propose to use to measure the achievement of the outcome, as well as your data collection and or measurement tool(s) (concepts explained shortly).

9. FAQ, implications for short term, cont’d • For those programs without DHHS identified long term outcomes in the FY 2007 RFP, you will be asked to describe your current or proposed program design and outcomes, using successful past performance or an evidence based justification. Additionally, you will be asked to identify what indicators you are or propose to use to measure the achievement of the outcome, as well as your data collection and or measurement tool(s).

10. FAQ, implications for short term, cont’d In other words, tell DHHS: What do you intend to do? (program design) Why do you intend to do it? How does what you are proposing contribute toward the long term outcome? (rationale/justification) In what way(s) do you expect clients to benefit? (short, intermediate, and long-term outcomes) How will you know if you have accomplished this? (indicators)

11. FAQ • Q: What are the implications of these changes for my Agency in the long term? • A: Indicators and data collection protocols for each program area will be developed through an inclusive process. OM will be part of program and contract requirements.

12. FAQ • Q: How will this initiative create work for my agency? • A: This initiative is more focused on reporting differently, versus reporting additional information. The required reporting will be part of your semi annual evaluation reports. • DHHS will seek to minimize the administrative burden.One way this will be accomplished is by soliciting Agency input to determine current capacity and potential technical assistance needs.

13. Example-Agency input • It is worth noting that DHHS has done an extensive review of existing outcome data currently collected to help formulate our thinking and strategy. Many of the examples used in this presentation are actual outcome data already collected; we are looking at the content currently being provided as a starting point.

14. Agency Input You may recall recently completing an Outcome Measurement Survey. These were sent to all agencies with purchase of service contracts. Some notable results, which have already been used to inform our progress on this initiative, are as follows:

15. Survey responses Agencies have widely interpreted definitions of “Outcome”; as such, there are widely varied applications of Outcome Measurement.

16. Survey responses • Most indicated an interest in an outcome measurement presentation. • Most indicated an interest in a formatted spreadsheet for measuring outcomes. • Vast majority are currently reporting outcomes. • Approximately half cited fundor requirements as the basis for deciding to report outcomes. It appears that many agencies are not finding an internal benefit to tracking outcomes, but are doing it because they “have” to.

17. Survey responses • Most already track outcomes for ALL programs. • Most common theme that emerged as “concern” was administrative/cost burden, with the second being data reliability and validity. • Vast majority have developed a logic model or other visual representation of main programs. • Most common methodology employed for measuring outcomes was United Way’s model.

18. Technical Assistance • DHHS will provide technical assistance throughout this process. • Let DHHS know what your technical assistance needs are.

19. On to Outcomes • This is intended to give you a sense of the process DHHS is using internally to determine outcomes, and may be helpful to you as you identify how your agency’s program will work toward a specific long term outcome. • DHHS will be using a methodology which is already familiar to many of you, and enjoys a broad base of support. It has become most familiar to practitioners through the United Way, but it is used more widely than that.

20. Some thoughts onTerminology Throughout this presentation, I will be using specific terminology. The concept behind the terminology is more important; however, because there is a wide use of terminology and interpretation of terminology among agencies, it is important that: • We adopt a common language for discussing Outcome Measurement concepts, and that • We use it consistently.

21. What are Outcomes and Why Measure Them? OUTCOMES- The intended benefit(s) for participants during or after their involvement with a program. You can also think of Outcomes as the OBJECTIVE of the program.

22. Assumptions and Organizational Challenges to implementing OM • Support for OM is based on the assumption that the information produced creates value that exceeds the time and effort to collect it. • The challenge, then, is to make sure that the information that OM produces is useful, and that collecting it is administratively as simple as possible. • DHHS wants OM information to serve useful purposes for all affected stakeholders.

23. Assumptions and Organizational Challenges to implementing OM • Concern about OM also comes from the belief that the outcomes measured are not an accurate or complete reflection of the program. • Our challenge is to identify outcomes which are appropriate. Your input will help

24. An Outcomes “Makeover” The Before • Served350 people. • Hosted10 resume writing seminars. • Provided150 hours of individual career counseling. • 25 staff attended Outcome Measurement training. Notice emphasis on delivery of service, or the program itself.

25. An Outcomes “Makeover” Gee, you sure did a lot, but what happened??? • Did your clients get jobs? • Did your clients learn anything? • Did your clients change their lifestyle? • In general, how were your clients affected or changed?

26. An Outcomes “Makeover” The After • 70% (160 out of 230) of clients over 50 years of age obtained employment . • Of the total, 65% were able to maintain employment for at least 1 year. Notice emphasis on the benefitor change to the person served. No (or little) emphasis on the actual program. In fact, the program can only be inferred from the outcome statement. These statements reflect the OBJECTIVE of the program.

27. What Exactly is An Outcome? Benefits for participants during or after their involvement with a program. Change in behavior Change in knowledge or skills Change in condition

28. Examples of Outcomes • Clients decreased emergency room visits. • Participants significantly reduced alcohol/drug use. • Immigrants were able to secure legal residency and citizenship. • Staff demonstrate proficiency with Outcome Measurement concepts.

29. Outcome or not? • Clients increase community activities. • Clients participate in skill training programs. • Residents feel safer. • In 2004, the Advocacy program assisted 75 individuals. • Clients reduce emergency room use.

30. Task 1: Choose the Outcomes You Want to Measure Hints: • Be thoughtful!! • Define Your Target Population. • Engage staff and clients in the discussion. • Choose meaningful/compelling outcomes. • Write outcomes so “John Doe” will be able to understand. • Develop a Logic Model.

31. Logic Model: A Picture of how a Program Works Inputs Activities Outputs Outcomes $ Staff Outpatient Clinic Referring agencies Individuals with mental illness Outside environment 75 psychiatric sessions 150 drug prescriptions 65 subsidized housing applications 175 crisis calls 80 resume writing/job placement sessions Psychiatric Services Pharmacy Services Nursing Services Housing Assistance 24 hour crisis hotline Vocational Services Reduce level of incarceration of adults with mental illness Reduce level of hospitalization of adults with mental illness

32. Levels of Outcomes In most cases, a program attempts to affect not one, but a series of changes for individuals, each building on the previous one. These can be thought of as the stepping stones toward the ultimate, or long term goal of the program. We can define these as: Initial Outcomes Intermediate Outcomes Long term Outcomes

33. Relationships among levels of outcomes Levels of outcomes can be thought of in terms of a series of interrelated milestones. The achievement of one sets the condition which increases the chances of achievement of the next. In other words, IF “a” happens, THEN “b” is more likely to follow.

34. Relationships among levels of outcomes • Example: IF job seekers learn appropriate job interview skills (initial outcome), THEN they will use appropriate job interview skills in the interview (intermediate outcome). IF they use appropriate interview skills, THEN they will obtain jobs (long-term outcome).

35. Relationships among levels of outcomes Similarly, it can work backwards: if you know the desired long term outcome for your program, you can ask yourself, “what will make this more likely to occur?” Example: Intended benefit (Long Term Outcome) for Employment Program: Clients get jobs. What will make this more likely to occur? Answer (one of many possible answers): Clients learn and demonstrate appropriate job seeking skills. ** There should be a theoretical/logical (research or practice supported) link supporting causality between the various outcomes.

36. Initial Outcomes Initial Outcomes-The most immediate changes and most heavily influenced by the program, and are typically expressed in terms of new knowledge, skills, or attitudes learned through the program.

37. Initial Outcomes Examples of Initial Outcomes: Program: Employment Initial Outcome: Clients learn appropriate job seeking skills. Program: Adult Day Services Initial Outcome: Clients increase awareness of community resources. Program: Outcome Measurement Initiative Initial Outcome: Staff demonstrate proficiency with Outcome Measurement concepts.

38. Intermediate Outcomes Intermediate Outcomes-A link between the program’s initial outcomes and long term outcomes, typically expressed in terms of changes in behavior connected to the new knowledge, attitude, or skill (initial outcome) learned through the program.

39. Intermediate Outcomes Examples of Intermediate Outcomes: Program: Employment Intermediate Outcome: Clients utilize appropriate job seeking skills. Program: Adult Day Services Intermediate Outcome: Clients increase utilization of public and private services in their community. Program: Outcome Measurement Initiative Intermediate Outcome: Staff apply Outcome Measurement concepts to programs.

40. Long Term Outcomes Long Term Outcomes-The ultimate objective of the program, typically expressed in terms of changes in condition or status. When defining Long Term Outcomes, it is important to select only outcomes which the program can be reasonably expected to influence.

41. Long Term Outcomes Examples of Long Term Outcomes Program: Employment Long Term Outcome: Clients are hired. Program: Adult Day Services Long Term Outcome: Clients generalize acquired skills to other settings.

42. Important note, Long Term Outcomes • Just because outside influences may affect achievement of an outcome, that does not mean the outcome is inappropriate. We can adjust for this by recognizing that outcomes will not be achieved 100% of the time. • As an example, consider the effect of the economy (outside influence) on an employment program. The state of the economy cannot rule out the outcome “clients obtain jobs”; rather, the existence of an outside influence means that we should expect to see less than 100% of the outcome achieved.

43. Important note, Long Term Outcomes • Long Term Outcomes are of greatest interest to DHHS and will be the primary focus of our efforts to define outcomes for each program.

44. Example Suppose we identify the long term outcome, “Clients get hired”. Well, we don’t necessarily care if the “means” to that end are the initial outcome “clients learn job seeking skills” or “clients are connected to warm job leads generated by job developers”. Either initial outcome can lead to the long term outcome. Depending on the particular agency’s expertise, service delivery method, or client mix, they may prefer one “means” to the other.

45. Example Another example would be the long term outcome, “clients are appropriate weight.” One program may choose to work toward this end by targeting the initial outcome, “clients are able to identify healthy food” and the intermediate outcome, “clients eat healthy food”, while another program may identify, “clients learn the benefits of exercise” as the initial outcome and “clients exercise at least 2x per week” as the intermediate outcome.

46. Implications of Different Means to a Common End • Therefore, you can consider the Long Term Outcomes identified for your program to represent the direction we want you to go in. You will be responsible for deciding the best way to get there. • In some instances, external factors prescribe program outcomes or processes. An example would be the new Federal Birth to Three Outcomes which will be required for all BT3 programs. Additionally, the growth in popularity of evidence based program design is putting increasing pressure on programs to adopt proven approaches to service delivery. These will likely play an increasing role in programs.

47. Task 2: Specify Indicators For Your Outcomes Hints: • Outcomes and Indicators are not the same thing. • Indicators-Measurable approximations of the outcomes you are attempting to achieve. They determine the extent to which the outcome has been met. • Indicators look at the outcome and ask, “How do you know?”

48. Task 2: Specify Indicators For Your Outcomes Think about the outcome of the program. Ask yourself: • How do I know this Outcome when I see it? • What does it look like when this Outcome happens? • What are the characteristics of someone who has achieved this Outcome?

49. Task 2: Specify Indicators For Your Outcomes Example: Program- Recreation. Outcome- Clients socialize with peers and non-disabled individuals. Indicator- Number of integrated community activities participated in during the previous month.

50. Examples of Program Indicators