Measuring Outcomes

1. Measuring Outcomes for quality and accountability

2. The changing landscape in mental health and rehabilitation • Increasing demand for accountability i wide range of service settings • Funding agencies demand that providers examine service effectiveness and measure consumer outcomes • Indications of quality improvement and capacity of management • Meet core values and get input from service consumers

3. Planning a program evaluation • Individual accounts and anecdotes • Concrete and standardized ways to measure service effectiveness • What is “best practice”?

4. Planning a program evaluation • Program administrators are often not well informed about outcome measurement • … and how best to link outcome measures to questions that they want to address related to services

5. The best starting point is to determine: • What outcomes need to be addressed? • Who is the evaluation information being developed for? • Who will receive information about the results? Example: Results can be disseminated to stakeholders such as funding and accreditation bodies, a board of directors, primary consumers, or family members.

6. A program evaluation effort can be started at any time – when a service starts or added later to an existing service

7. Clarifying outcomes to be measured • Conducting an evaluation, study, or quality assurance effort begins with understanding which outcomes need to be addressed

8. Steps towards clarification: • Get an understanding of your agency’s intended areas of effectiveness Example: The agency’s stated goal is “Increase in employment” • Ask whether the services your agency offers are likely to go beyond the intended outcomes to other unintended outcomes Example: A bonus goal is “Increase in self-esteem among service recipients”

9. Examine your agency’s mission statement, strategic plan, or results of former program evaluation efforts to get closer to your agency’s intended effects and thus the outcomes you should measure

10. Determine what domains need to be measured from these efforts (housing, employment, social inclusion, satisfaction, etc.) • Develop clear definitions or criteria for these domains so that the appropriate measures or instruments can be located

11. Selecting an outcome measure or instrument • Providers must have confidence in the measures • Choose high quality and appropriate measures • Look for existing measures that match the outcomes needing to be addressed, rather than using home-grown surveys or instruments developed just for the purpose at hand

12. Ensure that the measures have been tested and are known to be reliable and valid, so … • that you can trust what they measure, and … • that they measure what they say they should

13. Two important paradigm shifts • The need for reliable and valid measures of the subjective aspects of recovery • The need for measures that were developed from a consumer perspective

14. “The times they are a-changing” Providers, practitioners, managers, administrators, and policy makers can ease pressures they face in the present and in the future by … • making use of established, high quality measures and instruments, and • clarifying which effects and outcomes ought to be measured

15. The results from evaluations • … may be used not only to satisfy funding bodies and other stakeholders • - they will also provide useful information that can guide the direction of changes needed to create more effective ways of helping the people being served

16. Evaluation is an integrated component in program development Procedures Record keeping Activities Supervision structure Policy statements Job descriptions Evaluation mechanisms Mission