How to Encourage Utilization

1. How to Encourage Utilization Chapter 14 April 3rd and 5th, 2006

2. To encourage utilization, present reports in a way that gets attention!

3. To encourage utilization, present reports in a way that gets attention!

4. The multiple uses of Evaluations • Stakeholders are disposed to use the information • The evaluation affects the intentions to modify programs • Evaluation results in further exploration of particular questions • The evaluation might affects the programs evaluated • The evaluation affects the development of new laws • Evaluation information becomes part of the knowledge base and reduce the uncertainty of administrators, stakeholders, and people involve in program • Evaluation might bring change in practice

5. Obstacles

6. Obstacles to effective utilizationand to implementingfindings • Constraints on managers • Value conflicts among stakeholders • Misapplied methodology • Evaluating a program at arm’s length

7. Example – Border OfficersArmed? or Unarmed?

8. Dealing with mixed findings

9. Dealing with mixed findings • Don’t abdicate your responsability • Don’t take the easy way out • Show how to use the evaluation to improve the program

10. Using Evaluations when an innovative program seems no better than other treatments • When can evaluators be sure groups do not differ? • Are evaluations valuable even when no advantage for the innovations is found?

11. When can evaluators be sure groups do not differ? • Sufficient sample sizes? • Include an examination of the degree of implementation • What failed? • Were the measurement tools sufficient to capture the essence of the program? • Type II errors?

12. Type II error = Missing a program effect

13. How to avoid Type II error? • Raise alpha level above 0.5 • Examine the impact model- intermediate and final outcomes- specific characteristics of participants • Make a good effort to find a difference

14. Using Evaluations when an innovative program seems no better than other treatments • When can evaluators be sure groups do not differ? • Are evaluations valuable even when no advantage for the innovations is found?

15. Are evaluations valuable even when no advantage for the innovations is found? Oftenyes! If the results of an evaluation demonstrate that a standard program is just as effective as an innovative program we can conclude that we can achieve the same results with a less expensive program.

16. Work-Release Programs for Prisoners in the USAExample p. 277-278

17. Developing a learning culture • Work with stakeholders • Adopt developmental interpretations • Frame findings in terms of improvements • Treat findings as tentative indicators, not final answers • Recognize service providers’ needs • Keep evaluation findings on the agency’s agenda

18. The Evaluation Attitude • Humility won’t hurt • Impatience may lead to disappointment • Recognize the importance of the evaluator’s perspective • Work on practical questions • Work on feasible issues • Avoid data addiction • Make reports accessible • Seek evaluations of your own work • Encourage the development of a learning culture

19. Possible Trends • Help the organization make use of an evaluation even when findings are mixed or negative • Encourage stakeholders to look upon an evaluation as a development tool to use in making program improvements • Evaluators should be involved in discussions in order to explain the findings related to needs, implementation, or outcomes • To expect confusion in program design and limitations in implementation in the field and to recognize the constraints on the activities of the managers of service organizations and government agencies • See that each program is unique and what works in one place might not be appropriate for another site or with another population • Recognize the limitations of evaluation methods

20. Better evaluations based on… • Greater sensitivity to stakeholder needs • Deeper respect for importance of the context of a program • Less grandiose view of evaluation methods

21. Group Exercises

22. The End