Designing the Evaluation

1. Designing the Evaluation Sonal R. Doshi MS, MPH Program Evaluator CDC/OSTLTS/DPHPI TB Education, Training, and Evaluation Lights, Camera, Action: Setting the Stage for TB Elimination 18 September 2012 National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention Division of Tuberculosis Elimination

2. Session Outline • Introduction • Session objectives • Background • What is evaluation? • Research vs. evaluation • CDC Evaluation Framework • What is evaluation design? • Considerations for determining evaluation design • Type of evaluation designs • Exercise • Wrap-up

3. introduction

4. Session Objectives • Identify key elements for determining the evaluation design. • Describe the difference between process and outcome evaluation and other evaluation designs

5. Background

6. What is Evaluation? • “the process of determining the merit, worth, or value of something, or the product of that process.” Michael Scriven • “Evaluation is a field that applies systematic inquiry to help improve programs, products, and personnel, as well as the human actions associated with them.” * * AEA guiding principles - http://www.eval.org/aea09.eptf.policy.handouts.pdf

7. Research vs. Evaluation SystematicMethods Evaluation Research • Knowledge intended for use • Program- or funder-derived questions • Judgmental quality • Action setting • Role conflicts • Often not published • Multiple allegiances • Production of generalizable knowledge • Researcher-derived questions • Paradigm stance • More controlled setting • Clearer role • Published • Clearer allegiance 7

8. “Research seeks to prove, evaluation seeks to improve…” M.Q. Patton

9. CDC Framework for Program Evaluation MMWR, 1999. Framework for Program Evaluation in Public Health.

10. evaluation design

11. Evaluation and Evaluation Design • To improve program… • Art of evaluation design* • An “…exercise of the dramatic imagination” – L.J. Cronbach (pg 249) • Involves creating a design that is appropriate for a specific situation and particluar action or policy-making context • There is no single best plan for an an evaluation, not even for an inquiry into a particular program, at a particular time, with a particular budget • Any given design is necessarily an interplay of resources, possibilities, creativity, and personal judgements by the people involved * From Patton MQ. Utilization-focused evaluation: the new century text. 1997. Sage Publications, Inc.

12. Types of Evaluation Design • Many different ways to categorize evaluation designs • Examples • Randomized or true experimental/quasi-experimental/non-experiments • Summative/formative • Process/outcomes • Economic – CEA, CBA, and CUA • Mixed methods

13. Considerations for Evaluation Design • Determine expected uses by intended users • Judgement oriented • Facilitating improvements • Generating knowledge • Have a clear understanding of what, if any, decisions will be influenced by the evaluation • Reflect on the program evaluation standards • Utility • Feasibility • Propriety • Accuracy * Citations, references, and credits – Myriad Pro, 11pt

14. Considerations for Evaluation Design, cont. • Considerations from “Real World Evaluation”* • Understand clients information needs. • Understand the dynamics and timetable of the decision-making process. • Define the program theory on which the program is based in close collaboration with key stakeholders. • Identify Budget, time, and data constraints and prioritize their importance,. • Understand political context. • Answer these questions** • What is the purpose of the evaluation? • How will the information be used? • What will we know after the evaluation that we don’t know now? • What actions will we be able to take based on evaluation findings? * Bamberger M, Rugh J, Mabry L. RealWorld evaluation: working under budget, time, data, and political constraints. 2006. Sage Publications, Inc. ** From Patton MQ. Utilization-focused evaluation: the new century text. 1997. Sage Publications, Inc.

15. exercise

17. Evaluation Questions • What are the barriers to successful tuberculosis contact investigations in Utah? • Did implementation of the TB cohort review process as a program evaluation tool improve treatment and contact elicitation outcomes?

18. Wrap-up

19. Take Aways • This is evaluation; not research • Be clear on the evaluation question you are trying to answer • Be clear on the intended use of the evaluation • Be realistic • Others??

20. Thank-you! For more information please contact Centers for Disease Control and Prevention 1600 Clifton Road NE, Atlanta, GA 30333 Telephone: 1-800-CDC-INFO (232-4636)/TTY: 1-888-232-6348 E-mail: cdcinfo@cdc.gov Web: http://www.cdc.gov The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention. National Center for HIV/AIDS, Viral Hepatitis, STD , and TB Prevention Division of Tuberculosis Elimination

21. Designing the Evaluation Sonal R. Doshi MS, MPH Program Evaluator CDC/OSTLTS/DPHPI sdoshi@cdc.gov TB Education, Training, and Evaluation Lights, Camera, Action: Setting the Stage for TB Elimination 18 September 2012 National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention Division of Tuberculosis Elimination