Health Program Evaluation: Impact and Outcome Evaluation Designs

1. Health Program Evaluation:Impact and Outcome Evaluation Designs CHSC 433 Module 5/Chapter10 L. Michele Issel, PhD UIC School of Public Health

2. Objectives • Develop a sampling strategy • Develop a design for an effect evaluation • Appreciate the trade-offs between rigor and costs

3. Avoid the 3rd Failure Next slide adds a failure to the two discussed previously. Design the effect evaluation to avoid evaluation failure ~ the failure to detect program effects due to poor design, measurement, or sample of the evaluation.

5. Rigor Considerations inEffectEvaluation • Sampling strategy • Validity and Reliability • Related to design • Related to method • Related to measures • Power to find effects

6. Sampling • Plan for selecting participants in the evaluation • Plan for selecting/choosing a control or comparison group • Plan for reaching and accessing the evaluation participants or groups • Plan assures generalizability

7. Sample Size Influence • Cost of the evaluation • Time and effort required • Power to detect a difference made by the program

8. Power • Statistical power begins with sample size • Power is the degree of certainty that will be able to find a measurable and significant effect from the intervention • Is a trade-off between

9. Not so Random • Randomization Randomly assign to receive the program or not • Random selection Randomly select from among program participants and non-participating target audience for inclusion in the evaluation

10. Effect Evaluation Designs Refer to research or epidemiology textbook chapters on experimental quasi-experimental non-experimental designs

11. Design Considerations • Ethical issues with random selection or random assignment • Design to assess impact objectives • Timing of data collection relative to receiving the intervention • Who are the “controls”???

13. Design Cost vs Rigor

14. Measure Program Impact Across the Pyramid