Assessing Program Impact

1. Assessing Program Impact Chapter 8

2. Impact assessmentsanswer… • Does a program really work? • Does a program produce desired effects over and above what would have occurred without the intervention? • Are there important unintended effects?

3. Impact Assessmentscan be relevant at many points • Policy formation • Evaluating a pilot program • Ongoing programs – may need to modify or refine • high costs of programs – must continually evaluate their efficacy • demonstrations of effectiveness are required to renew grants • justify uses of public funds

4. Prerequisite conditions • impact assessments build on earlier forms of evaluation • the program’s objectives must be well articulated and have plausible, measurable outcomes • intervention is well implemented and has been in place for sufficient time

5. Equivalence • identical composition - same mixes in terms of their program-related and outcome-related characteristics • identical predispositions - equally disposed toward the project and equally likely to attain any given outcome status • identical experiences – over the time of observation

6. If two groups are “equivalent”, both are subject to the same degree of change induced by factors outside of the program. Any difference in outcome between them, therefore, should represent the effect of the program.

7. Two classes of Approaches: Experimental vs Quasi-Experimental Research Designs Randomized field experiment • the “gold standard” research design • Participants are randomly sorted into at least two groups – a control group and an intervention group Quasi-experiments • Nonrandomized in which participants (given intervention) are compared with nonparticipants (the controls)

8. Randomized Field Experiments Principal advantage: • it isolates the effect of the intervention being evaluated by ensuring that intervention and control groups are statistically equivalent except for the intervention received Best way to achieve equivalence!

9. Randomization is just not a “shot in the dark • Allocating targets to intervention and control groups requires considerable care • The evaluator must use an explicit chance-based procedure • Be careful when using lists • use statistical significance testing to judge whether a specific difference is likely to have occurred simply by chance

10. Data Collection Strategies Two strategies can improve the estimates of program effects • Make multiple measurements of the outcome variable • Collect data periodically during the course of an intervention

11. Units of Analysis • the units on which the outcome measures are taken • should be based on the nature of the intervention and the target units Be careful! • Observational vs. experimental units • Assumption of independence

12. Limitations of Randomized Experiments • These research designs are not applicable to all program situations • The program usually has to be stable and operationally mature for authentic results • If the program changes during the course of the experiment, it is difficult to differentiate between what program version produced what effects. • Difference between Experimental and Actual program delivery

13. Ethical Considerations • ethical qualms about randomization – deprives control groups of positive benefits If program resources are scarce • do you allot services by chance? • or insist that the most needy targets receive priority? The next chapter that will look at alternative designs.

14. “Perfect” vs “Good Enough” • time and resource constraints • intended use of the results • feasibility of design • ethical considerations • credibility • differences between experimental and actual intervention delivery • high turnover in policy considerations • integrity of a randomized experiment is easily threatened

15. Impact Assessments Overview • All impact assessments are comparative – with a group receiving alternative services or no treatment. • Ideally, the conditions being compared should be identical in all respects except for the intervention • All assessment involves establishing control conditions • The most valid results generally require more skills, more time to complete and more cost.

16. Beware! Rigorous Impact Assessments involve technical and managerial challenges and significant resources -sometimes has political pressures Is it justified by the circumstances?