Impact Evaluation Designs for Male Circumcision

1. Impact Evaluation Designs for Male Circumcision Sandi McCoy University of California, Berkeley Male Circumcision Evaluation Workshop and Operations Meeting

2. Our Objective: Estimate the CAUSAL effect (impact) of: interventionP(male circumcision) on outcomeY(HIV incidence)

3. Our Objective: Estimate the CAUSAL effect (impact) of: interventionP(male circumcision) on outcomeY(HIV incidence) • Since we can never actually know what would have happened, comparison groups allow us to estimate the counterfactual

4. 1 2 3 4 Evaluation Designs for MC IE

5. 1 2 3 4 Evaluation Designs for MC IE Not everyone has access to the intervention at the same time (supply variation) The program is available to everyone (universal access or already rolled out)

6. Cluster Evaluation Designs • Unit of analysis is a group (e.g., communities, districts) • Usually prospective Intervention Comparison

7. Cluster Evaluation Designs Case Study:Progresa/Oportunidades Program • National anti-poverty program in Mexico • Eligibility based on poverty index • Cash transfers • conditional on school and health care attendance • 506 communities • 320 randomly allocated to receive the program • 185 randomly allocated to serve as controls • Program evaluated for effects on health and welfare

8. Stepped Wedge or Phased-In Brown CA, Lilford RJ. BMC Medical Research Methodology, 2006.

9. Stepped Wedge or Phased-In Brown CA, Lilford RJ. BMC Medical Research Methodology, 2006.

10. Stepped Wedge or Phased-In Brown CA, Lilford RJ. BMC Medical Research Methodology, 2006.

11. Stepped Wedge or Phased-In Brown CA, Lilford RJ. BMC Medical Research Methodology, 2006.

12. Stepped Wedge or Phased-In Case Study:Rwanda Pay-for-Performance • Performance based health care financing • Increase quantity & quality of health services provided • Increase health worker motivation • Financial incentives to providers to see more patients and provide higher quality of care • Phased rollout at the district level • 8 randomly allocated to receive the program immediately • 8 randomly allocated to receive the program later

13. Selective Promotion • Common scenarios: • National program with universal eligibility • Voluntary inscription in program • Comparing enrolled to not enrolled introduces selection bias • One solution: provide additional promotion, encouragement or incentives to a sub-sample: • Information • Encouragement (small gift or prize) • Transport

14. Selective Promotion Universal eligibility

15. Selective Promotion Universal eligibility Selectively promote NoPromotion Promotion

16. Selective Promotion Enrollment Universal eligibility Selectively promote NoPromotion Promotion

17. Selective Promotion Brown CA, Lilford RJ. BMC Medical Research Methodology, 2006.

18. Selective Promotion Brown CA, Lilford RJ. BMC Medical Research Methodology, 2006.

19. Selective Promotion Brown CA, Lilford RJ. BMC Medical Research Methodology, 2006.

20. Selective Promotion Brown CA, Lilford RJ. BMC Medical Research Methodology, 2006.

21. Selective Promotion Necessary conditions: • Promoted and non-promoted groups are comparable • Promotion not correlated with population characteristics • Guaranteed by randomization • Promoted group has higher enrollment in the program • Promotion does not affect outcomes directly

22. Selective Promotion Case Study:Malawi VCT • Respondents in rural Malawi were offered a free door-to-door HIV test • Some were given randomly assigned vouchers between zero and three dollars, redeemable upon obtaining their results at a nearby VCT center

23. Dose – Response Evaluations • Suitable when a program is already in place everywhere • Examine differences in exposures (doses) or intensity across program areas • Compare the impact of the program across varying levels of program intensity Hypothetical map of program implementation levels

24. Dose – Response Evaluations Example for MC: • All clinics in a region offer MC, but their capacity is limited and there are queues • Some towns are visited by mobile clinics that help the fixed clinic rapidly increase MC coverage

25. Design Variations for MC IE

26. Random Allocation • Each unit has the same probability of selection • for who receives the benefit, or • who receives the benefit first • Helps obtain comparability between those who did and did not receive the intervention • On observedandunobserved factors • Ensures transparency and fairness

27. Unit of Randomization • Individuals, groups, communities, districts, etc

28. Matching • Pick a comparison group that “matches” the treatment group based on similarities in observed characteristics

29. Matching Region A - Treatment Region B - Comparison

30. Matching Region A - Treatment Region B - Comparison

31. Matching • Matching helps control for observable heterogeneity • Cannot control for factors that are unobserved • Matching can be done at baseline (more efficient) OR in the analysis

32. Enrolled versus Not Enrolled Consider a school-based pregnancy prevention program 10 schools in the district are asked if they would like to participate

33. Enrolled versus Not Enrolled No intervention 5 schools decline participation 5 schools elect to participate in the program Pregnancy Prevention Program

34. Enrolled versus Not Enrolled No intervention Pregnancy rate = 3 per 100 student years 2 per 100 student years Pregnancy Prevention Program

35. Enrolled versus Not Enrolled Schools in the program had fewer adolescent pregnancies… Can we attribute this difference to the program? No intervention Pregnancy rate = 3 per 100 student years 2 per 100 student years Pregnancy Prevention Program

36. Enrolled versus Not Enrolled Observed effect might be due to differences in unobservable factors which led to differential selection into the program(“selection bias”) No intervention Pregnancy rate = 3 per 100 student years 2 per 100 student years Pregnancy Prevention Program

37. Enrolled versus Not Enrolled • This selection method compares “apples to oranges” • The reason for not enrolling might be correlated with the outcome • You can statistically “control” for observed factors • But you cannot control for factors that are “unobserved” • Estimated impact erroneously mixes the effect of different factors

38. Choosing Your Methods • Two decisions to decide the design:

39. Choosing Your Methods Identify the “best” possible design given the context • Best design = fewest risks for error • Have we controlled for “everything”? • Internal validity • Is the result valid for “everyone”? • External validity • Local versus global treatment effect

40. Consider Randomization First • Minimizes selection bias • Balances known and unknown confounders • Most efficient (smaller Ns) • Simpler analyses • Transparency • Decision makers understand (and believe) the results

41. Choosing Your Methods • To identify an IE design for your program, consider: • Prospective/retrospective • Eligibility rules • Roll-out plan (pipeline) • Is universe of eligibles larger than available resources at a given point in time? • Who controls implementation? • Budget and capacity constraints? • Excess demand for program? • Eligibility criteria? • Geographic targeting?

42. Thank you