Testing Program Design

1. Testing Program Design David Yanagizawa-Drott Assistant Professor of Public Policy Evidence for Policy Design Harvard Kennedy School Oct 22, 2013

2. Testing Program Design By providing evidence for links in a program theory, testing design can help us to understand not only whether a policy works, but also why it works, which can feed back into program design.

3. Lecture Overview • Why test program design? • What are some aspects of program design we can test? • Cost-effectiveness of design variations • Program intensity • Interaction of program elements • Challenges for testing program design

4. Property tax enforcement in Pakistan Context: • Tax evasion costs developing countries an estimated $285 billion per year (Cobham, 2005), resulting in under-provision of public services. • In Punjab province, Pakistan, property tax collection is roughly 20% of the level in comparable countries (World Bank, 2006). Overall tax to GDP ratio was only 9.5% in Pakistan in 2008. • Corruption and poor tax administration contribute to lack of enforcement.

5. Property tax enforcement in Pakistan How can we increase enforcement of property taxes? What are some program design options for achieving this policy goal?

6. Simple program evaluation Does the program or policy have the intended impact on the intended beneficiaries? Evaluation strategy: Compare outcomes between a control group and onetreatment group What is the magnitude of the impact?

7. Testing design 1. Within a fixed budget, what program design will have the highest impact? Evaluation strategy: Compare outcomes between a control group and multiple treatment groups 2. Given different designs, with different costs, which design is most cost-effective? 3. How do changes in program intensity affect impact? 4. How do different program components contribute to overall impact?

8. Within a fixed budget, which program variation will give you the most impact? Total Population Target Popula-tion Not in evalua-tion Treatment Group 1 Treatment Group 2 Evaluation Sample Treatment Group 3 Random Assignment Control Group

9. Lecture Overview • Why test program design? • What are some aspects of program design we can test? • Cost-effectiveness of design variations • Program intensity • Interaction of program elements • Challenges for testing program design

10. Design variations: CCTs in Colombia You are trying to improve secondary schooling outcomes in Colombia using conditional cash transfers. Consider 2 different designs: • Standard: Payments of $15/month for 10 months conditional on 80% attendance (total cost: $150/year) • Savings: Payments of $10/month for 10 months conditional on 80% attendance, plus $50 one-time payment during enrollment period for the following school year (total cost: $150/year) Source: Barrera-Osorio, Bertrand, Linden, & Perez-Calle (2011). Improving the Design of Conditional Transfer Programs: Evidence from a Randomized Education Experiment in Colombia.

11. Suppose you are the policymaker: Which design would you opt for? • Standard • Savings

12. CCTs in Colombia Small changes to the design of an intervention can lead to substantial differences in impact. Source: J-PAL (2012)

13. If you have different program designs, with different costs, which design is most cost-effective? Total Population Target Popula-tion Not in evalua-tion Treatment Group 1 Treatment Group 2 Evaluation Sample Treatment Group 3 Random Assignment Control Group

14. Price Information Experiment Context: • Rural small-scale farmers often lack updated information about crop prices • Therefore, knowledge about current prices is low and price expectations are often imprecise or false • Getting price information is a priority for many farmers

15. Price Information: program theory Question: what is the most cost-effective way to get the information to farmers?

16. Price Information: intervention Experiment: • Rural Gujarat, India 2007-ongoing • Provide weekly price information for crops (cotton, castor, guar) • Tested two types of interventions: • Price information provided on village boards • Price information provided via SMS

17. Price Information: design Total sample 1080 108 villages X 10 respondents per village Treatment 1: Village board 286 Treatment 2: SMS 254 Control 540

18. Price Information: costs & benefits Cost/benefit considerations: • SMS intervention costs 65% more on average, and can’t reach the poorest farmers • SMS would have to be more than 65% more effective in order for benefit to justify costs.

19. How will changes in program intensity affect program impact? Target Population Not in evaluation High intensity Treatment group Moderate intensity Treatment group Evaluation Sample Treatment group Low intensity Random Assignment Control group

20. Balancing impact and cost Unclear ex-ante how impact will vary with changes in intensity… Decreasing returns Increasing returns Constant returns Outcome Outcome Outcome Intensity/Cost Intensity/Cost Intensity/Cost

21. Evaluating intensity: Banking in Indonesia Problem: Informal economy in Indonesia represents 23% of GDP. Theory: Drawing individuals and firms into the formal financial sector would decrease the size of the informal economy and foster development. Policy proposal: Offer incentive payments for unbanked households to open savings accounts. Source: Cole, Sampson & Zia (2011). Prices or Knowledge? What drives Demand for Financial Services in Emerging Markets?

22. Banking in Indonesia Question for evaluation: What is the impact of different levels of incentive payments? Intervention: Offer unbanked households Rp. 25,000 ($3), Rp. 75,000 ($8.50) or Rp. 125,000 ($14) to open savings accounts. Ben Olken

23. Which do you expect to have the greatest impact on bank account take-up? • Increasing the incentive payment from Rp. 25,000 to Rp. 75,000 (diminishing returns) • Increasing the incentive payment from Rp. 75,000 to Rp. 125,000 (increasing returns) • Neither – they will have the same impact (constant returns)

24. Banking in Indonesia Results: Higher incentives substantially increased the likelihood of opening a bank account, but returns appear to be diminishing. 3.8% points 5.4% points

25. How do different program components in cross-cutting designs contribute to impact? Target Popula- tion Not in evaluation Evaluation Sample Random Assignment

26. Complements v. substitutes • If the impact of two interventions used together is greater than the sum of their impacts when used alone, these two interventions are complements. • If the impact of two interventions used together is less than the sum of their impacts when used alone, these two interventions are substitutes.

27. Subsidies & information: complements or substitutes? Problem: In Zambia, subsidies have been found to increase take-up of preventive health products, such as water purification solutions, but subsidies are costly. Theory: Small subsidies might be more effective if consumers knew more about the product. Policy proposal: Provide product information alongside subsidies. Source: Ashraf, Jack & Kamenica (2011). Information and Subsidies: Complements or Substitutes?

28. Subsidies & information: complements or substitutes? Question for evaluation: Does information about water purification products complement subsidies, or is it a substitute? Intervention: Vary subsidy level and whether or not information is provided. Dan Bjorkegren

29. Do you think subsidies and information are complements, substitutes, or neither? • Complements • Substitutes • Neither

30. Subsidies & information: complements or substitutes? Results: Information, although ineffective in isolation, increases the effect of subsidies – the two program elements are complements. Information increases marginal impact of subsidies by 60%

31. Lecture Overview • Why test program design? • What are some aspects of program design we can test? • Cost-effectiveness of design variations • Program intensity • Interaction of program elements • Challenges for testing program design

32. The design “space” is infinite… Consider designing a microcredit program: • What range of loan size? • Group liability or individual liability? • Women, men, or both? • How many borrowers per group? • Repayment schedule? • Combine with health education or other interventions?

33. Testing cross-cutting design requires multiple experimental groups 2 Interventions T1: A Evaluation Sample 2 x 2 = 4 experimental groups T2: A & B Random Assignment T3: B Control group 3 interventions T1: A T5: A & C Evaluation Sample 2 x 2 x 2 = 8 experimental groups T2: A & B Random Assignment T6: C T3: B T7: A,B,C T4: B & C Control group Testing 4 interventions requires 2x2x2x2 = 16 groups!

34. More power → larger samples Testing multiple designs of a program generally means you are interested in detecting small differences in the magnitude of impact. • This requires more power, meaning larger samples.

35. Identifying the key questions To narrow the scope of an evaluation, it is helpful to return to your program theory. What links in my theory are uncertain? Are my hypotheses testable? Is the value of the information worth the cost? Relevant set of questions to be tested

36. Identifying the key questions By providing evidence for links in a program theory, testing design can help us to understand not only whether a policy works, but also why it works, which can feed back into program design.

37. Property tax enforcement in Pakistan How can we increase enforcement of property taxes? What are some program design options for achieving this policy goal?

38. Property taxes in Pakistan – what is the underlying cause of low enforcement? Theory Testable policy proposal Salaries too low: poor performance due to low value placed on jobs Raise wages, with threat of punishment if performance is below a minimum threshold Low-powered incentives: poor performance because remuneration not directly tied to results Tie wages to tax revenue, and/or other measurable outcome

39. Property Tax Experiment Experiment: • Developed partnership with Excise & Tax Department, Punjab • Large-scale randomized experiment, testing 2 treatments.

40. Property Tax Experiment Preliminary results: All treatment groups increased tax recovery over benchmarks, Conditional Fixed Wage shows greatest effect.

41. Review of concepts • Every intervention or program has infinite design possibilities. Programs can vary by: • Type of intervention, delivery mechanism, timing of delivery, etc. • Intensity of intervention • The set of program elements included • Testing different designs can inform questions of comparativeimpact and cost effectiveness, and clarify uncertainties in program theory. • Ultimately, testing design leads to better programs and policies.

42. Thinking about design Consider a policy problem in your context: • What is the goal of the policy? • What are some alternative program designs for achieving the goal? • Which design do you think is the most cost-effective? • What is your program theory? • What are the key design questions that evaluation could help you address?