1 / 28

Quasi-Experimental Methods

Quasi-Experimental Methods. Florence Kondylis (World Bank). Objective. Find a plausible counterfactual Reality check Every method is associated with an assumption The stronger the assumption the more we need to worry about the causal effect Question your assumptions. Program to evaluate.

aline-delgado
Download Presentation

Quasi-Experimental Methods

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Quasi-Experimental Methods Florence Kondylis (World Bank)

  2. Objective • Find a plausible counterfactual • Reality check • Every method is associated with an assumption • The stronger the assumption the more we need to worry about the causal effect • Question your assumptions

  3. Program to evaluate Fertilizer vouchers Program (2007-08) • Main Objective • Increase maize production • Intervention: vouchers distribution • Target group: • Maize producers • Farmers owning >1 Ha, <3 Ha land • Indicator: Yield (Maize)

  4. I. Before-after identification strategy Counterfactual: Yield before program started • EFFECT = After minus Before Counterfactual assumption: There is no other factor than the vouchers affecting yield from 2007 to 2008 years

  5. Questioning the counterfactual assumption Question: what else might have happened in 2007-2008 to affect maize yield ?

  6. Examine assumption with prior data Assumption of no change over time not so great ! >> There are external factors (rainfall, pests…)

  7. II. Non-participant identification strategy Counterfactual: Rate of pregnancy among non-participants Counterfactual assumption: Without vouchers, participants would as productive as non-participants in a given year

  8. Questioning the counterfactual assumption Question: how might participants differ from non-participants?

  9. Test assumption with pre-program data REJECT counterfactual hypothesis of same productivity

  10. III. Difference-in-Difference identification strategy Counterfactual: • Non-participant maize yield, purging pre-program differences between participants/nonparticipants • “Before vouchers” maize yield, purging before-after change for nonparticipants (external factors) • 1 and 2 are equivalent

  11. Effect = 3.47 – 11.13 = - 7.66 Participants 66.37 – 62.90 = 3.47 57.50 - 46.37 = 11.13 Non-participants

  12. Effect = 8.87 – 16.53 = - 7.66 Before 66.37 – 57.50 = 8.87 62.90 – 46.37 = 16.53 After

  13. Counterfactual assumption: Without intervention participants and nonparticipants’ pregnancy rates follow same trends

  14. 74.0 16.5

  15. 74.0 -7.6

  16. Questioning the assumption • Why might participants’ trends differ from that of nonparticipants?

  17. Examine assumption with pre-program data counterfactual hypothesis of same trends doesn’t look so believable

  18. IV. Matching with Difference-in-Difference identification strategy Counterfactual: Comparison group is constructed by pairing each program participant with a “similar” nonparticipant using larger dataset – creating a control group from similar (in observable ways) non-participants

  19. Counterfactual assumption: Unobserved characteristics do not affect outcomes of interest Unobserved = things we cannot measure (e.g. ability) or things we left out of the dataset Question: how might participants differ from matched nonparticipants?

  20. 73.36 Effect = - 7.01 66.37 Matched nonparticipant Participant

  21. Can only test assumptionwith experimental data • Studies that compare both methods (because they have experimental data) find that: • unobservables often matter! • direction of bias is unpredictable! Apply with care – think very hard about unobservables

  22. Summary • Randomization requires minimal assumptions needed and procures intuitive estimates (sample means !) • Non-experimental requires assumptions that must be carefully assessed • More data-intensive

  23. Example: Irrigation for rice producers + Enhanced Market Access • Impact of interest measured by: • Input use & repayment of irrigation fee • Rice yield • (Cash) income from rice • Non-rice cash income (spillovers to other value chains) • Data: 500 farmers in project area / 500 random sample farmers • Before & after treatment • Can’t randomize irrigation so what is the counterfactual?

  24. Plausible counterfactuals • Random sample difference in difference • Are farmers outside the scheme on the same trajectory ? • Farmers in the vicinity of the scheme but not included in scheme • Selection of project area needs to be carefully documented (elevation…) • Proximity implies “just-outside farmers” might also benefit from enhanced market linkages • What do we want to measure? • Propensity score matching • Unobservables determining on-farm productivity ?

  25. Thank You

More Related