1 / 40

Impact Evaluation: The case of Bogotá’s concession schools Felipe Barrera-Osorio World Bank

Impact Evaluation: The case of Bogotá’s concession schools Felipe Barrera-Osorio World Bank. October 2010. Outline. Hypotheses Detour: Evaluation of Programs Empirical strategy The data Results. Why concessions (a PPP) may increase quality of education?.

baylee
Download Presentation

Impact Evaluation: The case of Bogotá’s concession schools Felipe Barrera-Osorio World Bank

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Impact Evaluation: The case of Bogotá’s concession schoolsFelipe Barrera-OsorioWorld Bank October2010

  2. Outline • Hypotheses • Detour: Evaluation of Programs • Empirical strategy • The data • Results

  3. Why concessions (a PPP) may increase quality of education? • An application of an already proven pedagogic model • Concession schools are financially stable, ensuring the stability of the pedagogic model. • Freedom to choose (and fire) teachers and management staff • Better infrastructure • Concession schools work actively with the families and the community • “Affection deficit” and family problems are important issues in these communities • Nutrition is an important variable in educational outcomes

  4. Three hypothesis • Dropout rates are lower in concessions schools than in similar, public schools. • Nearby schools have lower dropout rates than public schools outside the influence of concessions. • Test scores are higher in concessions schools than in similar, public schools.

  5. Detour: Evaluating Programs

  6. What is Impact Evaluation? • Impact evaluation is a set of methods to identify and quantify the causal impact of programs • For example, What is the effect of a concession schools on drop-out rates and test scores?

  7. Why are these effects difficult to estimate? • The basic question: What would it happen in the absence of the program? (e.g., what is the contrafactual?) • We need the same individual with and without the program • For example, we need to observe John attending one concession school, and (the same) John attending a public school • …but it is impossible to observe the same individual in both states! • Solution: “build” the correct contrafactual • Find individuals who do not have the benefits of the program, but are very similar to the ones that have the program • For example, find David who is very similar in all characteristics to John…

  8. What is the problem? • In short, the problem is to find the correct comparison (control) group • The control group and the treatment group should have the same characteristics, observable and non-observable, before the beginning of the program • External factors will affect in the same way control and treatment group • An, usually, people self-select into programs, and therefore, the beneficiaries are (with high probability), different that the ones that did not enter into the program

  9. The basic intuition: only data before and after the program Y: Test scores Impact of the program? NO! We need a contrafactual Time t = 0 before program t = 1 after program Intervention

  10. We need the right comparison group Y: Test scores Y Treatment Impact of the program Control Time t = 0 before program t = 1 after program Intervention

  11. The basic intuition 2: only data after the program…. Y: Test scores Y Treatment Impact of the Program? Control Time t = 0 before program t = 1 after program Intervention

  12. We need the right comparison group! Y: Test scores Y Treatment Impact of the Program? NO! At t=0, two groups were very different… Control Time t = 0 before program t = 1 after program Intervention

  13. Four possibilities to find or construct the right control group • Prospective evaluation • Randomization of benefits • Randomization of entry (phase-in approach) • Randomization of information (encouragement design) • Retrospective evaluation • Regression discontinuity analysis • Instrumental variables • Differences in differences • Propensity and matching estimators

  14. Randomization • Lottery among individuals will separate the sample between winners and losers • Model of “over-subscription” • A lottery will create homogenous control and treatment group: they will be very similar in all characteristics, observable and non-observable.

  15. Regression discontinuity • When to use this method? • The beneficiaries/non-beneficiaries can be ordered along a quantifiable dimension. • This dimension can be used to compute a well-defined index or parameter. • The index/parameter has a cut-off point for eligibility. • The index value is what drives the assignment of a potential beneficiary to the treatment. (or to non-treatment)

  16. Intuition • The potential beneficiaries just above the cut-off point are very similar to the potential beneficiaries just below the cut-off point. • We compare outcomes for individualsjust above and below the cutoff point.

  17. Non-poor Poor

  18. Treatment effect

  19. Difference in differences • Estimating the impact of a “past” program • We can try to find a “natural experiment” that allows us to identify the impact of a policy. For example, • An unexpected change in policy could be seen as “natural experiment” • A policy that only affects 16 year olds but not 15 year olds • We need to identify which is the group affected by the policy change (“treatment”) and which is the group that is not affected (“control”). • The quality of the control group determines the quality of the evaluation.

  20. Intuition • Find a group that did not receive the program • …with the same pattern of growth in the outcome variable before the intervention • The two groups, treatment and control, have the same profile before the intervention

  21. Intuition: when it is right to use DD? Y: Test scores Impact of the Program Control Time t = -1 t = 0 t = 1 after program Slopes, before intervention (between -1 and 0) were equal Intervention

  22. Intuition: when it is right to use DD? Y: Test scores Impact of the Program? NO: wrong comparison group Control Time t = -1 t = 0 t = 1 after program Slopes, before intervention (between -1 and 0) were different Intervention

  23. Propensity and matching estimation • Find the comparison group from a large survey • Each treatment will have a comparison that has observable characteristics as similar as possible to the treated individuals • This method assumes that there is not self-selection based on unobservable characteristics: • Selection is based on observable characteristics

  24. How is this procedure done? Two steps • Estimation of the propensity score: The propensity score is the conditional probability of receiving the treatment given the pre-treatment variables. • Estimate the probability of been treated based on the observable characteristics • Estimation of the average effect of treatment given the propensity score • match cases and controls with exactly the same (estimated) propensity score; • compute the effect of treatment for each value of the (estimated) propensity score • obtain the average of these conditional effects

  25. …Back to Concessions

  26. Empirical Strategy for Concessions: propensity and matching estimation • Propensity score estimation: • “Find” controls that have similar characteristics as the treatment one • Matching estimation: • Once the controls are found, estimate impact:

  27. The data • C100 and C600: Rich information about schools • For 1999 and 2003: Administrative personnel, number and level of education of teachers, physicians and psychologists, students by grade and by age, students who failed a grade and dropped out. • For 1999: infrastructure variables like computers, rooms, labs, etc. • ICFEX 2003: Standardized test scores • Test scores at the individual level • Some information at the individual level

  28. Flavor of the (strong) result Concessions reduced the dropout rate significantely Concessions reduce the jump in the desertion in grade 6

  29. Test scores: Public schools have lower test scores Concessions and public non-concessions are “similar”

  30. Dropout Results: Probit estimates

  31. Dropout results: impact • Matching: • 10 nearest estimators, common support, balance groups • Direct Effect: reduction in 1.7 points dropout rates • Indirect Effect: reduction in 0.82 points

  32. Test Results: Probit estimates

  33. Test results: impact • Matching: • 10 nearest estimators, common support, balance groups • Effect over math test scores: improvement of 2.4% • Effect over language test scores: improvement of 4%

  34. Some conclusions (and some questions) • Concessions delivers: • Strong evidence of differences in dropout rates • Some evidence of impact over test scores • Colombian government is implementing the II stage of Concessions • What makes concessions different from public? • Is it the work with families and community? • Freedom of hiring teachers? • Pedagogic method?

  35. A discussion about the data • Limits of the data: • Not a randomized experiment • Still some source of bias • Ideally, we want to have more hard-evidence • Randomization of students who applied to concessions

  36. Why is this important? • Impact evaluation can provide reliable estimates of the causal effects of programs • Impact evaluation can potentially help improve the efficacy of programs by influencing design and implementation • Impact evaluation can broaden political support for programs • Impact evaluation can help in the sustainability of successful programs and the termination of programs that are a failure • Impact evaluation can help expand our understanding of how social programs produce the effects that they do

More Related