400 likes | 527 Views
Impact Evaluation: The case of Bogotá’s concession schools Felipe Barrera-Osorio World Bank. October 2010. Outline. Hypotheses Detour: Evaluation of Programs Empirical strategy The data Results. Why concessions (a PPP) may increase quality of education?.
E N D
Impact Evaluation: The case of Bogotá’s concession schoolsFelipe Barrera-OsorioWorld Bank October2010
Outline • Hypotheses • Detour: Evaluation of Programs • Empirical strategy • The data • Results
Why concessions (a PPP) may increase quality of education? • An application of an already proven pedagogic model • Concession schools are financially stable, ensuring the stability of the pedagogic model. • Freedom to choose (and fire) teachers and management staff • Better infrastructure • Concession schools work actively with the families and the community • “Affection deficit” and family problems are important issues in these communities • Nutrition is an important variable in educational outcomes
Three hypothesis • Dropout rates are lower in concessions schools than in similar, public schools. • Nearby schools have lower dropout rates than public schools outside the influence of concessions. • Test scores are higher in concessions schools than in similar, public schools.
What is Impact Evaluation? • Impact evaluation is a set of methods to identify and quantify the causal impact of programs • For example, What is the effect of a concession schools on drop-out rates and test scores?
Why are these effects difficult to estimate? • The basic question: What would it happen in the absence of the program? (e.g., what is the contrafactual?) • We need the same individual with and without the program • For example, we need to observe John attending one concession school, and (the same) John attending a public school • …but it is impossible to observe the same individual in both states! • Solution: “build” the correct contrafactual • Find individuals who do not have the benefits of the program, but are very similar to the ones that have the program • For example, find David who is very similar in all characteristics to John…
What is the problem? • In short, the problem is to find the correct comparison (control) group • The control group and the treatment group should have the same characteristics, observable and non-observable, before the beginning of the program • External factors will affect in the same way control and treatment group • An, usually, people self-select into programs, and therefore, the beneficiaries are (with high probability), different that the ones that did not enter into the program
The basic intuition: only data before and after the program Y: Test scores Impact of the program? NO! We need a contrafactual Time t = 0 before program t = 1 after program Intervention
We need the right comparison group Y: Test scores Y Treatment Impact of the program Control Time t = 0 before program t = 1 after program Intervention
The basic intuition 2: only data after the program…. Y: Test scores Y Treatment Impact of the Program? Control Time t = 0 before program t = 1 after program Intervention
We need the right comparison group! Y: Test scores Y Treatment Impact of the Program? NO! At t=0, two groups were very different… Control Time t = 0 before program t = 1 after program Intervention
Four possibilities to find or construct the right control group • Prospective evaluation • Randomization of benefits • Randomization of entry (phase-in approach) • Randomization of information (encouragement design) • Retrospective evaluation • Regression discontinuity analysis • Instrumental variables • Differences in differences • Propensity and matching estimators
Randomization • Lottery among individuals will separate the sample between winners and losers • Model of “over-subscription” • A lottery will create homogenous control and treatment group: they will be very similar in all characteristics, observable and non-observable.
Regression discontinuity • When to use this method? • The beneficiaries/non-beneficiaries can be ordered along a quantifiable dimension. • This dimension can be used to compute a well-defined index or parameter. • The index/parameter has a cut-off point for eligibility. • The index value is what drives the assignment of a potential beneficiary to the treatment. (or to non-treatment)
Intuition • The potential beneficiaries just above the cut-off point are very similar to the potential beneficiaries just below the cut-off point. • We compare outcomes for individualsjust above and below the cutoff point.
Non-poor Poor
Difference in differences • Estimating the impact of a “past” program • We can try to find a “natural experiment” that allows us to identify the impact of a policy. For example, • An unexpected change in policy could be seen as “natural experiment” • A policy that only affects 16 year olds but not 15 year olds • We need to identify which is the group affected by the policy change (“treatment”) and which is the group that is not affected (“control”). • The quality of the control group determines the quality of the evaluation.
Intuition • Find a group that did not receive the program • …with the same pattern of growth in the outcome variable before the intervention • The two groups, treatment and control, have the same profile before the intervention
Intuition: when it is right to use DD? Y: Test scores Impact of the Program Control Time t = -1 t = 0 t = 1 after program Slopes, before intervention (between -1 and 0) were equal Intervention
Intuition: when it is right to use DD? Y: Test scores Impact of the Program? NO: wrong comparison group Control Time t = -1 t = 0 t = 1 after program Slopes, before intervention (between -1 and 0) were different Intervention
Propensity and matching estimation • Find the comparison group from a large survey • Each treatment will have a comparison that has observable characteristics as similar as possible to the treated individuals • This method assumes that there is not self-selection based on unobservable characteristics: • Selection is based on observable characteristics
How is this procedure done? Two steps • Estimation of the propensity score: The propensity score is the conditional probability of receiving the treatment given the pre-treatment variables. • Estimate the probability of been treated based on the observable characteristics • Estimation of the average effect of treatment given the propensity score • match cases and controls with exactly the same (estimated) propensity score; • compute the effect of treatment for each value of the (estimated) propensity score • obtain the average of these conditional effects
Empirical Strategy for Concessions: propensity and matching estimation • Propensity score estimation: • “Find” controls that have similar characteristics as the treatment one • Matching estimation: • Once the controls are found, estimate impact:
The data • C100 and C600: Rich information about schools • For 1999 and 2003: Administrative personnel, number and level of education of teachers, physicians and psychologists, students by grade and by age, students who failed a grade and dropped out. • For 1999: infrastructure variables like computers, rooms, labs, etc. • ICFEX 2003: Standardized test scores • Test scores at the individual level • Some information at the individual level
Flavor of the (strong) result Concessions reduced the dropout rate significantely Concessions reduce the jump in the desertion in grade 6
Test scores: Public schools have lower test scores Concessions and public non-concessions are “similar”
Dropout results: impact • Matching: • 10 nearest estimators, common support, balance groups • Direct Effect: reduction in 1.7 points dropout rates • Indirect Effect: reduction in 0.82 points
Test results: impact • Matching: • 10 nearest estimators, common support, balance groups • Effect over math test scores: improvement of 2.4% • Effect over language test scores: improvement of 4%
Some conclusions (and some questions) • Concessions delivers: • Strong evidence of differences in dropout rates • Some evidence of impact over test scores • Colombian government is implementing the II stage of Concessions • What makes concessions different from public? • Is it the work with families and community? • Freedom of hiring teachers? • Pedagogic method?
A discussion about the data • Limits of the data: • Not a randomized experiment • Still some source of bias • Ideally, we want to have more hard-evidence • Randomization of students who applied to concessions
Why is this important? • Impact evaluation can provide reliable estimates of the causal effects of programs • Impact evaluation can potentially help improve the efficacy of programs by influencing design and implementation • Impact evaluation can broaden political support for programs • Impact evaluation can help in the sustainability of successful programs and the termination of programs that are a failure • Impact evaluation can help expand our understanding of how social programs produce the effects that they do