190 likes | 324 Views
Private involvement in education: Measuring Impacts. Felipe Barrera-Osorio HDN Education Public-Private Partnerships in Education, Washington, DC, March 31, 2010. The quest. Ideally, we want to know what is the effect of private participation in the provision of education
E N D
Private involvement in education:Measuring Impacts Felipe Barrera-Osorio HDN Education Public-Private Partnerships in Education, Washington, DC, March 31, 2010
The quest • Ideally, we want to know what is the effect of private participation in the provision of education • Does donation into an school produce better learning outcomes of students? • Does a donation that goes to teacher training, increases the pedagogic capacity of teachers and better education outcomes of kids? • Does a private funded computer program induce in the long run higher labor opportunities for individuals? • Does a Public-Private Partnership induce higher learning outcomes?
The quest (continuation) • Also, we want to know how we can improve design of programs that will lead to improvement of donations • If the program donate to schools, what is the best strategy to allocate resources: towards management, or towards educational management information systems, or towards teacher training? • Should we donate computers with specific software for education or generic software? • Can we improve efficiency of private donation by inducing involvement of the community?
Why is this important? • Impact evaluation can provide reliable estimates of the causal effects of programs • Impact evaluation can potentially help improve the efficacy of programs by influencing design and implementation • Impact evaluation can broaden political support for programs • Impact evaluation can help in the sustainability of successful programs and the termination of programs that are a failure • Impact evaluation can help expand our understanding of how social programs produce the effects that they do
These questions are not easy to answer • Impact evaluation is a new name for an old quest: What is the effect of programs? • A good evaluation will have the following characteristics: • A clear definition of the intervention: the concrete objective, what is being modified, the new set of incentives, and to whom the modifications apply. • A description of how the intervention is expected to achieve the final desired outputs: how the intervention will lead to the desired result. • A definition of the identification strategy that allows to attribute causal effects between an intervention and a set of outcome variables.
Definition of the program: objectives • Sometimes, the objectives are difficult to measure • For example, a program can have an objective such as “social cohesion” • Solution: formulate a concrete, measurable objective • For example, an objective can be to increase learning outcomes of students • A measurable, concrete outcome can increase accountability of a program
Definition of a program: an special challenge for the private sector • Usually, the evaluation is not about the private donation, but about the specific type of intervention • For example, a private individual who donate to schools for teacher training • The evaluation is not about the dynamics of the private sector into the schools, but about the effect of teacher training
How the intervention will lead to the desired result. • It is critical to understand the pathways in which the program operate • An example: a computer donation to schools • Is the program affecting the way teachers teach? • Are the students using the computers? • What are the students using the computer for?
Channels of transmission • PPP contracts allow more flexibility in the provision of education than in the public sector • Monitor critical areas of flexibility: payroll of teachers, length of shifts, firing and hiring of personnel • Usually private providers in PPP contracts are chosen by an open bid process based on quality criteria • Quality of providers: previous experience
Channels of transmission (cont.) • A PPP contract can achieve an optimal level of risk-sharing between the government and the private sector • Very difficult to quantify • The private sector may have higher standards in the delivery of education services • Analysis of educational outcomes like test scores, internal efficiency, etc • PPPs can induce competition in the market for education • Important to track entering and exiting students
Attributing causal effects • Why is so difficult to find the effects of a program? • Suppose a computer program: donation of private sector, schools receive them, teachers use them in classrooms, students have direct access to computer and software • Suppose that we follow two strategies: • First, we compare the students that receive the program before and after the program • Second, we compare students that receive the program versus students that did not receive the program
The basic intuition: only data before and after the program Y Impact of the program? NO! We need a contrafactual Time t = 0 before program t = 1 after program Intervention
We need acomparison group…. Y Impact of the program Control Time t = 0 before program t = 1 after program Intervention
The basic intuition 2: only data after the program…. Y Impact of the Program? Control Time t = 0 before program t = 1 after program Intervention
We need the right comparison group! Y Impact of the Program? NO! At t=0, two groups were very different… Control Time t = 0 before program t = 1 after program Intervention
Some possibilities to find or construct the right control group • Prospective evaluation • Randomization of benefits: a lottery to get benefits • Randomization of entry: a lottery to determine order of entry • Retrospective evaluation • Regression discontinuity analysis: groups are found using an index (e.g., if an individual score above certain number, she receives the benefits) • Differences in differences: information at baseline and follow up for a group that received the program and for a group that didn’t • Propensity and matching estimators: very detail data at some point before the program
When do we apply each method? • Ideally, randomization is first best • Individual / geographic randomization: program is a pilot and not universal • Phase-in randomization: program is universal and implementation is done in steps • RD • Program is targeted using an instrument • DD and Matching: • If there is good amount of information and the program is not universal
Who should be pay for an evaluation? • If the evaluation produces a public good the evaluation should be finance by governments / multilateral institutions. • If the evaluation produces a private good (e.g., the results are appropriated by the private agent), the evaluation should be finance by the private agent • If so, what are the incentives to provide a real, strong evaluation?