Felipe Barrera-Osorio Dhushyanth Raju (The World Bank)

1. Evaluating a test-based subsidy program for low-cost private schools: Regression-discontinuity evidence from Pakistan Felipe Barrera-Osorio DhushyanthRaju (The World Bank)

2. Overview of the study:Strategy and Results • Strategy of estimation • Since entry into the program is a function of a minimum student pass rates in the test (average at the school level), we used a Regression Discontinuity approach in two entry school waves (Phases 3 and 4) • We used both administrative and own collected data via phone survey to schools near the discontinuity point • Results • For Phase-4 applicants (sharp design), we find large positive impacts on the number of students, teachers, classrooms and blackboards. Students to teacher ratio remains constant. Impact emerging within a short period • Using Phase-3 (fuzzy design), we did not find statistically significant impacts.

3. Program setting • In general, Pakistan has poor performance in educational indicators • Participation rate in formal school (grade 1+) of children ages 6-15 was 65.7% • While 91.8% of individuals ages 12-16 completed primary school (grade 5), only 40.7% completed secondary school (grade 10) • Mean scores in national assessments less than 50% • Public sector the dominant provider of education with weak accountability and weak incentive systems. • Dramatic growth since mid 90’s of the private school sector (Andrabi et al 2007; Andrabi et al 2006; Das, 2006) • 25% of school-goers (6-17 year olds) in private school • Specially for the poor and in rural areas • Fees in private school are generally low: median annual fees per student in 2000 were 960 (rural area) and 751 (urban) rupees • The government formed an partnership with the private sector in order to increase enrollment and quality of education, specially for disadvantaged households

4. Foundation-Assisted Schools (FAS) program structure and coverage • Administration: Designed and administered by the Punjab Education Foundation, a semi-autonomous organization • Objectives: Increase school participation and achievement levels among children from disadvantaged households • Timeline: Program established in 2005 and expanded in phases (4 phases completed). • Coverage: • Districts: 18 out of the 35 districts in Punjab; 87% of schools in 7 districts; • Schools: 1,082 low-cost private schools; • Students: 474,000

5. FAS program: eligibility conditions • Initial benefit eligibility conditions • Minimum enrollment size of 100 students. • Pass a qualitative inspection conducted by PEF. • At least 67% of students tested have to score 33% or higher on the test. • Continued benefit eligibility conditions • Maintain minimum enrollment size of 100 students. • Eliminate all tuition and fees for all students: signboard outside school gate which announces tuition-free schooling and provides PEF contact information. • Report enrollment figures to PEF every month. • At least 67% of students tested have to score 40% or higher on the test. • First violation: Penalties (e.g., frozen enrollment). • Second violation: Permanent program disqualification.

6. FAS program: Benefits • Subsidy: Rs. 300 (US$4.3) per student per month. • Subsidy level set at upper-end of price range for low-cost sector. • Use of subsidy largely unfettered. • Teacher bonus: Rs. 10,000 (US$143) per teacher per year for 5 teachers in schools in which at least 90% of students in tested classes obtain a score of 40% or higher in the Quality Assurance Test. • 370% of mean monthly teacher salary at baseline. • School bonus: Rs. 50,000 (US$714) to the school in each district with the highest pass rate in the QAT per year. • 76% of mean monthly subsidy payment to schools given mean enrollment size at baseline.

7. FAS program: Tests • The entry test: Short-listing Quality Assurance Test (SLQAT): • To be part of the program, at least 67% of students tested have to score 33% or higher in the SLQAT • Pared-down version of QAT • Developed and administered by PEF. • Offered to two to three grades in a unannounced visit to the school • Continuation test: Quality Assurance Test (QAT): • For continuation in the program, at least 67% of students in tested grades have to score 40% or higher in the QAT • Based on samples of syllabi and textbooks used in low-cost private schools. • Selected subjects: English, Urdu, Mathematics, and Science. • Offered twice a year in two or three grades

8. FAS Program: Application process and Number of schools

9. Identification • Entry process: • Schools apply when a call for applications is issued. • Schools with qualifying applications are subject to a physical inspection • In phase-3 and phase-4, schools that pass the physical inspection are offered the SLQAT • Assignment to FAS program (treatment) based ultimately on school’s pass rate on SLQAT • To be part of the program, at least 67% of students tested have to score 33% or higher in the SLQAT • Virtually all schools that attained at least the cutoff, accepted treatment (take-up rate of 94% in Phase 3 and 98%, in Phase 4) • No treatment dropouts to date

10. Local smoothness in baseline mean outcomes • Comparison of means for those that marginally fail the SLQAT and for those that marginally pass: no statistically-significant differences, except for mean SLQAT score (scores positive correlated with pass rates). • LLR estimates: Some statistically-significant breaks but nothing systematic to suggest a breakdown of the conditional smoothness condition for RD identification.

11. Data used for the evaluation • Data before the program • Source: Administrative data. • Data on school characteristics and non-learning outcomes obtained from applications maintained electronically by PEF. • Data on SLQAT pass rates and average school test performance obtained from individual student test score data maintained electronically by PEF • Number of schools: 747 (Phase 3) and 830 (Phase 4) • Data after the program • Source: Phone interviews (schools phone numbers in applications) • Data collected on school characteristics • Number of schools (just passers and just failers): Phase 3: 192. Phase 4: 236 • Data collected 14 and 10 months after first subsidy payment to phase-3 and phase-4 entrants (partially spans two academic years)

12. Main findings • Large positive effects on enrollment, teachers, classrooms, and blackboards; within short treatment period of 10 months • Conservative estimates: +85 children (37% relative to baseline mean); +3.4 teachers (37%); +4 classrooms (47%); and +2.8 blackboards (27%) • The annual cost per student of increasing enrollment by 1% is 97 rupees (US$1.4) • Among the lowest cost-effectiveness ratios estimated

13. Ongoing studies • More evaluation research is planned with several data sources • Annual public school censuses • The Punjab Examination Commission test score data • Additional FAS program administrative data from PEF • Primary survey data from samples of low-cost private schools • Some example of key questions • What are the impacts of the FAS program on student learning • Does the FAS program have any impacts on the socioeconomic composition of their students (particularly in terms of gender and economic background) • Does the FAS program induce the displacement of students from neighboring public schools and to what extent