Report Cards: The Impact of Providing School and Child Test-scores on Educational Markets

1. Report Cards: The Impact of Providing School and Child Test-scores on Educational Markets Jishnu Das (World Bank) With: Tahir Andrabi (Pomona College) Asim Ijaz Khwaja (HKS, Harvard)

2. The Context Pakistan Rural India • Private Schools have expanded dramatically since the 1990s in South Asia

3. The Context: Its not what you think! • Although much discussion about madrassas, this is not where the action is!

4. The Context: The “New” Village Environment A village in Central Punjab A village in North Punjab • In our sample of 112 villages, there were 812 schools • 50% of rural Pakistan’s most populous province—Punjab—live in villages like one of these two

5. The Questions • Can better information about school performance take advantage of the market structure to improve educational outcomes? • What is the equilibrium impact of information on educational markets? • Quality • Price • Quantity • Is there a strong case for the provision of better information? • Address these issues using an experimental design in Punjab province, Pakistan

6. Precursors • Information can lead to a number of different types of results! • Positive: Information leads to greater accountability/verifiability, competition • (Bjorkman & Svenson (2008) – community-based health reporting/monitoring in Uganda; • Jin & Leslie (2003) – Restaurant Hygiene cards in LA • Rokoff and Turner (2008); Chiang (2008) – public school accountability in US; Hastings (2007) • Nothing: Information may be known, not understood/credible/believed; • Banerjee et al. (2007) - no learning improvements from information dissemination (Indian state) • Negative: Information may lead to greater cream-skimming/sorting (winner takes all) • Education : Chile (Urquiola and Mizala 2007) • Health: Dranove et al. (2003) – hospital outcomes in NY • Direct Manipulation: US – cheating teachers (Jacob & Levitt, 2003) • Gaps • All market-level studies are observational (Dranove and others, Urquiola and Mizala, Jin and Leslie. • Experimental work thus far either in cases where markets are sparse or market reactions not examined • First experimental equilibrium results on impact of information in education

7. Remainder of talk • A note on private schools • The data • The experiment • The Results • A note on the results

8. A note on private Schools Probably causal differences Learning • All unaided, very sparsely regulated, co-educational, mostly small “mom & pop” operations • Better learning than public schools

9. A note on Private Schools (II) • And cheaper, too!

10. Data (I) • Sample: • 112 villages from 3 districts in the Punjab, Pakistan • Randomly selected from list of villages with at least one private school in 2000 (3rd of villages & 50% of pop); somewhat bigger/richer than average village • Defining Schooling Markets: • Goal – capture parents/children complete choice set & schools’ potential market • Feasible: • 92 % of children attend village school (HH census) • Large distance effect – most primary-school children go within 15 minutes • Create 15 minute (30m for RYK) boundaries around village HHs (include some schools right outside village boundary) (Figure 3)

11. Data (II) LEAPs Project - www.leapsproject.org: • Baseline HH census (80,000) • Four Rounds (2004-2007): • School (823) Questionnaires: • General School Questionnaire • Class Teacher Questionnaire • Head Teacher Questionnaire • Educational Performance: • Child-Tests (Follow 12,000 plus children over 4 years) in English, Urdu and Mathematics • Household-Level Information • Detailed Household Interviews for randomly selected HHs (1,800) • Short school-based Child questionnaire (randomly select 10 in each school)

12. Data (III) • Survey Instruments & Timeline: • HH census (80,000 hhs) – 2003 • Round 1 (Baseline): • School-Based (Jan-Feb 04): (i) 823 primary schools + class 3 teachers; (ii) 800+ Class 3 teachers; (iii) 6,000 class 3 kids (brief info) • HH-Based (March-Apr 04): Detailed HH surveys (1,800); part matched on class 3 children • Child-Tests (Jan-Feb 04): 12,000+ Class 3 children – Norm-referenced test to maximize variation – Use Item Response Theory to get at underlying child knowledge; we administer (minimize cheating etc.) • Report Card Intervention – Sept/Oct • Round 2 (2005): Report all of Round 1 Surveys/Tests (96% children tracked)

13. The Experiment • After baseline, villages within each of 3 districts divided with equal probability into treatment and control • Report Card provided to each Class 3 kid parent in school-meeting – explain scores Parent Card 1: Child Info • In all 3 subjects (Maths, Urdu & English): • Child score and quintile • Child’s School score & quintile • Child’s village score and quintile • Quintile described as “needing a lot of work” to “very good”

14. The Report Card Intervention Parent Card 2: Village Schools Info • For all Primary schools in villages give : • School Name • Tested Children • School scores and quintiles in all 3 subjects • “Bundled-Impact”: • Information (child, schools) • Increase precision, verifiability • Meeting effect?

15. A note on the information • This is not value-added information • Why? • Feasible intervention • Theoretical considerations (who can back out VA better?) • Empirically doesn’t look too bad • Nevertheless, combination of selection and measurement error may lead to erroneous inference by parents

16. A further note on the information • Reliability vs. Measurement Error (Kane & Staiger) • Information is fairly reliable • Low measurement error of test • Large variation across schools - see Figure • Selection (into schools) • Value-added estimates? • Selection Not as severe (see learning gaps Figure) • Need to have Information be clear and understandable • Policy feasible/relevant • Households may be better able to back out value-added • village w/ 15 schools; test-score & (2) standard-error bands (computed using IRT)

17. What should we expect? • 3 Broad Classes of models • Symmetric information • Some unobservable components of quality for both schools and households • Asymmetric information: Price signals quality • Asymmetric information: Price does not signal quality • In Model 1 price declines for all schools; depending on structure of demand can get heterogeneous declines by initial quality; quality weakly improves • In Model 2 price declines more for initially higher performing schools; quality weakly improves • In Model 3 price/quality movements are ambiguous

18. Results: Quality Notes Learning: • Similar across subjects; holds 2 yrs after Attrition: • Unlikely concern: no difference in baseline scores for attritors between treatment and control samples Switching/Dropouts: • Results entirely driven by children who stayed in same school: • Few Switch Schools (5%); Gains similar if restrict to non-switching children • For gains to be attributable to switchers, need switchers to have gained 1.7sd, given numbers---highly unlikely!

19. Results: Quantity Notes Enrolment • Large increase in RC villages (almost 5 percent) • Entirely from Government schools, entry into Grade i Switching • No evidence of increased churning • But evidence of differential churning School Closures • Significant among initially low performing private schools

20. Results: Price (Private Schools Only) Notes Fees • Large Declines—24 percent across the board • Larger in initially higher performing private schools • These are reported by the school; we obtain identical results using reports from households

21. Results: Schools or Households? Notes Household • Little evidence of any big changes (consistent with Das and others 2009) • Children in bad private schools are now playing less • Sleeping more • Spending more time in school Schools • Private schools increase teacher eduation, textbooks • More time on task—fewer breaks

22. Conclusion • RC increase learning and/or fees drop – equity and efficiency both increase? • Results depend on pre-existing market conditions (parental demand; eductaional production function - school type vs. effort) • Cost of Intervention ~ fee drop • RC exercise cost $1 per child (testing, grading & dissemination) • Cost savings ~ $3/child in private schools (1/3rd of all children enrolled in private schools) • Welfare calculations? • Tricky: Typical Cost-Benefit calculations in LIC ignore welfare costs for providers → learning gains free of cost • BUT: complete welfare analysis has to factor in provider welfare loss – transfer from school to parents & decline in rents – need more structural approach • Policy Questions & Caveats: • Public vs (socially cheaper) Private sector • Intervention simultaneously improve private sector (equity, efficiency) and public sector – State’s role as information provider (rather direct regulator)

23. Further Notes or how I began to worry that this may actually lead to policy • Theory: Outlined 3 classes of theory; there are others • Question: Why changes in provider behavior, but no increased churning? • Alternative Question: (Hastings): Why switching but no change in provider behavior? • Answer: We don’t know the dynamic equilibrium process in control villages (ratcheting?) • Empirics: Is this a big effect? • CANNOT compare SD increases across tests • Can simulate changes from 0.05sd to 1.3sd by changing the test! • Have to link to some cardinal change (see Heckman) • Trying to calibrate to TIMMS using identical questions • Can answer: how big is this change relative to the world distribution • Longer-term effects (up to 5 years later) • How do we treat utility of providers in welfare computations? • Feasibility • This is proof of concept; mainstreaming is a different issue