Pitfalls of Participatory Programs: Evidence from a randomized evaluation in education in India

1. Pitfalls of Participatory Programs:Evidence from a randomized evaluation in education in India Abhijit Banerjee (MIT) Rukmini Banerji (Pratham) Esther Duflo (MIT) Rachel Glennerster (MIT) Stuti Khemani (The World Bank)

2. Motivation • Community participation is being pursued in policies and programs to improve the quality of public services • We evaluate the impact of mobilizing communities (using information & advocacy) to participate in publicly provided education services

3. The Study Location • In Uttar Pradesh (UP)—a “typical” district, Jaunpur

4. Institutional Context in UP • Village-level education committees (VECs), representing parents and village leaders, created by state education policy in India since the 1990s • VECs have specific powers to monitor quality, and manage public resources to improve quality • Can recruit community-based additional teachers

5. Findings from Baseline Survey • Baseline Survey (March – June 2005), 280 villages: • Learning outcomes • 30 households randomly selected in each village • all children between the ages of 7 and 14 tested on basic reading, writing, and math skills • Community participation and local governance • All VEC members interviewed • All government school head teachers interviewed • 10 households (from among the above 30) interviewed • School resources and school functioning • All government schools surveyed through interviews and direct observation

6. Have you heard of the VEC? Villagers Who Don't Know of a Village Education Committee Villagers Who Think there is a VEC Villagers Who Believe there is a VEC, But Can't Name Any VEC Members 5.0% 7.6% 92.4% Villagers Who Can Name Only One or Two VEC Members 1.1% 1.5% (the Pradhan and/or Headmaster) Villagers Who Can Name More VEC Members than Just the Pradhan and Headmaster * Based on 2,803 household surveys in 4 random blocks in the District of Jaunpur, UP. Each household is weighted by total number of households in village divided by number households surveyed in village. Parents don’t know that a VEC exists

7. VEC members don’t know their roles

8. Parents of children at low levels of learning tend not to know this…

9. Interventions to Encourage Participation: (1) Mobilizing communities through advocacy, and information about VECs • Small, informal meetings in each hamlet during 2 days in a village • Village-wide meeting on 2nd or 3rd day, with participation of key VEC members—Village Head and School Teacher • Distribution of pamphlets to VEC members listing and explaining their roles

10. Interventions to Encourage Participation: (2) (1) +Testing: Reading test for outcomes information, building monitoring capacity • In hamlet meetings, facilitators begin testing children; community invited to test children themselves and prepare hamlet-level “report cards” • In village-wide meeting, hamlet volunteers invited to present testing tools and “report cards”

11. Interventions to Strengthen Community Participation: (3) (2) + Teaching : Capacity Building to Directly Improve Learning • In village meetings, facilitators present Pratham’s program for improving reading • Offer to train any volunteers to hold reading classes

12. Experience with Interventions • 65 villages each received Interventions 1, 2, and 3 between September and December 2005 • Soon after a new village government was elected, and could constitute new VECs • Repeat visits in Jan-Feb 2006 to hand-out and explain pamphlets to VEC members • 85 villages served as controls

13. Experience with Teaching Intervention Local youth volunteered: 405 reading classes held No. of Reading Classes in Teaching Intervention Villages No. of Villages 12 10 10 10 8 7 6 6 4 4 4 4 3 3 3 3 3 2 2 2 1 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 16 No. of Reading Camps

14. Participation of 7453 children in reading camps in 55 villages: 135 children per village

15. Impact Evaluation End-line survey took place over March-June 2006 Specification (for each of K “families” of outcomes): • X: baseline levels of every outcome in the family • Difference-in-Difference estimates • White standard errors clustered by village • Additional controls in some specifications (parent’s caste, education, occupation, literacy; child’s age, gender, school status)

18. No impact on VEC activity • No impact on VEC awareness of additional teacher program

20. Only small impact on parent knowledge (only 5-10% of parents know about the VEC even at endline) • No impact on parent activism

21. Non impact on parent activism is confirmed by school teacher responses

22. No positive impact on school resources

23. No positive impact on teacher attendance or effort

24. No positive impact on student enrollment and attendance in government school

26. In “teaching” intervention villages, where reading classes were offered, small increase in children moving out of school, possibly because they were moving into reading camps

28. Improvement over time among illiterate children

29. Improvement over time among children who could recognize letters

30. Improvement over time among children who could read words or paragraphs

31. Summary of Impact • No impact at all on public schools, or on VEC activism • Learning improved because an offer to the community of a teaching program was taken up by local youth volunteers—reading classes were held • Children who enrolled in these classes made significant improvements in reading within a couple of months • No evidence, anecdotal or otherwise, of VECs, or Village Heads, or school teachers, supporting these volunteer-led reading classes, or mainstreaming the reading program in public schools

32. Why was there no impact on public schools? • Not because of lack of interest in education, or willingness to participate, or because it’s just too difficult to get children to learn • Contrast between impact outside versus within the public school system suggests that the real challenge lies in improving public provision

33. Conclusions • Significant barriers to participation, for social accountability of public providers • Instituting participation from outside, in lagging or disadvantaged areas, likely requires credible support from above—from government or influential NGOs • “Short route” to accountability (WDR 2004) likely requires going a long distance