Strengthening Accountability in Primary Education in Madagascar : Results of an Impact Evaluation

1. StrengtheningAccountability in Primary Education in Madagascar:Results of an Impact Evaluation Jee-Peng Tan & Cornelia Jesse, HDNED Chief Economist Office & Results for Development Institute Seminar “Demanding Good Governance - Inside & Out” March 25, 2010

2. Primary Education in Madagascar: Much Progress, but stillMany Challenges • Signs of progress: • Primarycompletion rate doubledfrom 35% in1999 to 71% in 2008 • Evidence of weaksector performance: • ½ of eachcohort of 1st graders does not finish the primary cycle; • Repetition rate stillhighat 18% in 2005 (30% in 2000) • Lowquality: in 2004-5 PASEC, average test score of 50% in Maths and Malagasy and 32% in French; deterioratedsince 1997-98 • Multiple systemic causes : • Inconsistencies in teacher allocation acrossschools; • Ineffective management of pedagogicalprocessesatschool and classroomlevels

3. A Problem of Internal Accountability Tasks essential for studentlearning are neglected: • Inadequate supervision of pupil and teacherabsenteeism e.g. more than 80% of directorsfail to report teacher absences to administratorsat the sub-district and district levels • Neglect of basic pedagogicaltasks e.g. 20% of teachersdon’tpreparedailylesson plans • Poor monitoring of and communication on studentlearning e.g. results of student tests and quizes are poorlyrecorded and communicated to parents, if at all; schooldirectorshardlyfollow-up on student performance: 3/4 don’tdiscusslearningoutcomeswiththeirteachers

4. What tasks are deemed essential? • Teacher: • Takes daily roll call • Prepares daily lesson plan • Prepared bi-monthly lesson plans • Monitors student learning • Has tested pupils during the past two months • Helps lagging students • Discusses student learning issues with the director • School director: • Keeps a register of enrollments • Signs off on daily roll call • Analyzes student absences on a monthly or bi-monthly basis • Reviews pupils’ test results • Takes stock of teacher absences • Informs sub-district or district officer about teacher absences • Follows up with teachers on lesson planning

5. Tighter Management to Improve Accountabilty Conceptual Intervention Framework & IE Design: • Workflow tools to clarify tasks and internal accountabilities; • Facilitation of meetings between school and community; • Better information flows within school and between school and community; • Structured training for teachers and school heads Leading to: • improvement in actors’ behavior through bottom up and top down accountability • better managed school • increased school quality • higher student learning

6. Key Questions for Policymakers • What is the impact of tighter management of processes on school functioning and student performance? • At what administrative level are management interventions the most effective (school, district or inspection level)?

7. Impact Evaluation Design (1) Method: Randomized experimental design over 2 school years Interventions: • Specify actors’ responsibilities & their mutual accountability the processes through: • Management Tools and Guides for key tasks (e.g. pedagogical, administrative) • Training • Focus attention on results to clarify goals through: • Report cards:School, district and inspection report cards • School meetings:Facilitated school meetings & development of school improvement plans based on school report cards

8. School & District Report Cards for Better Information Flow • Report cards for school directors, sub-district and district levels officers: • Complement the tools and proceses • Draw attention to schooling outcomes • Include comparative data, allowing a school to compare its outcomes with those of other schools • Serve as basis for dialogue and accountability

9. 15 CISCO AGEMAD 15 CISCO CONTROL 89 ZAP AGEMAD 84 ZAP NON-AGEMAD 80 ZAP CONTROL 303 Schools CISCO AGEMAD TREATMENT 1 303 Schools CONTROL Impact Evaluation Design (2) 303 Schools AGEMAD TREATMENT 3 303 Schools ZAP AGEMAD TREATMENT 2

10. Collecting Data Actors’ Behavior (direct effects): • Questionnaire from impromptu school visits in 1,200 schools, with information for 4,000 teachers • Questionnaires for District and Community admin. level • Collection and analysis of tools used in 40 schools (850 tools) Schooling outcomes (indirect effects): • Test scores from standardized tests in 3 subjects • National year-end school census data: flow rates, repetition, CEPE pass rate Timeline: • 2 school years, 2005-2007 • Baseline survey/test and post-intervention survey/test

11. Results: Effects on Actors’ Behavior

12. Results: Effects on Schooling Outcomes

13. Policy Implications • Prioritize school-level actors • “Cascade” training model alone, as currently defined, doesn’t work • Though results are encouraging, better management essentially entails changing peoples’ behaviors, which takes time and effort • Mainstream IE results into MoE activities • Need a champion from the start • Need early involvement of a national team, with good technical support • Necessary to sustain change in actors’ attitudes & behaviors • Use existing structures and mechanisms for scale up: • Tools, guides and training modules integrated into teacher training • Tool distribution, training and facilitated school meetings funded through the local catalytic funds based on regional, district and school performance plans and needs • Develop leaders to drive change in management practices • Discussion underway on collaboration in leadership training between Madagascar MoE and partner organization in another country

14. Stay tuned…Publications forthcoming • Africa Human Development Working Paper Series «Améliorer la gestion de l'enseignement primaire à Madagascar - Résultats d'une expérimentation randomisée » • Journal Article undergoing peer review «Managing for results in primary education in Madagascar: Evaluating the impact of selected workflow interventions »

15. It takes a village… • Government commitment: • Stable counterpart team (15 staff from MoE with coordinator) • Partner commitment: • Financial and technical assistance from AFD (via two staff) • WB team lead by Jee-Peng Tan and Cornelia Jesse, consisting of Gérard Lassibille and Trang van Nguyen (with in-country field coordinators) • Local NGO Aide et Action to assist with training • Financing: WB, AFD, MoE, EFA-FTI (EPDF), Irish Aid, Ministry of Foreign Affairs of Norway • Timeline: 2004 – 2007 • Total number of people involved: 50

16. The Perils of Data Collection… Thank You!