1 / 17

Operationalizing IE: Case Study example: Textbooks and Teacher Training in Sierra Leone

Operationalizing IE: Case Study example: Textbooks and Teacher Training in Sierra Leone. APEIE Workshop, May 10-14 2010. Operational Key Steps. IE Design – methodological issues. Available data and required data Identify control group and treatment group Randomization Transparency

prentice
Download Presentation

Operationalizing IE: Case Study example: Textbooks and Teacher Training in Sierra Leone

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Operationalizing IE:Case Study example: Textbooks and Teacher Training in Sierra Leone APEIE Workshop, May 10-14 2010

  2. Operational Key Steps

  3. IE Design – methodological issues • Available data and required data • Identify control group and treatment group • Randomization • Transparency • Sampling and sample sizes • Power calculations • Budget and access constraints • Identify outcome indicators

  4. Data Instruments • Utilize survey experts • Who do you survey? • Households, community, facilities, government… • Pilot instruments • Clear questions and coding • Includes all information needed for analysis and covariates

  5. Intervention Roll Out • Possible contamination of evaluation results! • Intervention benefits rolled out to control and treatment schools  evaluation compromised • Intervention or treatment schools only receive additional benefit  evaluation compromised • Monitor roll out • Train implementers – they need to know why! • Hire supervisors • Continue supervision throughout roll out

  6. Background • In June 2007, the World Bank and a team from the Sierra Leone Ministry of Education and Sports (MOES) met to rigorously evaluate the impact of two pilot studies on student performances. • Two questions • What is the effect of textbooks on student performance? • What is the effect of teacher training on student performances?

  7. How do we design the study to get rigorous results? • Random Selection • 5 districts in Sierra Leone • Kailahun, Kambia, Pujehun, Western Rural, Western Urban • Neediest schools in district (using EMIS data) • Student to teacher ratio, presence of latrine, school construction • 360 schools total • Random Assignment • Comparison schools: receive regular allotment of books • Books schools: receive additional books • Books and training schools: receive additional books + teacher training

  8. Timeline Sept: WB and MOE agree on pilot Oct: Met with local councilors to select districts TARGET Nov: Baseline data collection and teacher training manual developed Dec: Roll out of textbooks and teacher training Nov: Mid-term data collection 2007 2008 2009 June: Baseline data collection June: Roll out of textbooks REALITY NO teacher training up to this point Dec: Mid-term data collection

  9. What happened? • Met with several challenges on the ground that caused significant delays in the project: • Textbook distribution • Teacher Training • Data Collection

  10. Challenge 1: Textbook Distribution • Target date: December 2007 • Actual date: June 2008 • Number of books: • MOES accidentally distributed original books, so had to use Sababu books • had fewer books than project initially called for • Communication – information not filtering down • Sababu not properly informed • Service Providers unwilling/unable to change distribution • Headteachers in comparison schools unaware of project

  11. Challenge 1: Textbook Distribution continued • Textbook distribution: June 2008 • Data collection: December 2008 • Consequences: • Not enough books distributed in some districts to make a statistical comparison • Distribution happened very late and at a bad time (right before summer vacation) • Allowing for only a small window for students to utilize the extra textbooks • Some Comparison schools received books

  12. Challenge 2: Teacher Training • Target date: December 2007 • Actual date: has not yet occurred • Time and budget constraints • The ‘+’ part of the ‘intervention +’ so took a back seat to the textbook distribution

  13. Challenge 3: Mid-term Data Collection • Target date: November 2008 • Actual date: December 2009 • Delayed funding disbursement • Non-existent schools (poor quality of initial data) • Headteachers unaware of project – not willing for enumerators to collect data

  14. Solutions: • Back up list of schools on hand for enumerators • Better communication with all levels of organization involved • Constant project sensitization • Greater monitoring of distribution • Inclusion of Process Surveys • Allow for sufficient time for procurement and disbursement

  15. Next Steps • Analyze mid-term data • Was there a significant effect due to textbooks? • Has education in Sierra Leone improved in the past 2 years? • Where in the textbook distribution are there major informational stop gaps? • Move forward on teacher training • Write up findings and circulate report

  16. Conclusion • Impact evaluations a useful tool for policy considerations • But need to understand every process involved in the impact evaluation • Textbook distribution, not just textbooks • Study can be ineffective if you don’t have full buy-in and understanding at all levels of government, all service providers, and headteachers: • SENSITIZATION! • COMMUNICATION! • MONITORING!

  17. THANK YOU

More Related