Presented by Dr Dragana Sretenov, Learning and Impact Assessment Adviser, Save the Children UK

1. Education in Conflict Affected Fragile States: Global Evaluation 2008-2010 Nepal Presented by Dr Dragana Sretenov, Learning and Impact Assessment Adviser, Save the Children UK April 2009

2. Presentation Outline • Global Terms of Reference • Methodology • Findings in 4 Countries • Data Use Plan Note: Dr.Ruth Naylor is the consultant - Global Researcher leading this research. Angola

3. Global Terms of ReferenceResearch Questions & Country Focus • How have Save the Children’s project level interventions contributed to quality primary education for children affected by conflict? • Which project level interventions have had what impact on the education quality of children affected by conflict? Afghanistan: Child & Parent Participation Nepal: Schools as Zones of Peace Southern Sudan: Teacher Training Angola: Teacher Professionalism

4. Global Terms of Reference Objectives • Contribute to technical knowledge about what works and why • Contribute to internal and external documentation used in advocacy, communications, and fundraising • Contribute lessons learned into the internal organizational and operational decision making process Audience • Donors* • External Communities of Practice* • Governments & International Bodies • Headquarters policy advisors and country office programmers Timing • 2008 Midterm (formative) 2010 Final (summative)

5. Global Terms of ReferenceIn Country Methodology • Design: in-country participatory evaluation design and implementation. Country specific • Key Informants: students, child clubs, teachers, PTAs, partner organizations, parents • Data Collection: focus groups, lesson observations, interviews, learning assessment, secondary data • Sample: 6 schools per district, 3-5 districts per country

6. Global Terms of ReferenceParticipatory Design Agenda • Identify Research Team • Identify Advisory Committee • Narrow scope: effective, unique, innovative • Select intervention package • Map intervention program theory • Identify indicators for program theoretical framework • Identify informants • Develop tools • Pilot & Capacity Building

7. Global Terms of ReferenceTools and Sample Data Collection Plan (Angola)

8. Global Terms of Reference Afghanistan Questions? Nepal Southern Sudan Angola

9. Nepal: Schools as Zones of Peace

10. Program Theory: Schools as Zones of Peace Save the Children Partners Village Child Protection Committee School Management Committee Child Club • Bring together community members and political groups • Develop, agree on and display codes of conduct Items on codes of conduct: no armed activities in school; no weapons in the school; no use of children in political activities; no political meetings or graffiti in school; no arrest, abduction and torture of children and teachers; no school closure for meetings; no corporal punishment; no cattle grazing; no alcohol or smoking in school.

11. Nepal: Successes of SZOP • Increased sense of security in schools “Though some fear is still there with ongoing conflict, we feel safer than earlier and also do not feel fear while sending our children to school.” Parent, Mahottari • In 2007 project schools on average were open 12 days more than comparison schools (of 220 days) • Informants at over half of project schools reported improved student and teacher attendance • Evidence of beating was observed in one project school only “Instead of beating children, teachers now support and counsel them.” Parent, Palpa • 11 out of 16 project schools had child representatives in the SMC • 12 out of 16 project schools had developed thier own codes of conduct

12. Nepal: Challenges • Monitoring systems for SZOP not well established • Difficult to establish SZOP in contexts with limited experience of community participation in school development • Ensuring that SZOP is accepted and respected by political parties and armed groups where conflict is ongoing Nepal

13. Methodological Challenges Nepal • Partners implementing at the school level need to be involved in the design workshop • Sampling plans must adapt to local conditions (weather, migration, and conflict) • Time for staff capacity building should be built into piloting. • More emphasis needed during design on participatory research methods for working with children • School level data from districts to be collected prior to researcher arrival in country and made available at workshop • Checklist sent to country offices giving requirements in terms of what needs to be done before, during and after the visit, who should be involved and for how long

14. Afghanistan: Child and Adult Participation

15. Afghanistan: Child and Adult Participation for Quality Education Children’s participation in school • Student Councils • Student representation in PTA • Student led education survey Children’s participation in learning • Child-to-child health education • Training teachers in participatory methodology • Resource centres, mobile libraries Parents’ participation in school • PTA training and support

16. Afghanistan: Successes Student group and PTA success stories • Follow up on absent students • Recruitment of new teachers • Construction/maintenance of classrooms, boundary walls, latrines and school water supplies • Reduction of corporal punishment • Children learn about health through teaching it Training provided by Save the Children was more effective when it was followed up by support to carry out a concrete action

17. Afghanistan: Challenges • In two of the mixed project schools visited there were no females in the Student Councils • Only five of the 12 project schools visited had mothers in their PTAs, and three of these were girls’ schools • 12% of teachers in formal schools are female Participation of girls and women PTA training: spot the mothers

18. Methodological Challenges Afghanistan • Translation of reading assessment to local language/dialect is critical in cases where instruction is done in local language (e.g., Paghman ALC). Also, be careful about key words on assessment. • Difficult to record all information. Need to have group discussion immediately following visit. • Random selection of children for assessment can be difficult to orchestrate. • Security situation may cause last minute challenges / changes in plans • Full course of data collection is very time consuming and can be logistically challenging, with short hours and unpredictable constraints. • Very difficult to avoid the bias of the most remote schools being underrepresented for practical reasons.

19. Global Evaluation: Angola • How have Save the Children’s project level interventions contributed to quality primary education for children affected by conflict? • Which project level interventions have had what impact on the education quality of children affected by conflict? Afghanistan: Child & Parent Participation Nepal: Schools as Zones of Peace Southern Sudan: Teacher Training Angola: Teacher Professionalism

20. Angola: Supporting Teacher Professionalism Putting our knowledge to use

21. Angola: Teacher Professionalism School directors Provincial Department of Education (DPE) – main partner Supervisors Save the Children Other teachers Teachers Students Parents

22. Angola: Successes • 93% of the teachers in schools visited had some training. • Most teachers had been observed by supervisors and school directors • Half of teachers observed had a lesson plan • Respondents reported Improved teacher conduct (reduction in drunkenness and exchanging grades for money)

23. Angola: Challenges • Only ¼ of grade 3 students tested were able to read well enough for simple comprehension • There was no significant difference found between the teaching of trained teachers and that of untrained teachers • 7 cases of teacher misconduct identified (in 14 schools + 6 ALCs) • 12 out of 16 student focus groups reported physical and humiliating punishment in schools

24. Angola: Student Learning Findings • Only ¼ of grade 3 students tested were able to read well enough for simple comprehension • Over 50% of students tested were unable to read a single word • Most children tested could solve simple addition and subtraction problems, including word problems

25. Angola: Teaching and Learning THANK YOU!

26. Methodological Challenges Angola • Context of very rapid economic and social change including major changes in education (new school buildings, distribution of books, mass teacher training programmes). • School level data on dropout and changes in enrolment is highly influenced by migration (urbanisation) and construction of new schools, and hence does not reflect quality (or access gains). • Evaluators measurement of teaching quality is restricted by low exposure to good teaching.

27. Southern Sudan: Teacher Training

28. Southern Sudan: Teacher Training • Phase training • In-service training • Women into Teaching • Intensive English Course • Accelerated Learning Programme teacher training

29. Southern Sudan: Successes According to respondents, teachers trained by Save the Children: • know how to structure lessons • adapt lessons to the learner • have increased confidence • have improved English skills • use less corporal punishment Teachers with training from Save the Children scored significantly higher in lesson observations than those without.

30. Southern Sudan: Challenges • No formal follow up of the training • 60% of enrolled students absent from lessons observed. • Only 10% of teachers in the evaluation counties were female • Only 27% of students in the evaluation schools were female. • Dropout rate for girls was 16%, and for boys 5%

31. Southern Sudan: Learning findings • 19% of grade 3 students tested could not read a single word • Boys score higher then girls in reading, comprehension and mathematics • Multivariate analyses of the learning assessment results indicated no significant difference between the reading and comprehension scores of students from project and comparison schools • In mathematics students from comparison schools scored significantly less • Teacher training level did not appear to make a difference in the learning outcomes

32. Methodological Challenges Southern Sudan • There is a need for a higher level of involvement from the country office team during the evaluation design training. • Due to geographical and funding reasons the design workshop had to be separated from the training workshop for data collection, and the workshops were organised in 2 distant part of the country, which limited the possibility of the head office staff to take part in the data collection workshop.  • The difficulty in undertaking the education research in Southern Sudan is also related to inaccessibility of schools in some parts of the country during the rainy session when schools are fully running. Project schools in each county were to be selected at random but flooding in two of the counties restricted the number of accessible project schools, so there was a bias towards the more accessible schools.  • Comparison schools were chosen to match the project schools in terms of their distance from the nearest town as there was little other data available to base the selection on.  • The enumerators had limited knowledge of teaching processes and the classroom observation tool had to be modified and simplified. The originally developed classroom observation tool was used by the international and local researcher only. • There is a need to enable greater child participation in the data review process, which proved to be very productive in other countries.

33. Moving towards data use • From our evaluation we know: In Angola, around half of the children tested (project and comparison) could not read a word in a simple Portuguese text In Nepal, a quarter of the children tested could not read a single word in a simple Nepali text In Southern Sudan, of children tested, girls’ reading accuracy was 30% lower than boys (43% compared to 73%) In Afghanistan, 40% of children tested in SC supported schools could read and comprehend a simple text, whereas 1/32 students tested in comparison schools could So what do we do now?

34. Evaluation Findings Data Use Plan • Global Evaluation Executive Summary (4pgs) • Global Evaluation Midterm Report (20 pgs) • External sharing of lessons learned, e.g. GMR, AEA, Education in Emergency Conference • Follow up action plan at country office & headquarters • Linking with other evaluations on related topics

35. Education in Conflict Affected Fragile States: Global Evaluation 2008-2010 Questions? Nepal

36. Education in CAFS: Global Evaluation 2008-2010 • How have Save the Children’s project level interventions contributed to quality primary education for children affected by conflict? • Which project level interventions have had what impact on the education quality of children affected by conflict? Afghanistan Nepal • For more information: • Angola – SC Denmark Gretehe Markuson GM@redbarnet.dk • Afganistán – SC US Rita Oswald Cristiano rchristiano@savethechildren.org • Nepal – SC Norway Inguun Nakkim • ingunn.nakkim@reddbarna.no • Southern Sudan – SC UK Dragana Sretenov d.sretenov@savethechildren.uk • Cynthia Koons, M&E Manager, Inernational Save the Children Alliance cynthia@save-children-alliance.org Southern Sudan Angola