Policy and Program Evaluation

1. Policy and Program Evaluation Logical Framework What is Program Evaluation Case 1: Indonesia PQEP Case 2: Honduras Educatodos Case 3: School Quality Program Summing Up

2. Logical Framework

3. The Logical Framework Approach is a project design methodology. Also know by the initials of the method in German ZOPP (Ziel Orientierte Projek Planung, or Objectives Oriented Project Planning or OOPP). This methodology, the Logical Framework Approach, calls for a structured sequence of activities that leads to the production of a Logical Framework (a Document).

4. The main steps in this process include: Analysis of the Present Situation • Analysis of Stakeholder Groups and Institutions Affected by the Problem • Problems Analysis • Objectives Analysis • Alternatives Analysis • Activities Planning • Logical Framework

5. The Logical Framework was developed and popularized in the design of USAID projects in the end of the 1960s and has been adopted widely by development organizations. • Key themes: • Objectives Oriented • Target Group Oriented • Participatory

6. Universal Primary Education = Poverty Reduced • Quality of Education Increased  National Productivity Increased • Confict Resolution Program Implemented  Social Violence Reduced

7. Distinguishing between Immediate and Development Objectives: • Both outside project control. Project hypothesis. • Cause-Effect.

8. Increased productivity of the labor force • Increased literacy of the population Increased healthy practices in population Higher educational attainment

9. Increased value added in information technology industries • Greater scientific knowledge of college graduates • Greater peace and security • More sophisticated international relations • More knowledge and awareness of global issues

10. Projects transform inputs into outputs. Activities are the processes that create the outputs using inputs. These three are within the project control. • Outputs are results than can be guaranteed by the project as a consequence of activities • Activities are actions necessary to transform given inputs into planned outputs within a specified period of time • Inputs are the resources of a project necessary to produce the intended outputs

11. Outputs • 100,000 graduates of literacy campaign Activities • 10,000 literacy workshops conducted in 2001 Inputs • 100,000 literacy booklets • 2,000 literacy promoters • 200 literacy coordinators • 500 all purpose vehicles • 2,000 sites for training

12. External Factors. Necessary and Sufficient Conditions. • External factors are necessary for project success but outside the control of project management. • Outside and inside the project. • Project Environment.

13. Examples: • Participants engage in training opportunities • Trained participants exhibit skills gained in training • Complementary resources (e.g. textbooks) are available. • Jobs are available • Parents send their children to new schools.

14. Critical factors need to be integrated as project outputs, when possible. Alternatively they need to be monitored carefully. • Indicators and Means of Verification. • Should specify: • Target group • Quantity • Quality • Time • Location

15. Examples: • Increased literacy, defined as ability to read and understand a short text. • Ability to design a project using a logical framework approach

16. The Logical Framework Matrix is a useful tool to systematize the hypotheses implicit in the project. It can be used by individuals or project teams, and often teams design their projects first and then retrofit them into the Logical Framework Matrix. This approach, obviously, misses the most important benefits of the approach, the possibility to organize thinking and to facilitate communication and dialogue among a range of stakeholders about problems and options for intervention.

17. In its original and most appropriate form, however, the Logical Framework Matrix should be the end product of a Logical Framework Analysis (process). The Matrix is always understood as a living document, a compact expressing temporary agreements and understandings among those with a stake in a project. It can (and should) be amended from time to time, as implementation proceeds and more is learned about the effects of the project and about the problems it is trying to affect.

25. SITUATION ANALYSIS • Situation Analysis. • Stakeholder Analysis (also called Participation Analysis) • Problem Analysis. • Objectives Analysis. • Analysis of alternatives. • PROJECT DESIGN • Specific Planning of Activities. • Preparation of a Detailed Logical Framework.

26. Situation Analysis. • This is an initial report to kick off the conversation. It describes the main issues defining the problem. It could be a needs assessment, a proposed project and feasibility, or an education sector-review.  • Develop comprehensive picture of the interest groups, individuals and institutions involved. TARGET groups whose views will be given priority in analyzing problems.

27. 2. Stakeholder Analysis (also called Participation Analysis) • This consists of political mapping of the groups affected by the problem under consideration, or with the ability to influence the problem. This political mapping answers the question: which interests are affected by this problem? This process should clarify the groups and their specific interests.

28. 3. Problem Analysis. In this step of the Logical Framework Process key stakeholders negotiate an agreement on the problem, it’s underlying causes and principal effects. One way to proceed is to get participants in the workshop identify their own ideas of problems and then connect those ideas in a problem tree (a sequence of cause-effect relationships). Then a group discussion seeks to integrate individual ideas into a group problem tree.  • Identify major problems and main causal relationships between them.

29. 4. Objectives Analysis. • In this step statements about problems are translated into positive statements about objectives. An ‘objective tree’ is a sequence of cause-effect relationships necessary to bring about desired objectives. Multiple objective trees can be generated.

30. 5. Analysis of alternatives. At this stage participants assess the alternative courses of action proposed by each objective tree in terms of effectiveness, costs, feasibility, impact on priority groups, risks or other criteria.  • Criteria can be: • Technical • Financial • Economic • Institutional • Social/distributional • Environmental

31. 6. Specific Planning of Activities. • Select Development and Immediate Objectives. • Once a particular course of action has been agreed upon, participants in the workshop identify measurement indicators and means of verification for each objective and identify the necessary activities and inputs necessary to achieve the objectives

32. 7. Preparation of a Detailed Logical Framework. • LFA is a useful communication tool to be used not just during project design, but also during implementation and evaluation. • Define Project Elemenents (Inputs, Activities and Outputs) • Identify External Factors • Indicators Arbitrary at which level to pitch the project in a problem ladder or tree. Definition of focal problem is an arbitrary decision, reflecting value preferences. 

33. A LFA can be used at various stages of the project cycle: • Project Identification • Feasibility Study • Project Design • Detailed Planning (Planning for Implementation) • Monitoring (Implementation) • Project Review (Mid-term project evaluation) • Evaluation

34. What is evaluation? • Why is it done?

35. Evaluation of school improvement through an educational effectiveness model: The case of Indonesia’s PEQIP Project • Van Der Werf, G., B. Creemers, R. de Jong and E. Klaver. • Why was the evaluation done? • What was the program about? • What are the conclusions?

36. 1992 Government initiated the Primary Education Quality Improvement Project (PEQIP). Study conducted between 1992 and 1997 • School effectiveness study. • An integrated school effectiveness model was developed which took into account PEQIP inputs and intended outputs, school and classroom level indicators of effective schools in Western countries, indicators of effective schools in developing countries, and the local context of schools, teachers and pupils in Indonesia.

38. Research questions • How large are the differences in student achievement between schools? • Are between school differences related to school participation in PEQIP? • Which school and classroom factors, indicating PEQIP implementation, are related to between-school differences in student achievement?

39. PEQIP • Launched 1990 • 440 schools • Books and materials • School funds • New instructional approaches • Professional development for teachers and administrators • School clusters • Four components: • Teacher development • Educational management –and community participation— • Books and learning materials • Evaluation and monitoring

42. Study in two provinces. 27 PEQIP schools from all three participating subdistricts. In each subdistrict three clusters of schools participated. Each cluster had a center school (Inti school) and six to nine satellite schools (Imbas schools). Plus 14 control schools in each province. • Bahasa Indonesia and Mathematics pre and post tests. • Science post-test • School year 1996-1997 • PEQIP participation did indeed have effects on student achievement, after taking prior achievement into account, but these effects are small, not always significant, and not always present for every subject in every category of school. Large unexplained differences remain between schools after controlling for PEQIP participation and context. • Implementation of components is not higher in PEQIP schools. Pequip is not implemented better in PEQIP schools than in control schools, which explains why the participation in PEQIP in general was not very effective. Copy table 5 page 346 • Multilevel model to examine the extent to which the variables in PEQIP explain achievement. The variables indicating the implementation of PEQIP account for almost half of the school-level variance. • Pretests administered in fifth year of project.

43. Educatodos • Quality and Efficiency in a Complementary Middle School Program: The Educatodos Experience in Honduras • Marshall, J., M. Mejia and C. Aguilar

44. Focus on Middle School. Alternative schools. • Outcome attrition • Origins radio courses in primary grades for adults • Now audiotaped lessons in community centers • Evidence of high rates of attrition • We analyze attrition in 55 learning centers in 2003-5 • Predictors of staying in the program are use of community projects, having a remunerated facilitator and type of learning center. • Based on official curriculum, learning materials and facilitator. • Purpose examine relationship between features of learning centers and attrition.

45. Contracted to investigate root causes of attrition. • Examined 55 centers in 8 clusters. • Centers located in: • Primary schools • Businesses • Private homes • Vocational training centers • Churches • Only 32% of students remained in the program 2 years later. • 20% of the centers closed.

48. Frequency of community participation projects related to low attrition • Attrition is also lower where the facilitator receives more money • Also social status of the center • Facilitators’ pay related to effort. • Why did learning centers close? • No facilitators • No materials • Why did they dropout? Disproportionately interested in higher income. Perhaps dropouts had higher expectations. • Ability to pay is not key… do not focus exclusively on making schooling more affordable to poor people.

49. Studying School Autonomy in Mexico¿Who Benefits from School Based Management? Fernando Reimers and Sergio Cardenas

50. Background for this study • Evaluation of 4 large scale policies in Mexico in 2006 • The study of SQP effects in grade repetition, dropout and promotion (Murnane, Willett and Cardenas) • The study of Who Benefits, differences in learning outcomes and school organization.