Results-based Monitoring and Evaluation in Bank projects

1. Results-based Monitoring and Evaluation in Bank projects HD Learning Week. November 6, 2006 Jody Zall Kusek & Mohamed Khatouri World Bank

2. Session I- Overview of Results-Based Management Concepts • Managing for results.Use the information to improve decision making and steer development interventions towards clearly defined goals. Results-Based Management Approach Focus on the results obtained rather than just on the inputs used or the activities conducted. Traditional Management Approach Focused mainly on inputs and activities. • Better indicators + Monitoring = Better results Which statement is accurate ? 2. High quality Project design (specific focus on outcomes) + High quality implementation = Better results • Results agenda.Focus on development outcomes to be achieved by Bank operations, which requires high quality project design and supervision.

3. Session I- Overview of Results-Based Management Concepts Critical factors for defining results to be achieved • Socio- economic context: Results statement should reflect local needs and priorities • Local Capacity: Existing skills, leadership, and management capacity will impact on what can be implemented to achieve expected results • Resources: Level of resources will impact on what can realistically be achieved • Timetable: Results framework must identify the results (changes) to be achieved in the life of the program

4. Principles of Management for Development Results* Session I- Overview of Results-Based Management Concepts 1. Focusing the dialogue on results at all phases of the development process 2. Aligning programming, M&E with results 5. Using results information for learning and decision making 4. Managing for, not by, results 3. Keeping measurement & reporting simple & cost-effective Tools for performance measurement to increase the effectiveness of development interventions Results-based M&E *Roundtable on Managing for Results in 2004

5. Session I- Overview of Results-Based Management Concepts Why is Results-Based M&E Important? • Improving internal management • Focusing interventions/reallocating inputs to achieve results. Programs focused are better able to demonstrate results over time. • Better reporting: Programs have effective M&E system develop results-oriented reporting • Marketing successes • Capturing lessons-learned: Showing general program progress & sharing practices and innovations. • Secure political & public support: “If you can demonstrate results, you can win public support“ • Ensuring accountability • Preoccupation with results is a global issue : demonstrate results against money spent • Results are more important than processes: provide evidence that the program is producing longer term benefits.

6. Session I- Overview of Results-Based Management Concepts Indicator 50% 40% 20 % 30% Year 1 2 3 4 5 • Clarifies program objectives Monitoring • Link inputs and activities to results to be achieved • Translates results to be achieved into performance indicators • Periodically collect data on the indicators and compare actual results with target Regular collection and reporting of information to track whether actual results are being achieved as planned • Reports progress and alerts management to problems in implementation

7. Session I- Overview of Results-Based Management Concepts Indicator Withproject 50% 15 % 40% With outproject 30% Year 1 2 3 4 5 • Analyses why intended results were or were not achieved Evaluation • Explores unintended results • Assesses causal contributions of activities to results and validate hypothesis • Examines implementation processes Analytical efforts to answer specific questions about performance of a program activities. Oriented to answering WHY? And HOW? • Provides lessons learned and recommendations for improvement

8. Session I- Overview of Results-Based Management Concepts Attribution Gap Long-term Goal (Impact) Outcomes Activities Outputs Inputs Traditional M&E Results-based M&E Results Building Blocs PLANING FOR RESULTS Effects or behavior changes resulting from program outputs Products and services to be used to simulate the achievement of results Utilization of resources to generate products and services Resources committed to program activities Long-term, widespread improvement in society Results Implementation

9. Session I- Overview of Results-Based Management Concepts Examples of Results Chain Long-Term Goal (Impact) Outcomes Outputs • Teachers trained • Text Books provided Increase literacy rates Increased student completion rates Education Increased use of health clinics Improved maternal mortality • Doctors hired • Health workers trained Health Increased population is food secure Increased % of vulnerable population using saftey net programs • Increased coverage of crop-insurance programs Social Protection and labor

10. Session I- Overview of Results-Based Management Concepts Identify the sequence of inputs, activities, outputs, outcomes, and impacts Example of Results Chain (Health Sector) • Information is available for parents about the importance of breast feeding • Children in Local Community healthier • Fewer children are having diarrhea diseases • Mothers Parents breast feeding rather than using formula • New funds available to implement a health project to reduce babies mortality rates • Implement information campaigns on the importance of breast feeding

11. Session I- Overview of Results-Based Management Concepts Critical Elements of an Effective M&E System • Active involvement of customers and partners (planning, conducting, reviewing, & interpreting performance information) • Fully integrated into the government’s existing management systems at the central and local level • Responsive to information managementand policy makingof government institutions at the central and local level • Built around reporting requirements • M&E capacity in place for collecting, analyzing, and reporting performance information

12. Session I- Overview of Results-Based Management Concepts Operations Projects/Programs Policy Dialogue CAS AAA Design Logic of Bank Operations Country’s National Poverty Reduction Strategy (PRSP) Provides a central framework for donor coordination PRSP Strategic Goals Outcomes Align Projects and Programs Results with CAS & PRSP Results Country Assistance Strategy (CAS) to support the national development strategy Operationalize the CAS

13. Session I- Overview of Results-Based Management Concepts Exercise 1. Develop Results Building Blocs • Assignment • Break out into groups of 4-5 persons • Each group should develop a results building bloc (inputs, outputs, outcomes, & impacts) for at least 2 programs. Spend no more than 15 minutes per program • Select one case to report on in the plenary session.

14. Session II- Development of Results Framework Results Framework • Program’sstrategy for achieving specific objective • Identifies the Program Development Objective (PDO) and Intermediate Outcomes which are critical to achieve it • Illustrates the development hypothesis and the cause & effect relationships linking all levels of a program’s objectives (IFa and b happen THENc should happen) • An effective tool for: • Strategic clarity and planning & managing for results • Building consensus and ownership • Effective communication

15. Session II- Development of Results Framework • Causal Chain Hypothesis.Results are caused by some interventionsIF… THEN • IF we build health centers • THEN local population will use them to for health services • IF we train teachers on new pedagogic skills • THEN teachers will use these skills to improve the quality of education • Design with clear logic Reduction in the child mortality 1. Determine the key development hypothesis & its contribution to higher order impacts THEN incidence of preventable disease will be reduced IF expand child immunizations

16. Session II- Development of Results Framework • Design with clear logic (Cont.) 2. Develop a hierarchy of objectives showing the causal linkages 3. Determine what is possible in the operation (program or project) Reduction in Child Mortality Expanded health care system Improved nutrition Incidence of preventable disease reduced Increased immunizations Increased family health practices • DESIGN. Project success depends on the development of the right activities & assumptions Vaccines availability Public awareness Enhance regulations • IMPLEMENTATION. Each part of the results chain has a role to play in achieving the Project Development Objective Expand facilities Provide Funds

17. Session II- Development of Results Framework Program Development Objective (PDO) The PDO describes the effect that the program’s outputs will have on the beneficiaries in terms of changed behavior or improved performance. It defines the program’s success. • Focuson the outcome that the project can directly influence, given its duration, resources, and approach • Focus on the expected outcome for the primary beneficiaries At the close of the program, what problem has been solved for the key recipient of the program outputs ? • The PDO outcomes are one step beyond the products of the activities. • Keep the PDO statement clear, precise, and stated as realistic results— not overly broad or too ambitious Is the PDO precise ? PDO example: “To achieve improved, expanded, sustainable utilization of basic social and economic services and to support a governance system where local government and communities can gradually become mutually accountable.

18. Session II- Development of Results Framework Program Development Objective (Cont.) • Focus on what is going to be transformed (not what is going to be done) Which outcome example describe what is going to be transformed? Example 1: To provide grant financing to educate “at risk” groups in the risks of HIV/AIDS. Example 2: To increase the awareness of the dangers of HIV/AIDS of “at risk” groups. • PDO Outcomes. Capture single outcome by outcome statement Example: PDO objective To expand access and improve quality of primary education throughout the country. Outcomes to be achieved ? ……………… ………………

19. Session II- Development of Results Framework Performance Indicators • “What gets measured gets done” • “If you don’t measure results, you can’t tell success from failure” Indicator: A variable that provides accurate and reliable evidence about the achievement of a specific result • Work with stakeholders and technical experts to determine which indicators are most important • Avoid selection of too many indicators Less is Better! • Information about indicators should be easy to gather and useful for management decisions • Performance indicators should be identified at all levels of results chain. 7

20. Session II- Development of Results Framework Criteria for Selecting Good Indicators Performance indicators should beSMART: Specific; Measurable; Attributable; Realistic; Targeted • Specific: measure as closely as possible what is intended to measure Which indicator is more specific ? OutcomeIndicator Parents insure that 1.Increased utilization of clinics Children get treatment for 2. Increased use of malaria Malaria drugs • Measurable:clear and unambiguous about what is measured • % of health centers without stock out of drugs x, y & z for more than a week at a time Which indicator is measurable ? 2. % of health centers with availability of drugs 8

21. Session II- Development of Results Framework Criteria for Selecting Good Indicators (Cont.) • Attributable: attributable to the project’s or program’s efforts Which indicator is attributable ? 1. Life expectancy 2. % of children fully immunized at 1 year • Realistic: data obtained at reasonable cost with enoughfrequency 1. HIV prevalence among 15-24 year old pregnant women Which indicator is more realistic? 2. HIV prevalence among the total population • Targeted: it should be specific about the targeted population/area Which indicator is targeted ? 1. Percent increase in employment 2.Percent increase in employment of graduate of technical training center X in the first year after completion of training. 8

22. Session II- Development of Results Framework EXAMPLE Outcomes Indicators PDO Outcomes PDO: Increased use of health facilities • % of under 5 yr covered by DPT immunization • Increased coverage of clinics providing immuniatio • % of health deliveries carried out in public health facilities (%) • Improved quality of doctors/nurses Intermediate Outcome Indicators Intermediate Outcomes by Component Component I- Extension of Quality Health Services • % of facilities without 7-day stock outs of essential drugs (List of essential drugs defined) • Sustained availability of essential drugs Component II. Development of Human resources • % of facilities with minimum staffing norms (List of minimum staffing defined) • Reduced shortage of human resources

23. Session II- Development of Results Framework Each Indicator must have a M&E plan • Definition.Clear and precise definition of what will be measured and unity of measure. • Baseline & Targets • Baseline.Value of the indicator at the beginning of the program. Used as a point of comparison when measuring progress toward a specific result • Target values.The intended value of the indicator at the end of a specified point in time, against which actual results will be measured • Data Acquisition Method.Data source, frequency/schedule, and responsibility for its collection • Data Analysis & Reporting Method.Frequency/schedule, Analysis method, and responsibility of reporting. • Critical Assumptions.External factors that could significantly affect the achievement of the program results targets 7

24. Session II- Development of Results Framework Example of a M&E plan for one indicator 10

25. Session II- Development of Results Framework Impacts Define Missing Middle Outcomes Interventions Ways to Improve the development of Results Frameworks High level outcomes Example: Reduced maternal mortality • Intermediate outcomes which the program is expect to influence directly through its defined interventions during the program period. Intermediate Outcomes Example: Birth attended by skilled staff Processes, Policies, & Outputs Example: Provide essential obstetrical health services

26. Session II- Development of Results Framework Increased use of maternal health services Increased adoption of prevention methods Improved access to family planning Availability of ITNs Hygiene and sanitation practices Increased coverage of Antenatal, new born, emergency obstetric, & post-natal care Improved Communi-cation networks & ambulances Provision of contracep-tives and counseling to women and youth Information on family planning available Health services provided ( Heath workers trained, Health facilities built & rehabilitated, Pharmaceutical drugs purchased) Example: Health Sector Reduced Maternal Mortality

27. Session II- Development of Results Framework Data Collection Methods Rigor/Difficulty (Cost/Time/skills) Census Formal Methods Impact Evaluation (Experimental Design) Panel Surveys Direct Observation Rapid Appraisal Methods One-Time Survey Community Interviews Review of official records Focus Group Interviews Validity Reliability Credibility Key Informant Interviews Statistical Emphasis 1. Issue to be examined • Choice depends 2. Quality of the information needed 3. Time frame in which information is needed 4. Cost

28. Session II- Development of Results Framework Key Steps for planning an Evaluation 1. Clarify the evaluation purpose and audience 2. Identify the evaluation questions 3. Select appropriate methods based on the questions to answer 4. Prepare data collection and analysis plan Example. Which data collection method is more appropriate in providing an answer to each of the following questions? • What % of women using health facilities? • Why didn’t more parents send their kids to school? • Did the CDD project contribute to the increase in income of local communities?

29. Session II- Development of Results Framework Focus on outcomes all phase of the project cycle • PCN • Develop causal chain • Define appropriate PDO and outcomes to be achieved • Completion (ICR) • Review and validate the causal chain • Report on the Project outcomes • Design/PAD • Develop the results framework with SMART indicators • Develop the M&E plan with baselines & targets • Implementation (ISR) • Report on outputs & outcomes • Adjust project design (as necessary to achieve outcomes) 7

30. Session II- Development of Results Framework Exercise 2. Assess the quality of the Project Development Objectives (PDO) & outcome indicators of selected projects • Assignment • Break out into groups of 4-5 person • Review the quality of the PDO statement and its corresponding outcome indicators for 2 projects • Report on one case in the plenary session