Monitoring and Evaluation

1. Monitoring and Evaluation Dr. Raghu Bista Anil Kumar Gupta

2. Outline • Basics of M&E framework • Result orientation in practice: Monitoring tools, techniques and issues in selected projects

3. Implementation of Project/Programs • Once preparation is completed, finance is assured and design is finalized, project/program is forwarded for implementation. • During implementation phase, Project/program is put into action. Roads will be contracted and built, project activities will be started and completed, project output will be produced. So this is the investment phase of the project. Here, inputs are converted into outputs

4. Outputs of Project/Programs Implementation • The output of the implementation phase is the new road, new hydropower capacity, new bridge, new hospital department, new irrigation system, trained manpower, enhanced capacity, new product etc.

5. Project Implementation • To create expected outputs, projects are divided into work packages and activities, activity time and activity cost are estimated, time schedules are prepared. Likewise budget, resources and procurement plan are efficiently utilized

6. Project Control • During project/program implementation, control is management function to corrective action to address gap between and actual and planned performance. Clearly, the need to exert proper control mandates the need to monitor the proper activities and elements of the project

7. The Project Control Cycle 1. Setting a Goal 4. Taking Corrective Actions 2. Measuring Progress 3. Comparing Actual With Planned

8. Monitoring • “Monitoring is the collection, recording, and reporting of project information that is of importance to the project manager and other relevant stakeholders” (Mantel, Meridith, Shafer, Sutton & Gopalan, 2005). • Monitoring is a continuous function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds. (World Bank, 2011)

9. Purpose of Monitoring • To indicate at the earliest instance any shortcomings with regard to achieving intended objectives so that measures can be undertaken in good time • To monitor the development of the project as a whole, and its component projects, in relation to changes in the context and circumstances of their implementation. • To implement a rapid problem identification system as well as a system for internal communications to the various stakeholders.

10. Purpose of Monitoring Cont.. • To provide forecasts on cost, • To forecast resources and schedule • To facilitate evaluation process during and after activities, through the definition of specific indicators.

11. Process of Monitoring • Identification of Indicators • Collection of information • Analysis of information • Use of information for Action

12. Work Schedule and Monitoring

13. Project S-Curve as Tool for Cost Monitoring • Project S curve consists of presenting costs against time for both budgeted and actual expenditure.

14. 60 40 Cumulative Budgeted Cost 20 Cumulative Actual Cost 10 15 20 25 30 35 40 45 50 5 Elapsed Time (in weeks) The Project S-Curve Cumulative Cost ($ in thousands) $20,000 Negative Var

15. Evaluation • Evaluation can be defined as a process by which general judgments about quality, goal attainment, project/program effectiveness, impact and costs can be determined. It can therefore be conceptualized as a review of the whole project/program in order to assess its overall value and effects. • Evaluation is a systematic and purposeful undertaking carried out by internal or external evaluators to appraise relevance, effectiveness, efficiency, impacts as well as sustainability generated by the policies, plans, programmes, and projects under/after implementation. (National Planning Commission, 2013)

16. Evaluation at Different Level • Evaluation is also classified by subject such as project level, program level, sector Level, policy level.

17. Evaluation Type • Ex-ante Evaluation • Mid Term Evaluation • Completion Evaluation • Ex-post Evaluation (Impact Evaluation)

18. M & E During Programme/Project Period (National Planning Commission , 2013)

19. Type of Evaluation Start of Implementation Completion of Implementation 2. Implementation of projects 4. 1. 3.

20. Ideal situation-M& E Complementary Monitoring Evaluation

21. Difference Between M & E • Evaluation is assessing the entire project cycle. • Evaluation looks for lessons to be learned from both success and lack of success, and also looks for best practices which can be applied elsewhere. • It is concerned with the assessment of progress against predetermined goals, outcomes and their effects and impacts on stakeholders. • It is generally conducted by an independent outsider • Monitoring is a continuing function that takes place throughout the implementation of a project/programme • It focuses on the implementation of the project comparing what is delivered with what is planned • It is directly related with inputs, processes and outputs. • It is generally conducted by the implementing agency.

22. Result based M& E Video

23. Result Based Management • Result based management is perspective that focus on achievement of development results. • It is a management strategy that focuses on the processes, products and services contributing to the achievement of development results (NPC, 2013)

24. How the Results Come • Once a project is designed and implemented, it will produce outputs (For example 25 trained math teachers) • The trained math teachers will use improved methods to teach, the result will be increased math solving competency of students • In exam, number of students to pass exam will be increased with better marks as ultimate results and will be able to get scholarship

25. Results Chain Impacts Activities Outputs Outcomes Inputs

27. The Power of Measuring Result • If you do not measure results, you cannot tell success from failure • If you cannot see success, you cannot reward it • If you cannot reward success, you are probably rewarding failure • If you cannot see success, you cannot learn from it • If you cannot recognize failure, you cannot correct it • If you can demonstrate results, you can win public support (Osborne & Gaebler 1992)

28. Result Base M & E • Results-based monitoring is a process of incessantly collecting and analyzing information on the extent to which results from the implementation of a given plan, policy, programme or project have been achieved so as to compare it with the expectations. • Results-based Evaluation is an undertaking that appraises relevance, efficiency, effectiveness, and sustainability of a given plan, policy, programme or project by making comparison of the outcomes of implementation with the intended results. (NPC, 2013)

29. Essence of Result Based M & E • Traditional M & E is more focused on completion of assignment/program/policy • Result Based M & E gives emphasis on achievement of ultimate results from the intervention • So it focus on continuous monitoring and evaluation of impacts during project implementation and after project implementation to assure results

30. Typologies of Result Based Monitoring • Continuous • Sustainability

31. Evaluation Criteria • Relevancy • Effectiveness • Efficiencies • Impact • Sustainability

32. Impact Evaluation • The objective of this type of evaluation is to determine whether or not a given project is achieving the intended impacts • Impact project performance with training of project staff • Impacts on income of people by apple farming project • Impacts of irrigation on increment of income and reducing poverty

33. Impacts Assessment: Impacts vs. Counterfactual

34. Selection of Indicators of Project Outcomes • Indicators are “the variables used to measure progress towards goals” (Stem , Margoluis, Salafsky, & Brown, 2005) • Indicators are required to be selected for Monitoring and Evaluating Results

35. Indicators • An indicators indicates progress, success, and failure of a policy, program and project • Indicators are related: • How do I know the intended positive change has taken place? • What changes? In what way? (quantity, quality, time description of targets)

36. Types of Indicators I • Input indicators • Process indicators • Output/outcome indicators • Impact indicators • Time related indicator • Quantity related indicator • Cost indicators • Quality indicator

37. Steps to be Taken to Select Indicators • Precisely identify what we want to monitor or examine • List down the questions that need answers • Identify the kind of indicators that can be used • Review the methodological basis and robustness of these indicators • Find out what data will be needed to derive each indicator, and where and how to get this data

38. SMART Indicator • Specific • Measurable • Achievable • Relevant • Time-bound Examples of Indicators • Drop out ratio • Enrolment percentage

39. Targets and Baseline • Target is the intended results • Baseline is the existing scenario or reference point against which performance is compared • Target is the intended results • Baseline is the existing scenario

40. M & E Plan • The plan is managerial tool that: that specifies indicators, targets, sources of data, schedules, resources and responsibilities for M & E activities

41. Exercise

42. Methods of Evaluation • Quantitative • Qualitative • Mixed

43. Quantitative Tools • Questionnaire and survey • Existing database

44. Qualitative • Direct Observation • Focus Group Discussion • Interview • Case Studies • Document Studies • Analysis of SWOT

45. Timing of Research • When and How often to collect data necessary to provide required information

46. Data Analysis and Reporting • Once data are collected, they are analyzed or processed for reporting. This may include averaging the data, calculating ratio, developing trends etc. Finally, the analysed data are reported in some form.

47. Essential Elements of Evaluation Report • Executive Summary • Main Body • Outline of evaluation background (Objective, Strategy, Time/Duration, Role/ • Introduction, Evaluator/s) • Review of expenditure along with a narrative overview • Evaluation log-frame and descriptive summary • Results of each of the dimensions evaluated : Relevance, Effectiveness, Efficiency, Outcomes and impact and Sustainability • Conclusion • Summary of results, learning and recommendation • Utility of results for management • Justification of evaluation plan • Changes in implementation modality at different points in time

48. Essential Elements of Evaluation Report Contd.. Annexes • Initial outline used for implementation • Evaluation log-frame and matrix demonstrating progress status • Evaluation questions and results analyzed through the measurement of indicators • Unprocessed data obtained from interview, survey, direct communication and other means • Reference