Monitoring and Evaluation Project Cycle June 7 th , 2012, Basics of M&E

1. Monitoring and EvaluationProject CycleJune 7th, 2012, Basics of M&E Emergency & Rehabilitation Coordination Unit

2. Why learn the Project Cycle? • An organization is comprised of its projects • Understanding how the project cycle works is the first step to understanding how monitoring and evaluation fits into the day-to-day operations of the organization

3. Why learn the Project Cycle? • An organization is comprised of its projects • Understanding how the project cycle works is the first step to understanding how monitoring and evaluation fits into the day-to-day operations of the organization Project Cycle

4. Project Cycle

5. Need Identified: • A need can be identified in many ways: • Through previous projects • Through assessments • Through calls from Donors • Through calls for assistance from Government • Etc... Project Cycle

6. Proposal Development: • Numerous individuals are involved in designing a project. • Operations, Field Staff, M&E Staff, Reporting, etc. • The design of a proposal varies depending on the donor, but there are some common elements: • Background Information • Beneficiary Criteria • Implementation Plan • Risks • Monitoring and Evaluation • Log Frame • Budget Project Cycle

7. Proposal Development Continued: • Background Informationshould be linked to the identified need and should also explain why an organization is qualified to implement the project. • Beneficiary Criteria defines the type of beneficiaries the project will target (specifics vary, but ‘vulnerable’ is a general criteria) • Implementation Plan outlines the activities that will take place and where • Risks allows the organization to identify possible obstacles that may arise Project Cycle

8. Proposal Development Continued: • Monitoring and Evaluation in terms of the proposal, is how an organization provides accountability to the donor. • The Log Frame outlines what the project seeks to achieve and how those achievements will be measured • Budget Project Cycle

9. Donor Approval: • Once the Donor has approved and signed the proposal it becomes a binding contract. • The proposal is a legal document that states, on the one side, the donor will provide money to the organization, and on the other, that the organization will achieve certain results using the money in the manner outlined in the budget. Project Cycle

10. Recruitment & Procurement: • After the funds are released (and for some organizations, before) recruitment of staff and procurement of inputs starts. • This stage is primarily the responsibility of logistics and HR with assistance from Operations Project Cycle

11. Implementation: • General activities. This varies greatly program to program. In the case of FAO, implementation is largely covered by implementing partners which requires Letters of Agreement (LOAs) –which are also legally binding documents outlining how the Service Provider (SP) will use the funds/inputs provided. • In the case of FAO, though the addition of SPs is slightly more complicated, activities are usually pretty straight forward. Project Cycle

12. Mid-Term and Final Evaluations • Donors generally require at least one interim report, and sometimes more. They also require a final evaluation report. • These reports should cover how the project is fairing against the indicators outlined in the log frame. They must also note any actions that deviate from the original proposal. Project Cycle

13. Monitoring and EvaluationIntro to M&EJune 7th, 2012, Basics of M&E Emergency & Rehabilitation Coordination Unit

14. There are two primary reasons for the monitoring and evaluation of projects • 1) To ensure the project is progressing in the best way possible. • 2) To measure whether the project is reaching (and has reached) the goals outlined in the proposal. Why is M&E important?

15. One of the most common misconceptions about M&E is that it is designed to show the ‘bad’ in a project. • Many organizations do not report accurate data for fear that it will reflect badly on them. Common Misconception

16. The goal of M&E is to determine how to improve a project, rather than to police it. • All project designs, to some extent, are based on assumptions and though these may be rooted in experience, they can sometimes be mistaken. • It is important to identify where changes need to be made • Though ‘problems’ may arise, they do not necessarily reflect mistakes on the part of an organization or its staff Common Misconception Cont’d

17. By providing accountability and transparency organizations show themselves to be honest • Donors want to see that organizations are implementing the projects they proposed. Identifying and rectifying issues that arise in the field shows that an organization puts the implementation of the project (and thus the beneficiaries) first. Why M&E Benefits Organizations

18. UNDP states that “Monitoring and Evaluation should be Results Oriented and provide assessments of the Relevance, Progress and Success” of a project

19. How does M&E fit into the Project Cycle? As a technical team, M&E can assist in designing and implementing Needs Assessments Data collected for evaluation is used to describe the effects and outputs on an intervention M&E should always have the final say on Indicators Continued Monitoring M&E is responsible for providing the data for the Mid-Term Evaluation Monitoring

20. There are two main ways to collect data during a project: • 1) Basic implementation facts • These are generally recorded by all organizations and include “x number of inputs distributed to x number of beneficiaries” • 2) Perceptions of beneficiaries and stakeholders • Often both qualitative and quantitative data is collected to determine if the perceptions of the project are those predicted The important component for monitoring and evaluation is number 2 How do we Monitor/Evaluate Projects?

21. Qualitative • Deals with descriptions and observations • Cannot be measured • e.g. “what is your favorite food?” Quantitative • Deals with numbers • Can be measured • e.g. “how many times a day do you eat?” Sample Size • A portion that is used to represent a whole • In very simple terms, a piece of cake can determine what the whole cake should taste like • Based on scientific methodology Standard M&E terms

22. There are 3 standard methodologies for collecting data from project stakeholders. • 1) Key Informant Interviews • Held with a single person and often ‘open ended’ • Usually qualitative in nature • 2) Focus Group Discussions • Held with a group, normally between 6 and 12 people • Also usually qualitative • 3) Surveys • Cover a wide range of people • Usually more quantitative, though they often seek to quantify qualitative data through the use of multiple choice questions • Often based on a sample size. How do we measure perceptions?

23. Since emergency projects take place in a short amount of time and often cover a large population, qualitative data is not usually possible to collect, and takes too long to analyze Surveys are carefully designed based on project goals so that they can be analyzed and eventually determine how a project progressed using qualitative data in a way that can be quantified • Due to the large number of beneficiaries a sample size of the beneficiary population is usually used. Most Common Methodology for Humanitarian Projects

24. Database vs. Analysis • There is a significant difference in the needs of a database system and an analysis system • Database: • The primary concern for a database is how to store a large amount of data in a small space • Analysis • Analyzing information requires that the data itself be • 1) Clean = Spelling and formatting are uniform throughout the database and ‘other’ sections have been checked and reworded so they can be compared • 2) Simple to read • 3) In cases where the data is to be merged/compared with other datasets, the field headings and formatting needs to be exactly the same. Data Entry

25. Reporting Results • M&E Reports are quite different to general reports. The goal of an M&E report is to state results in a factual manner. • Though there may be some need to explain some results, conjecture should be avoided wherever possible • M&E uses information that can be verified though a technical process Reporting

26. Monitoring and EvaluationBasic Techniques for M&EJune 7th, 2012, Basics of M&E Emergency & Rehabilitation Coordination Unit

27. There are 4 standard methodologies for collecting data: • 1) Desk Review • Review existing information • 2) Key Informant Interviews • Usually held with a single person and often ‘open ended’ • Usually qualitative in nature • 3) Focus Group Discussions • Held with a group, normally between 6 and 12 people • Also usually qualitative • 4) Surveys • Cover a wide range of people • Usually more quantitative, though they often seek to quantify qualitative data through the use of multiple choice questions • Often based on a sample size. How do we measure perceptions?

28. A desk review can be an incredibly useful source of information • The first step of a desk review is to determine what data may already exist on the population group being targeted in the study (e.g. beneficiary population) • This data can serve as a baseline or may be useful for helping to design questions and determine what type of data collection tool to use Desk Review

29. Usually a One-on-One Open Ended Interview • Qualitative. Allows the interviewer to ask additional questions as the conversation progresses • Should try to keep it at one-on-one to help the person being interviewed feel comfortable (and thus more likely to share) • Should try to respect the position of the person being interviewed • E.g. if the person being interviewed is high level government, then it should be a very high level staff member conducting the interview Key Informant Interviews

30. How to record information • As there is likely only a general list of questions to ask, and the primary benefit is the ability to ask questions in an ad hoc manner, it is important to ensure there are detailed notes from such an interview. • Ideally the interview should be recorded but you must have the permission of the person being interviewed in order to record them. • Alternately take limited notes during the interview (if you write the whole time you may not hear important information, and it is awkward for the person speaking.) • Be sure to write detailed notes up right after the interview is complete Key Informant Interviews

31. Different Meanings of FGDs • In academic terms, a focus group discussion (FGD) is qualitative and involves a group of people discussing a given topic. (As opposed to answering a multiple choice question which does not involve the same discussion). However, the terminology is now widely used to describe group surveys, some of which may have pre-set answers. For the purpose of this discussion, FGD’s refers to the academic definition Focus Group Discussions

32. Why Conduct an FGD? • FGDs are used, primarily when you require general opinions, and are useful for helping confirm or reject assumptions. • E.g. To understand the different roles and responsibilities of men vs. women and how they’ve changed over a long period amongst a group of IDPs, you could hold FGDs with Men and Women. Focus Group Discussions

33. How to Conduct an FGD • FGDs should have a facilitator and a note taker. • Ideally they should involve activities that illicit responses. • The most important thing to remember with FGDs is that the participants should be doing the talking. It is very important to try not to LEAD their answers Focus Group Discussions

34. Difficulties Encountered Conducting FGDs Focus Group Discussions

35. First Questions: • What information do we need to know? • What ways can we collect this information? • Do we need a baseline to compare? • What type of data collection tool should we use? (Open ended qualitative or closed quantitative) Designing the Questions • In quantitative surveys, it is just as important to design the answers as the questions • Questions should be kept simple and in a format that is easy to use in the field and the language and terms used should be local where possible • For surveys that are often reused, they should be constantly examined and improved How is a Survey Designed?

36. Remember: • Always keep the reason for asking the questions in mind. • Keep it Simple • Try to keep is Short Survey

37. Monitoring and EvaluationAssessmentsJune 7th, 2012, Basics of M&E Emergency & Rehabilitation Coordination Unit

38. Important Component of Targeted Interventions • Donors want to know why an intervention is needed. • Detailed assessments help to illustrate why. • Needs assessments take many different forms, but can normally fall into ‘Rapid’ or ‘Detailed’ (though there are many in between) Assessments

39. Emergencies In an Emergency it is important to collect information as quickly as possible. • 72 Hours • 1 to 2 Weeks • 1 Month Rapid Assessments

40. A coordinated assessment is always complicated. • Important to ensure everyone agrees before moving on to the next step Coordinated Assessments

41. Steps: • Determine what the goals of the assessment are (what the final report should be focused on) • Create a list of general questions the assessment seeks to answer • Determine the tools to be used and assign the general questions to respective tools • Design the tools • Collect the Data • Clean and Analyze the Data • Produce the Report Coordinated Assessments

42. These should follow a similar process to the Coordinated Assessments General Assessments

43. Important Note: • Needs assessments determine where a project will take place and what it should focus on They should not be the sole responsibility of M&E or they may compromise the independence of the M&E department Assessments

44. Monitoring and EvaluationIndicators and Log FramesJune 7th, 2012, Basics of M&E Emergency & Rehabilitation Coordination Unit

45. Most important part of the proposal for M&E • Determines what the organization is contractually (legally) obligated to report on. • Within the log frame there are overall goals and activity specific goals • Overall Goal – What the project seeks to achieve as a whole • Specific Goals – What each activity will achieve Log Frames

46. The indicators listed under each activity must be reported on • They need to be MEASURABLE • They need to be SIMPLE • They need to be easily UNDERSTOOD and not open to misinterpretation Indicators

47. Look at the Sample Log Frame provided. Please take 10 minutes to go through it and identify indicators that you think are ‘Good’ or ‘Bad’ Sample Indicators

48. Using the same log frame, please work together to identify alternative indicators for each activity. Be sure to think about how they will be measured. Improved Indicators

49. Monitoring and EvaluationM&E PlanJune 7th, 2012, Basics of M&E Emergency & Rehabilitation Coordination Unit

50. Designing an M&E Plan • Examine the Indicators • Determine what information you must collect • Identify any additional information you should collect • Determine the tools you will need to use • Design the tools • Execute the Plan M&E Plan