Foundations in Monitoring and Evaluation

1. Foundations in Monitoring and Evaluation Niamh Barry Niamh.Barry85@gmail.com February 2010

2. Session outline • What is M&E? • Why do M&E? • What happens without M&E? • Tools and Approaches to M&E • Logical frameworks • Performance indicators • Practical work • Data collection and MOV’s • Reporting and reviewing • Documentation

3. What is M&E? • Monitoring – routine & regular collection, analysis & use of information to track progress towards goals • Evaluation: assessment of the extent to which a project is achieving or has achieved its stated goals Or • Monitoring: Improvement and development of the community. • Evaluation: A process design to show the relationship What's the difference? • Timing • Analysis What do we M&E? • Inputs • Process • Progress towards goals • Impact of actions against stated goals

4. What M&E is can not do • M&E ‘s fundamental tenet is to inform judgments about project performance • Measure things that are immeasurable! i.e. if all your activities do not have a goal then how can progress be measured??? • The key is in program design if your program design is weak and inconsistent then you will not have a quality M&E framework • First question should look at your program ‘is this the best way to achieve our goals’ then you can answer the question of how to best M&E the program • M&E should always be built in – it must work with the program design

5. Why do M&E? • M&E of development activities provides organizations with a better means for learning from past experience, improving service delivery, planning and allocating resources, and demonstrating results as part of accountability to donors & partners. • Within the development community there is a strong focus on results— this helps explain the growing interest in M&E.

6. Why do M&E, when we should focus on implementation? “If your not keeping score, your only practicing”

7. What happens without M&E? • If you do not monitor progress, you do not know if you can or are succeeding • If you do not monitor progress, you can never recognize program failures OR success • If you do not evaluate Impact, you do not know if you havesucceed or failed • You prevent learning and sharing • If you never M&E you may repeat programs that have no impact and are wasting organizational and target participants time, resources and donor funds.

8. Tools and Approaches to M&E • Various tools and methods exist to begin to help us M&E, the choice depends what is being being M&E (ed) Progress/impact indicators The logical framework approach Formal surveys – Baselines Needs assessments Participatory methods (FGDs) Cost-benefit and cost-effectiveness analysis Impact evaluation • Some of these tools & approaches are complementary; some are substitutes. Some have broad applicability, while others are quite narrow in their uses.

9. Questions?

10. What do we use? Performance indicators and Logical frameworks

11. A combination of logical framework and performance indicators is the most commonly used in long term programming and planning

12. Logical frameworks

13. Causal Pathway (same but with out indicators)

14. Causal Pathways • This is the initial step in creating the logical framework • It is a program planning tool that ensure that program activities will cause the desired impact • This is a very important step – all inputs must be enough to produce activities and these activities must eventually produce the goal • It’s logic is based on cause and effect or the ‘theory of change’ but you must be careful to think of challenges during this exercise i.e. you can not assume condom distribution will lead to condom use.

15. Simple Logical Framework

16. Terminology • Input-resources needed in the project • Activities- process of using inputs to achieve results • Output –the results of the completed project activities • Outcome/objectives-takes place when the project targets uses outputs as anticipated • Impact/goal-long term change/effectiveness of the project • Many other terms are used-but effectively mean the same thing …goals-objectives-results-effects-purpose…the point is to know when to use them in the process Outside project control

18. Important Notes on Log Frames • You must know what you want to set out to achieve before doing the log frame, is your programme: • Causal: strategies aimed at directly changing a situation • Persuasive: strategies aimed at influencing a situation or thinking about a situation • Supportive: strategies aimed at influencing the environment with which a focal problem is situated • If you do not know exactly what you want to do then you can’t realistically expect it to be M&E(ed)

19. Important Notes on Log Frames Remember that change happens of a period of time (Theory of Change) That is, how a particular project is anticipated to contribute to social change through time. Often and incorrectly the vertical logic is used to systematically disaggregate the problem – without consideration of time

20. Right and Wrong Approaches Goal :a wall Objectives :the bricks that make up the wall Outputs :the sand and cement that make up the bricks Activities :the molecules that makes up the grains of sand and cement Goal :significant and lasting changes anticipated in the lives of the ultimate beneficiaries (e.g. households with improved livelihoods) Objectives: Change to action – peer educators using knowledge and increased capacity Outcomes :changes in knowledge/attitude/practice influenced among intermediary stakeholders (e.g. Peer educators with better capacity) Outputs :deliverables of the project implementing team aimed to initiate the change process (e.g. training of peer educators) Activities: Trainings, follow ups

21. Questions?

22. Performance indicators

23. Performance Indicators • Performance indicators are measures of inputs, processes, outputs, outcomes, and impacts for development projects, programs, or strategies. • Only beneficial when supported with sound data collection—perhaps involving formal surveys—analysis and reporting, • Indicators enable us to track progress, demonstrate results, and take corrective action to improve service delivery. • Participation of key stakeholders in defining indicators is important because they are then more likely to understand and use indicators for management decision-making.

24. Developing indicators • Working with indicators isnotdifficult • Indicators are no more than criteriathat will help us monitor and evaluate our project • To develop them, we have to ask ourselves: What aspect will tell us whether or not something went well? Whether we achieved our objectives or not? Whether we followed our plans?

25. Example of Performance Indicators • A program has the goal to increase community knowledge of HIV prevention and access to HIV testing treatment and care through targeted health talks • What could the performance indicators of this programme be? • Number of community members reached • Improvement in community members knowledge on HIV • Improvement in number of people testing • Improvement in the number of people receiving treatment • Increase in condom use • Reduction in HIV transmission

26. Important Note on Indicators • When we talk about any of the below type of indicators: Increase Decrease Improvement Change Etc • We can only measure these if we have a BASELINE to compare it with • A baseline is information on whatever we wish to impact prior to our program intervention.

27. Important note on Indicators Keep them SMART S – Specific M - Measurable A – Achievable R – Realistic T – Timely

28. Resources to Help on Indicator Development

29. Questions?

30. Devising your own Logical frame work Practical work

31. Case study • Country XX has one of the highest burdens of TB worldwide. XXX has quite a number of challenges that correspond to this high burden of TB: • Limited clinical infrastructure • Poorly trained clinic staff • Low health seeking behavior in the community • Stigma regarding TB • Low knowledge of TB • Low adherence/Lost to follow up for those on medications

32. An TB NGO establishes itself in country XX and wants to decrease the incidence of TB. Of course to do this it most address all of the challenges. • The NGO’s overall Goal is the decrease the incidence of TB mortality and morbidity in country XX • Design a logical framework with activities, Outputs, Objectives and the stated goal. • Start with your goal and then work with activities up! • Draft a number of indicators for each of these steps.

33. Key Questionsto Consider • How will you track or measure the success of this project? • What key indicators will allow to measure results of the work? • What are some sources and/or means of data collection for these indicators? • What challenges do you anticipate? What assumptions do you have?

34. Data collection and MOV’s

35. Means of Verification (MOV) • For every indicator you create you must be able to verify it. • That is you must be able to collect data to prove that it is true. Example Indicator: Decrease in the number of TB patients lost to follow up MOV: National TB registers Indicator: Increase in TB/HIV knowledge among clinic staff MOV: TB/HIV Knowledge questionnaire • Essentially these are the data collection tools you will use to monitor progress and evaluate impact .

36. Data collection Tools • These are integral to any M&E Framework with them you can not monitor progress or evaluate impact. • You must ensure all data collection tools capture the indicators you have established • When designing data collection tools – Less is more • Do not ask for unnecessary information • Do ensure the tool is user friendly • Do pilot test the data collection tool (if possible) before you finalise it • Do design the tools as you are also designing the indicators – this will tell you if your indicator is SMART

37. Important Notes • Does your data collection tool also need to collect baseline information? • Does each indicator require a new data collection tool? Are there MOVs such as National Registers, KAP surveys already available with information on your target population. • Remember: data collection tools are anything that collects information on your indicators Attendance lists Training evaluations National registers Questionnaires Monthly reports Clinic forms Site assessments Etc EtcEtc

38. QUESTIONS?

39. Reporting and reviewing

40. Reporting • One of the key reasons for implementing an M&E framework is because of reporting requirements • Most funding agencies require this annually • Additionally the organization should have its own reporting structure, not only to managers but to the beneficiaries and all stakeholders

41. Reporting ctd • Monthly reports on progress are recommended to ensure that the programme is progressing without challenges and to review progress made against targets • Monthly reports are more desirable as they will detect any challenges early • Additionally data collected should be entered into a program data base

42. Reviewing • All programs need an internal review and replanning process at least yearly. • Recommended that this take place as a prelude and guide to implementing new work plans • Reviewing should focus on program’s overall achievements, challenges and emerging issues, based on information obtained through the M&E processes described in the preceding slides. • Through this process, work plans are reviewed • The process should include Project partners and beneficiaries

43. Documentation

44. Documentation • Information provided by a strong M&E framework is invaluable • Organizations should always documenting: good practices, success stories and case studies. • Case studies will facilitate the sharing of ideas for action. • They will provide information on what people are doing and what's working, what communities have learned from their experience and how it made a difference. They will provide inspiration and show what's possible. • Disseminating the M&E findings including the documentation should form a critical part of the overall M&E activities

45. Concluding Messages • Keep it simple • Build it in • Meaningful as well as measurable • Develop evaluation capacity • Learn and do it yourself!

46. Thank you!