Measuring effects and impact: Analysis of practices in HI & partners projects

1. Measuring effects and impact: Analysis of practices in HI & partners projects Impact and evaluation seminar - 2 to 6 December, Phnom Penh

2. Findings of the meta-analysis • Method and corpus • A meta-evaluation, or meta-analysis of evaluations, is a study and synthesis of several other evaluations based on a set of pre-determined criteria. The aim is to learn lessons about the quality and impact of interventions and obtain the information resources necessary for improving future actions. • 18 external evaluations for 2012 • Based on an analysis chart, the meta-analysis examined 5 OECD criteria (relevance, effectiveness, efficiency, impact and sustainability), and 5 other criteria (gender, partnership, beneficiary participation, coordination and innovation) corresponding to approaches promoted by HI. Impact and evaluation seminar - 2 to 6 December, Phnom Penh

3. Findings of the meta-analysis • How is impact analysed in evaluations? • No specific methodology for measuring impact , but evaluators used participatory tools for measuring effects (e.g.: SWOT); • The evaluations did not allow any real conclusions to be drawn on the long-term impact of the projects, but more on the projects’ direct and immediate effects; • The lack of a baseline/endlinewas one of the main factors limiting the measurement of project impact; • Limitations of the indicators monitoring tools and process; • Some evaluations considered the positive and negative effects of the projects, as well as expected and unexpected effects.(e.g.: gender approach not taken into account). Impact and evaluation seminar - 2 to 6 December, Phnom Penh

4. Findings of the meta-analysis • Outcomes of the analysis of effectsfrom the evaluations: • The evaluation reports highlighted the projects’ effects on improving the lives of beneficiaries and their inclusion; • The projects also had a positive effect on the stakeholders’ “sphere” in terms of inclusion and respect for the rights of vulnerable and marginalised people (recognition of rights by local authorities, change in attitudes within communities, etc.); • Several evaluations mentioned positive effects that went beyond expected effects(e.g. in the Prison project of Madagascar programme); • It is essential for HI to invest resources and introduce processes for measuring the effects of certain activities which are crucial to project strategy, such as the training and capacity-building of the services’ staff • . Impact and evaluation seminar - 2 to 6 December, Phnom Penh

5. Measuring effects or impact: methods and tools used within the projects Policy makers and duty bearers Rights Monitoring Tools, PRDLI research, … Participation measurement tool DPOs (Vida Brasil), quality of life measurement tools (e.g. Asia), social participation survey (e.g. Togo), CAP surveys, Pre and Post Impact Assessment (Mines), Case studies… Tools/user satisfaction surveys, Pre and post-test training, Mapping tools, Mozambique/Kosovo survey… Users and/or beneficiaries, communities Service providers Impact and evaluation seminar - 2 to 6 December, Phnom Penh

6. Methods and tools: stakes A diversity of effects monitoring and measurement tools but a rather irregular implementation → How to share and to better use them? Few impact surveys (HIB Burundi/Colombia survey, Social participation survey Togo 2007-2009, Filariasis impact survey) • → Develop more surveys, with which objectives and which resources • Methodological limitations within effects and impact measurement: • Most often, absence of baseline • When baseline exists, no endline • Irregular participative approach in methods and tools • Balance between quantitative and qualitative tools • Need for reinforcing Monitoring-Evaluation tools • Better designing indicators of effects and impact data collection Impact and evaluation seminar - 2 to 6 December, Phnom Penh