Impact evaluation: External and internal stakes

1. Impact evaluation: External and internal stakes Impact evaluationseminar - 2 to 6 December, Phnom Penh

2. International context and stakes Uncertainty on aid, Paris Declaration 2005, achievement of MDG (Millennium Development Goals) and after-MDG • Highlights the Results-Oriented Management (ROM) • Focuses on results/effects/impact • Insists on participative approach Since the 90s, increasing initiatives to improving quality and impact of humanitarian action • Sphere, HAP Standards in Accountability and Quality, Code of behaviour, Don not harm, etc. • Increasing concern for understanding changes induced by humanitarian aid Impact evaluationseminar - 2 to 6 December, Phnom Penh

3. International context and stakes • Stake of accountability: Stakeholders have a moral obligation to allocate necessary means to the purpose of their mission and to prove the changes made by their action; • Stake of visibility: Need for demonstrating one’s added-value among an increasing number of stakeholders in the humanitarian sector and international solidarity; • Stake of financial funding: donors prescriptions to prove impact and/or changes along with the development of new tools and approaches (Value for money, Theory of change, etc. ) Impact evaluationseminar - 2 to 6 December, Phnom Penh

4. Crossing the red line.... Results Chain Inputs Outputs Outcome Impact How we keep busy How we make a difference

5. Définitions Outputs/Extrants/Produits: • The immediate results of the organisation’s activities – the processes, goods and services that it produces. The organisation controls its outputs. → 40 health professionals are trained on inclusive approach, reception and follow-up of people with disabilities Outcomes/Effets: • The likely or achieved short-term and medium-term effects of an intervention’s outputs. They contribute to the improvement of people’s life. The organisation only influences outcomes, besides external factors. • → Improved access to health services for persons with disabilities • → Improved health conditions of persons with disabilities Impact: • Long-term, sustainable changes in the conditions of people and the state of the environment that structurally reduce poverty, improve human well-being. Your organisation contributespartially and indirectlyto these enduring results in society or the environment. • → Improved quality of life of persons with disabilities Impact evaluationseminar - 2 to 6 December, Phnom Penh

7. A short story of impact 1950/60s 1970/80s 1980/90s 2000 → Evolution towards a multidimensional comprehension of impact, development of ex-post evaluations Elaboration of new approaches; theory of change, outcome mapping Impact analysis ex ante to anticipate the economic effects of interventions Taking account of unexpected and negative effects, one-dimensional models (Cost-profit analysis, social impact measurement, environmental impact measurement) Rural assessment participative approach, populations as actors of change Linear approach of change, target populations as objects of studies and not actors of change

8. Two main approaches • Usualmethods of impact evaluation: different types of studies and evaluation, sometimesrequiring important resources; • Approachesmeasuringeffects by beingintegratedwithin the intervention with the use of tools monitoring changes atdifferentlevels (short and mid-termeffects, sometimes impact) • E.g.: The planning, Monitoring and Evaluation tool of ACTIONAID, highlightslearning and ownership (ALPS for Accountability, learning and planning system) Impact evaluationseminar - 2 to 6 December, Phnom Penh

9. Why measuring impact? CREDIBILITY ADVOCACY IMPACT LEARNING ACCOUNTABILITY

10. Key questions on impact measurement • What has changed? • For whom? • How significant was it? • Will it last? • In what ways did we contribute to these changes? Impact evaluationseminar - 2 to 6 December, Phnom Penh

11. Responsibility of impact Impact evaluationseminar - 2 to 6 December, Phnom Penh

12. One or several definitions of impact? «  The systematic analysis of lasting or significant change –positive or negative, intended or not – in people’s lives brought about by an action or a series of actions  » (Roche, OXFAM) « Measuring the changes taking place as the result of an emergency project (The Good Enough Guide Impact Measurement and Accountability in Emergencies) Impact evaluationseminar - 2 to 6 December, Phnom Penh

13. Towards an open definition? Common elements of definition: • concept of change/effects, • changes are positive or negative, • changes are expected/desired or not Elements of definition adapted to each intervention type and environment: • the object of change, • the sustainable, significant or structuring aspects of change Impact evaluationseminar - 2 to 6 December, Phnom Penh

14. Impact measurement challenges • When initiating an impact evaluation? • Which approach? • What level of investigation? • What resources? • What participation of the different stakeholders? • What responsibility to changes? Impact evaluationseminar - 2 to 6 December, Phnom Penh

15. 2011-2015 Federal Strategy Chapter 2 - An approach centred on achieving a measurable impact “Handicap International reaffirms the importance of taking effective action, generating positive and measurable effects and introducing changes that bring about tangible improvements for the beneficiaries, and for people with disabilities in particular. For Handicap International, the development of services and facilitating access to these services are essential to achieving a positive impact on the living conditions of beneficiaries».

16. HI and partners’stakesin Impact measurement • Sharing common references and terminology • Proposing and implementing rigorous methods and tools adapted to projects and intervention contexts • Asking the good questions to each effect or impact measurement initiative: What, when, how, why? • Integrating effect or impact measurement within or Monitoring-Evaluation tools and systems • Supporting a balance between the quantitative and qualitative analysis • Analysing and taking into account negative effects of projects Séminaire Evaluation d'impact du 2 au 6 décembre, Phnom Penh

17. Seminar objectives • Identify and analyse HI and partners’ practices on effect and impact measurement, share experiences, draw recommendations for improving tools and methods; • Identify, by technical domain and by intervention context, the areas of change expected, the different levels of change and the related indicators; • Evaluate the possibility of integrating and implementing new approaches of change or impact measurement; criteria of choice and identification of resources; • Identify the ways of integrating change follow-up and measurement into Monitoring-Evaluation tools Séminaire Evaluation d'impact du 2 au 6 décembre, Phnom Penh

18. LET’S GO TO WORK! Impact evaluationseminar - 2 to 6 December, Phnom Penh