Monitoring and Evaluation: Learning from Development Co-operations

1. Monitoring and Evaluation: Learning from Development Co-operations Nicolina Lamhauge OECD Environment Directorate SEA Change CoP Webinar, 26 March 2012

2. 1. Outline • Context • Data and methodology • M&E approaches • Indicators • Baseline, milestones and targets • Conclusions

3. 2. Context • M&E for adaptation in the context of scaled-up climate finance • Already a large body of work on M&E for adaptation • Early work focused on categorisation of adaptation activities and an assessment of factors to be considered when developing M&E frameworks • Recent work has proposed M&E frameworks at the project and programme level • The theoretical frameworks have been translated into practice by some of the climate funds/mechanisms

4. 2. Context (cont.) • Development agencies have a long history in implementing projects and programmes in climate sensitive areas • Many of these include adaptation-like activities • Climate resilient infrastructure • Drought or flood resilient crops • Drawing on the experience of 6 bilateral agencies: • CIDA, DFID, DGIS, JICA, SDC and Sida

5. 2. Context (cont.) • By comparing the approaches used by the 6 agencies, the objective of the paper was to: • To get a better understanding of the particular characteristics of M&E in the context of adaptation • To see if there are any best practices in the choice and use of indicators for adaptation

6. 3. Data and methodology • The data consists of: • Documents for 106 projects and programmes • These include ex ante, interim and ex post evaluations • Most of the projects were directly provided by the agencies – some were also available online • The documents cover different themes and geographical areas

7. 3. Data and methodology (cont.) Regional focus of the sample

9. 3. Data and methodology (cont.)

10. 4. M&E Approaches • Result Based Management and the Logical Framework Approach are the most common M&E approaches across the 6 agencies • Most of the agencies distinguish between activities, outputs and outcomes • Approaches differ by the level of detail included in the evaluation documents • The standard logframe approach • The expanded logframe approach • The simplified approach

11. 4. M&E Approaches (cont.) The standard logframe approach

12. 4. M&E Approaches (cont.) The expanded logframe approach

13. 4. M&E Approaches (cont.) The simplified approach

14. 5. Indicators • Selection of indicators is a core component of M&E • Indicators: • Show how results will be measured • Provide an overview of change over time • Help programme staff prioritise inputs and communicate outcomes • Input, process, output and outcome indicators • Can be categorical, quantitative and qualitative

15. 5.1 Indicators on risk reduction

16. 5.1 Indicators on risk reduction (cont.)

17. 5.1 Indicators on risk reduction (cont.) • Project on adaptive capacity has 4 component: • Household capacity to innovate more climate resilient livelihood strategies • Improved capacity of communities to implement adaptation strategies • Increased capacity of local partner to raise awareness on climate change • Interaction by project partners with local stakeholders • Difficult to define objective indicators for concepts such as “adaptive capacity” • No. of households that seek out, test, adapt and adopt climate resilient livelihood strategies • % of households with new livelihood strategies

18. 5.2 Indicators on policy

19. 5.2 Indicators on policy (cont.) • Policy and administrative management are complex processes • The introduction of a policy will not necessarily result in its implementation • It is therefore important to consider the full set out indicators contributing to the same output or outcome: • No. and types of actors that support climate change adaptation initiatives • Level of integration in policy processes • Level of integration in strategies and programmes • No. of beneficiaries

20. 5.3 Indicators on education/training

21. 5.3 Indicators on education/training (cont.) • Combinations of indicators: • No. of educational material produced & extent of its use • No. of people trained & percentage of trained policy makers who apply the information • Difficult concepts to evaluate include: • Ability • capacity

22. 5.4 Indicators on research • Generally binary indicators • Climate scenarios developed • Climate tools developed to assess climate change vulnerabilities • Complementary indicators ensure that the research tools and scenarios are used in practice • Information from climate scenarios is integrated into national plans • Knowledge platforms become the basis for better information sharing • This provides a measure of long-term impact beyond the initial development of a climate tool

23. 5.5 Indicators on co-ordination

24. 6. Baseline, milestones and targets • Baselines provide a reference point against which results can be measured • This may require the application of climate projections • This requires a certain level of technical expertise • Targets provide a benchmark for evaluating achievements • May also change in the context of climate change • Milestones are useful for monitoring progress • Allow project staff to monitor progress and revise project components if needed

25. 6. Baseline, milestones and targets (cont.) • Indicator: Countries in South Asia co-operating at a regional level to invest in improving water management • Baseline: Major water insecurity with natural scarcity and variability, weak management, increasing demand, climate change, limited co-operation on water across borders, insufficient data sharing or joint investments to manage water variability, floods and droughts having significant impact. • Milestone (2011): 3 significant investment projects in development, with at least one involving co-operation between 2+ countries. • Target (2018): Substantial investment at scale in regional water management underway in 3 major river basins, reducing the impacts of climate change and reducing vulnerability of the 700 million people living in these basins.

26. 7. Conclusion • RBM and the logical framework approach are the most common M&E approaches • The type of activity will determine the choice of indicators • A combination of qualitative, quantitative and binary indicators are needed • If not carefully defined, qualitative indicators often require a value judgement by the evaluator • The use of complementary indicators is particularly important when measuring outcomes and impacts

27. 7. Conclusion(cont.) • Without carefully defined baselines, mid-term and final evaluations based on milestones and targets are difficult to conduct • Baselines are often based on assumptions of a static climate • To evaluate adaptation project it may be necessary to extend the timing of evaluation • Detailed indictors for every component vs broader vulnerability assessments

28. Thank you!Nicolina.Lamhauge@oecd.org