2 - Common monitoring and evaluation framework for evaluation of rural development programs

1. 2 - Common monitoring and evaluation framework for evaluation of rural development programs Jela Tvrdonova, 2012 Challenges of the CMEF & Ongoing Evaluation

2. Outline of the presentation • Strategic approach to RD • Common approach to the evaluation - the CMEF • Monitoring and evaluation –explanatory notes • Intervention logics – the role in the evaluation • Indicators • CMEF structure and content Challenges of the CMEF & Ongoing Evaluation

3. Strategic approach to rural development Focus on limited number of objectives Competitiveness – Environment – Quality of life Axes connected with strategic objectives Strong and dynamic agri-food sector Agriculture and forestry with high added value Employment and growth in rural areas Improvement of the governance in rural areas and mobilisation of the endogenous potential

4. Common approach to monitoring and evaluation Based on: • Exact definition of objectives in Regulation, Strategic Guidelines for RD and rural development programs • EU Strategic monitoring a national strategies • Definition of baseline indicators at the program start period • Suitable combination of output, result and impact indicators, • Common evaluation questions for all RD programs (94) • Single framework for all program intervetion

5. Content of the CMEF • CMEF – Common monitoring and evaluation framework – Handbook, Annexes, Guidance notes • Common and additional/program specific indicators • Common and additional/program specific evaluation questions • Intervetion logic of rural development programs and individual measures Challenges of the CMEF & Ongoing Evaluation

6. CMEF – legal base • EC Reg. 1698/2005 and 1974/2006 and its Annex VIII Challenges of the CMEF & Ongoing Evaluation

7. Monitoring and evaluation • Monitoring – measurement of inmediate outcomes and results at measure and axis level • Evaluation - measurement of long-term and rural areas effects/impacts of intervention, within the program development context • Comparissons and lessons learned for next interventions Challenges of the CMEF & Ongoing Evaluation

8. Monitoring • On-going process which monitors the gradual implementation of the program at the level of financial inpiuts, physical outputs and axes results • It subject of annual reporting on the program implementation • Instruments – common and additional indicators of output and results,

9. Strategic monitoring • Since 2010 and every other two years • Focus of the EC on the progress of national strategic plans and its objectives implementation • Assesment of the contribution of national plans to the EU Rural Development Strategy

10. Evaluation On-going evaluation is based on CMEF and country´s own methodology to carry on the evaluation during the program implementation Process based on annual reporting of result/impact indicators including periodical exercises: • Ex – ante: evaluation of relevance of the planned intervention and optimizing of the RDP budget • Mid-term and ex-post: for the assessing the progress related to the specific and overall/program objectives,

11. Monitoring and evaluation in program logic • Good program is basic ground for the high quality of monitoring and evaluation • The evaluation and the monitoring strats with the program intervention logic Challenges of the CMEF & Ongoing Evaluation

12. Intervention logic of RDP Change Needs Development context - all intervening factors Change observed in reality during and/or after intervention Envisioned change before intervention Overall/horizontal objectives of RDP Impact Specific/Axis objectives of the RDP Result Operational/measure objectives of the RDP Output Measures Inputs of the RDP Overall baseline socio-economic and environmental situation of rural areas, at baseline period Challenges of the CMEF & Ongoing Evaluation

13. Hierarchy o objectives and indicators • 5 types of indicators coresponding with the hierarchy of objectives • Hierarchy of objectives: - instrument helping to show how local activities contribute to the overall objectives, • It is composed of: - operational objectives (measure level) - specific objectives (Axes/priority areas level) - overall – cross cutting/strategic/horizontal objectives (at the program level) • Indicators copy objectives in the hieararchy

14. Hierarchy of indicators Hierarchy of objectives Overall/horizontal objectives of RDP Impact ndicator Specific/Axis objectives of the RDP Result indicator Operational/measure objectives of the RDP Output indicato Measures Input indicators Objectives related baseline indicators Context related baseline indicators Indicators in intervention logic of RDP

15. Role of indicators • Indicators • used as tools to assess how far the expected objectives have been achieved by measures or whole programmes • should be specific, measurable, available / achievable in a cost effective way, relevant for the programme, and available in a timely manner (SMART) • Indicators can not always be filled with quantitative statistical data; in some cases, indicators might also include qualitative assessments or logical assumptions Challenges of the CMEF & Ongoing Evaluation

16. The quality of indicators - SMART approach S – simple, specific M – measurable - data exist A – achievable – available – at low cost R – relevant – selected well for given program T - timely oriented

17. Types of indicators • Baseline indicators: they relate to general socio-economic context of the programme area (context-related baseline indicators) and to the state of the economic, social or environmental situation in direct relation with the wider objectives of the programme (objectives-related baseline indicators) • Financial execution (input) indicators: they refer to the budget or other resources allocated to the programmes • Output indicators: measure activities directly realized within programmes Challenges of the CMEF & Ongoing Evaluation

18. Types of indicators • Result indicators: measure the direct and immediate effects of the intervention and provide information on changes that have taken place • Impact indicators: refer to the benefits of the programme both at the level of the intervention but also more generally in the programme area. They are linked to the wider objectives of the programme Challenges of the CMEF & Ongoing Evaluation

19. Monitoring and Evaluation process Impact Result Ouputs Activities Inputs The process

20. Common indicators • Acommon set of baseline, output, result, and impact indicators for the RDPs (Art. 62 Reg. 1974/2006) “shall form the Common Monitoring and Evaluation Framework (CMEF)’’ (Annex VIII lists the common indicators) Challenges of the CMEF & Ongoing Evaluation

21. Additional indicators • Since common indicators may not fully capture all effects of programme activities, it is necessary to define additional indicators within the programmes (see Guidance notes A & K) Challenges of the CMEF & Ongoing Evaluation

22. Why additional indicators? • The CMEF provides the Common Framework to be applied throughout the EU, thus ensuring comparability • The specificities of each single programme cannot be fully reflected by this Common Framework • It is therefore important to complement the Common Framework by additional indicators in order to capture the full range of intended effects of a given programme • As general rule, a thorough analysis of the programme intervention logic can drive the choice of relevant additional indicators • Considering the limited number of common impact indicators and their broad scope, additional indicators are essential to overcome attribution gaps Challenges of the CMEF & Ongoing Evaluation

23. CMEF structure Handbook Annex 1: • Guidance notes - A: Choice and use of indicators, B: On-going evaluation , C: Mid-term evaluation Annex 2: • Guidance notes – D: Hierarchy of objectives, E – Measure Fiches Annex 3: • Guidance notes – F – K, Common indicators Fiches Annex 4 • Guidance J – O Other guidances Challenges of the CMEF & Ongoing Evaluation

24. CMEF guidance on indicators CMEF provides guidance for MAs in setting up indicators at each level of intervention and baseline indicators Annex 3 of the CMEF Handbook provides detailed description of all common indicators (Guidance notes F – K): • F: COMMON INDICATOR LIST (overview of all common indicators) • G. BASELINE INDICATOR FICHES (detailed description of indicators) • H. OUTPUT INDICATOR FICHES • I. RESULT INDICATOR FICHES • J. IMPACT INDICATOR FICHES Challenges of the CMEF & Ongoing Evaluation

25. Indicator Fiches Each indicator fiche contains the following elements: • Type of indicator • Related measures • Measure Codes • Definition of the indicator • Subdivision • Unit of measurement • Level of collection • Responsible actor for collection • Collection method/good practice • Sources • Registration frequency Challenges of the CMEF & Ongoing Evaluation

26. Measure Fiches Guidance on practical use of indicators in monitoring and evaluation of RDPs • Guidance note E provides the following on use of indicators within each particular measure: • Measure Code • Rationale of the measure • Target group • Target area • Common indicators • Link rationale of the measure and indicators • Evaluation questions Challenges of the CMEF & Ongoing Evaluation

27. RDP objectives Evaluation Question Indicators Baselines and their quantification, Target levels Overall strategic objectives Evaluation questions – cross-cutting Impact indicators Horizontal baselines, objective related, contextual Targets for impact indicators Axis specific objectives Evaluation questions Axis/sector specific Result indicators Axis based baselines, objective related, contextual Targets for results indicatiors Measure/operational objectives Common and program spec. Eval. questions Output indicators Baseline values of outputindicators Targets for output indicators Setting up the evaluation framework at the stage of Ex-ante and use of indicators Challenges of the CMEF & Ongoing Evaluation

28. Thank you for your attention! The Helpdesk of the European Evaluation Network for Rural Development jela@ruralevaluation.eu jelatvrdonova@gmail.com 260, Chaussée Saint – Pierre - Etterbeek - 1040 Brussels – Belgium Tel: +32 2 736 49 60 Email: info@ruralevaluation.eu http//enrd.ec.europa.eu/evaluation/