Common monitoring and evaluation framework for evaluation of rural development program me s

1. Common monitoring and evaluation framework for evaluation of rural development programmes Jela Tvrdonova, 2014

2. Content of the presentation • Strategic approach to rural development • Common approach to evaluation: legal background and CMEF • Monitoring and evaluation – what we are talking about? • What is evaluated? • Intervention logic in evaluation • Indicators • Evaluation questions • CMEF guidance

3. Strategicapproach to ruraldevelopment Focus on limited number of objectivesCompetitiveness – Environment – Quality of life Axes connected with strategic objectives Strong and dynamic agri-food sector Agriculture and forestry with high added value Employment and growth in rural areas Improvement of the governance in rural areas and mobilisation of the endogenous potential

4. Commonapproach to monitoring and evaluation • Exact definition of objectives in Strategic Guidelines for Rural Development, Council Regulation 1698/2005 (art. 77-87), EC Regulation 1974/2006 (art. 60-62), Health check, • National strategic plans and EU strategic monitoring • Single framework for all program interventions • CMEF: Common monitoring and evaluation framework – Handbook, Annexes, Guidance notes • Baseline indicators at the program start • Suitable combination of input, output, result and impact indicators, • Common evaluation questions for all RD programs • Guidance for ex ante, mid-term, ex post and ongoing evaluation

5. Monitoring and evaluation – what we are talking about? • Monitoring – measurement of immediate outputs and results at measure and axis level (different from strategic monitoring) • Evaluation - measurement of long-term effects/impacts of intervention, within the program development context (rural areas) Accountability, comparability and lessons learned for next interventions

6. Monitoring • On-going process which monitors the gradual implementation of the program at the level of financial inputs, physical outputs and results • Subject of annual reporting • Instruments – common and programme specific indicators linked to outputs and results,

7. Strategic monitoring • Since 2010 and everyothertwoyears • Focusofthe EC on theprogressofnationalstrategicplans and itsobjectivesimplementation • Assesmentofthecontributionofnationalplans to the EU RuralDevelopmentStrategy

8. Evaluation On-going evaluation is based on CMEF and country´s own methodology to carry on the evaluation during the program implementation Process based on annual reporting on result/impact indicators including periodical exercises: • Ex – ante: evaluation of relevance of the planned intervention and optimizing of the RDP budget, • Mid-term: for the assessing the progress related to the specific and overall/program objectives, • Ex post : fortheassessmentofimpacts and acievementstowardsobjectives. WILL PROGRAMME WORK AS IT SHOULD? IS PROGRAMME WORKING AS IT SHOULD? DID PROGRAMME WORK AS EXPECTED?

9. Monitoring and evaluationofprogramme • Good programme is the basis for the high quality of monitoring and evaluation • Normally the monitoring and evaluation is linked to programme intervention logic • But also specific topics, such as: • Technical assistance • Administrative arrangements • Delivery mechanism

10. What is aninterventionlogic? Intervention which logically responds to the most important needs of the targeted area • Hierarchy of objectives: • overall, • specific, • operational • Hierarchy • of expected effects: • impacts on the territory, • results on the • supported beneficiaries • immediate outputs • at project level Challenges of the CMEF & Ongoing Evaluation

11. Intervention logic of EU rural development programmes Challenges of the CMEF & Ongoing Evaluation

12. EU policy objectives RDP Intervention logic Context , itsdescription SWOT and needs assessment Complementarity Relevance Impacts Overall objectives EU/MS Programme level Coherence Results Specific objectives EU/MS Axislevel Effectiveness Operational objectives EU/MS Measurelevel Outputs Measures, projects and their management and implementation Inputs Efficiency Source: EENRD 2014

13. Role of indicators • tools to measure the programme effects and achievements towards expected objectives by measures or the whole programme • should be Specific, Measurable, Achievable, Relevant for the programme, and Timely (SMART) Indicators should be filled with quantitative statistical data; however in some cases, indicators might also be linked to qualitative assessments or logical assumptions

14. Types of indicators • Baseline indicators: they relate to general socio-economic context of the programme area (context-relatedbaseline indicators) and to the state of the economic, social or environmental situation in direct relation with the wider objectives of the programme (objectives-relatedbaseline indicators) • Financial execution (input) indicators: they refer to the budget or other resources allocated to the programmes • Output indicators: measure activities directly realized within programmes

15. Types of indicators • Result indicators: measure the direct and immediate effects of the interventionwithinthegroupofprogrammebeneficiaries and provide information on changes that have taken place • Impact indicators: refer to the benefits of the programme both at the level of the intervention but also more generally in the programme area. They are linked to the wider objectives of the programme

16. Impact Result Ouputs Activities Inputs Indicators are linked to processofprogrammeimplementation

17. Common indicators • A common set of baseline, output, result, and impact indicators for the RDPs (Art. 62 Reg. 1974/2006) “shall form the Common Monitoring and Evaluation Framework (CMEF)’’ (Annex VIII lists the common indicators)

18. Programe specificindicators • Since common indicators may not fully capture all effects of programme activities, it is necessary to define additional programme specific indicators (see Guidance notes A & K), which relate to programme specific issues. • They are developed if: • Programme contains programme specific objectives and common indicators are not sufficient to capture achievements • There is the need to evaluate specific issues such as delivery mechanism etc.

19. Purpose of Evaluation Questions • Define the focus of evaluations • Demonstrate the progress, impact, achievements, effectiveness, efficiency and relevance of rural development policy Common Monitoring and Evaluation System distinguishes EQs: • Common Evaluation Questions for Rural Development (CEQ-RD) • Programme-Specific Evaluation Questions (PSEQs) Source: EENRD 2014

20. Purpose of Evaluation Questions COMMON Eqs for rural development PROGRAMME SPECIFIC EQs Evaluation of matters relevant at the EU policy level Evaluation of matters relevant to the programme-specific policy Encourage the assessment of programme results and impacts Enhance comparability across RDPs Evaluation of specific RDP related topics Demonstrate the contribution of programme interventions Source: EENRD 2014

21. Clear and straightforward evaluation framework • Triangular consistency between objectives, questions and indicators which allows for: • Less EQ and more targeted towards policy • Clearer formulation of EQ and harmonization of all terms used • More evidence to answer EQ • Better planning the evaluation Policy objectives Evaluation questions and judgment criteria Indicators Source: EENRD 2014

22. How are the CEQs developed? 1. Common Evaluation Questions • Linked to RD policy objectives • Causal-effect questions (To what extent..?) • 1. Development of CEQs linked to the RD policy objectives • 3. Identification of relevant common rural development indicators • Common judgment criteria • Set the foundations to assess the success of intervention • Formulate explicit judgments on the basis common terms • Facilitate the identification of data, information and analysis needed • 2. Development of common judgment criteria • Common RD indicators • Additional information when necessary Source: EENRD 2014

23. CMEF – guidance Handbook Annex 1: • Guidance notes - A: Choice and useofindicators, B: On-goingevaluation , C: Mid-termevaluation Annex 2: • Guidance notes – D: Hierarchyofobjectives, E – MeasureFiches Annex 3: • Guidance notes – F – K, CommonindicatorsFiches Annex 4 • Guidance J – O Otherguidances

24. Set of common evaluation questions(ex post guidelines) • Original set of 150 common evaluation questions of CMEF has been simplified and reduced to the essential demand for knowledge from the European perspective. 3 groups of CEQs Programme-related: Community strategic priorities Health Check objectives 7 impact indicators TA and NRN Efficiency of RDP resources Measure-related: Contribution of the measure to its axis objectives Other measure effects and contribution to other axes objectives Leader approach-related: Community strategic priorities: Employment, diversification and governance Leader approach LAG contribution to LDS and RDP Source: EENRD 2014

25. RDP objectives Evaluation Question Indicators Baselines and their quantification, Target levels Overall strategic objectives Evaluation questions – cross-cutting Impact indicators Horizontal baselines, objective related, contextual Targets for impact indicators Axis specific objectives Evaluation questions Axis/sector specific Result indicators Axis based baselines, objective related, contextual Targets for results indicatiors Measure/operational objectives Common and program spec. Eval. questions Output indicators Baseline values of outputindicators Targets for output indicators M&E framework

26. CMEF guidance on indicators CMEF provides guidance for MAs in setting up indicators at each level of intervention and baseline indicators Annex 3 of the CMEF Handbook provides detailed description of all common indicators (Guidance notes F – K): • F: COMMON INDICATOR LIST (overview of all common indicators) • G. BASELINE INDICATOR FICHES (detailed description of indicators) • H. OUTPUT INDICATOR FICHES • I. RESULT INDICATOR FICHES • J. IMPACT INDICATOR FICHES

27. Indicator Fiches Each indicator fiche contains the following elements: • Type of indicator • Related measures • Measure Codes • Definition of the indicator • Subdivision • Unit of measurement • Level of collection • Responsible actor for collection • Collection method/good practice • Sources • Registration frequency

28. Measure Fiches Guidance on practical use of indicators in monitoring and evaluation of RDPs • Guidance note E provides the following on use of indicators within each particular measure: • Measure Code • Rationale of the measure • Target group • Target area • Common indicators • Link rationale of the measure and indicators • Evaluation questions

29. Ex post evaluationguidelines Part III Part I Introduction Part II Toolbox Provides additional practical tools for ex post evaluation preparing, implementation and reporting. Mainly Managing Authorities Introduces the process of the ex post evaluation, steps to be conducted and role of evaluation stakeholders Explains also specificities with respect to NRN For everyone Introduces ex post evaluation and its role in policy cycle Explains the scope and focus of ex post evaluation, legal requirements and common evaluation elements Mainly Evaluators Discusses and explains intervention logic, evaluation questions, indicators, methods and data Source: EENRD 2014

30. Evaluation. Whatit is about? EU policy objectives SWOT and needs assessment Basis of evaluation RDP Intervention logic Attribution of impacts Focus of evaluation Evaluation questions Indicators Overall objectives EU/MS Impacts Methods Results Specific objectives EU/MS Data Outputs Operational objectives EU/MS Measures and their implementation Measurement tools Collection of evidence Efficiency Source: EENRD 2014

31. Thankyouforyourattention jela@ruralevaluation.eu jelatvrdonova@gmail.com