1 - Monitoring and Evaluation

1. 1 - Monitoring and Evaluation Jela Tvrdonova 2012

2. Outline of the presentation • Strategy/program in development context • Strategy/program logic • Monitoring and evaluation in development context • Evaluation processes • Monitoring and evaliation framework

3. The strategy/program in the development context • The development of countries, regions are often governed via strategies and programs, which are time limited • In strategies/programs it is not possible to solve all problems/needs of particular area, country or region in the same time. • Problems/needs are targeted by strategy/program interventions • It is necessary to decide which areas of life are the priority to be targetedvia interventions of the strategy/program life time

4. The strategy/program in the development context Intervention contains: • priority areas, • overall and specific objectives, • measures and actions to be taken Intervention reflects: • SWOT • Identified needs and problems • Vision established by development stakeholders

5. Stakeholders Territory Vision Strategic objective I n t e r v e ntion Situation description SWOT Problem analysis and priorities Specific objectives Measures, activities, Financial plan Strategy/program framework and intervention logic

6. Interventions in the development context • Interventions are always functioning in certain contextual environment, which „goes its own way“ as well. • Example of the EU rural development programs

7. Figure. 1 Context of the program/strategy in rural development Rural areas as defined by national definition/all stakeholders in rural areas The area of intervention rooted in the EU Strategic guidelines EC reg. 1698/2005, eligible beneficiaries, EU objectives The intervention area rooted in National strategic plan for RD ,selected eligible beneficiaries – national objectives Other interventions and objectives • Example - Context of the development program in the EU

8. Contextofthestrategy/program and itseffects • In such a complex situation is sometimes complicated to evaluate effects of the intervention. • The overall situation is located in changing environment, which can influence the intervention of the program/strategy in many different ways.

9. Development program/strategy Other programs effects Other Non-program effects Direct effects on eligible beneficiaries In-direct effects on non-beneficiaries In-direct effects on eligible beneficiaries Program/strategy´s effects

10. Program/strategy logic • Strategies/programs are governance tools • They shall bring sustainable solutions to territory´s needs and challenges. • Program/strategy effects on the territory depend to the high extent on the quality of its intervention logic design. • It shall prove the efficient use of public and private funds in favour of whished change • In the ideal case the development of the area shall go with the line of its goals and objectives although still influenced by external factors • Only well established strategoes have realeffects on the territory in whished directions supported by sufficient public and private means

11. Change Needs, visions Context of development– all factors influencing development Change observable in reality during and after intervention Whished change in priority areas before intervention Overall /strategic goal Impact Specific objectives within development priorities Result Output Operational objectives of measures Measures Inputs Overall contextual baseline socio-economic and environmental situation before intervention Logic of the program/strategy

12. Monitoring and evaluation in development context • It is necessary to distinguish among various processes – audit, monitoring and evaluation

13. Audit and Monitoring • Audit , control is focused on the correct administrative and financial management of measures • Monitoring is continuous and systematic observation of financial and other inputs, implemented projects and activities and intermediate outputs within the implementation of particular measures of the local development strategy. It also contains the program results, although these are part of the evaluation as well

14. Evaluation • Evaluation is focused on the assessment of impacts of the program/strategy within certain development context, netted out from deadweigth, leverage, substitution, diplacement and multiplying effects • Evaluation means also looking at the effectiveness, efficiency and the relevance of particular measures and offers useful reflexion for re-formulating or overall change of policies.

15. Evaluation • In this process the quality of the evaluation substantially depends on the methods choosen • It also depends on the quality of available data and information collected in the monitoring processes or also contextually, which means that both processes must be interlinked and designed in interaction straight from the beginning.

16. Evaluation processes On going evaluation is helping us to collect data which help us to carry the evaluation exercise in time periods – ex-ante, mid term and ex post Periodical evaluations • Ex ante evaluation – independent study, which helps us to understand if the projected intervention is really going to enable to achieve envisioned change expressed in objectives. • Modeling approaches could represents solid methodological gound for this purpose

17. Evaluation processes Mid term evaluation • Is the independent exercise which is carried out in the middle of the strategy/program implementation. • The changes on the level of specific objectives/development priorities shall be observed and results shall be measured • Analytical methods are envisioned for this stage • At minimum evaluation framework for impacts shall be included

18. Evaluation processes Ex post evaluation • The last independent study is carried out at the end of the strategy implementation. • The impact of the strategy implementation shall be visible and strategic objectives shall be achieved • Analytical methods are envisioned at tis stage

19. Contextual monitoring and evaluation • This M/E is based on the prediction that the intervention of the strategy exist in the environment which might itself to strengthen or weaken our intervention instruments • It helps to distinguish to which extent the change was caused by our intervention and to which extent the change was caused by external factors • Contextual indicators help to carry this process

20. Self-evaluation • is the process which is carried out in the case when stakeholders create partnerships, which prepares,implements and evaluates the strategy or program • it shall help to steer/coach the strategy in such a way that objectives are followed • for this purpose any findings of not well prepared interventions within the self-evaluation process shall be discussed and necessary changes of the strategy shall be implemented as the lessons learned • this does not exclude the independent evaluation

21. Monitoring and evaluation framework – part of the strategy/program design • Already in the stage of the strategy/program design it is necessary to prepare monitoring and evaluation framwork • It contains: • To set up proper monitoring mechanism – indicators – input and output indicators (reflecting the activities and projects level) • To set up proper evaluation system – result (reflecting specific objectives) and impact indicators (reflecting strategic goals) • To design the system together with the setting up objectives, goals, priorities, measures and activities

22. Monitoring and evaluation framework contains • Evaluation questions • Indicators • Baselines • Targets All this is interlinked with the intervention logics of the overall program, priority and particular measures

23. Monitoring and evaluation framework • shall be part of the strategy from the beginning • It shall be organicaly incorporated into the strategy and shall be designed based on the situation analysis, needs and objectives, measures and activities

24. Partnership Territory Strategic objecive Impact Impact Indicators E v a l u a t i o n Vision I n t e r v e n t i o n Resource audit Analysis Priorities for development Specific objectives Result Result indicators Input indicators Measures Output Output Indicators Monitoring The strategy framework - completed

25. Evaluation questions • Common – cross strategy evaluation questions and specific (sector/priority/measures) – are those which shall be answered in the evaluation process. • Shall be well formulated so we will receive the answer if objectives were reached

26. Indicators Are instruments for the measurement of interventions Indicators are on the level of: Monitoring – input and output (physical financial) and results Evaluation – results (priority level) and impacts (the cross strategy level)

27. Indicators in monitoring and evaluation framework Inputs Activities Outputs Results Impacts • Long term qualitative changes and effects connected with intervention in certain environment • Achievement of the strategy objectives • Improved situation • Inmediate results on the priority level, qualitative changes connected directly with intervetion • New quality, changes • New knowledge • skills • attitude • nové správanie sa • Resources • money • people • supply • equipment • facilities • What plan does • Services • Training • Processing • Products orservices. Quantitative changes • Numbers of products or services

28. Baselines • Are base data we were able to collect in resource audit or in any other additional data collection and are available in existing data collection systems, which help us to understand the initial situation before the intervention starts • Accessible data shall be in balance with M/E indicators or vise versa – the preferably we shall select indicators which can be filled with data

29. Target levels Are wished situations in which the development priorities shall be after intervention is completed and express the quantitative level of the change E.g. unemployment decrease in 25% is the target level

30. Evaluation questions Indicators Baseline situation (situation analysis,SWOT) Target levels Objectives Strategic /overall objectives Cross strategy /horizontal evaluation questions Impact indicators Baseline values of impact indicators Target levels for impact indicators Specific objectives Horizontal evaluation questions relating tospecific objec/priorities Result indicators Baseline values of result indicators Target levels of result indicators Operational objectives, measures activities Measure specific evaluation questions Output indicators Baseline values of output indicators Target levels of output indicators Monitoring and evaluation framework

31. Example of the monitoring framework

32. Example of the evaluation framework

33. updatedswotanalysisRuralDevelopmentEvaluationSystem2007-2013November 2010Thankyouforyourattention! The Helpdesk of the European Evaluation Network for Rural Development jela@ruralevaluation.eu jelatvrdonova@gmail.com 260, Chaussée Saint – Pierre - Etterbeek - 1040 Brussels – Belgium Tel: +32 2 736 49 60 Email: info@ruralevaluation.eu http//enrd.ec.europa.eu/evaluation/