Evaluation of EU Structural Interventions

1. Evaluation of EU Structural Interventions Ex-ante Evaluation Mid-term / Updated Mid-term Evaluation Ex-post Evaluation Project Assessments

2. Königsteiner Strasse 116D-65812 Bad SodenGermany Phone: +49-(0)6196-654168Fax: +49-(0)6196-654178 Internet: www.prac.de

3. Purpose

4. Explanation on the Purpose The quality and adequacy of projects and fanding activities go hand in hand with the quality and adequacy of the programme in which the project is part of. A regional or sectoral development programme (as e.g. the EU Structural Funds) aims at fostering inter-regional convergence and/or stabilising important regional growth poles for the national economy. The purpose is not to simply subsidise institutions, business and individuals, Funding is the means and not the end. Programme evaluation has to consider the respective quality, adequacy and implementation of a programme. Therefore the criteria applied in programme evaluation need also be translated into the quality assessment of the projects materialising the programme. Hence, as a truism, a project can only be as good as the programme and the programme can only be as good as the projects.

5. 1. Ex-ante Evaluation

6. Structural Funds interventions are implemented in accordance to Operational Programmes (or Single Programming Documents) which must be (1) based on a reliable qualitative and quantitative analysis of the socio-economic situation of the region including its SWOTs and previous experiences, (2) the determination of a target-oriented development strategy, (3) the choice of appropriate measures to materialise the strategy, (4) suitable projects and (5) sound provisions governing financing, responsibilities, evaluation etc.. The assessment of the consistency of the above listed contents of an OP (or SPD respectively) is the core rationale of the Ex-ante evaluation. Further tasks to be carried out in the context of the Ex-ante Evaluation are (7) the quantification of the objectives and (8) the estimation of the overall socio-economic impact.

7. Objectives of the Ex-ante Evaluation • Assessment of the appropriateness of the plan in addressing the regional development problems; • Assessment of the consistency of analysis and concept; • Definition of suitable indicators and quantification of objectives;

8. Objectives ... • Analysis of the adequacy of the implementation and monitoring arrangements; and • Impact assessment

9. Lessons learned from past experiences; Difficulties and shortcomings experienced and to be tackled; Assessment of the validity of data used in previous programmes; Consideration of thematical evaluations; Consideration of experiences in other programmes; and Verification of the socio-economic analysis and the consistency of SWOTs Assessment of Appropriateness

10. Consistency of SWOTs • Strengths are static features to be supported • Weaknessesare static features to be reduced • Opportunitiesare dynamic features to be reaped • Threats are dynamic features to be forestalled

11. Consistency of Analysis and Strategic Concept • Are the strategic objectives well specified ? • Do the strategic objectives sufficiently respond to the needs identified in the socio-economic analysis? • Are the proposed measures justified by the strategy? • Is the strategy sufficiently coordinated with other related supra-national or national policies/programmes (nat. regional policy, NAP, CIs, Objective 3 etc.) ?

12. Basic indicators: Programme indicators: Measure related indicators: Performance indicators: Gross regional value added Supported investment No. of supported SMEs Disbursement/allocated funds Suitable Indicators (examples)

13. Quantification of Objectives • Related to INPUTS-OUTPUTS-RESULTS-IMPACTS • Determination of a realistic quantification of indicators • Comparison with results achieved in former programme periods • Proxy-indicators to monitor intangible effects

14. Appropriateness of the Implementation System • Management and implementation responsibilities; • Transparency; • Efficiency and effectiveness of inter-institutional co-ordination; • Competitiveness in project choice; • Accountability in line with national and community regulations (competition policy, additionality, horizontal priorities, innovation, NAP etc.); and • Partnership in programming and monitoring.

15. Impact assessment • Micro-economic net effects (notably employment, productivity, regional GDP); • Macro-economic net effects for large programmes (macro-econometric estimations and simulations on short-term demand, long-term supply, labour market and government sector)

16. 2. Mid-term Evaluation and its Update

17. Mid-term Evaluation • Update of the analysis of relevance; • Update of the analysis of consistency of priorities; • Analysis of the validity of the objectives; • Comparison of plan and achievements (vertical and horizontal); • Assessment of the implementation performance in terms of financial and administrative management; • Contribution to the decision on the allocation of the Performance Reserve;

18. Mid-term Evaluation ... • The responsibility of the mid-term evaluation is with the member state; • The evaluation is to be carried out by an independent assessor; • An update of the mid-term evaluation is to be elaborated one year before termination of the CSF/SPD; and • The evaluations are accompanied and approved by the monitoring committees.

19. Analysis at Programme Level • Were recommendations from former evaluations sufficiently considered? • Have the socio-economic problems (justifying the intervention) changed? • Is the SWOT analysis adequate? • Is the Programme still consistent and coherent? • Has the programme been implemented professionally?

20. Analysis at Priority/Measure Level • Financial implementation analysis • Effectiveness analysis (plan vs. achievement) • Efficiency analysis (unit cost comparisons) • Impact analysis (GDP, net employment) Cf. EU Commission/DG Regio: Working Paper 8

21. Update Mid-term Evaluation The role of the updated mid-term evaluation is to confirm the implementation of recommendations put forward in the mid-term evaluation and to cover fields of analysis which could not be evaluated earlier. Cf. EU Commission/DG Regio: Working Paper 9

22. 3. Ex-post Evaluation

23. The Ex-post evaluation is carried out under the responsibility of the EU Commission. It is elaborated as a global evaluation for each of the regional(Objective 1 and 2) and sectoral (Objective 3) programmes covering all targeted member states. The main purpose is to assess the quality, effectiveness and impacts of the programmes. It should clearly reveal the strengths and weaknesses of the planning process, the strategic approach and implementation. Lessons learned from this exercise should contribute to better policy formulation in current and future programming periods.

24. Scope of the Ex-post Evaluation • Analysis of the relevance of strategy and the expenditure at EU, national and regional levels; • Effectiveness analysis; • Efficiency and impact assessment; • Assessment of the delivery system; • Analysis of the Community value added; and • Lessons learned for the future and policy recommendations.

25. Are the strategies adopted by the regions sufficiently relevant? Are the strategies appropriately implemented in practice? Has there been any deviation of the interpretation of the strategy (new direction)? Is the philosophy of the strategy still in line with that of the prior evaluations? Were the funds accordingly allocated (dispersion vs. Concentration? Were the changes of the strategy relevant or arbitrary? Was the aid intensity relevant? The degree of coherence in the approaches across all CSFs/SPDs of Objective 1 or Objective 2 respectively? Relevance of Strategy and Expenditure

26. Effectiveness Analysis • Comparison of plan and realisation in terms of policy priorities (e.g. private sector support, innovation, business related infrastructure, HRD etc.); • Assessment of the formerly expected and realised performance of the policy instruments (e.g. financial engineering)

27. Efficiency and impacts • Value for money assessment; • Assessment of the net-effects (employment, productivity, regional GDP etc.); and • Macro-econometric model estimations for major programmes (notably Objective 1)

28. Delivery System • Assessment of the institutional management and implementation capacity; • Functioning of partnership; • Sufficiency of transparency; • Functioning of the monitoring system; • Functioning of the project selection procedures; • Appropriateness of the practical provisions to ensure the legal underpinnings; and • Functioning of the quality control mechanisms

29. Community Value Added • Economic and social convergence; • Internal and external Cohesion; and • Institutional improvements (e.g. better coordination)

30. `Lessons Learned` and Policy Recommendations • Conceptual level and supra-national regulations (EU regional policy formulation); • Justification of the financial contributions and/or the level of additionality respectively; and • Implementation level (management, M&E partnership etc.)

31. THE EVALUATION CYCLE Ex-ante Evaluation Programming Mid-term Evaluation Mid-term Update Implementation Adjustments Ex-post Evaluation Programme results

32. Project assessments

33. Project Assessment within Structural Funds Programmes • Rules and funding regulations laid down in the Programme Complement • Supra-national, national and regional funding regulations • Assessment of the scope and aim of the application • Control/audit • Funding approval … Proof of fund utilisation

34. Criteria listed the Programme Complement The PC includes the specific funding regulations and criteria for all measures of the programme. Any institution/individual applying for funding needs to comply to the provisions made in the PC. The PC defines The valid funding provisions The thematical/sectoral scope of funding The pre-conditions of funding (e.g. specific target groups) Eligible expenditure Ceiling of the subsidy Period of validity Geographical coverage

35. Funding regulations • Regulations of the EC (e.g. Council Regulation 1260/1999) • European competition law (e.g. Article 92) • National competition law • Regional funding provisions Note: all regulations need to be co-ordinated

36. Project assessment grid (I) • Is the project thematically suitable for funding under the respective measure? • Does the project contribute to the quantified objectives of the measure? • Is the project idea sufficiently original? • Does the project suggest synergy effects for the programme? • Is the required co-financing adequate?

37. Project assessment grid (II) • Is the applicant (or the group of applicants) part of the target group? • Is the applicant (or the group of applicants) trustworthy • …

38. Controlling/auditing • Internal controlling of the managing authority notifying fraud and irregularities • Monitoring of the project progress by the Managing Authority • Controlling and issuing of expenditure notes by the Paying Authority • 5% random audits carried out by the Independent Auditing Unit

39. Author: Rolf Bergs, 4 November 2004