Guidance document on ex-ante evaluation

1. Guidance document on ex-ante evaluation 15 March 2012 Joint presentation to the ERDF and ESF Evaluation Partnership Meeting

2. Ex ante evaluation: our approach What is different? Common Regulation & multi-fund programmes Stronger focus on results / Europe 2020 strategy More detailed Article 48 Links with Partnership Contracts, SEA Our common approach: Stick to Regulation (article 48 and related articles)-as proposed by Commission Focus on ex ante evaluators' role When necessary, give examples of differences between ESF/ERDF

3. Ex ante evaluation Guidance • Programme strategy Q&A • Indicators, monitoring and evaluation • Consistency of financial allocation • Contribution to Europe 2020 Q&A • Strategic Environmental Assessment • Evaluation Process Q&A

4. Programme strategy • Consistency of programme objectives • Coherence • Horizontal principles • Linkage between supported actions, expected outputs and results Q&A

5. Programme strategy – key reference points Art. 24(1) CPR: • Europe 2020 strategy • Common Strategic Framework • Partnership Contract Art. 87(2)(a)(i) CPR: • Country specific recommendations • national and regional needs

6. Consistency of programme objectives • Step 1: The ex ante evaluator should assess the challenges and needs identified by the programme. • Step 2: The evaluator should examine if these challenges and needs have been consistently translated into thematic objectives, priorities, investment priorities and corresponding specific objectives.

7. Step 1: Challenges & needs (1) • National & regional programmes: the evaluators should primarily base their assessment on the NRP, country-specific recommendations, European semester analyses • In addition: • National ERDF sectoral programmes: specific sectoral analysis & territorial priorities • Regional programmes: specific regional situation and needs • Specific challenges of sub-regional or functional areas or of specific target groups • Horizontal principles (Art. 7 & 8 CPR) Justification of specific regional challenges and needs

8. Step 1: Challenges & needs (2) • The ex ante evaluator should assess whether the identified challenges and needs are consistent with the analysis of disparities and development needs in the Partnership Contract. • The evaluators should also base their assessment on the Common Strategic Framework. • They should examine the prioritisation of the identified challenges and needs and suggest reviewing it, if appropriate.

9. Step 2: Programme objectives thematic objectives investment priorities Specific objectives • While appraising the consistent translation of the identified challenge & needs into programme objectives, the evaluators should focus on: • Their appropriate weight in the investment priorities • Precise formulation of specific objectives • Rationale and evidence for omitting major challenges or needs • The evaluators may recommend complementary analyses

10. Coherence Internal coherence The evaluators should assess: • Relationship between specific objectives • Coordination mechanisms in multifund programmes External coherence The evaluators should examine: • Analysis of programme contribution to other strategies/programmes and of the influence of these on the expected results of the programme • Planned integrated territorial approaches

11. Horizontal principles • The evaluators should assess the following elements under each principle (Art. 87(3) CPR): • Equality between men an women • Arrangements to integrate the gender perspective at the level of the programme and operations • Contribution to its promotion • Anti-discrimination • Specific actions in preparation and implementation of programme • Sustainable development • Specific actions in the selection of operations Opinion of national equality bodies

12. Linkage between outputs and results • Clear intervention logic: condition for good programming and evaluations! • Recommendation: use a logical framework for each priority axis or investment priority • The evaluator should examine: - What is the expected change? • How will outputs contribute to intended results? • Will the proposed actions effectively lead to these outputs? • What other factors could influence the expected results? • Would evidence suggest other approaches?

13. Linkage between outputs and results • Are the planned form of support the most effective? What is the rationale? (grants, prizes, repayable assistance and financial instruments and a combination: Art. 56) • Will the actions effectively meet the needs of specific territories or target groups?

14. Indicators monitoring and evaluation • Relevance and clarity of proposed indicators • Baseline and target values • Suitability of milestones • Administrative capacity • Data collection • Evaluation • Consistency of financial allocation • Contribution to Europe 2020 Q&A

15. Indicators, monitoring and evaluation Art. 24 CPR - For each priority axis, indicators to assess progress towards achievement of objectives - Basis for monitoring, evaluation, review of performance • Increased focus on results: role of well designed indicators and evaluations • New focus on territorial integrated approaches: reflected in monitoring and evaluations where relevant • Common indicators compulsory

16. Indicators, monitoring and evaluation • Relevance of indicators • Output indicators reflect the main operations • Result indicators are responsive to the policy (values evolve with development of operations) • Result indicators reflect the most important intended change • Common indicators - ERDF: used where relevant to operations and specific objectives / ESF: monitored under all investment priorities • Results may be close to the intervention, short or long term: depends on the intervention logic, on the intended change

17. Indicators, monitoring and evaluation

18. Indicators, monitoring and evaluation • Clarity of indicators • Unequivoqual title, clear definition • Normative interpretation • Robustness and transparence • Robustness: reliability of data source, representativeness of samples (survey) • Data sources for baselines and target values + definitions are publicly available • See ex ante conditionality (Annex IV CPR)

19. Indicators, monitoring and evaluation Baseline and target values • Availability of baseline data for result indicators? • latest available data (ERDF and ESF Regulations) • advice on sources and methods to collect data when necessary, or research to establish baselines • Realistic targets? • Compared to actions and form of support, financial allocations • For output indicators, compared to references (previous programmes or sectoral norms) • For result indicators, taking into account other factors

20. Indicators, monitoring and evaluation • Performance framework: a subset of indicators for each priority axis • Suitability of milestones • Do they capture essential information on the progress of a priority? • Can they be achieved at the review points? • Will data be available for 2017/2019 progress reports?

21. Administrative capacity • The ex ante evaluator should assess whether adequate human resources and administrative capacity for the management of the programme are in place • Including: Have possible previous bottlenecks at the level of MA and IB been tackled? • The evaluator may advice on how to address any issues

22. Data collection The ex ante evaluator shall assess whether the data collection procedures are suitable. Whether they: • Respond to all requirements for monitoring • including: is data available on time, in the required form and quality? How will the data be collected and stored? Will administrative data bases be used to reduce admin. burden? Etc. • Provide the data needed for evaluations • including: data to i.a. assess effectiveness, efficiency and impact? What are the mechanisms to ensure good quality data? Etc.

23. Evaluation The ex ante evaluator may advice: • On possible evaluation needs for on-going evaluations • On evaluation methodologies and resulting data needs • On setting up the evaluation plan

24. Consistency of financial allocation • The ex ante evaluator shall assess the consistency of the financial allocation on the basis of the identified challenges and needs and the proposed actions and selected forms of support • The evaluator may assess whether the different funds are adequately combined and contribute to integrated approaches

25. Contribution to Europe 2020 • Taking account of the proportionality principle and against the background of national and regional needs • the ex ante evaluator shall assess the programme’s potential contribution to Europe 2020 in the light of its thematic objectives and investment priorities.

26. Process • Strategic Environmental Assessment • Timing • Interactive and iterative process • Partnership and multi-level governance • Evaluation methods and financing • Final report and publication • An independent process • What requirements for evaluators? Q&A

27. Strategic Environmental Assessment • A new link with Ex ante: "shall incorporate, where appropriate, the requirements for SEA…" - "Where appropriate": direct impact on environment • "Incorporate": does not mean that the ex ante evaluator has to do SEA • "Requirements": environmental report, consultations, information, monitoring • How to incorporate SEA: still under discussion

28. Timing • Ex ante evaluation & Partnership contracts PC includes (Art. 14): • Summary analysis of ex ante evaluations justifying selection of the thematic objectives and the indicative allocation of Funds • Summary of main results expected for each thematic objective • Developping ToR … now - PC and OPs adopted Oct.-Dec. 2013 - 6 months negotiations: draft PC & OP April-June 2013 • Programme elaboration 6-8 months: starts Aug.-Oct. 2012 • Calls for tender launched mid 2012

29. Interactive and iterative process Work in stages Timing and interaction Steering group 29

30. Partnership and multilevel governance Art. 5 CPR • Strengthening with the objective of: • Increasing legitimacy • Build on wider expertise and knowledge • Ensure collective commitment and shared understanding of expected results • The evaluators should review if the relevant stakeholders are involved in the preparation of the programme as well as in its implementation, monitoring and evaluation • The evaluators should base their assessment on the Commission Staff Working Document “Elements for a Code of Conduct on Partnership” 30

31. Evaluation methods and financing • Mix of methods: - TBE: literature review, workshops, interviews, focus groups, peer reviews • Involve partners in programme design • Cost: - number of evaluation questions • methods • Extra tasks (quantification of baselines, complementary analyses)

32. Final report and publication • Submitted with the OP to the Commission - Main evaluation methods used - Changes and improvements to programme - Final assessement of OP - Strategic Environmental Assessment • Executive summary (+ in English) • Made available to citizens

33. An independent process Article 47 CPR: "carried out by experts functionally independent of the authorities responsible for programme implementation" No hierarchical link with the entity responsible for implementation: internal expert may contribute Balance between an interactive/iterative and independent process: critical jugements on the different elements of the programme and its coherence

34. What requirements for evaluators? • Knowledge of EU and national documents • Deep knowledge of context • Situation of areas or target groups with specific needs • Situation of economic sectors - Able to judge quality of socio-economic analysis • Experience in evaluations • Data needs for future impact evaluations • Relevance of indicators within the intervention logic (thematic evaluations)