Monitoring and evaluation of Cohesion Policy Austrian experience and perspectives

1. Slovak Society for Evaluation Monitoring and evaluation of Cohesion Policy Austrian experience and perspectives Bratislava, November 30th 2010 R. Hummelbrunner, Graz Austria

2. Regional and Cohesion Policy in Austria • Key features of regional policy in Austria • Regional policy is competence of “Länder” (regions - NUTS II) • National level: coordination, orientation (Federal Chancellery) • Mechanism for aligning regional policies at national level (ÖROK) • Cohesion Policy – Structural Funds in Austria • 8 OPs for Objective 2 at NUTS II level, 1 OP for Objective 1 – Phasing Out (Burgenland), 7 OPs for Objective 3 (ETC) - CBC strand • Programmes at national level for Employment and Rural Development • No other thematic / sectoral OPs at national level • Central Monitoring System for all ERDF Programmes • Austrian Conference for Spatial Planning (ÖROK) functions as network of MAs for ERDF Programmes

3. Financial Resources 2007-13 Source: ÖROK

4. Co-ordination / exchange of evaluations at national level Platform Evaluation (ÖROK) Cross-programme exchange, horizontal themes Evaluation Team Evaluation Team Evaluation Team Evaluation Team Evaluation Team Contracts Operational Programmes

5. Key innovation during 2000 – 2006: On-going evaluation • Voluntary activity of various ERDF Programmes • Objective 1 Programme Burgenland, five Objective 2 Programmes, four INTERREG IIIA Programmes, URBAN II Graz • Period between MTE and MTE Up-date (2003 – 2005) • Implementation of on-going evaluations • Integral element of evaluation contracts by MAs • Specification and steering mostly by Working Group at programme level • Focus: Result oriented (impact assessment) and/or process oriented (understanding of implementation and framework conditions) • Documentation: Internal papers, minutes of workshops/meetings, contributions for Annual Reports • Exchange / reflection on process and results of on-going evaluations at national level (ÖROK)

6. Evaluation components in Austria Programme period 2000-2006 Mid-Term Evaluation & Up-date embedded in on-going evaluation process 2000 - 2001 - 2002 - 2003 - 2004 - 2005 - 2006 On-going MTE Up-Date.

7. Relevance of evaluationforsteering (actors, functions) • Mid-Term Evaluation and Up-date • External requirement, timing and topics pre-defined Relevant for: • MAs (e.g. use of resources, n+2, achievement of targets) • Monitoring Committee (e.g. Programme changes, reallocations) • On-going Evaluation (between MTE and Up-date) • Utilization-focused: Information needs/questions of programme actors • Quick results for programme modifications, quality improvement Relevant for: • Implementing Bodies at Measure level (e.g. funding conditions, project selection and development, publicity and visibility of measures) • MAs (e.g. implementing mechanisms, results and impacts)

8. Experience with On-going Evaluations 2000 – 2006 • Value-added of On-going Evaluation • External view on programme implementation, „External mirror“ for MAs • New role of evaluators: „Coaching“ of programme actors in view of professionalising the entire programme system • In-depth treatment of selected topics / areas • Accumulation of insights into programme realities • Processing of information in small portions, easier to handle • Quick and flexible reaction on questions / problems of MAs • Rapid implementation of findings / recommendations • Timely reaction and adjustments at programme or measure level

9. Innovations 2007-13: Strategic governance process • Governance process for NSRF (strat.at) and nine Objective 2 OPs, to • Promote learning and exchange of experience on OPs • Provide ideas and practical knowledge for various stakeholders • Generate contributions for future regional policy (EU, national ) • Based on established networks and structures • Network of Managing Authorities (within ÖROK) • Steering Group (on-going definition of thematic priorities, work plan) • Main elements of governance process • Work commissioned by ÖROK (evidence, expertise) • Workshops to discuss/validate expert inputs • Events for wider public (dissemination, information)

10. Outline of strategic governance process in Austria Objectives and Strategy (Sub)objectives per Priority Strategy fields Community Strategic Guidelines Contributions to objectives NSRF STRAT.AT GOVERNANCE PROCESS NSRF (= M&E, interactive exchange, Reflection> institutional learning) • Operational • Programmes • Priorities • Actions- Mechanisms PROGRAMME GOVERNANCE (= Achievement of objectives, regional impacts) Inputs (Resources, Rules) Outputs (Projekts) Use of outputs Results Analysis Regional Impacts External influence Territorial framework conditions (physical, socio-economical, institutional)

11. Innovations 2007-13: Cross-programme activities • Analysis of common themes / challenges • Example: Assessment of Structural Funds implementation system in Austria („Governance Check“) • Joint reflection and learning • Example: 15 Years of INTERREG / ETC programmes in Austria - ‘Looking back - looking ahead’ • Comparative implementation analysis • Example: Cross-programme evaluation of national funding scheme for environment / climate change • Participation in larger scale cross-programme evaluations • Example: Cross-programme evaluation of ETC programmes in Central and Southeast Europe, Operational and Thematic aspects

12. Innovations 2007-13: ProcessMonitoring of Impacts • What is Process Monitoring of Impacts? • Tool for impact-led monitoring of an intervention • Focus on processes, which should lead to impacts • Use of Logic Models (impact diagrammes) • What is it for? • Illustrates causal relations between programme outputs and effects in a plausible manner • Enables a concise, systemic overview of impact patterns of complex interventions, e.g. programmes • Provides early information for programme management (MAs, IBs) required for effective steering

13. Monitoring of change processes • Impact chainsasbasicscheme Inputs are used to produce outputs (= projects) Outputs are used (by someone, in a specific manner) to reach results Results will lead to impacts (intended / unexpected) in a plausible manner Inputs Outputs Results Impacts Financial Table Typology of projects (output) to be supported Objectives at the level of Areas of intevention / Activities Objectives at the level of Priority or Program Commitments Output- Indicators Result- indicators Disbursements

14. Impact diagramme AoI A 4.1 – Softmeasures to strengthen innovations in entreprises Enterprises (especially SME) collaborate in networks (also with large enterprises) Increased linkages between enterprises at regional level / scale Advise for co-operation AF6 Enterprises (especially SME carry out reorganization processes Sustainable stabilization of enterprises Adaptation to international competition Enterprises (especially SME) carry out product finding processes New / improved products, services and production processes External expertise, process consulting Enterprises (especially SME) introduce new technologies Increase in employment / new jobs AF6 Enterprises (especially SME) gain new markets New contacts with clients/ new orders Enterprises (especially SME) conceive innovation-/ investment projects Services to sensitize for innovations AF6

15. Pilot applicationswithprogrammes Reg. Competitiveness 2007 - 2013 • Programme Styria • Based on impact diagrams established during programme preparation • Application with 5 Areas of Intervention (of totally 11) • Project sample (used for up-dating of impact diagrams): 70 • Projects approved (used for weighting of impact chains): 840 • Duration of pilot application: July - November 2009 • Programme Carinthia • Internal assessement of projects approved so far (ca. 40) based on impact diagrams (dating from ex-ante evaluation) • Assessment accompanied externally • Finalization (Report) October 2009

16. ApplicationwithprogrammeStyria 2007 - 2013 • Procedure / steps • Up-dating impact diagrams from the programming process (based on current version of internal „Program Complement“ ) • Attributing all (840) projects in line with their primary impact chain (basis: short descriptions contained in ERDF Monitoring system, information provided by IBs) • Weighting of impact paths according to number of projects and financial volume, calculation and graphic representation • Integrating indicators (from ERDF Monitoring database) to provide information on final achievements of processes • Validating and complementing impact hypothesis through interviews with (selected) project owners • Discussing findings / conclusions with IBs (Funding Authorities)

17. Impact diagramme AoI 4.1 – Softmeasures to strengthen innovations in entreprises 2 Enterprises (especially SME) collaborate in networks (also with large enterprises) Increased linkages between enterprises at regional level / scale Advise for co-operation AF6 40 Enterprises (especially SME carry out reorganization processes 65 Sustainable stabilization of enterprises 25 Adaptation to international competition Enterprises (especially SME) carry out product finding processes 143 New / improved products, services and production processes 40 External expertise, process consulting Enterprises (especially SME) introduce new technologies Increase in employment / new jobs 9 AF6 Enterprises (especially SME) gain new markets New contacts with clients/ new orders 2 Enterprises (especially SME) conceive innovation-/ investment projects Services to sensitize for innovations AF6

18. Impact diagramme AoI A 4.1 – Softmeasures to strengthen innovations in entreprises Weighting according to financial volume (public funds) 127.900,00 Enterprises (especially SME) collaborate in networks (also with large enterprises) Increased linkages between enterprises at regional level / scale Advise for co-operation AF6 352.997,53 Enterprises (especially SME carry out reorganization processes Sustainable stabilization of enterprises 789.269,49 436.771,66 Adaptation to international competition Enterprises (especially SME) carry out product finding processes New / improved products, services and production processes 3.620.240,85 1.652.266,92 External expertise, process consulting Increase in employment / new jobs AF6 Enterprises (especially SME) introduce new technologies 1.084.671,36 Enterprises (especially SME) gain new markets New contacts with clients/ new orders 439.728,68 Services to sensitize for innovations Enterprises (especially SME) conceive innovation-/ investment projects AF6

19. Pilot applications with programmes Territorial Co-operation • INTERREG IIIA programmes (2000-2006) • On-going evaluation Austria – Slovenia • Ex-post evaluation of Swiss participation in INTERREG III • ETC Cross-border Co-operation Programmes (ex-ante evaluation) • Austria – Slovenia • Austria – Bavaria • Lake Constance (AT / DE / CH / LIE) • ETC Trans-national Co-operation Programmes (ex-ante evaluation) • Central European Space • South East European Space • Studies / Work for INTERACT downloadable at www.interact-eu.net/mt

20. Results and experience gained so far Main results of method: Sound and quantifiable connection between supported projects and effects at programme level (expected results, impacts) Systematic capture (respectively clarification) of project types and categorisation of their use dimensions Capture and representation of linkages (within and between Areas of Intervention, with other programmes / interventions) Focus on supported project types and their use, thus early indication of likely achievement of impacts Limits of the method: Provides assumptions on impact paths, but does not capture the actual achievement of results / impacts However, they can be integrated (once achieved), e.g. via data input from ERDF monitoring system, surveys, project reports

21. Evaluation of OPs 2007 - 2013 • National Evaluation Plan 2007 – 2013 (gradually specified) • Preparation by ÖROK, imbedded in Governance process of NSRF • Combination of considerations of MAs and national-level actors • Considerations by MAs for on-going evaluations • Utility for MAs in implementing their tasks (e.g. Annual Reports, justifications for programme changes) • Strategic assessments (e.g. achievement of objectives/results, mid-term reviews) • Operative assessments based on data from Central ERDF Monitoring System (often done internally) • Work commissioned ad-hoc, programme specific approach

22. Thank you for your attention Richard Hummelbrunner ÖAR - Regionalberatung Alberstrasse 10, 8010 Graz, AUSTRIA T: +43/316/31 88 48 hummelbrunner@oear.at www.oear.at