Agnė Paliokaitė , Senior Policy Researcher , Public Policy and Management Institute, Lithuania

1. Cohesion Policy Support to Innovation in Lithuania: Lessons Learnt for Evaluation Presentation for DIRECTORATE-GENERAL REGIONAL POLICY "EVALUATION NETWORK MEETING" Brussels, 14 April 2011 Agnė Paliokaitė, Senior Policy Researcher, Public Policy and Management Institute, Lithuania

2. ROADMAP • Aims and challenges • Approach • Conclusions • Added value, strengths and limitations Insights, based on 4 studies carried by PPMI in 2010: - Twosystem level evaluationsfor the Knowledge Economy Forum and the Prime Minister’s Office; - ERAWATCH country report; - SF indicators’ system evaluation for the Ministry of Finance (RTDI measures case study).

3. WHY ‘SYSTEM’ EVALUATION? • System (portfolio) evaluation – evaluating policy portfolios, not individual programmes. • Retrospective: new political will put innovation high on the political agenda in 2009/2010. New ideas - need for revisiting the incrementally developed policy mix. • Prospective: need for rethinking the future priorities in the context of ‘Progress Strategy Lithuania 2030’ and the new 2014-2020 Structural Funds period.

4. AIMS OF SYSTEM EVALUATION • To analyse the extent to which SF funded innovation policy portfolio/mix reflects specific conditions and levels of the National Innovation System (NIS). • To analyse how the financial proportions fit to the policy agenda (the preferred ‘routes’). • To present preliminary insights on effectiveness in achieving set targets. • To draw conclusions on governance & monitoring system.

5. 1. Innovation system ‘health’: market, capability, institutional, network, system, and governance failures Innovation Policy and Governance development Hypotheses about bottlenecks Conclusions 2. Intervention logic and policy mixes Hypotheses about bottlenecks Conclusions 3. Extent to which outputs and results are achieved, critical factors EVALUATION FRAMEWORK Relevance (Are we doing the right things?) Effectiveness (Are we doing things right?) Based on: Arnold E. Evaluating research and innovation policy: a systems world needs systems evaluations, Research Evaluation, volume 13(1), 2004 sdafasdfasdfasdf

6. CHALLENGES AND LIMITATIONS • Timing: low absorption of funds at the time of evaluation (most measures started operation in 2009-2010). • Small scale evaluations.  Hence, inability to apply quantitative approach. • A ‘moving object’: innovation policy and governance reform (LIS 2010-2020, SITA); changes in the system of SF objectives. • Inability to rely on the system of quantitative indicators . • Innovation policy specific: M&E exceptionally difficult for innovation programmes: inherently qualitative and diffuse nature of innovation benefits. Long cause-effect chain.

7. QUALITATIVE APPROACH DATA COLLECTION ANALYTICAL TOOLS Assessment of the innovation system and the RTDI policy mix using the ‘system failures’ framework; Logical models and reconstruction of the policy intervention logic; Meta-analysis of previously carried out studies and analysis of trends in the theoretical debate; Data integrating methods: scenarios and road-mapping; Comparative analysis / benchmarking of other countries’ experience; Risk analysis, critical factors and analysis of policy options. • Semi structured interview programme with stakeholders and target groups (~30 in total); • Desk research: literature review, secondary and administrative data; • Expert panels (focus groups); • Triangulation principle applied for avoiding subjectivity and partiality of the data as well as guaranteeing impartial conclusions.

8. RESULTS 1: INTERVENTION LOGIC 2007-2013 Higher value added in the economy Higher private R&D investments Higher R&D collaboration between public and private sectors Higher private sector R&D capacity and potential Higher public R&D potential and capacity Stronger clusters Better innovation support services More and better researchers in public sector Public R&D infrastructure quality and access to business Better private R&D infrastructure More researchers in business More business R&D projects Younger researchers More researchers Higher researchers mobility Better qualified researchers ESF ERDF sdafasdfasdfasdf

9. RESULTS 2: NIS ‘HEALTH’ ASSESSMENT • Market failure (productive sector) • Institutional failure (knowledge infrastructure) • Capabilities failure • Networking failure • Framework conditions • Governance failure • Demand side (absorptive capacity)

10. RESULTS 3: POLICY MIX ‘ROUTES’ Public private R&D collaboration - Strengthening public R&D system NGOsHEIS, PRIs Firms Investments in private R&D base Investments into productivity • Heavily expanding and versatile, but ‘linear’ logic persists’. • Mainly follows two routes: (1) to strengthen public R&D base, and (2) to invest in R&D in R&D performing firms. Lack of critical mass to implement some objectives placed high on political agenda (e.g. R&D collaboration). “Valleys”, national complex programmes € 678,6m (MoES) Clusters & innovation support services € 92.7m RTDI Networks € 6.23m • Direct support to companies, € 732.4m (MoE): • Access to capital (€ 415m ) • process inovelties (€118.2m) • E-business, investments into production technologies Researchers in business: € 9.3m R&D in business, € 162.2m sdafasdfasdfasdf Source: PPMI, Knowledge Economy Forum, 2010

11. ‘HORIZONTAL’ EVALUATION ~ 1000 indicators ‘VERTICAL’ EVALUATION ~ 150 indicators EVALUATION OF SF MONITORING SYSTEM • Quantitative (statistical analysis) as well as qualitative (logical models and consensus building activities). • SMART framework (specific, measurable, achievable, timed..)

13. KEY CONCLUSIONS • Structural gap: Lack of innovation absorptive capacity in business and society; limited local market: the key barrier to knowledge intensive firms. • Policy myopia 1: Excessive focus on supply side measures and on ‘supporting the winners’ can be contradictory to the systemiccharacteristics of NIS • Policy myopia 2: Quantitative targets will be met99 percent, but it does not mean achievement of qualitative objectives. • Risk-averse approach to implementation due to limited capacity to evaluate innovation projects. • Hypothesis: onlya minor part of economy benefits from innovation measures.Financially marginal “soft” measures are important for behavioral additionality: project pipeline building, innovation brokering

14. RECOMMENDATIONS FOR INNOVATIVE POLICY • Governance allowing quality ideas entering the ‘market’: • Boosting capability to develop RTDI policy, strengthening project and programme level intelligence; novel approaches to funding; a stronger involvement of users in evaluation and funding. • Policy as a discovery process: • Promoting innovative, risky, flexible, “bottom-up” approaches; project pipeline building. • Empowering people to innovate (‘bottom-up’), and demand side: procurement, regulation, clusters along the value chain, networks around societal problems.

15. STRENGTH AND LIMITS OF APPROACH • Strengths: Focus on the NIS bottlenecks as opposed to the mechanical transfer of policy models that may not be the most relevant for the NIS. Allows for internal coherence and looking beyond the quantitative input/output indicators. From macro to micro level analysis (focus on important details). • Limitations of qualitative approach: lack of ‘hard’ data and evidence (e.g. as opposed to counterfactual analysis) for tracing the real change and explaining obtained effects. Object for the following evaluations. • Recommendation for following evaluations:look for behavioural additionality (knowledge spillovers, changes in innovation process related behavioural patterns, interaction additionality, etc.), quantifying impact of networks

16. THANK YOU FOR ATTENTION! Agnė Paliokaitė Senior Policy Researcher Public Policy and Management Institute +37061690469 agne@vpvi.lt www.vpvi.lt