Presentation

1. Evaluating the societal impacts ofPublic research organisations:A (belated) paradigm shift in the making Kaisa Lähteenmäki-Smith and Kirsi Hyytinen VTT Technology Studies

2. Presentation • The project on societal impact evaluation: research questions, methods, participating organisations and their differentiated working environments, steering processes and R&D ambitions • Evaluation model for societal impacts of publicly funded R&D organisations • Some key findings and conclusions and their implications for the evaluation model

3. Description of the project: questions • How are publicly funded R&D organisations adjusting their evaluation practices to the emerging ‘management by results’-based accountability demands set by the government? • How can different evaluation practices and methods be used for this purpose in assessing the societal impacts of R&D activities? • What are the similarities and differences, as well as lessons to be drawn between policy sectors (best practices, challenges, bottlenecks…)? • What are the implications of impact assessment on administrative practices, including steering processes, in respect of governmental actors and public research organisations?

4. Description of the project: organisations • MULTI-SECTOR TECHNOLOGY R&D INSTITUTE (VTT) is national research institute in technology working in close interface with both the business sector and policy-making. • DEFENCE SECTOR R&D INSTITUTE: The Finnish Defence Forces Technical Research Centre • RESEARCH INSTITUTE IN THE AGRICULTURE AND FOOD SECTOR, MTT Agrifood Research Finland • TWO POLYTECHNICS – one WITH ITS SUBSTANTIVE FOCUS IN THE AREA OF BUSINESS MANAGEMENT, ANOTHER MULTIDISCIPLINARY POLYTECHNIC

5. Motivations of impact assessment Internal and external motives : Change in management and administrative culture Demand for the accountability and result-driven planning Spread of a service-oriented culture – customer orientation Demand for transparency and accountability Compatibility and integration of policy objectives Selection of sectoral objectives and consolidation of sector objectives with inter-connected sectors Selection and compatibility of horizontal goals Demand for organisational development Identification of demands and multiple development challenges Supporting strategic planning – contributes to steering and priority-setting, allows for operational freedom in developing core competences and activities Prioritizing the activities

6. National approach to accountability • National and international dimensions of effectiveness • Customer-orientation: are the needs of the clients met? • Future orientation: are the visions and strategic targets set meeting the needs of the changing nature of society and of the policy environment? • Cost effectiveness • Socio-economic and societal dimension: Have the main external costs and effects been taken into consideration?

7. THE EMERGENT FRAMEWORK OF IMPACT EVALUATION

8. Key concepts 1/3- output, outcome, impact, effectiveness • The preservation of healthy oat grain compounds in the baking process • The healthy food product • Lower cholesterol levels • Improved national health • longer life-expectancy • economic effects Output Outcome Impact Effectiveness

9. Key concepts 2/3- impact typology IMPACTS ANTICIPATED UNANTICIPATED Kohdealueen ulkopuolella Outside target area Outside target area In target area In target area R&D impacts R&D impacts T&k- vaikutukset Useful Useful Other impacts Other impacts Detrimental Detrimental ( Mickwitz 2004)

10. Key concepts 3/3-impacts through time IMPACTS • IMMEDIATE • •networking • improved R&D efficiency • patent applications • publications • prototypes • INTERMEDIATE • •partnership-based cooperation • new/ improved products, services and processes • company growth • improved company competitiveness • cost savings • higher employment • strengthened expertise • technology transfer • standards, norms • support for decision making • public discourse • ULTIMATE • • improved industry competitiveness • higher investment • better safety • improved quality of life • promotion of regional development • improved awareness Benefits to economy and society Benefits to project participants -1 0 1 2 4 6 8 10 15 YEARS, OR MORE (Tassey 2003)

11. Dimensions of impacts and final beneficiaries • Economy, technology and commercialisation • Knowledge, expertise and human capital • Networking and social capital • Decision making and public discourse • Social and physical environment Own organisation Customer Society Public R&D-organisation

12. Evaluation model: some criteria • Relevance • Quantifiable nature • Reliability • Easy availability • Dynamics of change and trends • Strategic feedback loop

13. How to get information about impacts and effectiveness through evaluation • Stakeholder and client ’inputs’ into evaluation cycle • Follow-up studies • Client satisfaction surveys • Interactive discussion fora Preparation phase Ex ante evaluation Ex post evaluation Execution phase Concluding phase Mid-term evaluation

14. Impact assessment as a learning and steering tool Transparency, permanence, flexibility, predictability, legitimacy, justness Rationales for public support MOTIVES Market failure Transaction costs Risks and uncertainty Policy statements OBJECTIVES Public safety New enterprises and workplaces Welfare services Policy implementation ORGANIZATIONAL CAPACITY as the means (for doing ’the rights things’) Using the knowledge about changes amongst the stakeholders ADDED VALUE OF R&D To the organisation To the customer For society Definition of the new rationales Allocation of the resources Prioritizing Definition on the strategy Feedback