11 th of June 2012

1. Eu-SPRI Conference 2012 Fraunhofer ISI, Karlsruhe The use of indicators and evidence in governance and policy development of Science, Technology and Innovation Manuel Laranja - Instituto Superior de Economia e Gestão / Universidade Técnica de Lisboa mlaranja@iseg.utl.pt Nuno Boavida - Faculdade de Ciências e Tecnologia/ Universidade Nova de Lisboa nuno.boavida@campus.fct.unl.pt 11th of June 2012

2. Contends • Introduction • Indicators • The use ofevidence in Policy making • Proposition • From pure evaluative to mixed or combined approaches • The use of evidence and governance • Examples • Next steps

3. The use of indicators and evidence in governance and policy development of Science, Technology and Innovation Introduction Is there a renewed interest in evidence-based policy ? • Increasing large sums of public funding to R&D at European and National levels • Easiness of access to data – statistical indicators • The relatively poor performance of Europe in turning research into innovation (EC, 2001; Kok, 2004; Aho, 2006) • Growing use and interest from policy advisors, consultants, foresight and TA reporting • Growing interest on what works, rather than policy rationales informed by systemic theories of knowledge and technical change • And more generally, a new “utilitarian” view of European STI policy.

4. The use of indicators and evidence in governance and policy development of Science, Technology and Innovation Indicators(1/2) • We now have more indicators than ever before… • Not just R&D indicators but also “blue sky” innovation measurements e.g. CIS surveys • Also broader approached through multiple scoreboards and composite/indexes • PRO INNO Europe and ERAWatch • IUS – Innovation Union Scoreboard (EIS – European Innovation Scoreboard) • OECD STI scoreboard

5. The use of indicators and evidence in governance and policy development of Science, Technology and Innovation Indicators2/2 • Growing scientific and technical discussion around the purpose and methodologies used to gather data and build STI indicators(Godin, 2008; OECD, 2008; Grupp and Mogee, 2004; Barré, 2004). • STI indicators were not subject to extensive research, and may present confidence, comparability and overlapping problems (Feller-Länzlingeret al, 2010; Grupp and Schubert, 2010; Godin, 2008; Barré, 2004 ) • But surprisingly, in our view, • discussion on how are indicators used for policy making decisions (within the policy development process) is less frequent.

6. The use of indicators and evidence in governance and policy development of Science, Technology and Innovation The use ofevidence in Policy-making • Indicators can be subject to decontextualized interpretation, simplification by the media and political appropriation (Feller-Länzlingeret al. 2010). • An interesting approach, proposed by Perry 6 (2002), suggests that the situations in which policy makers find themselves, will shape which information from the complex set available is used and, most importantly, which information is rejected or at least downplayed. • Perry 6 (2002, p.7) argues that policy-making “always makes use of some evidence, but that there is a plurality – a limited plurality, indeed – of things that count as evidence, and what counts depends on where policy makers are situated”.

7. The use of indicators and evidence in governance and policy development of Science, Technology and Innovation Proposition • An exploratory framework that sets out two conjectures that may help to conceptualize the way in which evidence and indicators are used in STI policy decision making • i.e. how policy-makers look for, interpret and use evidence for reflection and policy development ?

8. The use of indicators and evidence in governance and policy development of Science, Technology and Innovation First: From pure evaluative to mixed or combined approaches • The use of evidence and indicators by STI policy-makers is a combination of a pure evaluative approach with ideas, conciliation of interests and considerations of popular support and acceptability of key constituencies. • That is: the use of evidence varies between a pure evaluative approach based on technically sound metrics and a combined approach.

9. The use of indicators and evidence in governance and policy development of Science, Technology and Innovation Second: the use of evidence and governance (1/2) • Another possible direction for research is whether the use of evidence and indicators is associated to styles of governance. Rigid hierarchical vs Flexible Network Governance • The more rigid hierarchical forms of governance, tend to assume that indicators inform decisions in a somewhat linear and mechanistic manner. The assumption is that more precise and higher quality indicators will always positively influence policy decision-makers.

10. The use of indicators and evidence in governance and policy development of Science, Technology and Innovation Second: the use of evidence and governance (2/2) • By contrast in the new multi-actor networked forms of governance, indicators must serve the ‘steering’ of a complex system of interactions requiring multiple information flows directed to different user segments, each with its own political agenda. • Can therefore Network forms of governance lead to a better set of consensual indicators compared with that obtained through centralised governance forms ?

11. Examples(1/2) • Portuguese TechnologyPlan 2005 -2009. Combinedapproachthatused EIS to steer a network governance • SIMPLEX 2008. CombinedapproachthatusedtheDoing Business rank – World Bank for impressionistic propaganda (and over simplification) • Evidence is amplified/rejected/downplayed to suit political intent. • Evidence becomes “policy-resonant” (Hezri and Dovers, 2006) i.e. they “strike a chord” with its intended target audience, hence becoming easier to communicate and often appropriated by policy-makers and by the media.

12. Examples(2/2) • Mobi.E– Charging Network for ElectricCars. Ideadriven. Evidenceused to suitthepolicyagenda. • NationalActionPlan for EnergyEficiency –PNAEE 2008. Theuse ofevidenceclose to a pureevaluativeapproach • The use of evidence specialised agencies or independent evaluators may be closer to a pure evaluative approach i.e. with the objective of building technically coherent measurement within their proper context sets upon to surmount technically sound advice.

13. The use of indicators and evidence in governance and policy development of Science, Technology and Innovation Centralized, Hierarchical mechanisms Propaganda Mobi.E Oversimplification SIMPLEX 2008 PANEE 2008 ADENE Pure evaluative, values technical evidence Combined mixed approaches Policy resonant Strikes a chord Steering Technological Plan 2005- 2009 Collective, multi-actor, multi-level, participative consensus

14. Next Steps • Gather more information on how are indicators (and other evidences) being used for policy development, through interviews to decision makers involved in STI policies, in Portugal and Germany

15. The use of indicators and evidence in governance and policy development of Science, Technology and Innovation Thank you Manuel Laranja mlaranja@iseg.utl.pt Nuno Boavida nuno.boavida@campus.fct.unl.pt