Rupert Pichler Wolfgang Hein Federal Ministry for Transport, Innovation and Technology

1. After Evaluation:Policy Consequences from Evaluations of transinstitutional RTD Funding in Austria Rupert Pichler Wolfgang Hein Federal Ministry for Transport, Innovation and Technology

2. Content • Context • Evaluation of FWF and FFF • Assessment of Competence Centre Programmes (CCPs) • matching political motivation and ToRs • findings, recommendations, political consequences and role of external players • what remains of the evaluations and why Intervention + Context = Impact (Pawson/Tilley, 1997)

3. Context • major evaluations of Austrian RTD system completed • changing needs and challenges • key factor of changes is spread of multi-actor programmes and PPP models through funding institutions • appropriateness of institutional and regulatory framework as underlying key issue • How are these factors reflected in terms of reference? • Do evaluators get into the feed back loops of policy-making? • How much of the results gets inevitably disposed of? • What prevails in the end, and why?

4. FWF/ FFF: Political Motivation • design of the two largest funding institutions stemming from the 1960s, remaining virtually unchanged • obviously no incentive for substantial change by new paradigms of the 1980s and 90s (collaborative technology programmes, EU FP) • political pressure for simplification of funding structures added to the funds since the late 80s • no evaluation ever carried out in the funds’ history, respective recommendation of R&D council • at the same time growing number of evaluations of programmes implemented outside the funds

5. FWF/FFF: Reflection in ToRs • evaluation commissioned by BMVIT alone, though reform efforts imminent at that time are supported by a multitude of players • BMVIT under pressure to justify efficiency and effectiveness of existing funding institutions, thus trying to safeguard its influence • at the same time examination by court of auditors pending which is clearly understood as to have role in decision making • positioning of the funds within a future scenario clearly addressed in ToRs • design of ToRs outsourced to ISI in light of political weight and financial dimension • contractors made aware of “killing fields”

6. FWF/FFF: ToRs – Central Issues • Are the instruments, procedures and structures adopted by the funds – according to their mission – appropriate to support the investigational and innovational behaviour of the relevant actors in an efficent and effective way? • What is the position of the two funds in the national, international and especially European science and innovation system and what recommendations can be made for future strategies?

7. FWF/FFF: ToRs – specific questions 1/2 • General Approach Are the objectives and the strategic orientation of the funds appropriate to pursue the intended effects of research funds? What is the impact of the funds activities on the corresponding science system and industrial RTDI? How are the funds positioned in comparison with corresponding international funds? • Performance/ Management Do the funds employ appopriate procedures to secure the quality of the supported project? What are the strengths and weaknesses in the performance of the funds?

8. FWF/FFF: ToRs – specific questions 2/2 • Cooperation Is the coordination of the funds with other national research and funding instruments and institutions suitable to realise feasible synergies? How do the funds coordinate, cooperate and communicate with the relevant actors of the science and innovation system, especially with each other and the corresponding ministries? • Internationalisation What are the strategies of the funds to secure their positioning and integration in the European research and innovation system?

9. FWF/FFF: Findings 1/3 – political context • beside in-depth analysis synthesis report tackles policy related questions at length based on the detailed results • however, intense integration into decision-making process achieved rather coincidentally while external expertise from court of auditors competes for political attention • involvement of evaluators into discussions with reform stakeholders paves the way to substantial influence • nonetheless, for a longer time during the process some actors remain unsure about the exercise’s political usefulness • multitude of stakeholders makes it easier for evaluators to become a voice that is heard

10. FWF/FFF: Findings 2/3 – major issues • merger of the two funds as advocated by substantial stakeholders not recommended, to the contrary merger of FFF with TIG, ASA, and BIT • organical expansion of FWF’s role to tackle use-oriented and thematic research; a substantial budget increase to cope with this progress and higher overheads on research grants in future • both funds require additional resources to generate strategic intelligence and their own strategies, strengthen their international roles and improve their communication within the Austrian funding system

11. FWF/FFF: Findings 3/3 – major issues • the governance of both funds was criticised as being close to self-serving with too little strategic capacities, leading to a role model fairly isolated within the system • the funds should be transformed into agencies and the power of their beneficiaries in the governance structures should be limited • the instruments of both funds need to be improved (more risk, more networks, simplification) • the strategic intelligence of both funds was found to weak in order to cope with implementation of the government’s strategic goals • surprisingly little windfall benefits of FFF funding

12. FWF/FFF: Recommendations 1/2 • business of ministries is to make policy, and of agencies to implement it – each needs to develop strategies at their own level • the division of labour between ministries and agencies needs to be transparent and modern • establishing clear performance contracts between the ministries and agencies in terms of objectives and how and when they are measured • correspondingly, it means absolutely forbidding interference from the policy or political level in operational matters such as personnel and project decisions

13. FWF/FFF: Recommendations 2/2 • ministries need to retain enough analytic capability to support their policy needs • responding to the policy needs of the ministries the agencies should propose, design and implement programmes and other instruments in the long run • agencies should be able to develop strategies to obtain economies of scope as well as scale, tackling policy needs of multiple ministries • beneficiaries must be important in the governance of agencies, but not have overall control, which should be exercised by a wider range of stakeholders on behalf of the taxpayers

14. FWF/FFF: Political consequences • the merger of the two funds was taken off the agenda after the evaluation had supplied ammunition to the opponents of the merger • the FWF’s governing institutions underwent reform in order to exclude beneficiaries and institutional stakeholders from decision-making • FFF was merged into new FFG (intended anyway) which is a “normal” state-owned company; FFF lost its autonomy • in practice, FFF mechanisms still exist as a part of FFG while a bigger debate over ex-FFF is on hold since it touches sensitive ground • while FFG struggles to become a one-stop-shop part of the old system survives for the sake of continuity

15. FWF/FFG: Remains • eventually, the evaluation’s expertise prevails over the court of auditors • beyond, a general willingness to devote attention to non-consultant style advice became obvious • the in-depth parts still hold treasures yet to be made use of • a surprisingly comprehensive debate on role of politics evolved over the issues of “autonomy” and “self-governance” and led to partial fulfillment of ambitions to regain political influence as a consequence of political responsibility • hypothesis: great visibility of an evaluation exercise can be achieved when a multitude of equal stakeholders is involved, none of which being in full possession of the process but each trying to get attention for own interests; this makes it easier for evaluators to play a role in decision-making as long as equal distance is kept to all parties

16. CCPs: Political Motivation 1/2 • substantial dimensions and visibility of competence centre programmes (aimed at bridging the gap between industrial and basic research) reached in 2003 • two co-existing programmes under responsibility of two ministries • funding terms of first centres nearing the end • widespread discontent with “competition” among ministries • strikingly different implementation

17. CCPs: Political Motivation 2/2 • development of perspectives obviously necessary • justification of future investment • though commissioned jointly by BMVIT and BMWA hidden agenda develops • common interest: resist outside pressure to merge and lose identity • concealed expectations to emerge as “winner” of the evaluator’s judgement and therefore secure a favourable position for upcoming battles

18. CCPs: Reflection in ToRs 1/3

19. CCPs: Reflection in ToRs 2/3 • General approach Does the conception of the programme correspond to the problem analysis? Is there missing cooperation between industry and science in research and innovation projects? Does the programme succeed in achieving its objectives? What is the impact of the funding activities on the corresponding science system and industrial R&D? • Management/ Performance Is the implementation of the programme efficent? Are the programmes´ instruments, procedures and structures appropriate to the objectives of the programmes and the needs to be funded?

20. CCPs: Reflection in ToRs 3/3 Perspectives of the CCPs • curiosity whether or not an empirical justification for two different programmes could be determined • vested interests to provoke a quality ranking not expressed in ToRs, though obvious to the evaluators Perspectives of existing CCs • pressing need for ideas about the CCs’ future thoroughly reflected • Will the centres continue to carry out long-term precompetitive research and development of high academic and economic relevance? • Should the centres be restructured and if so in which way?

21. nature and dimension of demand for CCPs remains to be determined both programmes address existing specific needs implementation is largely consistent with goals, but room for improvement K plus should remain science orientated; appropriate current funding budget; K ind/net should be distinguished as innovation-orientated programme; decrease of the funding rate; more transparency of the funding criteria; aspiration for cooperation both programmes should provide incentives to create synergies between the different CCs; cooperation is limited and programme management should enhance networking activities CCPs: Findings 1/2

22. CCPs: Findings 2/2 four models for the CCs’ future were specified • privatisation • termination • reinstallation in future CCP calls • institution building • common platform with public research organisations • close cooperation between the CCs and universities • establishing stand alones, possibly through mergers of CCs

23. CCPs: Role of other players • decision-making on CCPs exceeds the realm of those commissioning the assessment (BMVIT and BMWA) • some interests underestimated (Länder, pressure for institutionalization, R&D council, universities) • occasionally open criticism is voiced (e.g. concerning comments on implementation standards) • at the same time, a soft-spoken OECD study is at hand

24. CCPs: Political consequences • great relief, no pressure from expert opinions to end seemingly mere political duplication of programmes • it is not believed in earnest that the policy field as such is at stake • the models are immediately fed into ongoing negotiations with other stake holders by BMVIT • immediately after completion: assessment as valuable input for negotiations with Länder, universities and industry • one year later: design process for CC future (mainly in BMVIT domain) about to disintegrate, thus reducing the assessment’s immediate impact • two years later: unexpectedly, a “grand” solution is underway and a short-term financial aid for outgoing CCs agreed; some of the assement’s implications are turned upside down

25. CCPs: Remains • commissioning policy makers never in full possession of the decision process • awareness of that process and close involvement of evaluators enhances the exercise’s weight even though by doing so some standard rules are ignored • general awareness for real need for strategic decisions beyond mere political interests significantly increased • input on implementation valid and recognized • struggle over financial questions concerning the future of existing CCs triggers agreement to pursue a complete and even more ambitious relaunch which was not considered an option at the time of the assessment because of apparent priority of short-term needs • hypothesis: the more relevant an issue evaluated and the more honest the interest in the exercise, the greater the probability that findings – even if concrete recommendations were asked for – serve only as “germs” for further process, thus eventually achieving powerful effects