The Making of the FP6 Ex-post Evaluation

1. The Making of the FP6 Ex-post Evaluation Erik Arnold American Evaluation Association, Orlando 12 November 2009

2. The expert group … don’t blame them for what I say! • Ernst Th. Rietschel (Germany) - Chairman • Erik Arnold (United Kingdom) - Rapporteur • Antanas Čenys (Lithuania) • Andrew Dearing (United Kingdom) • Irwin Feller (United States of America) • Sylvie Joussaume (France) • Aris Kaloudis (Greece/Norway) • Lene Lange (Denmark) • Jerzy Langer (Poland) • Victoria Ley (Spain) • Riitta Mustonen (Finland) • Derek Pooley (United Kingdom) • Nicoletta Stame (Italy)

3. What’s changed in Framework Programme goals? • From FP5 “A successful and scientifically strong European industry and high quality of life for citizens” • To FP6 as an instrument for reaching Lisbon, Barcelona, ERA - introduction of (shifting) policy goals

4. Structure of the Framework Programme (€20bn 2002-6) • 6FP for RTD and Demonstration (93%) • Focusing and integrating Community research (76%) • Thematic priorities (65%) • Specific activities covering a wider field of research (7%) • Non-nuclear activities of the JRC (4%) • Structuring the ERA (15%) • Strengthening the foundations of the ERA (2%) • EURATOM (7%) • Priority thematic areas (5%) • Other activities in the field of nuclear technologies & safety (0.3%) • Nuclear activities of the JRC (2%)

5. The evaluation • The first evaluation of a single FP (after 3 ‘Five-Year Assessments’: Davignon; Majo; Ormala) • The most solidly evidence-based evaluation of the FP yet undertaken • Hence, broke the mould in moving from high-level discussion to an analysis with empirical ‘bottom’ • Evaluation issues • Rationale • Implementation • Achievements • Recommendations • + Vision

6. The Commission’s FP6 evaluation has focused on the new • New member states • New instruments (In toto, NoEs, ERANETs x2, ETPs, OMC …) • Activities of DG-ENTR • Behavioural additionality • International standing of FP6 (+China + Bilaterals) • Bibliometric profiling • Network formation in FP6 • Gender equality • Ethical review • Plus the traditional ‘participation survey’ across the board

7. The Member States and others’ evaluations • Continuation of traditional impact studies in some countries • Those who need evaluation the most do the least … • From snapshot to video: Sweden and the ‘Gang of 4’ (Sweden, France, Spain, Czech Republic) • Associated States: Norway, (Switzerland) • ‘Third Countries’: China

8. Achievements of FP6 • High volume of R&D at high quality • Quality-assured assessment processes • Participation by excellent researchers • Thematic priorities – two thirds of the FP; appear productive but evidence is general, unsegmented – we’d like to know more • EURATOM – captured the ITER global fusion facility. Fission? • Indirect but positive evidence on industrial competitiveness • Lack of institutional strategies has limited the FP’s strategic influence over the Knowledge Infrastructure • No ‘great leap forward’ in ‘structuring the ERA’– probably because we haven’t fully agreed what the ERA actually is • Integrated the New Member States • Gender, Joint Research Centre – unfinished business

9. Expert group recommendations • Better, more transparent design • Bigger role for the MS; avoid the tendency towards a Commission monopoly of initiative (eg ERC) • Develop focused strategies for ‘Third’ countries: OECD; BRICS, poor countries • Add bottom-up experimentation (compare NEST) • SMEs matter but stop the silly targeting of lower-capability firms • More research infrastructure; more ESFRI • More women • More young people and mobility • Radical overhaul of administration - no more tinkering • Broader evaluation approach - we know more but far from enough

10. Vision • A new, confident and outgoing Europe • A proactive partner in the global knowledge society • Much stronger global links and collaboration • Top down and bottom up combined • Grand Challenges • Great Ideas

11. Self-criticism: not enough policy or process • No discussion of the instrument mix at EU level • Research institutes omitted from the picture • No discussion of the division of labour between the Member States and the Framework • Did not take on the changed character of the ‘self-organised’ instruments or their implications for future R&D policy • In many places, failed adequately to differentiate among the instruments – too much momentum from the old Framework evaluation tradition • Inadequate treatment of the FP design process and the influence of internal incentives in the Commission (not entirely our fault … )

12. Evaluation, the Commission and the FP • Massive changes in extent and visibility of evaluation since the 1999/2000 reforms • Early signs of an ‘evaluation culture’ in places • Use of programme theory an improvement over the 1990s • Input - Output - Results - Impacts a useful corrective to the old “we can’t measure the impacts so let’s talk about the programme management” approach • Is there a risk of losing sight of the processes and the political science? • Still opportunities to improve the link between evaluation and the evolution of the FP

13. Thanks for listening erik.arnold@technopolis-group.com