1 / 22

Overview

Evaluation and monitoring of EC-funded research and innovation in ICT Carlos OLIVEIRA DG Information Society and Media (DG INFSO C3). Overview. 1) INFSO-C3: Evaluation and Monitoring Who we are 2) What we do ? Impact assessment Evaluation Monitoring 3) Plans for future activities

onella
Download Presentation

Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation and monitoring of EC-funded research and innovationin ICTCarlos OLIVEIRADG Information Society and Media(DG INFSO C3)

  2. Overview 1) INFSO-C3: Evaluation and Monitoring • Who we are 2) What we do? • Impact assessment • Evaluation • Monitoring 3) Plans for future activities • Data strategy • follow-up studies • …

  3. DG INFSO–C3Evaluation and Monitoring

  4. INFOSO C3 mandate: areas of responsibility Evaluation and monitoring Impact assessment Evaluation Monitoring Legislation FP7 - ICT Performance indicators Financial framework CIP - ICT/PSP Calls Strategy Legislation … Policy Other

  5. Why evaluate? • Better informed decision making • Better regulation and coherence with the Strategic Planning • Focus Upon Results and Transparency • Communication “Focus On Results: Strengthening Evaluation Of Commission Activities” SEC (2000)1051 - 26/07/2000 • Communication “Responding to Strategic Needs: Reinforcing the use of Evaluation” SEC(2007)0213

  6. Impact assessment Legislation Financial framework Strategy Policy Impact Assessment • Prospective evaluation (ex-ante) • The rationale for new initiatives • The choice of instruments • The expected impact • The monitoring instruments • Budgetary interventions: research and innovation programmes • Regulatory: roaming, universal services, public sector information • Other policy: ICT for energy efficiency, digitisation and digital preservation

  7. Evaluation FP7 - ICT CIP - ICT/PSP Legislation Other Evaluation Interim and ex-post evaluation • to which extent the objectives were achieved (effectiveness) • efficiency, • relevance, • coherence, • sustainability, • utility and/or community added value • contribution to broader strategic objectives

  8. Legal base Financial Regulation (art 27 – Sound and Financial Management, art. 60 Deelgation) FP7 legal base (art 7 of Decision 1982/2006 FP7 for research, tech. development and demonstration) Previous evaluations (“ex-post”) FP6-IST (Aho report) FP5-ICT (Gago report) Scope FP7-ICT ICT Theme of Cooperation art. 171 (JTIs) and 169 (AAL) Relationship FP7-ICT and eInfrastructures (FP7 Capacities) FP7-ICT Overall budget: 9 billion EUR over the full duration of the FP7 (2007-2013) FP7-ICT Interim Evaluation

  9. FP7-ICTICT theme(of Cooperation programme) • Specific objectives: • Improve competitiveness of European industry • Enable Europe to master and shapefuture developments in ICT • Strengthen Europe’s scientific and technology base • Ensure global leadership in ICT • Stimulate product, service and process innovationand creativity through ICT use • Ensure that ICT progress is rapidly transformedinto benefits for citizens, businesses, industry and governments • Help to reduce digital divide and social exclusion

  10. FP7-ICT Interim EvaluationEvaluation questions • Quality of research • Are we doing the right kind of research? • Are we mobilising the best people (Europe / worldwide)? • Progress towards objectives • ERA? Innovation? Knowledge triangle? • EU policy goals? • Quality of implementation; simplification • WP formulation? Mix of instruments? • Rules of participation? • Trust-based relationship?

  11. Sources of evidence • FP7-ICT Self-assessment • Challenges (technology, societal) • Survey + interviews of participants • Nature of the projects, • Perceptions and expectations of participants • Issues / barriers • Specific studies • PREDICT (EU R&D in ICT sector) • Portfolio of participants • Non-retained proposals • Patents and publications • Indicators of administrative efficiency (ttc, ttp)

  12. Monitoring Monitoring Monitoring – formative assessment • how are we progressing towards the objectives • Effectiveness and efficiency of specific processes Examples: • Integrated Programme Participation Analysis • Publications and patents of EU-funded research in ICT • Network analysis • Monitoring of calls for proposals Performance indicators Calls …

  13. IPPA reportIntegrated Programme Portfolio Analysis • Top beneficiaries by group (HE, research, industry, SMEs) • Analysis by challenge (i.e. research area) • Geographic breakdown (inc. EU and 3rd countries) • Analysis per instrument (collaborative projects IPs and STREPs), NoEs, CSAs

  14. IPPA reportIntegrated Programme Portfolio Analysis

  15. StudyPerformance Indicators • assess output performance of EU funded research in ICT using scientometric and bibliometric techniques • N° and relevance of publications (scientific articles, conference papers) • N° Patents • Extent of cooperation (across countries, regions, organisations) • Coverage 2005-2007 (FP5, FP6, part of FP7) Who: • KITeS Universitá Luigi Bocconi / Prof. Stefano Breschi • data collection • data cleaning and integration • benchmarking methodology

  16. Study Performance Indicatorsdata integration: publications

  17. Study Performance Indicatorsdata collection - patents

  18. What’s next … Lessons learned • Systematic data collection (completeness, accuracy) • Improved methodological approach • A more comprehensive set of indicators …

  19. Ongoing reflectiondata strategy for evaluation • Policy goals vs operational objectives => intervention logic • Top-down vs bottom-up (macro-economic sectoral analysis, portfolio project analysis, etc.) • Efficiency, effectiveness, critical mass? What indicators? • Causality? (eg research vs. innovation) • Targeted case-studies (not success stories) • Data collection / analysis / publication

  20. Further output analysis Other concrete project outcomes • Specifications / reference models • Software implementations Human capital • PhD, MSc students • Participants in doctoral schools Standards • industry initiatives leveraged by FP participation Project operations • resources devoted to specific activities (management, training, dissemination) • outcome of annual reviews (eg “continuation”, suspension, “red-flag”) • Administrative efficiency (eg cost of reviews)

  21. Thank you Contact: DG Information Society (and Media) INFSO-C3 Evaluation and Monitoring Unit Head of Unit: Anna Krzyzanowska Carlos Oliveira (Carlos.Oliveira@ec.europa.eu)

  22. Europa web site

More Related