1 / 20

Evaluating public RTD interventions: A performance audit perspective from the EU

Evaluating public RTD interventions: A performance audit perspective from the EU Hendrik Fehr, Gareth Roberts European Court of Auditors American Evaluation Society Portland, 3 November 2006. to discuss. Background the European Court of Auditors the European Union budget

fathi
Download Presentation

Evaluating public RTD interventions: A performance audit perspective from the EU

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluating public RTD interventions: A performance audit perspective from the EU Hendrik Fehr, Gareth Roberts European Court of Auditors American Evaluation Society Portland, 3 November 2006

  2. to discuss.... • Background • the European Court of Auditors • the European Union budget • What do we mean ‘performance audit’? • Current audit: The European Commission’s system for evaluating the impact of RTD interventions • why this audit, why now? • audit methodology • issues arising to date

  3. European Court of Auditors • Independent institution – external auditor of European Union • Reports sent to European Parliament and European Council of Ministers • 25 Members, based in Luxembourg, organised into 5 groups, covering: • agricultural policies • structural and internal policies • external actions • own resources, banking, admin • co-ordination, quality assurance • See www.eca.europa.eu

  4. European Union Budget (i)

  5. European Union Budget (ii) Research 6bn euro (approx)

  6. Framework Programmes for RTD (i) FP7 (avg 7bn pa) FP6 (avg 5bn pa) Run-in: developing new programme FP5 FP4 Shadow: implementation to the end 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

  7. Framework Programmes for RTD (iii) • directly managed by the European Commission • mainly extra-mural • multi-partner consortia (covering more than 1 EU Member State) • governed by contracts / grant agreements • several instruments / types of projects • networks of excellence • integrated projects • specific targeted research projects • specific support actions • etc

  8. What do we mean by ‘performance audit’? • Court’s mandate derived from treaty • provide a statement of assurance as to the reliability of the accounts and the legality and regularity of the underlying transactions, • examine whether the financial management has been sound • submit observations on specific questions and deliver opinions • Financial audits: • annual report / opinion • Performance audit: • management of research under FP5 • audit of effectiveness of instruments • audit of evaluation system • Opinions: • Rules of Participation for FP7

  9. What do we mean by ‘performance audit’? • According to the Court’s audit manual, a performance audit is • an audit of sound financial management, namely of the economy, efficiency and effectiveness. • to put it simply, examining • whether the right things are being done, and • whether things are being in the right way

  10. The focus of the current audit Overall audit question Does the European Commission have an adequate system for evaluating and assessing the impact of indirect RTD actions? Sub question 2: Are evaluation activities carried out in a comprehensive, coherent and independent manner and are relevant and useful results communicated on a timely basis, and are these results taken into consideration for future policy decisions? Sub question 1: Are there clear relationships between the policy objectives of the framework programmes and the criteria and indicators used to select, evaluate and assess the impact of RTD projects?

  11. Why this audit (now)? • because evaluation is relevant in the audit world..... • for transparency and accountability • for good governance and improving programme management • for informing policy decisions • assessed as an issue of particular risk because of... • the inherent difficulty in assessing the impact of public R&D interventions given the complexity of relationships between inputs, results and impacts, and, • the increasing levels of European Community funding • the concerns expressed in the Court’s earlier work about the information available about the outputs, outcomes and impacts of Community-funded research actions • to contribute to the development of an improved evaluation system for the future

  12. the audit methodology • Work focused on the Commission: • analysis of existing evaluation system • analysis of objectives and performance criteria • analysis of evaluation reports and • methodologies deployed • Work aimed at understanding good practice: • benchmarking visits to USA and Canada • literature review • stakeholder consultation within the Member • States • Work to validate credibility of findings • expert / stakeholder focus groups • contradictory procedure with the • Commission

  13. Issues arising from work to date....... legal and institutional framework intervention logic etc implementing RTD evaluation evaluation findings

  14. Legal and institutional framework • What are the main elements of evaluation governance? and what impact do these have?, eg: • Legal framework • Reporting results and link with budgetary processes • Internal evaluation and other standards • Quality assurance of the evaluation system • Who is responsible for monitoring the functioning of the evaluation system? • What influence do they have in practice? • Is a distinction between monitoring and evaluation

  15. Intervention logic etc • Is intervention logic clear from the outset? • Are objectives SMART? • How are performance indicators defined and targets set? How is performance against these targets monitored?

  16. Implementing RTD evaluation • How far is programme evaluation taken as a important priority by top management? • Is a robust evaluation strategy in place? If so, how well does it • consider how to deal with inherent complexity • explain intervention logic • take account of the role of evaluation in the policy cycle • strike a judicious balance between in-house capacities and external expertise • cover the portfolio of issues to be evaluated , program level challenges • deploy a mix of quantitative and qualitative methodologies etc • Are resources adequate for evaluation needs? • human and financial resources • data collection / IT systems

  17. Evaluation findings • What kind of information does the evaluation system produce? • To whom is it disseminated? • Do stakeholders consider it to be useful information?

  18. Comparing ‘performance audit’ and ‘evaluation’ • Similarities: • Methodologies and techniques • Differences: • Institutional context • Evaluee relationships • Reports Do you welcome such an initiative from an audit institution ?

  19. Contact details Gareth Roberts Auditor Tel: +352 4398 45724 gareth.roberts@eca.europa.eu Hendrik Fehr Head of Division Tel: +352 4398 45503 hendrik.fehr@eca.europa.eu Internal Policies (including Research) Division European Court of Auditors 12 rue Alcide de Gasperi L-1615 Luxembourg http://www.eca.europa.eu

More Related