Evaluation Seminar Czech Republic CSF and OP Managing Authorities

1. Evaluation SeminarCzech Republic CSF and OP Managing Authorities David Hegarty NDP/CSF Evaluation Unit Ireland

2. Programme Structure: 6 January, 2005 • 5 presentations • Context and Organisation of Irish CSF Evaluation Process • Ongoing Evaluation • Mid-Term Evaluation • Final (Update) Evaluation • Ex-Ante Evaluation and Lisbon Strategy

3. Session 1 Context and Organisation of Irish CSF Evaluation Process

4. Presentation Structure • History • Evolution of system through CSF1 and CSF2 • The evaluation system now • NDP/CSF programme structure • Organisation and key features • Role of NDP/CSF Evaluation Unit • Role of other stakeholders • The evaluation market: who does the evaluations?

5. History • Objective 1 region 1989 to 1999 • 2 CSF programmes • EU funds = 1.7% of GDP • Focused on economic convergence and tackling unemployment • Evaluation context • Little prior tradition of evaluation • Low evalution capacity baseline

6. Evaluation Drivers • Evaluation impetus driven by • compliance considerations • EU Commission pressure and support • political priority attached to EU funds • Leading to …. • gradual creation of evaluation structures (CSF1) • major expansion in evaluation capacity and output (CSF2)

7. Milestones in Capacity Development • Establishment of 3 internal OP Evaluation Units • ESF (1992), Industry (1993), Agriculture (1995) • Outputs • Indicators and monitoring systems • Detailed measure level and thematic evaluations • Appointment of 6 OP external evaluators • Advisory and “watch-dog” functions • Mid-term evaluation • Steering committees with oversight role

8. Milestones in Capacity Development • Establishment of CSF Unit in 1996 • function: co-ordination of OP work and good practice promotion • Outputs: reviews of existing practice and guidelines • ESRI CSF mid-term evaluation 1997 • Innovative, tranferable methodology • Recommendations implemented • A success story

9. Taking Stock • System reviewed in 1998 • Lots of activity but … • Little understanding of evaluation purpose • Inconsistent evaluation focus • Quality of work variable • Poor follow-up/utilisation systems • Relatively expensive and inefficient • Lessons informed design of current structures

10. The Evaluation System Now • Total public investment in NDP/CSF • NDP € 51 billion • CSF € 5.6 billion • Programme Structure • Economic and Social Infrastructure (23.8) • Employment and Human Resources (14.2) • Productive Sector (5.8) • BMW region (3.0) • S&E region (4.1) • Peace (.127)

11. Key Features of Evaluation System • Evaluation system extended to NDP • Centralised system, 1 Evaluation Unit • But OP managing authorities fully involved • Extensive interim evaluation effort • Follow-up procedures in place

12. Agreed Evaluation Approach • Formally adopted in CSF • 5 key evaluation criteria • Rationale • Relevance • Effectiveness • Efficiency • Impact • Provides anchor for evaluation system • Promotes understanding of evaluation purpose • Underpins work programmes and terms of reference

13. Role of NDP/CSF Unit • Whats our job: • to help OP managing authorities with monitoring and evaluation • What do we do? • Development of performance indicators • Advice on project appraisal techniques • Drafting terms of reference • Commissioning evaluations • Doing evaluations!

14. Organisational Issues • Unit funded under Technical Assistance OP • CSF budget of €3.2 million (evaluation measure) • Report through CSF MA to monitoring committee • Monitoring committee decides work programme • MC composition • CSF MA (Chair) • OP managing authorities • Commission • Unit an advisory member of CSF and OP monitoring committees

15. Who are we? • Staff complement • 1 Senior Evaluator (Head of Unit) • 4 evaluators • 2 administrative • Evaluator staff recruited through open competition • 3 year contracts • Requirements • Degree in relevant area (economics, public administration) • 3 years experience (evaluation, research, policy analysis)

16. EU Commission Role • Member of Technical Asssistance MC • Role in work programme approval • Member of Evaluation Steering Committees (more later) • OP and CSF monitoring committee membership • An active participant in evaluation process

17. Managing Authority and Implementing Ministry Roles • OP managing authorities on Technical Assistance MC • Work programme • Also on evaluation steering committees • Responsible for submisison of evaluations to monitoring committees • decision paper on evaluation recommendations • follow-up on agreed recommendations • Implementing departments on steering and monitoring comittees

18. Who does the evaluations? • Small evaluation market in Ireland • Economic Consultancy firms • Specialise in OP level work • ESRI • Macroeconomic modelling capacity needed for CSF level evaluation • Limited involvement of foreign firms • Some evaluations done internally