Proposed Organisation of Evaluation of the Romanian NSRF and Operational Programmes, 2007-2013

1. Proposed Organisation of Evaluation of the Romanian NSRF and Operational Programmes, 2007-2013 Niall McCann, Technical Assistance Project for Programming, Monitoring and Evaluation, MACSF

2. Basis for proposed model? • The Evaluation Standards, to be approved by the EWG • The latest draft EU Regulation on organisation of the management of Structural Instruments (21 Dec, 2005 Presidency Compromise proposal to the Commission’s 2004 “Proposal for a Council Regulation Laying Down General Provisions on the European Regional Development Fund, the European Social Fund and the Cohesion Fund,” [COM(2004)492 final], Articles 45-47.

3. Proposed evaluation model • The Ops will be subject to two types of evaluation: • 1. Operational • Interim • Ad hoc • 2. Strategic • Thematic • Strategic

4. 1. Operational Evaluations • 1.1 Two Interimevaluations will be commissioned for each OP: • 2009 – evaluation will concentrate on implementation progress to date • 2012 – will look towards the next programming period.

5. 1. Operational Evaluations • 1.2 Ad hoc evaluations will be triggered at Priority or Key Area of Intervention level, wherever: • monitoring data reveals a departure from goals initially set • a proposal is made to change the Operational Programme. • a request is made by the Monitoring Committee; • a request is made by the Evaluation Working Group.

6. 2. Strategic Evaluations • 2.1 Thematic evaluations will analyse a theme [e.g. an OP priority theme (“R&D”), a horizontal principle (“equal opportunities”), a management issue (“procurement”) in the context of several interventions within the Operational Programme. • 2.2 Strategic evaluations will examine the evolution of groups of programmes in relation to Community and national priorities.

7. Who will commission and perform the evaluations? • 1. Operational evaluations • Interim – commissioned by each MA, conducted externally, • Ad hoc – commissioned by each MA, conducted internally? • 2. Strategic evaluations • Thematic – commissioned by each MA, conducted externally • Strategic – commissioned by the MACSF ECU, conducted externally?

8. Evaluation Terms of Reference • Terms of Reference should be drafted for each evaluation, internal or external. • The template Terms of Reference for the OP interim evaluations as well as for the thematic evaluations will be drafted by the MACSF Evaluation Central Unit, following discussion within the Evaluation Working Group and delivered to each Managing Authority, who will adapt the Terms of Reference to their own OP.

9. Evaluation Steering Committees • Each OP will have its own Evaluation Steering Committee. • The Committee will be convened at the start of each evaluation process. • The core membership of the Evaluation Steering Committee will remain the same for each evaluation, and will, at the very least, include members from the Evaluation Central Unit, the head of the evaluation function in each MA, and the head of the Managing Authority.

10. Staffing of the evaluation function in the MACSF and the Managing Authorities • The MACSF and the Managing Authorities should ensure that an adequate number of staff is assigned to the evaluation function. • Full-time staffing of the MACSF ECU will be increased in 2006. • The Managing Authorities’ evaluation function should be assigned to a stand-alone, MA evaluation unit, ensuring that the unit does not have other tasks related to audit, monitoring, etc. • Decisions on whether to assign staff full-time to the MA evaluation unit can be taken later in 2006. • When selecting staff, MAs should ensure that: • Staff possess an adequate level of experience required; • Staff are available to attend all stages of the proposed MA training plan; • Job descriptions of the MA evaluation staff clearly state all the responsibilities that managing the MA evaluation function entails.

11. Evaluation Resourcing • The MACSF Evaluation Central Unit evaluation tasks shall be financed from the Technical Assistance OP. • MAs for the other 7 OPs shall ensure that adequate resources are provided for management of the evaluation function. • The OP Technical Assistance Priorities shall fund not only the evaluations to be commissioned, but also other evaluation related costs, such as participation of staff in conferences, etc. • Evaluation should become a stand-alone measure in the OP Technical Assistance Priorities.

12. Evaluation Procedures General Principles • The evaluation process will be based on 3 principles: • 1. A Multi-Annual Evaluation Programme; • 2. Operational procedures for each evaluation; • 3. Follow-up procedures to implement evaluation recommendations.

13. 1. A Multi-Annual Evaluation Programme • A MAEP for each OP will be proposed by the MA and agreed upon with the MACSF Evaluation Central Unit, on the basis of a template prepared by the ECU. • The MACSF ECU will propose a national MAEP on the basis of all the MEAPs received from the MAs. • The MAEPs should identify the number and types of the evaluations to be carried out as well as the themes of thematic evaluation reports.

14. 2. Operational Procedures for each evaluation • The MACSF will develop a Manual of Evaluation Procedures • They will deal with: • Designing and Approval of the Multi-Annual Evaluation Programme • Establishing the Evaluation Steering Committee • Designing ToRs and selection of the evaluator • Organising the Kick-off meeting • Drafting/Revision of First evaluation draft report • Approval of Final draft: the Evaluation Report • Publication of the report

15. 3. Follow-up of Evaluation Findings and Recommendations • All evaluation reports will include a recommendations table. • The table will describe the actions to be taken by each stakeholder to implement the recommendations. • The Monitoring Committee will decide whether to accept or reject each recommendation. • For recommendations approved by the Monitoring Committee: • a debriefing meeting of the Evaluation Steering Committee will focus on the means and the timing of implementing the recommendations. • the MA evaluation unit will then send the follow-up table to the relevant institutions (including DG Regio). • At the next Monitoring Committee meeting, the table will be examined and the relevant implementing authorities will report on their progress.

16. Main role of the MA evaluation function • Manage the evaluation activity for the OP, under the guidance of the MACSF ECU • Draft an OP Multi-Annual Evaluation Programme, to be updated annually • Participate in capacity building activities organised by the MACSF ECU • Propose to the OP Monitoring Committee and the MACSF ECU ad-hoc evaluations • Draft the Terms of References for and conduct ad hoc evaluations? • Convene, chair and act as Secretariat to the OP Evaluation Steering Committee

17. Main role of the MACSF Evaluation Central Unit • Manage the evaluation activity for the NSRF as a whole • Organise evaluation capacity building activity, in the form of training seminars, drafting guidelines, determining procedures, etc., for MA evaluation units and for itself • Convene, chair and call meetings of the Evaluation Working Group • Finalise the NSRF Multi-Annual Evaluation Programme, on the basis of the MA MAEPs submitted by the MA evaluation units • Guide, assist and coordinate the work of the MA evaluation units

18. Main role of the MACSF ECU (contd.) • Draft the template for the Terms of References for the OP interim and thematic evaluations, to be provided to the MA evaluation units • Draft the Terms of References and manage strategic evaluations at the NSRF level • Attend meetings of the OP Evaluation Steering Committees as a member, and advise the Committees on evaluator selection, quality controlling of evaluation reports, methodological issues, etc.