Evolution of Uval's work program: from support to evaluation

1. Evolution of Uval's work program: from support to evaluation Italian Evaluation Unit Open Days 3 – 5 July 2006 Laura Raimondo

2. Until April 2006 Ministry of Economy and Finance

3. Department for Development Policies CSF MA

4. UVAL’s tasks • UVAL founded in 1982 • Reformed in 1998 • Re-reformed in 2002 • Main tasks: • Promotion of data, information and knowledge for public policies • Support to administrations to modernize and innovate public action, • Evaluation of programs and capacity building for evaluation • Ex-ante assessement of projects

5. UVAL’S work program • Activities (not staff) are organized in four areas • Methods and studies for public policy • Modernization and administrative innovation • Program evaluation • Project appraisal • 104 activities • Support to other parts of the Department and to the Network of the Italian Evaluation Units is still a large part of work (e.g., project appraisal, support to programming for 2007-2013)

6. Area 1 Area 1 Area 2 Area 2 Division 3 Area 3 Division 4 Area 4 Methods and studies for public policies Metodi e studi per le politiche pubbliche Trasferimento metodi e sostegno attuativo Modernization and innovation of administrative action Program Evaluation Program Evaluation Project Evaluation Project Evaluation Tendenze e analisi delle politiche di sviluppo territoriale Trends and analysis of territorial development policies Sostegno attuativo ai servizi DPS e alle amm.regionali e centrali Support for implementation for DDPservices and for regional and central adm. Coordination of CSF evaluation system Coordination CSF evaluation system Development of guidelines and standards for the assessment of infrastructure and integrated projects Development of guidelines and standards for the assessment of infrastructure and integrated projects Rapporto territoriale Annual report Intese e APQ, impostazione e performance. Institutional agreements and FPA, support to design and implementation CSF evaluation and control of evaluation quality of OP Valutazione QCS e controllo qualità valutazioni dei PO Project of territorial public accounts and public finances Progetto conti pubblici territoriali e finanza pubblica Implementation of CSF performance reserve. Attuazione Premialità QCS. Metodi e indirizzi per la valutazione e  l'utilizzazione della valutazione Methods and guidelines for evaluation and use of evaluation Support for project cycle enhancement Support for project cycle enhancement Support and training to the Network of regional and central evaluation units Supporto e formazione Rete dei nuclei CSF working groups Gruppi di lavoro (GdL) del QCS. Conduction and commissioning of evaluation on instruments, plans and programmes Conduzione e committenza di valutazioni su strumenti, piani e programmi Certified assessment of specific sectorial programmes Certified assessment of specific sectorial programmes Comunicazione e pubblicazioni. Communication and publications Contributo indirizzi e raccordi attività DPS Input to DDP guidelines Metaevaluation of projects Metaevaluation of projects The Evaluation Unit (UVAL)

7. Program evaluation • Evaluation capacity development • Started with Mezzogiorno & structural funds • Now expanding to whole country & all development program • Huge task: guidelines, groups, observation of practice & processes, common work • Evaluations • Slowly growing • Legitimity (who asks for evaluations) • Resources (financial and staff)

8. Evaluation: where we started from • A “continental” public service: No orientation to results • Absence of social partners and policy makers • Push for evaluation from EU Structural Funds: • No national reinterpretation of general framework • Predominance of EC evaluative needs • Rigid framework • Compliance at best purely formal • No interactions • Low quality of evaluations • Little (if any) use of evaluations

9. Evaluation capacity strategy initially relied on: • EU evaluation requirements + • New context of policy making • More ambitious development policy • More emphasis on local knowledge • More emphasis on change • More responsibilities to regional governments • Greater importance to administrative capacity

10. Close links between administration capacity and development policy • Development policy requires administrative capacity • Evaluation capacity is part of administrative (and political) capacity • Evaluation results improve development policies if useful for key actors • More than one way to get there

11. A piecemeal strategy • developed over time • gradually introducing innovations • knowledge from day-to-day work • taking into account achievements and disappointments • We are still working on it

12. Some evaluation capacity building activities UVAL worked on • Drafting national Guidelines (2001-2002) • Efforts to open up the market (2001) • Setting up networks and discussion groups (from 2001) • Creation of evaluation units (2000-2002) • Meta-evaluation activities (from 2002) • Self assessment exercises (2002-2003) • Utilization of evaluation, evaluative knowledge, and implicit knowledge in discussing strategic lines for next programming period and for nat’l interventions (2005) • Orientation to ex-ante evaluation process and products and organization of training and discussion groups (2006)

13. UVAL role in the evaluation of Ob.1 CSF • Organizing the evaluation: • Suggested four segments (main contract, environment, cultural resources, macro model) • Evaluation questions: • Collected MA’s point of view • Drafted Terms of Reference • Dealt with EC, evaluator, MA • Steering Group: • Read and commented report drafts and reports • Informal dealings, formal meetings and feedback • Protected evaluators’ independence

14. A part of the Network of Italian Evaluation Units • Not ALL evaluation units: • Evaluation Units dealing with Development Policies • Within executive branch • Some Ministries and all Regions • There are others: Regional elected assemblies, Universities, other Ministries. • Mix of tasks (actually evaluation only a small part) • ex ante analysis, project appraisals, feasibility studies • direct evaluations (only one so far) • management of mid-term evaluations • Various roles in programming activities • Interaction with local authorities and coalitions on projects • UVAL plays a proactive role in the Network, but has no formal “central” role.

15. The National Evaluation System • UVAL – Ministry of Economy and Finance (Coordinator) • INEA – Ministry for agricultural policies • ISFOL – ESF Evaluation Unit – Ministry of Labour • Evaluation Units of MA’s Initially built to deal with evaluation of Objective 1 regions’ strategy and OPs Now on the way to be extended to all Italian Regions’ regional policies

16. Different ways to organise evaluation • External : Mid-term evaluation of CSF and OP • Internal: • Evaluation of the performance reserve system • Thematic evaluations (social inclusion, urban development, research) • Self-evaluation

17. EVALUATION OF EVALUATIONS, MID-TERM EVALUATION, UP-DATE 2006 2005 2004 2003 2002 MID-TERM EVALUATION OF OP and CSF • Fieldworks • Updating Mid-Term Evaluation Reports • Fieldworks • Intermediate Products and in-depth studies • Design • Early Reports • Fieldworks • Evaluation Questions • Organization of evaluation update • EC reads Evaluation reports • SNV reads reports for lessons and utilization Mid-Term Evaluation Report Up-dating Mid- term : Strategy of CSF EVALUTION OF EVALUATIONS • SNV reads preliminary products • Work with Ob.1 Units • Methodological talks SNV – Evaluators – MA’s – Units • SNV makes reports to CSF’s MC • SNV reads intermediate evaluation products • Working in team with Ob.1 Units (Questionnaire and Meeting in Oct).