Welfare Economics, Project and Programme Appraisal and Evaluation

1. Welfare Economics, Project and Programme Appraisal and Evaluation Lecture 2: Programme Evaluation David Hegarty 7 October, 2011

2. Structure • Part 1: Definition, purpose and key issues in programme evaluation • Part 2: Overview of methodological tools • Part 3: Evaluation capacity and practice in Ireland

3. Part I: Definition and Purpose • No single, preferred definition of evaluation • “Evaluation” is a common, everyday activity • People evaluate films, restaurants etc • Firms evaluate investments • “Evaluation is an elastic word that stretches to cover judgements of many kinds” (Weiss)

4. Some Definitions • “The process of collecting and analysing information and reaching conclusions on specific questions” (Dept. of Finance VFM guidance manual) • “Judgement of interventions according to their results, impacts and needs they aim to satisfy” (EU Commission) • “Evaluation is the process of determining the merit, worth, and value of things, and evaluations are the products of that process” (Scriven) • “The systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards, as a means of contributing to the improvement of the program or policy” (Weiss)

5. Key Features • Definitions point to some key features • Evaluation can be carried at various levels: policy, programme or project • A systematic exercise based on accepted social science research standards • Involves forming a judgement to be based on certain criteria • Focus can be either on the process (operation) of the programme or on its impacts (outcomes) • Purpose of exercise is to improve the intervention under evaluation

6. Evaluation Purposes: Why evaluate? • Planning • Is the programme justified? • Programme design • Resource allocation • Implementation • Is the programme working and/or how can the programme be improved? • Accountability • What was achieved?

7. Evaluation Purposes • Knowledge • What interventions work and in what circumstances? • Does the logic of the programme and its assumptions need to be questioned? • Development • Institutional performance and strengthening • Service quality

8. Evaluation Purposes • But… • evaluation sometimes used for other, “covert” purposes • Justify decisions already made • Postpone decisions • Public relations • Compliance • “A rational exercise often undertaken for non-rational reasons” (Weiss)

9. Summary • Evaluation can be seen as serving an overarching learning purpose “To learn through systematic enquiry how to better design, implement and deliver public programmes and policies” (EU Evalsed Guide)

10. Formative and Summative Evaluation • Summative evaluation • Accountability focus • What has been achieved? • Formative (“Process”) evaluation • Development or learning focus • How can we improve performance and delivery of programme? • “When the cook tastes the soup that’s formative evaluation. When the guest tastes it, that’s summative evaluation” (Scriven) • Most evaluations lie along this continuum • Combining elements of each • Both relevant and useful to public sector • Dept. of Finance VFM manual has strong summative emphasis

11. Key issues in programme evaluation • What's the basis for evaluation judgements? • Dept. of Finance VFM Manual refers to 5 main evaluation criteria • Relevance • Rationale • Effectiveness • Efficiency • Impact • Framework originally developed for evaluation of EU-funded programmes (CSF Evaluation Unit, 1996)

12. Evaluation Questions • Rossi et. al (2004) identify 5 main question types in evaluation of social programmes • Needs assessment • Assessment of programme theory • Assessment of programme process • Impact assessment • Efficiency assessment • DOF framework has stronger summative or economic emphasis

13. Relevance • Two main dimensions • Policy relevance • Domestic • EU • External relevance • What societal needs or problems does programme address? (needs assessment) • Is the programme “fit-for purpose”? • Implications of external changes for programme (continued relevance)

14. Rationale • Why is the State involved? • Is there a market failure? • “A necessary but not sufficient for government intervention to improve economic efficiency is that there is some form of market failure” (HM Treasury) • Could the problem be addressed through more direct means? • Danger of “second-best” solutions • Types of market failure • Public goods • Externalities • Redistribution

15. Effectiveness • Is the programme meeting its objectives? • Generally addressed at level of • Inputs: Is the money being spent? • Outputs: • Results or immediate benefits • Are the above in line with expectations? • If not, why not? • Almost a monitoring question • Effectiveness and impact questions often overlap • A lot depends on how objectives are framed

16. Efficiency • Some definitions: • “Efficiency in the public sector involves making the best use of resources available for the provision of public services” (Gershon UK efficiency review 2004) • “Optimising the ratios of inputs to outputs” (DOF VFM Manual) • Can be viewed in a number of ways • Reduced inputs for same level of service • Additional outputs for same level of inputs • Improved unit cost ratio • Changing mix of activities/outputs to better deliver a given objective for same input level • Using alternative delivery approaches, e.g., outsourcing to private sector • Efficiency a core, perhaps overarching, element of value for money agenda • Getting the best return from a given level of resources is the essence of value for money

17. Impact • What difference has the programme made? • To its beneficiaries • In terms of wider socio-economic objectives • Need to consider • Deadweight effects • Displacement • Unintended side-effects • So-called horizontal issues a sub-set of impact • Rural development • Poverty • Gender equality

18. Evaluation Cycle • Evaluation cycle is a function of the wider programme and policy cycles (see diagram) • Aim should be to conduct evaluations at the right time to influence programme design and policy formulation • Easier said than done!

19. Policy, Programme & Evaluation Cycles Policy Review Policy Formulation Programme Conclusions Programme Design Programme Implementation Policy Delivery Evaluation Source: The EVALSED

20. The Evaluation Cycle • EU Structural Fund regulations require evaluations at three stages • Ex ante (before) • Interim or ongoing • Ex post (after) • Value for Money reviews generally take form of ongoing evaluations

21. Ex Ante Evaluation • Focus: “to optimise the allocation of resources and improve the quality of programming” (EU Regulation) • A planning purpose • Key evaluation questions • What is the rationale for programme and is it robust? • Is the programme relevant or fit-for-purpose • Programme design issues

22. Interim Evaluation • Focus • Largely an implementation purpose • But much depends on programme maturity • Key questions • Relevance or continued relevance • Effectiveness • Efficiency

23. Ex Post • Mainly an accountabilitypurpose • what has been achieved and at what cost • summative in character • Not widely practiced in Ireland except for EU programmes

24. Evaluation Cycle and Focus

25. Overview of methodological tools Sourcing information and data Data analysis techniques Tools to inform evaluation judgements 25

26. Data Sourcing All evaluations require data “the raw material that once collected is organised, described, grouped, counted and manipulated by various methods and techniques” (EVALSED Guide) Primary and secondary data Primary data are data generated as a consequence of programme (uptake of services, data relating to beneficiaries) Secondary data are generated for other purposes and pre-exist the programme (e.g., socio-economic and administrative data) 26

27. Data Types Key distinction between quantitative and qualitative approaches Quantitative used to gather “hard” data Who, what, how many Expressed in terms of averages, ratios or ranges In practice much hard or quantitative data may be categoric or ordinal in nature Qualitative methods used to gather “soft” data Focus on understanding or why questions Quantitative/qualitative a continuum Distinction stronger in terms of analytical intent Quantitative for aggregation and generalisation Qualitative to understand complexity 27

28. Data sourcing techniques Main techniques/sources include Monitoring indicators Documentary analysis Administrative data Socio-economic data Beneficiary surveys Stakeholder interviews Focus groups Case studies 28

29. Data Analysis Techniques Once the data is collected, how do we analyse it? Main techniques include Statistical analysis SWOT analysis Econometric models Experimental designs Quasi-experimental designs (control groups) 29

30. Tools to inform evaluation judgements Having gathered and analysed the data, how do we arrive at evaluation judgements? Main tools Benchmarking Multi-criteria analysis Cost benefit analysis and cost effectiveness analysis (will be addressed in project evaluation stream) Economic impact assessment Macro Micro Intervention logic analysis Specialist thematic tools Gender impact assessment Strategic environmental assessment 30

31. Factors Affecting Choice of Method Programme type Stage in programme/evaluation cycle Evaluation purpose Evaluation scope and questions Data availability Resources 31

32. Factors Affecting Choice of Method 32

33. Part 3: Evaluation capacity and practice in Ireland • Concept of evaluation capacity • Influence and evolution of EU Structural Funds evaluation systems • Development of national programme evaluation processes • Expenditure Review Initiative • Value for Money and Policy Review Initiative • Where are we now?

34. The concept of evaluation capacity • Evaluation capacity concerns • the process of setting-up the necessary systems and infrastructures to undertake evaluation • Some definitions • “the development of national or sectoral evaluation systems” (MacKay, World Bank) • “the institutional, human, resource, skill and procedural base for conducting evaluations in public policy and public management systems” (Evalsed Guide) • Concerned with creating and sustaining factors that support evaluation in government sector

35. Key dimensions of evaluation capacity • Fair degree of consensus in literature as to key building-blocks • 4 key dimensions generally highlighted in literature • Architecture: organisation of evaluation function • Demand: is there an effective demand for evaluation? • Supply: evaluation resources (methods, resources, skills) • Institutionalisation: building evaluation into policymaking systems • The wider, cultural factors or conditions that determine the influence of evaluation on policy

36. Critical Success Factors • Key lessons as to factors needed to strengthen government evaluation systems • Substantive government demand essential • Incentives important for demand • Limitations of reliance on rules and regulations • Need to work on demand and supply sides in parallel • Need for evaluation champions • Adequate evaluation resources • Including good data systems • Importance of structural arrangements/architecture including links with other functions • Danger of over-engineering the system • Utilisation is key • A long-haul effort requiring patience, persistence and leadership

37. Evolution and development of EU evaluation systems • Evaluation context (as of late 1980s) • Little prior tradition of programme evaluation in Ireland prior to Structural Funds • evaluation limited in scope and largely peripheral to decision making • A low evaluation capacity baseline • Evaluation impetus driven by • compliance considerations • EU Commission pressure and support • political priority attached to EU funds • Leading to …. • gradual creation of evaluation structures (CSF1, 1989-1993) • major expansion in evaluation capacity and output (CSF2, 1994-1999)

38. Key Developments in Capacity Development • 1994 to 1999 CSF saw gradual establishment of programme evaluation structures • 3 internal evaluation units • 6 external evaluators to other programmes • By end-1996 each programme (9) had dedicated evaluation function • Central CSF Unit with a coordination and good practice promotion remit set up in 1996 • Lessons learnt influenced design of evaluation arrangements for 2000 – 2006 period

39. 2000 to 2006 Evaluation System: Key Features • Evaluation system extended from just EU-funded elements to entire NDP (€51 bn) • Applied to up to 20% of total public expenditure • Centralised system with 1 Evaluation Unit (NDP/CSF Evaluation Unit) • Main responsibilities and activities • Development of performance indicators • Advice on project appraisal techniques • Drafting evaluation terms of reference • Commissioning evaluations • Doing evaluations • Extensive ongoing evaluation effort

40. Development of National Evaluation Systems • Background/origins • Increased emphasis on public service management • Public management reforms in early/mid 1990s • International developments in public management • Key milestones • C&AG amendment Act, 1993 • Gave C&AG mandate to carry out VFM audits • And to examine adequacy of departments’ systems to evaluate effectiveness of their operations • 76 VFM audits to date • Launch of Strategic Management Initiative (1994) • Delivering Better Government (1996) • Public Service Management Act, 1997 • Departments required to produce Statements of Strategy and Annual Reports

41. Expenditure Review Initiative • Expenditure Review Initiative (ERI) • Non-EU evaluation system introduced in 1997 • ERI influenced by Australian system • A “whole of government” evaluation strategy • Objectives • to provide a systematic analysis of what is actually being achieved by expenditure in each programme; and • to provide a basis on which more informed decisions can be made on priorities within and between expenditure programmes

42. ERI: Key Features • Aim was to review all expenditure areas every 3 years • Programme of reviews agreed by each department with Department of Finance • Central Steering Committee and secretariat in DOF • Reviews undertaken by line departments and by programme managers • Department of Finance represented on steering committees

43. Evolution of ERI • ERI reviewed by C&AG in 2001 • Key findings • 3 year target not met, significant delays • Reviews focused on minor programmes • Quality highly variable • Limited influence on resource allocation • Number of reforms introduced • Establishment of network of reviewers and training supports • Independent quality review procedure • Efforts to track impact of reviews and review process generally

44. Evolution of ERI • Central steering committee (ERCSC) reviewed progress in October 2004 • Key findings • Taken time to for earlier reforms to take effect • Slippage in timeframe for completion of reviews • Topics selected for review relatively small scale • Evaluative capacity of departments variable • Process had led to improvements in approach to evaluation and evaluation culture of departments • Extent to which reviews driving resource allocation decisions unclear

45. ERI Review • Series of recommendations made by ERCSC • Changes to structures and reporting arrangements in departments • Independent steering committees • Reporting on review results • Intensify efforts to develop performance indicators • Use trainee analysts and graduates from IPA policy analysis masters course to support review process

46. Value for Money and Policy Review Initiative • ERI replaced in 2006 by “Value for Money and Policy Review Initiative” • Somewhat wider evaluation focus • Ninety reviews approved for 2006-2008 period • 2 per department per year • Guidance manual published 2007 • Mix of internal and external reviews • Target for number of reviews does not appear to have been reached • Progress rather uneven

47. Current Situation • Post 2007, EU funding very limited • EU funding of just €900 mn. for 2007 to 2013 • Limited evaluation requirements under EU regulations • Earlier NDP/CSF Evaluation Unit replaced by Central Expenditure Evaluation Unit in MOF • Responsible for evaluation of all national programmes • Main focus of Unit now on Value for Money and Policy Review Initiative • Undertaking and overseeing VFM reviews • Issuing guidelines • Unit also has important role in project evaluation area

48. Reflections on Irish Experience • EU requirements and external influences a key driver • EU funds contributed to development of capacity and expertise • Increased awareness and understanding of evaluation amongst policymakers • Led to creation of internal evaluation structures • And improved supply-side capacity in response to evaluation demand • Important long-term benefits of Structural Funds

49. Reflections on Irish Experience • Ireland now in a “post Structural Funds” era • Evaluation system no longer organised around EU Funds • Slow progress under ERI and VFM processes • Some signs of a loss of momentum in evaluation practice over recent years • Current economic difficulties means heavy emphasis is on expenditure control and reduction and broader expenditure review exercises • McCarthy review (2009) • Comprehensive Spending Review

50. Conclusions: Some quotes • “In the age of evaluation Ireland has been encouraged or even compelled by the pressure of very influential external forces to adopt a culture of evaluation. Despite the fact that this culture dates back over some three decades it remains a somewhat uneasy and unconvincing addition to the tools of governance” • “One thing seems clear: policy developments in the field of evaluation will continue to be largely driven by external pressures since there is very little evidence of an appetite for evidence-driven policy among senior political or public-sector leaders” (McNamara et al, Developing a Culture of Evaluation In Ireland, 2009)