150 likes | 156 Views
Making an Appropriate Choice of Methods to Evaluate Cohesion Policy Harvey Armstrong, University of Sheffield. Presentation to Joint Meeting of DG REGIO Evaluation Network and ESF Evaluation Partnership 8 July 2011, Gdansk. Key Fifth Cohesion Report themes (pp256-7).
E N D
Making an Appropriate Choice of Methods to Evaluate Cohesion PolicyHarvey Armstrong, University of Sheffield Presentation to Joint Meeting of DG REGIO Evaluation Network and ESF Evaluation Partnership 8 July 2011, Gdansk
Key Fifth Cohesion Report themes (pp256-7) • Move on from preoccupation with financial ‘absorption’ • Fewer, simpler policy priorities (‘concentration’) • Focus on results and impacts (i.e. performance). • More rigorous evaluation (e.g. CIE, CBA for quant; case studies for qual; beneficiary surveys for both quant and qual) • Triangulation • From productivity to well-being (happiness) and sustainability For the presentation I will focus on 4, 5 and 6
Making the right choice: The example of beneficiary surveys • CED priority (P6 – Targeted Action for Key Deprived Areas) in 1994-99 Yorkshire & Humber Objective 2 programme • ERDF Measure 6.21: ‘support for community based economic projects’ – 111 (of the 178) CED projects • Main types of projects funded: - Micro-finance and credit - Social enterprises (‘community businesses’) - Labour market access, training and ILM - Education - Targeted environmental improvements - Transport access to jobs - Pure capacity building
What we did • Two-stage sampling method: - managers of all the projects contacted for lists of beneficiary enterprises and other organizations - individual enterprises and organizations randomly sampled from the lists (simple random sampling) • Useable lists from 33 of the 111 projects, but all the biggest projects chased down • Random sampling of 242 organizations and enterprises (110 First Sector enterprises serving mainly non-local markets; 44 First Sector serving mainly local market; 15 community enterprises; 64 mixed social/market organisations; 9 purely social organisations)
Did we do anything right?Lots of nice new stuff on ‘why it works’
What did we do wrong (in Fifth Cohesion Report terms)? • Rigour • Should have used stratified sampling method • Only one bias check (sub-areas) • No proper confidence intervals • Triangulation • Could have used open questions and a third stage of SSIs as a qual-on-quant triangulation • Could have analysed secondary data sets for quant-on-quant triangulation • Well-being and sustainability • Beneficiaries = organisations. Wrong. • Should have done residents’ survey • Should have added some sustainability questions
Case studies: what’s being done well? • Example: ERDF 2000-06 Work Package 4: Structural Change and Globalisation • Thematic: effect of, and policy response to globalisation • Holistic: quantitative and qualitative • Quantitative: Three main elements: - By core team statistical analysis - By case study teams - Pilot beneficiary surveys; secondary data analysis • Qualitative: - In-depth analysis of documents - Semi-structured interviews with regional experts, policy makers and beneficiaries (176 in total)
Case studies: what’s being done well? • Multiple full case studies (12) • Transparent and logical selection criteria: ‘representative’ approach rather than extreme/deviant, maximum variation, critical or ‘pardigmatic’ (Flyvbjerg, 2006) • In-depth knowledge of context, including historical context (flexible – back to early 1980s for many) • Narrative: ‘telling the story’ • Mini-case studies • Common template but with flexibility for regional teams • Central core team • Generalization: accumulation of evidence; power of the ‘deviant’ case
Case studies: what next? • Rigour • Fundamentally a qualitative method, but needs more quant • Beneficiary surveys • Proper CIE and secondary econometric data analysis? • More rigorous qualitative methods • Triangulation • Already happening, but • Regional team SSI/document analysis and core team quant • Beneficiary survey as quant-on-qual by regional team • Well-being and sustainability • Potentially excellent method
From productivity to well-being: A warning from history: British enterprise zones in the 1980s • An urgent need to prioritize amongst the many well-being and sustainability measures. • ‘Concentration’ will narrow the field for each programme, but…….. still and awful lot, especially for well-being • A ‘must’ for all future evaluations: distribution of income and wealth • A warning from history: Erickson and Syms, Regional Studies, 1986. Salford-Trafford Park EZ, Manchester. • Tax reductions (especially property tax – ‘Business Rates’ • ‘Place prosperity’ versus ‘people prosperity’: who wins, who loses. Distribution: combined property tax (rates) plus real rents: 36% of value of the property tax exemption went to industrial tenants and 64% to initial property owners, many ex-regional. We may not like the results we get.
From productivity to well-being and sustainability: Challenges for other methods? • Heavily affected? - cost-benefit analysis - SWOT - Evaluability assessment, logic models, concept mapping - Delphi methods, focus groups, experts panels - Observational methods - Stakeholder consultation and participative techniques - Meta-analysis • Less heavily affected? - Input-output - Macro models CGE