Pushing at the Limits of Rigorous Evaluation Harvey Armstrong, University of Sheffield

1. Pushing at the Limits of Rigorous EvaluationHarvey Armstrong, University of Sheffield Presentation to Conference Evidence-Based Evaluation: Session Focus on Evaluation 14.45-16.45, 7 July 2011, Gdansk

2. At the quantitative frontiers • Where is further quant desirable? • How to drive quant down the chain: from pan-EU level to individual OP, regional-level case studies etc? • Some key frontiers: • CIE • Primary and secondary data analysis • At qual/quant interface where qual dominates: CED • Classificatory methods • Whatever happened to economic theory?

3. CIE: Some lessons from a bygone eraBritish regional policy 1963-1971 Moore and Rhodes, Economic Journal, 1973

4. What did we learn back then? • ‘Policy off’ (control group) does not exist • ‘Other variables’ make it hard to isolate policy treatment effect • Geographical spillover effects • Possible to make progress on 1-3, but at cost of rapidly rising complexity • Result: A priesthood of the cognoscenti

5. CIE: Busillo et al (2010)

6. What has changed since 1973? • Control group not ‘policy off’: still a problem. Bosillo et al recognise non-Obj. 1 regions get other SF and national assistance monies. Excluded non-Obj. 1 regions with over €1,960 per head (min Obj. 1 value). • ‘Other variables’: still a problem, especially if wish to make a more convincing “plausible causal interpretation”(Martini, 2009, p4) of results. • Geographical spillovers: probably a worse problem for Bosillo et al than for M&R. Spatial autocorrelation. • Cognoscenti effect: infinitely better econometrics than 1970s, and multiple methods used to check robustness. But at cost of complexity (implementation and dissemination).

7. Likely limits of CIE • Great in own right, and as triangulation method with macro-models (and economic growth theory time series and pooled/panel data analyses such as Esposti and Bussoletti, Regional Studies, 2009). Particularly at pan-EU level and for bigger individual member states. • Slow enhancement as ‘plausible causal explanation’, particularly as more ‘other variables’ are added. • But will we ever see CIE being routinely used at the level of an individual OP region or within thematic case studies?

8. Main constraints on beneficiary surveys • Over-ambitious? Three very different uses: - to directly ask ‘counterfactual’ questions - to provide data for an econometric CIE - as a combined quant/qual method to go past ‘what works’ into ‘why’. Especially combined survey and semi-structured interview programmes. • Cognoscenti problem. Good survey design, piloting, implementation, bias checking, coding up and statistical analysis are rare skills. Combine this with an econometric CIE needing data-matching and the ‘capacity’ barrier is serious. • Too often an ‘afterthought’ in evaluations. More an insufficient time issue than resources.

9. Beneficiary surveys and direct counterfactual questions

10. Beneficiary surveys and the ‘why it works’ question

11. Classificatory methods • Another set of methods we see more of at pan-EU and national level than at OP or regional level. • Two main types have great value: network visualisation; cluster analysis. Principal components analysis less useful. • Network visualisation is widely used for partnership and network analysis and sometimes for industrial clusters/districts. But usually just charts. • Network visualisation and cluster analysis have enormous future potential for illuminating ‘why it works’ questions.

12. Russo and Rossi, Evaluation, 2009: Regional innovation project in Tuscany 2000-04 • Network visualisation, but also…. Graph (network) = nodes and links. Graph theory measures: degree centrality, closeness centrality, betweenness centrality. • Cluster analysis. Cases = organisations (429). Cluster variables: numbers of projects involved in, number of projects in each of the four action lines (Western Tuscany, the Fashion District, Optoelectronics, Biotechnologies). Results: six significant clusters. Four (A-D) encompassing 361 organisations involved in only a single project each and 34 involved in two to four projects. But…… Cluster E = 16 organisations involved in two to three optoelectonics projects, and cluster F = eight organisations involved in more than five projects. E+F = only 24 organisations but 50% of the ERDF funding.

13. Network visualisation, graph theory and cluster analysis: Russo and Rossi, Evaluation, 2009: Regional innovation project in Tuscany 2000-04

14. Whatever happened to economic theory? • TBIE is almost always seen as realist evaluation. Particularly valuable at the ex ante evaluation stage – revealing the implicit theory of development. • Yet economic theory, especially regional growth theory, surely has much to contribute to understanding ‘why’. We surely do not wish to stick so closely with qual and ‘realist’ evaluation. • Quant applications of growth theories largely restricted to pan-EU or national studies. • We need: - more econometric testing of economic growth theory - much more reference and detailed unearthing of which particular growth theory(ies) the OPs and individual projects are resting upon.