Counterfactual impact evaluation: What is it, why do it?

1. Counterfactual impact evaluation:What is it, why do it? Daniel Mouqué Evaluation Unit DG REGIO

2. An example: support to enterprise & innovation • Some €79 billion of cohesion in 2007-13: the largest broad category of expenditure • Key instrument: investment/research grant • But also significant spending on loans/venture capital, advice, networking, incubators • With all this at stake, we should know exactly what we’re doing, right?

3. What should we know about enterprise support? We’re managing a programme – what should we know? • The context and needs (productive base, sectors, weaknesses etc…) • What we plan to change (Investment? Productivity? Employment?) • How we will change it: instruments, delivery, financial allocations • Activity/outputs (number of enterprises assisted etc) Question: is this enough?

4. In the long term, we want to know about impacts • Do the instruments work? In terms of increasing long run investment, productivity, employment, etc? • What is the optimum level of support? • Different effects of different tools? Better single instrument or mixed? • SMEs only or include larger enterprises?

5. In other words, we want to know... • What works? • How much impact does it have? • How to change/finetune it to get more impact? These questions apply to all cohesion policy fields: human resources, infrastructure, environment

6. monitoring to track implementation efficiency (input-output) Human behaviour But impacts are the tricky bit • impact evaluation to measure effectiveness (output-outcome) MONITOR EFFICIENCY INPUTS OUTPUTS OUTCOMES EVALUATE EFFECTIVENESS Source: Arianna Legovini and the World Bank (modified) Plans, programmes

7. How do we assess impacts? Traditionally in enterprise support: • Monitoring (but: « before/after » problem) • Beneficiary surveys • Opinion And enterprise support is one of the « good » areas – situation no better in training, infrastructure, environment

8. To truly know impacts, you must know… … What would have happened without the intervention Or in other words: The counterfactual

9. How do we find this mysterious counterfactual? • Can it be observed?

10. A time machine?

11. Sadly, only in Hollywood… Maybe someday? 

12. What do scientists do?

13. One thing scientists do to find counterfactuals: Compare twins

14. Source: www.webmd.com – smoking and sun are responsible here

15. Twins in cohesion policy? • Does this mean we can only provide training to twins? And only one of the two? • And what about enterprises? Or urban neighbourhoods in crisis?

16. The « law of large numbers » As n increases, random differences tend to average out Solution 1: large « n » - mobilising the power of statistics NB: « large » varies. But 20 or 50 may be enough

17. Solution 2: clever statistical matching techniques • Sometimes solution 1 is enough • But sometimes we need to use statistical techniques to find matches between the treated and non-treated populations We’ll come back to how this is done tomorrow…

18. But matching is not always straightforward

19. Examples of counterfactuals in practice • 100 innovation vouchers are randomly distributed between ~900 applicant firms, performance tracked • 500 long term unemployed in poor mental health – 250 receive standard support, 250 receive extra counselling • 70 deprived urban areas assisted. Performance on unemployment etc compared to neighbouring areas

20. To recap • It is crucial to know about impacts • But measuring impacts is far from straightforward, depends on human behaviour • « Traditional » techniques do not measure impact • We need a counterfactual, comparing performance of treated and non-treated • But counterfactuals are not the only useful technique…