Estimating net impacts of the European Social Fund in England

1. Estimating net impacts of the European Social Fund in England http://research.dwp.gov.uk/asd/asd5/ih2011-2012/ihr3.pdf Paul Ainsworth Department for Work and Pensions July 2011

2. Overview • ESF priorities in England; • Indicators and Impacts; • Methodology and Challenges; • Results; • Evidence-based policy; • Conclusions.

3. ESF in England

4. Indicators and Impacts Indicators and outcomes can provide lots of useful information about the Programme… • Demographic characteristics of participants; • Number of jobs achieved; • Number of qualifications achieved. But many participants would have achieved jobs or qualifications even without ESF support. Our driving question is… “How many participants found jobs and ended welfare receipt directly because of the ESF support they received?”

5. Main analytical challenges • We can never know the counterfactual outcome – what would have happened to a participant if they had not participated: - we therefore use the outcomes of non-participants as an approximation for what would have happened to participants. • ESF support is voluntary – people who choose to participate will be different from those who do not: - we select a comparison group of non-participants who are very similar to participants on a broad range of characteristics. • Participants and non-participants may be receiving support through other programmes. This could bias results: - we ensure the comparison group of non-participants have received similar levels of non-ESF support to ESF participants.

6. Methodology We used a Propensity Score Matching methodology. This allowed us to match the participant and non-participant groups on a large number of characteristics. • Age, gender, ethnicity, disability and other demographics; • Previous employment and welfare receipt; • Geographical distribution; • Deprivation indicators; • Support from other employment programmes.

7. Impact Results For unemployed participants, ESF support made little difference. After entering the ESF programme (at week 0) participants and non- participants flow off benefit at the same rate. For economically inactive participants in receipt of disability benefits, ESF support was much more effective. After entering the ESF programme (at week 0) participants flow off benefit much more quickly than non-participants.

8. Interpretation of findings Why is ESF more effective in helping inactive participants than unemployed participants in England? There are no definitive answers, but some likely explanations: • Unemployed participants tend to be less disadvantaged and closer to the labour market than inactive customers. Many would have achieved jobs even without ESF support. • Inactive participants have a lower base-level of employment support than unemployed participants. ESF support therefore offers more of a step-up in support for this group.

9. Evidence-Based Policy How has the impact analysis helped us to improve ESF for the future? The analysis fed into the strategy for the second half of the 2007-13 Programme. The focus of ESF is being moved away from short term unemployed participants towards more disadvantaged groups: • Inactive benefit recipients; • Income support recipients; • Families with multiple problems.

10. Conclusions • Net impact analysis can provide crucial information which cannot be achieved by standard indicators and outcomes: - is ESF support making a real difference to the outcomes achieved by participants? • To perform this kind of impact analysis you will need: • Good individual-level data on both participants and non-participants; • Local expertise – estimating impacts is highly complex, with many technical challenges to overcome, some which will be specific to your Programme and data; • Time, resource and patience – our analysis took about a year and a half.