Evaluation evidence and lessons from contracting for employment services in OECD Countries

1. Evaluation evidence and lessons from contracting for employment services in OECD Countries Dan Finn Centre for Economic and Social Inclusion and University of Portsmouth Welfare to Work Convention Birmingham, 11 July 2012

2. Contracting with external providers • In many countries Ministries, PES, Municipalities contract with other providers, but mostly on the basis of cost reimbursement/fixed-price and highly regulated contracts • Most ‘purchasers’ have used external providers to: • buy specialist services • increase capacity • ‘benchmark’/drive innovation in PES • competition for existing providers and PES • Developments include greater use of payment-by -results with replacement of public sector provision in one OECD country(Aus), extensive subcontracting of job matching & reintegration services (e.g. some US states, Netherlands). Others experimenting with private contractors to build capacity and/or ‘benchmark’ and drive innovation in PES (S. Korea, Germany, France, Sweden, Saudi Arabia, Brazil)

3. Advantages/risks of contracting • Advantages - cost-efficiencies, innovation, resources, flexibility, capacity, complementarity. • Most discussed risks include: • ‘Cream-skimming’ • ‘Creaming’ and ‘Parking’ (clients/areas) • ‘Gaming’ and fraud (for & non profits) • Other risks include: • Potentially high transaction costs (offset by longer contracts, fixed outcome prices?) • Loss of insight into service delivery – importance of contract and performance management • Accountability and transparency issues in exercising oversight of public spending, and supervising activities of commissioners and contractors. • Ability to transfer risk – response of purchaser to market failure and/or withdrawal from contracts

4. Evaluation evidence • Evidence/literature reviews reveal extensive range of case studies, ‘grey literature’, and varied quality academic research. • Most studies concern developments in countries and/or states at the forefront of ‘work first’ welfare and public sector reform – USA, Australia, Netherlands, GB, Germany • Hard to disentangle commissioning/market impacts from wider system changes, such as increased activation/compliance effects – evaluation research of welfare markets “in its infancy”. • Most comprehensive evaluation of Australian Job Network. Recent impact studies from GB and Europe

5. Job Network Evaluation • Aus Government committed to comprehensive evaluation • 3 phases to Departmental evaluation: • Stage 1, ‘implementation and market development’ (Feb 2000) • Stage 2, ‘progress report’ – early outcome data, assessed market development and any need for ‘fine tuning’ (Feb 2001) • Stage 3, ‘effectiveness’ – impacts and effectiveness of administrative arrangements, different types of assistance and different providers (May 2002). • Methods – included use of regular 3 month post programme monitoring surveys that commenced in 1996, with ‘net impact’ studies since published every few years. • Commitment also to an independent review of the new policy framework, undertaken by the Productivity Commission (June 2002). Independent review also undertaken by OECD (2001) • Evaluations found improvement in access, user satisfaction and performance, but also parking in the Intensive Assistance ‘Black Box’, leading to APM and Jobseeker Account

6. Findings from impact evaluations • Several Australian, 2 USA (Florida) and 12 English language European impact studies of contracts that paid for outcomes which used experimental/quasi experimental methods • Mixed results – most positive results from Australia, UK; less positive from studies in Germany, Sweden and France • Negative European findings linked to issues of poor contract design, early stages of contracting, and contrasted with typically increased performance from PES under competitive pressure • Interesting findings from process studies about how contracts changed service delivery strategies – PES more process oriented, subcontractors emphasised barrier reduction, job search/outcomes • Little focus on differential performance of contractors – likely that efficiencies and gains of contracting emerge over time (as found in Australian experience) resulting in fewer higher performers. • Australia has several longer term outcome studies which indicate that employment outcomes improved between 3 and 12 months, but mostly for those closer to the labour market - research showing movement of low paid workers to higher pay/skills (often associated with transition from casual and part-time to full-time work)

7. Some issues for evaluation • Market composition: • Findings suggest that improved performance associated with organisational learning and reduction in the number of providers – often linked with rationalisation/exit of existing providers with impacts on smaller, specialist providers. • Problems of ‘mission drift’ for the voluntary sector (e.g., sanctions, advocacy) and changing composition of third sector providers • Market incentives induce early service delivery innovation and improved management but evidence of ‘innovation plateau’ and ‘service delivery convergence’ in mature markets. • Evidence suggests process of frequent re-regulation to both resolve delivery problems, reflect learning and reshaping of contract incentives to meet new programme objectives. • Some re-regulations ‘ring fence’ resources for client services or create separate contracts for specific services. • Issues with administrative complexity in relationships between delivery of benefits, referral to contractors and interaction with sanctions/appeals/health/changing circumstances