Justice Data Lab: Facing the Third Sector

1. Justice Data Lab:Facing the Third Sector How can we develop the capacity of third sector organisations to engage with data? Scottish Universities Insight Institute Georgina Eaton & Tillie Paul, Ministry of Justice 4th February 2015

2. What will be covered • Aims and history of the Justice Data Lab • How does the Justice Data Lab work and how can it be accessed? • Findings to date • Receptions of the service • Challenges and successes

3. Launched in April 2013 ..to improve the evidencebase on successful rehabilitation.. ..by giving organisations working with offenders secure and legal access to aggregate re-offending data ..enabling them to better assess the impactof their work on re-offending Aim of the Justice Data Lab

4. Why are we doing the Justice Data Lab? • In 2012 we identified that charitable organisations in particular found it difficult to access re-offending data on their clients… • … this meant that they could not understand how effective their services were at rehabilitating offenders… • … and they were therefore unable to understand how their services could be improved, or have the evidence for further funding • It soon became clear that there was intense interest in this initiative from both public and private sector organisations too

5. Project timeline Spring 2014: Pilot extended for further year and announcement of improvements to the service December 2011: NPC approach MoJ about Data Lab idea December 2012: Announcement that Justice Data Lab would be piloted • 2013 2012 2014 2012: Feasibility Justice Data Lab Pilot April 2013: One year pilot goes live October 2013: First publication of Justice Data Lab products January 2012: Ministerial Approval to look into the feasibility of the Justice Data Lab

6. How does the Justice Data Lab work? Provider organisation • Individual level data • sent to MoJ • Aggregate data return MoJ Analysis and Matching

7. Accessing the Justice Data Lab service • Data upload template requirements: • Sample size must be at least 60 needed for robust analysis • Matching variables including: • full name • date of birth • gender • intervention start date • Thorough information on the service they provide • Send securely (CJSM or gsi account)

8. What does the Justice Data Lab not do? • X We won’t disclose individual level data • X We currently only give re-offending related outcomes • X There are ethical and practical considerations for assessing interventions aimed at certain groups of offenders (e.g. sex offenders, vulnerable persons) that mean that a Justice Data Lab analysis is not appropriate

9. Processing and matching the data • Match to Police National Computerfor demographics, criminal histories • Find correct sentence and re-offending follow-up period in MoJ administrative datasets • Link to employment and benefitsdata - Data share with DWP/HMRC • Create matched control groupof similar offenders who have not had the intervention from MoJ administrative datasets

10. Processing and matching the data • Propensity Score Matching to match individuals in the two groups to each other • Test differencesin re-offending for the groups • Statistical significance testing to determine whether there is a true differencebetween the groups

11. What is provided to Justice Data Lab users? The best estimates for the one year proven re-offending rate for offenders who received an intervention from WYJS, and a matched control group. • One year re-offending rate • Frequency of re-offending • Time to re-offending • Information on characteristics of both the treatment and control groups

13. Cumulative findings to date • Of the 121 reports published so far: • 27 reports indicated statistically significant reductions in re-offending on the one year proven re-offending rate • 87 reports indicated insufficient evidence to draw a conclusion about the effect on the one year proven re-offending rate • Of these 87, 11 reports detail statistically significant reductions in the frequency of re-offending • 7 reports indicated a statistically significant increase in re-offending on the one year proven re-offending rate

16. Specific reports – Blue Sky The best estimates for the one year proven re-offending rate for offenders who received an intervention from Blue Sky, and a matched control group. • Short term, full-time employment contracts. Aims to move them into onward full-time employment elsewhere. • The one year proven re-offending rate for 72 offenders employed by Blue Sky was 31%, compared with 43% for a matched control group of offenders with similar characteristics. • A reduction in re-offending between 1 and 23 percentage points. One year proven re-offending rate

17. Specific reports – Prisoners Education Trust Prisoners Education Trust submitted data relating to offenders who had a grant for Open University, distance learning courses, or Art and Hobby materials between 2002 and 2010. We carried out one overall analysis and four sub-analyses, the results are in the table below.

18. Specific reports – Prisoners Education Trust “We have already changed our approach to funding different course types as a result of the findings.” “Our charity and our funders know that learning in prison works – but now we have the evidence to prove it with this robust, hard-edged report carried out by MoJ statisticians.” Rod Clark, Chief Executive of Prisoners Education Trust

19. Reception of reports • Survey issued to organisations who have used the Justice Data Lab service showed that: • The expectations around the Data Lab had been met, although our customer service could be variable • The Justice Data Lab is a useful service for the third sector, helping to provide information on re-offending and impact • More information on outcomes(severity, re-incarceration rates, employment and benefits) would be helpful • Results had been used to understand / demonstrate impact internally and externally

20. Challenges • First time the sector has had transparency about their effectiveness • Claims about success vs. evidence • Understanding technical aspects / statistical literacy • Availability and quality of internal and external data • Resources for MoJ and organisations • Legalities for organisations 40% ↓ re-offending

21. Successes • Positive feedback from users • Organisations using results to change their services • Continual demand for service • Engaging with users • Service developments • Passing on experience to others

22. Award Winners! • Government Finance Insight Award 2014 • “Judges were impressed by the use of data in an innovative way, and the presentation of complex data in a way that is understood by all.” • Royal Statistical Society Excellence in Official Statistics Award 2014 • “Judges were impressed by the use of statistical techniques to assess success (or failure) in a critical area and by the exceptionally close way MoJ statisticians had worked with their users, mainly non-statisticians.”

23. How can we develop the capacity of third sector organisations to engage with data? • Data collection • Central data system – Charity Log? • Data manager/analyst • Access to data through government/local authorities • Open data • Asking for help – Academics, Government Statisticians and Think Tanks • Don’t be scared of data!!

24. Contact Details • Email: justice.datalab@justice.gsi.gov.uk • Accessing the Justice Data Lab service: • https://www.gov.uk/government/publications/justice-data-lab • Published reports: • www.gov.uk/government/collections/justice-data-lab-pilot-statistics • Government Statistical Service: • https://gss.civilservice.gov.uk

25. Propensity Score MatchingScale of propensity score 0 1 -1 Key Control offenders are matched to treatment offenders if the control offender’s propensity score are within a specified range away from the treatment offender’s propensity score. Treatment offenders Control offenders Matched offenders