Evidencing Outcomes

1. Evidencing Outcomes George Box Evidence-Based Practice Team, HMPPS September 2019 georgina.box@justice.co.uk PROTECTED

2. Have you made an impact? • What does your project/service/intervention set out to achieve? • What is your Theory of Change? • How can you demonstrate that it has achieved your intended outcome?

3. How do you demonstrate impact? • In order to demonstrate impact you can measure baseline indicators at the start of your service/intervention and then compare the results to a re-measurement of the baseline at least once (usually post intervention) • It is important to identify objective indicators and have a good understanding of how you are going to collect the evidence in the planning stages of your intervention

4. What is Evidence? • Evidence can take a range of forms depending on your service and outcome (surveys, observed behavioural change, interview data, numeric data etc.) • Intermediate outcomes are useful when you are restricted in the evidence you have available or the time in which you have to collect it • Intermediate outcomes are results or outcomes of a service/intervention that must occur in order to produce the final outcome. (e.g., increased employment before a later reduction in recidivism)

5. What is Evidence? An Example • What sort of evidence would help establish whether a drug treatment service had achieved its aim of reducing offending related to drug misuse? Intermediate outcomes - Fewer positive mandatory drug test results - Greater uptake of methadone replacement treatment - Greater attendance at Narcotics Anonymous meetings • Fewer assaults motivated by drug use • Results from a survey of drug users’ views of their likelihood of misusing drugs before and after use of the service Longer-term outcomes - Reduction in rates of drug-fuelled offending

6. How to collect evidence • Evidence gathering needs to be as simple and cost-effective as possible • Ensure that the data you are collecting relates to the outcome you are measuring • Pilot questionnaires and data collection methods • Keep detailed records of the recording processes as you go along and make changes if needed • There is value in combining different measures and different outcomes to gain a fuller understanding of the impact.

7. Understanding and presenting evidence • The way in which you assess your outcomes will be dependent on your method of data collection and the types of evidence you are collecting. • Judging the evidence and level of impact of an intervention can take multiple forms: • Narrative accounts of a service/intervention • Simple numeric data about, for example, the people who accessed the service (descriptives) • Case studies • Evaluations

8. What is evaluation? • A type of research which aims to identify how well a project/service/intervention achieves its aims • In CJ evaluations, the aim we are most often interested in is reducing reoffending. • Reconviction is most usual outcome • Evaluation of intermediate outcomes can be used when longer-term evaluations are not possible PROTECTED

9. For this grant, we are looking at evidence of impact • Impact Evaluation • An impact evaluation tries to identify whether or not a policy achieved the outcome it set out to achieve. • This requires identifying firstly whether the outcome was actually achieved, and secondly whether the policy in question, rather than one or more other factors, was responsible. PROTECTED

10. What are the ingredients necessary for a robust outcome evaluation? • A matched COMPARISON or CONTROL group • Large enough sample to detect a difference between groups • A well defined target group • Appropriate follow-up period • Qualitative / Quantitative Analysis – Mixed methods approach • Independent Peer Review PROTECTED

11. For this project we would like to see • Planned approach to evaluation and collecting evidence • Systems put in place to gather relevant data systematically • Randomised control trials would be great, but unlikely • Other methods for collecting impact data, such as using the Ministry of Justice Data Lab • Impact on intermediate outcomes related to reoffending PROTECTED

12. Communicating Findings • Be clear about your methodology (sample sizes / approach) • Be open about the limitations of the design • Include details on how generalizable the findings are • Be aware of ‘The Researcher’s Curse’ UNCLASSIFIED

13. Confirmation Bias • We all have a tendency to be seek out information that supports our existing views and dismiss counter evidence • Our credibility and integrity will be enhanced by inviting scrutiny of our design and findings; • Multiple independent peer reviews • Focus on the limitations not just the positive elements of the research • Look for alternate ways to interpret the data UNCLASSIFIED

14. Get some advice • You might want to have experts in evaluation involved in your project when bidding to design and carry out an evaluation • Consider using the MoJ Data Lab • Advice on evaluations can be found on the MoJ website and gov.uk • Plan your evaluation before you start the project so you know you are collecting the right data from the start • Make sure you have a well thought through theory of change PROTECTED

15. In Summary • Research is important. The evidence gathered as part of your research will be the basis for understanding if and how your intervention has been effective • Evaluating your work is not about proving that you did it right but to enable everyone to learn more about what works for whom and what doesn’t work and why PROTECTED