Better Policing Collaborative Police Knowledge Fund Introduction to Evaluation

1. Better Policing Collaborative Police Knowledge FundIntroduction to Evaluation Dr Susan Giles Sabina Enback & David Leather University of Liverpool Skills for Justice

2. BPC training goals within PKF • Objective 1: Workshop and seminar series funded by HEFCE • Aim is to help police staff develop the core skills and knowledge required to carry out EBP research and evaluation tasks. • Mutual exchange:- we can learn about the issues and problems you experience in applying EBP to your work

3. Overview for today • What is evaluation research and why should we be interested in it? • When to evaluate • What to evaluate • Common evaluation techniques • Do we need an evaluation partner or can we do it ourselves? • The role and value of data • How to make use of findings

4. Why evaluate? Evaluation • Services may become irrelevant or ineffective if we do not ask challenging ‘so-what?’ questions. This is not cost effective. • SERVICES LOSE FUNDING if they cannot evidence outcomes. Focus on outcomes helps to keep us accountable to service users, funders and society • We do not know how or why an intervention has worked if we do not evaluate • Interventions and services can be improved as a result of evaluation What is evaluation? A way of assessing whetherobjectives havebeen achieved A way of assessing theoutcomes of services forand with people whouse them A way of learning from successes and mistakes A way of publicising your work and sharing what you have learnt with others

5. What is EBP? • A ‘new normative value’ (Welsh and Harris; 2013:174) • Understanding ‘what works’ in responding to crime trends and social problems • Using robust research evidence to design, develop and implement innovative and effective solutions to prevent and reduce crime • Making the most efficient use of police resources

6. Sherman (1998; 2013) TARGETTING • Police should conduct and apply good research to target scarce resources on predictable concentrations of harm from crime and disorder. TESTING • Once police choose their high-priority targets, they should review or conduct tests of police methods to help choose what works best to reduce harm. TRACKING • Once police agencies use research to target their tested practices, they should generate and use internal evidence to track the daily delivery and effects of those practices, including public perceptions of police legitimacy.

7. Evaluation methods

8. Purpose of different evaluation questions Outcome questions: Whether your service has brought positive change to your users Outcome or Effectiveness evaluation Impact questions: What are the long term consequences for services and service users? Impact evaluation

9. Purpose of different evaluation questions Process questions: Whether you are providing the right quality services to the right people. Also considers how an intervention works Process or Efficiency evaluation Economic questions: Do the benefits of the intervention outweigh the costs? Is the intervention an effective use of resources? Economic evaluation

10. Evaluating evidence For what works in crime reduction, college of policing and UCL funded by ESRC have developed EMMIE 1. Effect size/s produced (the magnitude of any impact on crime), 2. Mechanism/s activated (how interventions work), 3. Moderators/contexts for the activation of the mechanism/s, (where it works best) 4. Implementation conditions that supported/obstructed delivery (how to do it) 5. Economic assessment of interventions (what it costs).

11. Critical issues in outcome evaluation Outcomes are the changes, benefits, learning or other effects that happen as a result of your work. They can be wanted or unwanted, expected or unexpected.

12. 6. 4. How evaluate? Six steps to the evaluation cycle – shared by psychological, health and community interventions. Reflecting on practice, learning from and sharing the findings 1. Setting measurable outcomes Evaluation design is central to intervention design 2. Selecting indicators and identifying sourcesof evidence 5. Analysing and presenting evaluation data Choosing methods and collecting and recording data 3.Planning intervention/evaluation

13. End point for service users – expressed as positive statements of change Four stages inoutcome setting Choose indicators: Qual/Quan measures to help evidence achievement of outcomes Stage 1: Identify the intended long-term outcomes of your work for service users Stage 2: Plan measurable outcomes for each of these long-term outcomes Think S.M.A.R.T, specific, measurable, achievable, realistic, time-based Stage 3: Decide on the type of service or intervention Which service/intervention can best deliver outcomes? Stage 4:Plot milestones Outcomes are not achieved immediately, c.f. stages of change model

14. OUTCOMES are defined as POSITIVE CHANGES for SERVICE USERS, ie changes to behaviour, cognition, feelings and awareness; sometimes referred to as ‘distance travelled’ Measured by indicator measures (variables) taken pre intervention (baseline) and post intervention. Example of single case research design to demonstrate whether programme has enabled change within individuals If you have no baseline measures – you need to establish evidence of behavioural changes as a result of intervention Inputs = The resources used to create the services offered – money, time, staff, premises Outputs = The services and products made available to service users –groups, sessions, activities

17. Design: Hierarchy of evidence c.f. extent to which funders and commissioners will fund future service provision based on hard (objective) and soft (subjective) outcomes Basic evaluation design: Between groups design with n outcomes This is the gold standard of efficacy evidence: Systematic reviews/meta-analyses of RCTs Level of evidence largely dependent in practice upon extent to which intervention has been developed Observational studies; with and without a control group

18. Embedded CPS lawyers in RIUs Discuss potential data sources that could be suitable measurable outcomes

19. Critical issues in process evaluation

20. Process Evaluation ....is about what makes a programme work. It defines what the conditions for success are. What needs to be done to achieve outcomes? • Did the programme ‘reach’ the right service users? • Is the programme successfully meeting a need in the local area? • Is there demand/interest in the programme? • Are the referral routes appropriate, used, successful? • Did the programme meet its objectives? • To what extent were the activities conducted according to the proposed timeline? By the appropriate personnel? • To what extent are the actual costs of the project implementation in line with initial budget expectations? • To what extent are the participants moving towards the anticipated goals of the project? • Which of the activities are aiding service users to move toward the goals? • What barriers were encountered? How and to what extent were they overcome?

21. Process Evaluation • Data is often qualitative and archival: • Stakeholder interviews • Focus groups • Observation • Documentary analysis (minutes of meetings, monitoring and progress reports) • Session feedback forms • Visual records • Reflective diaries • Process evaluation can identify conditions of success and failure which can feedback into successive and iterative development of intervention.

22. Black box approach ....is about what makes a programme work. It defines what the conditions for success are. What needs to be done to achieve outcomes? • While outcome and impact evaluation is essential to establish whether something works it does not address the questions of ‘why’ and ‘how’ – leaving an unopened ‘black box’ between inputs and outcomes Context Inputs Outcomes Intervention Mechanisms

23. Economic evaluation and cost benefit analysis

24. What is economic evaluation and why should we be interested in it? • Economic evaluation is an appraisal of the economic implications of alternate policies/interventions. This appraisal involves identifying, measuring and valuing costs and benefits of two or more interventions. This then allows for comparing these interventions. • This is of importance as policing budget is limited and one needs to understand what represents best value for money (best bang for a buck!) • Economic evaluation essentially means doing a cost benefit analysis

25. What does one need to do for a cost benefit analysis? • Need to identify the effect (causal identification of an intervention) actually occurs because of the intervention before one can do cost benefit analysis • This is important as several variables are changing at the same time and thus one has to be careful that one is not picking up a simple correlation which is not causative

26. Once we have causal identification we are in a position to calculate cost and benefits for the policy/intervention. • The first thing to keep in mind is that there are both explicit and implicit costs. • Explicit costs are actual costs of time, materials, training, etc. • Implicit costs are costs of not being able to do an intervention or time taken away of perusing an alternative. • Benefits can be seen as gains or as savings • Understand that costs and benefits have a static as well as dynamic component and learn to convert future costs and benefits into today’s monetary value – we call this net present value (NPV).

27. Cost benefit analysis of Lifeline • Costs:- • What costs do you envisage for this invention? • How might they be measured? • Benefits:- • What benefits do you envisage for this intervention? • How might they be measured?

28. 1. Costs to service providers · Capital - land - buildings - equipment · Running costs - paid staff - volunteers - administrative and managerial costs - consumables such as any equipment that’s needed to deliver any of the sessions, costs of computer software licenses, staff travel costs etc. 2. Costs to the offenders and their families in treatment · Out of pocket expenses - travelling and other direct expenses - any contribution to treatment costs · Leisure time and other costs associated with input to treatment Costs • pain, distress etc. associated with changing habits, or with process of treatment Lifeline costs = inputs

29. 3. Costs to other agencies or individuals · Referrals to other health or social agencies linked to the treatment · Increases in potential problems associated with treatment 4. Productivity costs - costs of lost productivity for staff, offenders and families during the time spent on the program Lifeline costs = inputs

30. Lifeline benefits = reduced spending for various agencies

31. Lifeline benefits = reduced spending

32. Cost benefit ratio = £1:£7First cohort has completed intervention – post measures will soon be gathered

33. How does evaluation shape strategy? • Police work with limited budgets (as do other crime prevention agencies) hence efficient use of resources require that only interventions with positive outcomes and NPV be used • In practice, there are other constraints that can operate: e.g. political ones-e.g. a particular policy is popular with citizens who may protest if it is withdrawn • Evaluation helps inform citizens by providing evidence for why we choose one intervention over another and thus may make change feasible

34. Do we need an evaluation partner or can we do it ourselves? There are strengths and weaknesses of internal versus external evaluations: • What are you evaluating (in force or partnership working? • Who is the evaluation for? • Skills/Knowledge • Budget considerations (10-20%)

35. Role and value of data

36. Evaluation Design, Data and Social Value • Understand how to write an evaluation protocol. You will discuss and formulate an evaluation plan for an intervention that you are working on • Further critical issues in analysis and presentation of data • Introduction to Social Return on Investment

37. A Short Course in Evidence-Based Policing • University Campus Suffolk (2016-2017) • Learning outcomes from two day course • an understanding of the principles and practices of evidence-based policing; • an ability to apply its principles and practices to real-life situations; • an analysis of a crime problem including causation and the limits of current attempts to address it; • knowledge of policies and legislation relevant to a crime problem; • an intervention appropriate to addressing a crime problem and justify its appropriateness for use.

38. Questions?