Measuring criminal court efficiency using DEA

Measuring criminal court efficiency using DEA Charlie Lee Forecasting and Model Development Unit (FMDU) CJ Sig Event March 2013

Agenda • Background - Background information about the project and why we are doing it. • Overview of the DEA methodology • Overview of the main points in the Data Envelopment Analysis methodology. • Developing the DEA model for the Crown courts - A walkthrough of things to consider when developing the DEA modelling. • Next steps

Background • HMCTS currently record vast quantities of management information related to the court process. • Regular performance reports and charts are produced and updated to keep track of various measures and targets related to court performance. • However, because there is so much data, it can be difficult to determine which courts are efficient and where to focus resources on. • Project aim • To explore a new analytical method of comparing the court level performance between each court (for the Crown Courts and magistrates’ courts) using a combined range of indicators. • Identification of most efficient courts for best practice • Identification of least efficient courts for improvement. • Identification of courts that are most similar, and why they differ in efficiency. • To provide HMCTS decision makers with useful information and give a new perspective on efficiency performance to enable better use of time and resources

What is Efficiency? • Efficiency can be defined as a measure of how well inputs are converted into outputs: Process Inputs Outputs • In its simplest form, efficiency can be described as: . • This can be applied to any input and output, for example: Sandwiches = Per Hour • Papers per Author = Goals Per Game = • A unit (e.g. a university, a football team, a cafe) can then measure their relative performance in two ways: • Against other units operating in the same field, where the higher their score, the higher their efficiency. • Against a theoretical benchmark, where the nearer to the benchmark they get, the higher their efficiency.

Measuring Efficiency with Multiple Inputs and Outputs • In most cases, combining multiple single efficiency ratios can be problematic as they can include contradictory views. For example, who is more efficient, Café A or Café B? • The most simple way around this problem is ratio analysis, which works by creating a weighted single score. • Although simple, this method requires an agreement on the weighting of scores which can lead to disagreements between units under investigation. • More advanced techniques such Data Envelopment Analysis can avoid this problem.

Data Envelopment Analysis (DEA) • Data Envelopment Analysis (DEA), also known as frontier analysis, is a linear programming methodology to measure the efficiency of multiple Decision Making Units (DMUs) when the production process presents a structure of multiple inputs and outputs. • Can handle multiple inputs and outputs • Relative peer to peer measure • Non Parametric approach Results • Gives a single efficiency score, although multiple units are likely to be 100% ‘efficient’. • Shows target input production and/or output consumption required to achieve efficiency. • Shows potential role models (efficient peers) for comparison.

How DEA works Stage 1 - Apply weights (ui,vi) to each of the input (xi) and outputs (yi) for the first unit. DEA selects the weights that maximises the units’ efficiency, which eliminates the need to create an agreed set of weighting between units. Outputs (v1) * Sandwiches sold (y1) (v2) * Tea sold (y2) Inputs (u1) * Number of Staff (x1) Café X Stage 2 – Apply the same weights to all other units and see if they can achieve a better rating. This can be a single unit or a selection of units to make a theoretical achievable unit point. Café B Café A Theoretical Cafe Stage 3 – Efficiency scores are created based on the difference between the efficient unit and the unit under investigation given the weighting Stage 4 – Repeat from stage 1 with next unit

Graphical Illustration Given an example with two outputs and a single input, an efficiency frontier can be created by drawing straight lines between units that are 100% efficient. In the example below this is Café A, Café B and Café C. It is then assumed that all points on this line are achievable and 100% efficient. When evaluating a units’ efficiency that does not lie on the frontier, a straight line can be drawn to the origin point. In the example, this is represented by the combination of the solid and dashed line between Café X and (0,0). Theoretical Achievable Point Café A The efficiency of Café X is therefore the ratio of the distance between 0 and Café X over the distance between 0 and the theoretical achievable point. Café B Café X Café C 0

DEA Formulation The model appeared first in ratio form, but can be converted to its linear form, making it easier to solve. It works by maximising the objective by varying the weights, but ensuring that applying those same weight to every other units will not lead to a score greater than 1. Ratio Formulation Linear Formulation Max Max Subject To Subject To

Orientation of DEA Input Orientation Attempts to minimise the inputs given the outputs Output Orientation Attempts to maximise the outputs given the inputs e.g. Keep the café’s staff numbers the same, but sell more sandwiches and teas to the regulars or attract new customers. e.g. Use fewer staff to serve the café’s regular customers who always order the same sandwiches and teas.

Economies of Scale Constant Returns to Scale (CCR) Assumes that economies of scale have no impact and everyone operates under the conditions that creating 100 units of outputs i requires exactly 100 times the effort as creating a single unit of output i. Variable Returns to Scale (BCC) Assumes that economies of scale have an impact on units, therefore acknowledging that smaller and larger units outputs may benefit or be affected by the units size. This will always lead to more units becoming 100% efficient. e.g. Decide if size affects how a café operates. A larger café may be more difficult to manage due to its size, but can buy its goods can in bulk. Small cafés may be too small to operate at optimal efficiency, but maybe better at retaining customer loyalty.

Developing DEA modelling for the Crown courts

Overview of the Criminal Courts in England and Wales Recorded Crime Out of Court Disposals Offence Detected CPS CPS discontinue Magistrates’ court Mags court proceedings Committed for Trial Sent for Trial Not guilty/ acquitted Guilty Committed for Sentencing Crown court Mags court Sentencing Appeal

Inputs and Outputs Selection • Possible Input measures

Inputs and Outputs Selection • Possible Output measures

Inputs and Outputs Selection Issue • The DEA method, in common with all linear programming are extreme point techniques. • The results are therefore sensitive to all inputs and outputs selected, as all variables can be seen as important as each other. • The more inputs and outputs selected, the more 100% efficient units there are. Solution • Careful selection of inputs and outputs. • Work closely with customers to determine priorities and include only a few of the most relevant inputs and outputs. • Check the data availability and quality is feasible for the analysis.

Investigating the case mix effect • The different types of cases tried in court will vary in their complexity as indicated by their average court duration time. • Due to regional variations, some courts may deal with more of a particular type of case, their case mix will vary. • e.g. Some courts may only deal with serious cases such as murders, whilst others may deal with more burglary and handling stolen goods. • Therefore, we need a method to fairly compare courts given their case mix. The average court duration times shown are for illustration only

Investigating the case mix effect • To deal with the case mix effect, we have created a weighting system by sub offence level based on the average court duration time for a particular sub offence compared to the average case in England and Wales. • For example, if the average court case is given a weighting of 1 point, then a Murder is worth 13 points, as the average court duration for a murder case may take 13 times longer than an average case. • Using this weighting system enables the courts to be fairly compared against each other when we consider their volumes of disposals as an output. The figures shown above are for illustration only

Crown Court Investigation To illustrate the sensitivities when selecting the parameters for DEA modelling, the following simplified models will demonstrate graphically how the results can vary. The models are run using an output orientation on the assumption that inputs are fixed by HMCTS and given these inputs they should be producing a certain amount of outputs. Model 1: One input and one output (Constant returns to scale) Inputs Sitting Hours Outputs Disposals Model 2: One input and one output (Variable returns to scale) Inputs Sitting Hours Outputs Disposals Model 3: One input and two outputs (Variable returns to scale) Outputs Disposals Early Guilty Pleas Inputs Sitting Hours

Model 1: one input and one output Efficiency Line DMU 4 DMU 3 DMU 2 DMU 1 The points on the chart represent the 76 Crown courts (DMU’s), disposals as an output are plotted against an input of sitting hours used. This is a standard ratio analysis model to illustrate the differences in results when scale is not considered.

Model 1: one input and one output DMU 4 is far away from the efficiency line and relatively not very efficient in this model. DMU 4 DMU 3 DMU 2 DMU 1 We can see the most efficient unit is DMU 1, and those furthest away from the efficiency frontier are considered to be the least efficient.

Model 2: One input and one output (Variable returns to scale) DMU 4 DMU 3 DMU 4 now lies on the efficiency frontier and is relatively 100% efficient compared to peers in this model. DMU 2 DMU 1 Using the same data points, if we choose a variable returns to scale approach, the efficiency frontier changes as more units are now captured as being most efficient. The efficiency score for the other units also change as efficiency is relative to the efficiency frontier.

Model 3: One input and two outputs (Variable returns to scale) DMU 4 • Using the same data points as before, we include an extra output measure (early guilty pleas) into the model. The same efficient units from the previous model are still efficient, but we now have a few more efficient units (DMU 5, 6, 7), the next chart shows why they are included. DMU 3 DMU 2 DMU 7 DMU 1 These courts (DMU 5, 6, 7) are not on the frontier for disposals but are considered 100% relative to peers. DMU 6 DMU 5

Model 3: One input and two outputs (Variable returns to scale) DMU 7 DMU 6 • The new efficient units (DMU 5, 6, 7) are efficient because they lie on the efficiency frontier for early guilty plea. This is an example of DEA using extreme points to consider both outputs simultaneously to give an overall result. DMU 2 DMU 5 DMU 3 DMU 1 DMU 4 performs relatively poor for EGP, but is considered 100% efficient relative to peers, when both outputs (disposals and EGP) are considered. DMU 4

In summary… • These basic models have illustrated that careful consideration must be given to select the appropriate parameters, input and output measures to include in model development. • Although the modelling is still developing the initial reaction from our customers have been very positive, they have been engaged in our meetings and very interested in the information this methodology can provide. • The resulting modelling will hopefully help our customers gain greater insight into their business and provide an alternative perspective on court performance, neither of which have previously been possible with the existing MI tools and performance reports

Next steps… • Continue to develop and refine the Crown court DEA modelling • Explore and evaluate the results from the selection of inputs and outputs • Drill down to the court level to find out what is driving the efficiency scores • Explore using weight restrictions for some outputs and evaluate the results • Apply a similar methodology to the magistrates’ court. • Magistrates’ court has over 220 courts across 7 regions in England and Wales • Possibility to model at regional level • Slightly different priorities and measures • Thank you!

Measuring criminal court efficiency using DEA