1 / 22

Evaluating methodological quality in the criminal justice system literature

Evaluating methodological quality in the criminal justice system literature. Dr Amanda Perry. Centre for Criminal Justice Economics and Psychology, University of York. Overview of the session:. Background to RCT in criminal justice Quality assessment

erasmus-hutchinson
Download Presentation

Evaluating methodological quality in the criminal justice system literature

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluating methodological quality in the criminal justice system literature

  2. Dr Amanda Perry Centre for Criminal Justice Economics and Psychology, University of York.

  3. Overview of the session: • Background to RCT in criminal justice • Quality assessment • The Maryland Scientific Methods Scale (SMS) • Problems associated with the SMS • Where next…….

  4. Background • Few RCT conducted in the UK • Systematic review revealed 125 conducted between 1957-2005 (Farrington & Welsh, 2005) • Concurrent findings from UK systematic review between 1990-2002 (Perry, McDougall & Farrington, 2005)

  5. Resistance to RCT… • Historical resistance • Ethical and moral reasons • Practical difficulties BUT………... New era Campbell Collaboration (2000) Department of Health/Home Office (2000)

  6. New age of RCT… Feasibility studies (Farrington & Joliffe, 2002) (Farrington et al., 2002) Matrix (2006) – Criteria for assessing feasibility • Current RCTs • Evaluation of Cognitive behavioural skills programmes(McDougall, Bowles, Perry & Clarbour, ongoing). • Evaluation of Restorative Justice Programmes (Strang & Sherman, 2006)

  7. Quality assessment… • Reviews of quality assessment tools: The medical field • Moher et al. (1995) identified 25 scales devised up to 1993. • Juni et al (1999) compared 25 scales for purpose of inter-rater reliability (r=.72).

  8. Quality assessment… • Reviews of quality assessment tools: The • Social Sciences • Gibbs (1989) – social work • STROBE: STrengthening the Reporting of OBservational studies in Epidemiologyhttp://www.strobe-statement.org/index.html • Maryland Scientific Methods Scale – Criminal Justice (Sherman et al., 2002)

  9. Quality assessment… • The Maryland Scientific Methods Scale (SMS) • (Sherman et al., 2002, based on work by Cook & Campbell, 1979) • Purpose of the SMS: • Simple (measuring internal validity) • To provide policy makers with information about the evidence • Aim to classify all programmes into 1 of 4 categories

  10. The SMS quality assessment.. Five point scale: Rating 1-5 Rating 1: Correlational study Rating 2: Pre and post test study Rating 3: Observational cohort with comparable group Rating 4: Quasi-experimental/controlled trial Rating 5: Randomised controlled trial

  11. The SMS quality assessment.. • Statistical conclusion validity • Was the statistical analysis appropriate? • Did the study have low statistical power to detect effects because of small samples? • Was there a low response rate or differential attrition? • Construct validity • What was the reliability and validity of measurement of the outcome?

  12. Evidence and use of the SMS: • What Works: Evidence from 2 or more studies reporting positive results scoring 3 or above on the SMS showing statistical significance and desirable effects and the preponderance of all available evidence showing effectiveness. • What Doesn’t Work: Evidence from 2 or more studies reporting negative results scoring 3 or above on the SMS…

  13. Evidence and use of the SMS: • What ‘s Promising: Reporting evidence from 1 study reporting positive results scoring 3 or above on the SMS…. • What Unknown: Evidence from 1 study with a negative or inconclusive result scoring 3 or above on the SMS…..

  14. Use of the SMS: ‘What works – an example’ • CCTV in car parks (evidence from Welsh and Farrington, 2002) • Street lighting (evidence from Painter & Farrington, 1997; 1999b;2001b; Farrington & Welsh, 2002) • Burglary reduction schemes (evidence from Ekblom, 1996a; 1996b)

  15. Limitations of the SMS… • Not fully assessing all threats to the validity of a study • Categorising study designs • Does not take into consideration at What cost?

  16. Limitations of the SMS… • Is designed to apply equally to all experimental units • Does not embrace all study designs • Method of drawing conclusions on what works based on statistical significance rather than effect size

  17. Improving the SMS…. • Farrington (2003) • Based on five key criteria: • Internal validity • Descriptive validity • Statistical conclusion/validity • Construct validity • External validity

  18. Improving the SMS … Information for policy makers 3 five point scales • Design • (internal validity) • Execution • (construct validity/statistical conclusion validity /sampling elements of external validity) • Reporting of the trial

  19. Beyond the SMS….. • Assessing the cost of an intervention • Adequate and standardised follow-up periods for outcome measures • (e.g., reconviction rates) • Encouraging journal editors to use a new scale/standard similar to CONSORT statement

  20. Beyond the SMS….. • Development of a specific quality assessment tool evaluating RCT in the criminal justice • To incorporate all elements of validity • To include an assessment of cost/cost effectiveness/cost-benefits of an intervention • To include guidance on adequate and standardised follow-up periods for outcome measures • (e.g., reconviction rates)

  21. SUMMARY • RCT rarely used in UK criminal justice system • Use of the SMS as quality measure • Room for improvement • Development of a new scale….

  22. CONTACT: Amanda Perry Centre for Criminal Justice Economics and Psychology University of York Email: aep4@york.ac.uk

More Related