Assessing the quality of evidence in realist synthesis

1. Assessing the quality of evidence in realist synthesis RAY PAWSON r.d.pawson@leeds.ac.uk

2. Systematic review: the classic ‘check list’ 1. Clarifying the question for review 2. Searching for primary studies 3. Appraising the quality of the studies 4. Extracting data from proficient studies 5. Synthesising the data 6. Disseminating the findings  + quality appraisal

3. Classic EBM standards Gold standard not achievable Gold standard not applicable. Gold standard relates only to ‘does the treatment work?’ Gold standard observed in breach as much as application New qualitative EBP standards complex abstract fragmented contradictory permissive compound pick and mix and therefore cannot act as a preliminary quality filter ‘Inclusion Criteria’ - reasons to be suspicious

4. Megan’s Law - Programme Theory STEP ONE Problem Identification Identify high-risk released sex offenders and create valid and reliable registers STEP TWO Public disclosure Issue bulletins, press releases, call meeting to identify released offenders to their community STEP THREE Sanction Instigation Community joins with police and probation to increase surveillance of suspicious behaviour STEP FOUR Offender response Community actions shame offenders and decrease opportunity of further offence

5. Study 1: did the law effect recidivism? A quasi-experimental trial Pre-intervention sample sex recidivism 22% Post-intervention sample sex recidivism 19% Pre-intervention sample arrest slow Post- intervention sample arrest significantly quicker Schram & Milloy

6. Pre-synthesis verdict on study one? • It is only ‘quasi’: Therefore uncontrolled, extraneous variable may be at work across the the two conditions • No details are given of the two conditions other than notification is absent and then present. We do not know ‘what it is about’ Megan’s Law that makes a difference Verdict    

7. Study 2- where does notification reach? a ‘prospective simulation’ 6could potentially respond to community notification 36previous offence 12 stranger predatory offences 136serious sex offences 6offender from out of state 24 known to victim 100 no previous offence Petrosino & Petrosino

8. Pre-synthesis verdict on study two? • Despite the growing use of ‘what if’ analysis in evaluation research, there are NO STANDARDS AVAILABLE to judge such a strategy • Ad hoc judgements may be used, in which case the research would probably deemed to be a mixed blessing Verdict   

9. Study 3: How did practitioners respond? Selected answers to open-ended questions “Caseloads are high, given the inordinate amount of time required in sex-offence supervision. Despite these heavy demands, agents and unit supervisors were found to be well trained and strongly motivated. The quality of supervision is high and the public is being well served by these professionals. No better evidence of this can be found in the very low recidivism rates for SBN cases.” “The Law is an unfunded mandate” “Management doesn’t understand the huge number of collateral contacts necessary for a sex offender caseload: family of defendant, victim’s family, D.A., clinician, employer and so on” “There is more pressure to baby sit with SBN cases simply because they are SBN cases” Zevitz and Farkas

10. Pre-synthesis verdict on study three? • Authors’ inferences seem to go some way beyond the sentiments uttered by the probation officials - e.g. NO evidence whatsoever is given on the proposition on ‘very low recidivism rates’ • Substantial research/subject contacts. Has the researcher gone native? Verdict    

11. Key findings synthesised Study 1: Following The introduction of the law, detection increase more sharply than deterrence. Study 2: The chances of community surveillance of stranger predatory offences remain low and offender may lie low Study 3: practitioner attention becomes increasingly focused on SBN cases because of community harassment

12. Post-synthesis verdict on the studies? • The crucial test is whether the studies are capable of warranting the more specific propositions sought in the synthesis. • Many of the stock in trade limitations of the studies do not apply limitations and they may be deemed fit for explanatory purpose Verdict    

13. Conclusions The worth of a study is determined in synthesis ‘Bad’ research can yield ‘good’evidence