Assessing what works research: What is good enough ? Jennifer Brown Mannheim Centre LSE

1. Assessing what works research: What is good enough? Jennifer Brown Mannheim Centre LSE J.Brown5@lse.ac.uk Paper for Canterbury Christchurch University Conference Evidence based policing: Beyond the RTC 22nd -23rdJune 2016

2. Starting position • “It is difficult to imagine anyone arguing that policy should be based on anything but the best available evidence” (Marston and Watts, 2003:144), • “rooting policy in evidence has all the appeal of motherhood and apple pie.” (Tilley and Laycock 2000:3)

3. Why does it matter • Exclusion of much relevant evidence • Justice and human rights • Narrowness of questions

4. Definition of EBP • Strict definition • research findings resulting from a medical model of randomised control trials and excludes those from non-scientific studies not achieving the gold standard of excellence of the RCT • Broader definition • drawing on a wider rand of inputs including practitioner experience and user consultations

5. Purposes • Decision making-Sherman • Change-Stanko • Legitimation-Weisburd and Neyroud • Tool-kit-Hough

6. Dramatis Personae • Strong experimentalists-police scientists • Quasi experimentalists • Crime scientists • Constructionists • Pragmatists

7. Maryland Scientific Scale • Level one - simple correlational analysis and represents the lowest level as there is no means to establish cause and effect (reconviction measured for intervention group only) • Level two - before and after study but which cannot rule out possible confounding effects (comparison of actual and predicted reconviction for intervention group only) • Level three- compares an experimental and comparable non experimental area at the same point in time (comparison of reconviction rates from treatment and unmatched controls) • Level four -multiple experimental and comparable control areas ( comparison of reconviction rates from treatment and controls matched on theoretically relevant factors) • Level five - RCT random assignments to the experimental and control conditions. (comparison of the reconviction rates from treatment and control groups with randomisation to groups)

8. Holy Grail? • There is "no gold standard methodology" (Laycock, 2005:8) • There is no toolbox to be had containing "a set of experimentally validated policy spanners” (Hough 2011:3).

9. “Wicked” Problem Quest • Who are the stakeholders? • What are the issues? • What are the constraints? • What are our assumptions? • What are the key decisions you have to make? • How will you make them?

10. More questions? • How does it work? • Why does it work? • On whom does it work or not work? • Are the changes sustainable? • Is it fair? • Is it transferable? • How do participants feel about it? • Are there any unintended consequences? • What do the general public think?

11. Accounting for differences • Epistemological bi-furcation • Within discipline fragmentation • Competition for ascendency - a kind of disciplinary imperialism. • The advent of the policy entrepreneur

12. Contingent repertoires • Pawson and Tilley • Quasi experimental paradigm has resulted in moribund evaluation being itself a contributory factor to the nothing works lament (p91) • Bennett • Their claims that we have learnt nothing from experiments ..seem unnecessarily nihilist and dismissive (p570)

13. Policy entrepreneur • Policing is political (Reiner, 1980) • Scientists are political (Bissett, 1972) • Both have policy entrepreneurs (Kingdon, 2003) • Someone willing to invest resources, energy and reputation to promote a particular approach

14. Politics of police research • Consensus • Controversy • Conflict • Contradictory • Crime control • Collaboration • Combative } extensions to the Reiner classification

15. Politics of Criminal/Forensic Psychology Research • Early accord • Parting of the ways • Not on speaking terms • Return to cordiality • Business as usual

20. Conclusion • "Like many wars, the methodological paradigm wars in criminology (and throughout the social sciences) may be based primarily on misunderstandings of what each side is really about. Quantitative researchers are often dismissive and condescending about qualitative research. Likewise qualitative researchers can spend a great deal of time criticizing quantitative research… when the generation that fought the qual v quant paradigm wars of the late twentieth century passes into retirement, I imagine that few of their successors will likely remember or understand what the fighting was all about." (Maruna, 2010:137)

21. Take home Messages • New wine old bottles • Problems with the Medical model • The dialogue of the deaf • Research imperialism • Some alternatives • Solving the wicked problem

22. The alternatives • A medical model using scientific experiments, the Maryland Scale and accepting only evidence approved by the Campbell Collaboration • A multidisciplinary model offered by Crime Science • A social science model comprising either/or both psychologists and criminologists offering a range of methods to suit the question • A mixed methods integration of quals and quants • A legal model resolving particular cases and developing a set of standards and principles • Other models such as pragmatic psychology which proposes the creation of a peer reviewed data base of systematic, rigorous, solution focused case studies.

23. A futile task?