From sifting to synthesis

1. From sifting to synthesis Evidence Base Camp: February 2014 Levin Wheller Practice Development Team Research Analysis and Information Unit

2. Recap on REAs While you were away… Peer review of sifting and the task ahead Mapping research evidence Systematic Mapping – the what and the why An example: Mapping Evidence on Public Order Tactics Appraising research evidence The Maryland Scale of Scientific Methods Some hypothetical examples Beyond Maryland: non-impact questions Qualitative methods Completing a pro forma Practical group work This session…

3. Recap

4. Explicit objective/ question Studies are selected on explicit criteria Thorough search for studies Each study is screened according to uniform criteria Reasons for excluding studies are clearly documented Systematic reviews and “what works”

5. Use systematic principles and the same process as a systematic review… …but make compromises given available time and resources Pragmatic and transparent approach Rapid Evidence Assessments

6. Demonstrate consistency in searching/ sifting Document search and sift process Process should be transparent and repeatable Specify required quality of evidence Systematic reviews only? Pre-post studies only? All ‘empirical’ papers? Be explicit/ transparent about the limitations of the approach Some key principles

7. The process (in a nutshell) Draft search terms Draft sift criteria Sift received abstracts Request full papers Read and ‘grade’ papers Write it up (‘synthesis’)

8. SiftingIdentifying truly relevant literature SearchingIdentifying potentially relevant literature SynthesisSummarising what relevant literature tells us

9. SearchingIdentifying potentially relevant literature

10. The Basics Boolean operators • Boolean is a powerful tool and can be used to create quite complex searches. The standard Boolean operators are: • AND • OR • NOT (or AND NOT) (advanced users only!) • Boolean is most effective when used in combination with: • PARENTHESES (round brackets) • TRUNCATION e.g. politic* • WILDCARDS e.g. randomi?e • DOUBLE QUOTES for specific phrases

11. Search terms ((((Domestic OR Familial OR Marital OR Spousal OR Partner) Near/2 (Abus* OR violen* OR control* OR coerc*)) OR "domestic homicide" OR "marital rape" OR "intimate partner violence" OR (coercive NEAR/2 (control OR behavio*r*))) AND (risk AND (predict* OR indicat* OR factor* OR management OR identif* OR assess* OR prevent* OR likelihood OR probability)) AND ("systematic review" OR "literature review*" OR trial* OR "RCT" OR experiment* OR evaluat* OR "best practice*" OR "good practice*" OR effective* OR assess* OR "What Works" OR impact* OR success*)) Riskc 1m papers DA/ DVc 75k papers ‘What works’c 2m papers Domestic violence search Initial search identified 1,531 abstracts to sift

12. Searching as a ‘blunt tool’ Research Question: What interventions have been shown to be effective in preventing or reducing theft from the person? So how do we end up with an abstract titled… Profile on circadian blood pressure and the influencing factors in essential hypertensive patients after treatment Theft from person synonyms include: Robbery, Mugging, Dipping, Pick pocketing, Assault with intent to rob, Snatching, Theft from person, Bag theft, Bag snatch, Personal theft…

13. Returned by theft search… Yuan Gang Qiu et al. Profile on circadian blood pressure and the influencing factors in essential hypertensive patients after treatment Non-dippers were defined as those whose nocturnal decrease in mean systolic BP and/or mean diastolic BP was < 10% of the daytime BP. Binary logistic regression analysis was used to evaluate the correlation between circadian blood pressure profile and factors as gender, age, height, body mass index (BMI), family history of premature cardiovascular disease, women under age 65 or men under age 55, smoking habits, grade of hypertension, and strategy of antihypertensive drugs. Similarly, the incidence of non-dippers in patients of overweight (24 </= BMI < 28) and obesity (BMI >/= 28) were 3.0 and 4.8 times of those in subjects of normal weight (P = 0.003 and 0.009, respectively). Compared with patients treated with long-acting calcium channel blockers (CCBs), patients treated with angiotensin-converting enzyme (ACE) inhibitors or angiotensin II receptor blockers (ARBs) alone had less prevalence of nondippers (OR = 0.139, P = 0.010).

14. SiftingIdentifying truly relevant literature

15. The same set of questions are applied to all abstracts to attempt to remove bias/ overt ‘judgement calls’. Typically, a lot of papers (up to 95%!) identified by the search are excluded at this stage of the process. Sift criteria

16. While you were away…

17. Peer Review: Second sift… ‘Red’ abstracts ‘Dip sample’ taken, e.g. one in 10 ‘red’ abstracts were peer reviewed Check for ‘false-exclusions’ ‘Green’ and ‘Amber’ abstracts All resifted to review against the inclusion criteria Check for ‘false-positives’ Decide where possible on ‘Ambers’ Final list of included studies sent to library…

18. Second sift… • Always expect to reduce the number of studies substantially through second sifting • Standard process for all REAs and SRs • Dual coding and often used and disagreements discussed between coders to make a final decision • Clarify queries/ refine criteria as a group • Speed of EBC does not enable all of this!!!

19. While you were away…

20. SynthesisSummarising what relevant literature tells us

21. Systematic Mapping… Methodology developed by EPPI-Centre Useful first step of synthesis process Mapping can: Describe the nature and coverage of research in the topic area Identify gaps in the research literature Describe the design of studies and direct further review work

22. Systematic Mapping… What does it involve: Collection of key information about each study into a pro forma/ template

23. Systematic Mapping… Collate pro forma results into a searchable bibliographic database

24. What is the evidence on Public Order Tactics?Country of focus

25. What is the evidence on Public Order Tactics?Research Method

26. Public Order Tactics - Mapped Overall: There is only limited research available on public order policing tactics. Only 38 papers were found to be relevant and based on empirical methods. Age: Most of the research (32/38 papers) has been published since 2000. Methods: Most papers used qualitative research methods, typically involving observations (n=19) or interviews (n=18). Only one study used experimental methods.

27. Public Order Tactics - Mapped Unit of study: Only five studies focussed directly on public order tactics. Instead, studies primarily focussed on an event or series of events (n=17) or the police more generally (n=10). Tactics: Of the 30 approved public order tactics, only nine were mentioned in the research evidence identified for this map. The most frequently mentioned tactics were containment (n=5), Police Liaison Teams (n=3) and CS smoke (n=3).

28. Appraising research evidence

29. Maryland Scientific Methods Scale (SMS).(AKA Maryland Scale) Developed by Sherman et al (1997) Established method for judging the quality of impact studies Useful for judging how much weight to give different studies identified in a ‘what works’/ ‘what’s effective’ REA Limited use for judging qualitative studies and questions asking ‘how’ things work

30. What does ‘good’ or ‘robust’ ‘what works’ evidence look like? Statements about ‘what works’ Statements about ‘what’s promising’ Study designs increasingly rule out potential alternative causes Study designs cannot rule out potential alternative causes Statements about possible impact

31. “Two months after the introduction of mobile information, officers were spending 55% of their time out on patrol” ‘Level 1’ Evidence Problems? • Can’t rule out other explanations • Time spent on patrol before mobile info? • Is 55% good/ bad/ average? • No before/ after measures • No comparison group Can’t say if mobile information makes a difference/ is worthwhile

32. ‘Level 3’ evidence We still can’t be sure that improvements are down to the intervention rather than other factors “Two months after the introduction of mobile information, officers in the Western District had increased their time on the street by 7 percentage points, while officers in the Eastern District had seen an increase of 4 percentage points…” Problems? • Still can’t rule out other explanations for change • Can’t control for differences between the two areas – different culture, different case load, management, etc.

33. RCTs - ‘gold standard’ evaluation design 2,000 officers in force 300 officers in treatment group Outcomes measured Get mobile data A random sample of 600 officers 300 officers in control group Outcomes measured Business as usual

34. 1. Participants are randomly selected from the population and randomly assigned to each group to minimise the chance of bias ‘Level 5’ Evidence - RCTs “Two months after the start of the trial, officers with access to mobile data were spending 55% of their time out on patrol, whilst officers in the control group were spending 43% of their time out on patrol” 2. Randomisation helps us rule out other potential explanations for differences in outcome measures 3. A control group means we can compare differences in outcomes between groups where the only difference is the intervention (e.g. mobile data 4. RCTs enable us to make strong statements about cause and effect We can say that officers with access to mobile data spend more time on the street… in this context.

35. Systematic reviews Overall, legitimacy interventions resulted in a large, significant increase in positive perceptions of police. Taken from: Mazerolle, L., Bennett, S., Davis, J., Sargeant, E. and Manning, M. (2013) Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence. Campbell Systematic Reviews 2013:1.

36. What does ‘good’ or ‘robust’ ‘what works’ evidence look like? Statements about ‘what works’ Statements about ‘what’s promising’ Study designs increasingly rule out potential alternative causes Study designs cannot rule out potential alternative causes Statements about possible impact

37. The Maryland scale is focussed on impact questions… and studies might answer a range of other questions… Beyond Maryland… • How ‘good’ a study is will depend upon how appropriate the methods used are for answering the research question… • Qualitative studies don’t fit into the Maryland dichotomies • You can have different quality qualitative work as well

38. Frameworks to assess/ appraise the quality of qualitative research use ‘principles’ rather than hard-and-fast rules Cabinet Office e.g. in 2003 sets key principles being that qualitative research should be: • contributory in advancing wider knowledge • defensible in design • rigorous in conduct • credible in claim Quality of Qualitative Research http://www.civilservice.gov.uk/wp-content/uploads/2011/09/a_quality_framework_tcm6-38740.pdf

39. Mapping the evidence in our 5 areas… Presenting key info in charts Drawing initial conclusions from studies in each area Some studies will not be impact studies Presentation of findings to NPLs and senior staff across the service… What we are doing at EBC

40. Completing a pro forma • 1. Basic info should be recorded • Reference (title/ author) • Date of publication • Country • 2. Check if the study should be included • Check the full paper against the inclusion criteria • If the study meets the inclusion criteria, complete the rest of the pro forma • If not, exclude the study and note your reasons why

41. Synthesis in practice Complete a pro forma for 2 articles 1. GMP study summary – 15 mins 2. An included article for your area – 15 mins Look at the methods of the study and try to ‘score’ it on the Maryland Scale Ignore the section on inclusion criteria Cover findings/ conclusions if you have time but the methods are your focus Some examples might not fit the scale…

42. Synthesis in practice GMP study summary 10 mins to individually complete as much of the pro forma as possible 5 mins to discuss with neighbour/ compare findings Group feedback

43. GMP study summary (1/2) The GMP procedural justice training experiment: Wheller et al (2013) UK Policing Test the impact of training on officer attitudes, officer behaviour and victim satisfaction Training 576 Random

44. GMP study summary (2/2) Training had a positive impact on officer attitudes, officer behaviour and victim satisfaction Training works (in this context!)

45. Synthesis in practice Included study from your area 10 mins to individually complete as much of the pro forma as possible 5 mins to discuss with neighbour/ compare findings Group feedback

46. Theft study summary (1/2) Does Heroin Prescription Reduce Crime?: Killias & Rabasa 1997 Policing/ Health Switzerland Study of the impact of heroin prescription programme on crime rates in Switzerland Drug prescription/ treatment 319 Non-random

47. Theft study summary (2/2) Number of outcomes – comparison of rates of (self reported) crime pre and post intervention. Heroin prescription reduces (self reported) criminality. Police statistics also show reductions in crime.

48. Wellbeing study summary (1/2) Mental, physical, and behavioural outcomes associated with perceived work stress in police officers: Gershon et al 2009 USA Policing Study links between perceived stress and impact of ‘coping’ on work stress and health None – non-impact study 1,072 Non- random

49. Wellbeing study summary (2/2) Outlines a number of statistical relationships drawn from the survey, e.g. between job stressors and work stress “results underscore the need to re-evaluate police training of recruits… to ensure they get training necessary to meet the daily challenges and demands of police work”

50. Career progression study summary (1/2) Encounters in social cyberspace: e-mentoring for professional women: Headlam-Wells et al 2006 UK Training Look at barriers for women in career progression and evaluate the effects of an e-mentoring programme E-mentoring programme 122 (89) Non-random