Searching for studies in complex areas

1. Searching for studies in complex areas Mary Dixon-Woods Department of Health Sciences University of Leicester Andrew Booth, University of Sheffield

2. A “typical” definition of an SR • "A systematic review is a review in which there is a comprehensive search for relevant studies on a specific topic, and those identified are then appraised and synthesised according to a pre-determined explicit method“ Klassen, Jadad, & Moher, 1998

3. Technical problems of searching • Effort invested in indexing systems and filters on electronic databases for quantitative research • Qualitative research still very elusive; abstracting is a problem (Shaw et al, 2004) • But searching isn’t just a technical difficulty

4. Greenhalgh, 2005 Protocol driven: 150 (30)Electronic database search* 126 (25) Hand search (32 journals) 24 (5) "Snowballing": 252 (51)Reference tracking 218 (44) Citation tracking 34 (7) Personal knowledge: 119 (24) Sources known to research team 85 (17) Social networks of research team 29 (6) Serendipitous 5 (1) Total in final report 495

5. CIS approach • Fixed, protocol-driven approach to searching has advantages and limitations • Emphasises more creative, organic, intuitive forms of searching • Accepts that exhaustiveness might not be possible or appropriate • Recognises that it may not always be possible to be “transparent”

6. Treat the question as “a compass rather than an anchor” –Eakin & Mykhalovskiy (2003) Not “Big Bang” searching Iterative “Spiral” Searching Subjecting a question to continual review and refinement

7. This requires: • Purposively searching for new insights (new databases, new disciplines, new search methods) – active sampling of the literature • “Lateral leaps” rather than “more of the same” • Actively seeking the “disconfirming case” rather than the “averaging effect” • Citation searching within divergent disciplines/schools of thought • Challenge: • “Sampling research is fundamentally not like sampling people. Unlike people, research papers have a vested interest in being different from one another, and are (in theory at least) only published if they are saying something new.”. Dixon-Woods, Bonas, Booth et al, 2006

8. Searching not reproducible, not necessarily even transparent • More we learn about SR methods the more we recognise that reproducible searching is a myth • Different questions, search strategies, selection of abstracts, inclusion criteria, selection of articles, methods of synthesis, level of analysis, implications…..and on, and on, and on! • Replace with criterion of “reproducible” with “rigorous” • Reporting of methods is not for reproducibility but to increase our “trust” of the research

9. Assessing comprehensiveness • The main purpose of a CIS is “interpretative” not “aggregative” • Sensitivity to evolving question/problem definition • Therefore the “point of termination” is not when you have identified ALL studies • It is when you no longer identify NEW insights (analogous to theoretical saturation) • Requires iterative, sampling-based approach

10. A different view of systematic • “‘Systematic’ is, by definition, ‘a system, plan, or organized method’ (Oxford English Dictionary [OED]), ‘methodical in procedure or plan’ (Merriam-Webster). If a systematic review is then….about method itself, then qualitative methods include valid principles and procedures with which to progress in reviewing literature ‘systematically’”. [Jones, 2004] • “A wide range of qualitative ‘methodical’ approaches are, therefore, available to perform or carry out a review in a particular field of interest—‘systematically and meticulously’ (OED)”. [Jones, 2004]

11. Systematic and Explicit • Simply means that we follow an (appropriate, but not specific) system and that we communicate what we have done • These are not markers of a systematic review – they are characteristics of ALL well-conducted, well-reported research

12. References - 1 Eakin, J & Mykhalovskiy, E (2003). Reframing the evaluation of qualitative research: Reflections on a review of assessment guidelines in the health sciences. Journal of Evaluation in Clinical Practice, 9(2), 187-194. Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005 Nov 5;331(7524):1064-5. Epub 2005 Oct 17. Klassen TP, Jadad AR, Moher D. Guides for reading and interpreting systematic reviews. I. Getting started. Arch Pediatr Adolesc Med 1998;152:700-704.

13. References - 2 Booth A (In press) “Brimful of STARLITE”: Towards standards for literature searches. JMLA (October) Dixon-Woods M, Bonas S, Booth A et al. How can systematic reviews incorporate qualitative research? A critical perspective. Qualitative Research, 6 (1), 27-44 (2006) Dixon-Woods M, Booth A, Jones D, et al. How can systematic reviews incorporate qualitative research? ESRC Methods Briefing No. 6 http://www.ccsr.ac.uk/methods/publications/documents/dixonwoods.pdf Jones K. Mission Drift In Qualitative Research, Or Moving Toward A Systematic Review Of Qualitative Studies, Moving Back To A More Systematic Narrative Review. The Qualitative report, 9(1), 95 - 112.http://www.nova.edu/ssss/QR/QR9-1/jones.pdf Shaw RL, Booth A, Sutton AJ, Miller T, Smith JA, Young B, Jones DR, Dixon-Woods M (2004). Finding qualitative research: an evaluation of search strategies. BMC Research Methodology 4:5 (16 Mar).