Protocol development and searching for qualitative evidence

1. Protocol development and searching for qualitative evidence Systematic review workshop K U Leuven 4-6 June 2012 Janet Harris - Updated material from Angela Harden, Three-day systematic review workshop, K U Leuven, 6th to 8th May 2011

2. Why develop a protocol? • Helps to plan the review and anticipate problems • Encourages comment and review • Aids transparency and audit • Can build in flexibility • Required by review organisations like Cochrane and Campbell

3. Elements of a review protocol • Title • Background • Review question/objectives • Inclusion and exclusion criteria • Search strategy • Screening • Quality assessment • Coding and data extraction • Synthesis methods • Conflicts of interest

4. Example of a review protocol that includes qualitative research • http://onlinelibrary.wiley.com/o/cochrane/clsysrev/articles/CD009105/frame.html • Leiknes KA, Berg RC, Smedslund G, Jarosch-von Schweder L, Øverland S, Hammerstrøm KT, Høie B. Electroconvulsive therapy for depression (Protocol). Cochrane Database of Systematic Reviews 2011, Issue 5. Art. No.: CD009105. DOI: 10.1002/14651858.CD009105.

5. Review questions • SPICE • Setting (where? in what context? • Perspective (who?) • Intervention (or phenomenon of interest) • Comparison (what else?) • Evaluation (how well – what result? • PICO • Population • Intervention • Comparison • Outcome

6. Searching systematically • Common principles and challenges • Approaches to searching • Search sources • Bibliographic databases • Filters for electronic searching • Summary of key points • BUT first……..

7. Searching is a skilled and time consuming activity • Tap into the skills of information, subject and systematic review specialists, to: • Identify sources to search • Deal with technicalities (e.g. of databases) • Plan time for developing and testing searches

8. Common principles • Searching is the act of seeking studies that might be relevant for the review’s question • The search strategy is guided by the review question(s) • Practically constrained (by database limitations, restricted time and resources available to reviewers) • As in other stages, reviewers: • Have a rationale for their searching methods: • Draft, test and implement a structured search plan (‘search strategy’); • Report fully on their methods and findings.

9. Common challenges • Profusion of published and unpublished material • Much hidden: only 50% abstracts presented at conferences are later published in full • Different databases use different terms to classify studies • Different databases cover different, largely discrete areas of literature • As individuals we are constrained by our own disciplinary and policy area knowledge.

10. Approaches to searching • Comprehensive searching • Considers ideal as having access to all studies that answer the review question. An unbiased sample is next best. • Purposive searching • E.g. to identify main themes in the literature • Searching plans may develop as evidence comes to light.

11. Sources of qualitative research (1) • Qualitative research may be published as: • Journal articles • Books and book chapters • Conference papers • Project reports • Dissertations • Any more formats?

12. Sources of qualitative research (2) • Qualitative research may be found through: • Bibliographic databases • Specialist research registers • Library catalogues • Conference proceedings • Personal contacts • Handsearching journals • Websites • Search engines (Google scholar) • Reference lists of relevant studies

13. Bibliographic databases • Subject specific • E.g. Medline, Embase, CINAHL, ERIC • Multiple disciplines • E.g. Social Science Citation Index, ASSIA • Format specific • Dissertation abstracts, Index to theses

14. Searching bibliographic databases • Two main approaches: • Topic only search • Topic plus qualitative filter • Two types of search terms • Controlled vocabulary • Free text

15. A note on terminology • No agreed definition of qualitative research • Houses a broad range of methodologies and methods • Indexing on databases is inconsistent • Titles and abstracts may not employ the word ‘qualitative’ but may use more specific terms • E.g. focus groups, interviews, grounded theory, discourse analysis, phenomenology, Glaser and Strauss, Nvivo, Nudist, views, perspectives, experiences, stories, narrative, themes, findings

16. Definitions range from relatively simple… “Research that derives data from observation, interviews or verbal interactions and focuses on the meanings and interpretations of participants” Holloway and Wheeler (1995) Holloway W, Wheeler S (1995) Ethical issues in qualitative nursing research, Nursing Ethics, 2:223-232

17. ….to elaborate and complex… Qualitative research is an umbrella term for an array of attitudes toward and strategies for conducting inquiry that are aimed at discerning how human beings understand, experience, interpret and produce the social world…it encompasses richly detailed descriptions and in-depth, particularised interpretations of persons and the social, linguistic, material and other practices and events that shape their lives and are shaped by them. Qualitative research typically includes, but is not limited to, discerning the perspectives of….the actors…point of view. Although both philosophically and methodologically a highly diverse entity, qualitative research is marked by certain defining imperatives that include a case orientation…to analysis, sensitivity to cultural and historical context, and reflexive accounting practices to optimise validity….. Sandelowski M (2003) Qualitative research. In M Lewis-Beck, AE Bryman & TF Liao (Eds) The Sage Encyclopedia of Social Science Research Methods. Thousand Oaks, Ca: Sage.

18. Optimal search filters • Aim to balance sensitivity and specificity • Some databases may have built in filters • Information specialists have developed and tested filters for others • Lots of room for further development and evaluation - build this into your reviews

19. Optimal qualitative filter for CINAHL • Various combinations of the following tested: • interviews.sh; interview.tw; attitude.sh; qualitative studies.sh; qualitative stu$.mp; thematic analysis.sh; audiorecording.sh; grounded theory.sh; study design.sh • Optimal combination: • Interview.tw OR audiorecording.sh OR qualitative stu$.mp sh= subject heading tw= textword - word or phrase in title or abstract mp = multiple posting – word or phrase in title, abstract of subject heading Wilczynski et al. (2007) Search strategies for identifying qualitative studies in CINAHL, Qualitative Health Research 17:705-710.

20. Evaluation of three strategies on six* databases *MEDLINE, EMBASE, CINAHL, British Nursing Index, ASSIA and SSCI Shaw et al. (2004) Finding qualitative research: an evaluation of search strategies, BMC Medical Research Methodology 4:5

21. Key points for developing your search strategy • Seek the input of an information specialist asap. • Undertake some initial scoping and searching. • Identify relevant sources. • Use a methods filter if your review only focuses on qualitative research. • Document each part of your search and its results. • Invest in good information management systems and people.

22. Further reading • Sandelowski M, Barroso J (2007) Handbook for synthesising qualitative research. New York: Springer (see chp 3) • FlemmingK, Briggs M (2006) Electronic searching to locate qualitative research: evaluation of three strategies. Journal of Advanced Nursing, 57: 95-100. • StansfieldC et al. (2010) Search wide and dig deep: Identifying ‘views’ research for systematic reviews. Paper presented at the Joint Colloquium of the Cochrane and Campbell Collaboration, Keystone, Colorado, USA, 18th to 22nd October. • StansfieldC et al. (forthcoming, 2012) Finding relevant studies. In Gough D, Oliver S, Thomas J (Eds) Systematic Reviews in the Social Sciences. London: Sage.