Searching for Relevant Studies

1. Searching for Relevant Studies Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide www.ahrq.gov

2. Systematic Review Process Overview

3. Learning Objectives • To substantiate why prudent literature searching is important • To describe important tenets of literature searching • To describe why transparency in the search process is important

4. Balancing Precision and Recall • Systematic reviews require comprehensive searches. • Searching many databases by using a generic term gives high recall but is very imprecise. For example: • A July 23, 2009, Ovid search for “cancer.mp or neoplasms” yielded 786,978 citations. • That is too many citations to review to yield the 323 citations on the role of tamoxifen and raloxifene in cancer, the reason for the search. • The goal is to carefully construct a search to keep precision high while maximizing recall. Precision (specificity): the proportion of retrieved articles that are relevant Recall (sensitivity): the proportion of potentially relevant articles retrieved by search Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

5. Presenting Popular Studies Is Not Enough • One evaluation compared the results of popular studies against others with populations and methodologies that were similar or superior. • Evaluation of 45 studies published from 1993 to 2000 in journals with high impact factors that reported positive findings and were cited more than 1,000 times. • Results were compared against other studies with the same comparators by employing populations and methodologies that were similar or superior. • Opposite/null findings or much more dramatic effects were found 31 percent of the time. • Eighty-three percent of nonrandomized studies and 23 percent of randomized controlled trials were subsequently contradicted. Ioannidis JP. JAMA 2005;294:218-28.

6. Where To Begin • When a general topic is proposed, it is tempting to begin by extensively searching for primary literature. • Before performing the extensive literature search, it is crucial to understand the topic, devise an analytic framework, ask clearly defined key questions, and understand the scope of the review to be conducted.

7. Specialized Electronic Databases • Although the topic area dictates the databases to be included, the following are common ones: • MEDLINE (General) • Cochrane Central (General) • EMBASE (General, more international in scope) • PsychLit/PsychINFO (Psychology) • AIDSLine (HIV/AIDS) • CINAHL (Nursing) • TOXNET (Adverse Events/Toxicology) Generally a minimum requirement for a thorough search of literature Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

8. MEDLINE Alone Is Not Enough • Study 1 (Sassi et al., 2002): sensitivity and specificity of using MEDLINE versus nine databases • This study focused on ability to retrieve economic analyses from January to March 1997. • MEDLINE-only searches had a sensitivity of 72 percent and a specificity of 75 percent versus nine databases. • Study 2 (Betrán et al., 2005): systematic review of the prevalence of maternal mortality and morbidity from 1997 to 2002 • Multiple databases were searched (MEDLINE, EMBASE, BIOSIS, LILACS). • 60 percent of citations were found in more than one database. • The MEDLINE search had 20 percent of nonreplicated citations, EMBASE had 7.4 percent, and LILACS had 5.6 percent. Sassi, F, et al. Med Care 2002;40:387-94; Betrán AP, et al. BMC Med Res Methodol 2005;5:6.

9. Identifying Key Articles To Refine a Pilot Search • Perform a pilot search and compare to see if output matches articles already identified by the research group, key informants, and manual searches of references of these articles. • Pilot searching is usually limited to MEDLINE and then adapted to other databases. • After pilot searching, refine the search. • Searching the literature is an iterative process. Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

10. Multiple Searches Help With anExtensive Literature Base (I) • Multiple searches within the same database may be necessary. • Efficacy and harms searches may be separate. • If the available literature is small, a single broad search is appropriate. • If not so, separate searches are prudent. Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

11. Multiple Searches Help With anExtensive Literature Base (II) • An efficacy search may include the disease, intervention, and terms for randomized controlled trials (RCTs). • Efficacy evaluations frequently are limited to RCTs. • Fewer inherent biases and higher quality evidence • A harms search may be broader and include just the intervention and harms and/or disease. • The paucity of harms data in published RCTs frequently requires that non-RCTs be included. • Authors would likely be interested in harms regardless of the reason for the intervention.

12. Boolean Operators Help ToBalance Precision and Recall • “OR” — makes the search broader • “AND” — makes the search more selective • “/” — means all subheadings are searched • When a search term is entered, you can selectively choose some subheadings (such as pharmacology for pharmacology studies). • Subheadings are generally imprecise; accepting all subheadings increases the yield. • “adj” — cuts down on miscellaneous citations • “Heart adj failure” would find only instances in which “heart” is next to “failure.” • “$” — truncates a word with different endings • “Analy$” would find words like analysis, analyses, analyze, and analyse • “.ti” or “.ab” — searches in the title or abstract for the word of interest • Can find citations where the word you search for is not a keyword • “.mp” or multiple posting — a text word search; does not need to be a key word

13. Harms Searching • If the specific adverse effects of interest are specified in the key questions, they can be searched for directly (e.g., rhabdomyolysis, cancer). • General harms subheadings for MEDLINE would include: • /adverse effects • /poisoning • /toxicity • /chemically induced • /contraindications • /complications • General harms subheadings for EMBASE would include: • /side effect • /adverse drug reaction • /drug toxicity • /complication • Consider using a database dedicated to harms such as TOXNET (http://toxnet.nlm.nih.gov/). Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

14. Filters or Hedges Are Validated Ways To Improve Precision and Recall • Filters have been developed and validated that can help balance precision and recall. • Cochrane Collaboration http://www.cochrane-handbook.org section 6.4.11 Search Filters, National Health Service • Centre for Reviews and Dissemination http://www.york.ac.uk/inst/crd/SysRev/!SSL!/WebHelp/SysRev3.htm • Scottish Intercollegiate Guidelines Network http://www.sign.ac.uk/methodology/filters.html • InterTASC Information Specialists’ Sub-Group http://www.york.ac.uk/inst/crd/intertasc/about.htm • Filters may not work in all circumstances Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

15. Example Filter (Hedge) for Finding Randomized Controlled Trials MEDLINE (Ovid) Search for Randomized Controlled Trials Using the Cochrane Highly Sensitive and Specific Search Strategy (Sensitivity and Precision Maximizing Version 2008) 1. randomized controlled trial.pt. 2. controlled clinical trial.pt. 3. randomized.ab. 4. placebo.ab. 5. clinical trials as topic.sh. 6. randomly.ab. 7. trial.ti. 8. 1 or 2 or 3 or 4 or 5 or 6 or 7 Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.

16. Example Filter for Finding Nonrandomized Studies MEDLINE (Ovid) Search for Observational Studies Using the Scottish Intercollegiate Guidelines Network Observational Study Medline Search Filter 1. epidemiologic studies/ 2. exp case control studies/ 3. exp Cohort Studies/ 4. case control.tw. 5. (cohort adj (study or studies)).tw. 6. cohort analy$.tw. 7. (follow up adj (study or studies)).tw. 8. (observational adj (study or studies)).tw. 9. longitudinal.tw. 10. retrospective.tw. 11. cross sectional.tw. 12. Cross-Sectional Studies/ 13. 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10 or 11 or 12 Coleman CI, et al. Comparative Effectiveness Review No. 18.Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.

17. Peer Review of Electronic SearchStrategies (PRESS) Checklist (I) • Items to consider before hitting enter: • Use of natural language terms (search as key word) • Appropriate word truncation and spelling variation • Appropriate use of limits such as language and years • Appropriate use of searching for field, publication type, and author Sampson M, et al. PRESS: peer review of electronic search strategies. 2008.

18. Peer Review of Electronic Search Strategies (PRESS) Checklist (II) • Items to consider before hitting enter: • Spelling errors/spelling in American and British English (e.g., anaesthesia and anesthesia, analyse and analyze, oestrogen and estrogen) • Line errors that occur when searches are combined by using line numbers (e.g., “1 AND 3” instead of “1 AND 4”) • Appropriate use of Boolean operators • Adaptation of search strategies for specific databases • Are all appropriate headings used (refer to the thesaurus for a specific database) • Appropriate use of subheadings and floating subheadings Sampson M, et al. PRESS: peer review of electronic search strategies. 2008.

19. Considering Publication Bias WhenDeveloping a Search Strategy • Publication bias is the tendency of certain types of trials (such as those with the largest effects) to be published. • Publication bias increases the risk that the observed effect might not reflect the true effect. • It may negatively impact consistency, precision, and magnitude of effect. • Expanding searches to include additional languages, citation tracking, hand searching, and grey literature can help identify and possibly minimize publication bias. Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

20. Example of Publication Bias • Study (Turner et al., 2008): 74 antidepressant studies registered with the U.S. Food and Drug Administration • 97 percent of positive studies published • 39 percent of neutral or negative studies published • 11 of 14 studies published in a way that conveyed the positive results but deemphasized the negative • When only published literature was meta-analyzed, a 32 percent increase in relative effect size occurred versus the more complete dataset of conducted trials. Turner EH, et al. N Engl J Med 2008;358:252-60.

21. Restricting Search by Languageand Impact on Publication Bias • It is not a requirement to search non-English–language publications. • Evidence-based Practice Centers should consider how inclusion or exclusion of non-English–language studies might or might not bias the search. • It is usually advisable to include non-English– language studies when the evidence base may be highly restricted in English-language but broader in non-English–language studies. • Such as, complementary and alternative medicine topics Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

22. English Only or All Languages (I) • Study 1 (Pham et al., 2008): evaluation of 42 systematic reviews, including 662 randomized controlled trials • For Western medicine, language-restricted systematic reviews did not introduce biased results when compared with language-inclusive ones, in terms of estimates of intervention effectiveness (risk odds ratio [ROR] = 1.02; 95% confidence interval [CI] = 0.83–1.26). • For complementary and alternative medicine, however, language-restricted systematic reviews resulted in a 63 percent smaller protective effect estimate when compared with language-inclusive reviews (ROR = 1.63; 95% CI = 1.03–2.60). Pham B, et al. J Clin Epidemiol 2005;58:769-76.

23. English Only or All Languages (II) • Study 2 (Vickers et al., 1998): evaluation of acupuncture trials conducted in England versus China, Japan, Russia, and Taiwan • Results of randomized controlled trials or controlled clinical trials in England were favorable 75 percent of the time. • Results in China, Japan, Russia, and Taiwan were favorable 99, 89, 97, and 95 percent of the time, respectively. • Researchers should carefully consider how to manage data from these countries. Vickers A, et al. Control Clin Trials 1998;19:159-66.

24. Citation Tracking (I) • Forward citation tracking: databases that provide information on articles cited by other authors • This method relies on the author’s choice to cite an article, rather than key words or indexing. • Can turn up unique items and can complement a traditional database search. • Databases currently in use are Web of Science, Scopus, Google Scholar, PubReMiner, and PubFocus. • Web of Science and Scopus have access fees but have the most developed search and export interfaces. • Google Scholar, PuReMiner, and PubFocus are free access but are not as developed. Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

25. Citation Tracking (II) • Backward citation tracing: reading references of key articles • New search terms, poorly indexed journals, or inappropriate limiters in the original search can be identified • “Related article” links: useful when designing and refining a search but not for a formal search • Difficult to perform systematically, report strategy with transparency, and reproduce the search Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

26. Hand Searching To Reduce Publication Bias • Not all journals or sources can be indexed in databases. • Some journals index only certain article types. • Abstracts, conference proceedings, and supplements may not be indexed. • Backward citation tracking and key informant discussion can identify these journals. • Ulrich has a useful Web site to identify journals in a specific topic area (available at: http://www.ulrichsweb.com/ulrichsweb/). • If Ulrich indicates an important journal is not indexed fully in databases used, search the journal by hand. Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

27. Benefits of Hand Searching • Study 1 (Hopewell et al., 2007): using MEDLINE to search 22 specialized journals to identify randomized controlled trials (RCTs) was compared with hand searching of the same journals • 117 of the 462 RCTs (25%) were not retrieved by MEDLINE because of a lack of the publication-type terms “randomized controlled trial” or “controlled clinical trial.” • Additional articles were missed in journals without full journal indexing. • Study 2 (Jadad et al., 1993): using MEDLINE to search nine pain and anesthesia journals was compared with hand searching • 13 percent of eligible articles were not retrieved with MEDLINE alone. Hopewell S, et al. Cochrane Database Syst Rev 2007;(2):MR000010; Jadad AR, McQuay HJ. Online J Curr Clin Trials 1993;(33).

28. Using Grey LiteratureTo Reduce Publication Bias • Grey literature: literature produced at all levels of government, academics, business, and industry that is not controlled by commercial publishers • Usually not systematically identified, stored, or indexed • Can help overcome publication bias • Least efficient type of literature to search and may be (or be perceived to be) of lower quality • Useful in areas with little published evidence, in fields that are rapidly changing (e.g., devices, surgery), or in fields that are highly interdisciplinary (e.g., mental health, alternative medicine) Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

29. Grey Literature • Study (Hopewell et al., 2008): reviewed five systematic reviews conducted to evaluate the impact of grey literature in meta-analyses of randomized controlled trials • All systematic reviews found more positive results with published literature than with grey literature (risk odds ratio = 1.09; 95% confidence interval = 1.03–1.16) Hopewell S, et al. Cochrane Database Syst Rev 2007;(2):MR000010.

30. Grey Literature: Regulatory Search • The approval process for new drugs and devices involves submission of data that may not be published or may be incompletely published elsewhere • Good check for publication bias • Regulatory Web sites: • Drugs@FDA; http://www.accessdata.fda.gov/Scripts/cder/DrugsatFDA/ index.cfm?fuseaction=Search.search_Drug_Name • FDA Center for Drug Evaluation and Research (CDER) Archives: http://www.fda.gov/cder/archives/default.htm • Dockets for FDA: http://www.fda.gov/ohrms/dockets/default.htm • Devices@FDA: http://www.accessdata.fda.gov/Scripts/cdrh/devicesatFDA/ • Health Canada Drug Product Search: http://www.hc-sc.gc.ca/dhp-mps/prodpharma/databasdon/index-eng.php • European Medicines Agencies European Public Assessment Reports for Authorized Medicinal Products for Human Use: http://www.emea.europa.eu/htms/human/epar/a.htm Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

31. Grey Literature: Online Registries • Online registries include trial name, type, and brief methodology • May contain results of completed but unpublished trials • Registry Web sites: • ClinicalTrials.gov: http://clinicaltrials.gov/ct2/search • ClinicalStudyResults.org: http://www.clinicalstudyresults.org/home/ • Current Controlled Trials: http://www.controlled-trials.com/mrct/ • Australian New Zealand Clinical Trials Registry: http://www.anzctr.org.au/trialSearch.aspx • Netherlands Trial Registry: http://www.trialregister.nl Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

32. Grey Literature: Abstractsand Conference Proceedings • Much grey literature is unlikely to be published. • When published, final data often differ from those in the conference abstract. • Use key informants to identify the most important meetings in the field of interest and hand search the abstracts and conference proceedings in journal supplements. • Some databases are also available: ProceedingsFirst; Biological Abstracts; Conference Papers Index; Conference Proceedings Citation Index (part of the Web of Science); and BioWizard Scholar Conference Abstract Search. Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

33. Grey Literature: Theses and Dissertations • Theses and dissertations are becoming increasingly available as institutions post them online. • Databases: • ProQuest Dissertation and Theses • The most comprehensive collection in the United States, with full text of dissertations through their Digital Archiving and Access program • Index to Theses in Great Britain and Ireland • Networked Digital Library of Theses and Dissertations (NDLTD) • Dissertation Abstracts Online Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

34. Grey Literature: Government Documents • Aside from governmental regulatory sites, other governmental agencies support or collect grey literature. • Computer Retrieval of Information on Scientific Projects (CRISP): http://crisp.cit.nih.gov • Searchable database of federally funded biomedical research • Health Services Research Projects in Progress (HSRPROJ): http://wwwcf.nlm.nih.gov/hsr_project/home_proj.cfm • Database of ongoing grants and projects in health services research • Science Accelerator: http://www.scienceaccelerator.gov/ • From the Office of Scientific and Technical Information and the U.S. Department of Energy • Defense Technical Information Center (DTIC): http://www.dtic.mil/dtic/ • From the U.S. Department of Defense, including the military health system Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

35. Grey Literature: General Sources • Grey Matters: http://www.cadth.ca/index.php/en/ cadth/products • A practical tool for evidence-based searching • Collated list of sources for grey literature appropriate for health sciences by the Canadian Agency for Drugs and Technology in Health (CADTH) • System for Information on Grey Literature in Europe (SIGLE): http://opensigle.inist.fr/ • Provides bibliographic references to grey literature in Europe • New York Academy of Medicine (NYAM): http://www.nyam.org/library.grey.shtml • Provides NYAM grey literature with a list of other organizations that do the same Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

36. Directly Contacting Researchers • Researchers can be contacted to determine whether or not projects in the grey literature are published in a peer-reviewed form. • Prolific researchers in the field can be contacted to see if they know of any citations that were missed by the literature search. • Search engine (e.g., Yahoo, Google) queries with the individual’s name or institution can be a good place to start. • Contact information in manuscripts is also a reasonable place to start. Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

37. Scientific Information Packets (SIPs) • SIPs are provided by the company manufacturing the drug or device. • SIPs contain information about products, including the product label and published and unpublished trials or studies. They: • can help to overcome publication bias by identifying trials that remain unpublished; • can identify soon-to-be-released trials that can ultimately be captured in comparative effectiveness reviews (CERs); and • allow drug and device manufacturers to be explicitly involved in helping provide a literature base to a CER. Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

38. Scientific Information Packets Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

39. Reporting Search Strategies: Transparency • Accurate and transparent reporting of search strategies is important. • Searches need to be updated toward the conclusion of a systematic review to ensure that no new information has come out in the intervening period. • Stakeholders and end-users want assurances that the systematic review was free of bias and that adequate search principles were followed. • Searches will need to be rerun if the systematic review is updated several years after it is completed. • Documenting the search strategy so that a skilled literature searcher can reproduce it is very important. Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

40. Transparent Reporting • Databases used • Dates covered • Search terms • Language restrictions • Nondatabase methods used • Inclusion/exclusion criteria • Full electronic search strategy • Publication-related restrictions • End date of search • List of excluded references • Qualifications of searcher • Number of references identified • QUOROM - or PRISMA-style flow diagram accounting for all references • Evidence of search effectiveness • Statement of filters employed • Description of sampling strategy Campbell R, Balshem H. J Clin Epidemiol 2010;63: in press.

41. Transparent Reporting: QUOROM Figure Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.

42. Transparent Reporting: PRISMA Figure Records identified through database searching (n = 56) MEDLINE (n = 42) Cochrane Central (n = 14) Additional records identified through other resources (n = 1) Identification Records after duplicates removed (n = 45) • Records excluded (n = 18) • Not a report in humans (n = 2) • Not in CF population (n = 1) • Not evaluating rhGH (n = 4) • Not a report of a new discovery (n = 11) Screening Records screened (n = 45) Full-text articles assessed for eligibility (n = 27) Eligibility • Full-text articles excluded, with reasons (n = 2) • Not a report in humans (n = 0) • Not in CF population (n = 0) • Not evaluating rhGH (n = 0) • Not a report of a new discovery (n = 2) Studies included in qualitative analysis (n = 25) Included Observational studies (n = 8) Repeat info of RCT (n = 6) Unique RTCs (n = 11) Studies included in quantitative analysis (n = 11)

43. Example of Two Searches • The next few slides present a comparison of a controlled trial search and an observational study search. • See how the search pieces are created and then come together. • Note the use of Boolean operators.

44. Example of Controlled Trial Search (I) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.

45. Example of Controlled Trial Search (II) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.

46. Example of Controlled Trial Search (III) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.

47. Example of Observational Study Search (I) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.

48. Example of Observational Study Search (II) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.

49. Example of Observational Study Search (III) Coleman CI, et al. Comparative Effectiveness Review No. 18. Available at: http://www.effectivehealthcare. ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productID=335.

50. Key Messages • Searching is important: balance precision and recall. • MEDLINE is not enough. • Multiple strategies improve completeness of search. • Filters are a good place to start. • More than one search may be needed (e.g., separately for “efficacy” and “harms”). • Report strategy with transparency.