Evidence from systematic reviews in public health and the work of the CRD

1. Evidence from systematic reviews in public health and the work of the CRD Kath Wright Amanda Sowden CRD November 2004

2. Background • 1998 - Our healthier nation: a contract for health • 1999 - Saving lives: our healthier nation • cancer, CHD and stroke, accidents, mental illness • 2000 - Evidence from systematic reviews of research relevant to implementing the “wider public health” agenda. Centre for Reviews and Dissemination http:www.york.ac.uk/inst/crd/wph.htm • cancer, CHD and stroke, accidents, mental health, education, social care and social welfare, crime drugs and alcohol • 2002 - Review of evidence from systematic reviews relevant to implementing the wider public health agenda. CRD. • cancer, CHD and stroke, accidents, mental health, sexual health, alcohol & drug misuse, smoking, immunisation , environment, crime & violence, elderly, children, miscellaneous

3. CRD’s 2002 update • A two part document • Part one is a qualitative study to elicit the views of policy makers and practitioners on the need for evidence and its presentation • Part two is a synthesis of existing systematic reviews relevant to the public health agenda • Draft report produced, peer reviewed, comments incorporated • Ongoing discussions with the DoH Policy Research Programme about dissemination of the findings • Health Development Agency now included in these discussions under their Wider Determinants of Health programme • Aim is to pilot a new series of publications based on the “Wider Determinants of Health” project

4. CRD’s 2002 update: views of policy makers and practitioners (1) • Envisaged that the results of this would help inform how the information collected would be presented and disseminated • Qualitative study using interviews (23), focus groups (35 individuals in 4 groups), responses (11) to e-mail discussion list • Directors of public health and public health observatories, PCT executives, senior health promotion/development managers, public health leads in PCTs • Data collected was analysed using N6 software

5. CRD’s 2002 update: views of policy makers and practitioners (2) • Findings: • Range of views on evidence based policy and practice • majority unsure why systematic reviews important • less than 50% had heard of systematic reviews • lack of critical appraisal skills • Amount of evidence • information overload a common experience • short “snappy” documents preferred • Access to evidence • not everyone had access to the Internet • Differences between NHS and non-NHS personnel

6. CRD’s 2002 update: synthesis of systematic reviews (1) • Identification, selection, appraisal and synthesis of existing systematic reviews on the effects of interventions and programmes relevant to the wider public health agenda • Inclusion criteria – • have systematic methods been used? • Is the topic of public health interest? • Originally planned to use data extraction, validity assessment, synthesis of study results as inclusion criteria • Later decided that these would be considered in the quality assessment of each included review

7. CRD’s 2002 update: synthesis of systematic reviews (2) • Arranged by topic area e.g. accidents • Each chapter gives details of the size of the problem and the current policy targets e.g. to reduce death rates by one fifth • Following this, the included SRs are briefly listed by: • interventions giving evidence of effect e.g. provision of free fire alarms • interventions giving no evidence of effect - none in this section • interventions giving insufficient evidence of effect e.g. parental education programmes aimed at increasing use of child safety seats in motor vehicles • interventions giving evidence of harm or potential harm e.g school based driver education programmes • Implications for policy, practice and research outlined e.g. • despite poisoning being the second most frequent cause of accident and injury in children, no identified reviews addressed this issue

8. CRD’s 2002 update: synthesis of systematic reviews (3) • Attewell RG, Glase K, McFadden M. Bicycle helmet efficacy: a meta-analysis. Accident Analysis and Prevention 2001:33:345-352. • Objective - to assess the efficacy of helmet use in preventing serious injury to cyclist • Inclusion/exclusion criteria - cyclists who crash and suffer head, brain or facial injury and/or death • Participant description - 4 studies children, 2 studies adults, 10 studies mixed • Intervention description - wearing of bicycle helmets in crashes • Results - gives ORs for efficacy, injury reduction figures as % e.g. head injury 45%, brain injury 33%, facial injury 27%, death 29% • Author Conclusion - helmets prevent serious injury and death • CRD comments - research question, inclusion criteria, synthesis - good; search adequacy, study details - fair ; validity assessment - N/A

9. Literature searching - problems • Range of primary sources • Volume of literature • Large number of potentially relevant databases • Variability in databases: • abstracts, • interface, • indexing • imprecise terminology • Internet searching

10. Literature searching - sources • Sources used • Cochrane Database of Systematic Reviews • Database of Abstracts of Reviews of Effects • additional databases e.g. EMBASE, HMIC, Sociological Abstracts, ASSIA, Health Technology Assessment database, NRR, guidelines databases (e.g. HSTAT, NGC, USPSTF) • websites • contact with subject experts

11. Cochrane Database of Systematic Reviews • Didn’t use a search strategy • Instead identified all new or substantially updated reviews on the database for issue numbers 2000 no 1 through to 2002 number 1 • Update searching done later to cover issue numbers 2002 numbers 2 and 3 • Reviewers scanned lists for potentially relevant reviews and protocols • Identified 80 potential reviews

12. Database of Abstracts of Reviews of Effects • Identifies reviews by: • handsearching • contact with HTA organisations • regular website searching • database searching • Current Contents Clinical Medicine • MEDLINE • CINAHL • ERIC • Biosis • AMED • PsycINFO

13. Benefits of searching DARE • A number of different databases and other sources have already been searched • No need to use a “systematic review” search filter • The included reviews have already met agreed quality criteria. See http://www.york.ac.uk/inst/crd/faq4.htm • So, saves time and effort and avoids many of the recognised problems of searching in this area • Nevertheless, still undertook 18 separate searches to cover all the topic areas • Identified 3955 potential reviews initially • 273 on public DARE • 3682 not on public DARE

14. EMBASE • When this project began EMBASE wasn’t searched for DARE • Wanted to include it to ensure relevant SRs were not missed • The searches undertaken demonstrate some of the problems referred to earlier about searching in this field: • volume of literature – especially as we were restricting our search to 2000, 2001 and 2002. • lack of tested systematic reviews search filter for EMBASE

15. EMBASE • Approach 1. Tried to identify records unique to EMBASE by developing comparable EMBASE and MEDLINE strategies that could be run across both databases, retrieving common records and removing them. This approach failed, demonstrating the difficulties of simply “translating” search strategies from one database to another. • Approach 2. Relied upon using a SR strategy with no subject terms specified. This approach failed as it retrieved too many records (17,633) for the project (or Endnote) to cope with • Approach 3. Amended the SR search strategy to make it less sensitive and combined it with subject terms. The 18 strategies identified 4137 unique records

16. Other database searching • Had to adopt a pragmatic approach re searching for systematic reviews e.g. • HMIC review* or guideline* • Sociological Abstracts review* or overview* or metaanal* or meta-anal* or metanal* or guideline* • Assia – review$ or overview$ or metaanal$ or meta-anal$ or metanal$ or guideline$

17. Guidelines databases and listings • HSTAT, National Guidelines Clearinghouse, Canadian Task Force on Preventive Care, US Preventive Services Task Force • All web-based • Interfaces don’t offer very sophisticated search options e.g. searching by date was often the most helpful approach (rather than searching by topic or study design) • Had to rely on downloading title listings from the websites and sifting

18. Literature searching issues….again • Range of primary sources • mainly focused on journal literature, aware that non-journal literature important in this area • Volume of literature • in the time available had to undertake quite a narrow search (in terms of databases used) and modify sensitivity of search strategies • Wide number of potentially relevant databases • didn’t search Caredata, Social Science Citation Index, AgeLine or other possibly relevant databases • didn’t have access to some possibly relevant databases e.g.Criminal Justice Abstracts • had to adapt each search strategy according to availability of indexing etc • Variable quality of search interface • usual access route to Assia is a web interface so used Datastar option. • Variable indexing options • limited number of study types available as subject headings • Internet searching • heavily reliant on scanning, sifting and downloading

19. Research in literature searching in public health and the social sciences databases • CRD’s experience is similar to that of others, see: • Grayson, L, Gomersall, A. (2003). A difficult business: finding the evidence for social science reviews. London, ESRC UK Centre for Evidence Based Policy and Practice. • http://www.evidencenetwork.org/home.asp • Popay, J and Katrina Roen. (2003). Using evidence from diverse research designs. London, SCIE. • http://www.scie.org.uk/publications/reports/report03.pdf • HDA/CRD/Cardiff University (HEBW) Collaborative Group. (2004). Indexing in databases of relevance to public health • http://www.hda.nhs.uk/evidence/indexing_home.html • Bradshaw, J. R., S. Baldwin, et al. (2000). Methodologies for socially useful systematic reviews in social policy. York, University of York • http://www.regard.ac.uk/research_findings/L141250001/report.pdf • Jackson, N, Waters E and the Guidelines for Systematic Reviews of HealthPromotion and Public Health Interventions Taskforce. (2004). The challenges of systematically reviewing public health interventions. Journal of Public Health 26 (3); 303-307.