Systematic Reviews Practicalities and Realities

1. Systematic Reviews Practicalities and Realities Alison Brettle, Research Fellow (Information) Salford Centre for Nursing, Midwifery and Collaborative Research University of Salford

2. Session Overview • Practical overview of systematic reviews and what you need to conduct one • Discussion

3. What is a systematic review? • An overview of primary research studies conducted according to explicit and reproducible methodology • A rigorous method of summarising research evidence • Shows what we know and don’t know about a topic area • Provides evidence of effectiveness (or not) by summarising and appraising relevant evidence

4. Systematic reviews aim • To find all relevant research studies (published and unpublished) • To assess each study on basis of defined criteria • Synthesise the findings in an unbiased way • Present a balanced and impartial summary of the findings taking any flaws into consideration

5. Systematic review models • Medical/Health care • Cochrane Collaboration, NHS Centre for Reviews and Dissemination • Usually includes “high quality” research evidence – RCTs • Often includes meta-analysis (mathematical synthesis of results of 2+ studies that addressed same hypothesis in same way) • Social care/Social Sciences • SCIE, EPPI Centre, Campbell Collaboration • Often include wider range of studies including qualitative • Often narrative synthesis of evidence

6. Systematic review process • Define/focus the question • Develop a protocol • Search the literature (possibly 2 stages scoping and actual searches) • Refine the inclusion/exclusion criteria • Assess the studies (data extraction tools, 2 independent reviewers) • Combine the results of the studies to produce conclusion– can be a qualitative or quantitative (meta-analysis) • Place findings in context – quality and heterogeniety of studies, applicability of findings

7. Task • You are part of a systematic review team and this is your first meeting. Your task is to draft a protocol by the end of the meeting then present to the other groups

8. Components of a protocol • Background/context leading to question • Review question • Inclusion/Exclusion criteria • Methods for • Identifying evidence • Selecting studies • Extracting studies • Assessing quality of studies • Synthesising the results • Disseminating the findings

9. Lessons learned in systematic reviews

10. Workplace physical activity interventions: a systematic review of the research evidence Lindsey Dugdill, Alison Brettle, Claire Hulme, Serena McCluskey, Andrew Long

11. Background • Policy • Increased PA important for prevention of chronic illness • Trend on a worldwide and UK scale that PA levels are declining

12. Objectives • To identify which types of workplace PA intervention were effective in changing PA behaviour for different sectors of the workforce and different types of workplace • To ascertain which aspects of intervention design and delivery contributed to effectiveness and contributory motivators, barriers and facilitators

13. Lesson 1. Focus the question • More focussed the question, easier to undertake literature search and decide on relevant studies • Need to maintain a balance between a clearly defined topic and a meaningful and relevant question • Can help scope the literature to help develop the question • Refine question as part of review process

14. Focussing the question • Options: • empirically led -> literature scoping -> narrowing down topic on the basis of ‘what is there’ in the evidence base • theoretical and conceptual -> concept mapping, explore the meaning given to the concepts, embracing user and carer notions of desired outcomes (based on previous research) themselves central to a ‘social care’ perspective • Funder/policy driven

15. Lesson 2. Scope the literature • Provides an indication of literature available and what needs to be examined • Helps focus the question • Helps clarify initial inclusion and exclusion criteria

16. Lesson 3. Write a protocol • Plan what you are going to do in the review • Set out the background and objectives • Outline the resources you will use • Establish inclusion/exclusion criteria • How will data be extracted (what will be extracted) • How will you synthesise literature

17. Literature search 12 electronic databases 1996-2007 Selected websites of known organisations Citation tracking Could also use hand-searching of journals “grey” literature

18. Lesson 4. Use multiple sources of information • Databases only part of the picture • Handsearching identifies further references • Citation tracking adds more references • SMI review – 96 studies (73 databases, 23 handsearching/citation tracking) • Counselling – 47 studies (aprox 10 handsearching, 4 citation tracking)

19. Lesson 5. Refine your search plan as you carry out the literature searches Theory • Develop your search plan • Everything that follows flows smoothly and easily Practice • Modify your approach as the search progresses • Use spiral approach to refine • Ongoing discussions within team

20. Search spiral • Scoping (initial database searches) • Refining (indepth searches of range of sources) • Confirming (hand searches, citation tracking)

21. Inclusion • Intervention aimed to increase PA • Aimed at employed adults • Initiated or endorsed by employer • Outcome measures included a measure of PA • Evidence of behaviour change • English • Published post 1996 • Took place in UK, Australia, NZ, Europe

22. Exclusion • Self employed or unemployed adults • Adults needing specialist advice regarding PA • No evidence of behaviour change or PA measure • Insufficient detail reported in study • Cost effectiveness • Located in US or Asia

23. Lesson 6. How will you refine the inclusion/exclusion criteria? • Tighter the criteria = • less papers to review • BUT • will your review draw any meaningful conclusions? • Focussing the question v quality of studies

24. Overview of search and appraisal process

25. Appraisal • 38 papers representing 33 studies • Each paper appraised by 1 reviewer, using a standardised protocol, 20% sample reviewed by 2 reviewers and each checked as put on database • Quality of papers ++ high, + good, - poor evidence (using protocols/guides provided by NICE) • Categorised by intervention and quality rating • Summary table • Conclusions on the whole derived from those classified as high or good

26. Lesson 7: Know what you want to find out • Select your critical appraisal/data extraction tool carefully • Make sure it captures the elements that you wish to write about in your final report • Think what data need to extract • Narrative review v meta analysis • Quality of studies – what evidence are you going to include? How are you going to assess the quality • Do you need to refine the inclusion/exclusion criteria as you go along?

27. Findings: published systematic reviews • Inconclusive review level evidence that workplace PA interventions were effective in increasing PA

28. Findings: Stairwalking • 7 studies (1 high, 2 good, 4 poor) • 4 demonstrated that posters and signs can increase stair use in the short term • Little evidence of effectiveness

29. Findings: Walking interventions • 4 studies (2 good, 2 poor) • 3 used pedometers (objective measure) • All 4 also used self reported counts • Evidence that workplace walking interventions can increase daily step counts

30. Findings: Active travel • 3 studies (1 good, 2 poor) • Evidence from 1 study that a walking and cycling to work campaign using written materials can increase walking to work in economically advantaged women

31. Findings: multi component interventions • 16 studies (2 high, 1 good) • Wide range of interventions • Difficult to attribute which bit of the intervention was causing the effect • Limited evidence that counselling has effect on workplace PA

32. Limitations • Quality of evidence • Exclusion of studies from US and Asia • Most of studies took place in large public sector workplaces • Reliance on self report measures

33. Lesson 9. Set up a system to keep track of the review process • Use a package such as reference manager or endnote to keep track of references • Large reviews need method of keeping track of data extraction, exel, access or specialised software • Update at every stage of the process • Compiles references at the end • Need a hard copy system too for the actual papers and a system for keeping track of results • If you do not know where you are up to – it is unlikely that your review will be systematic

34. System to manage process – the physical • Putting material into endnote • Direct export from many sources • From a file • Direct input • How are you going to manage the hard copies • What about your notes? • What systems do you use?

35. Reducing and organising • Using groups and adding notes to Endnote • Using a screening tool (once you’ve ditched the really irrelevant ones)

36. Data extraction/Critical Appraisal • Tools • Many widely available • Can adapt to suit your own study • Could design own • Assessing quality – systems available – be explicit if you are going to do this • Storing the data • Template in word/Table in word • Excel spreadsheet • Other?

37. Useful resources – systematic reviews • Cochrane Collaboration • http://www.cochrane.org/ • http://www.cochrane.org/docs/irmg.htm • Centre for Reviews and Dissemination http://www.york.ac.uk/inst/crd/ handbook for conducting systematic reviews, • http://www.york.ac.uk/inst/crd/methods.htm • Searching for systematic reviews http://www.york.ac.uk/inst/crd/revs.htm • EPPI-Centre – Stages of a review • http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=89 • SCIE - The conduct of systematic research reviews for SCIE knowledge reviews • http://www.scie.org.uk/publications/details.asp?pubID=111