Review

1. Review Professor Jan Illing Centre for Medical Education Research School of Medicine, Pharmacy and Health Durham University

2. Overview of the lecture • The role of the literature review in research • Types of evidence: Logic of aggregation vs. logic of configuration • Different types of literature review • Narrative • Systematic • Meta Analysis • Realist synthesis • Synthesising qualitative research • Overview of the stages of a systematic review: • research question, inclusion and exclusion criteria, searchers of title, • abstract and papers, data extraction critical appraisal, assessing quality. • Reviews: Cochrane collaboration, Campbell collaboration, BEME

3. The role of the literature review • Need to understand the topic and what is already known • The review places research in the context of a body of literature • Informs what still needs to be researched, before it is known • Learn about the methods used previously – what methods are appropriate, or not, and where new approaches are needed • e.g. need for a qualitative approach • Avoids repeating what is known, or repeating studies that have already failed. • Provides information on the strength of available evidence, • quality of the studies • and identifies gaps

4. How does a literature review help you? • Helps to understand how one study is related to others • Highlights conflict in findings • Can suggests new interpretations and new insights • Can identify what research is needed next

5. Different types of literature review answer different types of question

6. Types of evidence • Aggregate (counting evidence) vs. Interpretive (seeks understanding) • Each study adds ‘weight’ Broadens or deepens understanding • Studies add more weight Add insight to a pattern or emerging view • with sample size, Add theory • study design and • strength of findings • Reporting statistics Reporting themes

7. Types of review: Narrative • Narrative – also called non-systematic, traditional and conventional • Provides a snap shot of prevalence knowledge • Criticisms include: • The selection and inclusion of papers are not clear, • Incomplete and sometime missing • Risk of bias from selection of included studies • Not easily to reproduce • Conclusions are questionable from the quality of methodology used and quality of reporting

8. Systematic literature review • Unlike the narrative review, the systematic approach aims to search all (not just part) of the relevant literature in a systematic way, aiming to use – • “a reproducible method for identifying, evaluating, and synthesising the existing body of completed and recorded work produced by researchers, scholars, and practitioners” (Fink 2005) • Systematic reviews aim to inform decisions and need to be made on the best available evidence (Booth et al, 2012) • Seen as ‘a scientific activity’ (Mulrow, 1995) • Often continuously updated (Cochrane reviews)

9. Systematic literature review • Weaknesses • The criteria used to identify the studies can also be a source of bias • or a limitation (Song et al., 2010) • Limited by year published e.g. last 10 years may miss important research published before this time • Limited by language e.g. publication in English may focus on positive findings, publications in authors own language may report more negative findings • Limiting searches to well-known databases can result in missing information. • E.g. An audit of sources for a review of complex interventions (including qualitative evidence) found that only 30% were identified from databases and hand searches. About half of studies were identified by ‘snowballing’ and another 24% by personal knowledge or personal contact (Greenhalgh 2005).

10. What a review adds • Can help answer important questions that inform the value of an intervention, policy, practice, and identify what new research to do • Can identify if finding tend to be consistent across a theme • Areas of conflict • “Disagreements among findings are valuable… conflict can teach us a lot” • Conflicts often identify important differences in sample and context • May indicate a need for more research • Could be used to illustrate a theory • Highlight poor research methods.

11. Hierarchy of research evidence

12. Stages of a systematic review • Devise a research question • Develop a protocol – objectives, search strategy, inclusion and exclusion criteria • Search literature – define search terms and databases and relevant grey literature, and searches from references of key papers • Search for relevant studies abstract, then papers • Data extraction and critical appraise studies • Synthesis of findings • Interpretation • Write up

13. A Scoping Review • Provides a snap shot on a topic • The focus is on the range of studies that have been done • They may focus on an issue • e.g. what theoretical approaches have been used to study x • There is no attempt to examine quality • (Arksey and O’Malley, 2005)

14. Meta Analysis • Helps to answer questions about interventions when sample sizes need • to be increased • Systematic search with inclusion/exclusion criteria • Focus is to find studies on the same topic using the • same study design e.g. RCT • Combines results of quantitative studies • Conducts statistical tests by combining data • The analysis is numerical • focus on the size of the effect

15. Realist synthesis • Realist synthesis is used mainly for the assessment of complex interventions • Starts with a systematic search of the literature • Uses an interpretive approach, looking for patterns • Aims to explain how outcomes occur • Theory-driven approach • Using evidence from qualitative and quantitative research methods • The goal is to identify mid-range theories

16. Context Mechanism Outcome • The realist research question is often summarized as: • "What works for whom under what circumstances, how and why?" • Realist inquiry is based on a realist philosophy of science (reality can be known indirectly) and considers the interaction between context, mechanism and outcome • Each study is read and the reviewers seek to identify: • the contextual (C) influences • that are hypothesized to have triggered the relevant mechanism(s) (M) • and generated the outcome(s) (O). Known as CMOs • “Education is a complex intervention which produces different outcomes • in different circumstances” (Wong et al 2012 Medical Education p89-96)

17. Development of theory • Realist synthesis looks for mechanisms and middle-range theories • The aim is to test and build theory • Mechanisms can be defined as: • “…underlying entities, processes, or structures which operate in particular contexts to generate outcomes of interest” (Astbury and Leeuw, 2010) • Mechanisms tend to generate outcomes, and are usually hidden • and are sensitive to changes in context. • Middle-range theories are developed from real data, so can be tested. • Programme theory explains how the intervention is expected to reach its objectives.

18. Realist synthesis and secondary searches • Realist reviews start with a systematic review of the literature • Assess the included papers and identify the CMOs • Often involve a second search to find more studies to help explain • and test out possible theories • Appraising the evidence • Relevance - can the study contribute to theory building or theory testing • Rigour - whether the method used to generate data is credible and trustworthy

19. Example: Realist synthesis of the literature on workplace bullying in the NHS • A systematic literature review focuses on research design • and outcomes, but a realist review looks for patterns in context. • A realist review aims to explain why an intervention may work • in one situation but not in another • Findings from a realist synthesis focused on producing a • theory to explain patterns in the data. • The findings from the study on workplace bullying highlighted • that interventions were more likely to be successful if the • senior managers were fully committed to them. • Illing, J, et al (2013). Evidence Synthesis on the occurrence, causes, consequences, prevention and management of bullying and harassing behaviours to inform decision-making in the NHS. HMSO.

20. Qualitative reviews • The focus is on understanding rather than effectiveness • The focus on effectiveness has privileged quantitative evidence over • qualitative, and reduced consideration of other options to understand • more about the impact of interventions. • Complexity and context • also need to be considered • by practitioners and policy-makers

21. Searching for qualitative research • “It is increasingly being recognized that evidence from qualitative studies that explore the experience of those involved in providing and receiving interventions, and studies evaluating factors that shape the implementation of interventions, have an important role in ensuring that systematic reviews are of maximum value to policy, practice and consumer decision-making” (Mays 2005, Arai 2005, Popay 2005). • Databases have improved filters to include qualitative research for MEDLINE (Wong 2004), CINAHL (Wilczynski 2007), PsycINFO(McKibbon 2006) and EMBASE (Walters 2006). • However, identifying qualitative research is problematic due to the varied use of the term ‘qualitative’ (Grant 2004). • Others argue for a purposive sampling approach, where searching is driven by the need to reach theoretical saturation and the identification of the ‘disconfirming case’, may be more appropriate (Dixon-Woods 2006).

22. Critical appraisal of studies • Weaknesses and flaws • Need to consider which are fatal and which are non-critical • There is no such thing as the perfect study and avoid nihilism (Sackett, 1997) • Need to consider how much a weaknesses impacts on the findings • Right research design and method for the research question • Can the results be relied upon (valid, true) • Are the results trustworthy, not just due to chance (reliable) • Rigour – the results are not biased, confounding factors e.g. age and gender • Limitations from selection bias, measurement of outcome, and analysis

23. Assessment of quality • Quality and relevance check lists vary by discipline • For novice researchers a check list can be helpful, • Provides a list of everything that should be considered. • Aim to use check lists that are widely accepted or validated • Checklists that use a scoring system are not recommended • (Higgins and Green, 2008)

24. The problem with scales • The use of scales for assessing quality is discouraged • in Cochrane reviews. Offers appealing simplicity, • not supported by evidence (Emerson 1990, Schulz 1995b). • A score involves assigning • ‘weights’to items in the scale, • difficult to justify the weights assigned. • Scales have been shown to be • unreliable assessments of validity (Jüni 1999) • They are less transparent to users of the review. • Preferable to use simple approaches for assessing validity that can be fully reported (i.e. how each trial was rated on each criterion)

25. Assessment of quality:in qualitative studies • Numerous tools are available to assess the quality of qualitative research • (Cote, 2005, Medical Teacher). • However, three structured appraisal approaches were compared and • found that structured assessment tools may not produce greater • consistency of judgements (Dixon-Woods, 2007Journal of Health Services Research and Policy) • Check lists • Evidence for Policy and Practice Information (EPPI) • Joanna Briggs Institute (JBI) Web site: www.joannabriggs.edu.au/cqrmg • Spencer et al. 2003 conducted a review of many of the current appraisal frameworks and checklists, which may be helpful • Critical Appraisal Skills Programme 2006 : http://www.phru.nhs.uk/Pages/PHD/resources.htm • Expert judgement is also an important factor

26. Quality prompts in qualitative research • Are the aims and objectives clearly stated? • Is the research design clear and • appropriate for the research question? • Are the processes of obtaining the data clear? • Is sufficient data provided to link interpretation • and conclusions back to the data? • Is the method of analysis appropriate and adequately explained? • (Dixon-Woods et al, 2006, BMC Medical Research Methodology)

27. Quality indicators for all studies. • Is the research question(s) or hypothesis clearly stated? • Is the subject group appropriate for the study (number, characteristics)? • Are the methods used (qualitative/quantitative) reliable and valid for the research question and context? • Completeness of ‘data’ Have subjects dropped out? attrition rate < 50%? • Control for confounding Have multiple factors/variables been removed or accounted for where possible? • Are the statistical or other methods of analysis used appropriate? • Is it clear that the data justify the conclusions drawn? • Could the study be repeated by other researchers? • Does the study look forwards in time (prospective) rather than backwards? • Ethical issues, Were all relevant ethical issues addressed? • Triangulation, Were results supported by data from more than one source? • (Buckley et al, 2009, BEME review 11).

28. Collaborations • Cochrane collaboration • Joanna Briggs Institute • Campbell collaboration, • Best Evidence Medical Education

29. The Cochrane Collaboration • The Cochrane Collaboration was set up over 40 years • ago by the epidemiologist Archie Cochrane. • The Cochrane Collaboration was set up to promote • Clinical trails, using systematic review and meta-analysis • to synthesise data. • Evidence was rated strong or weak according to standard • criteria which included research design and statistical effect size. • Helpful in making decisions about treatments – single variables, • focused on outcomes.

30. Other approaches to systematic review • In Health • Joanna Briggs Institute (http://www.joannabriggs.edu.au) • The central focus not only effectiveness, but also appropriateness, meaningfulness and feasibility of health practices and delivery methods. These questions are often answered by considering other forms of research evidence • Also W K Kellogg Foundation (http://www.wkkf.org) • Aim to improve health care practice, but do not place randomised controlled as high up in the hierarchy. • In Medical Education • Best Evidence Medical Education (http://bemecollaboration.org) • Realist synthesis – RAMESUS project –what works, for whom, under what circumstances.

31. Campbell collaboration • The Campbell collaboration hosts reviews on: • education, • crime and justice, • social welfare and • international development.The Campbell Collaboration is an international research network that produces systematic reviews of the effects of social interventions. • http:wwwcampbellcollaboration.org/

32. Best available evidence • Traditionally, the evidence based practice movement has focussed on the results of quantitative evidence (considering the RCT as the gold standard) to answer questions of effectiveness. • Joanna Briggs Institute (JBI) has as its central focus not only effectiveness, but also appropriateness, meaningfulness and feasibility of health practices and delivery methods. • Best answered by considering other forms of evidence. The JBI regards the results of well-designed research studies grounded in any methodological position as providing more credible evidence than anecdotes or personal opinion. However, when no research evidence exists, expert opinion can be seen to represent the ”best available” evidence.

33. Conclusions Literature reviews provide a range of purposes, including knowing what has already been done and what needs to be done The approach used depends on the question and purpose of the review There are a range of different types of review: narrative, systematic, meta analysis, realist synthesis Most of these start with a systematic review Assessment of quality can challenging, useful tools to support focus Collaborations Cochrane collaboration, Campbell collaboration, BEMEeffectiveness, As well as effectiveness we need to think about using reviews to answer questions about appropriateness, meaningfulness and feasibility

34. Thank youQuestions? j.c.illing@dur.ac.uk Centre for Medical Education Research School of Medicine, Pharmacy and Health Durham University