Mixed method reviews

1. Mixed method reviews Data analysis and synthesis

2. Complexity “Some problems are so complex that you have to be highly intelligent just to be undecided about them” Lawrence J. Peter

3. Narrative synthesis (NS) • An approach to the synthesis of findings from multiple studies that relies primarily on the use of words and text to summarise and explain the findings of the synthesis • Designed for integrating effectiveness and implementation data, but can be used to synthesise evidence for a wide range of questions including effectiveness, experience, need, process, implementation etc.

4. Advantages Assists in review team communication and documentation Assists in making the processes of synthesis available to others for validation Assists with the presentation and communication of the results of synthesis Using a synthesis method

5. Guidance • Pope C, Mays N, Popay J (2007) Synthesising Quantitative and Qualitative Evidence: a guide to methods. Milton Keynes,Open University Press • Popay J et al. (2006) Guidance on the Conduct of Narrative Synthesis in Systematic Reviews. Available at: http://www.lancs.ac.uk/shm/research/nssr/research/dissemination/publications/NS_Synthesis_Guidance_v1.pdf

6. A framework to increase the transparency and trustworthiness of the synthesis Includes description of tools and techniques Could be used to synthesise the findings from mixed evidence sources A guide to the whole process of systematic review – only the stage of synthesis A cookbook of tools and techniques to be selected at random Aims to be accessible, but not a guide for complete novices in research What NS is, and isn’t

7. Relationship to systematic review Question development Theoretical framework Search/data extraction Quality assessment Descriptive analysis Preliminary synthesis Interpretive analysis Final modelling/synthesis Theory testing /development Iterative not linear Looking for patterns in data Generalise on basis of theory

8. Phases of NS • Phase 1: Develop a theoretical framework* • Phase 2: Descriptive analysis + develop a preliminary synthesis • Phase 3: Interpretive analysis: explore relationships in the data + construct synthesis • Phase 4: Assess the robustness of the synthesis • NB Not linear stages: phases are iterative *not always required

9. Theoretical framework • Purpose: to inform decisions about the review question, contribute to interpretation of findings, and to assess the applicability of the findings • Sources: • Research question • Topic • Theory • Hypotheses

10. Descriptive analysis • Purpose: to organise findings to provide an initial description of patterns across the included studies • Tasks: select, group, juxtapose • Tools: • Textual descriptions • Groupings and clusters • Tabulation • Transform to a common rubric • Vote count • Thematic analysis • Content analysis

11. Descriptive analysis

12. Descriptive analysis SELECT GROUP JUXTAPOSE

13. Descriptive analysis • Textual descriptions • Groupings and clusters • Tabulation • Transform to a common rubric • Counting • Thematic analysis • Content analysis

14. Tools for describing: examples • Textual descriptions: a short ‘story’ with key messages, significant information that acts as a ‘viewing platform’. • Ordering/grouping: finding shared features and clustering studies along those lines • Tabulation: Tables summarising key features of study objectives, design, context, findings • Common rubric: Using a common framework to allocate material from multiple studies • Counting: Using occurrence or vote counting to consider the frequency of phenomena in the data • Content/thematic analysis: qualitative analysis

15. EXERCISE: description • Select summary information from the narrative description to put in a table, making sure that the table will fit data from all three studies • Look at the types of clients, and context. Can you spot potential groupings? • Look at the findings across the three studies. Can you spot potential categories or themes?

16. Output: preliminary synthesis • Descriptive narrative and tabular summary of the studies and findings • Specification of analytical framework/common rubric • Events, participants, contexts? • Stages, phases, time? • Ordering, grouping? • Positives, negatives? • Hierarchies, key issues? • Preliminary synthesis • Testing the theory with a subset of data to make sure it works

17. Exploring relationships in the data • Purpose: to consider relationships between: • study results and key aspects of those studies (within) • factors across included studies (between) • Tasks: compare, combine, translate, integrate • Tools: • Identifying mediators and moderators • Sub-set analysis • Conceptual mapping • Translation: reciprocal/refutational • Qualitative case description

18. Exploring relationships in the data COMPARE COMBINE TRANSLATE INTEGRATE RELATIONSHIPS ACROSS STUDIES RELATIONSHIPSWITHIN STUDIES

19. Exploring relationships in the data Mediators and moderators Sub-set analysis Conceptual mapping Translation: reciprocal/ refutational Qualitative case descriptions RELATIONSHIPS ACROSS STUDIES RELATIONSHIPS WITHIN STUDIES

20. Exploring relationships: tools • Moderators and mediators: categorising and viewing the relationship between concepts according to their action on the phenomena of interest • Sub-set analysis: Looking at actions, results or themes in groups, dependent on characteristics • Visual representation: • Concept mapping: links between concepts are labelled and placed in space to express a relationship • Ideas webbing: concepts organised in trees, hierarchies • Causal/cognitive mapping: representation aims to describe direction of influences, key issues in real life • Translation: reciprocal/refutational: grouping and integrating study findings in terms of level of agreement/fit in argument/conclusions • Qualitative case description: writing a short narrative summary across studies, concepts, themes, arguments

21. Output: Synthesis • Rigorous exploration of alternative interpretations of relationships between and across studies • Some level of translation and integration with explanation of (e.g.) • agreement/refutation • commonalities/outliers • order/groupings • priorities/emphasis/frequency • Identification of moderators (influencing factors), mediators (mechanisms of action) and sub-groups (components of programme theory)

22. EXERCISE: interpretation • Look at what you think might be the key concepts/themes drawn from the findings of the 3 studies • Play with the concepts/themes spatially to explore the potential relationships and patterns across concepts • Map the concepts/themes ACROSS STUDIES to denote some aspect of the relationship between them

23. Assessing robustness • Purpose: to assess the strength of evidence for drawing conclusions about the results of the synthesis in relation to methodological quality, relevance, trustworthiness • Tasks: assess relationship of synthesis to original review question, programme theory, study quality and review method • Tools: • Assessing theoretical fit • Validity assessment • Weight of evidence • Critical reflection • Triangulation • Checking with authors

24. Assessing robustness REVIEW QUESTIONS PROGRAMME THEORY

25. Assessing robustness REVIEW QUESTIONS Best Evidence Synthesis Weight of evidence Validity assessment Critical reflection Checking with authors Triangulation STUDY QUALITY REVIEW METHOD PROGRAMME THEORY

26. Tools: assessing robustness • Best evidence synthesis: prioritising/choosing studies with the strongest design • Weight of evidence: using a criterion of strength or value to interpret the synthesis findings e.g. effect size, repeatability, harmony, coherence • Use of validity/credibility assessment: using the results of critical appraisal to interpret synthesis • Critical reflection: using the review author’s use of critical tools/techniques/methods to consider results e.g. agreement, alternative explanations, assessment of theoretical fit, parsimony, elegance • Checking with authors: involving the authors of the primary studies in comment, reflection, or building theory/synthesis • Triangulation: using the results of other data sources to compare the results of the synthesis

27. Exercise • How do you think the methods and quality of the original studies would affect the interpretation of the findings?

28. Output: robustness • Identification of any limitations in the process of review and synthesis • Assessment of the credibility and trustworthiness of the synthesis product, seen through the provenance, quality, and strength of the evidence • Assessment of the significance of the theoretical contribution • Identification of its potential use and further development

29. Review product suggestions • Formal report: PRISMA guidelines • method • analysis: description, interpretation • conclusion: theoretical contribution • implications • Practitioner report: • content, • examples, visual illustration • best practice guidance • tools, resource lists

30. Common problems • Volume of material • Level of material • Heterogeneity of material • Disaggregation • Crossing disciplinary boundaries • Lack of guidance on synthesis method • Time • Skill in original research methodology