Meta-analysis and systematic reviews in qualitative research: Mission impossible?

1. Meta-analysis and systematic reviews in qualitative research: Mission impossible? Ellen MacEachen (Institute for Work & Health, University of Toronto) Scott Reeves (Li Ka Shing Knowledge Institute, University of Toronto) Qualitative Inquiry Group Seminar University of Toronto, March 27, 2009

2. Overview – 4 parts 1. Where did systematic reviews come from & why do we do them? 2. Brief overview of the systematic review process 3. Cases that draw out systematic review issues • Case I (Scott) • Case II (Ellen) 4. Reflections on the conduct and usefulness of systematic reviews of qualitative studies

3. 1. What are systematic reviews? • Positivistic roots / standardization • Synthesis of primary studies • Multiple check points / dual reviewers / transparency • Assess quality of evidence (only include ‘rigorous’ studies) • Inference through statistical analysis • Newer Qual types (meta-ethnography… )

4. Why do SRs? Useful: • Knowledge translation (policymakers/ practitioners) • Initial entry into field (academics) • Help define field (academics) • Outline areas of future research (acad/policy) • Grant applications (academics)

5. 2. Brief overview of the systematic review process A walk through the ‘recipe’ followed for systematic reviews

7. What keywords are used in the search? What is the scope of the search? Languages? Dates? What stakeholders are involved and at what stages? What databases will be searched? Medline Total = 1620 EMBASE Total = 1683 CINAHL Total = 443 PsycINFO Total = 1174 Sociological Abstracts Total = 279 ASSIA Total = 74 ABI Inform Total = 381 EconLit Total = 58 Business Source Premier Total = 585 Content Experts’ Refs Total = 840 How will we find papers not listed on databases? Other Total = 240 Quality control process for what is being included/excluded Step 1: Library Search What articles are included/excluded? Step 2: Study Relevance Excluded at this level n = 4256 Studies considered for QA (qualitative, quantitative and mixed methods (n =609 ) Supplemental articles from reference list n = 63 Two reviewers per paper, consensus approach Step 3: Division of QNT/QL Studies Excluded at this level n = 20 Excluded at this level n = 13 How will we assess quality? Two reviewers per paper at this stage, consensus approach Step 4: Quality Appraisal What data will we extract? Studies passing quality appraisaln = 5 Step 5: Data Extraction Studies passing quality appraisal n = 14 Step 6: Finding Synthesis n = 19

8. 3. SR ‘cases’ Case I – Making decisions about how to construct systematic reviews: the early days Interprofessional education (Scott) Case II – Struggles with the paradigm of systematic reviews: later days Work & health in small businesses (Ellen)

9. Interprofessional education (IPE) Case I: Making decisions about how to construct systematic reviews: the early days

10. SETTING PARAMETERS Discussion/consultation/agreement: definitions, approaches, processes REVIEW QUESTION what kind of IPE, under what circumstances, produces what kind of outcomes? LITERATURE SEARCHES Retrieval & screening Quantitative studies: Quality assessment, data extraction, synthesis SYNTHESIS Different types of IPE and associated outcomes Qualitative studies: Quality assessment, data extraction, synthesis Mixed method studies: Quality assessment, data extraction, synthesis

11. Context information • Policymaker demands for IPE evidence (early 1990s) • SR – political response • In dark about SR process – new activity • ‘Side line’ work (enthusiasm)

12. SR processes • Inclusive approach to review team – 9 members • Conceptually inclusive (implicit/explicit IPE) • Methodologically inclusive (qual/quan) • Open stance – seeking guidelines/standards • Pragmatic – trial & error • Abstraction – pre-determined categories

13. Thinking about Quality quality of study (appropriate design, sampling, recruitment, validity, reliability) quality of information (good contextual info, explicit rationale, clear research questions, clear results) (e.g. CASP, EPPI Centre, Popay et al 1998)

14. Large and small SRs • Big was good: 107 studies Practitioners/policymakers - larger numbers (width) • Small was better - 21 studies Researchers – small numbers (higher quality)

15. Some reflections • Lots of discussion/debate create parameters – first IPE review • Proceed with caution • One step forward, two back • Gradual movement: from inclusion to exclusion

16. Some reflections • Different types of qualitative research difficult to synthesize • Qual / Quan / mixed methods – more problems • Quality assessment – best effort (pragmatics) • SR team dynamics

17. Case II: Systematic review on work and health in small businesses (Ellen) • Struggles with the paradigm of systematic reviews • 2 parts: • qualitative review • mixed method review

18. REVIEW QUESTION What understandings, processes and interventions influence OHS in SBs? SUB-QUESTIONS Qualitative literature: How do SB workplace parties understand and enact processes related toOHS? Quantitative literature: Do OHS interventions in SBs have an effect on OHS? STAKEHOLDER CONSULTATION LITERATURE SEARCH Retrieval, screening of T&As FOCI FOR IN-DEPTH REVIEW Qualitative studies: Quality assessment, data extraction, synthesis Quantitative studies: Quality assessment, data extraction, synthesis STAKEHOLDER CONSULTATION OVERALL SYNTHESIS Understandings, processes and interventions that influence OHS in SBs

19. Struggles in a Qualitative review • How to observe the broad focus that can be taken with qualitative research while also subjecting it to the ‘recipe’ of a SR process • The SR process is systematic, not exploratory. • Pre-defined parameters

20. Struggles in a Qualitative review • How to synthesize the findings & insights of Qualitative studies while also preserving their context and theory • The systematic review ‘extracts’ findings to answer a specific question. • What insights do we end up with? What do we gain/lose?

21. Struggles in a Qualitative review • What is being counted/considered in a qualitative systematic review? • Varying approaches to data extraction. • pre-set-findings categories? • grounded theory process?

22. Struggles in a Qualitative review • What counts as a qualitative study? • Studies can self-identify as “qualitative” but not use recognizable qualitative methods. • E.g. Participatory studies • Our criteria: Some qualitative data & some qualitative analysis of it

23. Struggles in a Qualitative review • How to handle interesting data from studies that don’t meet quality criteria? • “Nuggets” Pawson (2006). • Descriptions of process that are a ‘nugget of gold’ for the question being asked.

24. Struggles in a Mixed Method review • How to achieve a steady process for both Qualitative & Quantitative sides of the review • Quantitative team-- focused on definitions, outcomes • Qualitative team--no fixed categories for outcomes. • Qual & quan work happens at different times--affects possibilities for concurrent synergy.

25. Struggles in a Mixed Method review • Theoretical differences about ‘conflict of interest’ during the quality assessment process • Qualitative team--sensitive to social/power relations • heated issue, agreed to disagree.

26. Struggles in a Mixed Method review • Differences about ‘levels of evidence’ • QN additive approach • E.g. "best evidence synthesis guidelines”. • “How much evidence is there that this has an effect?” • QL studies—not about effectiveness • Some additive approach; e.g. concepts in 3 studies = theme • No overall conclusions about strength of evidence

27. IV. Reflections on the conduct and usefulness of systematic reviews • SR process developed from Quantitative paradigm; we are trying to adapt Qualitative literature to it… …Can this be meaningfully done?

28. Issues Reviews are not entirely systematic • Messiness • Paired reviewers: anti-bias ‘ideal’ • Politics of agreement/consensus BUT: • Power relations among reviewers • Assessing papers: own merits or against overall standard? • QA approaches change over course of the review

29. Issues • The issue of the podium • Arrogance: decisions on inclusion of peer reviewed papers • A side effect of systematic reviews • Claims of ‘ownership’ of a field from SR work

30. Issues • Debates: • Can the synthesis integrate research using different methods/theories • Although there are multiple descriptions/ explanations of data, these all ultimately relate to some underlying reality/truth (Bondas & Hall 2007, Mays et al 2005)

31. Some gains • Think carefully about papers • Learn a lot about a field • Read a lot of poor papers (learn: good, bad ugly) • Hone critical appraisal skills (teaching, journal reviewing/editing) • Opportunities to engage with people (outside academia)

32. Thank you • Questions? Comments?