James Thomas EPPI-Centre, Social Science Research Unit, Institute of Education, University of London

1. Mixing different types of research in systematic reviewsMethodological Challenges for the Twenty First CenturyManchester, 22-23 November 2007 James Thomas EPPI-Centre, Social Science Research Unit, Institute of Education, University of London

2. RESEARCH SYNTHESIS/ SYSTEMATIC REVIEWS • A particular type or class of reviews • Bridge between research and policy and practice • Usually question-driven • Use explicit methods, taking steps to increase trustworthiness • Observational research or ‘Research on research’

3. Setting question and developing protocol Defining studies (inclusion and exclusion criteria) Searching exhaustively (search strategy) Describing the key features of studies Assessing their quality/weight of evidence Synthesising findings across studies Communication and engagement Typical stages of a systematic review

4. Effectiveness Screening and diagnosis Exploring risk or protective factors Observational associations between interventions and outcomes Questions about prevalence Questions about meanings and process Methodological questions Economic questions Types of questions for systematic reviews* * from Petticrew and Roberts (2005)

5. Examples of synthesis methods • Statistical meta-analysis • Meta-ethnography • Grounded theory • Thematic analysis • Realist synthesis • Critical interpretive synthesis • Bayesian synthesis Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A (2005) Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy 10: 45-53.

6. Why synthesise quantitative research? Meta-analysis refers to the analysis of analyses . . . the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature. Glass, 1976, p 3

7. Does sex education increase the use of contraception amongst young people? From DiCenso et al. (2002) Interventions to reduce unintended pregnancies amongst adolescents: a systematic review of randomised controlled trials. British Medical Journal 231: 1426-1434

8. Why synthesise qualitative research? “…the full contribution of qualitative research will not be realised if individual studies merely accumulate and some kind of synthesis is not carried out…there are generalisations to be made across qualitative research studies that do not supplant the detailed findings of individual studies, but add to them” Britten et al. (2002)

9. Deriving descriptivethemes

10. ‘Mixed method’ systematic reviews (1/2) • Policy and practice concerns often precede, or go beyond, questions of effectiveness • Different types of questions likely to be answered by different types of findings • Different types of findings may require different types of synthesis

11. ‘Mixed method’ systematic reviews (1/2) Single reviews with three syntheses • Effect sizes from trials pooled using statistical meta-analysis • Findings from qualitative studies synthesised using thematic analysis • Synthesis 2) used to interrogate synthesis 1)

12. An example of a ‘mixed method’ review Children and healthy eating: a systematic review of barriers and facilitators* *Thomas J, Sutcliffe K, Harden A, Oakley A, Oliver S, Rees R, Brunton G, Kavanagh J (2003a) Children and Healthy Eating: A systematic review of barriers and facilitators. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London (The full report of this review is available at the EPPI-Centre website http://eppi.ioe.ac.uk/EPPIWeb/home.aspx) :.

13. REVIEW PROCESS Searching, screening and mapping Focus narrowed to ‘fruit &veg’ Synthesis 2: Qualitative studies (n=8) 1. Quality assessment 2. Data extraction 3. Thematic synthesis Synthesis 1: Trials (n=33) 1. Quality assessment 2. Data extraction 3. Statistical meta-analysis Synthesis 3: Trials and qualitative studies

14. Synthesis 1: Statistical meta-analysis

15. Line-by-line coding(Synthesis 2)

16. Children consider taste, not health, to be a key influence on their food choice Food labelled as healthy may lead children to reject them (‘I don’t like it so it must be healthy’) Buying healthy foods not seen as a legitimate use of their pocket money Synthesis 2: Thematic analysis 1) Children don’t see it as their role to be interested in health. 2) Children do not see future health consequences as personally relevant or credible. 3) Fruit, vegetables and confectionary have very different meanings for children. 4) Children actively seek ways to exercise their own choices with regard to foods. 5) Children value eating as a social occasion. 6) Children recognise contradiction between what is promoted and what is provided.

17. Brand fruit and vegetables as ‘tasty’ rather than ‘healthy’. Reduce health emphasis of messages Do not promote fruit and vegetables in the same way within the same intervention. Create situations for children to have ownership over their food choices. Ensure messages promoting fruit and vegetables are supported by appropriate access to fruit and vegetables Preparing for synthesis 3 1) Children don’t see it as their role to be interested in health. 2) Children do not see future health consequences as personally relevant or credible. 3) Fruit, vegetables and confectionary have very different meanings for children. 4) Children actively seek ways to exercise their own choices with regard to foods. 5) Children value eating as a social occasion. 6) Children recognise contradiction between what is promoted and what is provided.

18. Synthesis 3: Across studies

19. Synthesis 3: Across studies Increase (standardised portions per day) in vegetable intake across trials Little or no emphasis on health messages

20. ‘Added value’ of mixing methods • Operationally, the method is simple, but conceptually, it is strong • Integrates ‘quantitative’ estimates of benefit and harm with ‘qualitative’ understanding from people’s lives • Facilitates a critical analysis of intervention studies from the point of view of those targeted by interventions - and vice versa • Preserves the integrity of the findings of the different types of studies

21. James ThomasEPPI-CentreSSRU18 Woburn SquareLondon, WC1H 0NR Email: j.thomas@ioe.ac.uk The methods described here can be found in: Thomas et al (2004) Integrating qualitative research with trials in systematic reviews. British Medical Journal 328:1010-1012 For full details of the systematic review discussed in this paper and other EPPI-Centre reviews please see our website: http://eppi.ioe.ac.uk/