Joanne McKenzie, School of Public Health and Preventive Medicine 19 th Cochrane Colloquium

1. Methods for synthesizing evidence of the effects of healthcare interventions in systematic reviews of complex interventions (including statistical approaches) Joanne McKenzie, School of Public Health and Preventive Medicine 19th Cochrane Colloquium Madrid 2011

2. Google’s take on complexity … www.nicholsoncartoons.com.au Methods for synthesizing evidence of the effects of healthcare interventions

3. Acknowledgements • Sue Brennan, Monash University, Australia • Sophie Hill, La Trobe University, Australia

4. Components of complexity in a systematic review Summary to synthesis (pros and cons) Available methods Outcome categorisation Conclusions and questions raised Outline Methods for synthesizing evidence of the effects of healthcare interventions

5. Splitting Lumping Condition One condition (e.g. diabetes) Any condition Outcome Consistent outcomes (e.g. test ordering) Diverse outcomes Setting One setting (e.g. primary care) All settings Intervention Narrow(e.g. audit cycles) Any form(e.g. any QI intervention) Study design Multiple designs(e.g. RCT, ITS, CBA) One design(e.g. RCT) Methods for synthesizing evidence of the effects of healthcare interventions October 2011 5

6. Summary Synthesis Vote counting Summary of effect estimates Meta-analysis • descriptive statisticsbox and whisker plots • texttabular • texttabularharvest plots • meta-analysispredictive intervalsforest plots Exploring heterogeneity • sub-group analysismeta-regressiongraphical approaches Methods for synthesizing evidence of the effects of healthcare interventions October 2011 6

7. Summary • Results of studies summarised: • in the text of a publication without the use of a synthesis method • in tables, providing a structured method for presenting data Pros Text: Provides an assembly of the available research meeting the inclusion/exclusion criteria. Tables: More likely to report all results of all outcomes (i.e. may be less likely to selectively include results). Results available for others to synthesize. Cons Text: Results summarised, not synthesized. Little structure to reporting results may lead to selective reporting (privileging of findings above others). Interpretation of results difficult/not possible. Tables: Results summarised, not synthesized. Overwhelming amount of information which is difficult for a reader to interpret (often multiple outcomes per study). Methods for synthesizing evidence of the effects of healthcare interventions October 2011 7

8. Summary • Results of studies summarised: • in the text of a publication without the use of a synthesis method • in tables, providing a structured method for presenting data Pros Text: Provides an assembly of the available research meeting the inclusion/exclusion criteria. Tables: More likely to report all results of all outcomes (i.e. may be less likely to selectively include results). Results available for others to synthesize. Cons Text: Results summarised, not synthesized. Little structure to reporting results may lead to selective reporting (privileging of findings above others). Interpretation of results difficult/not possible. Tables: Results summarised, not synthesized. Overwhelming amount of information which is difficult for a reader to interpret (often multiple outcomes per study). Methods for synthesizing evidence of the effects of healthcare interventions October 2011 8

9. Methods for synthesizing evidence of the effects of healthcare interventions October 2011 9

10. Methods for synthesizing evidence of the effects of healthcare interventions October 2011 10

11. Outcome Any outcome: attitude (6), knowledge (10), skill/behaviour (6), process (27), patient (18) Setting Condition Any setting: ambulatory (22), inpatient/nursing home (10), mixed clinical (3), educational (4) Any condition/aspect of care: preventive care, diabetes, asthma, hypertension, HIV, renal failure, palliative care, coronary heart failure, stroke, pain management, falls prevention, neonatal infection, anticipatory guidance, compensation,wait times Intervention Any form: QI education for trainees (10), QI education for nontrainees (5), ‘other’ interventions with an educational component (5), education within a QI collaborative (19) Study design Multiple designs: RCT (8), nonrandomised trial (14), pre/post or time series (17) Quality improvement education for clinicians Boonyasai, JAMA 2007 Lumping Splitting Methods for synthesizing evidence of the effects of healthcare interventions October 2011 11

12. Study 1 Study 3 Study 2 • 27/39 studies measured process outcomes (3 example studies) • 15 clinical areas [Boonyasai JAMA 2007]

13. Outcome categorisation (EPOC) Example CQI review: • Healthcare professional performance (binary, continuous) • e.g. adherence to recommended practice • Patient outcomes (binary, continuous) • e.g. pain, quality of life, function, mortality • e.g. patient experience of care, patient evaluation of care co-ordination, length of stay • Other outcomes • e.g. resource use [Brennan Cochrane Database Syst Rev 2009] Methods for synthesizing evidence of the effects of healthcare interventions October 2011 13

14. Outcome categorisation (CCRG) Table 4.2: Outcomes of importance to consumers, communication and participation: a new taxonomy [Hill Wiley-Blackwell 2011] Methods for synthesizing evidence of the effects of healthcare interventions October 2011 14

15. Synthesis • Vote counting: • “Is there any evidence of an effect?” • # studies showing harm compared with # studies showing benefit (regardless of stat. sig. or size of results) • Sign test used to assess the stat. sig. of evidence of an effect in either direction Pros Provides a method for synthesizing effects when standard meta-analytical methods difficult to apply (e.g. variances of effect estimates not available). Cons Provides no information on the magnitude of effects (e.g. equal importance given to risk difference of 5% and 50%) . No account of differential weighting across the studies. Problems when stat. sig. used to define # positive and # negative studies (unit of analysis errors, underpowered studies). Methods for synthesizing evidence of the effects of healthcare interventions October 2011 15

16. Synthesis • Vote counting: • “Is there any evidence of an effect?” • # studies showing harm compared with # studies showing benefit (regardless of stat. sig. or size of results) • Sign test used to assess the stat. sig. of evidence of an effect in either direction Pros Provides a method for synthesizing effects when standard meta-analytical methods difficult to apply (e.g. variances of effect estimates not available). Cons Provides no information on the magnitude of effects (e.g. equal importance given to risk difference of 5% and 50%) . No account of differential weighting across the studies. Problems when stat. sig. used to define # positive and # negative studies (unit of analysis errors, underpowered studies). Methods for synthesizing evidence of the effects of healthcare interventions October 2011 16

17. Effectiveness of teaching quality improvement to clinicians [Boonyasai JAMA 2007] Methods for synthesizing evidence of the effects of healthcare interventions October 2011 17

18. Harvest plots Crowther Res Syn Meth 2011

19. Synthesis • Summary of effect estimates: • “What is the range and distribution of effects?” • ‘Median-of-medians’ approach (EPOC). • One outcome chosen per outcome category (selection process independent of result & stat. sig.). Effect size associated with this outcome used to ‘characterise’ the outcome of the study. Pros Provides a method for synthesizing results when difficult to undertake a meta-analysis (e.g. missing variances of effects, unit of analysis errors). Provides information on the magnitude and range of effects (IQR, range). Cons Does not weight effects; small studies are as influential as large studies. Doesn’t use all available data for a particular outcome category. Methods for synthesizing evidence of the effects of healthcare interventions October 2011 19

20. Synthesis • Summary of effect estimates: • “What is the range and distribution of effects?” • ‘Median-of-medians’ approach (EPOC). • One outcome chosen per outcome category (selection process independent of result & stat. sig.). Effect size associated with this outcome used to ‘characterise’ the outcome of the study. Pros Provides a method for synthesizing results when difficult to undertake a meta-analysis (e.g. missing variances of effects, unit of analysis errors). Provides information on the magnitude and range of effects (IQR, range). Cons Does not weight effects; small studies are as influential as large studies. Doesn’t use all available data for a particular outcome category. Methods for synthesizing evidence of the effects of healthcare interventions October 2011 20

21. Printed educational materials: effects on professional practice and health care outcomes [Farmer Cochrane Database Syst Rev 2008] Methods for synthesizing evidence of the effects of healthcare interventions October 2011 21

22. Audit and feedback: effects on professional practice and health care outcomes [Jamtvedt Cochrane Database Syst Rev 2006 ] Methods for synthesizing evidence of the effects of healthcare interventions October 2011 22

23. Synthesis • Meta-analysis: • “What is the average intervention effect?” (random effects meta-analysis) • Prediction intervals can be calculated to complement information of random effects meta-analysis; “What is the potential effect of an intervention in an individual study?” Pros Provides a combined estimate of average intervention effect (random effects), and certainty in this estimate (95% CI). Weights estimates of effect; small studies are (generally) less influential compared with large studies. Predictive intervals can be calculated; helpful when there is unexplained heterogeneity. Forest plots display study effect estimates and CIs; can display pooled effect; familiar. Cons Requires variances of the effects. Argued that a meta-analytic estimate (average effect) may be of little value when there is heterogeneity. Particularly if there is inconsistency in the direction of effect. Methods for synthesizing evidence of the effects of healthcare interventions October 2011 23

24. Synthesis • Meta-analysis: • “What is the average intervention effect?” (random effects meta-analysis) • Prediction intervals can be calculated to complement information of random effects meta-analysis; “What is the potential effect of an intervention in an individual study?” Pros Provides a combined estimate of average intervention effect (random effects), and certainty in this estimate (95% CI). Weights estimates of effect; small studies are (generally) less influential compared with large studies. Predictive intervals can be calculated; helpful when there is unexplained heterogeneity. Forest plots display study effect estimates and CIs; can display pooled effect; familiar. Cons Requires variances of the effects. Argued that a meta-analytic estimate (average effect) may be of little value when there is heterogeneity. Particularly if there is inconsistency in the direction of effect. Methods for synthesizing evidence of the effects of healthcare interventions October 2011 24

25. Exploring heterogeneity • Sub-group analysis, meta-regression, other statistical approaches: • “What factors modify the size of the intervention effect?” • Can be used to investigate components (‘active ingredients’) of multifaceted interventions which may modify effects. Pros Provides hypotheses regarding what (set) of factors might be necessary for the intervention to be effective. Cons Observational analysis; may suffer confounding bias; aggregation bias; overfitting and spurious claims of association. Investigation of intervention components when there are many is difficult (e.g. assumptions of additivity, correlation between combinations of components). Requires variances of effects, measurement of factors. Technical issues with baseline compliance. Methods for synthesizing evidence of the effects of healthcare interventions October 2011 25

26. Exploring heterogeneity • Sub-group analysis, meta-regression, other statistical approaches: • “What factors modify the size of the intervention effect?” • Can be used to investigate components (‘active ingredients’) of multifaceted interventions which may modify effects. Pros Provides hypotheses regarding what (set) of factors might be necessary for the intervention to be effective. Cons Observational analysis; may suffer confounding bias; aggregation bias; overfitting and spurious claims of association. Investigation of intervention components when there are many is difficult (e.g. assumptions of additivity, correlation between combinations of components). Requires variances of effects, measurement of factors. Technical issues with baseline compliance. Methods for synthesizing evidence of the effects of healthcare interventions October 2011 26

27. Audit and feedback: effects on professional practice and health care outcomes [Jamtvedt Cochrane Database Syst Rev 2006] Methods for synthesizing evidence of the effects of healthcare interventions October 2011 27

28. Re-analysis of audit & feedback review Authors used theory (behaviour change) to categorise intervention components (feedback, performance target, action plan), and investigated if components modified the effects. [Gardner Soc Sci Med 2010]

29. Conclusions • Diversity of interventions, settings, conditions, outcomes, and study designs complicates the synthesis of evidence. • A range of ‘synthesis’ approaches are available; some are clearly better than others. • Limitations in quantitative synthesis should be acknowledged, but may be preferable to “qualitative interpretation of results, or hidden quasi-quantitative analysis …” [Ioannidis BMJ 2008] • Before making a decision not to synthesize data, review authors should consider what readers/decision makers might do (e.g. selection of favourable effects, count up #favourable results or stat. sig. results). Methods for synthesizing evidence of the effects of healthcare interventions October 2011 29

30. Conclusions • Diversity of interventions, settings, conditions, outcomes, and study designs complicates the synthesis of evidence. • A range of ‘synthesis’ approaches are available; some are clearly better than others. • Limitations in quantitative synthesis should be acknowledged, but may be preferable to “qualitative interpretation of results, or hidden quasi-quantitative analysis …” [Ioannidis BMJ 2008] • Before making a decision not to synthesize data, review authors should consider what readers/decision makers might do (e.g. selection of favourable effects, count up #favourable results or stat. sig. results). Methods for synthesizing evidence of the effects of healthcare interventions October 2011 30

31. Conclusions • Diversity of interventions, settings, conditions, outcomes, and study designs complicates the synthesis of evidence. • A range of ‘synthesis’ approaches are available; some are clearly better than others. • Limitations in quantitative synthesis should be acknowledged, but may be preferable to “qualitative interpretation of results, or hidden quasi-quantitative analysis …” [Ioannidis BMJ 2008] • Before making a decision not to synthesize data, review authors should consider what readers/decision makers might do (e.g. selection of favourable effects, count up #favourable results or stat. sig. results). Methods for synthesizing evidence of the effects of healthcare interventions October 2011 31

32. Conclusions • Diversity of interventions, settings, conditions, outcomes, and study designs complicates the synthesis of evidence. • A range of ‘synthesis’ approaches are available; some are clearly better than others. • Limitations in quantitative synthesis should be acknowledged, but may be preferable to “qualitative interpretation of results, or hidden quasi-quantitative analysis …” [Ioannidis BMJ 2008] • Before making a decision not to synthesize data, review authors should consider what readers/decision makers might do (e.g. selection of favourable effects, count up #favourable results or stat. sig. results). Methods for synthesizing evidence of the effects of healthcare interventions October 2011 32

33. Questions raised • Are arguments for not undertaking a meta-analysis based on too much clinical and methodological heterogeneity consistent with the ‘median-of-medians’ approach? • Are there other statistical approaches that may make better use of available data? E.g. meta-regression methods that adjust for correlated effects within studies (e.g. Hedges Res Syn Meth 2010). • The measures of effect used in complex reviews typically adjust for baseline imbalance (e.g. adj. RR, adj RD, adj OR). Do these estimators achieve the desired effect? • Will New Zealand win the 2011 Rugby World Cup? Methods for synthesizing evidence of the effects of healthcare interventions October 2011 33

34. References • Boonyasai et al. Effectiveness of teaching quality improvement to clinicians: a systematic review. JAMA 2007;298(9):1023-37. • Brennan et al. Continuous quality improvement: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2009, Issue 4. 10.1002/14651858.CD003319.pub2. • Crowther M, Avenell A, MacLennan G, Mowatt G. A further use for the Harvest plot: a novel method for the presentation of data synthesis. Res Syn Meth 2011 • Gardner B et al. Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Soc Sci Med 2010;70(10):1618-25. • Farmer et al Printed educational materials: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2008(3): CD004398. • Hedges et al. Robust variance estimation in meta-regression with dependent effect size estimates. Res Syn Meth 2010;1(1):39-65. • Hill et al. Identifying outcomes of importance to consumers, communication and participation. In Hill S (ed). The Knowledgeable patient: Communication and participation in health. Wiley-Blackwell 2011. • Ioannidis JP et al. Reasons or excuses for avoiding meta-analysis in forest plots. BMJ 2008;336(7658):1413-5. • Jamtvedt et al. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2006(2):CD000259. Methods for synthesizing evidence of the effects of healthcare interventions