Evidence-based clinical practice - Risk for errors Chris tian Gluud Copenhagen Trial Unit

1. Evidence-based clinical practice - Risk for errors Christian Gluud Copenhagen Trial Unit Centre for Clinical Intervention Research Rigshospitalet Copenhagen University Hospital

2. Evidence-basedclinical practice • The patient’s values • (concerns, expectations, preferences) • The best clinical research evidence

3. Levels of evidence

4. Important aspects of clinical research- all levels of the hierachy Systematic errors (domains) Systematic errors (design) Random errors (play of chance)

5. Levels of evidence Risk of systematic errors (domain) Risk of systematic errors (design) Risk of random errors

6. The hierarchy of evidence Ia Systematic review of randomised clinical trials with low risk of systematic errors and low risk of random errors Ib Single randomised clinical trial with low risk of systematic errors and low risk of random errors !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! II Cohort study III Case-control study IV Case series

7. Dangers in non-randomised studies • Confounding by indication and unmeasured confounding • Biological mechanisms • - Limited time of diseases • - Cyclical progression of diseases • - When do we see patients? • Psychological mechanism • - We see what we want to see (BIAS)! • - We believe what we want to believe (astrology)!

8. Deeks et al (2003) and the International Stroke Trial • 19,435 patients with ischaemic stroke • 467 clinical sites • Randomised to aspirin versus placebo • Dead or dependent at 6 months • OR 0.95; 95% CI 0.89 to 1.01

9. Deeks et al (2003) and the International Stroke Trial Deeks et al. resample 100 pts/group making - Small randomised trials (n=14,000) from randomised groups within centres - Small controlled cohort studies (n=14,000) from two centres

10. Deeks et al. 2003 Randomised clinical trials

11. Deeks et al. 2003 Controlled cohort studies

12. Deeks et al. 2003 Logistic regression Controlled cohort studies

13. Deeks et al. 2003 Propensity scores Controlled cohort studies

14. Povl Heiberg 1897 • Heiberg knew the solutions - large numbers ! - randomisation ! - blinding ! - independent research !

17. Levels of evidence

18. Important aspects of randomised clinical trials and meta-analyses Systematic errors (domains) Systematic errors (design) Random errors (play of chance)

19. Generation of the allocation sequence Allocation concealment Blinding Incomplete outcome data (intention-to-treat) Outcome reporting bias Industry bias Other components associated with bias Domains associated with bias risk

20. Meta-analysis of several trials Low risk of bias High risk of bias Overall

21. Ratio of odds ratios (ROR) Odds ratio of trials with unclear or inadequate component (high risk of bias) divided by odds ratio of trials with adequate component (low risk of bias)

22. Control of selection bias • Generation of the allocation sequence • Low risk of bias • Computer system, • table of random numbers, or similar • High risk of bias • Not described or quasi-randomised • (excluded)

23. BRANDOSequence generation 112 meta-analyses with 944 randomised clinical trials 696 (73.7%) with high risk of bias ROR 0.89 (95% CI 0.82 to 0.96)

24. Control of selection bias • Allocation concealment • Low risk of bias • Central independent unit, • sealed envelopes (only if perfect placebo) • High risk of bias • Not described or open table of random • numbers

25. BRANDOAllocation concealment 146 meta-analyses with 1292 randomised clinical trials 916 (70.9%) with high risk of bias ROR 0.93 (95% CI 0.87 to 0.99)

26. Control of detection bias(reporting bias and observer bias) • Blinding • Low risk of bias • Identical placebo or comparator • High risk of bias • Not described or not blinded

27. BRANDOBlinding 104 meta-analyses with 1057 randomised clinical trials 467 (44.2%) with high risk of bias ROR 0.87 (95% CI 0.79 to 0.96)

28. The hierarchy of evidence Ia Systematic review of randomised clinical trials with low risk of systematic errors (bias) and of random errors (play of chance) Ib Single randomised clinical trial with low risk of systematic errors (bias) and of random errors (play of chance) II Cohort study III Case-control study IV Case series

29. High risk of systematic errors Low risk of systematic errors Low risk of random errors High risk of random errors

30. Domains associated with bias risk Generation of the allocation sequence Allocation concealment Blinding Incomplete outcome data (intention-to-treat) Outcome reporting bias Industry bias Other components associated with bias

31. Industry sponsorship and research outcome • Lundh et al, The Cochrane Library 2012 • Cross-sectional studies, cohort studies, systematic reviews and meta-analyses that quantitatively compared primary research studies of drugs or medical devices sponsored by industry with studies with other sources of sponsorship

32. Favourable beneficial results

33. Favourable harms results

34. Favourable conclusions

35. Important aspects of randomised clinical trials and meta-analyses Systematic errors (domains) Systematic errors (design) Random errors (play of chance)

36. Design errors affecting external validity of randomised trials include ‘wrong’: • Centres • Participants • Experimental intervention • Control intervention • Goal - explanatory or pragmatic • Trial structure - parallel group, crossover, etc. • Objective - superiority, equivalence, non-inferiority • Outcome • Unit of analysis

37. Important aspects of randomised clinical trials and meta-analyses Systematic errors (domains) Systematic errors (design) Random errors (play of chance)

38. Mean number of patients per intervention arm (SEM) in 383 randomised trials published in Gastroenterology from 1964-2000 (Kjærgard et al. 2002)

39. False positive results (type I error) False negative results (type II error) Random errors in small trials

41. TA(C)E for hepatocellular carcinoma – systematic review

42. TA(C)E for hepatocellular carcinoma – trial sequential analysis

43. The pernicious yin-yang interplay between random errors and systematic error PUBLICATION BIAS

44. Randomised clinical trials – most often false due to the people behind! Control intervention • Experimental intervention • systematic errors (domains) • systematic errors (design) • PUBLICATION bias “…why most research findings are false!” JP Ioannidis

45. THANK YOU !

46. The Cochrane Collaboration An international network of professionals, preparing , maintaining, and disseminating systematic reviews of the effects of health care

47. Archie Cochrane (1979) ”It is surely a great criticism of our profession that we have not organised a critical summary, by speciality and subspeciality, adapted periodically, of all relevant randomised trials”

48. The Cochrane Collaboration www.cochrane.org 670 000 randomised trials Now 5000 systematic reviews 500 new reviews per year 500 updated reviews per year JIF 6.2

49. Answer to the question: When is it acceptable to make therapeutic decisions based on patient series, patient-control studies, and cohort studies?

50. Answer to the question: When is it acceptable to make therapeutic decisions based on patient series, patient-control studies, and cohort studies? Only if you have exophtalmus producing intervention effects!