Making sense of results - a workshop for healthcare librarians

1. Making sense of results- a workshop for healthcare librarians Dr Amanda Burls 2nd UK Clinical Librarian Conference

2. Objectives • To look at how results can be presented • To understand what a meta-analysis is • To be able to interpret a “blobbogram” • To be able to make sense of tests for “statistical significance” • To explore how uncertainty in results can be summarised and understand: • P-values • Confidence intervals • To have fun!

3. Making sense of results • How are results summarised?

5. How many of you have attended a critical appraisal skills workshop? • What sort of study design were you appraising? • What are the key things you remember?

6. Critical appraisal of any study design must consider • Validity • Can the study (results) be trusted? • Results • What are the results and how are they (or can they be) expressed? • Relevance • Do these results apply to the local context?

7. Warning! • Everything I say from now onwards assumes that the results being considered come from an unbiased study!

8. How are results summarised? • Most useful studies compare at least two alternatives. • How can the results of such comparisons be expressed?

9. Expressing results:What did the study show? • Patients with backache: • 100 randomised to receive a firm mattress • 100 randomised to receive a medium mattress • After 3 months: • 80 get better in the firm mattress group • 20 get better in the medium mattress group • How would you summarise this for a friend?

10. Summarise • 80 out of 100 (80%) better in firm mattress group • 20 out of 100 (20%) better in the medium mattress group • 4 times as likely to get better with a firm mattress • An extra 60% of people get better with a firm mattress

11. How were the results summarised? • There are two basic ways to summarise results of studies that compare two or more groups: • Difference (take them away) • Ratio (divide)

12. The blobbogram!

13. Blobbogram Line of no difference between treatments less more

14. Blobbogram - Difference (taking away) Line of no difference between treatments less more 0

15. Blobbogram - ratio (dividing) Line of no difference between treatments less more 1

16. A randomised placebo-controlled trial

17. Well conducted RCT – no bias • Five people with backache received Potters • Five people received placebo • 4 out of 5 with Potters got better • 2 out of 5 with placebo got better

19. No backache at 3 months(Results of our Potters tablet versus placebo trial) Potters Placebo Favours placebo Favours Potters

20. No backache at 3 months(Results of our Potters tablet versus placebo trial) Potters Placebo Favours placebo Favours Potters

21. No backache at 3 months(Results of our Potters tablet versus placebo trial) Potters Placebo Favours placebo Favours Potters

23. No backache at 3 monthsDo you think this study proves Potters works? Potters Placebo Favours placebo Favours Potters

24. It could be due to chance! • What if there had 1000 people in each arm and 800 got better with Potters and only 200 got better on placebo? • Would you believe Potters works now? • So how many people would you want in each arm to believe the trial?

25. P-value in a nutshell

26. The Null Hypothesis Uumm.....

27. 1 0 So what does p=0.5 mean? So what does p=0.1 mean? So what does p=0.05 mean? Absolutely certain Impossible

28. p = 0.5 • quite likely - evens chance - 50:50 - 1 in 2 p = 0.001 • very unlikely - 1 in 1000 p = 0.01 • unlikely - 1 in 100 p = 0.05 • fairly unlikely - 1 in 20 - 5 times in 100

31. Moral: Any observed difference between two groups, no matter how small, can be made to be “statistically significant” - at any level of significance - by taking a sufficiently large sample.

32. Question: How can we express uncertainty due to chance? • Answer: the p-value • But is there a better answer?

33. Introduction to confidence intervals • CIs are a way of showing the uncertainty surrounding our point estimate.

34. No backache at 3 months(Results of our Potters tablet versus placebo trial) Potters Placebo Favours placebo Favours Potters

35. No backache at 3 months(Results of our Potters tablet versus placebo trial) Potters Placebo Favours placebo Favours Potters

36. No backache at 3 months(Results of our Potters tablet versus placebo trial) Potters Placebo Favours placebo Favours Potters

37. No backache at 3 months(Results of our Potters tablet versus placebo trial) Potters Placebo Favours placebo Favours Potters

38. Clifton 1993 Clifton 1992 Hirayama 1994 Marion 1997 Total (95%CI) .1 .2 1 5 10 Hypothermia vs. control In severe head injury Mortality or incapacity (n=158) RR 0.63 (0.46, 0.87) RR

39. Hypothermia vs. control In severe head injury Mortality or incapacity (n=158) Clifton 1993 Clifton 1992 Hirayama 1994 Marion 1997 RR 0.63 (0.46, 0.87) Total (95%CI) .1 .2 1 5 10 RR

40. Hypothermia vs. control In severe head injury Mortality or incapacity (n=158) Clifton 1993 Clifton 1992 Hirayama 1994 Marion 1997 RR 0.63 (0.46, 0.87) Total (95%CI) .1 .2 1 5 10 Favours intervention RR Favours control

41. Clifton 1993 Clifton 1992 Hirayama 1994 Marion 1997 Total (95%CI) .1 .2 1 5 10 Hypothermia vs. control In severe head injury Mortality or incapacity (n=158) RR 0.63 (0.46, 0.87) Favours intervention RR Favours control