Kevin McConway ABSW Summer School 8 July 2019

1. Reading a research paperWhy the full story probably isn’t in the press release, and how to find out what really is Kevin McConway ABSW Summer School 8 July 2019

2. It’s a long way from research to the public Research Paper or conference talk Press release Press story Headline Reader

3. From media release to press story, or how to spin and be spun

4. The science • People whose heart stops, outside a hospital, are in great danger… • …particularly if their heart doesn’t restart after CPR and/or defibrillation. • Advice was to inject adrenaline, but concerns had been expressed. • Clinical trial in the UK, 2014-2018. • Patients chosen at random to get either injections of adrenaline or salt solution (placebo).

5. Using adrenaline in cardiac arrests results in 40% more people leaving hospital alive But the survivors’ risk of severe brain damage increases by 13 percentage points.

7. […] https://www.theguardian.com/science/2018/jul/19/routine-treatment-for-cardiac-arrest-doubles-risk-of-brain-damage-study

8. So why the made-up top line? • Both top lines are telling the truth, based on the data from the research. • There are two ways to describe changes numerically – absolute and relative. • Nothing mathematically wrong with either of them, but they can give very different impressions to the reader.

9. https://www.nejm.org/doi/full/10.1056/NEJMoa1806842

10. Data from the trial report

11. Using adrenaline in cardiac arrests results in 40% more people leaving hospital alive But the survivors’ risk of severe brain damage increases by 13 percentage points.

12. Percentages • Change in something as a percentage is • My rent was £500, goes up to £550. Increase is £50. As percentage of the original rent that’s • But awkward if my rent was already a percentage (of my pay, say). • Goes up from 20% of my pay to 22% of my pay. • Change is 22% - 20% = 2 percentage points. • As a percentage of the original level this is

13. Using adrenaline in cardiac arrests results in 40% more people leaving hospital alive But the survivors’ risk of severe brain damage increases by 13 percentage points. Survival comparison is relative. The survival rate was 3.2% with adrenaline, 2.3% with placebo. The adrenaline rate is higher than the placebo rate. Brain damage comparison is absolute. It compares the rates by subtraction. Difference is 31.0% - 17.8% = 13.2 percentage points.

14. Using adrenaline in cardiac arrests results in less than one percent more people leaving hospital alive But nearly doubles the survivors’ risk of severe brain damage. Survival comparison is absolute. Difference is 3.2% - 2.3% = 0.9 percentage points. (They should have called them ‘percentage points’.) Brain damage comparison is relative. The rate was 31.0% with adrenaline, 17.8% with placebo. The adrenaline rate is higher than the placebo rate. Maybe that’s close enough to ‘nearly double’…

15. Messages about percentages • Percentages can be useful for comparing things, because they give an idea of how big the change is compared to the previous level. • Suppose I was paid £20 for giving a lecture, and it goes up to £30. • The increase is . Pretty good! • But if I was originally paid £200, a £10 increase would be much smaller in relative terms, as a percentage. Only 5%. • But a problem of terminology if you’re looking at relative changes in percentages. • The gap between two percentages is measured in percentage points. The change from 20% to 35% is 15 percentage points.

16. What do you want from a study of a potential risk factor or intervention? • Want to know: • How likely is the outcome in exposed people? (Absolute risk) • How likely is the outcome in unexposed people? (Baseline for comparison) • How do these compare?

17. Based on study of taking vitamin D supplements to reduce risk of acute respiratory tract infections. (Martineau et al., BMJ, February 2017, https://doi.org/10.1136/bmj.i6583 )

18. But the absolute risk isn’t always in the research paper • There can be good reasons for that. • They might give the relative risk a.k.a. risk ratio:RR = So for the Vitamin D study, RR = 39/42 = 0.93. • Sometimes, again for good reasons, the paper reports odds ratios (OR)or hazard ratios (HR)instead. These are also relative. • Ask the scientists to give you useful numbers.

19. Placebo Adrenaline Died before hospital discharge Survived with severe brain damage Survived, no severe brain damage

20. Why absolute risk matters • The impact of a new treatment, or a harmful exposure, depends on the absolute risks involved, not just on the RR. • If the baseline risk (in unexposed people) is very small, doubling or halving it may not be very important. • Twice not very much is still not very much. • But this can be a big issue in relating the statistical results to individuals in a story.

21. Quiz! A dietary supplement reduced the risk of having at least one cold in a year by 20%, according to an RCT. In a given year, people who don’t take the supplement have a 50% chance of getting a cold. If a large group of people take the supplement, what’s their chance of having at least one cold next year? • 30% • 40% • 44% • I’m a journalist, not a numbers person. You’re the numbers person – you tell me!

22. What if the paper reports only relative measures? • Ask the researchers • Absolute risks might be in the paper somewhere, just well hidden. • But are you the best person to be digging in the paper for them? And they might not be there anyway. • Press the researchers for an answer! Your readers want the numbers, or at least some clear indication of importance.

23. Messages about absolute and relative comparisons • You can give, or get, quite a different impression depending on which you use. • So you can slant what you write, deliberately (but please don’t!) or accidentally… • …or you can be taken in by a misleading comparison in a press release. • Giving only relative comparisons, e.g. of risk, can be very misleading. Double a very small risk is still very small. • Use absolute comparisons if at all possible. • You may have to ask the researchers, or a statistician, or the Science Media Centre. • If the study is about a disease or condition, at least write something about how common it is.

24. Press releases and when to go beyond them

25. When to go beyond the press release: some suggestions • Exaggerations in press releases happen • Check with: • Science Media Centre (or equivalent in other countries) • author • paper • Reasons for smelling a rat include: • Health advice • X “causes / is linked with” Y • Animal study that extrapolates to people • Borderline or lack of statistical significance • Things not controlled for (confounders) • Niche group of people • Research not peer reviewed • Often omitted from press releases: study limitations, funding, conflicts of interest

26. Reading the paper • Takes time, has jargon • But, typical structure (don’t need to read whole thing) • Abstract – summary (group studied, study design, findings, stats results) • Title – useful but sometimes like news headline • Discussion – usefully has ‘strengths’ and ‘limitations’ • Introduction, or (simpler!) ‘What is already known on this topic’/‘What this study adds’ • Usually less useful (for journalists): • Methods – for confounders • Results – usually enough in Abstract, but sometimes release concentrates on others • Usual order: Title, Abstract, Introduction, Methods, Results, Discussion • … but sometimes Methods comes last.

28. Statistical significance and p values

29. Statistical significance • It doesn’t mean what you (possibly) thought it meant. • Nothing directly to do with real-world importance • A (flawed?) way of judging whether some finding can be explained away by chance variability. • …usually based on the p value.

30. p values • It almost certainly doesn’t mean what you thought it meant. • Number between 0 and 1 (or 0% and 100%). • The smaller, the less likely it is that a finding is just chance (more or less!) • Common convention: if p < 0.05 (that is, 5%) a result is statistically significant. • But it’s just a convention (and doesn’t work well with big data)

31. p values • E.g. “people on the new drug had, on average, lower blood pressure than people on the old drug (p = 0.03)”. • DOESN’T mean: • There’s a 0.03 (or 3%) probability that the result was due to chance. • DEFINITELY DOESN’T mean • There’s a 0.97 (or 97%) probability that the result was NOT due to chance. • DOES mean: • Assume that the new drug has the same average effect on blood pressure as the old drug. Then there’s a 3% probability that a difference like this (at least as extreme as this) will be observed. • So what IS the probability that the result was due to chance? • You can’t say from this information (and often you can’t say at all)

32. But statistical significance still matters • p value is too big – plausible there’s no effect, just chance variation. • But still don’t know the probability that it is just down to chance. • If the treatment (or whatever) has absolutely no effect, there’s a 5% chance that each statistical test will report a significant effect. • Doing multiple statistical tests is like having lots of shots on goal – increased chance one will score (be significant). • Hence risk of “data dredging”, existence of (numerous) statistical methods to correct for multiple testing, etc. (This is a form of p-hacking.) • One alternative approach uses confidence intervals – range of (statistically) plausible values for something the study measured.

34. Systematic review and meta-analysis of 13 studies worldwide (though most results depended on 3). • Paper abstract says “Although not reaching statistical significance, all the summary effect estimates for the risk of stillbirth were systematically elevated[…]”. • The press release did mention a few caveats, but not the lack of significance. Is it enough to say ‘suggestive’? • The press release gave the relative risk for just one pollutant, PM2.5, but no absolute risks, and it’s unclear how big that relative risk is anyway. • (For later: Everything is based on observational studies so we can’t conclude that the pollution causes the stillbirths anyway.)

35. From the paper’s Abstract: https://oem.bmj.com/content/73/9/573 From the press release: https://www.bmj.com/company/wp-content/uploads/2016/05/air-pollution-stillbirth.pdf

36. Messages about statistical significance, p values and confidence intervals • Statistical significance is a (flawed?) way of judging whether some finding can be explained away by chance variability, usually based on the p value. • Other things being equal, which they hardly ever are, the smaller the p value, the less likely it is that a finding is due to chance alone. • Common (but flawed) convention: if p<0.05, the result is statistically significant. • But you still don’t know how likely it is that the result is due to chance. • Calculating many p values increases the chance of finding false positive results. • Often better to use confidence intervals – ranges of (statistically) plausible values for a quantity of interest.

37. Types of study Important because this determines strength of findings and confidence in results

38. Study types Roughly in decreasing order of strength of evidence: • Randomized controlled trial (RCT) (or better still, a good systematic review of RCTs) • Prospective cohort study • Retrospective cohort study • Case-control study • Cross-sectional study on individuals • Ecological study (doesn’t mean what you might think it means) • Animal studies (don’t fit on the same strength of evidence scale, really)

39. Study types Roughly in decreasing order of strength of evidence: • Randomized controlled trial (RCT) (or better still, a good systematic review of RCTs) • Prospective cohort study • Retrospective cohort study • Case-control study • Cross-sectional study on individuals • Ecological study (doesn’t mean what you might think it means) • Animal studies (don’t fit on the same strength of evidence scale, really) “observational”

40. (December 2017) (December 2017) (December 2017) (November 2017) (January 2018) (January 2018)

41. Randomized Controlled Trials

42. RCTs • Why controls? • Why randomize? • Avoid bias • Blinding

43. Why are systematic reviews better than a single RCT? • More data • (Usually) more settings • Can look for heterogeneity • The statistical approach to putting the data together is meta-analysis.

44. Statistical analysis in RCTs • Possibility of “cheating” – carry out lots of tests, some will be significant even if there’s no real effect. • RCTs registered in advance, stating primary and secondary outcomes. • Things not registered: Ad hoc/post hoc analysis. Be careful how you write about these! (And the press release might not tell you which they are.)

45. Observational studies in epidemiology (etc.)

46. The general idea in epidemiology • When you can’t do an RCT (e.g. epidemiology of things that are possibly bad for people). • There’s exposure to some (potential) risk • There’s an outcome (disease, early death, etc.) • Want to know whether exposure changes the chance of the outcome. • But all you can do is observe what people do or did. No experimental manipulation (as in an RCT). So these are observational studies.

47. Cohort studies • Choose some group of people (the cohort) that includes some exposed to risk and some not exposed. • Prospective cohort study: follow them up and record whether they have the outcome or not. • Expensive, can take a very long time, loss to follow-up, problems with rare outcomes • Retrospective cohort study: choose cohort after the outcome has occurred (or not) and look back at past exposure to risk. • Possible bias in choosing cohort, recall bias or loss of records, and still doesn’t work well with rare outcomes

48. Case-control studies • Find some people with the outcome in question – the ‘cases’. • Find some people who are sufficiently ‘like’ the cases, but they don’t have the outcome in question – the ‘controls’. • So participants chosen on the basis of outcome. • Look back (retrospective) at exposure. • Issues of recall bias (remembering or finding records about the past)

49. Meta-analysis • A systematic review should follow a standard procedure. • Sounds like (and often is!) a good idea: put together information from many studies of the same thing. • The statistical approach to putting the data together is meta-analysis. • But not all meta-analyses are good science; problems with: • Study quality • Levels of evidence • Publication bias

50. Randomized controlled trial (December 2017) Prospective cohort study (December 2017) Retrospective cohort study (December 2017) Case-control study (November 2017) Cross-sectional study (January 2018) Animal experiment (January 2018)