Statistics in Medicine

1. Statistics in Medicine Unit 6: Overview/Teasers

2. Overview • Type I and Type II errors and statistical power; pitfalls of p-values • Overview of statistical tests

3. Teaser 1, Unit 6 • A prospective cohort study of 34,079 women found that women who exercised >21 MET hours per week (60 minutes moderate -intensity exercise daily) gained significantly less weight than women who exercised <7.5 MET hours per week (p<.001) • Widely covered in the media. Headlines: • “To Stay Trim, Women Need an Hour of Exercise Daily.” • “New Exercise Goal: 60 Minutes a Day” • Physical Activity and Weight Gain Prevention. JAMA 2010;303:1173-1179.

4. Teaser 1, Unit 6 • A prospective cohort study of 34,079 women found that women who exercised >21 MET hours per week (60 minutes moderate -intensity exercise daily) gained significantly less weight than women who exercised <7.5 MET hours (p<.001) • Widely covered in the media. Headlines: • “To Stay Trim, Women Need an Hour of Exercise Daily.” • “New Exercise Goal: 60 Minutes a Day” • Physical Activity and Weight Gain Prevention. JAMA 2010;303:1173-1179.

5. How big was the effect? • How much less weight do you think the high exercise group gained compared with the low exercise group over 3 years? • Write down a guess!

6. Teaser 2, Unit 6 Abstract OBJECTIVES: The aim of the pilot study was to determine the efficacy of dietary n-3 PUFA docosahexaenoic acid (DHA) in patients with atopic eczema. METHODS: Fifty-three patients suffering from atopic eczema aged 18-40 years were recruited into this randomized, double-blind, controlled trial and received either DHA 5.4 g daily (n = 21) or an isoenergetic control of saturated fatty acids (n = 23) for 8 weeks. At weeks 0, 4, 8 and 20 the clinical outcome was assessed by the SCORAD (severity scoring of atopic dermatitis) index. RESULTS: DHA, but not the control treatment, resulted in a significant clinical improvement of atopic eczema in terms of a decreased SCORAD [DHA: baseline 37.0 (17.9-48.0), week 8 28.5 (17.6-51.0); control: baseline 35.4 (17.2-63.0), week 8 33.4 (10.7-56.2)]. What should we conclude from these results? Did DHA beat placebo?

7. Statistics in Medicine Module 1: Type I and type II errors

8. Hypothesis Testing The Steps: 1.Define your hypotheses (null, alternative) 2.Specify your null distribution 3.Do an experiment 4.Calculate the p-value of what you observed 5.Reject or fail to reject the null hypothesis Follows the logic: If A then B; not B; therefore, not A.

9. Summary: The Underlying Logic of hypothesis tests… Follows this logic: Assume A. If A, then B. Not B. Therefore, Not A. But throw in a bit of uncertainty…If A, then probably B…

10. Error and Power Note the sneaky conditionals… • Type-I Error (also known as “α”): • Rejecting the null when the effect isn’t real. • Type-II Error (also known as “β “): • Failing to reject the null when the effect is real. • POWER (the flip side of type-II error: 1- β): • The probability of seeing a true effect if one exists.

11. Your Decision The TRUTH God Exists God Doesn’t Exist Reject God BIG MISTAKE Correct Accept God Correct— Big Pay Off MINOR MISTAKE Think of…Pascal’s Wager

12. Your Statistical Decision True state of null hypothesis H0 True (example: the vaccine doesn’t work) H0 False (example: the vaccine works) Reject H0 (ex: you conclude that the vaccine works) Type I error (α) Correct Do not reject H0 (ex: you conclude that there is insufficient evidence that the vaccine works) Correct Type II Error (β) Type I and Type II Error in a box

13. Error and Power • Type I error rate (or significance level): the probability of finding an effect that isn’t real (false positive). • If we require p-value<.05 for statistical significance, this means that we are permitting a false positive rate of 5% (1 in 20). • Type II error rate: the probability of missing an effect (false negative). • Statistical power: the probability of finding an effect if it is there (the probability of not making a type II error). • When we design studies, we typically aim for a power of 80% (allowing a false negative rate, or type II error rate, of 20%).

14. Statistical power Statistical power is the probability of finding an effect if one exists.

15. Factors Affecting Power 1. Size of the effect 2. Standard deviation of the characteristic 3. Bigger sample size 4. Significance level desired

16. Sample size calculations • Based on these elements, you can write formal mathematical equations that relates power, sample size, effect size, standard deviation, and significance level…

17. Represents the desired power (typically .84 for 80% power). Sample size in each group (assumes equal sized groups) Represents the desired level of statistical significance (typically 1.96). Standard deviation of the outcome variable Effect Size (the difference in means) Example: formula for difference in means

18. Statistics in Medicine Optional Module 1X: Sample size formulas, derivations

19. Distribution, difference in means • T-distribution (Z for n>100) • Mean=true difference in means • Standard error:

20. Distribution, difference in proportions • Z-distribution • Mean=true difference in proportions • Standard error:

21. Power and sample size Power = What’s the probability that we will correctly reject the null hypothesis when the alternative hypothesis is in fact true? I.e., what’s the probability of detecting a real effect?

22. Can we quantify how much power we have for given sample sizes?

23. Example 1: difference in proportions Rejection region. Any value >= 6.5 (0+3.3*1.96) Power= chance of being in the rejection region if the alternative is true=area to the right of this line (in yellow) Null Distribution: difference=0. For 5% significance level, one-tail area=2.5% (Z/2 = 1.96) Clinically relevant alternative: difference=10%.

24. Example 1: difference in proportions Rejection region. Any value >= 6.5 (0+3.3*1.96) Power here: Power= chance of being in the rejection region if the alternative is true=area to the right of this line (in yellow)

25. study 1: difference in proportions, smaller sample size Critical value= 0+10*1.96=20 Z/2=1.96 2.5% area n=100 Power closer to 15% now.

26. Example 2: difference in means Critical value= 0+0.52*1.96 = 1 Clinically relevant alternative: difference=4 points Power is nearly 100%!

27. Example 2: difference in means, greater outcome variability Critical value= 0+2.58*1.96 = 5 Power is about 40%

28. Example 2: difference in means, smaller effect size Critical value= 0+0.52*1.96 = 1 Power is about 50% Clinically relevant alternative: difference=1 point

29. Factors Affecting Power 1. Size of the effect 2. Standard deviation of the characteristic 3. Bigger sample size 4. Significance level desired

30. 1. Bigger difference from the null mean Null Clinically relevant alternative average weight from samples of 100

31. 2. Bigger standard deviation average weight from samples of 100

32. 3. Bigger Sample Size average weight from samples of 100

33. 4. Higher significance level Rejection region. average weight from samples of 100

34. Sample size calculations • Based on these elements, you can write a formal mathematical equation that relates power, sample size, effect size, standard deviation, and significance level… • **WE WILL DERIVE THE MEANS FORMULA SHORTLY**

35. Represents the desired power (typically .84 for 80% power). Sample size in each group (assumes equal sized groups) Represents the desired level of statistical significance (typically 1.96). Standard deviation of the outcome variable Effect Size (the difference in means) Example: formula for difference in means

36. Represents the desired power (typically .84 for 80% power). Sample size in each group (assumes equal sized groups) Represents the desired level of statistical significance (typically 1.96). A measure of Variability of a proportion Effect Size (difference in proportions) Example: formula for difference in proportions

37. Derivation of sample size formula….

38. Example 2: difference in means, effect size=1.0 Critical value= 0+.52*1.96=1 Power close to 50%

39. Critical value= 0+standard error (difference)*Z/2 Power= area to right of Z= SAMPLE SIZE AND POWER FORMULAS

40. Power= area to right of Z= Power is the area to the right of Z. OR power is the area to the left of - Z. Since normal charts give us the area to the left by convention, we need to use - Z to get the correct value. Most textbooks just call this “Z”; I’ll use the term Zpower to avoid confusion.

41. All-purpose power formula…

42. Derivation of a sample size formula… Sample size is embedded in the standard error….

43. Algebra…

45. Sample size formula for difference in means

46. Example • You want to calculate the sample size needs for a study comparing male doctors and female doctors. You want to detect a difference of 3.0 IQ points between two groups. If you expect the standard deviation to be about 10 on an IQ test for both groups, how many people would you need to sample in each group to achieve power of 80% (corresponds to Z=.84) 174/group; 348 altogether

47. General sample size needs when outcome is binary:

48. Statistics in Medicine Module 2: P-value pitfalls: statistical vs. clinical significance

49. Statistical vs. clinical significance • Trivial effects may achieve statistical significance if the sample size is large enough.

50. Example • A prospective cohort study of 34,079 women found that women who exercised >21 MET hours per week (60 minutes moderate -intensity exercise daily) gained significantly less weight than women who exercised <7.5 MET hours (p<.001) • Widely covered in the media. Headlines: • “To Stay Trim, Women Need an Hour of Exercise Daily.” • “New Exercise Goal: 60 Minutes a Day” • Physical Activity and Weight Gain Prevention. JAMA 2010;303:1173-1179.