Sample Size and Power

1. Sample Size and Power Steven R. Cummings, MD Director, S.F. Coordinating Center

2. The Secret of Long Life • Resveratrol • In the skin of red grapes • Makes mice • Run faster • Live longer

3. What I want to show • Consuming reservatrol prolongs healthy life

4. Sample Size Ingredients • Testable hypothesis • Type of study • Statistical test • Type of variables • Effect size (and its variance) • Power and alpha

5. My research question • I need to plan the study • My question is Does consuming reservatrol lead to a long and healthy life?

6. What’s wrong with the question? • I need to plan the study • My question is Does consuming reservatrol lead to a long and healthy life?

7. What’s wrong with the question? Does consuming reservatrol lead to a long and healthy life? • Vague • Must be measurable

8. Consuming resveratrol • Most rigorous design: randomized placebo-controlled trial • Comparing red wine to placebo would be difficult • Resveratrol supplements available and widely used

9. Measurable (specific) • Consuming resevertrol = taking resveratrol supplements vs. taking placebo • Prolong healthy life = reduces all-cause mortality Do people randomized to get a resveratrol supplement have a lower mortality rate than those who get a placebo?

10. In whom? Do people randomized to get a resveratrol supplement have a lower mortality rate than those who get a placebo? • Must study a sample from the larger ‘target population’ • What is the target population?

11. In whom? • Elderly men and women (≥70 years)

12. The research hypothesisThe ‘alternative’ hypothesis Men and women > age 70 years randomized to get a resveratrol supplement have a lower mortality rate than those who get a placebo. • Cannot be tested statistically • Statistical tests can only reject null hypothesis - that there is no effect

13. The Null Hypothesis Men and women > age 70 years randomized to receive a resveratrol supplement do not have lower mortality rate than those who receive placebo. • Can be rejected by statistical tests

14. Ingredients for Sample Size  Testable hypothesis • Type of study • Statistical test • Type of variables • Effect size (and its variance) • Power and alpha

15. Type of study • Descriptive • Only one variable / measurements • What proportion of centenarians take resveratrol supplements? • Confidence interval for proportions • What is the mean red wine intake of centenarians? • Confidence interval for the mean

16. Sample size for a descriptive study • “What proportion of centenarians take resveratrol supplements?” • How much precision do you want? • Sample size is based on the width of the confidence interval (Table 6D and 6E) • For example, assume that 20% of centenarians take resveratrol • I want to be confident that the truth is within ±10%

18. Type of study • Analytical: comparison • Cross-sectional • Mean red wine intake in centenarians vs. 60-80 years old • Randomized trial • Elders who get resveratrol have lower mortality than those who get placebo

19. Ingredients for Sample Size  Testable hypothesis  Type of study: analytical (RCT) • Statistical test • Type of variables • Effect size (and its variance) • Power and alpha

20. Type of statistical testsDepends on the types of variables

21. Types of variables? • Dichotomous • Treatment or placebo • Continuous • Walking speed

22. The types of variables? Men and women > age 70 years randomized to receive a resveratrol supplement do not have lower mortality rate than those who receive placebo • Dichotomous: reseveratrol or placebo • Dichotomous: mortality rate • 3-4% per year*; 3 year study: 10% • Statistical test: Chi-squaree (Tables 6B) * ~ mean annual male @ 78 yrs

23. Ingredients for Sample Size  Testable hypothesis  Type of study: analytical (RCT)  Statistical test  Type of variables • Effect size (and its variance) • Power and alpha

24. Effect sizethe hardest part Considerations • What is likely, based on other data? • Pilot study • Estimates from biomarkers • What difference is important to detect? • “We don’t want to miss a ____ difference” • What can we afford to find?

25. Resveratrol pronged survival of mice fed high calorie diet ~ 25% Baur, Nature 2006

26. The effect of resveratrol on mortality rate? • What is likely, based on other data? • Pilot study • Estimates from biomarkers • What difference is important to detect? • “We don’t want to miss a _1%_ difference” • What can we afford? • 1%: too expensive; 5%: cheap * ~ mean annual male @ 78 yrs

27. The effect of resveratrol on mortality rate? • Finding a smaller effect is important to health • Power to find a larger effect is important for your budget • Too small! vs. too large!

28. The Science of Effect SizesToo large! Too small!Just right. • Smaller effect is important to health • Larger effect is important for your budget

29. Effect size Men and women > age 70 years randomized to receive a resveratrol supplement do not have lower mortality rate than those who receive placebo • Placebo rate: 10% • Resveratrol rate: 8% • Chi-squared (Table 6B.2) * ~ mean annual male @ 78 yrs

30. Ingredients for Sample Size  Testable hypothesis  Type of study: analytical (RCT)  Statistical test  Type of variables  Effect size (and its variance) • Power and alpha

31. I will need to convince people • The result must be statistically significant Customarily, P<0.05 AKA • Probability of a type I error (oops, we lied) • (alpha) = 0.05

32. I will need to convince skeptics • Very small chance that we are fooling you (alpha) = 0.01 P<0.01 • Smaller means larger sample size

33. Two-sided vs. one-sided  • Use 2-sided  if the result could go the opposite way you want • 1-sided reduces sample size somewhat • You may believe that your effect could only go one way! • Resveratrol could not increase mortality! • Be humble. • The history of research is filled with results that contradicted expectations • A 1-sided test is almost never the best choice

34. If it’s true, I don’t want to miss it • The chance of missing the effect () customarily 20% AKA • Type II error •  (beta): 0.20 • Power = 1- 0.80

35. I really don’t want to miss it •  = .10 • Power (1- ) = 0.90 • Greater power means larger sample size

36. We have all of the ingredients  Testable hypothesis  Type of study: analytical (RCT)  Statistical test: Chi-squared  Effect size 10% vs 8%  Power: 0.90; alpha: 0.20

37. From Table 6B.2Comparing two proportions

38. From Table 6B.2 • Sample size: 4,401 • Per group • Total: 8,802 • Does not include drop-outs • 20% drop-out: 11,002 total sample size

39. Alternatives • Tweak : one-sided • Almost never appropriate • Tweak the power: 0.80 • Modest effect: 3,308 (6,616 total)

40. From Table 6B.2Comparing two proportions

41. Alternatives • Tweak and  • = 0.20 • 3,308/group; 6,616 total • Also increase the effect size • 10% vs. 6%

42. From Table 6B.2Comparing two proportions

43. Alternatives • Tweak and  • = 0.20 • 3,308/group; 6,616 total • Also increase the effect size • 10% vs. 6% • 930 / group; 1,680 total • Big difference, still not affordable • Not believable

44. Alternatives: a new hypothesis • Change the outcome measure • Continuous measurement • A ‘surrogate’ for mortality rate • Strongly associated with mortality rate • Likely to be influenced by resveratrol • Walking speed

45. Mice on resveratrol • Mice fed resveratrol • Live 25% longer • Are significantly faster • Have greater endurance

46. Increased gait speed (0.1 m/s) in 1 year and survival over 8 years Faster by ≥0.1 m/s Slower ~20% decreased mortality rate

47. What you need to know about a continuous variable • Outcome: change in walking speed • Mean value in the population • Effect size • Change in walking speed • Variability in the change

48. What you need to know about a continuous variable • Outcome: change in walking speed • Mean value in the population = 1.0 m/sec • Effect size • Change in walking speed • 1.0 to 1.1 m/sec • Variability in the change

49. Variability • No variability • Extremely reproducible • Relatively small sample size • Highly variable • Poor reproducibility • Relatively large sample size • Assessed by the Standard Deviation

50. Variability • Standard deviation for the measurement • Cross-sectional: 0.25 m / sec • However, we are interested in change • Standard deviation of change in speed?