1 / 32

Section V

Section V. Sample Size and Power Multiple hypothesis testing. Sample size (n) based on estimation precision-CI width. Can plan sample size so that standard errors (SEs) and the corresponding confidence intervals are sufficiently small. (How small? How small do you need it to be?).

ifama
Download Presentation

Section V

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Section V Sample Size and Power Multiple hypothesis testing

  2. Sample size (n) based on estimation precision-CI width Can plan sample size so that standard errors (SEs) and the corresponding confidence intervals are sufficiently small. (How small? How small do you need it to be?)

  3. Sample size for precision/CIs π = proportion with TB in the population (prevalence) p = proportion with TB in a sample of size n, SE(p) = √[π(1- π)/n] 95% confidence interval for π: p ± 1.96(SE) Precision: want to estimate true prevalence (π) within ± 6% Solve for n: 1.96(SE) = 1.96 √[π(1- π)/n] = 0.06 n = 1.962π(1- π)/(0.06)2 = 3.84 π(1- π)/0.0036 Can estimate π using the observed p or use maximum at π=0.5 If π=0.15, n = 3.84 (0.15)(0.85)/0.0036 = 136 At π=0.50, n = 3.84 (0.50)(0.50)/0.0036 = 267 (worst case) Rule of thumb: For 95% CI for π, conservative n for precision w is n = 1/w2

  4. Sample size based on hypothesis testing

  5. Hypothesis test decision table

  6. Alpha versus Power The top distribution shows the sampling distribution of a test statistic under the assumption that delta is zero (the null hypothesis is true). α The bottom distribution shows the true population distribution (unknown at the time of testing), with a true population delta=2.5. 1 - β 1 - β

  7. Power calculationZpower = Zobs – Zα Zobs=1.50/1.18 = 1.27 (<1.96, so not statistically significant, p=0.20) Zpower = Zobs - Z α= 1.27 – 1.96 = -0.69 From the Gaussian table, Zpower=-0.69 yields power about 25%.

  8. Interpretation of power If test is “statistically” significant (p < α), we have a “positive” or “significant” outcome (& accept the false positive probability of α). If test is not statistically significant (p > α) either there is no relationship (“negative” outcome) or sample size is inadequate (inconclusive outcome). If power is low for a given δ, results are inconclusive, not negative. If power is high, results are affirmatively negative.

  9. Power when testing difference between 2 means Two independent groups, each with sample size n = 2 (Zpower + Zα)2 (σ/δ)2 Zα= 1.96 and Zpower = 0.842 (for power of 80%), so n = 2(0.842 + 1.96 )2 (σ/δ)2 = 15.7 (σ/δ)2 or n/group approximately ≈ (range/δ)2 (since 15.7 ≈16, 16(σ/δ)2 = (4σ/δ)2 and the range ≈4σ )

  10. Power for increasing delta δ =2.5 δ = 0 δ = 3.5 Areas under the curves and right of the vertical line are α for the black curve and power for the other curves. The power is larger for the red curve than for the blue.

  11. Power Summary Power increases as • True difference (δ) increases • Sample size (n) increases • α increases (less strict significance criterion) • Patient heterogeneity (σ) decreases Generally, we set α = 0.05 & power = 1 – β = 0.80 To determine n, we need to estimate δ and σ. Often we use values of δ/σ for the calculation For time to event outcome (survival), n also depends on f/u time.

  12. Sample Size Checklist Effect size (δ) = smallest clinically important difference (n increases as δ decreases) Variability = patient heterogeneity = group SD (n increases as variability increases) Power = prob of detecting the effect, set at 80% or higher (n increases with power) α level = prob of rejecting when δ=0, usually set at 0.05, two-sided (n decreases with larger α) ** for time to event (survival) outcomes *** Time of comparison = how long it takes to achieve the effect (n decreases with time) Follow up time = time each patient is followed (n reduced if patients followed longer). In survival “n” is the number who have the outcome. Also consider percentage who will agree to participate patient accrual rate, dropout / loss rate

  13. Sample size for selected δ/σ - means(2 means difference= δ, SD=σ, two-sided α=0.05)

  14. Sample size for two proportions

  15. Pseudo replication Most variation is between persons, not within person. Two blood samples on n=10 is not a sample size of 20. Observed value = true population mean + between person variation (σp) + within person variation (σe) Example: To estimate the mean 1. Compute a mean for each person using her “m” observations per person. 2. Compute the group mean from the “n” person means. SEM = √[σp2/n + σe2/nm], usually σe < σp

  16. Statistical vs Medical “significance” (“A difference, in order to be a difference, must make a difference”–Gertrude Stein?). Average drop in weight (kg) after 3 months

  17. Multiple Hypothesis testing Multiple efficacy endpoints /outcomes Multiple safety endpoints/outcomes Multiple treatment arms and/or doses Multiple interim analyses Multiple patient subgroups Multiple analyses

  18. Who killed Tweety Bird?

  19. Did Sylvester do it?

  20. Motivation (class discussion) Tweety Bird is murdered by a cat who left a DNA sample. The particular DNA profile found in the sample is known to occur in one of every one million cats. There is also about a 0.1% false positive rate for this test. Is the level of evidence (guilt) equal in these two scenarios? • Sylvester only is tested and is a match. 2. A DNA database on 1 million cats, including Sylvester, is searched and Sylvester is a match.

  21. Motivation(class discussion) The “disease score” ranges from 2 (good) to 12 (worst). Scenario A: Due to prior suspicion (prior information), only patients 19 and 47 are measured and both have scores of 12. We report that they are “significantly” ill. Scenario B: The score is measured on 72 patients. Only patients 19 and 47 have scores of 12. We report that they are “significantly” ill.

  22. Is the amount of “evidence” or “belief” that patients 19 and 47 “really” are very ill (have “true” score of 12) the same in both scenarios? The data for patients 19 and 47 are the same in both scenarios. Most would agree that, if both patients were retested (confirmation step), and came out with lower scores, this would decrease the belief that there “true” score is 12. If they came out with 12 again, this would increase the belief that the true score is 12.

  23. Multiple testing  “If you torture the data long enough, it will eventually confess”  Two different situations for new arthritis treatment compared to aspirin. A. Only pain (0-10) and swelling (0-10) are measured. Both are significantly better at p < 0.05 on the new treatment compared to aspirin. • Ten different outcomes measured: pain, swelling, activities of daily living, quality of life, sleep, walking, bending, lifting, grinding, climbing. Only the two that are significant are reported after all 10 are evaluated.(fraud?) Confirmatory studies specify outcomes in advance. Misleading to report only statistically significant results.

  24. How to really lie with stats for fun and profit • Bet on the horse after you know who won (Movie -> The “Sting”) 2. Send financial advice after you know how the market did (example in class)

  25. Multiple Testing Assume all results are reported, not just the significant ones. Bonferroni: out of m independent tests at 0.05, num significant by chance alone, even when all null hypotheses are true (assumes independence).

  26. Multiple testing-What to do? Option 1: Use nominal alpha level for significance. Creates too many false positives. Option 2: Use Bonferroni criterion –Declare significance if p < α/m if “m” tests are made. Has too many false negatives. Option 3: Use Hochberg criterion – a compromise

  27. Hochberg criterion Rule for m (not necessarily independent) significance tests. Keeps overall false positive rate at α. 1)Sort the “m” p values from lowest to highest. 2)Declare the ith ordered p significant if it is less than (i/m)α. If p > (i/m)α, this & all larger p values are declared non significant. Oveall type I error rate (FWER) is ≤ α.

  28. Hochberg Example for m=5, α=0.05 ip value (i/5)α (i/5) 0.05 1 p1-smallest α/5 0.01 2 p2 (2/5)α 0.02 3 p3 (3/5)α 0.03 4 p4 (4/5)α 0.04 5 p5-largest α 0.05

  29. No adjustmentvs Hochberg vsBonferroni

  30. Multiple testing & primary outcomes As “m’, the number of outcomes, increases, individual αi for each outcome must be smaller so n must be larger if overall α is to stay constant (ie at α=0.05). But not all outcomes are equally important. Designate important outcomes “primary” & the rest secondary so m is only the number of primary outcomes. Assumes less concern if there is a false positive finding among secondary outcomes. Must designate primary vs secondary outcomes in advance, before study results are known. It is not fair to declare which outcomes are primary and which are secondary based on their p values.

  31. Statistical Analysis Plan Statistical models and methods to answer study questions Conclusions = data + models (assumptions) Each specific aim needs a stat analysis section. Sample size and power follows the analysis plan. Outline: •Outcomes: denote primary & secondary •Primary predictors or comparison groups •Covariates/confounders/effect modifiers •Methods for missing data, dropouts •Interim analyses (for efficacy, for safety)

  32. Common Methods Univariate analysis Continuous outcome: Means, SDs, medians Time to event: Survival curves Discrete: Proportions Multivariate analysis Continuous outcome: Linear regression,correlation Positive integers: Poisson regression Binary (yes/no): Logistic regression Time to event: Proportional hazard regression ANOVA and t-test are special cases of linear regression

More Related