Sample Size Considerations for Answering Quantitative Research Questions

1. Sample Size Considerations for Answering Quantitative Research Questions Lunch & Learn May 15, 2013 M Boyle

2. National Children’s Study in the US Proposed Birth Cohort 100,000 to age 21 Planning Costs 2000-2006: $54.7M Implementation Costs 2007-2011: $744.6 Sample Size Justification: ?

3. What is Statistical Power? • The statistical power of a test is the probability of correctly rejecting H0 when it is false. In other words, power is the likelihood that you will identify a statistically significant effect when one exists

4. Types of Power Analysis A priori: Used to plan a study (Question: What sample size is needed to obtain a certain level of power)? Post hoc: Used to evaluate a study faced with a constrained sample size (Question: Do you have a large enough sample to detect a meaningful effect)? [Types of constraints: (1) a completed study; (2) a proposed study with limited number of eligible subjects; (3) a proposed study faced with limited resources]

5. Elements of Power Calculations • Effect size ∆ • Measurement variability SD • Type I error Alpha (α) typically specified at p=0.05, 2-tailed • Type II error Beta (β) typically specified at p=0.20 • Power = 1-β; typically 0.80 • Sample Size

6. Hypothesized distributions, effect sizes and error rates Effect Size ∆ Type I Type II Measurement Variability +/- 1 SD

7. Decisions Disease Status Present Absent Medical Diagnosis +ve Test Result -ve correct false +ve false -ve correct Population Status H0 false H0 true[H1 true] Hypothesis Testing correct Type II 1-αβ Accept H0 Decision Reject H0 [Accept H1] Type I correct α 1-β (power)

8. Example Power Calculation H0: At 2 years of age, the IQs of newborns randomly allocated to the NFP program will be no different than newborns allocated to usual care. H1: At 2 years of age, the IQs of newborns randomly allocated the NFP program will be 5 points higher. Effect size ∆ = SD = Alpha (α) = Beta (β) = Power = Sample Size ?

9. Example Power Calculation H0: At 2 years of age, the IQs of newborns randomly allocated to the NFP program will be no different than newborns allocated to usual care. H1: At 2 years of age, the IQs of newborns randomly allocated the NFP program will be 5 points higher. Effect size ∆ = 5 SD = 15 Alpha (α) = 0.05 2-tailed Beta (β) = 0.20 Power = 80 Sample Size 146 per group

10. n=292

11. FACTORS THAT INFLUENCE SAMPLE SIZE PLANNING AND STATISTICAL POWER

12. Sample Size Planningand Power 1. Error rates Type I (α) -smaller α requires larger sample sizes -2-sided tests requires larger sample sizes Type II (β) Statistical power: -smaller β (more power) requires larger sample sizes [Use conventional levels & worry about the trade- offs between effect size and sample size]

13. Sample Size Planningand Power 2. Effect Size ∆ “What is the minimally important effect based on clinical, biological or social implications of the findings?”

14. Sample Size Planningand Power • Effect size ∆ • What do you know about the nature of the effect – its scale of measurement and its perceived importance to practice, policy, resource allocation (e.g., infant mortality; dollars; self-esteem)? • What do previous empirical studies tell you about achievable effects?

15. Sample Size Planningand Power 1.Effect Size ∆ • Can you generate a consensus among your investigative team on a minimally important effect? • Is it reasonable to use conventional estimates of small, medium and large? • Are you limited by the dollar amount you can request?

16. Sample Size Planningand Power • The measurement scale of the dependent variable: discrete, ordinal, interval -interval level measurements require smaller samples • The variability of the dependent variable in the general population (SD, Variance) -lower variability requires smaller sample sizes

17. Sample Size Planningand Power • The statistical test -simple estimation; differences between groups; correlation and prediction. The test must be appropriate for the question and data. A key element in sample size planning 5. Sample distribution, for example, exposed versus not exposed) -balanced is the most powerful

18. Sample Size Planningand Power 6. Attrition loss of subjects -higher attrition leads to lower power 7. Measurement reliability -complicated: if true variance is constant and error variance is reduced statistical power will increase

19. Sample Size Planningand Power 8.Study costs – what the market will bear 9. Analytical complexity – what to do when your models require much more information than you can get?

20. Adding Complexity • Multilevel Model yij = β0j + β1z0j+(u0j + eij) y H0 The association between neighbourhood affluence measured on resident 4-16 year olds in 1983 and years of education assessed in 2001 will be = 0.00 standard units x H0∆ = β1z0j > 0.20 Neigh Affluence

21. Estimates • 2-level balanced data, nested model • Significance level = 0.025 (to get 0.05 2-tailed) • Number of simulations per setting = 100 • Response variable = normal • Estimation method = IGLS • Fixed intercept = yes • Random intercept = yes • Number of explanatory variables = 1 • Type of predictor = continuous

22. Estimates • Mean of the predictor = 0.0 • Variance of the predictor at level 1 = 0.0 • Variance of the predictor at level 2 = 1.0 • Smallest/Largest # units at L1 (increment) • Smallest/Largest # units at L2 (increment) • Estimate β0 = 0 • Estimateβ1= 0.15 • Estimate L2 variance 0.05 • Estimate L1 variance 0.95

24. Comments • Ask specific, quantifiable research questions • Consult with colleagues about clinical, biological and social importance of your outcomes • Move from simple to complex hypotheses. Complex models – SEM, Multilevel – can require you to provide an enormous number of parameters. • When estimating sample size requirements for complex models, you will inevitably use standardized variables

25. Comments • Estimating sample size requirements is part game, subject to practical constraints (limited resources and subjects) and convincing reviewers that you know what your doing • Take a ‘reasoned’ approach – most reviewers will have no clue what you are going on about • The hardest part of the process is acquiring the information you need.