Statistical Power for Cluster-Level Moderators - Overview of Moderation Effects

1. Cluster Level Moderation Jessaca Spybrook Western Michigan University

2. Cluster Level Moderation • Outline for Session • Brief conceptual overview • Examples and models • Power/Minimum detectable effect size difference (MDESD)

3. Conceptual Overview • Move beyond main effect of treatment • Consider under what conditions a program works • Critical questions for program developers, funders, policymakers, school personnel, and many others

4. Conceptual Overview • Examples • Village level intervention • Access to health care • Local politics • Size of village • Adult literacy rates in village

5. Conceptual Overview • Example • The Gambia Data, School-based intervention • Condition of buildings • Number of classrooms in the school • Number of teachers in the school • One or two shifts of school per day • Presence of library • Number of working toilets

6. Two Types of Moderator Effects • Treatment with Binary • Example • Treatment by Presence of Library • Treatment with Continuous • Example • Treatment by Number classes per school

7. Example • Scenario • The Gambia data (2009) • Students nested in schools • 2,657 -> 1,573 students (pupils data) • 271 -> 173 schools (head teacher data)

8. Example • Variables • DV: • Numbers of words read correctly in 60 seconds (reading fluency) [S2Q3_PP] • IVs at L2: • Library [Q216] • WSD indicator [WSD] • Library x WSD [TRMTBYLI] • Product of centered Lib x WSD [CENTWSDL] • School mean reading fluency 2008 [MEAN08]

9. Guiding Question • Guiding Question • Research suggests that the presence of a library in a school is an important condition for academic success. To test this theory, we are interested in examining whether the effect of WSD is different in schools with a library compared to schools without a library.

10. The Model Level 1 (students): Yijis reading fluency for student i in school j is the mean reading fluency for school j rij is the random error associated with student i in school j, Level 2 (schools): is the average school mean reading fluency for control schools with no library is the average difference between control schools with a library and without a library is the average difference between treatment and control for schools with no library is the increase or decrease in the treatment effect for schools with a library compared to schools without a library u0j is the random error associated with school mean, conditional on the X’s,

11. Interpretation Combined Model: See output

12. Results/Interpretations • Note that these are not statistically significant  • Average reading fluency for students in control schools with no library is 33.70 • Average increase in reading fluency for students in control schools with library compared to those without a library is 0.76 • Average treatment effect for schools with no library is -4.68 • Treatment effect for schools with a library is higher than for schools without a library by an average of 4.16

13. Centering • Two options • Uncentered (previous example) • Center treatment, moderators, and then compute product of the centered variables for the moderator effect

14. Centering • In the context of moderator effect, centering will: • Reduces multicollinearity • Change the interpretation, estimates, and standard errors of intercept and main effects • NOT change the interpretation, estimate, or standard error of the moderator effect • See output • Choice depends on research questions

15. Statistical Power for Cluster Level Moderators • Suppose planning a CRT of intervention similar to WSD in a similar context • Use this study to plan future CRT

16. Statistical Power Model: To increase precision, include pre-test. New Model:

17. Statistical Power Estimated treatment effect: Variance of treatment effect:

18. Statistical Power Hypothesis Test: F statistic: Under the alternative hypothesis, F statistic follows non-central F distribution with non-centrality parameter:

19. Statistical Power Standardized Noncentrality Parameter: where

20. Estimating Design Parameters • ICC • Unconditional model (see output) • R2 • Lib, WSD, Interaction, School-level mean reading fluency in 2008 as “pre-test” (see output)

21. MDESD • Minimum detectable effect size difference Note: These values assume half the clusters assigned to treatment and cluster moderator, alpha = 0.05, power = 0.80, and

22. MDESD • Magnitude • Context specific • Outcome (proximal vs. distal) • Intervention (level of intensity) • Strength of moderator

23. MDESD • Magnitude • How to estimate? • Literature • Similar programs • Pilot study • In our example • Moderator effect:4.00 • Standardized moderator effect:4.00/(10.23+21.05) = 0.13 • Based on our parameters, assuming 100 kids per school, need about 380 schools

24. Implications • Cluster level moderators • Challenging given cost constraints! • Need lots of clusters! • If priority, need to consider in design stage • Consider 3 cases • Balanced • Unbalanced • Unbalanced

25. Implications • Case 1: Balanced 60 15 T-L, 15 T-NL 15 C-L, 15 C-NL 15 per group * 4 = 60 total clusters 30 NL 30 L 15C 15 T 15 T 15C 30 T 30 C

26. Implications • Case 2: Unbalanced 60 20 T-L, 10 T-NL 20 C-L, 10 C-NL Harmonic mean (20, 10) Harmonic mean = 13.3 Effective sample size = 13.3*4=53 total clusters 20NL 40L 20C 10T 20T 10C 30 T 30 C

27. Implications • Case 3: Unbalanced 60 Unbalanced case 10T-L, 20T-NL 20T-L, 0 C-NL No C-NL group! 20NL 40L 30C 20T 10T 0C 30 T 30 C

28. Next Steps • Practice session in lab • Questions/comments via video session