Optimal Design for Longitudinal and Multilevel Research

1. Optimal Design for Longitudinal and Multilevel Research Jessaca Spybrook July 10, 2008 *Joint work with Steve Raudenbush and Andres Martinez

2. Outline • Overview • Examples • 2-level HLM • 3-level HLM • 3-level Randomized Block Design • Writing up a Power Analysis • Exercises

3. Overview Tool for planning studies HLM notation Connection to Howard’s formulas Warning: Optimal Design is easy to use but can also be easily misused

4. Connecting the Language Larry OD 2-level HLM Cluster Randomized Trial 3-level HLM Three level model with treatment at level three 3-level RBD Multi-site Cluster Randomized trial

6. Organization of Optimal Design • Optimal Design for Group Randomized Trials • Cluster Randomized Trial (2-level HLM) • Three level model with treatment at level 3 (3-level HLM) • Multi-site cluster randomized trial (3-level RBD)

7. Cluster Randomized Trial (2-level HLM) Example Program: A new math curriculum is randomly assigned and implemented at the school level. Research question: Is the new math curriculum more effective than the traditional one? Standardized math tests scores used as outcome measure. Based on previous research: • 18% percent of the variation lies between schools. (Students nested within schools). • Researchers expect the treatment to boost test scores by 0.25 standard deviations. With 90 students per school, how many schools are needed to detect the above effect size with 80% power?

8. Cluster Randomized Trial (2-level HLM) Example • How many levels are in this study? • What is the unit of randomization? • Is there blocking? • What is the MDES? • What is a good estimate of the intraclass correlation? • What information is missing?

9. Cluster Randomized Trial (2-level HLM) J – the total number of clusters n – the total number of individuals per cluster Rho – the intraclass correlation R-squared – the percent of variation explained by level-2 covariate Delta – the effect size

10. Three level model with treatment at level three (3-level HLM) Example Program: Third grade math curriculum implemented at the school level (group-level intervention). 40 schools are randomized. There are 6 teachers within each school and 25 kids per teacher. Research question: Is the new math curriculum more effective than the traditional one? Standardized math tests scores used as outcome measure. Based on previous research: • 8% percent of the variation lies between classes. 12% of the variation lies between schools. What is the MDES for power of 0.80? Is this reasonable?

11. Three level model with treatment at level three (3-level HLM) Example How many levels are in this study? What is the unit of randomization? Is there blocking? What is the MDES? What are good estimates of the intraclass correlations? What information is missing?

12. Three level model with treatment at level three (3-level HLM) K- the total number of schools J – the total number of classrooms per school n – the total number of individuals per classroom Rho pi (level-2 ICC)– the intraclass correlation at classroom level Rho beta (level-3 ICC) – the intraclass correlation at school level R-squared – the percent of variation explained by level-3 covariate Delta – the effect size

13. Multi-site Cluster Randomized Trial (3-level RBD) Example Program: The following will be repeated across districts. 6 schools will be randomly assigned to either the new math curriculum or the regular curriculum. 150 students will be tested in each school. Causal question: Is the new math curriculum more effective than the traditional one? Standardized math tests scores used as outcome measure. Based on previous research: • 25% percent of the variation lies between districts. • We expect blocking will account for 30% of the variation. • Researchers expect the treatment to boost test scores by 0.20 standard deviations. How many districts are needed to detect an effect size of 0.20 with 80% power?

14. Multi-site Cluster Randomized Trial (3-level RBD) Example • How many levels are in this study? • What is the unit of randomization? • Is there blocking? • If so, how much of the variation is explained by blocking? How are blocks being treated – as fixed or random effects? If random effects, what is the variation of treatment effect across schools? • What is the MDES? • What is the intraclass correlation prior to blocking? • What information is missing?

15. Multi-site Cluster Randomized Trial (3-level RBD) • K- the total number of blocks/sites (districts) • J – the total number of schools per district • n – the total number of individuals per school • Rho – the intraclass correlation • R-squared – the percent of variation explained by level-3 covariate • B – percent of variation explained by blocking • Sigma sq sub delta – effect size variability (specifying greater than 0 treats blocks as random effects) • Delta – the effect size

16. Writing up a Power Analysis • Differs depending on the design (again it is critical to know your design!)

18. Exercise 1 Suppose a group of researchers want to examine the effectiveness of a professional development program intended to improve reading instruction for elementary teachers. They plan to randomly assign schools to the treatment (the new pd).Within each school, they select four 2nd grade teachers to implement the program. Each teacher has approximately 25 students. The outcome of interest is student reading achievement. They expect that students in the treatment schools will improve their reading scores by 0.25 standard deviations. They plan to use last years students posttest as a covariate. How many schools are necessary for 80 percent power? (Hint: use the best estimates for the intraclass correlation(s))

19. Exercise 2 Suppose that a team of researchers plan to evaluate the effectiveness of a scripted whole school reform model. They plan to randomly assign entire schools to either receive the new reform model or continue with their current practices. On average, there are 300 students in each school. They expect that students in the treatment schools will outperform students in the control schools by 0.20 standard deviations on a particular math test. They estimate that approximately 20 percent of the variation lies between schools. They plan to use last years math scores as a covariate. How many schools do they need to have 80 percent power? What if they only tested 50 kids in the school, how does that affect the power?

20. Exercise 3 Suppose that a team of researchers recruited 6 districts for a study of the effectiveness of a new science program. They plan to randomly assign 4 schools within each district to either the new curriculum or their current curriculum and to test 150 students within each school. They expect that by blocking on district they will remove 25 percent of the variation in science outcome scores. They estimate an intraclass correlation of 0.24 and have a pre-test covariate that they think will explain approximately 60 percent of the variation in test scores. What is the smallest effect size they can detect with power of 0.80? Is this reasonable? (Hint: remember to go through the steps of selecting the appropriate design and all the relevant parts.)