Analysis of Experiments

1. Analysis of Experiments Larry V. Hedges Northwestern University Prepared for the IES Summer Research Training Institute July 7 – 17, 2008

2. Software Many kinds of software can provide valid analyses of experiments with complex designs • SAS Proc Mixed • STATA XTMixed • SPSS GLM • HLM We will focus on HLM because it is very general and easy to use, but if you want to use SAS Proc Mixed, we will try to accommodate you in using that here The choice of programs should be based on your level of comfort

3. What is HLM? HLM is not a complete statistical package like SAS, SPSS, or STATA HLM is a software tool that works with other statistical packages such as SPSS, SAS, or even EXCEL Data management and preparation (recoding and creation of variables) is done in the general package (e.g., SPSS) HLM then takes the data intended for analysis and performs various complex analyses on the data

4. How Does HLM Work? HLM reads your data and creates summary files from it Multivariate Data Matrix (or MDM) files There is one MDM file for each level of the data analysis. It essentially is the information necessary to carry out the analyses at each level of the HLM model Every variable that you plan to use in the HLM analysis must be included in the MDM file at the level you intend to use it. If you decide to include new variables, you must create new MDM files Your data must be sorted by the ID variable that links different levels. If it is not, you will get an error message

5. Specifying Analyses Know the design Think through the levels of the design that will be included in the analysis Decide on the inference model for each level Do I want to generalize to a larger universe than just the units in the sample? Different levels may have different inference models Levels that exist in the design may be left out of the analysis model (but be sure you know what you are doing!)

6. Specifying Analyses Know the design Generally Covariate effects should be fixed effects Treatment effects should be random effects when the design permits it (e.g., randomized blocks designs) Sometimes, there are too few units at a level to permit making it a random effect, even when this is theoretically sensible (e.g., 1 – 2 classrooms in a hierarchical design that assigns schools)

7. Specifying Analyses(Centering) Centering is a transformation of the independent variables In simple (simple random sample) designs, centering is just subtracting the mean If Xi is the independent variable, the centered variable is where is the mean of the Xi’s in the sample Thus the mean of the centered variable is 0

8. Specifying Analyses(Centering) What does centering do? It changes the meaning of the intercept Consider a regression equation Yi = β0 + β1Xi + εi And the regression equation with the centered predictor Equating the centered and uncentered regression see that Centering also changes the precision of estimate

9. Specifying Analyses(Centering) In complex (multilevel) designs, there is more than one kind of centering: • Centering by grand mean • Centering by group mean Grand mean centering is subtracting the grand mean Group mean centering is subtracting the group mean These centering methods affect interpretation of the intercept

10. Specifying Analyses(Grand Mean Centering) What does grand mean centering do? It changes the meaning of the intercept in the ith cluster Consider a regression equation Yij = β0i + β1iXij + εij And the regression equation with the centered predictor Equating the centered and uncentered regression see that Centering also changes the precision of each estimate

11. Specifying Analyses(Group Mean Centering) What does group mean centering do? It changes the meaning of the intercept in the ith cluster Consider a regression equation Yij = β0i + β1iXij + εij And the regression equation with the centered predictor Equating the centered and uncentered regression see that Centering also changes the precision of each estimate

12. Specifying Analyses(Group/Grand Mean Centering) Generally it is useful to group mean center covariates This permits the group mean to be used as a covariate at the group level But the major reason to center is to increase precision of estimates, particularly estimates of variances of intercepts (which figure into standard errors of estimates of treatment effects) Centering method should not greatly affect the results of analyses of experiments

13. Specifying Analyses(Weighting) Weighting is sometimes used when samples are selected with the probability of selection in a stratum not proportionate to the size of the population Sometimes people are tempted to weight in experiments so that the weighted proportions equal some preset (population) values This is most defensible in experiments with randomized block designs If treatment effects are heterogeneous, different weightings typically lead to different average treatment effects

14. Creating MDM Files First start the HLM Program by clicking on the icon Next go to the file menu • Select make new MDM file, • Select from Stat Package input and select the relevant package Then indicate that you want an HLM2 (two level analysis) or HLM3 (three level analysis)

15. Creating MDM FilesSelect Level 1 Variables Select the radio button for nesting persons in groups Under level 2 specification Select “Browse” and select the file that has the level 2 variables Select “Choose Variables” under level 2 specification • Select one variable as the ID variable (this variable is used to link the level 1 and level 2 data) • Select all of the other variables needed for the level 2 model Select “Make MDM” Then select “Check Stats” Then select “Done”

16. Creating MDM FilesSelect Level 2 Variables Select the radio button for nesting persons in groups • Select the file that has the level 1 variables • Select one variable as the ID variable (this variable is used to link the level 1 and level 2 data) • Select all of the other variables needed for the level 1 model Indicate if there is missing data and if so, how it will be handled • When making the MDM file • When doing analyses