Meta-Analysis using HLM 6.0

1. Meta-Analysis using HLM 6.0 Yaacov Petscher Florida Center for Reading Research

2. Why use HLM? • Nested structure • Necessity of special models • Variation at both subject and study levels

3. “Special Case” of HLM • If ES are based on n ≥ 30, we assume approximate normal distribution with sampling variance assumed to be known • V-known models run in Interactive Mode • Time to brush up on your DOS command code!

4. Standardized Mean Difference • No raw data for us • Must rely on stats to be converted to single metric • Many types of statistics that may be used Z t M/SD χ² F p-value r r²

5. Level-1 (Within-Studies) Model = any standardized effect measure from study j = the corresponding population parameter = sample error associated with d

6. Level-2 (Between-Studies) Model = grand mean effect size = regression coefficients = study characteristics (moderators) = level 2 random error

7. Combined Model

8. Estimation Empirical Bayes Estimator where

9. EB Estimates • Level 1 • May be used as a shrinkage estimator to identify potential outliers • Shrinkage in the direction of the grand mean • Level 2 • Supplying the grand mean provides an estimate of the conditional shrinkage • Shrinkage towards a value that is conditional on the amount of prior contacts (WEEKS)

10. The following example will be run using data from Raudenbush & Bryk Chapter 7 data (pg 211) The Effect of Teacher Expectancy on Pupil IQ

12. Some Calculations • Need the Conditional Variances • Since d in this model is Fisher’s r to Z transformation, the formula is Since we’re not given n we need to calculate another way…..ideas?

13. YAY! • Simply square your standard error

14. Data File Prep Considerations • Since meta-analysis in HLM is a V-known model, only one data file is used

15. Data File Prep Considerations, cont • Four key features to data prep (assume using SPSS) • Column 1 = ID in character format • Column 2 = ES estimates • Column 3 = Variance estimates • Column 4-n = Potential level-2 predictors

16. Formatting • Variable View • Column 1 • String, width = 2, decimal = 0 • Columns 2-n • Numeric, width = 12, decimal = 3 • Save as a Fixed ASCII (.dat) file • Hold onto your output, you’re gonna need it! • You should now have a list of the variables and associated Format statements

17. HLM in Batch Mode • Bring up your computer’s Command Prompt • Typically found in “Accessories” • By default, you should see C:\> • If not, type c: and hit enter • At this point you want to locate HLM • Type dir, hit enter • cd program files, enter • cd HLM6, enter • hlm2, enter • We’re now ready to begin!

18. HLM2- MDM File Creation C:\Program Files\HLM6\hlm2 Will you be starting with raw data? y Is the input file a v-known file? y How many level-1 statistics are there? 1 How many level-2 predictors are there? 1 Enter 8 character name for level-1 variable number 1: Zes Enter 8 character name for level-2 variable number 1: weeks Input format of raw data file (the first field must be the character ID) format: (a2, 3f12.3) What file contains the data: e:\test.dat Enter name of MDM file: e:\test.mdm 19 groups have been processed C:\Program Files\HLM6>

19. Specifying UC HLM Model C:\HLM6> hlm2 e:\test.mdm SPECIFYING AN HLM MODEL Level-1 predictor variable specification Which level-1 predictors do you wish to use? The choices are: For ZES enter 1 Level-1 predictors? (Enter 0 to end) 1 Level-2 predictor variable specification Which level-2 variables do you wish to use? The choices are: For WEEKS enter 1 Which level-2 predictors to model ZES? Level-2 predictor? (Enter 0 to end) 0

20. ADDITIONAL PROGRAM FEATURES Select the level-2 variables that you might consider for Inclusion as predictors in subsequent models. The choices are: For WEEKS enter 1 Which level-2 variables to model ZES? Level-2 variable? (Enter 0 to end) 0 Do you wish to use any of the optional hypothesis testing procedures? n OUTPUT SPECIFICATION Do you want a level-2 residual file? n How many iterations do you want to do? 10000 Do you want to see OLS estimates for all of the level-2 units? n Enter a problem title: lvl1 Enter name of output file: e:\lvl1.lis

21. Results for UC Model ----------------------------------------------------------------------------------------- Standard Approx. Fixed Effect Coefficient Error T-ratio d.f. ----------------------------------------------------------------------------------------- For ZES, B1 INTRCPT2, G10 0.084376 0.052039 1.621 18 ----------------------------------------------------------------------------------------- ----------------------------------------------------------------------------------------------- Random Effect Standard Variance df Chi-square P-value Deviation Component ------------------------------------------------------------------------------------------------ ZES, U1 0.13896 0.01931 18 36.25115 0.007 ------------------------------------------------------------------------------------------------ Significant variability exists in true-effect sizes

22. EB Estimation Level 1 Since there are no predictors at Level 1, the last term is omitted, leaving us with

23. Using Excel to Calculate EB Using the variance component from lvl 1 Model, create Lambda using the formula function

24. Supplying the grand mean ES into the Excel formula function allows us get the EB estimates See the difference in results?

26. Specifying CL2 HLM Model C:\HLM6> hlm2 e:\test.mdm SPECIFYING AN HLM MODEL Level-1 predictor variable specification Which level-1 predictors do you wish to use? The choices are: For ZES enter 1 Level-1 predictors? (Enter 0 to end) 1 Level-2 predictor variable specification Which level-2 variables do you wish to use? The choices are: For WEEKS enter 1 Which level-2 predictors to model ZES? Level-2 predictor? (Enter 0 to end) 1

27. ADDITIONAL PROGRAM FEATURES Select the level-2 variables that you might consider for Inclusion as predictors in subsequent models. The choices are: For WEEKS enter 1 Which level-2 variables to model ZES? Level-2 variable? (Enter 0 to end) 0 Do you wish to use any of the optional hypothesis testing procedures? n OUTPUT SPECIFICATION Do you want a level-2 residual file? n How many iterations do you want to do? 10000 Do you want to see OLS estimates for all of the level-2 units? n Enter a problem title: lvl2 Enter name of output file: e:\lvl2.lis

28. Results for CL2 Model ------------------------------------------------------------------------------------------------- Standard Approx. Fixed Effect Coefficient Error T-ratio d.f. P-value ------------------------------------------------------------------------------------------------- For ZES, B1 INTRCPT2, G10 0.408572 0.087146 4.688 17 0.000 WEEKS, G11 -0.157963 0.035943 -4.395 17 0.000 ------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------ Random Effect Standard Variance df Chi-square P-value Deviation Component ------------------------------------------------------------------------------------------------ ZES, U1 0.00283 0.00001 17 16.53614 >.500 ------------------------------------------------------------------------------------------------

29. EB Estimation Level 2 Since then = 0 and we’re left with Using G10 and G11, we can calculate the EB estimates for Level 2

31. EB Estimation Level 2 • Since our Level-2 predictor takes on one of four different values, the shrinkage is towards one of the four points.

32. End (for now)