Random Effects & Repeated Measures

1. Random Effects & Repeated Measures Alternatives to Fixed Effects Analyses

2. Questions • What is the difference between fixed- and random-effects in terms of treatments? • How are F tests with random effects different than with fixed effects? • Describe a concrete example of a randomized block design. You should have 1 factor as the blocking factor and one other factor as the factor of main interest.

3. Questions (2) • How is a repeated measures design different from a totally between subjects design in the collection of the data? • How does the significance testing change from the totally between to a design to one in which one or more factors are repeated measures (just the general idea, you don’t need to show actual F ratios or computations)? • Describe one argument for using repeated measures designs and one argument against using such designs (or describe when you would and would not want to use repeated measures).

4. Fixed Effects Designs • All treatment conditions of interest are included in the study • All in cell get identical stimulus (treatment, IV combination) • Interest is in specific means • Expected mean squares are (relatively) simple; F tests are all based on common error term.

5. Random Effects Designs • Treatment conditions are sampled; not all conditions of interest are included. • Replications of the experiment would get different treatments • Interest in the variance produced by an IV rather than means • Expected mean squares relatively complex; the denominator for F changes depending on the effect being tested.

6. Fixed vs. Random

7. Single Factor Random • The expected mean squares and F-test for the single random factor are the same as those for the single factor fixed-effects design.

8. Experimenter effects (Hays Table 13.4.1)

9. Sum of Source DF Squares Mean Square F Value Pr > F Model 4 3.48150000 0.87037500 10.72 <.0001 Error 35 2.84250000 0.08121429 Corrected Total 39 6.32400000

10. Random Effects Significance Tests (A & B random/within)

11. Why the Funky MS? • Treatment effects for A, B, & AxB are the same for fixed & random in the population of treatments. • In fixed, we have the population, in random, we just have a sample. • Therefore, in a given (random) study, the interaction effects need not sum to zero. • The AxB effects appear in the main effects.

12. Applications of Random Effects • Reliability and Generalizability • How many judges do I need to get a reliability of .8? • How well does this score generalize to a particular universe of scores? • Intraclass correlations (ICCs) • Estimated variance components • Meta-analysis • Control (Randomized Blocks and Repeated Measures)

13. Review • What is the difference between fixed- and random-effects in terms of treatments? • How are F tests with random effects different than with fixed effects?

14. Randomized Blocks Designs • A block is a matched group of participants who are similar or identical on a nuisance variable • Suppose we want to study effect of a workbook on scores on a test in research methods. A major source of nuisance variance is cognitive ability • We can block students on cognitive ability.

15. Randomized Blocks (2) • Say 3 blocks (slow, average, fast learners) • Within each block, randomly assign to workbook or control. • Resulting design looks like ordinary factorial (3x2), but people are not assigned to blocks. • The block factor is sampled, i.e., random. The F test for workbook is more powerful because we subtract nuisance variance. • Unless blocks are truly categorical, a better design is analysis of covariance, described after we introduce regression.

16. Randomized Blocks (3)

17. Review • Describe a concrete example of a randomized block design. You should have 1 factor as the blocking factor and one other factor as the factor of main interest. • Describe a study in which Depression is a blocking factor.

18. Repeated Measures Designs • In a repeated measures design, participants appear in more than one cell. • Painfree study • Sports instruction • Commonly used in psychology

19. Pros & Cons of RM

20. RM – Participant ‘Factor’

21. Drugs on Reaction Time Order of drug random. All Ss, all drugs. Interest is drug. Drug is fixed; person is random. ‘1 Factor’ repeated measures design. Notice 1 person per cell. We can get 3 SS: row, column, and residual (interaction plus error).

22. Total SS

23. Drug SS

24. Person SS

25. Summary Total = 1491.8; Drugs = 698.2, People=680.8. Residual = Total –(Drugs+People) = 1491.8-(698.2+680.8) =112.8

26. SAS Run the same problem using SAS.

27. 2 Factor, 1 Repeated DV=errors in control setting dials; IV(A) is dial calibration - between; IV(B) is dial shape - within. Observation is randomized over dial shape.

28. data d1; input i1-i4; cards; 0 0 5 3 3 1 5 4 4 3 6 2 4 2 7 8 5 4 6 6 7 5 8 9 data d2; set d1; array z i1-i4; do over z; if _N_ le 3 then a =1; if _N_ gt 3 then a =2; sub = _N_; b = _I_; y=z; output; end; proc print; proc glm; class a b sub; model y = a b a*b sub(a) sub*b; test h=a e=sub(a)/htype=1 etype=1; test h=b a*b e=sub*b/htype=1 etype=1; run;

29. Summary Note that different factors are tested with different error terms.

30. SAS & Post Hoc Tests Run the same problem using SAS. The SAS default is to use what ever is residual as denominator of F test. You can use this to your advantage or else over-ride it to produce specific F tests of your desire. If you use the default error term, be sure you know what it is. Post hoc tests with repeated measures are tricky. You have to use the proper error term for each test. The error term changes depending on what you are testing. Be sure to look up the right error term.

31. Assumptions of RM Orthogonal ANOVA assumes homogeneity of error variance within cells. IVs are independent. With repeated measures, we introduce covariance (correlation) across cells. For example, the correlation of scores across subjects 1-3 for the first two calibrations is .89. Repeated measures designs make assumptions about the homogeneity of covariance matrices across conditions for the F test to work properly. If the assumptions are not met, you have problems and may need to make adjustments. You can avoid these assumptions by using multivariate techniques (MANOVA) to analyze your data. I suggest you do so. If you use ANOVA, you need to look up your design to get the right F tests and check on the assumptions.

32. Review • How is a repeated measures design different from a totally between subjects design in the collection of the data? • How does the significance testing change from the totally between to a design to one in which one or more factors are repeated measures (just the general idea, you don’t need to show actual F ratios or computations)? • Describe one argument for using repeated measures designs and one argument against using such designs (or describe when you would and would not want to use repeated measures).