The Two-Group Randomized Experiment

1. The Two-Group Randomized Experiment

2. The Basic Design R X O R O • Note that a pretest is not necessary in this design. Why? • Because random assignment assures that we have probabilistic equivalencebetween groups

3. The Basic Design R X O R O • Differences between groups on posttest indicate a treatment effect. • Usually test this with a t-test or one-way ANOVA. • Why no pretest?

4. Internal Validity History Maturation Testing Instrumentation Mortality Regression to the mean Selection Selection-history Selection- maturation Selection- testing Selection- instrumentation Selection- mortality Selection- regression Diffusion or imitation Compensatory equalization Compensatory rivalry Resentful demoralization R X O R O Examples

5. Experimental Design Variations • The posttest-only two group design is the simplest; there are many variations. • To better understand what the variations try to achieve, we can use the signal-to-noise metaphor.

6. what we see Signal to Noise What we observe can be divided into

7. what we see Signal to Noise What we observe can be divided into Signal

8. what we see Signal to Noise What we observe can be divided into Signal Noise

9. Signal to Noise Experimental designs can take two approaches:

10. Signal to Noise Experimental designs can take two approaches: Signal Focus on (enhance) thesignal

11. Signal to Noise Experimental designs can take two approaches: Signal Focus on (enhance) thesignal (what is this?)

12. Signal to Noise Experimental designs can take two approaches:

13. Signal to Noise Experimental designs can take two approaches: Noise (what is this?) or reduce the noise

14. Signal to Noise Experimental designs can take two approaches: Noise or reduce the noise

15. Signal to Noise Signal enhancers • Factorial designs • Covariance designs • Blocking designs Noise reducers

16. Noise and Signal in Significance Tests • For interval and ratio dependent variables, you can conduct a difference in means test: Signal Noise

17. Factorial Designs

18. A Simple Example R X11 O R X12 O R X21 O R X22 O Time in Instruction 1 hour per week 4 hours per week Setting In-class Pull-out Factor 1: Level 1: Level 2: Factor 2: Level 1: Level 2:

19. A Simple Example Time in Instruction Factors: Major independent variables Setting

20. A Simple Example Time in Instruction 1 hour/week 4 hours/week In-class Levels: subdivisions of factors Setting Pull-out

21. A Simple Example Time in Instruction factors 1 hour/week 4 hours/week levels Group 1 average Group 3 average In-class Group 2 average Group 4 average Setting Pull-out Usually, averages are in the cells.

22. Multiplicative Notation A 3 x 4 factorial design How many factors? How many levels? How many cells with averages?

23. Multiplicative Notation A 3 x 4 factorial design The number of numbers tells you how many factors there are. There are 2 factors because there are 2 numbers.

24. Multiplicative Notation A 3 x 4 factorial design The number values tell you how many levels are in each factor.

25. Multiplicative Notation A 3 x 4 factorial design The number values tell you how many levels are in each factor. Factor 1 has 3 levels. Factor 2 has 4 levels.

26. The Null Case Time 1 hr 4 hrs The lines in the graphs below overlap each other. 5 5 5 Out Setting 5 5 5 In 5 5

27. A Main Effect • A consistent difference between levels of a factor • For instance, we would say there’s a main effect for time if we find a statistical difference between the averages for the hours of instruction between groups

28. Main Effects Time 1 hr 4 hrs 5 7 6 Out Main Effect of Time Setting 5 7 6 In 5 7

29. Main Effects Time 1 hr 4 hrs 5 5 5 Out Main Effect of Setting Setting 7 7 7 In 6 6

30. Main Effects Time 1 hr 4 hrs 5 7 6 Out Main Effects of Time and Setting Setting 7 9 8 In 6 8

31. An Interaction Effect • When differences on one factor depend on the level you are on on another factor • An interaction is between factors (not levels) • You know there’s an interaction when can’t talk about effect on one factor without mentioning the other factor

32. Interaction Effects Time 1 hr 4 hrs 5 5 5 Out The in-class, 4-hour per week group differs from all the others. Setting 5 7 6 In 5 6

33. Interaction Effects Time 1 hr 4 hrs The 1-hour amount works well with pull-outs while the 4 hour works as well with in class. 7 5 6 Out Setting 5 7 6 In 6 6

34. Advantages of Factorial Designs • Offers great flexibility for exploring or enhancing the “signal” (treatment) • Makes it possible to study interactions • Combines multiple studies into one

35. Randomized Block Designs

36. The Basic Design R X O R O

37. The Basic Design R X O R O R X O R O R X O R O R X O R O

38. The Basic Design Homogeneity on the dependent variable (observations in group one tend to have higher levels of the dependent variable than observations in group two, etc.) R X O R O R X O R O Homogeneous groups R X O R O R X O R O

39. Randomized Blocks Design R X O R O R X O R O Each replicate is ablock. Homogeneous groups R X O R O R X O R O

40. Randomized Blocks Design • Can block before or after the study • Can block on a measured variable or on unmeasured perceptions • Is a noise-reducing strategy

41. 1 0 0 9 0 8 0 7 0 6 0 5 0 4 0 3 0 2 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 How Does Blocking Reduce Noise? Posttest Pretest

42. 1 0 0 9 0 8 0 7 0 6 0 5 0 4 0 3 0 2 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 How Does Blocking Reduce Noise? Range of x Posttest Pretest

43. 1 0 0 9 0 8 0 7 0 6 0 5 0 4 0 3 0 2 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 How Does Blocking Reduce Noise? Range of x Posttest Variability of y Pretest

44. 1 0 0 9 0 8 0 7 0 6 0 5 0 4 0 3 0 2 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 How Does Blocking Reduce Noise? Range of x Posttest Variability of y Mean difference Pretest

45. 1 0 0 9 0 8 0 7 0 6 0 5 0 4 0 3 0 2 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 How Does Blocking Reduce Noise? For block 3 Posttest Pretest

46. 1 0 0 9 0 8 0 7 0 6 0 5 0 4 0 3 0 2 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 How Does Blocking Reduce Noise? For block 3 Range of x in block Posttest Pretest

47. 1 0 0 9 0 8 0 7 0 6 0 5 0 4 0 3 0 2 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 How Does Blocking Reduce Noise? For block 3 Range of x in block Posttest Variability of y in block Pretest

48. 1 0 0 9 0 8 0 7 0 6 0 5 0 4 0 3 0 2 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 How Does Blocking Reduce Noise? For block 3 Range of x in block Posttest Mean difference Variability of y in block Pretest

49. 1 0 0 9 0 8 0 7 0 6 0 5 0 4 0 3 0 2 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 How Does Blocking Reduce Noise? For block 3 Range of x in block Posttest Mean difference Variability of y in block Same mean difference, but lower variability on both x and y Pretest

50. Conclusion • Instead of having one treatment effect based on the full variability of y, you have three treatment effects based on reduced variability (but with the same mean difference). • The average of the three estimates gives a less noisyestimate than the nonblock one.