Basic Experimental Design

1. Basic Experimental Design Larry V. Hedges Northwestern University Prepared for the IES Summer Research Training Institute June 18 – 29, 2007

2. What is Experimental Design? Experimental design includes both • Strategies for organizing data collection • Data analysis procedures matched to those data collection strategies Classical treatments of design stress analysis procedures based on the analysis of variance (ANOVA) Other analysis procedure such as those based on hierarchical linear models or analysis of aggregates (e.g., class or school means) are also appropriate

3. Why Do We Need Experimental Design? Because of variability We wouldn’t need a science of experimental design if • If all units (students, teachers, & schools) were identical and • If all units responded identically to treatments We need experimental design to control variability so that treatment effects can be identified

4. A Little History The idea of controlling variability through design has a long history In 1747 Sir James Lind’s studies of scurvy Their cases were as similar as I could have them. They all in general had putrid gums, spots and lassitude, with weakness of their knees. They lay together on one place … and had one diet common to all (Lind, 1753, p. 149) Lind then assigned six different treatments to groups of patients

5. A Little History The idea of random assignment was not obvious and took time to catch on In 1648 von Helmont carried out one randomization in a trial of bloodletting for fevers In 1904 Karl Pearson suggested matching and alternation in typhoid trials Amberson, et al. (1931) carried out a trial with one randomization In 1937 Sir Bradford Hill advocated alternation of patients in trials rather than randomization Diehl, et al. (1938) carried out a trial that is sometimes referred to as randomized, but it actually used alternation

6. A Little History Studies in crop variation I – VI (1921 – 1929) In 1919 a statistician named Fisher was hired at Rothamsted agricultural station They had a lot of observational data on crop yields and hoped a statistician could analyze it to find effects of various treatments All he had to do was sort out the effects of confounding variables

7. Studies in Crop Variation I (1921) Fisher does regression analyses—lots of them—to study (and get rid of) the effects of confounders • soil fertility gradients • drainage • effects of rainfall • effects of temperature and weather, etc. Fisher does qualitative work to sort out anomalies Conclusion The effects of confounders are typically larger than those of the systematic effects we want to study

8. Studies in Crop Variation II (1923) Fisher invents • Basic principles of experimental design • Control of variation by randomization • Analysis of variance

9. Studies in Crop Variation IV and VI Studies in Crop variation IV (1927) Fisher invents analysis of covariance to combine statistical control and control by randomization Studies in crop variation VI (1929) Fisher refines the theory of experimental design, introducing most other key concepts known today

10. Our Hero in 1929

11. Principles of Experimental Design Experimental design controls background variability so that systematic effects of treatments can be observed Three basic principles • Control by matching • Control by randomization • Control by statistical adjustment Their importance is in that order

12. Control by Matching Known sources of variation may be eliminated by matching Eliminating genetic variation Compare animals from the same litter of mice Eliminating district or school effects Compare students within districts or schools However matching is limited • matching is only possible on observable characteristics • perfect matching is not always possible • matching inherently limits generalizability by removing (possibly desired) variation

13. Control by Matching Matching ensures that groups compared are alike on specific known and observable characteristics (in principle, everything we have thought of) Wouldn’t it be great if there were a method of making groups alike on not only everything we have thought of, but everything we didn’t think of too? There is such a method

14. Control by Randomization Matching controls for the effects of variation due to specific observable characteristics Randomization controls for the effects all (observable or non-observable, known or unknown) characteristics Randomization makes groups equivalent (on average) on all variables (known and unknown, observable or not) Randomization also gives us a way to assess whether differences after treatment are larger than would be expected due to chance.

15. Control by Randomization Random assignment is not assignment with no particular rule. It is a purposeful process Assignment is made at random. This does not mean that the experimenter writes down the names of the varieties in any order that occurs to him, but that he carries out a physical experimental process of randomization, using means which shall ensure that each variety will have an equal chance of being tested on any particular plot of ground (Fisher, 1935, p. 51)

16. Control by Randomization Random assignment of schools or classrooms is not assignment with no particular rule. It is a purposeful process Assignment of schools to treatments is made at random. This does not mean that the experimenter assigns schools to treatments in any order that occurs to her, but that she carries out a physical experimental process of randomization, using means which shall ensure that each treatment will have an equal chance of being tested in any particular school (Hedges, 2007)

17. Control by Statistical Adjustment Control by statistical adjustment is a form of pseudo-matching It uses statistical relations to simulate matching Statistical control is important for increasing precision but should not be relied upon to control biases that may exist prior to assignment

18. Using Principles of Experimental Design You have to know a lot (be smart) to use matching and statistical control effectively You do not have to be smart to use randomization effectively But Where all are possible, randomization is not as efficient (requires larger sample sizes for the same power) as matching or statistical control

19. Basic Ideas of Design:Independent Variables (Factors) The values of independent variables are called levels Some independent variables can be manipulated, others can’t Treatments are independent variables that can be manipulated Blocks and covariates are independent variables that cannot be manipulated These concepts are simple, but are often confused Remember: You can randomly assign treatment levels but not blocks

20. Basic Ideas of Design (Crossing) Relations between independent variables Factors (treatments or blocks) are crossed if every level of one factor occurs with every level of another factor Example The Tennessee class size experiment assigned students to one of three class size conditions. All three treatment conditions occurred within each of the participating schools Thus treatment was crossed with schools

21. Basic Ideas of Design (Nesting) Factor B is nested in factor A if every level of factor B occurs within only one level of factor A Example The Tennessee class size experiment actually assigned classrooms to one of three class size conditions. Each classroom occurred in only one treatment condition Thus classrooms were nested within treatments (But treatment was crossed with schools)

22. Where Do These Terms Come From?(Nesting) An agricultural experiment where blocks are literally blocks or plots of land Here each block is literally nested within a treatment condition

23. Where Do These Terms Come From?(Crossing) An agricultural experiment Blocks were literally blocks of land and plots of land within blocks were assigned different treatments

24. Where Do These Terms Come From?(Crossing) Blocks were literally blocks of land and plots of land within blocks were assigned different treatments. Here treatment literally crosses the blocks

25. Where Do These Terms Come From?(Crossing) The experiment is often depicted like this. What is wrong with this as a field layout? Consider possible sources of bias

26. Think About These Designs A study assigns a reading treatment (or control) to children in 20 schools. Each child is classified into one of three groups with different risk of reading failure. A study assigns T or C to 20 teachers. The teachers are in five schools, and each teacher teaches 4 science classes Two schools in each district are picked to participate. Each school has two grade 4 teachers. One of them is assigned to T, the other to C.

27. Three Basic Designs The completely randomized design Treatments are assigned to individuals The randomized block design Treatments are assigned to individuals within blocks The hierarchical design Treatments are assigned to blocks, the same treatment is assigned to all individuals in the block

28. The Completely Randomized Design Individuals are randomly assigned to one of two treatments

29. The Randomized Block Design

30. The Hierarchical Design

31. Randomization Procedures Randomization has to be done as an explicit process devised by the experimenter • Haphazard is not the same as random • Unknown assignment is not the same as random • “Essentially random” is technically meaningless • Alternation is not random, even if you alternate from a random start This is why R.A. Fisher was so explicit about randomization processes

32. Randomization Procedures R.A. Fisher on how to randomize an experiment with small sample size and 5 treatments A satisfactory method is to use a pack of cards numbered from 1 to 100, and to arrange them in random order by repeated shuffling. The varieties [treatments] are numbered from 1 to 5, and any card such as the number 33, for example is deemed to correspond to variety [treatment] number 3, because on dividing by 5 this number is found as the remainder. (Fisher, 1935, p.51)

33. Randomization Procedures You may want to use a table of random numbers, but be sure to pick an arbitrary start point! Beware random number generators—they typically depend on seed values, be sure to vary the seed value (if they do not do it automatically) Otherwise you can reliably generate the same sequence of random numbers every time It is no different that starting in the same place in a table of random numbers

34. Randomization Procedures Completely Randomized Design (2 treatments, 2n individuals) Make a list of all individuals For each individual, pick a random number from 1 to 2 (odd or even) Assign the individual to treatment 1 if even, 2 if odd When one treatment is assigned n individuals, stop assigning more individuals to that treatment

35. Randomization Procedures Completely Randomized Design (2pn individuals, p treatments) Make a list of all individuals For each individual, pick a random number from 1 to p One way to do this is to get a random number of any size, divide by p, the remainder R is between 0 and (p – 1), so add 1 to the remainder to get R + 1 Assign the individual to treatment R + 1 Stop assigning individuals to any treatment after it gets n individuals

36. Randomization Procedures Randomized Block Design with 2 Treatments (m blocks per treatment, 2n individuals per block) Make a list of all individuals in the first block For each individual, pick a random number from 1 to 2 (odd or even) Assign the individual to treatment 1 if even, 2 if odd Stop assigning a treatment it is assigned n individuals in the block Repeat the same process with every block

37. Randomization Procedures Randomized Block Design with p Treatments (m blocks per treatment, pn individuals per block) Make a list of all individuals in the first block For each individual, pick a random number from 1 to p Assign the individual to treatment p Stop assigning a treatment it is assigned n individuals in the block Repeat the same process with every block

38. Randomization Procedures Hierarchical Design with 2 Treatments (m blocks per treatment, n individuals per block) Make a list of all blocks For each block, pick a random number from 1 to 2 Assign the block to treatment 1 if even, treatment 2 if odd Stop assigning a treatment after it is assigned m blocks Every individual in a block is assigned to the same treatment

39. Randomization Procedures Hierarchical Design with p Treatments (m blocks per treatment, n individuals per block) Make a list of all blocks For each block, pick a random number from 1 to p Assign the block to treatment corresponding to the number Stop assigning a treatment after it is assigned m blocks Every individual in a block is assigned to the same treatment

40. Sampling Models

41. Sampling Models in Educational Research Sampling models are often ignored in educational research But Sampling is where the randomness comes from in social research Sampling therefore has profound consequences for statistical analysis and research designs

42. Sampling Models in Educational Research Simple random samples are rare in field research Educational populations are hierarchically nested: • Students in classrooms in schools • Schools in districts in states We usually exploit the population structure to sample students by first sampling schools Even then, most samples are not probability samples, but they are intended to be representative (of some population)

43. Sampling Models in Educational Research Survey research calls this strategy multistage (multilevel) clustered sampling We often sample clusters (schools) first then individuals within clusters (students within schools) This is a two-stage (two-level) cluster sample We might sample schools, then classrooms, then students This is a three-stage (three-level) cluster sample

44. Precision of Estimates Depends on the Sampling Model Suppose the total population variance is σT2 and ICC is ρ Consider two samplesof size N = mn A simple random sample or stratified sample The variance of the mean is σT2/mn A clustered sample of n students from each of m schools The variance of the mean is (σT2/mn)[1 + (n – 1)ρ] The inflation factor [1 + (n – 1)ρ] is called the design effect

45. Precision of Estimates Depends on the Sampling Model Suppose the population variance is σT2 School level ICC is ρS,class level ICC isρC Consider two samplesof size N = mpn A simple random sample or stratified sample The variance of the mean is σT2/mpn A clustered sample of n students from p classes in m schools The variance is (σT2/mpn)[1 + (pn – 1)ρS + (n – 1)ρC] The three level design effect is [1 + (pn – 1)ρS + (n – 1)ρC]

46. Precision of Estimates Depends on the Sampling Model Treatment effects in experiments and quasi-experiments are mean differences Therefore precision of treatment effects and statistical power will depend on the sampling model

47. Sampling Models in Educational Research The fact that the population is structured does not mean the sample is must be a clustered sample Whether it is a clustered sample depends on: • How the sample is drawn (e.g., are schools sampled first then individuals randomly within schools) • What the inferential population is (e.g., is the inference these schools studied or a larger population of schools)

48. Sampling Models in Educational Research A necessary condition for a clustered sample is that it is drawn in stages using population subdivisions • schools then students within schools • schools then classrooms then students However, if all subdivisions in a population are present in the sample, the sample is not clustered, but stratified Stratification has different implications than clustering Whether there is stratification or clustering depends on the definition of the population to which we draw inferences (the inferential population)

49. Sampling Models in Educational Research The clustered/stratified distinction matters because it influences the precision of statistics estimated from the sample If all population subdivisions are included in the every sample, there is no sampling (or exhaustive sampling) of subdivisions • therefore differences between subdivisions add no uncertainty to estimates If only some population subdivisions are included in the sample, it matters which ones you happen to sample • thus differences between subdivisions add to uncertainty

50. Inferential Population and Inference Models The inferential population or inference model has implications for analysis and therefore for the design of experiments Do we make inferences to the schools in this sample or to a larger population of schools? Inferences to the schools or classes in the sample are called conditional inferences Inferences to a larger population of schools or classes are called unconditional inferences