Experimental Design & Analysis

1. Experimental Design & Analysis Analysis of Covariance; Within-Subject Designs March 13, 2007 DOCTORAL SEMINAR, SPRING SEMESTER 2007

2. Outline • Blocking vs. analysis of covariance • ANCOVA • Within-subject designs • Greenwald

3. Randomized Block Design • Completely randomized designs based on assumption that random assignment diminishes systematic bias • Blocks are experimental counterparts of covariates • Identify “nuisance” variable • Randomly assign within blocks

4. Randomized Block Design • Example: Experiment about effect of 4 types of training programs on learning • Completely randomized design: N=60, a=4 levels, randomly assign 15 subjects to each of 4 levels • Randomized block design: assume intelligence could confound results, categorize subjects according to IQ or GPA (high, medium, low), assign 20 to each “block” and randomly assign within each block to one of 4 conditions

5. Randomized Block Design • Code block as a factor • Do not want treatment x block interaction • More blocks require more subjects

6. Analysis of Covariance • The main purpose of ANCOVA is statistical control of variability when experimental control can not be used • Statistical method for increasing power of ANOVA by reducing mean square error (within-condition error) • ANCOVA can be used to correct for extraneous variables and rule out rival explanations

7. Analysis of Covariance • ANCOVA is like ANOVA on the residuals of  the values of the dependent variable, after removing the influence of the covariate, rather than on the original values themselves • In so far as the measures of the covariate are taken in advance of the experiment and they correlate with the measures of the dependent variable they can be used to reduce experimental error • Based on partial correlation analysis

8. Analysis of Covariance • Examples of covariates • If we have a theory about two different methods of learning and our dependent variable is memory, level of education, intelligence, IQ, experience and age may all be covariates • If we have a theory about the effect of management style on firm profitability, some covariates may be firm size, competition, length of CEO tenure • If we have a theory about the relationship between number of outstanding shares and share price, we may consider market capitalization, earnings per share and IPO year as covariates • If we have a theory about the interactive effects of advertising and sales force training, some covariates to consider may be brand equity of product, size of sales force and past advertising

9. Analysis of Covariance • One way of understanding ANCOVA is in terms of deviations from a common regression line • If the regression lines of the dependent variable vs. covariate have same slopes but different intercepts, the effect of the treatment is consistent • If the regression lines of the dependent variable vs. covariate have different slopes, the effect of the treatment depends on the value of the covariate

10. Analysis of Covariance • ANCOVA is based on the sums of squares and sums of products • It is used to test the same hypothesis as standard ANOVA • The only difference is that each of the sums of squares are adjusted on the basis of the covariate variable • The covariate reduces the amount of experimental error • In calculations, reduce degree of freedom by 1 for each covariate

11. ANCOVA Assumptions • Dependent variable continuous • Covariate may be continuous (like regression) or discrete (like ANOVA) • Factor has discrete levels • Variables are normally distributed • Relationship is linear

12. ANCOVA • Multiple covariates possible • Should be theoretically motivated • How much variance does covariate account for?

13. Within-Subject Designs • Advantages of within-subject designs • Economy • Power • In fully crossed designs, random assignment permits the assumption of equivalence (subject comparability). In within-subject designs, subjects are the same, thus removing source of unwanted variability and reducing the error term • Usefulness • Allows study of behavioral/attitudinal change and learning

14. Within-Subjects Designs • Within-subjects designs involve applying all treatments to the same individuals • Think about within-subjects designs as a 2-way ANOVA where the columns are the treatments and the rows the individuals, with one observation per cell • This design introduces the idea of individuals as a random factor that crosses or intersects other factors in the design • Within-subjects, or repeated measures design, has advantages and disadvantages • Advantage is that each individual serves as his or her own control, thus a source on experimental error resulting from individual difference is controlled for • Disadvantage is that it introduces dependencies across the treatments, known as carryover effects

15. For a 2x2 fully-crossed factorial N=40 (different subjects in each cell) For a 2x2 within-subject design N=10 (same 10 subjects in each cell) a2 a2 a1 a1 n=10 n=10 n=10 n=10 b1 b1 b2 b2 n=10 n=10 n=10 n=10 Within-Subject Designs: Economy

16. Within-Subject Designs: Power • Consider example in which 1 factor (A) is manipulated • Because factor A is completely crossed with subjects factor S, we denote as AxS • Analogous to two-factor design AxB in which variability is SStotal = SSA + SSB + SSAB + SSerror • Examine claim of greater power • SSerror = SStotal – SSeffects • In one-factor example: SSerror = SStotal – SSA - SSS

17. Within-Subject Designs • Consider complete within-subjects 2x2 design (each subject sees a1b1, a1b2, a2b1, a2b2 conditions) • Sources of variability: A, B, S, AxB, AxS, BxS, AxBxS • To test effects: compare mean square of effects (A, B, AxB) with mean square for effects with subjects (MSAxS, MSBxS, MSAxBxS)

18. Within-Subjects Designs • Example • Let’s say that we are interested in the effect of different types of exercise on memory • We use two treatments, aerobic exercise and anaerobic exercise • In the aerobic condition, participants run in place for five minutes, after which they take a memory test • In the anaerobic condition they lift weights for five minutes, after which they take a different memory test of equivalent difficulty • In a within-subjects design all participants begin by running in place and taking the test, after which the same group of people lift weights and then take the test • We compare the memory test scores in order to answer the question as to what type of exercise most aids memory