Experimental Design

STAT E-150Statistical Methods Experimental Design

The design of an experiment typically takes place before the data is collected. It is the design of the experiment that determines the model. • Some basic vocabulary: • The individuals used in the experiment are called experimental units. When the units are human beings, they are called subjects. A specific experimental condition applied to the units is called a treatment. • The explanatory variables in the experiment are often called factors. A specific value of a factor is called a level of the factor. • A response variable is a measure of the outcome of the experiment.

The design of an experiment first describes the response variable(s), the factor(s), and the layout of the treatments. Here is an example: What are the effects of repeated exposure to an advertising message? In an experiment designed to investigate this question, undergraduate students viewed a 40-minute television program that included ads for a digital camera. Some students saw a 30-second commercial; others saw a 90-second version. The same commercial was shown either one, three, or five times during the program. After viewing, all of the students answered questions about their recall of the ad, their attitude toward the digital camera, and their intention to purchase it.

What are the effects of repeated exposure to an advertising message? In an experiment designed to investigate this question, undergraduate students viewed a 40-minute television program that included ads for a digital camera. Some students saw a 30-second commercial; others saw a 90-second version. The same commercial was shown either one, three, or five times during the program. After viewing, all of the students answered questions about their recall of the ad, their attitude toward the digital camera, and their intention to purchase it. Who were the subjects?

What are the effects of repeated exposure to an advertising message? In an experiment designed to investigate this question, undergraduate students viewed a 40-minute television program that included ads for a digital camera. Some students saw a 30-second commercial; others saw a 90-second version. The same commercial was shown either one, three, or five times during the program. After viewing, all of the students answered questions about their recall of the ad, their attitude toward the digital camera, and their intention to purchase it. Who were the subjects? The subjects were undergraduate students

What are the effects of repeated exposure to an advertising message? In an experiment designed to investigate this question, undergraduate students viewed a 40-minute television program that included ads for a digital camera. Some students saw a 30-second commercial; others saw a 90-second version. The same commercial was shown either one, three, or five times during the program. After viewing, all of the students answered questions about their recall of the ad, their attitude toward the digital camera, and their intention to purchase it. What are the factors of the experiment?

What are the effects of repeated exposure to an advertising message? In an experiment designed to investigate this question, undergraduate students viewed a 40-minute television program that included ads for a digital camera. Some students saw a 30-second commercial; others saw a 90-second version. The same commercial was shown either one, three, or five times during the program. After viewing, all of the students answered questions about their recall of the ad, their attitude toward the digital camera, and their intention to purchase it. What are the factors of the experiment? The factors are the length of the commercial and the number of repetitions.

What are the effects of repeated exposure to an advertising message? In an experiment designed to investigate this question, undergraduate students viewed a 40-minute television program that included ads for a digital camera. Some students saw a 30-second commercial; others saw a 90-second version. The same commercial was shown either one, three, or five times during the program. After viewing, all of the students answered questions about their recall of the ad, their attitude toward the digital camera, and their intention to purchase it. What are the levels of each factor?

What are the effects of repeated exposure to an advertising message? In an experiment designed to investigate this question, undergraduate students viewed a 40-minute television program that included ads for a digital camera. Some students saw a 30-second commercial; others saw a 90-second version. The same commercial was shown either one, three, or five times during the program. After viewing, all of the students answered questions about their recall of the ad, their attitude toward the digital camera, and their intention to purchase it. What are the levels of each factor? The length of the commercial has two levels: 30 seconds and 90 seconds. The number of repetitions has three levels: 1, 3, and 5 repetitions.

What are the effects of repeated exposure to an advertising message? In an experiment designed to investigate this question, undergraduate students viewed a 40-minute television program that included ads for a digital camera. Some students saw a 30-second commercial; others saw a 90-second version. The same commercial was shown either one, three, or five times during the program. After viewing, all of the students answered questions about their recall of the ad, their attitude toward the digital camera, and their intention to purchase it. What is (are) the response variable(s)?

What are the effects of repeated exposure to an advertising message? In an experiment designed to investigate this question, undergraduate students viewed a 40-minute television program that included ads for a digital camera. Some students saw a 30-second commercial; others saw a 90-second version. The same commercial was shown either one, three, or five times during the program. After viewing, all of the students answered questions about their recall of the ad, their attitude toward the digital camera, and their intention to purchase it. What is (are) the response variable(s)? The response variables are the recall of the ad, the attitude toward the camera, and the intention to purchase the camera.

Here is a diagram of this experimental design:

The second step of the design is to define how experimental units will be assigned to treatments. Comparison of the effects of treatments can only be valid if the treatments are applied to similar groups of experimental units. This requires at least two groups; one is often a control group.

The most important principle of statistical design is randomization, which involves using chance to select the sample from the population and to assign subjects to treatment. This assignment, then, does not depend on any characteristic of the subjects and does not rely on the judgment of the experimenter in any way. This protects against bias which favors certain outcomes, allows us to draw cause-and-effect conclusions, and provides a justification for using a probability model.

Does talking on a hands-free cell phone distract drivers? Undergraduate students "drove" in a driving simulator equipped with a hands-free cell phone. How quickly does the student respond when the car ahead brakes suddenly? Twenty students (the control group) simply drove. Another twenty students (the experimental group) talked on the phone while driving.

Here is a diagram of the design of this experiment: Randomization produces groups of subjects that we expect to be similar in all respects before the treatment is applied.

Comparative design (comparing the effects of several treatments) helps ensure that influences other than the treatment (in this case, the cell phone) operate equally on all groups. Any differences, then, must be due either to the treatment or to chance (i.e. the random assignment). Recall that a small p-value is an indication that it is unlikely that the results we see are due to chance alone.

Randomization F-Test (One-Way ANOVA) 1. Observed F: Compute the value of F as usual. (FObs) 2. Randomization distribution: a. Rerandomize = create a random reordering of the data. b. Compute the value of F for the rerandomized data (FRand) c. Repeat this many times and record the values of FRand 3. Find the proportion of values of FRand that are greater than or equal to FObs This proportion is an estimate of the p-value. Both the F-test and the Randomization F-test use a quantitative response variable and one categorical predictor, and both test whether the observed differences in group means are too large to be due to chance alone.

Repeated Measures Analysis of Variance In this analysis, the groups are not independent. Instead, we have one sample of subjects and take multiple, or repeated, measurements on each subject.

Here’s an example: Consider this data for four subjects over three treatments, where the response variable is number of trials to success on some task: If you look at the treatment means, you see a very slight difference. But look also at the subject means: It is apparent that Subject 1 learns quickly under all conditions, and that Subjects 3 and 4 learn very slowly. These differences among the subjects are responsible for the differences within treatments, but they really have nothing to do with the treatment effect.

A repeated measure is a multivariate response in which the same variable is measured on each subject several times, possibly under different conditions. The hypotheses are the same as they are for a one-way ANOVA. The analysis is also the same: first a test for the equality of the means, and then appropriate tests to investigate any differences that are found.

The assumptions are: • Independent observations within each treatment • Normal populations within each treatment • Equal population variances within each treatment • Sphericity • Note that there is an additional assumption of sphericity, which assumes that the pairwise differences among treatment levels have equal variances. Mauchly's test for sphericity is commonly used to test this assumption.

Example 1: A study was conducted to examine whether the anxiety a person experiences affects performance on a learning task. Subjects with varying levels of anxiety performed a learning task across a series of four trials and the number of errors made was recorded. Does the number of errors made by subjects change significantly across the four learning trials? H0: μtrial1= μtrial2 = μtrial3 = μtrial4 Ha: the means are not all equal

We will use statistical tests to assess whether the assumptions are met. First, we can use the Kolmogorov-Smirnov results in the Tests of Normality table to assess normality as we did earlier; since all p-values are greater than .05, we do not reject the null hypothesis of normality.

We will use Mauchly's test to test the hypothesis that the variances of the differences between conditions are equal. Since p = .112 we cannot reject the null hypothesis, and can conclude that there are no significant differences between the variances of the differences. We now know that the condition of sphericity is met. Since the conditions for the test have been met, we can continue.

We will consider the results in the first row of this table, where sphericity is assumed. The results (F = 127.561, p = .000) indicate that we can reject the null hypothesis and conclude that the number of errors made by subjects did change significantly across the four learning trials.

Example 2: • This data was collected in a study investigating factors associated with the risk of developing high blood pressure, or hypertension. The subjects were all college students. We would like to see if there is evidence that diastolic blood pressure changes significantly during three different conditions: resting, doing mental arithmetic, and immersing a hand in cold water. The null hypothesis is that blood pressure does not change under these stressors: • H0: μrest = μarith = μcold • Ha: the means are not all equal

First we can see what the descriptive statistics for this data suggest. It appears that the means are different for the three stressors, but we will need to analyze the data to see if the difference is significant.

First we will check the conditions for this test. We can use the results in the Tests of Normality table to assess normality as we did earlier; since p = .200, we do not reject the null hypothesis of normality.

However, the results of Mauchly's test are significant (p is close to 0), and we conclude that the condition of sphericity is not met. However, we can still continue with this analysis even when the assumption of sphericity is violated.

The table below includes additional information that can be used in this situation: You can use these results if the sphericity assumption is met, as we did in the first example; however, this is not the case in this analysis. The next lines show results for other tests where adjustments to the degrees of freedom have been made because the sphericity assumption was violated.

The table below includes additional information that can be used in this situation: The Greenhouse-Geisser test is commonly used in this situation. Those results are significant (F = 73.754, p = .000) and so we reject the null hypothesis. We can conclude that this indicates that diastolic blood pressure changes significantly during the various mental and physical stressors investigated in this study.

But where are these differences? We can compare the means using the Bonferroni method: These results indicate that there are no significant differences between the stressors when compared pairwise. All p-values are reported as .000.

You may also choose to create a graph that illustrates and supports the analysis. This graph supports the conclusion that the means of the three groups are different:

Another way to investigate differences between pairs of groups is by performing appropriate paired-sample t-tests. For example, you can compare the subjects' diastolic blood pressure in two of the conditions: resting, and after immersing a hand in cold water. Note that in this test, we are testing the mean difference between the groups, not the difference of the means. Paired sample t-tests are used to compare two means when the samples are not independent, so that a value in one sample can be paired with a corresponding value in the second sample. For example, a subject's resting diastolic blood pressure can be paired with the same subject's diastolic blood pressure after submerging a hand in cold water.

Assumptions and Conditions Paired data assumption: The data must be paired. Independence assumption: The groups are not independent, but the differencesare independent. Randomization condition: Pairs are randomly chosen, or treatments are assigned randomly to the members of each pair Normal population assumption: The population of differences is nearly normal (Check with a histogram and/or a Normal probability plot of the differences.)

When the conditions are met and the null hypothesis is true we can model the sampling distribution of this statistic with a Student’s t-model with n-1 degrees of freedom, where n is the number of pairs. We test the hypothesis H0: μd =Δ0where the d’s are pairwise differences, and Δ0 is almost always 0.

H0: μd = 0 H0: μd ≠ 0 Here are the results: We will reject the null hypothesis (t = -10.590, p = .000) and conclude that there is a difference between resting diastolic blood pressure and diastolic blood pressure after immersing a hand in cold water.

SPSS Instructions for Repeated Measures ANOVA 1. Choose > Analyze > Descriptive Statistics > Explore In the Explore dialog box, choose the columns you wish to include in your analysis, and choose Plots as the display. Then click on Plots.

In the Explore Plots dialog box, select Normality plots with tests, deselect Stem-and-leaf and select None under Boxplots. Click on Continue and then click on OK in the main dialog box.

2. Choose > Analyze > General Linear Model > Repeated Measures Enter an appropriate Within-Subject Factor name; in this case it is stressor. Type in the number of levels, then click Add. You will see the results in the next box.

2. Choose > Analyze > General Linear Model > Repeated Measures Now click on Define. You will see the Repeated Measures dialog box like the one shown on the left. Select the columns containing data for the factor levels in order by highlighting the column name and clicking on the arrow that will move the name to the list of variables. Then click on Plots.

2. Choose > Analyze > General Linear Model > Repeated Measures In the Repeated Measures: Profile Plots dialog box, click on the factor and move it to the Horizontal Axis. Then click on Add to add this selection to the Plots: box. Click on Continue.

SPSS Instructions for Paired Sample t-tests Choose > Analyze > Compare Means > Paired-Samples t-test Indicate the columns you wish to compare, and then click on OK:

Experimental Design