Designing Experiments

Designing Experiments Section 5.2

What You’ll Learn • The difference between an observational study and an experiment • Why we use a randomized, comparative experiment • The four principles of experimental design • How to design experiments using • Completely Randomized Design • Randomized Block Design • Matched Pair Design • Repeated Measures Design

Observational vs Experiment • Remember that in an observational study the researcher simply observes and records the information.----NO treatment is applied, NO manipulation by the researcher occurs. • We are unable to infer “cause and effect” when doing observational studies

Observational vs Experiment • In an experiment the researcher applies some treatment or manipulates some factor. • When experiments are done properly, we are more likely to be able to infer a cause-and-effect relationship

Experiments • In this section we will learn how to properly design randomized, comparative experiments. • Researchers study the relationships between at least two variables. The explanatory variable, “factor”, is the variable the researcher manipulates. We observe what effect that manipulation has on the response variable

Comparative Experiments • In order to measure the effect of a treatment, it is desirable to have more than one group to compare results. • This may mean we are comparing two or more levels of a factor. • It may also mean that we are comparing a single level of a factor to a group which does not receive the treatment

Four Principles of Experimental Design • When designing these randomized, comparative experiments, we must remember to follow the four principles of design if we are to infer cause-and-effect relationships. • The next four slides will describe these principles

Control • Sources of variation in a variable can come from many places. In order to attribute the differences we see to the explanatory variable, we will attempt to make the conditions for all groups as similar as possible. We must be careful to identify any possible sources of variation besides the treatment and then control for them as best we can.

Randomize • Randomization is just as important in designing an experiment as it is in selecting a sample. • Randomizing which individuals get which treatments helps us to spread the variation between individuals out across the groups. Again it helps to make the groups as alike as possible to begin with, so differences we see in the response variable can be attributed to the explanatory variable.

Replication • There are two kinds of replications in comparative experiments • Applying the treatments to more than one individual allows us to estimate the variability of the responses since we wouldn’t expect exactly the same response from each individual • Replication of an entire study gives us confidence in our results as many experiments do not begin with a random sample from the population. This step allows us to generalize to the larger population with more confidence.

Blocking • Blocking, like stratification in sampling, allows us to identify a variable we know will have different results and control for that variable. • By placing all individuals in one group by the identified variable (for example---gender) we can then use randomization on the resulting groups to all levels of the treatment. (in other words, it is like we are running separate experiments on each gender)

Which principles do we use? • The first three principles: • Control • Randomization • Replication must be present in every experiment • The last principle may or may not be appropriate depending on the situation.

Experiments • In the last section we learned that in order to make inferences about populations we must use randomness when choosing the individuals for our samples • Because of ethics, this is not always possible when we perform experiments, especially on living beings • So how do we ensure that our results will generalize to a larger population?

Inference to a Larger Population • In order to generalize the results from an experiment to individuals beyond our participants, we design the experiments using the four principles of experimental design. • Let’s look at how the four different designs we will consider might look.

Migraines • A pharmaceutical company has developed a new medicine that they believe will be more effective in treating people who suffer from migraine headaches. Researchers plan to enlist several people who suffer from migraines in a test. • Let’s see how we could design the four different types of experiments.

Designing the Experiment • When beginning the design phase, it is very useful to identify the following: • The subjects studied • The factors in the experiment, and the number of levels for each • The number of treatments • The response variable to be measured • Any variables we wish to use blocking to control

Migraines • Subjects: individuals who suffer from migraine headaches • Factor(s): Medication • Level(s): 2—new and old medication • Treatment(s):2—new and old meds. • Response Var.: level of pain relief • Blocking Var.: Possibly gender

Completely Randomized Design New Medication-50 subjects Compare level of pain relief as reported by subjects Random Assignment 100 Subjects Old Medication-50 subjects Along with the diagram, you need to describe how the first three principles of experimental design have been accomplished----see next slide!

Completely Randomized Design • Control • Each group will be given instructions as to the amount of medicine, timeframe for taking the medicine, timeframe for reporting pain relief, and general instructions. • Randomization • To accomplish the random assignment, each subject as they enter the facility will be assigned a random number between 001 and 100, those with numbers 001-050 will be assigned to the new medication. The remaining subjects will be assigned to the old medication. • Replication • We have included more than one individual in each treatment group.

Randomized Block Design • In previous research it is known that women and men respond differently to medication, and report pain and pain relief differently. • To control for this variable, you decide to block for this variable.

Randomized Block Design New Medication- 25 subjects Compare level of pain relief as reported by subjects 50 Women Random Assignment Old Medication-25 subjects 100 Subjects Block by Gender New Medication- 25 subjects Compare level of pain relief as reported by subjects 50 Men Random Assignment Old Medication-25 subjects When we have equal number of subjects in each group this is known as a balanced design. It is always desirable but not always possible. See next slide for written description of the design

Randomized Block Design • Blocking: • We have identified a variable we know may be responsible for differences we see in pain relief, so we divided our group of participants into groups by gender which created two groups in which the individuals were alike with respect to gender. • Control: • Direct control can be accomplished in the same way we set up in the completely randomized design. • Randomization: • We need to number the women subjects as they arrive with a number between 01-50, those with numbers between 01-25 will take the new medication, the rest will get the old medication. • Repeat the above randomization method with the group of men • Replication: • Again, we have applied the treatments to more than one subject

Matched Pairs Design • In a matched pair design, we will try to match two subjects on a variety of variables. The more alike our two subjects are to begin with, the better chance we have of attributing differences we see to the treatment. • So, how do we accomplish the principles of design? Read on.

Matched Pairs • Blocking: • By matching pairs of subjects with respect to identified variables (maybe gender, age, how long they have suffered with migraines, ect.) we will have created groups (blocks) of two. This means that a matched pair design is really a form of blocking. • Control: • Direct control will again be accomplished as in the previous two designs. We want to make sure that the conditions are the same for each subject within the pair. • Randomization: • We can randomly assign which subject within each pair will receive the new medication in a variety of ways: toss a coin, generate a random number between 0-9, with odds getting the new medication, ect. • Replication: • Replication happens because of the way in which we report our response variable. Instead of finding the average pain relief level of all those who took the new medication, we first find the difference in pain relief between each pair (Old-new or New-old). We can then look at the average difference in pain relief. If this average is zero or close to zero then we would conclude that the new medication does not provide better treatment. (we’ll find out how far from zero this difference must be to be significant in second semester)

Repeated Measures Design • A repeated measures design is a form of matched pair design where each subject serves as their own control. • In other words, every subject will take both the new medication and the old medication and our response variable will again be the difference in pain relief for each subject. • So let’s see how the principles are accomplished.

Repeated Measures Design • Blocking: • We create “blocks” by having each subject serve as their own control and apply both treatments to each subject • Control: • Direct control will be accomplished by advising the subjects to follow all the directions during both treatments. In this situation we may have subjects wait for a designated period of time before starting the second treatment in order to minimize any residual effects from the first treatment. • Randomization: • Instead of randomly assigning subjects to treatments, we will randomly assign which medication the subject will take first. This randomization needs to be done separately for each subject. • Replication: • Replication happens because of the way in which we report our response variable. Instead of finding the average pain relief level of all those who took the new medication, we first find the difference in pain relief for each subject (Old-new or New-old). We can then look at the average difference in pain relief. If this average is zero or close to zero then we would conclude that the new medication does not provide better treatment. (we’ll find out how far from zero this difference must be to be significant in second semester)

Describing the Designs • Although diagrams can be used with any of the four designs, there should always be a written narrative describing how the principles have been included in the design. Of particular importance is the description of how randomization has been accomplished.

To Tell or Not To Tell • Informed consent to be part of an experiment is very important in any kind of experiment involving humans, however in order to obtain results that are meaningful it is also necessary for us to minimize the effects that knowledge of the study may have on our response variable. • So, how do we stay ethical and obtain valuable results? Using a technique we call blinding will allow us to accomplish this.

Blinding • There are two main classifications of people who can affect the outcome of an experiment. • Those who could influence the results (subjects, administrators of treatments, etc) • Those who evaluate the results • When all the individuals in either category are unaware of which group each subject has been assigned, the study is considered “blind” • When the individuals from both groups are unaware of which group each subject has been assigned the study is said to be “double-blind” • We could have made any of the designs we illustrated “double-blind”

Informed Consent • So how does a blind or double-blind experiment address the issue of informed consent? • A subject should have know that they are a part of an experiment, and that the possibility exists for them to be assigned to one of several groups. This way the subject can either choose to participate or not. What they won’t be told is which group they will be assigned.

Additional Resources • The Practice of Statistics—YMM • Pg 265-284 • The Practice of Statistics—YMS • Pg 290-307 • Against All Odds—Video #13 • http://www.learner.org/resources/series65.html

Designing Experiments