Chapter 8 Experimental Design

1. Chapter 8Experimental Design RCS 6740 Supplemental

2. Refresher of the Experimental Method • Experimental Method: A method of determining whether variables are related in which the researcher manipulates the INDEPENDENT variable and controls all other variables (DEPENDENT) either by randomization or by direct experimental control (Cozby, 2004).

3. Refresher of the Experimental Method Random Sampling vs. Random Assignment • Random Sampling: Using a random table of numbers, a researcher selects people from a sample to become participants in a study • Random Assignment: Participants (already chosen using random sampling) are assigned to groups using a random table of numbers

4. Refresher of the Experimental Method Cont. • So, a true experimental study needs: • Randomization and • Control • Independent Variable: The variable that is manipulated to observe its effect on the dependent variable. • Dependent Variable: The variable that is the subject’s response to, and dependent on, the level of the manipulated independent variable

5. Refresher of the Experimental Method Cont. • Example: To measure the effects of crowding (Independent variable) on cognitive performance (Dependent variable), participants are randomly assigned to one of two groups: • Group 1 takes a test in a crowded room • Group 2 takes a test alone in a room Is this a good example of the experimental method?

6. Refresher of the Experimental Method Cont. • Yes, if you accounted for and minimized other differences between the groups! • Confounding Variable: A variable that is not controlled in an experiment and that can effect the dependent variable. • So, from the study on crowding and cognitive performance, what are possible confounding variables?

7. Refresher of the Experimental Method Cont. Possible Confounding Variables in the Crowding Experiment • Groups took tests in different rooms, one with a window, one without • One group took an algebra test, the other took a statistics test • One group took the test at 7:30 AM, the other at 7:30 PM • One group was proctored by Dr. Spitznagel, the other by Snoop Dog

8. Refresher of the Experimental Method Cont. • Good experimental designs involve eliminating confounding variables that may result in alternative explanations • When results of an experiment can confidently be attributed to the effect of the independent variable on the dependent variable, a study is said to have good Internal Validity. • Therefore, researchers need to design experiments that account for an minimize possible alternative explanations

9. Types of Experimental Design • Basic Experiments • Posttest only Design • Pretest-Posttest Design • Independent Groups Design • Repeated Measures Design • Counterbalancing • Matched Pairs Design • Developmental Research Designs • Cross-Sectional Design • Longitudinal Method • Sequential Method

10. Coding for Lecture • To simplify the designs, the following coding system will be used throughout the remainder of this lecture • O = observation or measure • X = experimental intervention or manipulation • R = random assignment to groups

11. Basic Experiments • The most basic experimental design has two variables • Independent Variable • Dependent Variable • The independent variable has two Levels • Experimental Group (Usually receives treatment) • Control Group (Usually does not receive treatment) • A study can also have two different amounts of an independent variable • 10 mg of Prozac for one group and 20 mg of Prozac for another group • Example: A Randomized and Controlled study looking at the effects of exercise (Independent) on body fat (Dependent) • Group 1 exercises 3 times a week for 6 weeks • Group 2 does not exercise at all for three weeks Researchers will compare the body fat of those who exercise to those who do not. The question is, when should body fat be measured?

12. Basic Experiments: Posttest Only Posttest Only Design • A researcher using a Posttest only design must: • Obtain two equivalent groups* • Introduce the independent variable • Measure the effect of the independent variable on the dependent variable * To eliminate selection differences among participants, a researcher needs to either randomly assign participants to conditions or have the same participants participate in both conditions

13. Basic Experiments: Posttest Only (R) X O (R) O (R) X (Exercise/IV) O (Body Fat Measure/DV) (R) O (Body Fat Measure/DV) • Upon completion of all interventions, the researcher will measure the effects of the independent variable on the dependent variable to look for statistically significant differences

14. Basic Experiments: Posttest Only • Posttest only design using our exercise example: • Participants are randomly assigned to one of two groups. Group 1 exercises 3 times a week for 6 weeks and Group 2 does not exercise for 6 weeks. After 6 weeks, both groups will undergo a test to gauge their body fat. It was found that people in the exercise group had 5% less body fat than the people who did not exercise. Results of this study are statistically significant.

15. Basic Experiments: Posttest Only Advantages of the Posttest Only Design • Less expensive and time consuming than Pretest-Posttest • Does not sensitize participants to what you are studying • Pretests are not usually given in the real world so this design provides the researcher with the most confidence for findings generalization • In this design the possibility of the interaction of the pretest effects and the independent variable are not present, therefore that source as a potential threat to external validity is eliminated

16. Basic Experiments: Posttest Only Disadvantages of Posttest Only Design • Does not enable a researcher to account for mortality • Mortality: People dropping out in studies • Smoking Study example • Does not establish a baseline • Does not give a researcher before and after data to analyze statistically

17. Basic Experiments: Pretest-Posttest Design • With a Pretest-Posttest design, a researcher must: • Obtain two equivalent groups* • Introduce the independent variable • Measure the effect of the independent variable on the dependent variable * A larger sample size increases the likelihood that the that the groups will differ. A good rule of thumb is 20 to 30 people per condition

18. Basic Experiments: Pretest-Posttest Design (R) O X O (R) O O • Participants are randomly assigned to one of two groups. Group 1 exercises 3 times a week for 6 weeks and Group 2 does not exercise for 6 weeks. Before the study begins, a measure of body fat is taken from participants in each group. After 6 weeks, both groups will undergo the same body fat test. It was found that the people in the exercise group reduced their body fat (pretest to posttest) by a mean of 10%. The people in the non-exercise group reduced their body fat by a mean of 1%. Results of this study are statistically significant.

19. Basic Experiments: Pretest-Posttest Design Advantages of the Pretest-Posttest Design • Allows a researcher to account for mortality • Establishes a baseline • Enables a researcher to do before and after analyses of data • Allows the researcher to establish whether or not groups were random/equal to begin with • Allows researcher the opportunity to select participants for a study • Matching (discussed in detail later in lecture) • Smoking Example: I smoke regularly but answers only 1 a day on pretest

20. Basic Experiments: Pretest-Posttest Design Disadvantages of the Pretest-Posttest Design • More costly and time consuming than Posttest only design • May sensitize participants to the study • A researcher can disguise the pretest by: • Having a different experimenter give it in a different setting • Embed the pretest with a few irrelevant measures • Does not allow for confidence in generalizing results as pretests are not given in the real world

21. Combination of Pretest only and Pretest-Posttest Designs • The Solomon Four Group Design combines pretest-posttest and posttest only designs • 1/2 participants receive pretest and posttest • 1/2 participants receive only posttest • Randomly assigned • To examine and control for effects of pre-test • If there is no impact of the pretest, posttest scores for both groups (pre and pre/post) should be equivalent

22. Solomon Four Group Design: No Pretest Effect (Posttest Scores)

23. Solomon Four Group Design: Pretest Effect (Posttest Scores)

24. Assigning Participants to Experimental Conditions • Independent Groups Design • Participants are randomly assigned to various conditions so that each participates in only one group • Repeated Measures Design • Participants participate in all conditions and are eventually assigned to all levels of the Independent Variable

25. Independent Groups Design • Participants are randomly assigned to each condition • Example: Using our exercise and body fat study, participants are randomly assigned to one of two groups • Group 1 exercises 3 times a week for 6 weeks • Group 2 does not exercise for 6 weeks

26. Repeated Measures Design • Participants participate in all groups/conditions and are measured repeatedly on the dependent variable after each condition • Example: To study the efficacy of Speech and Language Pathology treatments on Aphasia, each participant is assigned to 3 groups: Melodic Intonation, Errorless Traditional, and Errorless Mapping. After completion of each treatment, the participants will take the Jackson Speech Assessment Inventory. • Remember, each participant participates in each group

27. Repeated Measures Design Advantages of the Repeated Measures Design • Fewer Participants are needed • Saves time and money • Repeated Measure Designs are extremely sensitive to statistical differences (significant) that are not as evident with the Independent Groups Design • Allows individual differences and random error to be seen

28. Repeated Measures Design

29. Repeated Measures Design Disadvantages of the Repeated Measures Design • Different conditions must be presented in a particular sequence • Order Effect: The order of treatments may effect the dependent variable • Example: In our speech study, lets say that everyone received Melodic Intonation first. Most of the participants improved on the Jackson Speech Assessment Inventory after the other two treatments (ELT & ELM) due to a technique learned from Melodic Intonation. Can you attribute this gain to the effects of ELT or ELM?

30. Repeated Measures Design Types of Order Effects • Practice Effect • Improvement in performance as a result of repeated practice • Fatigue Effect • Drop in performance due to being tired, bored, or distracted • Contrast Effect • Responses to the second condition of an experiment are altered because both conditions are contrasted to one another • Crime Study Example • Seeing a Mild Crime First and then Seeing a Severe Crime Second • Seeing a Severe Crime First and then seeing a Mild Crime Second Basically, seeing one first will influence how you perceive the other

31. Repeated Measures Design • There are two way to account for and minimize order effects • Employ Counter Balancing Techniques • Ensure that adequate time elapses between conditions

32. Counterbalancing • Counterbalancing entails including all possible orders of treatment/conditions in an experiment • It helps researchers identify any order effects • Example: Lets use our speech study and focus on ELM and ELT…

33. Counterbalancing

34. Counterbalancing • Counterbalancing can be extended to experiments with three or more groups • What if the number of possible orders becomes too big? • One study has 3,628,800 orders! A Latin Square controls for order effects without having all possible orders

35. Latin Square • A Latin Square is a limited set of orders constructed to ensure that: • Each condition appears at each ordinal position • Each condition precedes and follows each condition one time • For experiments with an extreme number of orders, a researcher may choose to randomly select a few orders to construct a Latin Square

36. Latin Square (3 Groups, 3 Treatments)

37. Randomized Blocks • Some Repeated Measures Design experiments with multiple treatment orders are repeated over and over again to look for effects • Example: Assigning participants a “lucky number” and seeing what mood it puts them in. • Each repetition of the basic experiment (i.e. assigning of the first lucky number) is called a Block of Trials • To control for order effects, the assigning of the numbers should be done in a random manner

38. Time Interval Between Treatments • In addition to counterbalancing, researchers need to carefully control how much time elapses between treatments/interventions • Examples: • Giving a sufficient rest period after a month of therapy • Allowing enough time to let a drug wear off • Also, ensuring that too much time does not elapse as participants may become upset and drop out

39. When to Choose an Independent Groups Design or a Repeated Measures Design • Basically, time, money, and number of participants will affect your choice • Also, it will depend on how your study generalizes to the “real world” • Example: If you want to look at how characteristics of a defendant affect a juror, you would probably use an Independent Groups Design (jurors focus on only one defendant) • If you wanted to see the effects of job applicants on employers, you would probably use the Repeated Measures Design (employers interview multiple applicants)

40. Matched Pairs Design • The Matched Pairs design is a way of assigning participants to groups by using, and focusing on, certain participant characteristics (Matching Variable) • The goal is to achieve the same equivalency of groups as the Repeated Measures Design without having the same participants in multiple conditions

41. Matched Pairs Design Steps of the Matched Pairs Design • Obtain a measure of the matching variable from each participant • Example: IQ scores • Rank the participants from highest to lowest on the matching variable • Example: (136, 126, 118, 118, 103, 101) • Match pairs that are approximately equal on the matching variable • Example: (136 and 126, 118 and 118, 103 and 101) • Finally, randomly assign members of each pair to one of two treatment groups • Example: 136 goes to ELT and 126 goes to ELM, 118 goes to ELM and 118 goes to ELT, 103 goes to ELT and 101 goes to ELM

42. Analysis of Covariance • Analysis of Covariance is another way to statistically control the correlation between a subject variable (e.g. IQ scores) and the dependent variable. • Basically, information on the subject variable is collected after the fact and analyzed statistically to determine if it is causing any effect on the dependent variable

43. Developmental Research Designs • Cross-Sectional Method • Longitudinal Method • Sequential Method

44. Cross-Sectional Method • In the cross-section method individuals at different ages are studied at one point in time on the dependent variable. • Health care use (DV) and age (IV) • Example: Health care use of a 10, 15, 20, 30, 45, and 60 year old person are examined at the same time • This method saves time, but uses different participants to study the effects of the independent variable (aging). • Beware of the Cohort Effect where change may be based on comparisons between cohorts (same age group of people) rather than the developmental change of age • Did your Grandparents have a Playstation?

45. Longitudinal Method • The longitudinal method studies the same individuals over time and take measurement on the dependent variable at specified times. • Example: Health care use of a group of 10 year olds is gauged. The same group will have their health care use gauged at 15, 20, 25, and 30 years of age Good design but people drop out, move, and pass away

46. Sequential Method • The sequential method is a mixed method beginning with the cross-sectional method and then using the longitudinal method for at least one more set of measurements on the dependent variable • Example: Health care use of 55 and 65 year olds are gauged. Then, 5 years later, both groups’ use of health care will be gauged again

47. Questions or Comments ?