Internal Validity and Basic Research Design

1. Internal Validity and Basic Research Design

2. Internal Validity • the approximate truth about inferences regarding cause-effect or causal relationships • only relevant in studies that try to establish a causal relationship • Or wherever the language of the study infers a causal relationship

3. Internal Validity internal validity is a “zero-generalizability concern”

4. Establishing Cause and Effect • 3 Criteria for a causal relationship to exist: • temporal precedence • covariation of cause and effect • no plausible alternative explanations • Example: • Does drinking beer make you happy?

5. Establishing Cause and Effect 1. temporal precedence • beer comes before happiness 2. covariation of cause and effect • if beer then happy, if no beer then not happy • the more beer then the more happy

6. Establishing Cause and Effect 3. no plausible alternative explanations are these plausible? “Never drink alone”…social causes? To drink, you need to have cash…it’s an economic difference? Drinkers experience smoke too…it’s all about the ciggies? When you drink you go to the loo more often…something to do with bladder swelling/emptying?

7. Identifying Possible Threats • Possible studies testing whether beer makes you happy? measure happiness give beer X O KNR 164 measure happiness measure happiness give beer X O O KNR 164 measure happiness measure happiness give beer X O O KNR 164 (8am) O O KNR 164 (9am)

8. Internal Validity Threats single-group threats multiple-group threats social interaction threats

9. Single Group Threats • history threat • some event (or series of events) occurring during the study that affects the outcome • maturation threat • observed effect due to normal maturation or development of subjects

10. Single Group Threats • testing threat (pre-post only) • taking the pre-test affects performance on post-test • instrumentation threat (pre-post only) • change in tests (observations, measures) is responsible for change in outcome

11. Single Group Threats • mortality threat • non-trivial or non-random dropout affects the results • regression threat • it is not the treatment causing the effect, but rather the posttest scores are simply moving back toward the population mean (up or down)

12. Solving Single Group Threats • Add a control group • the only difference between the control group and the treatment group should be the presence or absence of the treatment • sometimes this means you need multiple control groups

13. Multiple Group Threats • all are selection bias threats • the critical question is: “were the groups equal (on the measure of choice) at the start of the study?”

14. Internal Validity Threats single-group threats multiple-group threats social interaction threats

15. Multiple Group Threats • selection-history threat • some event (or series of events) occurring between the pre- and post-test affects the groups differently • selection-maturation threat • there are differential rates of normal development between the pre-test and post-test for the groups

16. Multiple Group Threats • selection-testing threat • the effect of taking the pre-test varies between groups • selection-instrumentation threat • changes in the tests across time vary between the groups

17. Multiple Group Threats • selection-mortality threat • differential non-random dropout occurs between the groups, so that post-test scores differ • selection-regression threat • differential rates of regression to the mean cause post-test scores to differ across groups.

18. Solving Multiple Group Threats • randomly assign to groups • random assignment ensures no systematic difference between groups (and therefore decreases the possibility that any observed affect is due to selection bias) • there have to be a reasonable number of participants in each group for the random assignment to work • can’t just randomly assign two people to two groups and expect them to be equal!

19. Internal Validity Threats single-group threats multiple-group threats social interaction threats

20. Social Interaction Threats • diffusion or imitation of treatment • control group learns about treatment somehow and does it on their own • control group becomes more like treatment group • compensatory rivalry • control group knows about treatment and becomes competitive with the experimental group, which affects how they respond on the posttest • control group becomes more like treatment group

21. Social Interaction Threats • resentful demoralization • basically the opposite of compensatory rivalry • upon finding out about the treatment the control group members become discouraged, angry, etc. • probably perform worse on post-test than they would have • compensatory equalization of treatment • the researcher somehow affects the outcome by treating the control group favorably in some way

22. Social Interaction Threats • experimenter bias • the belief in the treatment in some way causes the experimenter to behave differently around the two groups, thus turning the findings into a self-fulfilling prophecy

23. Reducing Social Interaction Threats

24. Guiding Questions for Critiquing the Internal Validity of Research Do the researchers claim in any way that the relationships they uncovered in their study are casual in nature (e.g., the intervention caused changes in the dependent variable, the treatment lead to group differences in the dependent variable)? If so, do you think that the causal inference made by the researchers is reasonable or justifiable given the design of the study? [This is the key Internal Validity question] The following questions should help answer Question #2. Consider the type of design (e.g. are there multiple groups, how were subjects assigned to groups, is there a pre and post-test). Does their design provide evidence of temporal precedence between the cause and effect (i.e. did the cause happen before the effect)? Does their design provide evidence of covariation between the cause and effect (i.e. is there an effect when the cause is present, but not when it is not)? Are there plausible alternative explanations for the relationship between the cause and effect? If so, what are they (use the potential single group, multiple group, and or social interaction threats to internal validity as a guide)?

25. Practice • identify potential internal validity issues (using the threats) for each of the following studies

26. The purpose of the study was to examine the relationship between exercise participation and happiness. A total of 300 students served as subjects, all of whom were recruited from a personal fitness class at a small liberal arts school in the mid-western portion of the United States. Students were led through a 30 minute traditional step aerobics class during one of their regularly scheduled class periods. At the end of the class, the students completed the 5-item Happiness subscale from the General Mood Inventory. Results indicated that the students were extremely happy, thus supporting the positive effect of exercise on happiness.

27. 2. A teacher wanted to determine whether allowing students to choose the topics covered in introduction to exercise science class would translate into better understanding of the subject. At the beginning of the semester students in both sections of KNR 164 were given a 100-item exam assessing general knowledge about exercise science. During the course of the semester, students who signed up for Section 1 of the class (MWF 8am) were given a choice by the teacher about which topics they would like to cover during the class period. Students who signed up for Section 2 of the class (MWF 6pm) followed a standard curriculum. At the end of the semester, students in both sections were given different versions of a 200-item exam covering key concepts of exercise science. Results indicated that the students in the experimental class (i.e., those who were allowed to choose the topics) scored significantly higher on the end of the semester exam, suggesting that the type of teaching style affected learning.

28. 3. The purpose of the study was to test whether a mental imagery training package resulted in better athletic performance. Freshman football players from ISU were randomly assigned to a control and experimental group. During a private session at the beginning of the season the players assigned to the experimental group were given 2 hours of instruction on using mental imagery, and were encouraged to use the technique for 10 minutes before each practice and game during the season. The players assigned to the control group also had a private meeting at the beginning of the season where they watched the movie “Remember the Titans”. At the end of the season, the amount of playing time and the head coach’s rating of overall performance for each athlete were compared to see if the imagery training was effective. Athletes who received the imagery training were found to perform better than those who did not receive the training.