Lecture 8 – Internal Validity Threats continued

1. Lecture 8 – Internal Validity Threats continued EPSY 640 Texas A&M University

2. TESTING • PRETESTING OR REPEATED TESTING • proactive or retroactive interference • memory • change in cognitive schema or structure

3. TESTING SOLOMON 4 GROUP DESIGN: 1 R O X O 2 R O O 3 R X O 4 R O

4. TESTING • Willson and Putnam (1982) conducted a meta-analysis of studies using pretests and concluded that pretesting can effect both cognitive and psychological testing. The largest effects can be expected to occur within about two weeks of pretesting, and effects on achievement and intelligence tests is expected to be greater than effects for attitude or psychological tests.

5. MORTALITY • LOSS OF CASES FROM GROUPS NONRANDOMLY • POTENTIAL FOR PARTICULAR CONDITION TO CAUSE DROPOUTS • NEED FOR INITIAL INFORMATION ON ALL PARTICIPANTS TO EXAMINE CHARACTERISTICS OF DROPOUTS

6. REGRESSION TOWARD MEAN • EQUIVALENT TO IMPERFECT CORRELATION BETWEEN VARIABLES • EXISTS WHENEVER A SAMPLE IS SELECTED AND ITS MEAN IS NOT EQUAL TO THE POPULATION MEAN, THEN MEASURED ON ANOTHER VARIABLE

7. REGRESSION TOWARD MEAN • REGRESSION OCCURS WITH: • GIFTED AND TALENTED RESEARCH • RESEARCH IN SPECIAL EDUCATION • RESEARCH ON ANY NONREPRESENTATIVE GROUPS

8. REGRESSION TOWARD MEAN AND MATCHING • MATCHING ALWAYS PRODUCES REGRESSION EFFECTS UNLESS THE TWO SAMPLES MATCHED HAVE THE SAME POPULATION MEANS

9. REGRESSION TOWARD MEAN AND MATCHING • In a study of a new intelligence test researchers gave the test to students who had been previously evaluated with another intelligence test and found to have scored over 120, qualifying them for gifted and talented status. The researchers found that the students averaged below 120 on the new test and concluded that the new test did not adequately assess gifted and talented status. What they actually found was the regression effect due to the lack of perfect correlation between the two intelligence tests. A simple calculation from their reported results conformed quite well to the expected lowering of score from one assessment to the other.

10. Statistical regression due to matching group 1 group 2 matching mean for selecting variable Outcome correlates .7 with selecting variable: Group 1 predicted score on outcome = +.7SD Group 2 predicted score on outcome = +.35 SD Thus, .35SD difference has been created with no treatment +1SD + ½ SD

11. HISTORY • CHANGES AND EVENTS OUTSIDE TREATMENT CONDITIONS THAT AFFECT OUTCOMES • Local or national events of stress • Weather • Political change

12. HISTORY • Campbell and Ross (1968) and Glass (1968) evaluated the effect of a speed limit law in Connecticut enacted to begin in January, 1955. Politicians claimed within several months that the law had reduced significantly auto deaths, using data from the previous several years to support their claim. Indeed, statistical analysis supported a significant drop in deaths in the months following the implementation of the law, even accounting for typical drops occurring after the Christmas season. Glass (1968) noted that an analysis of the surrounding states showed a similar drop even though no laws had been enacted in them. An investigation of weather conditions at the time showed that a series of blizzards at the time severely restricted driving, particularly high speed driving that would have contributed otherwise to auto deaths. He and Campbell and Ross (1968) discounted the legal experimental account in favor of the historical explanation. Willson (1973), reported in Glass, Willson, and Gottman (1975) demonstrated that the legal effect was not significant when adjusted for the comparable effects in other adjacent states