Design

1. Design Ch. 6 – Internal Validity

2. Internal Validity • Cause and effect • Cause… “Function: transitive verbInflected Form(s): caused; caus·ingDate: 14th century1:to serve as a cause or occasion of :MAKE2: to effect by command, authority, or force- caus·ernoun” *from Merriam-Webster online

3. Internal Validity • Cause and effect • Effect… “Function: nounEtymology: Middle English, from Middle French & Latin; Middle French, from Latin effectus, from efficere to bring about, from ex- + facere to make, do -- more at DODate: 14th century1 something that inevitably follows an antecedent (as a cause or agent) ” *from Merriam-Webster online

4. Internal Validity • Cause and effect Internal validity is a “zero-generalizability concern”

5. Internal Validity • Cause and effect • So…did the treatment you used cause the measurement you took to be what it was? • Or…to what extent did the treatment you used influence the measurement you took?

6. Internal Validity • Establishing cause and effect • Example: Does beer make you happy? • Temporal precedence beer happiness

7. Internal Validity • Establishing cause and effect • Example: Does beer make you happy? • Covariation of cause and effect If beer then happy If not beer then not happy If more beer then more happy If less beer then less happy

8. Internal Validity • Establishing cause and effect • Example: Does beer make you happy? • No plausible alternative explanations “Never drink alone”…social causes? Rank – most plausible? - least plausible? To drink, you need to be flush…it’s an economic difference? Drinkers experience smoke too…it’s all down to ciggies? When you drink you go to the loo more often…something to do with bladder swelling/emptying?

9. Internal Validity • Single group threats • Sticking with the beer idea… • Imagine you take a group of students, and give them a dose of beer. You might measure their happiness just once at the end, or you might measure them both before and after the beer dose…

10. Internal Validity • Single group threats • Sticking with the beer idea… Give beer Measure happiness Measure happiness Give beer Measure happiness

11. Internal Validity • Single group threats • Sticking with the beer idea… • History Threat • It’s not the beer. It’s some other major historical event that influenced your group’s happiness… • Perhaps on the day of the test, Saddam decided to pack it in, and everyone simultaneously found out that the recession was a big mass-induced dream state, awaking to find that the world was infinitely happier than they had first thought? Plausible?

12. Internal Validity • Single group threats • Sticking with the beer idea… • Maturation Threat • As people get older, they get happier. You took your first measurement during your group’s teenage angst years, then waited until the wisdom, wonder and general delight of 40ish life settled in before taking the second measurement • [can be anything related to general “internal” change in people – any “big event” external to the person causing the change is generally regarded as a history threat] Plausible?

13. Internal Validity • Single group threats • Sticking with the beer idea… • Testing Threat • (Pre-post only) • Measuring happiness made participants focus on their degree of happiness – if they were unhappy, they resolved to buckle up, and cheer up. If they were happy they were made aware of this and were therefore delighted to be so well-adjusted Plausible?

14. Internal Validity • Single group threats • Sticking with the beer idea… • Instrumentation Threat (pre-post again) • A couple of possibilities… • The group are scored with different instruments on each occasion. The second is more sensitive than the first. The group’s happiness doesn’t change, but the scores arising from the instruments differ anyway • The person giving out the happiness instrument is such a happy-go-lucky lass that she infects everyone else with her sense of “joi de vivre” (experimenter as instrument)- the longer she’s around the happier you feel Plausible?

15. Internal Validity • Single group threats • Sticking with the beer idea… • Mortality Threat • As the study wears on and the beer starts to have its inevitable effect, many people are past the stage of being happy and are approaching that other unfortunate stage of influence…but they aren’t around for the happiness measurement at the end of the study Plausible?

16. Internal Validity • Single group threats • Sticking with the beer idea… • Regression Threat • [The Fugawe Tribe] • When you look at the scores of your group on the pre-test, you notice that they are unusually low… • This situation, where for any reason scores are just by chance suppressed on the first measurement, will probably result in artificially higher scores (in comparison to the first) on the second reading • It’s not that the treatment is having an effect, it’s that the scores are just going back to normal • Normally occurs when selecting a group by pre-test scores… Plausible?

17. Internal Validity • Single group threats • Sticking with the beer idea… • Regression Threat • You want to find out if the effect even works on really unhappy people. So you get 200 grad students to fill out a happiness questionnaire, and select only the lowest 25% for your study. • Lo and behold, after a good dose of beer, they all score higher on the hap-o-meter! • Why? Plausible?

18. Internal Validity • Solving single group threats • Add a control group • The only difference between the control group and the treatment group should be the presence or absence of the treatment • Sometimes this means you need multiple control groups • Often, it still fails…

19. Internal Validity • Multiple group threats • All are selection bias threats • The critical question is: “were the groups equal (on the measure of choice) at the start of the study” • The general case is simply a selection bias. Other specific differences follow

20. Internal Validity • Multiple group threats Single-group threats – taken care of now Multiple-group threats – still a problem

21. Internal Validity • Multiple group threats • More examples with the hoppy one… • Selection-history threat • Say you got your 2 groups from grad classes • On the day of the test, one of the groups’ classes has a nasty test while the other has a normal class • What is causing the difference in happiness now?

22. Internal Validity • Multiple group threats • More examples with the hoppy one… • Selection-maturation threat • Say you’re looking at the long-term effects of beer on happiness (not temporary as now) • Say also that one group (the drinkers) is selected from among the student population, while the other is selected from among the faculty • It might be easy to imagine that the older group is somewhat more stable with respect to the happiness measure than the younger

23. Internal Validity • Multiple group threats • More examples with the hoppy one… • Selection-testing threat • Generally, that one group is for some reason more sensitive to be influenced by the first test than the second • Say you get the beer group to volunteer from a pub, while the non-beer group volunteers from a teetotal social club • The pub group also (unknown to you) score higher on a neuroticism scale. This leads them to be more easily influenced by knowledge of their score on a happiness scale…and thus alters their responses second time around

24. Internal Validity • Multiple group threats • More examples with the hoppy one… • Selection-instrumentation threat • Say we have our happy-go-lucky experimenter from the single group threats operating here too, but she can’t do both groups (schedule can’t allow), so she gets her partner (who is rather gloomy…funny how that seems to happen) to monitor the other group

25. Internal Validity • Multiple group threats • More examples with the hoppy one… • Selection-mortality threat • Even if you add another (non-beer) group, it is quite probable that your non-beer group will be intact at the end of the study, while your beer group might suffer substantial drop-out

26. Internal Validity • Multiple group threats • More examples with the hoppy one… • Selection-regression threat • Say you select your volunteers from a pub on a weekday afternoon. The others are selected at random from grad research classes • Trouble is, folk who drink at a pub on a weekday afternoon tend to be a depressed lot…and thus are more likely to improve their scores on a subsequent post-test • [this is fiction, folks!]

27. Internal Validity • Solving Multiple group threats • Randomly assign to groups • Random assignment ensures no systematic difference between groups • We’re still not done yet though…

28. Internal Validity • Social interaction threats Single-group threats – taken care of by adding control group Multiple-group threats – taken care of by random assignment Social interaction threats

29. Internal Validity • Social interaction threats • One more time with the amber nectar (as Ben Franklin once said, “beer is proof that God loves us and wants us to be happy”) • Diffusion or imitation of treatment • Imagine you pluck all your participants from a college hall of residence. They are aware of each other’s participation, and the other group’s treatment. On seeing the beer group go off for their treatment, the non beer group think “that’s a good idea” and troop off to the liquor store…

30. Internal Validity • Social interaction threats • One more time… • Compensatory rivalry • In some way the groups become aware of each other’s treatments and of the measures, and something about them sets up a different behavior that influences the final measurement • For example, the non-beer set may for some reason not want to be judged as less happy” than the beer group second time around.

31. Internal Validity • Social interaction threats • One more time… • Resentful demoralization • Say the non-beer group are aware the beer group is getting the beer • Say also the non beer group enjoy a beer • They’re jealous • This unhappiness finds its way into the post-test

32. Internal Validity • Social interaction threats • One more time… • Compensatory equalization • Say the non-beer group are aware the beer group is getting the beer • Say also the non beer group enjoy a beer • Say the experimenters are aware of this • They treat the other group with extra kindness and comfort to offset the perceived slight at the lack of beer

33. Internal Validity • Social interaction threats • And one extra I’d add here (repeated from construct validity) • Experimenter bias • The belief in the treatment in some way causes the experimenter to behave differently around the two groups, thus turning the findings into a self-fulfilling prophecy

34. Introduction to design

35. Types of designs

36. Types of designs