Experimental Design: Threats to Validity

1. Experimental Design: Threats toValidity

2. EXPERIMENTS: The independent variable is manipulated todetermine its effect on the dependent variable(s)whilst holding all other potential influences onthe dependent variable(s) constant

3. Does the design allow us to answer thequestion, or might there be alternativeexplanations for the results obtained? VALIDITY OF A STUDY INTERNAL VALIDITY: To what extent does the design of the studyallow us to attribute changes in the dependentvariable(s) to the effects of changes in theindependent variable?

4. THREATS TO INTERNAL VALIDITY Changes in the dependent variable may be dueto a variety of extraneous factors (confoundingor potential independent variables), rather thanto the manipulation of the independent variable

5. How meaningful are the results whenapplied to the real world? VALIDITY OF A STUDY EXTERNAL (ecological) VALIDITY: To what extent does the design of the studyallow us to generalise the results to populationsother than that from which the sample was drawn,or to similar populations in different settings orat different times?

6. THREATS TO INTERNAL VALIDITY Threats relating to… The passage of time Selection of participantsTesting and manipulations

7. Example 1: Teaching styles THREATS RELATING TO THE PASSAGE OF TIME MATURATIONChanges within the participants due to thepassage of time (developmental, ageing,hunger, fatigue etc.)

8. Example 2: Veggie study THREATS RELATING TO THE PASSAGE OF TIME HISTORYEvents occurring between pre- and post-test in addition to the experimental manipulation

9. Example 3: Foul play study THREATS RELATING TO THE PASSAGE OF TIME INSTRUMENTATIONChanges in the way the dependent variableis measured (measurement errors,different tests, calibration problems, etc)

10. 54321 1 2 3 4 5 Unsupervised Intentions Supervised 1 2 3 4 5 Example 4: Exercise adherence Pre-test Post-test 1 2 3 4 5

11. 54321 1 2 3 4 5 Unsupervised Intentions Supervised 1 2 3 4 5 Example 4: Exercise adherence Pre-test Post-test 1 2 3 4 5 MORTALITY Differentialdrop-out from groups betweenpre- and post-test

12. RANDOMISED PRE-TEST POST-TESTCONTROL GROUP DESIGN R O1 X O2 R O3 O4 Controls for threats to internal validity due tothe passage of time provided that: Randomisation works (groups equated at pre-test) There is no differential mortality between groups

13. RANDOMISED PRE-TEST POST-TESTCONTROL GROUP DESIGN R O1 X O2 R O3 O4 Threats relating to the passage of timeare controlled because they shouldmanifest themselves equally in each group

14. Dependent Variable Pre-test Post-test R O1 X O2 R O3 O4

15. R O1 X O2 R O3 O4 Dependent Variable Pre-test Post-test

16. THREATS RELATING TO SELECTIONOF PARTICIPANTS SELECTIONBias resulting from differential selection when assigning participants to groups

17. Recovery time Steroid Shoulder ACL Pre-operative anxiety and recovery from anaesthesia. A study was conducted to determine whether a brief pre-operative relaxationprocedure could reduce recovery time in surgical patients with sports injuries.Athletes about to undergo three different types of surgery were assigned toone of three conditions. Those about to receive a steroid injection for frozenshoulder were assigned to a relaxation condition in which they listened to a30 min. relaxation tape immediately prior to transfer to the operating theatre.Athletes about to undergo repair of ruptured anterior cruciate ligament wereassigned to a placebo condition in which they listened to a 30 min. tape-recordedshort story. Athletes about to undergo reduction of dislocated shoulder actedas no-treatment controls.

18. Dissociation and endurance performance (Morgan et al., 1983). A study was performed to evaluate whether a mental dissociation strategy couldfacilitate endurance performance. 27 navy personnel underwent a submaximalexercise test to determine VO2 max. Participants returned to the lab 48hours later and ran to exhaustion on a treadmill at an exercise intensity of80% VO2max. Time to exhaustion was recorded. Participants returned again48 hours later and were randomly assigned to a treatment group or a controlgroup. The treatment group were instructed in how to employ a dissociationstrategy whilst running. Time to exhaustion at 80% VO2max was assessed again. Results showed that significantly more of the dissociation participantsincreased their endurance time between trials two and three compared tocontrol participants (78% of dissociation participants versus 31% of controlparticipants; p < .05). This suggests that the dissociation strategy facilitatesendurance performance.

19. Dissociation and endurance performance (Morgan et al., 1983). A study was performed to evaluate whether a mental dissociation strategy couldfacilitate endurance performance. 27 navy personnel underwent a submaximalexercise test to determine VO2 max. Participants returned to the lab 48hours later and ran to exhaustion on a treadmill at an exercise intensity of80% VO2max. Time to exhaustion was recorded. Participants returned again48 hours later and were randomly assigned to a treatment group or a controlgroup. The treatment group were instructed in how to employ a dissociationstrategy whilst running. Time to exhaustion at 80% VO2max was assessed again. Results showed that significantly more of the dissociation participantsincreased their endurance time between trials two and three compared tocontrol participants (78% of dissociation participants versus 31% of controlparticipants; p < .05). This suggests that the dissociation strategy facilitatesendurance performance.

20. THREATS RELATING TO SELECTIONOF PARTICIPANTS SELECTION X MANIPULATION Effects of the manipulation only hold forthe particular population sampled A threat to external validity

21. Attitudes (mean) Control Treatment Pre-test Post-test Example 5: Drugs intervention

22. THREATS RELATING TO TESTINGAND MANIPULATIONS EFFECTS OF TESTINGThe effects of taking the pre-test on scoreson the post-test Post-test scores may be affected by practice,memory etc. Pre-test sensitisation

23. Example 5: Drugs intervention Attitudes (mean) Pre-test Post-test

24. THREATS RELATING TO TESTINGAND MANIPULATIONS REACTIVE EFFECTS OF TESTINGTesting X manipulation interactionThreat to external validity Pre-testing changes the participants’ responsesto the manipulation. The results obtained frompre-tested participants may be unrepresentativeof results that would be obtained from thosewho only received the manipulation

25. REACTIVE ARRANGEMENTS The experimental setting influences participantsto respond differently to how they would respondin a real-life setting Research settings are unnatural situationsso we might expect participants to behaveunnaturally

26. Dissociation and endurance performance(Morgan et al., 1983). Long distance runners in Tibet have used thistechnique to run 300 miles non-stop in 30 hoursand many successful marathoners are known to use similar techniques… By dissociating in thisway you will not feel the same amount of fatigue,pain or discomfort you experienced during the lasttrial…

27. REACTIVE ARRANGEMENTS DEMAND CHARACTERISTICS (Orne, 1962) “… the totality of cues which convey an experimental hypothesis to the subject.”

28. Dissociation and endurance performance(Morgan et al., 1983). Long distance runners in Tibet have used thistechnique to run 300 miles non-stop in 30 hoursand many successful marathoners are known to use similar techniques… By dissociating in thisway you will not feel the same amount of fatigue,pain or discomfort you experienced during the lasttrial… Please go as long as you can in order to makeour experiment a success.

29. NON-SPECIFIC TREATMENT EFFECTS Placebo/attention effects Hawthorne effects Expectations of benefit Credibility of the treatment

30. Rosenthal’s rats (1963) EXPERIMENTER EFFECTS The Pygmalion effect Rosenthal & Jacobson (1968)

31. RANDOMISED PRE-TEST POST-TESTCONTROL GROUP DESIGN R O1 X O2 R O3 O4 Controls for threats relating to the passage of timeprovided that: Randomisation works (groups equated at pre-test) There is no differential mortality between groups All potential independent variables includingnon-specific treatment effects are held constant

32. R O1 X O2 R O3 O4 Threats relating to selection and testingare controlled because participants arerandomly assigned to groups EXCEPT: Because there is a pre-test, the reactiveeffects of testing are not controlled

33. Alsocontrols for threats to internal validityprovided that: Randomisation works (groups equated at pre-test) There is no differential mortality between groups All potential independent variablesincludingnon-specific treatment effectsare held constant POST-TEST ONLY CONTROL GROUP DESIGN R X O1 R O2

34. R X O1 R O2 PLUS:This design also controls for reactive effectsof testing because there is no pre-test BUT:We don’t know if the groups were equated at pre-test

35. EXTERNAL VALIDITY With the exception of the post-test only design’scontrol for reactive effects of testing, logically, wecannot control for threats to external validity In an internally valid study we can onlydemonstrate that the effects (of X) hold underthe specific conditions of the study e.g. The effects of X hold for(pre-tested)participants from theparticular populationsampled at thispoint in timein this particularplace … etc.