Experimental Research Validity and Control

1. Experimental ResearchValidity and Control

2. Visualizing Designs • Structure of a research project • Shows at a glance the major components of an experiment • Design notation • Useful for seeing the temporal sequence of the experiment • Block notation • Useful for visualizing conditions, interactions

3. Design Notation • Elements: • Observations/Measures: O • Treatments/Programs: X • Groups: Each group has its own line • Assignment to group: • R = Random assignment • N = Nonequivalent groups • C = Assignment by cutoff • Time: Moves from left to right

4. Design Notation • Pretest-posttest design, two groups R O X O R O O

5. Design Notation R O1 X O1,2 R O1 O1,2

6. Classifying designs R X O R O Posttest-only randomized experiment N O X O N O O Pre-post nonequivalent groups quasi-experiment Posttest-only nonexperiment X O

7. Time 4 hr/wk 1 hr/wk 20 20 Inside Setting 20 20 Outside Block Notation • Each axis contains one IV

8. Block Notation • Each block contains one condition Independent Variable: Type of Noise Experimental group 2 Experimental group 3 Experimental group 1 White noise Music No noise

9. Music study from slide 7 in design notation R X1 O R X2 O R O

10. Establishing Cause and Effect • Temporal Precedence • Cause has to happen before the effect • Covariation of Cause and Effect • Show there is a relationship between X and Y • No Plausible Alternative Explanations • Any other cause that can bring about the effect

11. Establishing Cause and Effect

12. Validity • Measure-Related Concerns • Construct Validity • Are you measuring what you think you’re measuring? • Reliability • Is your measure consistent in its evaluation of the same individuals? • Conclusion Validity • Are the conclusions you reach about the relationships in your data reasonable?

13. Validity • Measure-Related Concerns • Construct Validity • Reliability • Conclusion Validity • Method-Related Concerns • Internal Validity • External Validity

14. Validity • Internal Validity • Is the independent variable the only possible explanation of the results shown? • Protecting internal validity allows us to eliminate potential alternative explanations for the outcome of an experiment • Extraneous variables • Influence the DV without your intent or consent!

15. Experimental Control • Extraneous Variables • Nuisance variables • Make the effects of the IV more difficult to determine by affecting variation within groups • Participant characteristics • Unintended influences of experimental situation

16. Experimental Control • Extraneous Variables • Confounding variables • Vary systematically with the IV • Influence the difference between groups • The differences observed between groups could be attributed to the IV or the confound

17. Experimental Control • Extraneous Variables • Experimenters • Physiological differences • Rosenthal effects

18. Experimental Control • Extraneous Variables • Participants • Demand characteristics • Good participant effect • Response biases

19. Internal Validity • Exercise control to protect internal validity before the experiment is conducted • Random Assignment • Elimination • Constancy • Counterbalancing • Single- and Double-blind Experiments • Altering Response Sets

20. Basic Control Techniques • Randomization • Individuals have an equal chance of being assigned to any group in the experiment • Allows us to assume groups are roughly equivalent before administering the IV

21. Basic Control Techniques • Elimination • Extraneous variables are completely removed from the experiment

22. Basic Control Techniques • Constancy • An extraneous variable is reduced to a single value experienced by all participants • Standard control technique • Hold environment, temperature, lighting constant

23. Basic Control Techniques • Balancing and Counterbalancing • Balancing • Achieves group equality by distributing extraneous variables to all groups • Counterbalancing • Controls order effects by presenting treatments in a different sequence • Order or sequence effects • Carryover effects • Differential carryover

24. Basic Control Techniques • Within-Subject Counterbalancing • Within-Group Counterbalancing • Each treatment must be presented to each participant an equal number of times • Each treatment must occur an equal number of times at each testing session • Each treatment must precede and follow each of the other treatments an equal number of times

25. Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 Participant 6 P C P C P C C P C P C P The Cola Wars!

26. Participant 1 Participant 2 Participant 3 Participant 4 Participant 5 Participant 6 P C DP P DP C C P DP C DP P DP P C DP C P The Cola Wars!

27. Basic Control Techniques • Single- and Double-Blind Experiments • The experimenter (single-blind) or both the experimenter and participants (double-blind) are unaware of the participants’ condition

28. Basic Control Techniques • Altering response sets • Rewrite survey items so that some negative responses indicate agreement and some positive responses indicate disagreement • Check for socially desirable responses in your survey and/or potentially undesirable cues

29. Internal Validity • Exercise control to protect internal validity before the experiment is conducted • Assess the internal validity of the experiment afterward

30. Threats to Internal Validity • History • Significant events occur between measurements X O R O X O R O O O X O

31. Threats to Internal Validity • Maturation • Participants change over time in the experiment O X O

32. Threats to Internal Validity • Testing (Pretest Sensitization) • Repeated tests cause changes in DV; practice effects O X O GRE

33. Threats to Internal Validity • Instrumentation • Changes in measurement due to changes in the measurement device (human or machine) O X O Paper GRE Computer GRE

34. Threats to Internal Validity • Statistical regression • Extreme scores regress toward the mean • Difficult to maintain extreme scores over repeated measures

35. Threats to Internal Validity • Selection • Selecting participants into groups that were not equivalent before the experiment N X O N O R X O R O

36. Threats to Internal Validity • Attrition or Mortality • Experimental dropouts

37. Threats to Internal Validity • Diffusion of treatment • Participants in one group communicate information from the IV to the other group

38. Threats to Internal Validity • Interactions with Selection • Maturation • History • Instrumentation

39. How Important is Internal Validity? • It is themost important property of any experiment • An experiment without internal validity cannot have external validity • With no internal validity, you can have no confidence in your results

40. External Validity • How far can you generalize your results beyond your experiment? • 3 types: • Population • Environmental • Temporal

41. Threats to External Validity • Testing/Treatment Interaction • Pretest may make reaction to the IV different from those not tested • Selection/Treatment Interaction • Effect is found only for a specific group of participants

42. Threats to External Validity • Reactive Arrangements • A la Hawthorne effects • Demand characteristics • Participant Characteristics • Specific animal species

44. Threats to External Validity • Reactive Arrangements • A la Hawthorne effects • Demand characteristics • Participant Characteristics • Specific animal species • Experimental participant populations: • College students • White male Americans Experimental setting alters participants behavior

45. Is External Validity Necessary? • May want to find out if something can happen • We may be predicting from the real world to the lab • If we can show something happens in the lab’s unnatural setting, we may have more confidence in the phenomenon • We may study something without a real-world analogy