Evaluating the Experiment from the Inside: Internal Validity

1. Evaluating the Experiment from the Inside: Internal Validity Taking a Broader Perspective: The Problem of External Validity Handling a Nonsignificant Outcome

2. An experiment is internally validwhen the effects on the dependent variable are due to the independent variable. An internally valid experiment is free of confounding. What is internal validity? Evaluating the Experiment from the Inside

3. A manipulation check evaluates how well the experimenter manipulated the experimental situation. A manipulation check determines whether subjects followed directions and were appropriately affected by our treatments. What is a manipulation check? Evaluating the Experiment from the Inside

4. Subjectsexpect their data to be discarded if they guess the experimental hypothesis, and don’t volunteer this information to the experimenter. Experimenters don’t want to test additional subjects and may take subject reports at face value. What did Orne (1969) mean by a pact of ignorance? Evaluating the Experiment from the Inside

5. Debrief subjects after the experiment and convey that you want to know if they guessed the hypothesis. Provide incentives for guessing the hypothesis. How can researchers overcome this problem? Evaluating the Experiment from the Inside

6. Selecting the wrong statistical test Using a t-test to analyze ordinal data. Improperly using a statistical test Calculating multiple t-tests. Drawing the wrong conclusions from the test Reporting p = .07 as a trend. Which mistakes could produce this threat to internal validity? Evaluating the Experiment from the Inside

7. An experiment is externally validwhen its findings can be extended to other situations and populations. Explain external validity. The Problem of External Validity

8. 1. The experiment must be internally valid. 2. The experimental findings can be replicated. What two requirements must an externally valid study satisfy? The Problem of External Validity

9. The findings can be extended to a larger group than our sample. Generalizing across subjects is critical to the external validity and usefulness of experimental findings. What does it mean to generalize across subjects? Why is this important? The Problem of External Validity

10. The samples used in psychological research are often biased and may not represent the larger population. The samples may not always represent even college sophomores since we heavily depend on volunteers. Which problems prevent us from generalizing across subjects? The Problem of External Validity

11. Experimental variables like anger may have multiple operational definitions. When we generalize from our experimental results, we move from discussing our specific operational definition of anger to discussing the concept of anger itself. Explain generalizing from procedures to concepts. The Problem of External Validity

12. It is dangerous to generalize from a single experiment’s operational definition of anger. We cannot be sure of the reliability or validity of our procedures. Why is this a problem in research? The Problem of External Validity

13. A study achieves research significancewhen its findings clarify or extend knowledge gained from previous studies and raise implications for broader theoretical issues. What is research significance? The Problem of External Validity

14. We should question novel findings when they contradict prior findings that have been successfully replicated. The burden of proof is on the experimenter who claims novel findings to explain this discrepancy. When should we question novel findings? The Problem of External Validity

15. We want to generalize beyond the laboratory to increase the external validity of our findings. Why do we want to generalize beyond the laboratory? The Problem of External Validity

16. Since extraneous variables are uncontrolled in real world setting and operate in complex combinations, they can modify the influenceof our individual variables. What is the problem in extending laboratory findings? The Problem of External Validity

17. The trade-off is between the laboratory’s more precise control of extraneous variables and the field experiment’s greater realism and external validity. What is the trade-off between laboratory and field experiments? The Problem of External Validity

18. Hanson (1980) found that more laboratory than field studies reported a positive correlation between reported attitudes and behavior. What did Hanson (1980) find about the agreement between laboratory and field studies? The Problem of External Validity

19. We can’t confirm external validity until additional studies are completed in field settings. Researchers can increase and verify the external validity of laboratory findings using aggregation, multivariate designs, nonreactive measurements, field experiments, and naturalistic observation. When can we extend results from an experiment to everyday life? The Problem of External Validity

20. Aggregationisthe grouping together and averaging of data to increase external validity. Combining the results of experiments with different subjects and methodologies increases the generality and external validity of our findings. What is aggregation? What is its logic? The Problem of External Validity

21. Meta-analysis uses statistical analysis to combine and quantify data from many comparable experiments to calculate anaverage effect size. How is this logic shared by meta-analysis? The Problem of External Validity

22. Aggregation establishes external validity by combining the results of experiments performed using different subjects, stimuli and/or situations, trials or occasions, and measures. How does aggregation establish external validity? The Problem of External Validity

23. A multivariate designstudies multiple DVs. For example, a study of repetitive strain places a computer keyboard at different distances from the subject IV) and measures the effect on three different muscle groups (3 DVs). What is a multivariate design? The Problem of External Validity

24. Multivariate designs allow us to study the effect of an independent variable on combinations of dependent variables. These designs better simulate the complexity of the real world than univariate designs and provide more detailed information. What is the advantage of multivariate designs? The Problem of External Validity

25. We analyze multivariate experiments with a multivariate analysis of variance (MANOVA). How are multivariate experiments analyzed? The Problem of External Validity

26. Accept the outcome, don’t reframe your result as “almost significant.” Examine the experimental procedures for design flaws. If the design appears sound, decide whether the hypothesis was reasonable. How should researchers handle a nonsignificant outcome? Handling a Nonsignificant Outcome

27. Check for possible causes of a nonsignificant outcome like: confounding extraneous variables that increase within-subjects variability weak manipulation of the IV How should we handle the possibility of faulty procedures? Handling a Nonsignificant Outcome

28. inconsistent or flawed procedures ceiling and floor effects insufficient power How should we handle the possibility of faulty procedures? Handling a Nonsignificant Outcome

29. If previous studies supported the hypothesis and ours did not, look for differences in experimental design or sample. If there was no previous support and our design and execution were good, we may have to revise or discard our hypothesis. How should we handle the possibility of a faulty hypothesis? Handling a Nonsignificant Outcome