Laboratory v. field experiments

1. Laboratory v. field experiments The tradeoff between internal and external validity

2. Natural sciences • In natural sciences, most experiments have been undertaken in carefully controlled laboratory conditions • Seeking laws of physics, etc. • Inanimate objects are unlikely to react differently in controlled and uncontrolled conditions • The main goal is to avoid contaminating the experiment via the introduction of third variables • Far more exacting and finer measurement is possible under controlled conditions

3. Laboratory experiments in social science • Closely controlled social science experiments do have the advantage of limiting the impact of third variables, but the unnatural situation presents problems for applying the findings outside the laboratory • “Artificiality” of laboratory setting

4. Human behavior is sensitive to the environment within which it occurs • People act differently in a laboratory than in the natural world • Several characteristics of laboratories are thought to influence behavior • The very third variables controlled in the lab may be the ones that determine behavior in the real world • So, findings from laboratory experiments may only be valid for laboratory environments

5. An example • New commercials are tested in controlled conditions • Eye tracking • Liking for commercials • Influence on purchase interest • May try to provide less artificial conditions for study • Simulated living room • Commercials that test high in lab experiments often do not work very well when used in real marketing campaigns

6. So, experiments move out of the lab: • Researchers want to retain some of the advantages of the experiment: • Ability to manipulate/introduce the independent variable and to control how much of it is presented • Time order—which comes first • While sacrificing some of their ability to control third variables • The goal is to improve our ability to generalize our findings to the real world

7. The field experiment • One way to do so is to carry out a ‘field experiment’ • The researcher still manipulates the independent variable, but she does so within the natural world • For example, police try randomly stopping motorists and testing for alcohol to see if alcohol-related crashes decline

8. Information Resources, Inc. • BehaviorScan system

9. What problems do we encounter? • Greatly reduced ability to prevent third variable contamination of results • The crusading doctor in MHHP • History effects • Cycles • What can you do? • Measure/monitor likely alternative explanatory variables • Question subjects about sources of influence • Multiple manipulations of the independent variable over time • Multiple measurement waves • Multiple dependent measures • None will be perfect

10. Expense • Tradeoff between extensive and intensive study • Budget constraints on number of sites, etc. • Access/permission • Some research may present concerns to authorities, citizens, etc. • Gain authorization/support prior to entering the field • Maintain good relationships with community leaders, etc. throughout the intervention/research

11. Capturing ‘natural experiments’ • Sometimes unusual or unique events occur • ALAR scare • Service disruptions • Political campaigns • School shooting • Institution of Wi-Fi in an area

12. “Natural experiments” • Because most such events are unplanned, the ability to prepare for them is limited • May keep a research group, materials and resources ready for certain types of events • Must engage in ‘firehouse research’ gathering as much data as possible in a short time • Inefficient, and may miss important data • However, real-world events, etc. may provide very valuable data—may have both internal and external validity

13. Use statistics, research analysis models that are appropriate to experimental studies