Comparative Study: Web vs. Lab Approach

1. Web vs. Lab John H. Krantz Hanover College

2. Approach • Not focus on technology • Focus on the research question • What is best way to answer question

3. Approach • Web is a tool • to use in determining best way to ask question • Much as field experiments have become a tool: • e.g. Milgram, S. , Bickman, L., & Berkowitz, L. (1969).  Note on the drawing power of crowds of different sizes.  Journal of Personality and Social Psychology, 13(2), 79-82. • While this example is trivia it shows an example of a study not possible in the laboratory

4. Approach • What this implies: • For some questions web is a superior method to lab • May depend on your situation as we all use what we have available

5. Web Distributed User or remotely controlled environment and location and some factors are unknowable Accidental sampling of large population Potentially anonymous Requires computer Lab Confined Experimenter controlled environment or at least determinable Accidental sampling generally but random sampling of restricted population is possible Experimenter is known to be present or near May not have computer Characteristics of Web vs. Lab

6. Characteristics of Web vs. Lab • Similarities • A certain act of belief is needed to accept self-report responses • Cannot control state and previous state or events of participant • This is an issue in control • Random assignment is possible

7. Reasons for Use – Musch & Reips (2000) • In descending order (7 very important – 1 not important) • Large N 5.5 • High statistical power 4.5 • Speed data collection/participants from other countries 3.6 • External validity 3.4 • Low cost 3.2

8. Reasons for Use • Possible responses not possible on paper • Dynamic and interactive (but any computer would do) • Access to special populations • More precise replication • Reduction of experimenter effects • Comparability to local sample • New set of variables to study

9. Reasons for Concern • Ethical issues • Informed consent/debriefing • Data security • Data validity • Subject falsification/collaboration • Loss of experimental control • Equipment/environment Variation

10. Reasons for Concern • Computers are necessary • Lack of ability to clarify instructions and informed consent • New class of variables that may impact responses • Does this last one sound familiar?

11. Importance of Problems Faced • Musch & Reips (2000) • In descending order • Lack of control of behavior 3.6 • Lack of control of motivation 3.4 • Inability to ask questions 3.3 • No control over hardware 2.9 • Nonrepresentative sample 2.8 • Ethical problems 1.5

12. Issues regarding the sample • Web is stated to be not representative • Questions • How many labs have representative sample? • Do we know what person variables are significant or do we control out of the assumption of a possible effect? • The greater diversity of samples (Krantz & Dalal, 2000 and “College Sophomores”) may allow this to become a research question instead of assumption

13. The Web Study In Context • Relevant Dimensions • Methodological Issues • Subject-Related Issues • Ethical Issues • A Decision Guide

14. Relevant Dimensions – Method • Statistical Power • Refers to noisy data • Not in all cases will noise be better on Web • Ruppertsberg et al (2000). • Web, n = 151, s = 12.5, SE = 1.02 • Control, n = 14, s = 7.4, SE = 1.9 • Effect Size or Robustness of the Effect • Basically is it easy to obtain the result

15. Relevant Dimensions – Method • Sample Bias • Is the effect sensitive to the characteristics of the sample? • External Validity • Does this effect or relationship happen outside of laboratory • Does this effect or relationship happen away from a computer

16. Relevant Dimensions – Method • Need for Control • Stimulus • Problematic Dimensions • Color • Luminance • Temporal • If can keep on one page • Relative spatial dimensions are fine

17. Example Study: Column Taper • Description: • Doric columns appear to lean when vertical • Apparently due to taper of column width • Method • Three columns. Adjust so out two same position as inner (vertical or horizontal) • Validation • Results

18. Manipulation Validation

19. Vertical Results

20. Relevant Dimensions – Method • Need for Control • Measurement • Need will be sensitive to type of measurement • Think of all question order effects • Now type of response objects • Environment • Effect of Web Known • Has the method been examined on Web • Time for data collection • Response time for results

21. Relevant Dimensions – Subject • Special Populations • Ethnic groups • Age • Education • Other targeted populations (Buchanan, 2000) • Self-Monitoring: found groups predicted high and low on scale to validate response

22. Relevant Dimensions – Subject • Interaction with Subjects • Need for Clarification • As part of research design • Demand Characteristics • Web removes us, but our design might still provide some hints

23. Relevant Dimensions – Ethics • Sensitivity of Ethical Issues • Deception • Potential for harm • Effects of Fraud by Participants on Data • Faking responses • Participant Collaboration

24. A Decision Guide - Method Sensitivity of Study to/Need:

25. Decision Guide – Method

26. Decision Guide – Subject

27. Decision Guide – Ethics

28. Factors not Included in Decision Guide • Did not include ease or cost • Also not important factor in early researchers, Musch & Reips, 2000 • These factors should not drive design criteria.

29. Conclusions • Web and lab distinct environments • Each method has strengths • Match method to research question • Still useful to examine methods • Study locally as well as globally • Differences do not necessarily imply web is less valid than lab • Differences could be meaningful • Differences could imply problem in lab envirnment