Pitfalls in Experimental Research

1. Pitfalls in Experimental Research Threats to Internal and External Validity

2. conceptual slippage faulty manipulation loose procedures order effects experimenter effects fudging researcher personal attribute effect researcher unintentional expectancy effect sensitization recording errors history and maturation basement and ceiling effects Hawthorne effect social desirability bias selection bias (sampling) intersubject bias mortality (attrition) overview of threats to validity

3. conceptual slippage • a gap between how a variable is conceptualized and how it is actually operationalized in an experiment • example: prejudice • It is difficult for a scale to capture the full richness of a concept • remedies: • limit the scope of the variables to be studied • don't over-claim the nature and/or purpose of the investigation • triangulation; use multiple outcome measures

4. faulty manipulation • failure to manipulate a variable, or create a stimulus condition as intended • example: Janis & Feshbach's (1953) study on fear appeals • remedies: • include a manipulation check: Teven & Comadena (1996) office aesthetics and teacher credibility • cover story can disguise true purpose of an experiment • use neutral observers to judge stimulus materials

5. loose procedures • ambiguity or imprecision in the experimental protocol • example: Raffetto (1967) study on sensory deprivation and hallucinations • remedies: • run a pilot study • script out all instructions to participants

6. order effects • the order, sequence, or placement of items in a series can influence participants' perceptions • example: Schuman, Presser, & Ludwig (1981) survey on attitudes toward abortion • remedies: • shuffle the order of items within a questionnaire, e.g. use multiple versions of the questionnaire • alter the order if different questionnaires are completed together

7. experimenter effects • fudging: • example: Stephen Gould "IQ Argument" • researcher expectations • example: intercessory prayer • cueing • example: Rosenthal (1966) unintentional paralinguistic and kinesic cues by experimenters • demand characteristics of the experimental situation

8. experimenter effects, continued • researcher observational biases • problem of “observer drift,” e.g., changing standards or criteria • problem of “observer bias” • remedies: • minimize interaction with participants • use multiple experimenters (different genders, different ethnicities, etc.) • use experimenters who are “blind” to the purpose of the study

9. sensitization • pre-test measurements may bias post-test measurements • solution: • avoid pre-tests, if possible • use different pre- and post-test measures

10. recording errors • errors in entering, translating, or transforming data • example: Kennedy & Uphoff (1939) extrasensory perception study • remedies: • use double-checks during data entry • pre-code surveys, questionnaires, etc.

11. history and maturation • history: events happening outside the experiment that influence participants' responses within the experiment • maturation: internal changes in participants during an experiment (developmental, emotional, etc.) • remedies: • keep experiments short • sequester participants if possible

12. ceiling or basement effects • ceiling effect: responses exceed capacity of measuring instrument; responses are “off the scale.” • floor or basement effect: responses are below threshold of measuring instrument • remedies: • use sensitive, precise measures • use established scales, instruments

13. Hawthorne effect • the mere knowledge by a participant that he/she is being observed may alter his/her natural behavior • example: Western Electric plant in Illinois (circa late 1970's) • remedies • use a cover story • use unobtrusive measures

14. social desirability bias • Self reports are often distorted because respondents want to present themselves in the best possible light • Examples: in person interviews involving embarrassing topics; infidelity, spouse abuse, cheating, etc. • Remedies • Emphasize anonymity of responses, importance for obtaining genuine responses, no correct answers • use of indirect questioning: asking respondents to project how they think other persons would answer

15. selection bias • failure to gather a random or representative sample, or a reliance on self-selected groups • example: the proverbial "college freshman" used in many social science studies • remedies: • use a random, representative, sufficient sample (easier said than done) • use random assignment within the experiment

16. inter-participant bias • Participant interaction may taint responses, both inside and outside of the experiment • example: social proof and laughter • example: surveys done in large lecture halls • remedies: • isolate participants from one another • ask participants not to disclose the purpose of an investigation to others

17. mortality or attrition • Subjects may drop-out or cease participating in a study. The ones who drop out may be different from the ones who remain. • example: long-term “longitudinal” research • example: studies on intimates • remedies: • keep studies short • provide incentives for participating