Lecture 5

1. Experiments Lecture 5

2. Agenda • Labs (update and questions) • STATA Introduction • Intro to Experiments and Experimental Design

3. Experiments vs Non-Experiments • Experiment is a study in which a treatment is introduced • The type of experiment (such as a true, randomized experiment) depends on several factors • Non-Experiments include responses from natural groups (e.g., most surveys) http://imgs.xkcd.com/comics/the_data_so_far.png

4. Famous Experiments Tuskegee Study Milgrim Obedience Study Stanford Prison Experiment

5. Two Essential Criteria in True Randomized Experimental Design • (1) Independent Variables must be manipulated (usually by experimenter, sometimes by context) • (2) Participants must be assigned randomly to various conditions or groups Courtesy http://psychology.ucdavis.edu/SommerB/

6. Active versus Attribute Independent Variables • Active independent variable(s): • The I.V. is “given” to the participants, usually for some specified time period. It is often manipulated and controlled by the investigator. • Attribute independent variable(s): • A predictive, defining characteristic of individuals. Cannot be manipulated.

7. Randomization in Sample and Assignment • Random Sample • System for choosing participants from a population • Generally, the larger the sampling population the better your generalizability becomes. • Random Assignment • Method for assigning participants randomly to experiment conditions

8. True Experiments • True experiments protect against both time and group threats to internal validity by randomly assigning subjects to treatment and control groups. The treatment (independent variable) is active. • If we cannot randomly assign subjects to different groups, then it is a quasi-experiment. The independent variable is active. • If we cannot randomly assign subjects to groups because the groups contain the attribute of interest, and if we give all groups the same treatment, then it is an associational non-experiment. The independent variable is not active.

9. Pre-test, experimental manipulation and post-testing • Pre-test: allows us to check group equivalence before the intervention X is introduced. • Experimental manipulation: An independent variable (X) that the experimenter manipulates. • Post-test: allows us to check group equivalence after intervention X has been introduced. Hypothesis  Random Assignment  Measure D.V.  Treatment  Measure D.V. (pre-test) (post-test)

10. Common types of true experiments O X O Pretest-Posttest Control R O O X O Post-only Control R O O X O (1) O O (2) R Solomon 4-group X O (3) O (4)

11. Example: Pen Study • Question: Do individuals in Japan and the US make differential choices about ‘unique’ versus ‘less unique’ items when given a choice?

12. Pen Study • Independent Variable • Cultural difference: Japanese students compared to US students • Assignment • Subjects were not randomly assigned because they already fell into one of the two groups. • Dependent Variable: • Pen layout (3 of one type, 1 of another) • Would they choose the ‘common’ pen or the ‘unique’ one?

13. Example: Trust-Building Study • Question: Do increased risk-taking behaviors over time increase interpersonal trust?

14. Trust-Building Study • Independent Variable • Experiment Condition (2 conditions): • Fixed partner on every trial, cannot control amount to entrust to partner • Fixed partner on every trial, can control amount to entrust to partner • Assignment • Random assignment of participants to one of the 2 conditions. • Same experiment conducted in Japan and US, and comparisons made between the two studies. • Dependent Variable • Cooperation rate (i.e., whether they returned the coins to the partner or not)

15. Measurement and Design Validity • Measurement Concerns • Construct Validity • Design Concerns • Internal Validity • External Validity • Ecological Validity

16. Construct Validity How do we know that our independent variable is reflecting the intended causal construct and nothing else? • “Face” validity deals with subjective judgement of appropriate operationalization • “Content” validity is a more direct check against relevant content domain for the given construct.

17. Internal Validity Internal Validity deals with questions about whether changes in the dependent variable were caused by the treatment.

18. ? ? Cause Effect ? ?

19. Threats to Internal Validity • History • additional I.V. that occurs between pre-test and post-test • Maturation • Subjects simply get older and change during experiment • Testing • Subjects “get used” to being tested • Regression to the Mean • Issue with studies of extremes on some variable

20. Contamination and Internal Validity • Demand Characteristics • Anything in the experiment that could guide subjects to expected outcome • Experimenter Expectancy • Researcher behavior that guides subjects to expected outcome (self-fulfilling prophecy)

21. General Demand Characteristics • Evaluation Apprehension • Solutions • Double-blind experiments • Experiments in natural setting (i.e., subjects do not know they are in an experiment) • Cover stories • Hidden measurements

22. Reducing the role of the experimenter: solving expectancy effects • Naïve experimenter • Those conducting study are not aware of theory or hypotheses in the experiment • Blind • Researcher is unaware of the experiment condition that he/she is administering • Standardization • Experimenter follows a script, and only the script • “Canned” Experimenter • Audio/Video/Print material gives instructions

23. And More! • Selection Bias • Issue with non-random selection of subjects • Mortality • Departure of subjects in the experiment • Diffusion, Sharing of Treatments • Control group unexpectedly obtains treatment • Other ‘social’ threats? • Compensatory rivalry, resentful demoralization, etc.

24. Three threats to external validity (generalizability) in experiments • Setting • Population • History External Validity– How far does the given experiment generalize to similar groups, individuals, etc?

25. Ecological Validity

26. The Validity Tradeoff: Truth and Myth Internal Validity External & Ecological Validity Balance is important between the types of validity, but internal validity is usually (if not always) the more important factor.

27. True Experiments in the Field • Some experiments can be conducted in a real-world setting while maintaining random assignment and manipulation of treatments Milliman (1986) Study of music tempo and restaurant customer behavior Cheshire and Antin (2008) Study of Incentives and Contributions of Information in an Online Setting

28. Natural Experiments • 1998 Total Solar Eclipse: testing temperature of sun’s corona

29. Pro’s and Con’s of Experiments • Pro’s • Gives researcher tight control over independent factors • Allows researcher to test key relationships with as few confounding factors as possible • Allows for direct causal testing • Con’s • Often very small N; enough for statistical purposes but not ideal for generalizability • Sometimes give up large amounts of external validity in favor of construct validity and direct causal analysis • Require a large amount of planning, training, and time– sometimes to test relationship between only 2 factors!

30. Additional considerations before using experiments • Cost and Effort • Is the effort worth it to test the concepts you are interested in? • Manipulation and Control • Will you actually be able to manipulate the key concept(s)? • Importance of Generalizability • Are you testing theory, or trying to establish a real-world test?