Analyzing Experimental Data

1. Analyzing Experimental Data

2. Shouldn’t Design Be Enough? • use of manipulation, control, & randomization should be enough – data speak for itself? • Problem – theories no longer simple a causes b hypothesis testing. • Complexity of theory means optimal design can be computationally difficult. • Material presented here on optimal design is mainly about how to design an experiment to test a “complete” theory – one that can be expressed in a maximum likelihood function for data observed.

3. On Optimal Experiment Size • Issues ignored in analysis – usually data generated from experiments can have a use beyond experiment & for other currently unimagined uses • There is a fixed cost to running an experiment that should be considered, analysis is only on marginal costs.

4. Sources for Material on Optimal Design • El-Gamal, Mahmoud A., Richard D. McKelvey, & Thomas R. Palfrey, “Computational Issues in the Statistical Design and Analysis of Experimental Games,” International Journal of Supercomputer Applications, vol. 7, no. 3, fall 1993, 189-200. • El-Gamal, Mahmoud A. & Thomas R. Palfrey, “Economical Experiments: Bayesian Efficient Experimental Design,” International Journal of Game Theory, 1996, 25:495-517.

5. Standard FTT Design • Researcher decides ex ante (w/o any statistical analysis) an interesting experiment to run. • Then attempts collect as much data as financially feasible, non-statistically considering • importance of experiment as piece of research agenda, • how high payoffs need to be to make subjects interested in making optimal decisions, • length of time one can keep subjects in laboratory, etc.

6. Standard FTT Design • After data collected, researcher approaches same way as non-experimental data – hypothesizing form of data generating process (either structural of reduced form), & proceeding with estimation & hypothesis testing. • However, as noted earlier, “point” predictions in most FTT fail, because most game theoretic models suffer from zero likelihood problem. • Possible observe data our model predicts could never happen (e.g. subjects choosing strictly dominated strategies).

7. Zero Likelihood Problem • Most game theoretic experiments ignore this issue & concentrate on relationship predictions, using standard hypothesis testing techniques. • But ideally, better to devise theoretical model which allows for statistical estimation of a likelihood function – true structural estimation. • Zero likelihood problem means a failure of theory, theory should be corrected. • For decision-theoretic situation not that difficult, just add some random error to individual’s behavior (what implicitly do in regression equations explaining individual behavior)

8. Zero Likelihood Problem • Most game theoretic experiments ignore this issue & concentrate on relationship predictions, using standard hypothesis testing techniques. • But ideally, better to devise theoretical model which allows for statistical estimation of a likelihood function – true structural estimation.

9. Zero Likelihood Problem • Adds huge computational complexity to solving games. • One solution is Quantal Response Equilibrium (QRE) concept devised by McKelvey & Palfrey. • McKelvey, R.D. and Palfrey, T.R. (1995). “Quantal Response Equilibria in Normal Form Games.” Games and Economic Behavior. 7, 6–38.

10. Problems w/ Standard FTT Design • While QRE or something similar (see other references in Methods & Models) can solve zero likelihood problem, still disconnect between theoretical purpose of experiments & standard design of FTT experiments. • Experiments are costly. • Some experimental designs may discriminate between models so poorly as to render them useless. • Why not attempt to get most bang (statistical result) for bucks . . .

11. Optimal Experimental Design: Step 1 • 2 game theoretic models, both solve zero likelihood problem & applied using maximum likelihood methods. • Class of experiments parametized by some vector . • Typically  correspond to payoff structures. • Can, from likelihood functions compute the Kullback-Liebler information number which measures how informative given design is expected to be if model were correct.

12. Kullback-Liebler Information # • Let X be space of all possible data sets under all designs proposed. • Denote a typical data set by x. • Let the likelihoods of a given data set x  X under design  for each of n competing models be l1(x;), l2(x;), . . . ln(x;).

13. Kullback-Liebler Information # • Given a collection of priors on models 1, 2, . . . , n, say p1, p2, . . . Pn, we can define for each model the Kullback-Liebler information number measuring how much information a given design is expected to be if that model were correct.

14. Kullback-Liebler Information # • Information number of model 1 under design  is:

15. Kullback-Liebler Information # • Design that maximizes expected separation between model 1 & other n – 1 models, if model 1 were indeed correct model, is

16. Kullback-Liebler Information # • If want to maximize overall informativeness of design, weight information numbers by prior on each of the models & choose

17. Optimal Experimental Design: Step 1 • El-Gamal, McKelvey, & Palfrey give algorithm to do this maximization problem in “Computational Issues . . .” • Required Cray computer in early 1990s . . . • Give example in paper – pretty complex – would be nice to be able to translate this to more political science context – but it may be that computationally this is too difficult? • Most FTT & PTT still use ad hoc approach on payoffs (including Cal Tech guys).

18. Optimal Experimental Design: Step 2 • Still problem – how many observations need? • Need stopping rule . . . • Assume chosen  & have two rival models. • EMP propose stopping rule belongs to family of Wald’s sequential probability ratio tests (SPRTs)

19. Optimal Experimental Design: Step 2 • Assume loss of selecting correct model 0 & cost of selecting wrong model K. • SPRT has property of minimizing expected sample size (& hence expected cost of set of experiments) in class of all tests with same type I & type II error probabilities.

20. Optimal Experimental Design: Step 2 • SPRTs take following form: • Continue sampling until likelihood ratio between two models crosses one of two boundaries. • If upper boundary is crossed, accept model whose likelihood appears in numerator of likelihood ratio, & if lower boundary is crossed, accept other model.

21. Optimal Experimental Design: Step 2 • A number of approximations to compute optimal stopping boundaries have been proposed. • Most popular involve using Wald’s approximation of type I & type II error & expected stopping time.

22. Optimal Experimental Design: Step 2 • Berger suggests a further approximation based on cost per experiment c being much smaller than loss of selecting wrong model K. • Resulting rule sets boundaries of A & B. • Let  be prior on model 1 being correct & I(I) information number for model i as calculated above.

23. Optimal Experimental Design: Step 2 • Stop & accept model 1 if likelihood ratio of model 1 to model 2 is greater than B, stop & accept model 2 if likelihood ratio is less than A, & continue sampling otherwise.

24. Optimal Experimental Design: Step 2 • Again, rarely do experimentalists do any of this. • Why?

25. Ethical Issues in Experimentation

26. Are Ethical Issues in Experiments Different? • Academics “mess” with real world all the time • As teachers – can influence students view of world – ethics of teaching rational choice • As researchers in publishing work – newspapers pick up conclusions, general public believe them . . . • As policy advisors – policy makers actually solicit advice under assumption more knowledgeable, principal agent issues.

27. How Are Experiments Different? • Transparency of motives • Even if not being deceptive, usually not “honest” about motives • When deception is involved, potentially more serious.

28. How Are Experiments Different? • Effect of experiment? • In experiments in lab with subjects “paid” can argue “less” of an effect than those where influenced for other reasons (I.e. please teacher, see advisor as having more information) • However, not so true in field experiments – informed consent issues

29. How Are Experiments Different? • Inducing of real decisions • In lab or field, experimenter induces subjects to make real decisions may not make if experimenter did not intervene • Students can tell teacher what s/he wants to hear, but don’t necessarily have to alter real world behavior. • Same for readers of journal articles or policy makers who solicit advice.

30. How Are Experiments Different? • Inducing of real decisions • However, not option for subject in experiment. • Moreover, in making these decisions subjects may learn something about themselves not like (defect in PD, unacknowledged racist preferences)

31. Should We Experiment? • Consider Cost/benefit approach • Theoretically if benefits > costs, worth doing.

32. Problems with Benefits • Benefits theoretical – posit experiments will reveal important information based on priors, but could be wrong, even if design is “optimal” • Benefits in politics “debatable” – I.e. do not have consensus about many of things study – debate over normative implications of political processes means perceived benefits depend on normative preferences of researcher.

33. Problems with Benefits • Politics involves real political outcomes & extent experiments involve real world of politics, benefits may mean winners & losers in real world.

34. Problems with Cost/Benefit Approach • Costs also difficult to measure or perceive – as discussed above. • Alternative approach – • Decide a priori some costs not worth bearing regardless of benefit • Provide transparency as much as possible both in lab & field • Acknowledge & incorporate potential disagreement over theoretical benefits

35. The FutureEITM: Experimental Implications of Theoretical Models What Should be the Relationship between Experimental and Theoretical Work in Political Science?

36. Experimental Work within Political Science, The Present • Some small # of researchers do game theory testing (FTT for formal theory testing) • Larger # do political psychology (usually trained like/as social psychologists) (PTT for psychological theory testing) • Newer atheoretical experimental work (ATT for anti theory testing)

37. Experimental Work within Political Science, The Present • Very small percentage of overall empirical research in political science • Modal political science department does not have an experimentalist & if they have one it is a PTT type

38. Experimental Work within Political Science, The Present • In fact, FTT work is stagnant (maybe declining within discipline?) • While PTT is focusing on ways to increase external validity (moving the lab to the mall, making experimental situation in lab more realistic, CATI survey experiments) • & ATT work is expanding

39. Experimental Political Science outside the Discipline, The Present • # of experimental economists look at political science related questions (FTT) • Some psychologists (usually more “communications” scholars) do political science questions (PTT) • Occasionally gets in political science journals but not generally read or cited within political science

40. In Contrast, within Experimental Economics • While are few with psychology backgrounds, take it as given that common economics model & role of game theory & PTT in economics looks much closer to FTT, more communication between groups (PTT in minority) • Modal economics department still doesn’t have an experimental economist – but top departments are hiring them & tend to be FTT types • No ATT that I know of . . .

41. Why the Difference?Theory 1 – Political Science is Economics Delayed • Evidence in favor: • FT work in PS expanding to new problems, becoming important in areas not “colonized” like the presidency, comparative, etc. • More FT training in graduate programs

42. Why the Difference?Theory 1 – Political Science is Economics Delayed • But several observations against: • Demand, renewed, that FT work be FTT work – general & pervasive unwillingness to accept FT without additional T • FTT experimental work has not increased (maybe declined) & more externally valid PTT & ATT are increasing (not high rates, but not declining as FTT)

43. Theory 2 – Political Science has its own Path • Traditionally, political scientists have emphasized context of data studied unlike psychologists or many economists • Most models are different (more institutional detail) • Much more emphasis on empirical relevance of models • Emphasis on non-experimental data over experimental data because of “external validity”

44. Theory 2 – Political Science has its own Path – Implications • No perceived need for political science as a discipline to replace game theoretical economists or psychologists, both of whom study human behavior in general senses • In fact, some FT work by political scientists seeks publication in economics journals & economists to work with (similar behavior among Political Psychologists) • Experimental work that has more perceived external validity is always viewed as more useful to political scientists

45. Problems • Knowledge trapped – few political scientists know or understand developments in FTT & PTT outside of discipline. • For example, even though grad students may have FT training, rarely refers to FTT experimental work. • Methods classes may present some experimental work, but typically PTT & by political scientists • Persistent misunderstanding of the flexibility of FTT experimental work to address external validity issues.

46. FTT & External Validity Issues • False assumption of many political scientists • Only way to deal with real political world is to use either non-experimental data, field experimental data or highly realistic PTT generated data. • But when working with FTT can add, in a controlled fashion, external validity. • Means that FTT experimental work in political science needs to be different from that in economics.

47. Proposed Game Plan for FTT Experimental Political Science • Take an existing popular formal theory in political science – Baron/Ferejohn Legislative Bargaining game • Traditional experimental economics approach – strip to bare bones & test basic behavior predicted (combination of ultimatium & dictator games) • Still need this type work • But need to take this & build in institutional & subject pool detail (on gradual basis) so can see where disconnects between theory & empirical world matter

48. Proposed Game Plan for FTT Experimental Political Science • Requires a team approach over an individualistic or even small group approach • Interactions with PTT – incorporating some of methods of PTT that increase external validity to testing explicit FT