Survey Experiments: Illuminating Theoretical and Methodological Controversies

1. Survey Experiments: Illuminating Theoretical and Methodological Controversies Harold Clarke EPPS, University of Texas at Dallas email: clarke475@msn.com

2. Utility of Survey Experiments • Survey experiments – increasingly popular over past decade • Sold as way to study causal processes that cannot be investigated with traditional cross-sectional & panel surveys • Randomization as cure for omitted variable biases & endogeneity problems that bedevil causal inference in observational studies • Survey experiments – external validity - sophomores in labs are not representative! • Other problems – truncated variance, small N’s, functional form forgotten by ANOVA, model specification

3. Survey Experiment Vehicles • In-person probability surveys – ANES, BES, GSS • Mail-back surveys • RDD telephone surveys – impact of TESS – NSF funds • Internet surveys – general populations - BES Continuous Monitoring Survey (CMS), BES Rolling Campaign Panel Survey (RCPS), now ANES • Internet surveys – specific populations – Survey Monkey, Qualtrics -> e.g., students, employees, patients, party members • Amazon Mechanical Turk too!

4. Internet Survey Experiments • Internet methodology – exciting possibilities • Huge N’s – massive statistical power • Inexpensive – vastly expand research opportunities – e.g., CMS • Technical tools – video, audio, feedback to respondents, intricate randomizations

5. Theory & Method • Often think of experiments as theory or, at least, hypothesis testing • Hypotheses of interest may concern measurement of key concepts • Survey measures may be affected by context in which questions are asked • Internal – question ordering effects • External – political/economic conditions • Gaines, Kuklinski & Quirk (2007) discuss the latter in terms of survey experiments, but it is of general interest • Use survey experiments to investigate how external context affects survey responses or relationships among variables – cross-level interactions

6. External Context Survey Experiment: Inglehart and Post-Materialism • Testing the impact of macro-economic context on responses to Inglehart’s Materialism-Postmaterialism question battery in the Euro-Barometer surveys • Does Inglehart’s battery really measure what he says it does? • Or, is it contaminated by changing economic context • Why does this matter?

7. Inglehart’s Value Shift Thesis & His Values Battery Inglehart famously argued there has been a “value shift” from materialist to postmaterialism driven by rising affluence in post-war Western democracies. Everybody jumped on this bandwagon! Four-Item Values Battery administered in bi-annual Euro-Barometer surveys Items: • Fight Rising Prices • Maintain Order in the Nation • Give People More Say in Government • Protect Freedom of Speech Respondents asked to select the most important, then select the next most important of the 3 remaining items.

8. Figure 1. Why The Battery Might Be Affected by Macro-Economic Context

9. Figure 2. Germany – Canonical Case

10. Values Battery Survey Experiment • Clarke’s Chance Encounter at Zuma • Dinner With Max Kaase • What are you working on Harold? • Studying how Macro-Economic Conditions Affect Responses to Inglehart’s Values Battery • Nice aggregate results, but need to nail it at the individual level • Survey experiment! • Parallel work being done by Kaase in Germany • Let’s work together!

11. The Experiment • National Samples of Canadian & German Electorates • Split-Sample Design • Half-One gets traditional Values Battery; Half-Two gets Values Battery with Unemployment item substituted for Inflation Item • At end of Canadian survey, ask other battery – Do people change their value classification in a 15-minute survey?! • Results?

12. Figure 3. Alternative Values Batteries, Responses - Canada

13. Figure 4. Materialists & Post-MaterialistsRival Values Batteries - Canada

14. Results & Implications • Same results in Germany! • Significant portion of the hypothesized value shift an artifact of interaction between his measure and changing macroeconomic context • Example of Gaines’ et al. caveat in their Political Analysis article • Simple experiments can have big payoffs

15. Three Other Examples of Survey Experiments • The Dynamics of Partisanship – Does party identification have an individual dynamic? One aspect of controversy concerns question wording – A task for survey experiments! • Does Mode Matter? - Do Traditional In-Person and Internet Election Studies Tell the Same Story? • The Endogeneity of Ideal Points? – Leveraging the Power of the Internet With a Feedback-to-Respondent Experiment to determine if “De Gustibus” is a viable psephological anthropology.

16. Dynamics of Partisanship • Longstanding Controversy! • Campbell et al. v. Fiorina et al. • Green et al. • In Canada, Richard Johnston & Andre Blais said if only we changed the “or what?” in first ANES party id question to “or none of these” we would find: • 1. fewer identifiers • 2. more stable identifiers

17. Survey Experiment on Stability of Party Identification in Canada • Split Half Samples in 2000-2002 Political Support in Canada Panel Surveys • One Half Gets Traditional Wording; Other Half Gets Revised Wording • Results: • Slightly fewer identifiers with revised wording • Rates of partisan stability almost exactly the same • Lots of partisan instability in both versions • Survey experiment results reinforce MMLC analyses of US, UK, Canadian and German multi-wave panel data Clarke & McCutcheon POQ 2009 and Stegmueller, Neundorf and Scotto, POQ 2011.

18. Figure 8. Dynamics of Party Identification, 2000-2002 PSC Panel: Stable Identifiers Traditional and Revised Question Wordings

19. Does Mode Matter? Are Ideal Points Endogenous? • Mode – see modecomp1.ppt • Ideal Points – see endogjepoprev.pdf

20. Conclusions I • Survey experiments are useful tools! • Survey experiments can address big theoretical & methodological controversies • Internet surveys put survey experiments “in reach” for many researchers • CCAP, CCES, CMS – example vehicles • Qualtrics – sophisticated platform – many universities subscribe – no cost

21. Conclusions II • Survey experiments not a panacea • Example of ongoing controversies over implications of endogenity of economic evaluations in party support and voting models • Never resolve controversy with traditional cross-sectional or panel survey designs! • Survey experiments very unlikely to simulate real economic conditions • Way forward - use CMS monthly surveys 1997- 2010 & VEC modeling – see Snark Paper • Data Driven Solutions • Not Necessarily Expensive