1 / 28

Survey Experiments

Survey Experiments. Defined. Uses a survey question as its measurement device Manipulates the content, order, format, or other characteristics of the survey as a treatment. Methodological Issues. Missing Data Matching Both can be an issue in experiments other than surveys. Missing Data.

ivor-cross
Download Presentation

Survey Experiments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Survey Experiments

  2. Defined • Uses a survey question as its measurement device • Manipulates the content, order, format, or other characteristics of the survey as a treatment

  3. Methodological Issues • Missing Data • Matching • Both can be an issue in experiments other than surveys

  4. Missing Data • Some observations missing data on the DV or IVs • If missing at random, not a problem to drop from the analysis • But usually not missing at random • Deleting non-random missing causes bias

  5. Missing Data II • Data can also be missing intentionally: • Some cases not “treated” • Possible to “guess” what would have happened to a subject had they been in another treatment group • Allows within-subject comparison of two treatments, the one they received and the one they could have received

  6. Solution: Imputation • Suppose Yi = a + b1Xi1 + b2Xi2 +ei • But Yi missing for some observations • Xi1 and Xi2 not missing • Regress Y on Xi1 and Xi2 for all non-missing observations • Use b1 and b2 to calculate predicted Ypi

  7. Better Yet: Multiple Imputation • Ypi is a predicted value with uncertainty • Multiple imputation predicts multiple values for Ypi drawn from a distribution of predicted values • 5 or so predicted Ypi sufficient for inference, no need for many • Gary King’s Amelia program available free on-line

  8. Matching • Experiments can be pre-matched to avoid large random sample • Match subjects on important characteristics such as • Sex • Race • Age • Education levels • Other traits?

  9. Matching • Often necessary in field experiment when randomization more difficult to control • propensity scoreis the probability of an observation being assigned to a particular treatment in a study given a set of known variables. • Propensity scores reduce selection bias by equating groups based on these variables

  10. A Theory of Nonseparable Preferences in Survey Responses

  11. Question • Why do people change their answers to survey questions if the order of questions changes? • Does changing survey responses indicate that people do not have well-formed opinions

  12. Theory • Nonseparable Preferences: What a person wants on one issue depends on what she gets on another issue • Separable Preferences: What a person wants on every issue is independent of what they get on other issues

  13. Measuring Nonseparable Preferences

  14. Method • Randomize the order of pairs of survey questions • For some issues, aggregate responses different across question order • Each subject answers questions in order • Issue 1 then Issue 2 • Issue 2 then Issue 1

  15. Method • Impute what subject would have answered had they heard questions in different order • For each question we then have Yi (if first) – Yi (if second) • One of these will be imputed for each person since they cannot answer a question both first and second in the order • First study to analyze individual differences in question orders, not simply aggregate differences

  16. Conclusions • Nonseparable preferences explain question order effects • Political information level does not • Response instability not due to uninformed respondents

  17. Are Survey Experiments Externally Valid? JASON BARABAS and JENNIFER JERIT American Political Science Review 2010

  18. Question • Many survey experiments expose subjects to different information to show effect of on responses • In a survey experiment, subjects are a “captive audience” that must pay attention • Do the same information effects appear in the real world • Compare survey experiments with natural experiments

  19. Method • Survey experiments give people to political information about immigration and medical care • Pre-post survey also in field during change in medical insurance and immigration • Ask respondents which media sources they use • Is the effect of information in the survey experiment as large as in the natural experiment?

More Related