Bias in Political Communication Experiments

1. Bias in Political Communication Experiments Jamie Druckman & Thomas Leeper Dept. of Political ScienceNorthwestern University

2. Mass Communication Effects • Political communication research is “one of the most notable embarrassments of modern social science” (Bartels 1993). • “Compelling concepts that have had a major impact in political science and communications scholarship” (Iyengar2010). • Key method behind progress experiments (survey & lab). • Randomly expose some respondents to one message (e.g., hate group rally as a free speech issue)…. • …others to another message (e.g. rally as a public safety issue), and… • …measure the effect (e.g., support for the rally).

3. Mass Communication Effects • But what about time? What happens before and after the experiment? Does it matter? • Claim: Ignoring what happens prior to the experiment (i.e., “pre-treatment”) has produced a bias portrait of communication effects that may: • overstate the malleability of the public (see Barabas and Jerit 2010). • miss the identification of potentially two groups of citizens: malleably reactive and dogmatists. • contradict much macro opinion formation research.

4. Outline • Psychology of pre-treatment effects • Framing effects and defining pre-treatment • Attitude strength • Hypotheses  conditions when pre-treatment effects occur • Laboratory experiment • Manipulates pre-treatment environment and attitude strength (via processing mode) • Survey experiment • Election exit poll experimental survey • Measures pre-treatment environment and attitude strength correlate • Survey experiment • Conclusion

5. Framing • Framing effects  In the course of describing an issue, a speaker’s emphasis on a subset of relevant considerations causes individuals to focus on these considerations when constructing their opinions. • Example: Politicians, media frame hate group rally request as a: • free speech issue citizens focus on speech considerations  citizens support right to rally • public safety issue citizens focus on safety considerations  citizens oppose right to rally • Experiments  random exposure to framed communication as a news article or in question wording (e.g., on a survey).

6. Framing • Many other examples: • campaign finance (free speech or corruption?) • abortion (rights of mother or rights of unborn child?) • gun control (right to bear arms or public safety?) • affirmative action (reverse discrimin. or remedial action?) • welfare policy (humanitarianism or overspending?) • social security (individualism or shared security?) • elections (economy or foreign affairs?) • A Central Means of Elite Influence on Public Opinion. • These effects ≠ valence framing effects (Tversky and Kahneman)

7. Pre-treatment • Pre-treatment environmentcontext prior to exposing experimental participants to the stimulus (frame). • Pre-treatment effect aspect of the prior context affects responses to the stimulus (frame). • Example: • Prior to the hate group rally experiment, respondents view news coverage using the free speech frame and become more supportive…. • …another free speech exposure in the experiment does not further move opinion. • Mistaken conclusion of no effect. • Possibly don’t/can’t publish the experiment!

8. Pre-treatment • “Experimenters implicitly assume…that respondents enter the survey as clean slates… [despite the fact that] there is inevitably some possibility that respondents enter the experiment having already participated in a similar experiment, albeit one occurring in the real world” (Gaines et al. 2007).

9. Conditions for Pre-treatment Effect • Exposure/Attention Absent exposure and attention to information, there is no “pre-treatment.” • Durable Influence  • Absent initial influence, there is no “effect.” • Absent durability, the initial influence will not impact experimental reactions (i.e., the effect must sustain until the time of the experiment). • Depends on time between pre-treat and experiment. • Depends on nature of the attitude formed in response to the effect Attitude strength

10. Attitude Strength • Attitude Strength  attitudes that persist and resist change. • Stems from attitude features (e.g., extremity) and the formation/updating process. • Two dynamics that affect attitude strength: 1. Processing Mode 2. Need to Evaluate

11. Attitude Processing Mode • Individuals form/update attitudes in relatively more memory-based or online fashion (Haste & Parke 1986, Lodge et al.). • On-line (OL): • immediately integrate information (i.e., frames at time t in the pre-treatment environment) into an overall evaluative summary, • store it, and • recall it when needed (i.e., at t+1, in an experiment). • Thus, impact of earlier information sustains  attitudes formed at time t pre-treatment are stronger (i.e., durable, resistant change in time t experiment).

12. Attitude Processing Mode • Memory-based (MB): store information (frames) in memory, do not evaluate it until asked for an attitude at which point retrieve what can be recalled and integrate. • May not recall items from distant past. • Thus, impact of earlier information does not sustain  attitudes formed at time t pre-treatment are weak and information may be forgotten. • Pre-treatment effects are more likely to occur among OL processors since earlier effects sustain and generate resistance to later influence (i.e., in the experiment).

13. Need to Evaluate • Need to Evaluate (NE)  “individual propensity to engage in evaluation.” • More likely to assess and retain information (once exposed). Generate stronger attitudes, similar to OL processing. • Thus, impact of earlier information sustains  attitudes formed at time t pre-treatment are stronger (i.e., durable, resistant to change in time t experiment). • Pre-treatment effects are more likely to occur among high NE individuals since earlier effects sustain and generate resistance to later influence (i.e., in the experiment).

14. Hypothesis Pre-treatment effects (e.g., leading to no experimental stimulus effect) will be more likely to occur when individuals are: • exposed and attentive to earlier communications similar to the experimental stimuli; and • form/update their attitudes in ways that promote strength. This occurs among on-line processors and high NEs. Motivated Reasoning Corollary • Motivated reasoning  reject information that is inconsistent with prior opinions. • Those with stronger attitudes are more likely to engage in motivated reasoning. • Those with strong attitudes (i.e., OL, high NE) not only will not be influenced by a “repeated communication” but may reject contrary communication.

15. Experiment 1 • Laboratory experiment with 744 participants (mostly students), Spring 2010. • Two Issues (both salient but not currently intensely debated) • Support for the Patriot Act (increases law enforcement power to combat terrorism) (measured on 7-point scale). • Support for a state owned gambling casino (measured on 7-point scale).

16. Experiment 1 • Pro/Con Strong Frames (in news articles): • Patriot Act: • Pro  Protection from terrorism • Con  Violation of Civil Liberties • Casino: • Pro  Economic benefits (e.g., tax relief) • Con  Social costs (e.g., addiction, debt)

17. Experiment 1 • Manipulated OL or MB processing mode (using conventional psychological approach) • OL respondents instructed to evaluate articles for their impact in increasing or decreasing support. Told they will later report opinions. • MB  respondents instructed to evaluate articles for their “dynamic” nature (i.e., use of action-oriented words).

18. Experiment 1 Procedure  Four waves, 5 days apart. • Background survey, assigned to condition that varied: (a) pre-treatment environment, (b) processing mode, (c) survey frame, AND --received two relevant pre-treatment frames, along with processing manipulation. • Received pre-treatment frame articles, along with processing manipulation. • Same as #2. • Received survey question using no, con, or pro frame.  virtually no attrition because compensation contingent on full completion.  virtually no coverage of these issues during the experiment or several months prior.

19. Conditions Expect survey frames will have scant effects in the OL conditions, but will impact MB processors (and the no-pre-treated individuals).

20. Patriot Act Survey Frames • Control “The Patriot Act was enacted in the weeks after September 11, 2001, to strengthen law enforcement powers and technology. What do you think—do you oppose or support the Patriot Act? Choose one number on the following 7-point scale.” • Con “…technology. Under the Patriot Act, the government has access to citizens’ confidential information from telephone and e-mail communications. As a result, it has sparked numerous controversies and been criticized for weakening the protection of citizens’ civil liberties…” • Pro“…technology. Under the Patriot Act, the government has more resources for counterterrorism, surveillance, border protection, and other security policies. As a result, it enables security to identify terrorist plots on American soil and to prevent attacks before they occur…”

21. Casino Survey Frames • Control “A proposal is being considered for the Illinois state government to operate a land-based gambling casino. What do you think—do you oppose or support the proposal for a state-run gambling casino? Choose one number on the following 7-point scale.” • Con “…Some say that a state-run casino will have severe social costs, such as addiction and debt..” • Pro “…Some say that the revenue from the casino would provide tax relief and help to fund education…”

22. Results • Simple mean comparisons, robust to controls. • Similar results for two issues; here present Patriot Act results. • Gradual disaggregation of conditions…

23. Patriot Act Results • Strong survey framing effect.

24. Patriot Act Results Non-manipulated • Strong survey framing effects for non-manipulated and MB. • NO SURVEY FRAMING effects for OL. • Effects apparent in merged data stem entirely from non-manipulated and MBs.

25. Patriot Act Results Non-manipulated • Non-manipulated = MB. • Aggregate results may stem from particular sub-groups. • Aggregate effect sizes underestimate impact on affected groups (e.g., MB moved .10 more than merged data from pro frame).

26. Patriot Act Results • Non-effect among OL is illusionary!

27. Patriot Act Results • Pre-treatment effects: • Pro pre-treatment frames significantly increased support, regardless of survey frame. • Con pre-treatment frames significantly decreased support, regardless of survey frame. • Repeated frame in survey had minimal impact, and contrary frame was rejected (motivated reasoning).

28. Patriot Act Results • No evidence of pre-treatment effects for MB processors. • Significant survey framing effect in each case, regardless of the pre-treatment environment.

29. Patriot Act Results: Belief Importance ***p£.01; **p£.05; *p£.10 for one-tailed tests, relative to OL Processors in contrary pre-treat environment and received survey frame condition. • OL processors who received a survey frame contrary to their pre-treatment environment, viewed that argument as significantly less important  evidence of motivated reasoning (rejection of contrary information). • Are these types of effects evident outside of manufactured lab setting?

30. Experiment 2 • Exit poll experimental survey that measures pre-treatment environment and attitude strength correlate, with 338respondents. • Issue: Support for a proposed state owned gambling casino during the 2006 IL Gubernatorial campaign (measured on a 7-point scale). • Tracked media (i.e., pre-treatment) environment: • Pro-casino frames: economic benefits, (entertainment). • Con-casino frames: social costs (corruption, morality). • Exit poll survey experiment. • Measure attention and likelihood of enduring opinion (via Need to Evaluate item).

31. The Campaign(pre-treatment environment) • 2006 IL Gubernatorial pitting Blagojevich (D) vs. Topinka (R). • Aug. 23: Topinka proposes state owned casino to raise revenue (t = 1). • Aug. 24: Chicago Tribune Comment: Topinka is “framing the contest… just as it needs to be framed: How can a grossly overcommitted state gov. bend financial trend lines that point inexorably toward ruin.” • Sept. 9: Corruption accusation against Blagojevich of taking a personal payoff for a state job. • Others follow! • Nov. 7: Election Day (t = n). • Coded campaign coverage in the Chicago Tribune from t = 1 (Aug. 23rd) to t = n (Election Day). • Most prominent issues: economy (budget) & corruption.

32. The Campaign • Early campaign discussions of the casino were framed in terms of economic benefits > 75% of the time.

33. Predictions • Attentive high NEs will be significantly more likely to form and maintain casino opinions upon exposure to the early campaign information (e.g., with economic frame focus).  • Less susceptible to the economic frame later (already influenced) and the social costs frame (reject it). • More supportive of the casino proposal.

34. Exit Poll • Election Day exit poll (t = n). • Random sample of polling stations in north Cook County. ($5 for participation). • Measured NE (i.e., 1 item measure) & Campaign Attention (i.e., newspaper reading during campaign period), etc. • Attentive voters  greater knowledge, discussion, interest in politics. • Attentive / High NE voters  above median on NE and attention (n = 111). • Four relevant experimental conditions: • No frame – asked extent of casino support. • Economic frame  pro casino. • Social costs of gambling  con casino. • Economic-Social costs  dual frame.

35. NE Measures Some people have opinions about almost everything; other people have opinions about just some things; and still other people have very few opinions. What about you? Would you say you have opinions about almost everything, about many things, about some things, or about very few things? almost many some very few everything things things things Compared to the average person, do you have a lot fewer opinions about whether things are good or bad, somewhat fewer opinions, about the same number of opinions, somewhat more opinions, or a lot more opinions? a lot fewer somewhat about the somewhat a lot more opinions fewer opinions same more opinions opinions Some people say that it is important to have definite opinions about lots of things, while other people think that it is better to remain neutral on most issues. What about you? Do you think it is better to have definite opinions about lots of things or to remain neutral on most issues? definite remain opinions neutral

36. Results • All = Large survey framing effect. • Non-attentive/Low NE  large survey framing effect.

37. Results • Attentive/High NE  NO survey framing effect. Significantly higher support across conditions. • Pre-treatment effects  No experimental effects but was earlier influence. • Other evidence suggests motivated reasoning for those receiving social costs frame.

38. Summary • The existence of an experimental effect can be misleading as it: • may stem from a subgroup that formed weak attitudes (e.g., MB) on the issue (Gaines et al. 2007, Barabas and Jerit 2010), and • may understate the effect size among those individuals. • The non-existence of an experimental effect can be misleading as it: • may stem from a large number of individuals forming strong attitudes (e.g., OL) in response to communications prior to the experiment. • Such individuals were limited in our studies, but may be pervasive in other contexts. • Hillygus and Jackman (2003)  conventional effects > debate effects.

39. Summary • Given publication biases (e.g., Gerber et al. 2010), experimental studies may over-state the existence of effects, and thus: • The mass public, on average, is less malleable and holds more stable opinions than would be suggested by the aggregation of experimental results. • The mass public may be bi-modal  malleably reactive and dogmatically invulnerable. • Caveat those who formed strong attitudes were affected by earlier stimuli.

40. Summary • Varying levels of stability in macro and micro-level studies of opinion may stem, in part, from different issue foci: • Macro studies  longstanding salient issues that generate stronger attitudes (e.g., Gallup’s most important problem surveys) • Micro over-time studies  relatively novel and specific issues (e.g., ballot proposition, new candidate, regulation of hog farms, campaign finance).

41. Conclusions • Opinions are not fixed in time. Time dynamics need study  priors, pre-treatment, durability. and post-treatment effects. • Failure to account for these dynamics  inferential errors. • Precise effects depend on attitude strength. • What should public opinion researchers do? • Define time period of study (as a unit of analysis). • If goal is to evaluate impact of an argument, test for pre-treatment effects. • Identify prior rhetorical context. • Test with distinct populations or times. • Develop theories of over-time effects

42. END

43. Casino Results

44. Casino Results

45. Casino Results

46. Casino Results