Response Biases in Survey Research

1. Response Biases in Survey Research Hans Baumgartner Smeal Professor of Marketing Smeal College of Business, Penn State University

2. Response biases when a researcher conducts a survey, the expectation is that the information collected will be a faithful representation of reality; unfortunately, this is often not the case, and survey researchers have identified many different sources of error in surveys; these errors may contaminate the research results and limit the managerial usefulness of the findings; if the response provided by a respondent does not fully reflect the “true” response, a response (or measurement) error occurs (random or systematic); response biases (systematic response errors) can happen at any of the four stages of the response process, are elicited by different aspects of the survey, and are due to a variety of psychological mechanisms;

3. The relationship between observed measurements and constructs of interest • The total variability of observed scores consists of trait (substantive), random error, and systematic error (method) variance. • Random and systematic errors are likely to confound relation-ships between measures and constructs and between different constructs. • They also complicate the comparison of means. T M E T1 T2 E2 E1 M1 M2

4. Outline of the talk • Misresponse to reversed and negated items • Item reversal and negation, types of misresponse, and mechanisms • Reversed item bias: An integrative model • Eye tracking of survey responding to reversed and negated items • Item grouping and discriminant validity • Extreme and midpoint responding as satisficing strategies in online surveys • Stylistic response tendencies over the course of a survey

5. The issue of item reversal Should reverse-keyed items (also called oppositely-keyed, reversed-polarity, reverse-worded, negatively worded, negatively-keyed, keyed-false, or simply reversed items) be included in multi-item summative scales? If reversed items are to be used, does it matter whether the reversal is achieved through negation or through other means? What’s the link between reversal and negation, what types of MR result, what psychological mechanisms are involved, and how can MR be avoided?

6. Item reversal vs. item negation • authors often fail to draw a clear distinction between reversals and negations and use ambiguous terms such as ‘negatively worded items’, which makes it unclear whether they refer to reversed or negated items, or both; • examples from the Material Values scale (Richins and Dawson 1992): • It sometimes bothers me quite a bit that I can't afford to buy all the things I’d like. • I have all the things I really need to enjoy life. • I wouldn't be any happier if I owned nicer things.

7. Item negation • items can be stated either as an assertion (affirmation) or as a denial (disaffirmation) of something (Horn 1989); • negation is a grammatical issue; • classification of negations in terms of two dimensions: • what part of speech is negated (how a word is used in a sentence: as a verb, noun/pronoun, adjective, adverb or preposition/conjunction); • how the negation is achieved (by means of particle negation, the addition of no, the use of negative affixes, negative adjectives and adverbs, negative pronouns, or negative prepositions);

10. Item reversal • an item is reversed if its meaning is opposite to a relevant standard of comparison (semantic issue); • three senses of reversal: • reversal relative to the polarity of the construct being measured; • reversed relative to other items measuring the same construct: • reversal relative to the first item • reversal relative to the majority of the items • reversal relative to a respondent’s true position on the issue under consideration (Swain et al. 2008);

11. Integrating item negation and item reversal

12. Misresponse to negated and reversed items MR → within-participant inconsistency in response to multiple items intended to measure the same construct;

13. Using reversed and negated items in surveys: Some recommendations although responding to reversed items is error prone, wording all questions in one direction does not solve the problem; negations should be employed sparingly, esp. if they do not result in an item reversal (note: negations come in many guises); polar opposite reversals can be beneficial (esp. at the retrieval stage), but they have to be used with care;

14. An integrative model of reversed item bias:Weijters, Baumgartner, and Schillewaert (2012) • two important method effects: • response inconsistency between regular and reversed items; • difference in mean response depending on whether the first item measuring the focal construct is a regular or reversed item; • three sources of reversed item method bias: • acquiescence • careless responding • confirmation bias

15. The survey response process (Tourangeau et al. 2000) Attending to and interpreting survey questions (careless responding) Comprehension Generating a retrieval strategy and retrieving relevant beliefs from memory (confirmation bias) Retrieval Integrating the information into a judgment Judgment Mapping the judgment onto the response scale and answering the question(acquiescence) Response

16. Empirical studies • (net) acquiescence and carelessness explicitly measured; • confirmation bias modeled via a manipulation of two item orders in the questionnaire, depending on the keying direction of the first item measuring the target construct; • three item arrangements: • grouped-alternated condition (related items are grouped together and regular and reverse-keyed items are alternated); • grouped-massed condition (items are grouped together, but the reverse-keyed items follow a block of regular items, or vice versa; • dispersed condition (items are spread throughout the questionnaire, with unrelated buffer items spaced between the target items);

18. Results for Study 2 both NARS (gNARS = .33, p < .001) and IMC (gIMC = .31, p < .001) were highly significant determinants of inconsistency bias; the effect of NARS on inconsistency bias was invariant across item arrangement conditions, as expected; the effect of IMC did not differ by item arrangement condition; the manipulation of whether or not the first target item was reversed (FIR) did not affect responses (although in the first study the effect was significantly negative); the effect of FIR did not differ by item arrangement condition;

19. Eye tracking of survey responding(with Weijters and Pieters) eye-tracking data may provide more detailed insights into how respondents process survey questions and arrive at an answer; eye movements can be recorded unobtrusively, and eye fixations show what respondents attend to while completing a survey;

20. Eye tracking study • 101 respondents completed a Qualtrics survey and their eye movements were tracked; effective sample size is N=90; • Design: • each participant completed 16 four-item scales shown in a random sequence; • the fourth (target) item on each screen was an RG, nRG, PO, or nPOitem (4 scales each);

21. Areas of interest AOI1a to AOI1e AOI3a AOI2a AOI3b AOI2b AOI3c AOI2c AOI3d AOI2d AOI5b AOI5a AOI4a AOI4b

22. Fixation durations for various AOI’s

23. Determinants of total fixation durationfor fourth item (logaoi23dplus1) Note: These results are based on a mixed model with respondent and construct as random effects.

24. Determinants of misresponse Note: These results are based on a mixed model with dist=gamma and construct as a random effect.

25. Item grouping and discriminant validity (Weijters, de Beuckelaer, and Baumgartner, forthcoming) • question whether items belonging to the same scale should be grouped or randomized: • grouped format is less cognitively demanding and often improves convergent validity; • random format may reduce demand effects, respondent satisfacing, and carryover effects, as well as faking; • effect of item grouping on discriminant validity: • grouping of items enhances discriminant validity (Harrison and McLaughlin 1996); • item grouping may lead to discriminant validity even when there should be none;

26. Method • 523 respondents from an online U.S. panel • questionnaire contained the 8-item frugality scale of Lastovicka et al. (1999) and 32 filler items; • frugality scale presented in two random blocks of 4 items each, with the 32 filler items in between • Condition 1: 1-2-3-4 vs. 5-6-7-8 • Condition 2: 1-2-7-8 vs. 3-4-5-6 • within blocks item order was randomized across respondents;

27. Estimated models

28. Results

29. Results (cont’d)

30. Results (cont’d)

31. Extreme and midpoint responding as satisficing strategies in online surveys(Weijters and Baumgartner) when respondents minimize the amount of effort they invest in formulating responses to questionnaire items by selecting the first response that is deemed good enough, they are said to be satisficing; when respondents put in the cognitive resources required to arrive at an optimal response, they are optimizing (Krosnick1991); the effectiveness of procedural remedies to prevent or at least reduce satisficing (MacKenzie & Podsakoff2012) is limited; post hoc indices designed to identify satisficersoften exhibit limited convergent validity and unambiguous cutoff values are often unavailable;

32. Satisficing in online surveys (cont’d) online surveys are likely to contain data from respondents who are satisficing, but what will be the consequences? we review satisficing and related measures that have been proposed in the literature and propose a new measure called OPTIM; we investigate the effect of satisficing on two stylistic response tendencies (ERS and MRS) and we demonstrate that the direction of the relationship varies across individuals;

33. The concept of satisficing the notion of satisficing is consistent with the view of people as cognitive misers (Fiske and Taylor 1991); satisficing is conceptually similar to carelessness, inattentiveness, insufficient effort responding, and content-nonresponsive, content-independent, noncontingent, inconsistent, variable or random responding; Krosnick (1991) argues that in weak forms of satisficing each of the four steps of the response process (comprehension, retrieval, judgment, response) might be compromised to some extent, whereas in strong forms of satisficing the second and third steps might be skipped entirely;

34. Measures of satisficing

35. Measures of satisficing (cont’d) a single-category measure is unlikely to assess satisficing adequately; direct measures of satisficing are desirable (esp. response time measures); bogus items and IMC’s have limitations; response differentiation for unrelated items might be a good outcome-based measure;

36. A new measure of satisficing • optimizing as the time-intensive differentiation of responses to items that are homogeneous in form but heterogeneous in content: OPTIM=log(TIME*DIFF) • survey duration: • input side of effort (indicator of the cognitive resources invested by a respondent); • time taken to complete the survey (in minutes), rescaled to a range of 0 to 10; • response differentiation: • output side of effort (indicator of optimizing for heterogeneous items); • DIFF = (f1+1)*(f2+1)*(f3+1)*(f4+1)*(f5+1), rescaled to a range of 0 to 10;

37. ERS and MRS as satisficing strategies • previous research suggests that both ERS and MRS may be used as satisficing strategies (even though ERS and MRS tend to be negatively correlated), although the empirical findings have not been very consistent; • different respondents may use different satisficing strategies: • some respondents may simplify the rating task by only using the extreme scale positions (resulting in increased ERS); • others may refrain from thinking things through and taking sides (resulting in increased MRS);

38. Method two online studies with Belgian (n=320) and Dutch (n=401) respondents; in dataset A 10 heterogeneous attitudinal items and in dataset B Greenleaf’s (1992) ERS scale; these items were used to construct the ERS (number of extreme responses), MRS (number of midpoint responses) and DIFF measures; survey duration was measured unobtrusively; use of a multivariate Poisson regression mixture model of ERS and MRS on OPTIM;

39. Model

40. Regression estimates by class

42. Dataset A Dataset B

43. Discussion • OPTIM as an unobtrusive measure that integrates several aspects of optimizing/satisficing; • across two distinct samples, three satisficing segments emerged: • extreme responders • midpoint responders • acquiescent responders • OPTIM is useful if a continuous measure of satisficing is required, but it may be less useful as a screening device for careless responders;

44. Stylistic response tendencies over the course of a survey (Baumgartner and Weijters) • three perspectives on stylistic responding: • nonexistence of response styles (complete lack of consistency); • instability of response styles (local consistency); • stability of response styles (global consistency); • Weijters et al. (2010) showed that • the nonexistence of response styles was strongly contradicted by the empirical evidence for both extreme responding and acquiescent responding; • there was a strong stable component in the ratings; and • there as a weaker local component (as indicated by a small time-invariant autoregressive effect);

45. Unresolved questions how do stylistic response tendencies evolve over the course of a questionnaire? prior research has only considered the effect of stylistic responding on the covariance structure of items or sets of items and has ignored the mean structure; are there individual differences in both the extent to which stylistic response tendencies occur across respondents and the manner in which stylistic response tendencies evolve over the course of a survey? prior research has not emphasized heterogeneity in stylistic response tendencies across people;

46. ALT model

47. Integrated ALT model for NARS and ERS

48. Method data from 523 online respondents; each participant responded to a random selection of eight out of 16 possible four-item scales shown on eight consecutive screens in random order; eight separate response style indices were computed for both (net) acquiescence response style or NARS (i.e., respondents’ tendency to express more agreement than disagreement) and extreme response style or ERS (i.e., respondents’ disproportionate use of more extreme response options); the design guarantees that there is no systematic similarity in substantive content over the sequence of eight scales across respondents;

49. Results

50. NARS and ERS trajectories