Misresponse to Reversed and Negated Items in Surveys: A Review

1. Misresponse to Reversed and Negated Items in Surveys: A Review Bert Weijters Hans Baumgartner

2. Motivation • Should reversed items be included in multi-item summative scales? • The use of reversed items in surveys often leads to problems (low reliabilities, poor fit, distorted factor structure); • Reversed items control for acquiescence, serve as cognitive “speed bumps” and may encourage more complete coverage of the construct’s domain of content; • If reversed items are to be used, does it matter whether the reversal is achieved through negation or through other means? • What’s the link between reversal and negation, what types of MR result, what psychological mechanisms are involved, and how can MR be avoided?

3. Item reversal vs. item negation • Authors often fail to draw a clear distinction between reversals and negations and use ambiguous terms such as ‘negatively worded items’, which makes it unclear whether they refer to reversed or negated items, or both; • Examples from the Material Values scale (Richins and Dawson 1992): • It sometimes bothers me quite a bit that I can't afford to buy all the things I’d like. • I have all the things I really need to enjoy life. • I wouldn't be any happier if I owned nicer things.

4. Empirical data we analyzed items from volumes 1 through 36 of JCR (1974 till the end of 2009) and volumes 1 through 46 of JMR (1964 to 2009); we included all Likert-type scales for which the items making up the scale were reproduced in the article and factor loadings or item-total correlations were reported; total of 66 articles in which information about 1330 items measuring 314 factors was provided; of the 1330 distinct items in the data set, 608 came from JCR and 722 from JMR;

5. Item negation • Items can be stated either as an assertion (affirmation) or as a denial (disaffirmation) of something (Horn 1989); • Negation is a grammatical issue; • Classification of negations in terms of two dimensions: • what part of speech is negated (how a word is used in a sentence: as a verb, noun/pronoun, adjective, adverb or preposition/conjunction); • how the negation is achieved (by means of particle negation, the addition of no, the use of negative affixes, negative adjectives and adverbs, negative pronouns, or negative prepositions);

8. Item reversal • an item is reversed if its meaning is opposite to a relevant standard of comparison (semantic issue); • three senses of reversal: • reversal relative to the polarity of the construct being measured; • reversed relative to other items measuring the same construct: • reversal relative to the first item • reversal relative to the majority of the items • reversal relative to a respondent’s true position on the issue under consideration (Swain et al. 2008);

9. Item reversal (cont’d) in our data set of 1330 items, between 83 and 86 percent of items were nonreversed (depending on the definition of reversal); the proportion of factors (or subfactors in the case of multi-factor constructs) that do not contain reversed items was 70 percent; only 8 percent of factors (out of 314 factors) were composed of an equal number of reversed and nonreversed items (i.e., the scale was balanced);

10. Integrating item negation and item reversal

11. Cross-classification of negation and reversal

12. Misresponse to negated and reversed items MR → within-participant inconsistency in response to multiple items intended to measure the same construct;

13. Theoretical explanations of MR:Comprehension • Careless responding (Schmitt and Stults 1985): • respondents fail to pay careful attention to individual items and respond based on their overall position on an issue [RMR]; • more likely when a reversed item is preceded by a block of nonreversed items; • Remedies: • alert Rs to the presence of reversed items, avoid tedious surveys, and be careful with involuntary Rs; • increase Rs’ ability and avoid distractions; • use balanced scales, alternate the keying direction, and disperse the items;

14. Theoretical explanations of MR:Comprehension (cont’d) • Reversal ambiguity: • Rs may not view antonyms as polar opposites [POMR]; • contradictories vs. contraries: • Antonyms can be contradictories or contraries, depending on whether they are bounded or unbounded (Paradis and Willners 2006); • Negations of the same core concept are clearly contradictories, but things are more complicated when the core concept differs; • simultaneous disagreement is more likely when items are worded extremely (McPherson and Mohr 2005); • “Buddhism’s ontology and epistemology appear to make East Asians relatively comfortable with apparent contradictions” (Wong et al. 2003, p. 86) [RMR];

15. Theoretical explanations of MR:Comprehension (cont’d) • Remedies: • use more sophisticated procedures to identify appropriate antonyms (formulate linguistic contrasts in two stages; see Dickson and Albaum 1977); • may be particularly useful in cross-cultural research; • bounded antonyms have to be pretested and unbounded antonyms have to be used with care; • extreme statements should be avoided;

16. Theoretical explanations of MR:Retrieval • Item-wording effects: • Confirmation bias (Davies 2003; Kunda et al. 1993); • Directly applicable to antonymic reversals; • For negation reversals, confirmation bias can lead to MR if a non-negated polar opposite schema is readily available (Mayo et al. 2004); • Remedies: • Use polar opposite reversals to get richer belief samples, even though they may increase apparent MR; • Negation reversals have few retrieval benefits;

17. Theoretical explanations of MR:Retrieval • Positioning effects: • Dispersed PO items reduce carryover effects and can increase coverage, but the task is more taxing for Rs and internal consistency may suffer; • Item similarity may determine whether Rs engage in additional retrieval when items are grouped together; • Remedies: • The use of dispersed antonyms should encourage the generation of distinct belief samples; • Avoid very similar (negated) statements when items are grouped;

18. Theoretical explanations of MR:Judgment • Item verification difficulty (Carpenter and Just 1975; Swain et al. 2008): • MR is a function of the complexity of verifying the truth or falsity of an item relative to one’s true beliefs, which depends on whether the item is stated as an affirmation or negation [NMR]; • Remedies: • Negations are problematic because they increase the likelihood of making mistakes (remember there are many types of negations); • Negated polar opposites are most error-prone; • Mix of regular and PO reversals should be best;

19. Item verification difficulty MR Negation Affirmation Truth value False True

20. Theoretical explanations of MR:Response • Acquiescence: Rs initially accept a statement and subsequently re-consider it based on extant evidence; the first stage is automatic, the second requires effort (Knowles and Condon 1999) [RMR]; • Remedies: • Although response styles are largely individual difference variables, situational factors may be under the control of the researcher (e.g., reduce the cognitive load for Rs); • Problems with online surveys;

21. Theoretical explanations of MR:Response (cont’d) • Asymmetric scale interpretation: the midpoint of the rating scale may not be the boundary between agreement and disagreement for Rs (esp. if the response categories are not labeled; cf. Gannon and Ostrom 1996) [RMR]; • Remedies: • Use fully labeled 5- or 7-point response scales;

22. Consequences of MR • Effect on scale reliability: [based on 4 studies] RG . .68 nRG .60 PO .57 nPO .49 • Effect on correlations between items measuring the same construct: • when content saturation is high, social desirability is low, and endorsement frequencies are moderate, the correlation between the four types of items can be quite high, although the findings showed evidence of POMR (Jackson and Lay 1968); • Ahlawat (1985) showed that the correlations between nPO and other items were particularly low;

23. Consequences of MR (cont’d) • Effect on factor structure: [based on three studies] • The factor structure can emerge clearly when the items are high in content saturation and free of social desirability, but RMR and POMR are apparent; • In Schriesheim and Eisenbach (1995), RG items performed best and nPO items did very poorly; • Effect on correlations with other scales: • Holden et al. (9185) showed that item negation was more detrimental to criterion-related validity than item reversal;

24. Example item with 4 negations Top management in my company has let it be known in no uncertain terms that unethical behaviors will not be tolerated.