Social Desirability Effects in Survey Research

1. Social Desirability Effectsin Survey Research Gov 1011 Shauna L. Shames shames@fas.harvard.edu

2. Social Desirability: Def. & Examples • What Weisberg would classify as "Measurement Error Due to Respondents," a sub-category of "Response Accuracy" (see Figure 2.1, from Weisberg Chapter 2, p. 19 • Phillips and Clancy (1972) definition: • "Broadly conceived, 'social desirability' as a response determinant refers to the tendency of people to deny socially undesirable traits or qualities and to admit to socially desirable ones" (923).

3. Def. & Examples, Cont’d • Sociological literature divides into two components (see Phillips and Clancy (1972): 'Some Effects of 'Social Desirability' in Survey Studies," in AJS, and also earlier work by the same authors): • - trait desirability (personality construct, has to do with how the respondent describes self to the interviewer or within the survey) • - need for social approval (quality of the measurement items – has to do with the relationship between the interviewer/survey giver and the respondent, and the respondent's quest for social approval from interviewer or survey giver)

4. Roots of Concept • The idea of social desirability traces its roots back to the sociology and psychology concept of response sets, "where the individual responds to item content in such a way as to portray himself in other than a true light (e.g. responding to items in terms of the social desirability of the answers)" (O'Neill 1967, 95). • Response set: "a personal tendency to respond in a specified way within a testing or interview situation" (Smith 1967, 87).

5. Roots, Cont’d • Converse in the early 1960s shocked political science (and himself) by finding that most people's opinions are ill-considered, inconsistent, and pretty ignorant – in his words, lack "constraint" (Converse 1964). • Recently, John Zaller has thrown a similar monkey wrench into the gears of public opinion research, with the major contention of his 1992 book (The Nature and Origins of Mass Opinion) being that public opinion surveys do not reveal deep underlying preferences of the public, but rather are "top of the head" replies by people trying to come up with answers to survey questions!

6. Some Effects of Soc. Desirability • Some of the sociological and political science social desirability studies reveal, for instance, that people will say they voted when they haven't, will inflate their reported incomes, will describe themselves in socially-sanctioned ways (such as saying they are happier than they are in their marriages), and will edit politically "incorrect" responses (such as prejudice)

7. Effects, Cont’d • Several studies by Bishop and co-authors found that respondents are differentially likely to offer opinions on fictitious issues, when asked to do so in a survey. Not exactly that all respondents will offer such opinions, but that certain people are more likely to do so than others (included lower status or less educated respondents). So people who are sensitive to being thought dumb or uneducated are more likely to overcompensate, by offering opinions on nonexistent political issues (see, for example, Bishop 1986). • Cross-comparative survey of women in US and Costa Rica – has to do with cultural connotation of topic, so saying "Have you had as many children as you want" in Costa Rica implies that if a woman says yes, she's going to have an abortion (even if she really does have enough for her) (Class Notes, 3/1/07) •  Acquiescence: the tendency to say yes repeatedly (generally believed that Latinos have more of a tendency to acquiesce to survey questions, because of cultural socialization – Class Notes, 3/1/07) • 1980s study on whether racism has declined in America: puzzle that white Americans say they are less racist on surveys, but strongly resist programs designed to achieve racial equality, such as school desegregation and affirmative action (McConahay, Hardee, and Batts 1981)

8. Effects of Study Mode • In a recent study, Presser and Stinson (1998) found that people were about a third less likely to report weekly attendance of religious services when the survey was self-administered rather than conducted by an interviewer.

9. Mode Effects, Cont’d • This turns out to have major impacts on our understanding of trends in religious service attendance: "This difference in measurement approach… alter[s] the observed trend in religious attendance over time: In contrast to the almost constant attendance rate recorded by conventional interviewer-administered items, approaches minimizing social desirability bias reveal that weekly attendance has declined continuously over the past three decades" (137). • Weisberg suggests (see Chap. 12) that different modes of survey call up different “scripts” and therefore invite different social behaviors • Face-to-face interview in a person’s home calls up an “interviewer as guest” script, and respondents act more polite. Telephone interviews call up a “solicitor” script, while self-administered paper surveys summon a “multiple-choice exam” script, both of which invite possible negativity.

10. “Fixes” for Soc. Desirability Effects in Surveys • 1.) Change mode of survey (self-administered surveys have greater chance of what Steven Colbert calls "truthiness") • For a sensitive survey, want self-administered (could be web or paper) • 2.) Carefully reword questions to (as far as possible) remove any implications or cues

11. Fixes, Cont’d • 3.) Deflect attention away from respondent (not "what do YOU think about racial conflict," but "what does your NEIGHBOR think") • example: recent WHP studies (see most recent study at: http://thewhitehouseproject.org/v2/press/2007/February/20070216-PressRelease.html )

12. References • Bishop, George F.; Tuchfarber, Alfred J.; and Olendick, Robert W.: 1986. "Opinions on Fictitious Issues: The Pressure to Answer Survey Questions," Public Opinion Quarterly Vol. 50, No. 2 (Summer): 240-50. • Converse, Philip: 1964. "The Nature of Belief Systems in Mass Publics,: in David Apter, • ed., Ideology and Discontent. New York: The Free Press, pp. 206-261. • McConahay, John B.; Hardee, Betty B.; and Batts, Valerie: 1981. "Has Racism Declined in America? It Depends on Who is Asking and What is Asked," The Journal of Conflict Resolution, Vol. 25, No. 4 (December): pp. 563-579. • O'Neill, Harry W.: 1967. "Response Style Influence in Public Opinion Surveys," Public Opinion Quarerly Vol. 31, No. 1 (Spring): 95-102.

13. References, Cont’d • Phillips, Derek L. and Clancy, Kevin J.: 1972. "Some Effects of 'Social Desirability' in Survey Studies," American Journal of Sociology Vol. 77, No. 5 (March): 921-40. • Presser, Stanley and Stinson, Linda: 1998. "Data Collection Mode and Social Desirability Bias in Self-Reported Religious Attendance," American Sociological Review Vol. 63, No. 1 (February), pp. 137-145. • Smith, David Horton: 1967. "Correcting for Social Desirability Response Sets in Opinion-Attitude Survey Research," Public Opinion Quarterly Vol. 31, No. 1 (Spring): pp. 87-94. • Watson, P.J.; Morris, Ronald J; Foster, James E.; and Hood, Ralph W., Jr.: 1986. "Religiosity and Social Desirability," Journal for the Scientific Study of Religion Vol. 25, No. 2 (June), pp. 215-232. • Weisberg, Herbert F.: 2005. The Total Survey Error Approach. Chicago: University of Chicago Press. • Zaller, John: 1992. The Nature and Origins of Mass Opinion. New York: Cambridge University Press.