Context Effects: Interpreting the Question

1. BA 531: Survey Methods Context Effects: Interpreting the Question Sharon Shavitt Department of Business Administration Survey Research Laboratory • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

2. Survey Orientation Context Effects: Interpreting the Question A Survey is… A set of cognitive tasks A social encounter (a conversation) • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

3. Respondent’s Tasks Context Effects: Interpreting the Question • Understanding and interpreting the question • Generating an opinion or judgment • Formatting the response • Editing the response • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

4. Respondent’s Tasks Context Effects: Interpreting the Question • Understanding and interpreting the question • Generating an opinion or judgment • Formatting the response • Editing the response • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

5. Respondent’s Tasks Context Effects: Interpreting the Question • Understanding and interpreting the question • Understand the literal meaning • Infer the intended meaning • Conversational logic • Response alternatives • Question context • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

6. Understanding/Interpreting the Question Context Effects: Interpreting the Question Where do you live? • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

7. Understanding/Interpreting the Question Context Effects: Interpreting the Question Where do you live? Literal meaning is clear, but inference about intended meaning will be strongly influenced by interview location. • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

8. Context Effects: Interpreting the Question Context Effects: Interpreting the Question • How satisfied are you with this course? • How satisfied are you with this curriculum in general? • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

9. Context Effects: Interpreting the Question Context Effects: Interpreting the Question • How satisfied are you with this course? • How satisfied are you with this curriculum in general? Don’t ask specific questions before general ones (e.g., rate course, then curriculum). Respondent assumes answer to 2nd question should be nonredundant with 1st. • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

10. Respondent’s Tasks Context Effects: Interpreting the Question • Understanding and interpreting the question • Generating an opinion or judgment • Formatting the response • Editing the response • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

11. Respondent’s Tasks Context Effects: Interpreting the Question • Understanding and interpreting the question • Generating an opinion or judgment Some questions request an evaluation of a target of judgment. What is your expected grade in this course? Rate the instructor’s overall teaching effectiveness. Evaluations are strongly influenced by context (making criteria salient, setting a standard of comparison, etc.) • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

12. Evaluation Context Effects: Interpreting the Question What is your expected grade in this course? Rate the instructor’s overall teaching effectiveness. Evaluations are strongly influenced by context (making criteria salient, setting a standard of comparison, etc.) • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

13. Retrieving or Computation Context Effects: Interpreting the Question • Understanding and interpreting the question • Generating an opinion or judgment Some questions require retrieving instances from memory or making a computation about the frequency of some event. ( How often do you…) • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

14. Longer or Shorter is Better? Context Effects: Interpreting the Question How many times do you study in a typical week? How many times do you study in a typical week? This might include studying at the library, reading a textbook while on a bus or train, looking at the course website, or going over your notes after class. • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses? • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

15. Respondent’s Tasks Context Effects: Interpreting the Question • Understanding and interpreting the question • Generating an opinion or judgment • Formatting the response • Nature of a frequency scale can dramatically affect responses. How to address this? • Should we use broad labels? (“often,” “sometimes,” “seldom” ): Annie Hall • Vague quantifiers: Provide evaluative as well as frequency information. Cannot disentangle. • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

16. Autobiographical Reporting: Formatting the Response Context Effects: Interpreting the Question • How many hours do you watch TV each week? • Less than 1 hour • Between 1-3 hours • Between 3-5 hours • Between 5-7 hours • More than 7 hours • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

17. Autobiographical Reporting: Formatting the Response Context Effects: Interpreting the Question • How many hours do you watch TV each week? • Less than 7 hours • Between 7-9 hours • Between 9-11 hours • Between 11-13 hours • More than 13 hours • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

18. Respondent’s Tasks Context Effects: Interpreting the Question • Understanding and interpreting the question • Generating an opinion or judgment • Formatting the response • Editing the response • Should we ask threatening questions multiple times to enhance reliability? • How can we make it OK to say yes (or no)? • Are shorter questions better than longer ones? • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?

19. Threatening Questions: Response Editing Context Effects: Interpreting the Question Did you vote in the recent election? Voting is not necessarily convenient for every person. Did you happen to vote in the recent election? • Should we ask specific questions before general ones? (e.g., rate course, then curriculum) • What about with attitude questions? • Does nature of the frequency scale affect responses?