Introduction to Questionnaire Design

1. Introduction to Questionnaire Design 2013-2014 Workshop in Methods November 8, 2013 Ashley Bowers, Stacey Giroux, and Lilian Yahng

2. Why Questionnaire Design Matters • Do you think the sports media treat African American athletes differently than white athletes? • YES • NO • 60% of respondents answer “YES, DIFFERENTLY” • Do the sports media treat African American athletes and white athletes the same or differently? • SAME • DIFFERENTLY • 40% of respondents answer “DIFFERENTLY” Center for Survey Research

3. Cognitive Aspects of Survey Methodology Example: Do you think the sports media treat African American athletes differently than white athletes? (Yes/No) Question-Answering Process: • Comprehension • Retrieval • Judgment • Reporting: • Formatting into response options • Editing answer Tourangeau (2000) • Captures both the ideal and nonideal case – e.g., misunderstanding, satisficing • Respondents can go wrong at any of these stages • Not necessarily an orderly progression; several can be happening simultaneously • Survey as artificial construct • Survey within a social context • (social exchange theory) • (cooperative/conversational principle) • (social desirability) Center for Survey Research

4. TOP TEN • Conceptual Variability • Problematic Wording • Vague Quantifiers • Response Options • Order Effects • Recall/Retrieval Difficulty • Estimation Difficulty • Attitudinal Questions • Sensitive Topics • Formatting Center for Survey Research

5. #1 - Conceptual Variability • Words have many meanings: • variability across respondents • divergence from researcher’s definition • Suessbrick, Schober & Conrad (2000) administered CPS Tobacco Use Supplement followed by post-test about interpretation: Most frequent interpretation held by only 53.8%! Center for Survey Research

6. Do you think children suffer any ill effects from watching programmes with violence in them, other than ordinary Westerns? • Belson (1981) determined that respondents interpreted children, ill effects and violence in numerous ways • e.g. “children”: < 8 years, < 19 - 20 years • children as students • only 8% interpreted question as intended Center for Survey Research

7. The “same” question, with some definitions: • Do you think that children suffer any ill effects from watching TV with violence in them, other than ordinary Westerns? By children I mean people under 14, by ill effects I mean increased aggression at school or at home, increased nightmares, inability to concentrate on routine chores, and so on. By violence I mean graphic depictions of individuals inflicting physical injuries on themselves or others, depictions of individuals wantonly damaging property or possessions, abusive behaviors or language to others, and so on. • Improved? • Additional words can clarify intended meaning but this may lead to unwieldy questions, as above. • Tradeoff between clarity and complexity • Possible confusion when everyday terms are defined (Griceanimplicatures) Center for Survey Research

8. #2 - Problematic Wording Are you satisfied with our prices and customer service? Do you favor or oppose not allowing drivers to use cell phones while driving? Given the world situation, the government protects too many documents by classifying them as SECRET or TOP SECRET. (Agree/Disagree) How many minutes does it usually take you to commute to work? In the past month, have you crossed the street from one side to another in order to avoid going near someone you thought was frightening? Now that you've seen how you can save time, would you buy our product? AVOID: • Double-barreling • Negatives • Complex syntax • Hidden assumptions • Leading questions Aim for a ~6th grade reading level. Center for Survey Research

9. #3 - Vague Quantifiers • Non-numerical terms for quantity have different numerical interpretations • Belson (1981) found “few” (in over the last few years) meant: • “no more than two years” (7/59 respondents) • “seven or more years” (19/59) • “ten or more” (11/59) • Particularly problematic in response options How often do you feel really excited? Very often, pretty often, not too often or never? • If R says more than never, Schaeffer & Bradburn asked for number • For educated and younger Rs, “pretty often” and “very often” were associated with larger numbers Center for Survey Research

10. #4 - Response Options Why Response Options Matter Response Alternatives and Television- Watching (Schwartz & Bienias, 1990) Don’t Forget to Check: • Balanced scale • Number of response options: 5 to 7? (Krosnick et al.) • Label all scale points • Mutually exclusive and exhaustive • Midpoint (e.g., Neither/Neutral) option? • “Other (Please Specify)” option? • “Don’t Know” option? • “Not Applicable” option? Center for Survey Research

11. #5 - Order Effects • Ordering of Questions • Part/whole effects (marital satisfaction/general satisfaction) – ask general question first • Related content – Fewer people say taxes are too high when first asked several questions about whether government spending for various programs should be increased • Ordering of Response Options • Recency effect: tendency to endorse last option in list • most likely when interviewer reads to respondent • Primacy effect: tendency to endorse first option in list • most likely when respondent reads to self (self-administered) or predictable scale used Center for Survey Research

12. #6 - Recall/Retrieval Difficulty When did you purchase your car radio? Center for Survey Research Tourangeau (2000)

13. Recall tasks can involve a complicated process for respondents (cognitive burden) • Telescoping • Situating an event between landmarks • More recent, more likely to recall • Greater impact/salience, • more likely to recall • Is the respondent in a position to know the answer? (e.g., parent’s income on college student survey) Center for Survey Research

14. #7 - Estimation Difficulty • In the past twelve months, since [DATE], how many different times have you been admitted to a hospital as a patient overnight or longer? • In the past year, how many times have you seen or talked with a medical doctor or a physician’s assistant about your health? • During the past 4 weeks, how much have you been bothered by emotional problems (such as feeling anxious, depressed, or irritable)? • How many hours of television did you watch yesterday? • How many hours of television did you watch last month? • In a typical week, how many hours of television do you watch? • We all, from time to time, compare our lives with those of other people, such as co-workers, family members, friends, neighbors, people you went to school with, and so on. We would now like to ask you some questions about how you see yourself in relation to others. • Counting, general impressions, and hypotheticals can be difficult for respondents • Reference period/point? • Ask in a series of questions? • Try a longer question (using familiar words, examples)? Center for Survey Research

15. #8 - Attitudinal Questions • Do you think marijuana should be made legal, or not? • On the average, [racial minority] have worse jobs, income, and housing than white people. Do you think these difference are mainly due to discrimination? • How satisfied are you with how your life has turned out so far? Please use a scale of 1 to 10, where 1 is extremely dissatisfied and 10 is extremely satisfied. • Factual vs. attitudinal questions • Measuring subject states of respondents • “No right or wrong answers” • May not be fixed • No mental “card catalog” of topics • Capturing weak attitudes and nonattitudes • “…or have you not thought much about this?” • Midpoint, No Opinion, Don’t Know options Center for Survey Research

16. #9 - Sensitive Topics Do you happen to jog or not? (Gallup) The Casual Approach: Do you happen to have murdered your wife? The Numbered Card Approach: Would you please read off the number on this card that corresponds to what became of your wife? (Hand card to respondent.) (1) Natural death (2) I killed her (3) Other (What?) The Everybody Does It Approach: As you know, many people have been killing their wives these days. Do you happen to have killed yours? The Other People Approach: Do you know any people who have murdered their wives? How about yourself? Bradburn (2004) In talking to people about elections, we often find that a lot of people were not able to vote because they weren’t registered, they were sick, or they just didn’t have time. How about you—did you vote in the elections this November? (ANES) Some people say that most people can be trusted. Others say you can’t be too careful in your dealings with people. How do you feel about it? (GSS) Center for Survey Research

17. Sensitive answers • A short list of topics shown to be sensitive: • Pap smears, mammograms, disease prevention, library card ownership, voting, reading, exercise, income, charitable giving, drinking, gambling, sexual activity, illegal activity. • Note: List possibly susceptible to culture, time, gender, etc. • Demographics: early or later in questionnaire? • Income: try broad ranges, if initially refused. Center for Survey Research

18. #10 - Formatting KEEP IT SIMPLE Center for Survey Research

19. Visual Cues and Heuristics Special Issue: Topics on Survey Measurement and Public Opinion POQ (2013) 77 : S1 For web surveys: • Middle means typical • Left and top mean first • Near means related • Up means good • Like means close Center for Survey Research

20. The Good, the Bad, and the Ugly • Do you have access to high speed internet, either in your home, at work, or somewhere you can easily get to like your public library or a nearby relative? • Do you agree or disagree that teens should not be fined for not obeying the local outdoor smoking ordinance? • What is the status of this house? (Indonesian Family Life Survey) • Self-owned • Occupying • Occupying illegally without rent • Occupying illegally with rent • Rented/contracted • Do you agree or disagree with the following statement: the federal government has gotten totally out of control and threatens our basic liberties unless we clear house and commit to drastic change. (FOX News Poll) Center for Survey Research

21. Workshop • Scenario: The Happiness Task Force has hired your team to develop a survey to gauge respondents’ happiness. • Rules: Draft five survey questions, and you must include at least: • One demographic question • One question that uses a Likert scale (e.g., 1=strongly disagree …. 5=strongly agree) • One additional closed-ended question • One open-ended question Center for Survey Research

22. Focus group Cognitive expert review Cognitive interview Field pretest Center for Survey Research

23. Focus Group http://lovestats.wordpress.com/dman/ Center for Survey Research

24. Focus Group • Guided discussion among 6-10 led by a moderator • How do people think about this topic? • Have we missed something? • Language people use • Technology: Online, teleconference, two-way • Study protocol development Center for Survey Research

25. Sample Focus Group Protocol • What is your job title and what kind of work do you do? • Main activities? • Change much from day to day? • When you think about job skills, what kinds of things come to mind? • Skills shared with others in this group? • Skills required? Center for Survey Research

26. Cognitive Expert Review • Experienced survey researcher/methodologist or cognitive psychologist • Not substantive expert • Written review with follow-up discussion • Measurement problems • Solutions (e.g., rewording, reordering) • Testing options Center for Survey Research

27. Center for Survey Research

28. CSR: Might need to be a little more specific – an alternative from the Current Population Survey is: Are you employed by government, by a private company, a nonprofit organization, or are you self-employed or working in the family business? Center for Survey Research

29. CSR: This will be a tough question and may be subject to inaccuracy. Would recommend asking if they have provided help in a specific area (e.g., help with food) – and then, how many times have they provided help in that specific area and to how many different people. Center for Survey Research

30. CSR: Do you need both Q5 and Q6? I am not sure how much additional information Q6 gains you for additional respondent burden. CSR: Might be good to be consistent across the items – you use two years here, then five years, then last year – may confuse respondents. CSR: Will respondent know the level of political influence of other people? CSR: Large number of scale points. Would be good to ensure that respondents are able to make distinctions between such a large number of points. I would expect that there would be concentration of responses at a limited number of points. Might also consider giving the respondent a showcard with the response scale as this might make it easier for them to provide a response that uses more of the scale. Center for Survey Research

31. Cognitive Interview • One-on-one interview with trained interviewer(s)/researcher(s) • Protocol: survey questions + item-by-item probes • Respondent “thinks aloud” as he/she answers each question • Directed probes after questions • Protocol just a guide Center for Survey Research

32. “Thinking Aloud” • Windows example before interview • Early probing: “Tell me more about that”, “Tell me what you are thinking” • Concurrent • May be difficult

33. Common Directed Probes (from Willis 2005, p. 48)

34. http://lovestats.wordpress.com/dman/

35. Center for Survey Research

36. Examples – AVS • Revised: Scientists should modify human genes to cure serious diseases. Center for Survey Research

37. Examples – AVS • Original: How good does it make you feel when you see the American flag flying? EXTREMELY GOOD VERY GOOD NOT VERY GOOD NOT AT ALL • Finding: R feels neutral • Revised: Changed scale and included neutral option • Original: If I have worked hard, I deserve to have time for fun and pleasure. • Finding: Pleasure not viewed positively • Revised: If I work hard, I deserve time to enjoy life. Center for Survey Research

38. Pilot/Field Pretest/Pretest • Small scale version of main study (n=20-30+) • Mirror procedures of main study • Use debriefing questions • Identify common problems from debriefing forms (observe/talk with interviewers) • Examine your survey data Center for Survey Research

39. Debriefing Questions • Were there any words or phrases in any of the questions that you felt were unclear? • Please list any questions that you felt were not easy to read or required you to read them more than once. • Were there any questions that you felt you did not have enough knowledge to answer? • Were there any questions that you felt were too personal or were uncomfortable answering? • One of our major challenges in conducting this study will be obtaining participation from busy executives like yourself. Please provide any thoughts you might have about what we could do to encourage participation. Center for Survey Research

40. Debriefing Questions • Did you have technical difficulties with any of the following tasks while completing this survey? (Select all that apply.) • Seeing the text of a question Seeing all of the responses for a question Entering an answer to a question Going back to a previous question Changing an answer to a question Exiting the survey Finding out how to contact us Figuring out how far along you were in the survey I didn’t have any technical difficulties with the survey Center for Survey Research

41. Original: How important to you are greenways and trails? • Finding: Rs do not understand greenway. • Revised: Add a definition. • Original: How satisfied or dissatisfied are you with greenways and trails? • Finding: New residents unable to answer. • Revised: Must live in the city for 6+ months. • Original:What is your date of birth? • Finding: Rs uncomfortable with question. • Revised: Use age range. • Original:How would you rate: Timely arrival of products on back order? • Finding: Never had back ordered product. • Revised: Asked screener question. Added reference period. Center for Survey Research

42. Blank Center for Survey Research

43. http://lovestats.wordpress.com/dman/ Center for Survey Research

44. Reliability and Validity (from Singleton and Straits 2005, pp. 90-105) • Reliability • Internal consistency • Test-retest reliability • Construct validity • Intercorrelations • Discriminant validity • Convergent validity • Known groups Center for Survey Research

45. References Questionnaire design: Bradburn, N. et al. (2004). Asking Questions. Converse, J., and Presser, S. (1986). Survey Questions: Handcrafting the Standardized Questionnaire. Fowler, F.J. (1995). Improving Survey Questions. Tourangeau, R. et al. (2000) . Psychology of Survey Response. Focus groups:Krueger, R., and Casey, M. (2000). Focus Groups. Cognitive interviewing: Willis, G. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Reliability/validity: Singleton, R.A., Jr., and Straits, B.C. (2005). Approaches to Social Research. Data collection: Dillman, D et al. (2009). Internet, Mail and Mixed Mode Surveys. Couper, M. (2008). Designing Effective Web Surveys. Oishi, S.M. (2003). How to Conduct In-Person Interviews for Surveys. Center for Survey Research

46. Thank You! For more information: Ashley Bowers Stacey Giroux Lilian Yahng afbowers@indiana.edu sagiroux@indiana.edu lyahng@indiana.edu Center for Survey Research http://csr.indiana.edu Center for Survey Research

47. Cognitive Testing Plan • Recruiting and scheduling subjects • How? Convenience sample, usually recruit through personal networks, Craigslist, social media, flyers, newspaper ads, community groups • What types? Representative of the population you are studying, make sure some respondents will go down all paths in questionnaire • Incentive usually given (unless volunteers) - $25-$50 • Generally, 5-25+ subjects per round (often only time and $$ for 1 round but 2-3 is desirable) • Train multiple interviewers • Involve in questionnaire development • Learn background on technique and probing Center for Survey Research

49. Cognitive Testing Plan (cont’d) • Mode of data collection? In-person may work better but can do by telephone • Where to conduct? Quiet lab or non-lab setting • Record interview and/or take notes for each interview • How long of an interview? 15 minutes – 2 hours. 1 hour recommended (may be able to get through a 20 minute questionnaire) • Hold debriefing and may write a report Center for Survey Research