Survey Design: Wording and Question Order Issues

1. STAT 472 Survey Design Wording Issues Questionnaire design issues

2. Wording Issues • There are two wording issues: • Discussed earlier, is to use simple vocabulary and grammar to minimize confusion • The effects of specific words or phrases on the respondent Example: what is the difference between forbid and not allow? Both terms have the same meaning, but many people are willing to not allow something than to forbid(emotional reaction) it. Less well educated respondents are most influenced by minor wording differences.

3. Example: Respondents were asked whether they thought television news was impartial. large numbers of respondents had ignored the word impartial. Less than 50% interpreted the word with its proper meaning Over 25% ignored it or had no idea of its meaning Others gave it unusual meanings 10% thought it was directly opposite to its true meaning Researchers need to be cautious, because some wording effects (e.g., the difference between forbid and not allow) remain the same for decades, while other effects may appear.

4. Questionnaire Design Issues • Length of Survey • Question Order • Nonresponse • Format and Layout • Total Design Method

5. Length of Survey or Questionnaire • Researchers prefer long questionnaires or interviews because they are more cost effective • There is no absolute proper length. The length depends on the survey format and on the respondents characteristics • In telephone interview: 10-20 minutes • In mail questionnaire: 3 to 4 pages (for highly educated and a salient topic: 15 pages) • In face-to-face interview: 1 hour (in special situations, 3-5 hours)

6. Question order or Sequences • There are two question sequence issues • How to organize the items in the overall questionnaire • Context effects of answering specific questions before others.

7. Organization of Questionnaire You should sequence questions to minimize the discomfort and confusion of respondents. A questionnaire has opening, middle, and ending questions. After an introduction explaining the survey, make opening questions pleasant, interesting, and easy to answer to help a respondent feel comfortable about the questionnaire. Avoid asking many boring background questions or threatening questions first Organize questions into common topics. Mixing questions on different topics causes confusion. Orient respondents by placing questions (e.g., "Now I would like to ask you questions about housing"). Make question topics flow smoothly and logically, and organize them to assist respondents' memory or comfort levels. Do not end with highly threatening questions, and always end with a "thank you”.

8. Order Effects It may influence respondent answers, it’s appear for people who lack strong views, less educated, and for older or those with memory loss. Example: For a woman having an abortion rises if the question is preceded by a question about abortion being acceptable when a fetus has serious defects, but not when the question is by itself or before a question about fetus defects. Previous questions can influence later ones in two ways: • through their content (i.e., the issue) • through the respondent's response

9. For example, a student respondent is asked, "Do you support or favor an educational contribution for students?" Answers vary depending on the topic of the preceding question. If it comes after, "How much tuition does the average U.S. student pay?” respondents interpret “contribution“ to mean support for what students will pay. If it comes after "How much does the Swedish government pay to students?” respondents interpret it to mean a contribution that the government will pay.

10. Responses can be also influenced by previous answers, because a respondent having already answered one part will assume no overlap. For example, a respondent is asked, "How is your wife?" The next question is, "How is your family?" Most respondents will assume that the second question means family members other than the wife because they already gave an answer about the wife.

11. Question Order Effects

12. Use a funnelsequence of questions - that is, ask more general questions before specific ones (e.g., ask about health in general before asking about specific diseases) • Or, divide the number of respondents in half and give half of the questions in one order and the other half in the alternative order. then examine the results to see whether question order mattered. Context Effects. Researchers found powerfulcontext effects in surveys. As a practical matter,two things can be done regarding context effects If question order effects are found, which order tells you what the respondents really think? The answer is that you cannot know for sure.

13. Example: A telephone survey on two topics were studied: Crime and attitudes toward a new anti-drunk-driving law A random half of the respondents heard questions aboutthe drunk-driving law first; the other half heard about crime first (to see whether there was any context effect-a difference by topic order). They found that respondents who were asked about the drunk-driving law first expressed lessfearabout crime than did those who were asked about crimefirst. Respondents answer all questions based on a context of preceding questions and the interview setting. A researcher needs to remember that the more ambiguous question's meaning, the stronger the context effects, because respondents will draw on the context to interpret and understand the question, previous questions on the same topic and heard just before a question can have a large context effect.

14. Order effects • Questionnaire : • Opening  middle  closeting questions -Start with Introduction explaining the survey -opening questing an easy question to be answered. - Avoid asking boring or threatening questions

15. Mixing questions on different topics causes confusion • Arrange questions on the same topic together, and introduce the section with a short introductory. • Don’t end with highly threatening questions, and always end with a “Thank you” • Use Funnel sequence of questions: Ask more general questions before more specific ones,