Survey Contents

1. Survey Contents Creating an Effective Questionnaire

2. Survey Contents • Devote time to design the questionnaire up front—you cannot change your questions in mid-survey! • Remember that your goal is to create a valid and reliable instrument, accurately measuring the concepts that you are interested in. • Remember that your respondents will be “Lazy Thinkers.”

3. Survey Contents • When creating questions, consider the effects of the following elements: • Type Constraints, if any, placed on responses • Wording Clarity of questions and response options • Order Relative placement of questions in the instrument • Content Topics the questions cover

4. Survey Contents • Type Constraints, if any, placed on responses • Open-ended A survey question to which the respondent replies in his or her own word, either by writing or by talking • Closed-ended or fixed-choice A survey question that provides preformatted response choices for the respondent to circle, check, mark, etc.

5. Survey Contents • Type Constraints, if any, placed on responses • Open-ended • Respondents are encouraged to give full reaction • Anticipate amount of space one will need to give a complete answer • Try to limit to exploratory questions • Too many will tire subjects • Lots of work to standardize • False belief that you got the full answer • Most commonly used are specific such as, “What is your age in years?”

6. Survey Contents • Type Constraints, if any, placed on responses • Closed-ended or fixed-choice • Response options are limited • Yes/no • True/false • Multiple choice with an “other” option • Race: White, Black, Hispanic, Asian, Other • Likert Scales 1 2 3 4 5 Strongly Agree Neither Disagree Strongly Agree A nor D Disagree

7. Survey Contents • Type Constraints, if any, placed on responses • Closed-ended or fixed-choice • Rating Scales • Ask respondents to rate something like a person, place, thing, idea, attitude, etc. on a numbered scale, often Likert scales • These are dominant form in the GSS • Semantic Differential a fifth grader is Bad Good Awful ⊲ ⊲ ⊲ ⊲ ⊳ ⊳ ⊳ ⊳ Nice • Ranking Scales • Force R’s to put a set of items in order • For example: Place the following animals in order of their ability to understand humans from lowest ability to highest ability. Dog, Cat, Slug

8. Survey Contents • Type Constraints, if any, placed on responses • Closed-ended or fixed-choice • Other Types • Magnitude Estimation (rare) Links a number to an attitude and numerically compares other attitudes relative to it • Split or unfolding questions General followed with clarifying questions • Funneling Starts general and narrows to specific questions • Inverted Funneling Uses specific thoughts to prompt general opinion • Responses to Vignettes

9. Survey Contents • Type Constraints, if any, placed on responses • Closed-ended or fixed-choice • Special Issue: Coding • Assigning numbers to response options so computers can operate with them • Keep conventional logic and data analysis in mind when determining your coding schemes for responses • Code Dummy Variables with 0 = absence and 1 = presence • Increasing magnitude should be reflected by increasing codes • The GSS is horrifically coded!

10. Survey Contents • Type Constraints, if any, placed on responses • Closed-ended or fixed-choice • Special Issue: Coding • If your variable were Religiosity, and you asked, “how religious do you consider yourself?” Then you should not have a coding scheme like: 1, very religious 2, somewhat religious 3, barely religious 4, not at all religious It should be reversed!

11. Survey Contents • Type Constraints, if any, placed on responses • Closed-ended or fixed-choice • Special Issue: Coding From the general social survey codebook: Do you Feel that you're constantly under stress? CONTENT CODE SAMPLE Yes 1 3681 No 2 6422 Don't know 8 51 Not stated 9 595 This should be No = 0 and Yes =1 because the idea of the question is about stress. In binary logic, 1 equals the presence of something.

12. Survey Contents • Type Constraints, if any, placed on responses • Closed-ended or fixed-choice • Special Issue: Coding From the general social survey codebook: How frequently do you participate in bowling? CONTENT CODE SAMPLE 2-3/month 1 13 1-2/week 2 90 3+/week 3 5 This is actually appropriate, because the higher the code number, the more the person bowls.

13. Survey Contents • When creating questions, consider the effects of the following elements: • Type Constraints, if any, placed on responses • Wording Clarity of questions and response options • Order Relative placement of questions in the instrument • Content Topics the questions cover

14. Survey Contents • Wording Clarity of questions and response options • Respondents should understand your questions • Pre-testing is a very effective way to see if they do • Examples of bad questions: • Do you hate taxes? • The answer will tell you little because our society has many different taxes, loves to complain about them, but thinks they are necessary • Do you hate getting up in the morning? • AIDS prevention researchers learned to avoid words such as “intercourse,” “homosexual sex,” and “intravenous injection” because many do not understand these words

15. Survey Contents • Wording Clarity of questions and response options • Guidelines for good questions: • Be direct • Maintain simplicity • Be specific • Take the role of your respondent The following points elaborate on these four main themes.

16. Survey Contents • Wording Clarity of questions and response options • No complex rhetoric, syntax, or disciplinary slang or jargon. • Do not expect them to learn new information just answer a question. (Sometimes, you may establish context with a short paragraph then ask a series of short, specific questions.) • Avoid phrasing questions to seem too personal or direct, especially when dealing with culturally sensitive issues. For example, “Do you abuse your kids?”

17. Survey Contents • Wording Clarity of questions and response options • Avoid Double-barreled questions. They contain two questions in one. For example, “Do you think that students and Professors should be given discounts on sports tickets?” • Avoid Double-negative questions. For example, “Do you disagree that professors should not be required to help students outside of class?” • Avoid hypothetical questions (unless you are studying the sociology of hypothetical situations). You will not gain useful information from these questions. For example, “If men could have children, would your husband or boyfriend stay with you?”

18. Survey Contents • Wording Clarity of questions and response options • Avoid ambiguous questions. • “Do you teach your children to effectively function?” “Does your boss engage you in interactive dialogue?” “Effectively function” and “interactive dialogue” are subject to interpretation. • Words such as "usually" or "normally" mean different things to different people. “Do your customers normally complain?” has a variety of interpretations. • Avoid biased questions. Avoid making one response option look more suitable than the other, using emotionally loaded terms, or using unbalanced response categories. • “Don’t you think that suffering terminal cancer patients should be allowed to be released from their pain by choosing death?” • Alabama pays teachers more than similar states. Alabama should: spend more, keep spending the same, reduce spending little, reduce spending some, reduce spending a lot, dramatically reduce spending

19. Survey Contents • When creating questions, consider the effects of the following elements: • Type Constraints, if any, placed on responses • Wording Clarity of questions and response options • Order Relative placement of questions in the instrument • Content Topics the questions cover

20. Survey Contents • Order Relative placement of questions in the instrument • Opening questions should be simple and introduce the topic of the survey. • Try not to mix topics. Put like things into sets of questions. • Avoid framing later questions with topics that can be linked to them in previous questions. For example, one should not ask about attitudes toward crack use right before asking about attitudes toward the urban poor. This will invoke stereotypes about the poor.

21. Survey Contents • Order Relative placement of questions in the instrument • Sensitive questions should never be at the beginning. Put in middle. May need opening paragraph for them. • Consider need to transition between types of questions. Write transitory questions or directions. • Place your most important questions earlier—R’s may fatigue before answering them if they are later.

22. Survey Contents • Order Relative placement of questions in the instrument • Do not put in too many complex format questions (e.g., “skipping” or “go to next section”). R’s will get confused/frustrated. • Closed-ended questions are easier to answer. Put open-ended questions later in survey. Investment of time up to that time will likely keep them through the open-ended.

23. Survey Contents • When creating questions, consider the effects of the following elements: • Type Constraints, if any, placed on responses • Wording Clarity of questions and response options • Order Relative placement of questions in the instrument • Content Topics the questions cover

24. Survey Contents • Content Topics the questions cover • You should make the topic of the survey clear to R’s. Do not surprise them with questions about unrelated topics. • If including sensitive questions, provide a rationale for asking them. Explain how honest answers will be helpful to others. • Questions relevant to deviance should include normalizing statements. For example, “Many people use drugs for a variety of reasons. Have you ever used ecstasy (or x, e, MDMA) to feel closer to other people?”

25. Survey Contents • Content Topics the questions cover • The more sensitive the topic, the more reassurance of confidentiality should be stated. • Never open a survey with questions about sensitive topics, and do not end surveys with questions that will negatively linger in R’s minds. • Continually reassure R’s that there are no right or wrong answers to questions—just truthful or not. • Questions that require lots of specific details or a good memory are typically useless. If they are necessary, employ techniques to prompt recollection.

26. Survey Contents • When creating questions, consider the effects of the following elements: • Type Constraints, if any, placed on responses • Wording Clarity of questions and response options • Order Relative placement of questions in the instrument • Content Topics the questions cover • Now… Some General Advice

27. Survey Contents • General Advice • Always consult other surveys first • They will give you good ideas for exploring your topic • They may provide good examples of how to ask questions • You avoid “reinventing the wheel” • Especially useful for validated scales • Be sure that you have permission to use • Sociological items are most often free, psychology and medicine typically charge • Always pre-test your instrument

28. Survey Contents • General Advice • When constructing and editing your survey, ask yourself: • Do your question types allow you to get the answers you need? • Is every question integral to your research? (Do not waste R’s time.) • Does a topic warrant more than one question? • Do any questions seem to appear out of nowhere? • Is each question general enough to cover the entire sample’s experiences, but specific enough to get useful data? • Would you respond to these questions? • Do you understand specifically what each question is asking about?

29. Survey Contents Social Psychology of Survey Response Consider these issues: • Not everyone will have an opinion on every topic. Consider “no opinion” as an option. • People want to appear to agree. Consider negative and positive statements. • R’s may lose track and choose salient options (such as first in series). Keep questions simple. • Response set problem: In a series with the same response options, R’s may just quickly check the same response for all questions.

30. Survey Contents Social Psychology of Survey Response Consider these issues: • Bad memory leads to: • Forward telescoping (reporting that events occurred more recently than in reality) • Backward telescoping (reporting events further back than in reality). • Salient events are overreported • Mundane events are underreported • “Habitual” events will fill in for lost information. Provide aides to recall such as reference points, landmark events, etc. Use limited time frames in questions.

31. Survey Contents Social Psychology of Survey Response Consider these issues: • Fatigue effect. Boredom or tiring. Use transitions, vary questions or response options, place easier questions near the end, or use a “jazzed up” format. • Self bias. People try to assert a vision of themselves and are the center of their universe. Try to elicit factual information rather than impressions by avoiding personal evaluations and by disguising implications about persons