Survey Contents

1. Survey Contents Creating an Effective Questionnaire

2. Survey Contents • When creating a questionnaire, remember: • Get it right at first—one cannot change questions in mid-survey! • The goal is to measure concepts as accurately as possible (validity and reliability). • Respondents will be “Lazy Thinkers.”

3. Survey Contents • Always consult other surveys first • They give good ideas for measuring concepts • They may provide valid ways to ask questions • Avoids “reinventing the wheel” • Always pre-test theinstrument

4. Survey Contents • When creating questions, consider the effects of the following elements: • Type Constraints, if any, placed on responses • Wording Clarity of questions and response options • Order Relative placement of items in the instrument • Topics Issues the items cover

5. Question TypesConstraints, if any, placed on responses • Open-ended avoid A survey question to which the respondent replies in his or her own word, either by writing or by talking • Closed-ended or fixed-choice preferred A survey question that provides preformatted response choices for the respondent to circle, check, mark, etc.

6. Question TypesFixed-choice • Closed-ended or fixed-choice examples (not exhaustive): • Two choices: Are you afraid to walk around your neighborhood at night? ___ Yes ___No (Level: Dichotomous)

7. Question TypesFixed-choice • Rating Scales, Asking respondents to rate something (e.g., an attitude, belief) on a numbered scale, often a Likertscale (lick-urt) • LikertItem, or “Likert Scale” (common in GSS): The police in my community are mostly courteous. 1 2 3 4 5 Strongly Agree Neither Disagree Strongly Agree A nor D Disagree • Semantic Differential a juvenile delinquent is Bad Good Awful ⊲ ⊲ ⊲ ⊲ ⊳ ⊳ ⊳ ⊳ Nice (Level: Ordinal used as Interval-Ratio)

8. Question TypesFixed-choice • Ranking Scale, Asking respondents to put a set of items into order Place the following crimes in order of severity: ___Armed Robbery ___Home Invasion ___Stalking (Level: ordinal) • List In which type of community do you live? Pick one: __Urban __Suburban __Rural __ Other (please specify) (Level: nominal) Please indicate whether you were the victim of a crime last year (Mark all that apply) __Assault __Auto Theft __Robbery __Other (specify) (Level: series of dichotomous)

9. Question TypesLazy Thinkers • Consider “neutral” or “no opinion” a valid option, but do not allow if fence sitters will likely hide their real opinions • Example: Avoid neutral for attitudes such as racial, sexual orientation, religious . . . • Questions about past behaviors that ask for specific details or rely on good memory are typically useless. • Example: Avoid: How many glasses of wine did you consume last month? _____ Use: About how many glasses of wine do you drink in a typical week? a. None b. 1 or 2 c. 3 to 6 d. 7 to 14 e. more than 14 • Bad memory leads to: • Forward (backward) telescoping: reporting that events occurred more recently (further back) than in reality • Salientevents beingover-reported • Mundane events beingunderreported • “Habitual”filling in for lost information.

10. Question TypesLazy Thinkers • Because of agreement bias, use negative and positive statements, such as those in the self-esteem scale. • R’s may tire or lose track in a series of questions and choose salient options (such as first in series). Example: Response set problem, where R’s quickly check the same response for all questions.

11. Survey Contents • When creating questions, consider the effects of the following elements: • Type Constraints, if any, placed on responses • Wording Clarity of questions and response options • Order Relative placement of items in the instrument • Topics Issues the items cover

12. WordingClarity of questions and response options • Respondents should understand questionnaire items • Guidelines for good phrasing: • Be direct • Maintain simplicity • Be specific • Take the role of the respondent The following points elaborate on these four main themes.

13. WordingClarity of questions and response options Some things to do to make questions less unclear and less incomprehensible to reduce inaccuracy: • Avoid Double-barreled questions. They contain two questions in one. • Example Avoid:“Do you think that students and Professors should be given discounts on sports tickets?” • Avoid Double-negative questions. • Example Avoid:“Do you disagree that professors should not be required to help students outside of class?”

14. WordingClarity of questions and response options • Avoid complex rhetoric, syntax, or disciplinary jargon. • Example Avoid: Please rate your impressions of the interactional sequences with your subordinate at the biennial review. • Avoid teaching a new word or information. • Example Avoid: In this questionnaire, deflection is the discrepancy between fundamental cultural sentiments and transient, situated impressions in the relevant semantic space. • Sometimes, however, one must establish a particular definition • Example OK: In this questionnaire, Binge Drinking refers to 5 or more drinks in an hour for men or 4 or more drinks in an hour for women.

15. WordingClarity of questions and response options • Avoid ambiguity. • Example Avoid: • Do you teach your children to effectively function? • “Do your customers complain normally?” • Avoid biased questions: • Emotionally leading terms: • Example Avoid: Should political prisonerswho aresuffering inhumane conditions be rescued by American forces? • Making a particular response more appealing: • Example Avoid: Given the success of three strikes laws, do you believe that California should expand this law to cover other crimes?

16. WordingClarity of questions and response options • Avoid biased questions (continued) • Avoid unbalanced response options. Example Avoid: Rate your professor’s friendliness? Poor . . . Satisfactory . . . Quite Warm . . . Exceptional Example From the GSS Avoid: What is your opinion about a married person having sexual relations with someone other than the marriage partner –is it always wrong, almost always wrong, wrong only some times, or not wrong at all? Always wrong Almost always wrong Wrong only sometimes Not wrong at all

17. Survey Contents • When creating questions, consider the effects of the following elements: • Type Constraints, if any, placed on responses • Wording Clarity of questions and response options • Order Relative placement of items in the instrument • Topics Issues the items cover

18. OrderRelative placement of items in the instrument Central goal, to maintain respondents’ engagement. • Opening questions: simple, introduce survey topic. • More important questions come earlier, avoiding “missing data” if R’s fatigue while taking survey. • Sensitive questions are in middle,neverat the beginning. Explain their necessity. • Ending questions: routine, demographic. • Avoid, but if necessary, put open-ended questions toward the end. • Ever notice how much having to type an answer makes you want to leave a consumer survey, and if required, ensures that?

19. OrderRelative placement of items in the instrument • Try not to mix topics. So put like questions into sets. • Example: 10 item self-esteem scale is 10 contiguous items. • May need transition statements between types of questions, or may need transitory questions or directions. • Example: “In this next set we ask about personal thoughts…”

20. Questionnaire Outline • Introduction, topical questions Transition • Central research questions • Set of related questions Transition • Set of related questions Transition • Sensitive questions, justification Transition • Routine, demographic questions

21. OrderContext Effects • Avoid context effects, framing subsequent questions with ideas about topics potentially related to them. Doing so may invoke stereotypes, bias, or otherwise influence R’s thoughts. • Example Avoid: • Those who use crack should be given longer prison terms. _____Yes _____No • The government should spend more money on improving the lives of the urban poor. _____Yes _____No

22. OrderFatigue Effects • Avoid fatigue effects. • Use only as many questions as necessary • Vary question complexity or response options • Use an appealing format or interface. • Do not ask R’s to navigate complex format questions (e.g., “skipping” or “go to next section”). • Paper-and-pencil imposes limits • Electronic questionnaires can, however, create seamless navigation

23. Survey Contents • When creating questions, consider the effects of the following elements: • Type Constraints, if any, placed on responses • Wording Clarity of questions and response options • Order Relative placement of items in the instrument • Topics Issues the items cover

24. TopicsIssues the items cover • Make survey topic(s) clear. • No surprises with questions about unrelated issues. • Example: The GSS explains that it is an “omnibus” survey and will have a variety of questions • Consider Self Bias and Belonging. • People (Lazy Thinkers): • assert a particular vision of themselves • are the center of their universe • view themselves as good • do not want to appear to not belong • Explain that there are no right or wrong answers, only honest or not. Honest answers will be helpful to others.

25. TopicsIssues the items cover • Focus on factual information, avoid impressions • Example: Avoid: Does your boss sexually harass you? Better: Has your boss ever told you a sexual joke? Has your boss ever touched your body in a way that was too familiar? and so forth • Remove social stigma when asking questions about criminal or deviant behavior • Include normalizing statements Example: “Many people use drugs for a variety of reasons. Please indicate which of the following you have you ever used.”

26. TopicsIssues the items cover • For personally sensitive questions: • Provide a rationale for asking them. • Avoid phrasing that seemspersonal or directed at respondent. Example Avoid: “Do you abuse your kids?” Example Better: “Have you ever spanked or hit your child with enough force to leave a bruise or mark on the skin?” • Do not open or close a survey with personal questions. Example Avoid: Now the final question: Have you ever been so sad that you attempted to commit suicide? _____ Yes _____ NO

27. Survey Contents • When creating questions, consider the effects of the following elements: • Type Constraints, if any, placed on responses • Wording Clarity of questions and response options • Order Relative placement of items in the instrument • Topics Issues the items cover THE END!