Developing survey questions

1. Developing survey questions

2. Problems that careful questionnaire design can alleviate • Consistency effect – respondents attempt to make their later answers fit the earlier ones • Fatigue effect – questions toward the end of the survey are not carefully answered • Redundancy effect – all questions in the same style are answered in the same way

3. Question types • There are two major types of questions • Closed-ended questions • Open-ended questions

4. Closed-ended questions • The respondent is provided with a limited number of response options • Each response option has a numerical value or code it represents

5. Closed-ended questions • Examples: • Are you: • 18-34 years of age (1) • 35-54 years of age (2) • 55+ years of age (3)

6. Closed-ended questions • If you were about to purchase a digital TV, which source of information, if any, would you consult first? • 1. Magazines • 2. Advertising • 3. Salesperson • 4. Product brochure • 5. Friends • 6. Other • 7. Would not consult information source

7. Closed-ended questions • Please say whether you • Strongly agree (5), • agree somewhat (4), • neither agree nor disagree (3), • disagree somewhat (2) • or strongly disagree (1) with the following statement: “People who spend over 400 dollars on a cell phone are vain.”

8. Open-ended questions • The question is standardized, but the respondent answers in her own words. • The interviewer usually ‘probes’ at least once for expansion and/or specificity after the initial response. • Example: • “What features would the ideal science fiction movie have?”

9. Closed-ended v. open-ended • Closed-ended questions take less time to complete • Closed-ended questions take less time to enter into the database • Closed-ended questions allow the respondent to interpret her position in relation to the question

10. Closed-ended v. open-ended • Open-ended questions allow the respondent to elaborate and to make connections you may not have anticipated • Answers to open-ended questions must be ‘coded’ in order to be interpreted in a quantitative manner • Respondents often prefer open-ended questions and are bored and frustrated by too many closed-ended questions

11. Closed-ended v. open-ended • Because they take longer and are more difficult to deal with, open-ended questions should be used sparingly • Mix open-ended questions through the survey instrument, breaking it up and adding interest

12. Question wording • Concrete questions are best a. Are you happy with the service at Radio Shack? b. Better: Were you served within 5 minutes the last time you went to Radio Shack?

13. Question wording • Avoid abbreviations • In your view, does USC provide a liberal arts education worth its yearly tuition? • In your view, does the University of South Charleston provide a liberal arts education worth its yearly tuition?

14. Question wording • Avoid slang and colloquial expressions • Should president Obama take his economic advisers behind the woodshed? • Avoid jargon and technical terms • Should a summative evaluation of Head Start be commissioned by the U.S. government? • Better: Should the U.S. government commission a history of Head Start to review its activities and accomplishments?

15. Question wording • Avoid biasing words and phrases • The left-wing student group protesting U.S. presence in Sumatraclaims that American presence there is unwarranted. Do youagree?? • An exception:If necessary,use loaded questions to prompt response, but be careful, respondents may be offended • Parents get really angry at their children sometimes. In the past week, have you been really angry at your son?

16. Question wording • Avoid double-barreled questions • Do you think the University of Kentucky should subsidize student tuition and staff benefits? • Would you say that the decision to increase President Lee Todd’s salary by 67% will be a boon to the University of Kentucky or don’t you care? • You could think it would be a boon but not care

17. Question wording • Avoid negative wording • Double negatives are especially bad • “Please indicate whether you strongly agree, agree, neither agree nor disagree, disagree or strongly disagree with the following statement: • The United Nations should not have more authority to intervene in a nation’s military affairs. • Either emphasize NOT when asking the question or else reword the question in a positive manner

18. Proper scale construction • Response options on scales should be • 1) Mutually exclusive • 2) Equivalent • 3) Exhaustive

19. Mutual exclusivity • A given respondent or item should not fit into more than one category • Example: • Are you: • a) Single; b) Married; c) Divorced; d) Widowed; e) Separated, or f) Living in a couple relationship but not married • In one interview I heard, a 70+ year old woman said “All of the above”

20. Category equivalency • Categories in a scale must differ on a single dimension • Example: Would you say that the service in the shoe department is: • a) fast • b) slow • c) polite • d) knowledgeable

21. Exhaustiveness • All respondents or items must be classifiable in one of the categories • Example: Does your Herald-Leader arrive on your doorstep by 6 AM: a) Always b) Seldom c) Never • What about “usually”?

22. Scales used to measure knowledge, attitudes, beliefs • Developing measures of knowledge, attitudes, and/or beliefs is a challenging task. • While anyone can come up with a measure, it is hard to develop a truly valid measure of KAB’s

23. Main types of attitude measures • Semantic differentials • Likert scales • Thurstone scales

24. Semantic differential • Good __ __ __ __ __ __ __ Bad • Weak __ __ __ __ __ __ __ Strong • Beautiful __ __ __ __ __ __ __ Ugly

25. Likert scales • Example: The Employment Self Esteem Scale • INSTRUCTIONS: Please rate how strongly you agree or disagree with each of the following statements by placing a check mark in the appropriate box. • (Strongly Disagree/Somewhat Disagree/ Somewhat Agree/Strongly Agree)

26. I feel good about my work on the job. • On the whole, I get along well with others at work. • I am proud of my ability to cope with difficulties at work. • When I feel uncomfortable at work, I know how to handle it. • I can tell that other people at work are glad to have me there. • I know I'll be able to cope with work for as long as I want. • I am proud of my relationship with my supervisor at work. • I am confident that I can handle my job without constant assistance. • I feel like I make a useful contribution at work. • I can tell that my coworkers respect me.

27. Guttman scale • Statements are ordered so that they represent increasing agreement with or acceptance of one position on a dimension • If respondent accepts one statement higher on the scale, he should also accept those below it on the scale

28. A type of Guttman scale:Bogardus social distance scale • I am willing to permit immigrants to live in my country • I am willing to permit immigrants to live in my community • I am willing to permit immigrants to live in my neighborhood • I am willing to permit immigrants to live next door to me • I am willing to have immigrants for friends • I am willing to have an immigrant marry my son/daughter • Agreement with item 3 implies agreement with items 1 and 2, and so on

29. Thurstone scales • Thurstone scales include a number of statements that have been developed to represent their position or magnitude on some dimension • The average of the scores of the statements agreed with by the respondent provides her placement on the scale

30. People with AIDS are like my parents. • Because AIDS is preventable, we should focus our resources on prevention instead of curing. • People with AIDS deserve what they got. • Aids affects us all. • People with AIDS should be treated just like everybody else. • AIDS will never happen to me. • It's easy to get AIDS. • AIDS doesn't have a preference, anyone can get it. • AIDS is a disease that anyone can get if they are not careful. • If you have AIDS, you can still lead a normal life. • AIDS is good because it helps control the population. • I can't get AIDS if I'm in a monogamous relationship.

31. Response option considerations • “Don’t know” option • Researchers debate whether the interviewer should explicitly ask whether the respondent doesn’t know an answer • Fear that the DK option gives the respondent an easy out without thinking about the question • If an explicit “Don’t Know” option is provided, you’ll get more DKs but your conclusions may actually be more valid

32. “Neutral” option • Should scales have a middle option (odd-numbered) or no neutral point (even-numbered)? • Even-numbered scales force respondents to take a stand

33. Indexes • Indexes combine scores from multiple items to generate a single score for each respondent on some variable of interest • Thurstone scales are indexes • Some variables are complex, needing several items to provide a valid measure • The items could be organized in a single dimension (unidimensional) or in several dimensions (multidimensional)

34. Unidimensional indexes • Summated index • measured by a set of indicators that can be added together to derive a single, overall score • Items are related among themselves (correlated), (if you score high on one item you should score high on the other ones) • Make sure the items are not too highly correlated

35. Multidimensional indexes • If the construct you are measuring has more than one significant subconcept, you probably would gain by using a multidimensional index • Example -- communicative credibility is composed of three subconcepts: • Authoritativeness • Trustworthiness • Dynamism • To measure communicative credibility you would need to ask questions related to each of the subconcepts

36. So: • Questionnaire design is the most important part of survey analysis • Developing valid measures of ‘black box’ phenomena is especially difficult • Writing good questions is probably the most important part of questionnaire design • A number of common problems in question wording exist

37. Closed-ended questions are simpler and faster but open-ended questions allow respondents to provide more personalized information and are more interesting for them • A number of forms of attitude/belief measures have been developed over the years • Commonly used types are Likert scales, semantic differentials, Thurstone scales, Guttman scales

38. Using multiple items to measure a single construct and then combining them increases measurement validity