Survey and Questionnaire Design

1. Survey and Questionnaire Design MAR 6648: Marketing Research January 20, 2010

2. Overview • How do you write survey questions? • What are some common traps survey writers fall into? • How do we avoid them? • What about special circumstances? • Embarrassing questions, etc.?

3. Survey Research • Descriptive research • Survey modes and methods: • Telephone surveys • Mail surveys • Face-to-face surveys • At home • Mall intercepts • Web-based surveys

4. Advantages of Surveys • Can collect a great deal of data, including • Attitudes, interests, opinions, behaviors • Classification variables • Can be employed in virtually any setting • Can be inexpensive, depending on format

5. Disadvantages of Surveys • Effective implementation requires considerable judgment • Social interaction of interview may lead to errors • Subject to demand and honesty concerns

6. First Big Point: Biases • Basic Tenet: Ask a question, and you’ll get a response! • Translation: Garbage in, garbage out!

7. Typical problems in wording questions

8. How questions create answers

9. How questions create answers

10. How questions create answers • A survey for Black Flag said: • “A roach disk… poisons a roach slowly. The dying roach returns to the nest and after it dies is eaten by other roaches. In turn, these roaches become poisoned and die. How effective do you think this type of product would be in killing roaches?” • Not surprisingly, 79% said effective! • A Gallup poll sponsored by the disposable-diaper industry asked: • “It is estimated that disposable diapers account for less than 2% of the trash in today’s landfills. In contrast, beverage containers, third-class mail and yard waste to account for about 21% of the trash in landfills. Given this, in your opinion, would it be fair to ban disposable diapers?” • 84% said no!

11. How questions shape answers • Some examples • Behavioral frequency questions • “How often do you…?”

12. Behavioral Frequency Questions • How many hours of TV do you watch on an average week day? Please choose the category from the list below that best describes your behavior. Marketers Question: What % of class watches more than 2.5 hrs of TV?

13. Behavioral Frequency Questions • Scale represents distribution in ‘real’ world • Values in middle reflect “average” / “usual” behaviors • Values at an end (or ends) reflect extreme behavior • Low Frequency Scale • Lower estimates of “average” • Avoid marking extreme high box • High Frequency Scale • Makes marking “>2.5 hours” more acceptable • “Response Scale Effect”

14. How questions shape answers • How much money do you have in savings? • 1-7; 1 = $0, 7 = $500 and over • 1-7; 1 = $500 and under, 7 = $500,000 and over • How wealthy are you?

15. How questions shape answers • Some examples • Behavioral frequency questions • Aided versus unaided questions

16. Aided and Unaided Questions • What do you consider to be the most important thing for children to prepare them for life? • 1. _______________ • 2. _______________ • 3. _______________ • 4. _______________ • What do you consider to be the most important thing for children to prepare them for life? • __Good education • __Inheritance • __Moral values • __To think for themselves • __Good friendships • __Others (please specify)

17. How questions shape answers • Some examples • Behavioral frequency questions • Aided versus unaided questions • Framing of questions

18. Framing Effects • Question forms that lead respondents to give a certain response • Example: Allow versus forbid Yes No 5% 95% 81% 19% Q: Do you think the U.S. should forbid public speeches against democracy? Q: Do you think the U.S. should allow public speeches against democracy?

19. Framing Effects • Question form • Choice of example/source

20. Framing Effects • Question form • Choice of example/source • Surveyor power: Mispredicting political outcomes (Bischoping & Schuman 1992) • Perhaps partly motivational? • People who feel good about themselves show less of this bias (Cohen 2003)

21. How questions shape answers • Some examples • Behavioral frequency questions • Aided versus unaided questions • Framing of questions • Scales associated with questions • Odd versus even numbered scales • Semantic versus Stapel scales

22. How questions shape answers • Some examples • Behavioral frequency questions • Aided versus unaided questions • Framing of questions • Scales associated with questions • Sequence of questions

23. Sequence of Questions Q1: General Colin Powell, the hero of the Gulf War, recently decided to become a member of a political party. Do you happen to know which party that is? Q2: Now rate this party. Q1: General Colin Powell, the hero of the Gulf War, recently decided to become a member of a political party, but decided not to run for the presidential elections. Do you happen to know which party that is? Q2: Now rate this party. • Specific – General Questions • Happiness and dating (How many dates have you had recently?) • Order 1 (Happiness then dates): Correlation = 0.03 • Order 2 (Dates then happiness): Correlation = 0.67

24. Sequence of Questions • Difficultly in answering one question can influence subsequent answers • List three (twelve) ways in which you are happy (unhappy) with your partner. • How happy are you with your partner?

25. Sensitive Questions • Respondents may not be willing to answer some questions, due to embarrassment, legal issues, etc. • This may result in • Item nonresponse • Termination • Distortion

26. Asking the Embarrassing Question • The casual approach • “Do you happen to have killed your wife?” • The numbered card approach • “Would you please read off the number on this card which corresponds to what became of your wife?: • 1: Natural Death • 2: I killed her • 3: Other • The everybody approach (counter biasing method) • “As you know, many people have been killing their wives these days. Do you happen to have killed yours?” • The other people approach • “Do you know any people who have murdered their wives? How about yourself?” Barton, POQ, 1958

27. How to ask a sensitive question • Personal information – Build rapport, legitimize yourself • Embarrassing info – Downward bias • Counterbiasing statement • “Recent studies indicate that men often use their wives’ hair spray. Do you use your wife’s hair spray?” • Projective techniques

28. Projective Techniques

29. Key Points • When wording questions… • Use simple words • Use clear words • Avoid leading questions • Avoid implicit alternatives • Avoid implicit assumptions • Avoid double-barreled questions • Consider frame of reference

30. Key Points • Don’t forget about question sequence • It can influence the nature of the respondent’s answers and can be the cause of serious error in the results • Use a simple and interesting opening question • Ask general questions first • Place uninteresting and difficult questions late in the sequence • Arrange questions in logical order

31. The Basic Steps of Questionnaire Design • Decide on question content • Decide how to ask each question • Response format • Question wording • Decide on question sequence • Funnel approach (general to specific) • Decide on physical characteristics • Carry out pretest, revise, and final draft

32. So… • What is a good survey question? • How are we to interpret survey question output?

33. A Managerial Problem In a few weeks, Verizon will start selling a version of the iPhone compatible with its network. How likely is it that you will buy a Verizon iPhone? • Temptation: • Report those numbers to your manager as is • Problem: • Those numbers have no direct value • What does “somewhat likely” mean in real life? Very Unlikely Somewhat Unlikely Somewhat Likely Very Likely 15% 35% 35% 15%

34. A Managerial Problem In a few weeks, Verizon will start selling a version of the iPhone compatible with its network. How likely is it that you will buy a Verizon iPhone? • Solutions: • 1. Benchmark • What do past surveys say? Very Unlikely Somewhat Unlikely Somewhat Likely Very Likely 15% 35% 35% 15% Actual Purchase Rate

35. A Managerial Problem In a few weeks, Verizon will start selling a version of the iPhone compatible with its network. How likely is it that you will buy a Verizon iPhone? • Solutions: • 1. Benchmark • What do past surveys say? • 2. Focus on questions you can answer • Relative responses on the same items Very Unlikely Somewhat Unlikely Somewhat Likely Very Likely 15% 35% 35% 15%

36. A Managerial Problem In a few weeks, Verizon will start selling a version of the iPhone compatible with its network. How likely is it that you will buy a Verizon iPhone? • Solutions: • 1. Benchmark • What do past surveys say? • 2. Focus on questions you can answer • Relative responses on the same items • 3. Ask other questions/use other methods Very Unlikely Somewhat Unlikely Somewhat Likely Very Likely 15% 35% 35% 15%

37. Summary • Survey design is harder than it looks • Be sure to think about what the question suggests about the answer • Order and scale points matter, too • Be sure to keep respondents’ comfort in mind