Survey Design and Measurement

1. Survey Design and Measurement Jeremy Kees, Ph.D.

2. Some practical issues…. • Qualtrics Research Platform • Free you under VSB’s “site license” • Extremely user friendly, but also very robust • www.qualtrics.com

3. Some practical issues…. • Amazon Mechanical Turk • The most inexpensive way to collect consumer data • Extremely user friendly, but also very robust • www.mturk.com

4. Online Survey (created by you and housed on Qualtrics’ server) • Create HIT (Human Intelligence Task) on Mturk • Description of your study and a (Qualtrics) link to it Mturk workers (survey responders) “work” on your HIT (i.e., they take your survey) Data is recorded by Qualtrics. Participants who complete the survey are given a code to input into Mturk. Those that enter a valid code, get paid. Everyone is happy   

5. The Mturk data I collected today…. • N=200 • Cost = $100 • Data collected in less than 1 hour • Demographics • Mean age = 36 • 56% male • 76% Caucasian • 80% at least some college • 41% are college grads • Median income = $35-50k • Highly engaged!

6. Formulate Problem Stages in the Research Process Determine Research Design Design Data Collection Method and Forms Design Sample and Collect Data Analyze and Interpret the Data Prepare the Research Report

7. Surveys / Questionnaires • The most common measurement instrument when quantitative data is sought • Descriptive research • Experiments • Modeling • Etc….

8. Developing Surveys • Good, well-specified research objectives lead to good surveys • Research design dictates what types of questions should be used • Exploratory research = unstructured script • Confirmatory research = structured survey

9. Desirable Characteristics • Brief • Objective • Specific • Relevant

10. Survey Methods • Usually should determine administration method prior to developing items • Can dictate what types of questions you should ask • Internet panels have become the most efficient and versatile method to collect data • Phone is still a viable option • Mall intercepts can still be useful • Mail/fax makes little sense anymore

11. mall internet panel tests tests members household size 2.8 2.9 3.0 average age 40.5 39.2 37.2 employed 71% 72% 69% white 86% 88% 89% male 20% 21% 15% college 40% 43% 46% Correlation between Responses: mall vs. internet internet test/retest reliability purchase intent .86 .94 frequency .94 .97 liking .85 .91 price / value .90 .99 Mall Intercepts vs. E-Panels

12. Internet Phone Time survey took to administer 12.5 19.4 minutes Upon completion, would respondent participate in future studies? 35% yes 26% yes More experienced Internet Users x Used rating scale extreme “endpoints” more frequently x E-Panels vs. Phone Jeff Miller and Alan Hogg “Internet vs. Telephone Data Collection” Burke White Paper series 2 (4) (www.burke.com). Also see Ashok Ranchhod and Fan Zhou “Comparing Respondents of E-Mail and Mail Surveys,” Marketing Intelligence & Planning 19 (2001), 254.

13. Types of Questions • Screening Variables • Independent Variables • Dependent Variables • Classification Variables • Segmentation • Moderators • Attention Filters

14. Types of Primary Data Demographic / Socioeconomic Characteristics Psychological / Lifestyle Characteristics Attitudes / Opinions Awareness / Knowledge Intentions Motivation Behavior What, how much, where, when, how, who Purchase behavior vs. use behavior E.g., --- basic hierarchy of effects models Example (CWL Study) Primary Data: Overview

15. QUESTION WORDING - General Guidelines • Use simple words and questions • Avoid ambiguous words and questions • Avoid leading questions---be objective • Avoid implicit alternatives • Avoid generalizations and estimates ---Be specific • Avoid double-barreled questions

16. What is your income? • $10,000 or less………………….1 • $10,000 to $25,000……………..2 • $25,000 to $50,000……………..3 • $50,000 to $75,000………..…….4 • $75,000 to $100,000..……..…….5 • $100,000 or more…………..……6 What is the problem and how would you revise the question?

17. Is the speed and efficiency of the drive-in teller services at your regular bank…..(READ CATEGORIES) • Very Satisfactory………………………4 • Somewhat satisfactory…………………3 • Somewhat unsatisfactory………………2 • Very unsatisfactory……………….……1

18. Question Wording • It is good practice to use scales whenever possible • Likert or semantic differential • Multi-item

19. Itemized Rating Scales • The respondents are provided with a scale that has a number or brief description associated with each category. • The categories are ordered in terms of scale position, and the respondents are required to select the specified category that best describes the object being rated. • The commonly used itemized rating scales are the Likert and semantic differential

20. Types of Scales • Nominal scales: those that use only labels • Ordinal scales: those with which theresearcher can rank-order the respondents or responses • Interval scales: those in which the distance between each descriptor is equal • Ratio scales: ones in which a true zero exists

21. Nominal Which of the soft drinks in the following list do you like? (Check ALL that apply): ___Coke ___Dr. Pepper ___Mountain Dew ___Pepsi ___Seven Up ___Sprite Ordinal Rank the soft drinks according to how much you like each (most preferred drink = 1, and least preferred drink = 6): ___Coke ___Dr. Pepper ___Mountain Dew ___Pepsi ___Seven Up ___Sprite Interval Please indicate how much you like each soft drink by checking the appropriate position on the scale: dislike like a lot dislike like a lot Coke ____ ____ ____ ___ Dr. Pepper ____ ____ ____ ___ Mountain Dew ____ ____ ____ ___ Pepsi ____ ____ ____ ___ Seven Up ____ ____ ____ ___ Sprite ____ ____ ____ ___ Ratio Please divide 100 points among these soft drinks To represent how much you like each: ___Coke ___Dr. Pepper ___Mountain Dew ___Pepsi ___Seven Up ___Sprite 100 Examples… 21

22. Itemized Rating Scales • Likert Scales • requires the respondents to indicate a degree of agreement or disagreement with each of a series of statements about the stimulus objects Strongly Disagree Neither Agree Strongly disagree agree nor agree disagree 1. Wal-Mart sells high quality merchandise. 1 2X 3 4 5 2. Wal-Mart has poor in-store service. 1 2X 3 4 5 I like to shop at Wal-Mart . 1 2 3X 4 5 4. Wal-Mart has low prices . 1 2 3X 4 5

23. Itemized Rating Scales • Semantic Differential Scales • End points associated with bipolar labels that have semantic meaning SEARS IS: Powerful --:--:--:--:-X-:--:--: Weak Unreliable --:--:--:--:--:-X-:--: Reliable Modern --:--:--:--:--:--:-X-: Old-fashioned

24. Decisions for Itemized Scales • Number of scale items • More is better, but there is a diminishing return around 11 points (Nunnally 1978) • 7-point scales are customary • Enough to discriminate • Allows for a scale midpoint • Manageable • Odd/even number of categories • Forced vs. non-forced

25. Why Multi-Item Scales?? Construct Abstract Concept “Unobservable” “Latent” “Psychological” **Single items are typically not sufficient to assess unobservable constructs

26. Multi-Item Scales are More “Reliable” • True Score Test Theory • All measures have • “True” Score • “Error” (Random and Systematic) • Good measures minimize the systematic error component of the score • Types of Reliability • Inter-Rater • Test-Retest • Internal Consistency (Cronbach’s Alpha)

27. Specify Domain of the Construct Step 1: Generate Sample of Items Step 2: Collect Data Step 3: Purify Measure Step 4: Assess Validity Step 5: Developing Sound Measures

28. Question Sequencing • After you have developed your measures, think about the order in which they should be asked

29. QUESTION SEQUENCING - General Guidelines • Use (more) simple, interesting opening questions • Use the funnel approach, asking broad questions first, and follow with more specific questions • Carefully design branching questions • Skip/display logic • Ask for classification information last • Place more difficult or sensitive questions near the end

30. QUESTION SEQUENCING - General Guidelines Question ordering #1 1 – EVALUATION OF FAT LEVEL OF A PRODUCT 2 – EVALUATION OF OVERALL PROD. NUTRITIOUSNESS 3 – EVALUATION OF OVERALL PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE Question ordering #2 1 – EVALUATION OF OVERALL PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE 2 – EVALUATION OF LEVEL OF PROD. NUTRITIOUSNESS 3 – EVALUATION OF FAT LEVEL OF PRODUCT

31. FOP Labeling Study • We were interested in consumer evaluations of: • Facts Up Front • All On-Package Labeling • Front-of-Package Nutrition Info Why was question sequencing critical??

32. Tips for Maximizing Participation • Offer an incentive ($$$) • Importance/relevance of the research project and its purpose • Completing the questionnaire will take only a short time • Answers are anonymous or confidential • Reminder 2-3 days after the initial ask

33. Attention Filters • Always include an attention filter to ensure that you are getting “quality” respondents • Eliminate “click throughs”

34. Attention Filters (Case Study) • Advertising Experiment • Very stringent screening criteria • Total # that started the study = 15,458 • Number that qualified = 870 • Incidence Rate (IR) = 5.6% • Number that qualified and passed the attention screener = 451 • 48% failed the attention filter!!! • NOT GOOD, criticalmix!

35. “Easy” Attention Filter

36. “Difficult” Attention Filter

37. And finally, remember the golden rule…. Do unto your respondents as you would have them do unto you!!

38. Team Assignment #2 • Refine your research questions • Need to be clear, concise, and “testable” • Based on your research questions • Design 2 potential studies that could address your research questions • Explain the benefits and weaknesses of each approach • Pick the “best” design and explain your decision (Note: Don’t worry about measurement or sampling too much---you’ll have your chance to do that later)

39. Team Assignment #3 ** Don’t start on this assignment until you’ve read Fowler (CH 6-7) • Based on your research design • Write a paragraph about what your measurement instrument is supposed to accomplish • Make a list of what should be measured to accomplish the goals of the study • Develop your measurement instrument

40. Team Assignment #3 • Deliverables include: • A very clean, polished version that you could use to actually collect data • This means you will need to carefully think through all of the issues we covered tonight (e.g., set-up, ordering, length, multi-item scales, etc.) • Intro paragraph and variable list (see previous slide) (Note: Don’t worry about defining your sample--you’ll have your chance to do that next week)