Questionnaire Development Survey Methods Sampling Fundamentals

1. Questionnaire DevelopmentSurvey MethodsSampling Fundamentals AE B37 - Week 2 � 15 January 2002 MM

2. Questionnaire Developmentand Survey Methods Further readings: Churchill, Iacobucci Chapters 7, 8 Malhotra Chapter 6, 10 Aaker et al. Chapter 10, 12

3. Sampling methods Further readings: Churchill, Iacobucci Chapters 10-11 Malhotra Chapter 11-12 Aaker et al. Chapter 14-15

4. Questionnaire design A questionnaire is a formalised set of questions for obtaining information from respondents Questions must translate the needed information Questionnaire must encourage cooperation Questionnaire should minimise response error Response errors: Inaccurate / misrecorded / misanalysed answers (due to the researcher, to the interviewer or to the respondents)

5. Questionnaire Design Process Specify the information needed Specify the type of interview method Determine the content of individual question Design the question to overcome any respondent inability/unwillingness to answer Decide on question structure Determine the question wording Arrange the question in proper order Identify the form and layout Reproduce the questionnaire Eliminate bugs by pretesting

6. 1. Specify the information needed This relates to the formulated MR problem It can be helpful, prior to design the actual questionnaire, to define a blank table where the desired data will be stored (e.g. Excel spreadsheet) It is important to specify the needed information having a clear idea of the target population (different types of respondents)

7. 2. Type of interview (survey methods) The questionnaire will be strictly conditional to the survey method

8. Telephone interviews Traditional interviewing (a phone, a pencil and a questionnaire) Computer Assisted Telephone Interviewing (CATI): computerised questionnaire administered to respondents through the phone

9. Personal interviews In-home Mall-Intercept Computer-Assisted (CAPI) with interviewer

10. Mail surveys Mail interviews (Fax just for businesses) Mail panels

11. Electronic interviews E-mail (ASCII/text message) Web-based (HTML/Java)

12. Response and costs

13. 3. Determine the content of individual question Is the question necessary? Unnecessary question should be eliminated, unless they serve for other purposes (involvement, disguise the purpose of sponsorship, etc.) Is a single question sufficient? Do you think that organic products are healthier and animal-friendlier? What does a no-answer mean? Why do you eat Sainsbury pizza? Potential different interpretations: because the cheese is better or because Sainsbury is closer to my place (attributes or knowledge of it)? What do you like about Sainsbury pizza as compared to other pizzas? Why did you first buy Sainsbury pizza?

14. 4. Overcoming problems in answering It is necessary to consider any factor that might lead to an unanswered question or an inaccurate answer Lack of information (do they answer anyway?) What did you eat as a dessert for Easter? Lack of memory (avoid omission, telescoping or creation effects) Incapacity to articulate certain responses For certain vague questions, multiple choice is preferable An unanswered question due to incapacity may lead to abandon the questionnaire Unwillingness to answer (sensitive information, too much effort, the question/context is perceived as inappropriate)

15. Techniques to get sensitive questions answered Hide the question among a group of �innocent� questions State that the behaviour of interest is common or the usefulness of an answer Use the �third-person� technique Provide categories instead of asking for figures Use randomised techniques (but you lose any linkage with other questions).

16. Randomised techniques Please flip a coin. If you get a head, please answer to question A, if you get a tail, please answer to question B. Are you enjoying this lecture? Are you a female? YES NO

17. Interpretation of randomised questions We got the following results for the question: YES: 20% NO: 80% We know that 38% of our respondents are female and 62% are male We know that the probability of getting a head or a tail is 50%

18. Results

19. 5. Choosing question structure Unstructured question (open-ended, free response) Good as first questions on a topic Less biasing influence (but interviewer bias) Coding of responses is costly and time-consuming Structured questions Multiple Choice (A, B or C?) � order bias Dichotomous (Yes or No + Don�t know) � question wording bias Scales (from 1 to 10)

20. Primary scales Nominal (Are you employed/non employed/student) Ordinal (order the following brands according to your preferences�) Interval (What is the temperature today?) Difference can be compared The 0 point is arbitrary Ratio (what were last year sales?) The 0 point is not arbitrary

21. Secondary scales Paired comparison: Coke vs Pepsi Rank Order: 1. Brie 2. Mozzarella 3. Stilton 4. Cheddar Constant sum: Allocate points to each alternative (with a constant total sum) � e.g. percentage points Q-Sort (many objects): 100 cards to be arranged in 10 piles, where piles go from �least highly agreed with� to �most highly agreed to� Magnitude estimation: how much do you agree from 0 to 100? Continuous rating: place a mark on a continuous line Paired comparison: Coke vs Pepsi Rank Order: 1. Brie 2. Mozzarella 3. Stilton 4. Cheddar Constant sum: Allocate points to each alternative (with a constant total sum) � e.g. percentage points Q-Sort (many objects): 100 cards to be arranged in 10 piles, where piles go from �least highly agreed with� to �most highly agreed to� Magnitude estimation: how much do you agree from 0 to 100? Continuous rating: place a mark on a continuous line

22. Itemised ranking scales Likert scale This cheese is soft Strongly disagree 2. Disagree 3. Neither 4. Agree 5. Strongly Agree Semantic differential This cheese is: Soft Hard Stapel scale This cheese is soft -5 -4 -3 -2 -1 +1 +2 +3 +4 +5

23. 6. Wording Define the issue Use ordinary words Avoid ambiguous words (no �usually�, �a bit��) Avoid leading questions (suggesting the answer) Avoid implicit alternatives (do you like to drive?) Avoid implicit assumptions (are you in favour of multiple choice tests? [�if this reduces the likelihood of top marks?] Avoid generalisation and estimates (how much do you spend in food every year?) Use positive and negative statements (advisable to use dual statements for different respondents; e.g. Is this cheese soft? Is this cheese hard?)

24. 7. Order of questions Use good opening questions Ask first basic information (target variables) Ask classification and identification questions at the end Place difficult and sensitive question towards the end General questions should precede specific questions Follow a logical order (flow chart)

25. 8. Form and Layout Check position of questions in the page No use of different colours (little effect, more complicated) Divide questionnaire into parts Number questions Number questionnaires (but risk of loss of anonimity) �

26. 9. Reproduction of the questionnaire Quality of paper Professional appearance Avoid splitting questions across pages �

27. 10. Pretesting Test preliminary the questionnaire on a small number of respondents, considering all previous issues. Any questionnaire can be improved. Better by personal interview (regardless of the actual survey method, a second pretesting may be carried out for some methods) Use a variety of interviewers for personal interviews Respondent is asked �to think aloud� Debriefing (go through the questionnaire with the respondent after he has finished to compile it)

28. Sampling A sample is a subgroup of the population selected for the study Sample statistics allow to make inference about the population parameters, through estimation and hypothesis testing

29. The sampling design process Define the target population, its elements and the sampling units Determine the sampling frame (list) Select a sampling technique Sampling with/without replacement Probability/Nonprobability sampling Determine the sample size Precision versus costs The marginal value in terms of precision of additional sampling units is decreasing Execute the sampling process

30. The sampling techniques Nonprobabilistic samples Convenience sampling Judgmental sampling Quota sampling Snowball sampling Probabilistic samples Simple random sampling Systematic sampling Stratified sampling Cluster sampling Other sampling techniques

31. Representativeness A sample can be considered as �representative� when it is expected to exhibit the average properties of the population

32. Selection bias Improper selection of sample units (ignoring a relevant �control variable� that generate bias), so that the values observed in the sample are biased and the sample is not representative. Example: A survey is conducted for measuring goat milk consumption, but the interviewers just select people in urban areas, that on average drink less goat milk.

33. Convenience sampling Only �convenient� elements enter the sample

34. Judgmental sampling Selection based on the judgment of the researcher

35. Quota sampling Define control categories (quotas) for the population elements, such as sex, age� Apply a �restricted judgmental sampling�, so that quotas in the sample are the same of those in the population

36. Snowball sampling A first small sample is selected randomly Respondents are asked to identify others who belong to the population of interests The referrals will have demographic and psychographic characteristics similar to the referrers

37. Simple random sampling Each element of the population has a known and equal probability of selection Every element is selected independently from other elements The probability of selecting a given sample of n elements is computable (known)

38. Systematic sampling A list of N elements in the population is compiled, ordered according to a specified variable Unrelated to the target variable (similar to SRS) Related to the target variable (increased representativeness) A sampling size n is chosen A systematic step of k=N/n is set A random number s between 1 and N is extracted and represents the first element to be included Then the other elements selected are s+k, s+2k, s+3k�

39. Stratified sampling Population is partitioned in strata through control variables (stratification variables), closely related with the target variable, so that there is homogeneity within each stratum and heterogeneity between strata A simple random sampling frame is applied in each strata of the population Proportionate sampling: size of the sample from each stratum is proportional to the relative size of the stratum in the total population Disproportionate sampling: size is also proportional to the standard deviation of the target variable in each stratum

40. Cluster sampling The population is partitioned into clusters Elements within the cluster should be as heterogeneous as possible with respect to the variable of interests (e.g. area sampling) A random sample of clusters is extracted through SRS (with probability proportional to the cluster size) 2a. All the elements of the cluster are selected (one-stage) 2b. A probabilistic sample is extracted from the cluster (two-stage cluster sampling)

41. Basic SRS sample statistics (unknown pop. variance)

42. Finite population correction factor Large samples (more than 10% of N) tend to overestimate the population standard deviation of the mean (proportion)

43. Level of confidence a and z parameter

44. Determining sample size Factors influencing sample size (n): Size of the population (N) Variability of the population (sX) Desired level of accuracy (q) Level of confidence (a) Budget constraint

45. An example Our aim is to estimate the average weekly consumption of beer in pints per student (x) in the University Student Union We don�t know the population variability, but we may roughly assume a large population standard deviation (s) of 4 pints We want to estimate the value with an accuracy (q) of 0.5 pints The target population (students at the university of Reading) has 13,151 units (N) We want to determine the sample size for a Simple Random Sampling, choosing a level of confidence of a=0.95 (za/2=1.96)

46. Sample size

47. Have a look at the assignments Any question?