Survey Research

1. Survey Research Neuman and Robson Ch. 8

2. Introduction to Survey Research • Survey research is a very old, and a very popular, research technique • 1880 survey by Karl Marx in France • 25000 questionnaires sent to workers • Late 19th century “Chocolate Sociologists” • Rowntree and Cadbury • Community surveys to study poverty

3. Topics appropriate to survey research: • descriptive, explanatory, and exploratory research purposes • best method available to collect original information about a population • To measure attitudes and orientations

4. General features of survey research • respondents are chosen through probability sampling procedures • Systematic questionnaire or interview procedures • Sophisticated statistical techniques

5. Strengths and Weaknesses of Survey Research: • Advantages • Reliability • Can describe large population • Flexibility • Standardized • Disadvantages • Validity • Superficiality • Cannot modify questionnaire in field • Artificiality

6. Mail Surveys and Self-Administered Questionnaires • Hand-administered to a group or mailed to sample • Mail distribution and return practices • Monitoring returns • Follow-up mailings • Response rates: 50% adequate, 70% very good • Additional factors: • Sponsorship • Inducement to respond

7. Disadvantages of mail surveys: • Requires simple questions • Cannot observe respondents • No opportunity for probe • No control over conditions or who responds • Not suited for people who are illiterate or for whom English not first language • Low response rate can be a problem

8. Face-to-Face Interview Surveys • Very expensive • Interviewers require training • Use interview schedule • Appearance and demeanour crucial • Need familiarity with questionnaire • Must follow question wording exactly • Must record responses exactly • Also record other events during interview

9. Advantages and Disadvantages of Face-to-Face Interview Surveys • Advantages: • Can probe for responses • Can keep respondent “on task” • Disadvantages: • Cost and training • Interviewer bias

10. Telephone Surveys • Can reach 95% of households • Use of telephone interview schedules • Selecting a sample for telephone surveys • Telephone directory • Random Digit Dialing (RDD) • Selecting a respondent from a household

11. Telephone Surveys • Can achieve higher response rates • Can decrease "don't knows" and "no answers” • Interviewers can help clarify • Telephone surveys have many advantages of face-to-face but without high cost

12. Advantages and Disadvantages of telephone surveys • Advantages: • cheaper to carry out than face-to-face interviews • less time and effort • more impersonal • Disadvantages: • large number of tries to achieve a small number of successes. • less motivation generated among respondents • No visual cues/aids possible • Limited interview length

13. Computer-assisted telephone interviewing (CATI) • interviewers sit at a computer terminal with a screen • answers given are directly entered into the computer • Very fast • Often used in polling (i.e. Video: Ask a Silly Question)

14. Internet (Web) Surveys • In U.S. 69.6% of population now use Internet • Canada 67.5% of population (2007) • Use same basic design as mail surveys • Advantages: • Fast, cheap • Can use graphic and visual effects • Disadvantages: • Coverage – not all groups use computers • Privacy – need for encryption

15. Comparison of Survey Methods Criteria Interview Mail Telephone Web _________________________________________________________________ 1. Cost high low moderate very low 2. Response rate high low high moderate 3. Level of control of high low moderate low situation 4. Applicability to moderate high moderate high geographically dispersed populations 5. Applicability to high low high moderate heterogeneous populations 6. Obtaining detailed high moderate moderate low information 7. Speed slow slow fast fastest

16. Secondary Analysis of Survey Data • New analysis of data collected for another purpose • Use of shared data and data archives (i.e. Census and GSS) • Advantages • cheaper and faster • Access to larger datasets • Disadvantages • problems of validity • Questions may not meet your needs • Sample may not be adequate

17. Questionnaire Construction • usually includes a variety of demographic questions, and one or more scales and indexes aimed at collecting data on attitudes, beliefs, or behaviours • purpose of a survey is to try to collect information about a sample in order to make generalizations about the larger population

18. The general structure of a good questionnaire: • 1. Title • 2. A brief introduction and explanation of research. • 3. A section of general demographic questions re: gender, age, marital status, education. Note: more sensitive demographic questions (i.e. income, home value, etc.) might be better placed at the end of the questionnaire. • 4. Section of non-threatening or less sensitive behavioural and attitudinal questions. • 5. A section near the end with any sensitive or threatening questions. • 6. Any additional demographic information. • 7. Conclusion with brief statement again thanking respondents for their time and effort

19. Additional Considerations: • 1. Keep questionnaire as short as is reasonably possible. • 2. Appearance of the questionnaire is important. Don't crowd your questions, and use an easy-to-read font. Leave some space between questions. • 3. Use bold font and underlining for titles/headings. • 4. Use examples and sample questions for clarity, but be careful not to introduce bias. • 5. Include an open-ended question at the end of the questionnaire asking for any additional information and/or feedback. This question sometimes provides some interesting information!

20. Constructing the questions: • Create questions that use indicators which will specifically address your hypotheses. • Use existing scales or measures whenever possible. • You can also modify a scale or index to meet the needs of your particular study if an existing scale is not quite appropriate. • The advantage in using an existing scale is that its reliability and validity are already known.

21. Types of questions to use: • a. Simple questions that use a concrete indicator (i.e. What is your age in years?___). • b. Likert style questions are used for measuring simple attitudes, beliefs, emotions, or behaviours (i.e. When you think back to your high school years, you feel: 1. Very unhappy 2. Somewhat unhappy 3. Don't feel anything 4. Happy 5. Very happy) • c. Matrix questions are useful when you have a number of related items. Matrix questions are composed of a series of questions that have identical response categories which are presented in table format with the response category labels at the top of the chart.

22. Types (cont.) • c. Guttman scales: These are useful for measuring the strength or intensity of an attitude. • d. Rank-ordering: Rank ordering is a method that works well with a small number of objects, such as 10. For example, instead of rating how interesting each of a set of subject areas are, you could ask the respondent to simply sort them in order of most to least interesting. • e. Paired comparisons: In this method, you present items two at a time, and ask respondent to pick which one has more of some attribute. For example, you can present behavioural problems and ask which one is more serious.

23. Closed or open-ended question responses • A closed response set refers to those questions which have fixed categories for answers (uses quantitative analysis) • Open-ended questions allows for free responses (qualitative analysis)

24. Filter questions and contingency questions • Filter questions are useful for sorting respondents. • Contingency questions allow respondents to skip portions of the survey that may not be relevant to them.

25. Principles of Question-Writing • 1. Keep questions as short and concise as possible. • 2. Choose your wording carefully. • 3. Try not to ask questions beyond a respondent's capabilities. • 4. Avoid emotional language and the use of "loaded" words. • 5. Watch out for prestige bias in your questions. • 6. Don't use leading questions like "You don't smoke, do you?" • 7. Avoid ambiguity and vagueness.

26. Principles (cont.) • 8. Don't use double-barrelled questions. • 9. Avoid false premises or assumptions. • 10. If possible, don't ask respondent about future intentions. The link between intentions and future behaviour is tenuous. • 11. Try not to use negatives and double negatives in your questions. • 12. Watch for explanatory statements that may bias the answer to the question. Watch out also for questions that might influence the answers to subsequent questions.

27. Other Issues • Aiding recall • Provide special instructions or extra time • Ask month by month, week by week, etc. • Threatening questions • People under-report illness, disability, deviance, illegal activity, income/wealth • Create “enhanced questions” or embed activity in list of more serious activities

28. Issues (cont.) • Social desirability bias • People tend to over-report socially desirable behaviours (being cultured, voting, giving to charity, being good spouses or parents, etc.) • Try to minimize importance of these activities or present alternatives in questions • Knowledge questions • Make sure questions are appropriate level • Phrase questions so respondents feel comfortable not knowing the answer