Lecture 4

1. Lecture 4 Survey Design and Data Coding

2. Overview • What is a survey • Question design and considerations • Testing a survey instrument • Data considerations (data coding)

3. What is a Survey? • To survey: • “act of looking or seeing, observing” • Research Surveys • Qualitative interviews, focus groups • Specific, systematic, quantitative data-collection instruments

5. Example Surveys • US Census • General Social Survey • Online Surveys

6. Some General Considerations for a Survey • Must have a logic for each question • Must have a logic for the question responses (if provided) • Must have a logic for the sequencing of questions • Must have clear wording • Must have clear formatting • Must take into account the sample population that will actually take/use the survey instrument • Culture, language, interpretive ambiguity

7. Question Logic • Avoid redundant questions unless you have a reason • Exception: Sometimes it is a good idea to ask two questions that tap same concept (first as a scale, then as a categorical decision) • Example (ranks): • First ask respondent to rate 1-10 how much they agree with several statements. • Next, ask respondents to rank the statements by how much they agree with them (i.e., 5 statements, rank them 1-5).

8. Question Logic (continued) • Avoid asking unnecessary questions • Example: Survey on computer usage • Socio-demographic questions (age, gender)? • Risk behavior questions (drug use, etc)?

9. Question Response Logic: Scales versus Categories • As a basic rule, metric scales with more range are better than binary or categorical responses (when appropriate). • Example: Happiness  • Are you happy or sad today? • How happy are you on a scale of 0-10 (0 is least happy, 10 is most happy)? • With general scales and Likert-scales, consider having no “middle” category (neutral, no opinion).

10. Question Response Logic: Categorical Responses • Responses must be mutually exclusive • Example (bad): Where do you live? • Berkeley, San Fran, Bay Area, Other • Responses must be exhaustive • Example (bad): What kind of computer do you have? • PC, Mac • Use ‘Don’t Know’, ‘Other’, ‘Not Applicable’ when absolutely necessary

11. Question Sequence • Static order for questions needs to have some rationale/logic • Grouping similar items together • Scattering similar items throughout survey • Personal demographic questions work best at end of survey (response rate and completion) • Randomization for all respondents

12. Clear Wording / Leading Questions • Clear Wording • Example: • “What ISP do you use?” • “If you have Internet service at home, what company or service provider do you use for Internet access?” • Leading Questions • Example: • “Don’t you think we should support our troops in Iraq?” • “How strongly do you agree or disagree with the following question: ‘We should support our troops in Iraq’”

13. Clear Formatting, Logic • Not all questions apply to everyone • Example: • “How much do you spend on gas heat each month?” • Branching is a possible solution • Example: • “Do you have gas heat? If yes, go to next question. If not, skip to question #3. • Condense when possible to avoid unnecessary branching. • Example: • “How much do you spend on gas heat each month? (write 0 if you do not have gas heat)

14. Know your sample population • Regional language and terminology • Cultural differences • How you conduct survey can influence your valid sample • Door to door? • Registered telephone directory? • Internet-based survey?

15. Replication and Using Existing Survey Instruments • ALWAYS a good idea to find other surveys that are used in your area of interest. • Especially with large, funded surveys the questions may have been tested for reliability. • Allows for comparisons between different samples if the question wording is the same. • If a question or set of questions is accepted as a good operationalization of the concept you are interested in, you don’t want to reinvent it unless you really intend to argue that your measure is more appropriate.

16. Pre-Testing and Pilot Studies

17. Testing a Survey Instrument • Pre-testing versus Pilots • Pre-tests: Focus on individual questions or the entire survey instrument/questionnaire. • Pilots: Usually larger scale than pre-testing, involve testing the entire survey procedure.

18. Pre-Testing and Pilots • Pre-tests and Pilots are always necessary, unless the survey in its existing form has already been given before. • Pre-testing and Pilot studies should have a large enough response rate so that you can actually find problems! • Example: You want to survey 100 undergrads for a small study. You may need to at least pre-test on a 20% sample with different undergrads than your intended valid sample (i.e., pre-test on 20 undergrads from your intended population, but not students who could end up in your final survey)

19. Testing your Questions • Does the respondent’s comprehension of question meaning match that of the researcher? • Does the researcher put too much of an expectation of recall on the respondent?

20. Ways to test your questions • Behavior coding– interview some respondents as you give the survey questions and keep track of requests for clarification. • Ask pretest respondents to rephrase your questions in their own words. • Panels of ‘experts’: give your questions to groups of individuals for comments/suggestions.

21. The pitfalls of skipping the testing stage • Best case scenario: you get some or most of what you wanted to get– but often an uphill battle with justifying your operationalizations and wording choices. • Worst case scenario: you get wild differences in responses, respondents don’t understand key questions, large incompletion rate, money and time spent on conducting survey is wasted (except for your newfound appreciation for pre-testing and pilots)

22. After the Survey: Data Coding and Error Checking

23. Data Coding • The coding considerations start with the survey itself. • You develop a codebook that records what the possible numeric responses will be for every question. • No open-ended questions unless absolutely necessary for other reasons.

24. Making data numeric • Use numbers to represent variable values • Assign a numeric value to all of the values that your variables can take. • Example: Gender (Male, Female) Male=0, Female=1. • Develop a systematic way of handling missing data! • You must enter a value for missing data– otherwise you will not know if missing is due to input error, N/A, skipped question, etc. • Example: use numeric codes that would not normally make sense for the variable (e.g., -9 for Missing, -8 for Not Applicable, etc).

25. Other tips for creating your dataset • Use ID numbers– always, no exceptions! seriously! • Datasets get manipulated and resorted constantly. Without ID’s, errors cannot be corrected, outliers cannot be identified. ID’s should allow you to match any case in the dataset with an actual survey taken by that individual. • Use conventional data structure. • Rectangular format, each row is a case and each column is a single variable.

26. Error Checking Data • Why? • Solves problems that may occur later • Makes sure your entire analysis is not bogus • You may accidentally engage in coitus more often as you get older….WHAT?!

27. Marital Coital Frequency • Jasso and Guillermina (1985) “Marital Coital Frequency and the Passage of Time: Estimating the Separate Effects of Spouses’ Ages and Marital Duration, Birth and Marriage Cohorts, and Period Influences” (American Sociological Review) • Major Findings of the Study: • Controlling for cohort and age effects, negative period effect • Controlling for period and cohort effects, wife’s age had a positive effect • Both findings differ significantly from earlier studies of the same topic.

28. Coitus: Part Deux • Kahn and Udry (1986) “Marital Coital Frequency: Unnoticed Outliers and Unspecified Interactions Lead to Erroneous Conclusions” (American Sociological Review) • Major Findings: • In the Jasso study, 4 cases were coded as 88– MISSING DATA CODES!!! • 4 more cases had very large studentized residuals (each was also very different from the first survey) • Missed an important interaction between length of marriage and wife’s age • Dropping the 8 outliers from the sample of more than 2000 cases drastically changed the findings

29. How to Error Check data • Know what you are looking to check, use appropriate methods: • Descriptives • Frequencies • Cross-tabulations

30. Error Checking Examples • Checking Original Variables for Errors • Frequencies • Descriptives • Checking and Setting “Missing” codes • Recoding and Creating New Variables from Existing Variables • Frequencies • Cross-Tabulations

31. Example: • Class Data Set