Quick and Painless Introduction to Survey Methodology

1. Quick and Painless Introduction to Survey Methodology R. Michael Alvarez PS 120

2. Testing Theories or Models • Experimental data: expensive, and has validity problems • Quasi-experimental data: aggregate election statistics, other data. Suffers from various problems. • Survey data: data about individual voters

3. Fundamentals of Surveying • Population: all elements of interest, usually in a geographic area • Sample: subset of population • Sample frame: list of sample (addresses, phone numbers, email addresses, etc)

4. Basic Typology of Surveys • Probability designs: population elements have a known (at least in theory) probability of selection into the sample. • Nonprobability designs: population elements have an unknown probability of selection into the sample • All of the statistical tools we use to study survey data are based on probability designs!

5. Literary Digest 1936: What Went Wrong?

6. Literary Digest Methodology • Sent out 10 million straw ballots, using a list drawn from auto registration lists and telephone books. • 2.3 million were returned, about 25% response. • Flawed sample (overrepresented rich and Republicans) • Low response rate

7. Literary Digest Fiasco Reforms Polling • Underlying flaws of Literary Digest straw polls revealed --- not using a scientific sampling procedure • Others, especially Gallup, Roper and Crossley began to work to find better ways of generating samples … • The Literary Digest soon folded!

8. Problems Continue, 1948

9. New Sampling Techniques Were Flawed! • Before 1948, they used “quota sampling” • Each interviewer is assigned a fixed quota of subjects to interview from certain demographic categories … gender, age, education, residential location. • Once they met their quota, the interviewer could select anyone they desired until they conducted all their required interviews

10. Quota Sampling • It’s not necessarily a stupid idea, as long as the underlying data (Census data?) used to construct the parameters of the sample are okay. • But, what can happen is that interviewers end up working to talk with people who are easy to contact. In 1948 that tended to be people in nice neighborhoods, with fixed addresses and phones (ie, Republicans).

11. Random Sampling • In the 1950’s, most scientific surveys shifted to the use of random sampling • For example, Gallup in 1956 moves to the use of random selection methods and seems to generate more accurate presidential election forecasts thereafter

12. Gallup’s Track Record

13. Basic Introduction to Sampling • Concept: The population (or universe or target population). • The population is the entire set of units to which a survey will be applied. Individual members of the population are called units or elements.

14. More on sampling ... • Next, we need a list of population units from which we can draw a sample. • This list is called the SAMPLE FRAME • The basic property of a sample frame is that every unit in the population has some known chance of being selected into the sample by whatever method is used to select units

15. Then ... • Probability sample: units are selected using a method that insures that each unit has a known, nonzero probability of being included. • Nonprobability sample: units are selected and inclusion probabilities are unknown (quota sampling …)

16. Simple Random Sampling • All elements of population have equal probability of being sampled • Cluster sampling: population is divided into clusters or groups, and clusters are sampled. Why? Cost and simplicity. • Stratified sampling: population is divided into subpopulations, or strata, and sampling occurs within strata. Why? Strata might be of interest or require different methods of analysis.

17. Sampling Error • Best way to think of survey error is in the context of proportions (percent saying “yes” or “no”). • Standard error of a proportion in SRS: • se(p) = sqrt[ ( p(1-p) )/( n - 1 )]

18. An Example of Survey Error

19. An Example of Nonresponse Error? March 2001 CSLP RDD

20. NES Response and Refusal Rates Response rate: interviews net of refusals and respondents who cannot provide an interview (e.g., language, etc)

21. Misreporting: Voting in Recent Federal Elections Note: Percentage of voting age population

22. Item Nonresponse • Don’t know is necessary in any survey, so that people can tell you if they don’t have an opinion • Due to uncertainty, vague questions, or respondent unwillingness to answer some questions

23. Should Gov’t Provide More Services? 1996 NES

24. Certainty of Responses? Senator Position on Abortion Scale, Alvarez and Franklin 1993

25. Question Wording and Order? • Would you say that traffic contributes more or less to air pollution than industry? (45% traffic primary contributor to 32% industry). • Would you say that industry contributes more or less to air pollution than traffic? (57% industry primary contributor to 24% traffic) • Wanke et al. 1995

26. Types of Surveys • Self-administered questionnaires (mail, web) • cheap • but: • low response rates • uncertainty about who completes questionnaire

27. Types of Surveys • Telephone: RDD/CATI • quick, random? • Uncertainty about respondent, difficult to ask complex questions, must be short

28. Types of Surveys • Face-to-face (on doorstep, exit polls) • highly accurate, high response rates • very expensive to implement • interviewer biases are problematic

29. Internet surveying --- the future? • Cheap to implement • Quick in the field, quick with analysis • Can implement complex designs, for example, use multimedia

30. Basic types of Internet surveys • Probability designs • Nonprobability designs • Mixtures of probability and nonprobability

31. Intercept-based surveys of visitors to particular web sites known email lists (students, etc) Probability-based Internet surveys

32. Nonprobability Internet surveys • Entertainment surveys • Self-selected surveys • Volunteer survey panels

33. Sampling error (difference between sample and pop.) Coverage error (deviation between sample and frame) Systematic sampling error; error in frame Nonresponse (unit) bias Nonresponse (item) bias Question wording or ordering effects Interviewer error; coding mistakes Surveys are not perfect!

34. Sample size Sampling methodology (probability or non-probability) Estimated sampling error Survey response rate Questionnaire design and question wording Item response rates Intuition: do the results make sense? How do I evaluate survey results?

35. Caltech’s NationalPublic Relations Initiatives March 11, 2003

36. Brief recap of survey methodology • Survey conducted by ICR • Wednesday, February 12-Sunday February 15 • Omnibus survey • N=1010 • Tabulation presents weighted results, weighted to map to American adult population

37. Questions • Considering what you might have seen or heard about the California Institute of Technology, also known as Caltech, in Pasadena, California, which of the following best describes your opinion of Caltech’s reputation. Would you say Caltech’s reputation is excellent, good, fair, or poor?

38. Questions • How did you hear about Caltech? (not asked to those unable to answer 1). • What do you think Caltech is best known for (not asked to those unable to answer 1).

39. Questions • Now, as I read each of the following topics, please tell me, generally speaking, whether or not you are interested in the topic: voting, the brain, climate changes, astronomy, earthquakes, nano-technology, detecting gravity waves

40. Questions • And considering those topics in which you said you had an interest, how do you usually get news and information about these topics? (asked to only those who were interested in at least one topic)

41. National Awareness of Caltech

42. Caltech Awareness Successes

43. Caltech Awareness Challenges

44. Caltech’s Reputation

45. Media Relations Focus • National and northeast TV - visit them, pitch them, invite them to campus. • Households with children • Senior-oriented media

46. Evaluate the Caltech Awareness Survey • Technical evaluation • Substantive evaluation • Policy evaluation