Survey Design

1. Survey Design

2. So Many Surveys • Political ‘push polls’ • Trying to influence, not collect data • Marketing polls • Some are just advertisements • News Polls

3. How to write a good survey • Think in terms of ‘questionnaire’, not questions • How are questions related to each other? • Have a clear topic • A research question • Know what concepts you are measuring • Think in terms of hypotheses, IVs and DVs • Every question should have a clear reason for inclusion

4. The Instrument – the whole package • Make it look/sound appealing • Clear directions that make it sound interesting/important • shorter is always better (i.e. 15 minutes) • Consistency is good • Use similar response formats for questions

5. Introductions and Instructions • What is the study about? • Informed and Voluntary Consent • Institutional Review Boards • Contact information for researcher • Let them know they can have the results

6. Example of Instructions

7. Question Order • Early questions make identities salient • Example: Political ID and Political Issues • So, Put space between the questions • Early questions should not be threatening • How often do you watch pornography? • Have you ever been convicted of a crime? • Early questions should not be boring • How old are you? • Of course, this all depends • Different survey settings call for different strategies • It’s a judgment call

8. Bad question order example (and probing)

9. Do previous questions influence answers?

10. Social Desirability • Sensitive Topics • sex, drugs, crime, embarrassment, church attendance, etc…. • People will sometimes tell you who they wish they were, not who they really are • Advice: write questions so that undesirable answers are made to sound ok.

11. Social Desirability Example • 1. How often do you attend mass at this parish? • Daily • Weekly • Few times a month • Few times a year • I’m a visitor today • I don’t know

12. Open and Closed Ended • Closed ended: give responses • Open ended: let respondent give comments • If closed: • Mutually exclusive and exhaustive • Allow don’t know / not sure ? • If open: • Will have to code after data collection • Might effect length of interview • Could be quicker or slower

13. Open vs. Closed Example • How many people do you consider close friends? ________ • How many people do you consider close friends? • 0 • 1 to 5 • 6 to 10 • More than 10

14. Avoid Double Barreled Questions • A single question that really asks two separate things. • Watch out for this if you use the word “and” • Try not to preface any question with something people might disagree with • Do you think the United States government should raise taxes and spend more money on education?

15. Double Barreled

16. Skip Patterns • Contingency Questions • Answer to a question sends you down a different path • If yes, go to question number X • Use it to shorten survey for respondent • Too many can get complicated • Use with caution on a self administered survey

17. Skip Pattern Example

18. Negative or Biased Terms • Negative terms may be confusing • The death penalty should be illegal • SDA DA N A SA • What does it mean if you disagree? • Avoid Biased Terms • Biased questions may encourage a certain answer • Depends on connotation of words

19. “Forbid” versus “Not Allow” • Do you think the U.S. should forbid public speeches against democracy? • Do you think the U.S. should allow public speeches against democracy? • “Americans are much more willing to not allow speeches than they are to forbid them.”

20. Some general tips • If asking about the past, use a time period people can remember • “In the last week…” • Only ask about personal experiences • Don’t ask about other people’s impressions • Beware of hidden contingencies • In the past month, have you crossed the street to avoid someone you found frightening?

21. Response Rate • % of those sampled who participated • Larger RR usually means more representative data • General Guidelines, no clear rule • Shoot for 70% • Happy with 50% • Don’t be surprised with 30% or lower

22. Methods of Administration – Mail Survey • Self Administered • Relies on Participant Enthusiasm • Low Response Rates • Appearance is Most important in this case • Give an Incentive? • Slow • Send a prior letter, maybe call • Send the survey • Send a reminder letter to those who haven’t replied • Contact again

23. Methods of Administration –Phone Survey • Random Digit Dial (RDD) • Very Fast • Very Expensive • Participation gets worse every year • Cooperation Rates • 20% to 30%, or lower • Statistically, in 30% range is not that different from in 60% range as long as sample is representative

24. Methods of Administration-In person interview • Best response rate & Most Expensive • Contact ahead of time to arrange meeting • Sit with respondent while collecting data • Well trained interviewers • Dress, Race, Gender, etc. • Threatening Situations • Mixed Methods • Computer Assisted • Self administered portions of an in person interview

25. Methods of Administration-Web Survey • Still a new method, very few clear guidelines • Visually, I’d say follow same guidelines as a a mail survey • How do you get your sample? • Do you trust online identities? • Can be very affordable • Survey Monkey

26. General comments on survey error • Sample surveys are way to generate estimates • There are multiple sources of error • Sampling error • Low response rates • Poorly written questions