Data Collection Toolkit III: Surveys

1. Data Collection Toolkit III: Surveys Research Methods for Public Administrators Dr. Gail Johnson Dr. G. Johnson, www.ResearchDemystified.org

2. Surveys Are Everywhere • End of term evaluation of faculty performance • Political polls • Policy preferences surveys • Employee surveys • Customer satisfaction questionnaires Dr. G. Johnson, www.ResearchDemystified.org

3. Types of Surveys • Self-administered • Participants complete on their own • Mail, email and web-based • Administered • Researcher asks the questions • Phone survey • Interviews Dr. G. Johnson, www.ResearchDemystified.org

4. General Survey Guidelines • Keep it simple, clear, easy, short • Respect respondents’ time and intelligence • Tell them how they were selected and why their participation is important • Do no harm: keep responses confidential Dr. G. Johnson, www.ResearchDemystified.org

5. General Guide: • Establish procedures that are followed • Maintain accurate records of definitions, coding • Pre-test, pre-test, pre-test, pre-test • Verify accuracy of coding, data input Dr. G. Johnson, www.ResearchDemystified.org

6. Reminder: Writing Questionnaires • Use clear, simple language. • Ask questions that are unambiguous. • Encourage a range of responses. • Avoid yes or no questions. • Ask only one question at a time. • Focus on current experiences. • Leave exits (no opinion, not applicable). • Pre-test, pre-test, pre-test. Dr. G. Johnson, www.ResearchDemystified.org

7. Closed-Ended Questions • Single question with a defined response set: How satisfied or dissatisfied are you with the program? (Select only one) • Very satisfied • Somewhat satisfied • Neither satisfied or dissatisfied • Somewhat dissatisfied • Very dissatisfied • No opinion Dr. G. Johnson, www.ResearchDemystified.org

8. Open-Ended Question: Fill in the Blank • Useful at the end of the survey: • Ask for “any other comments?” or “what one change would you make to the program?” Dr. G. Johnson, www.ResearchDemystified.org

9. Types of Surveys • Mail questionnaire • Cyber-survey (email or web-based) • Phone survey Dr. G. Johnson, www.ResearchDemystified.org

10. Option: Mail Surveys • Use mail questionnaires when: • Seeking opinions and viewpoints • Target population is literate and large • Quantitative analysis is needed • Capacity to do statistical analyses exists Dr. G. Johnson, www.ResearchDemystified.org

11. Survey Options: Mail • Usually very structured, using closed-ended questions • Relatively inexpensive (costs are printing and mailing) • Long time to develop and implement • Close-ended responses are easy to analyze • But if you use a wrong word, unforgiving of mistakes • Should take less than < 30 minutes • Better for sensitive questions or confidential issues Dr. G. Johnson, www.ResearchDemystified.org

12. Do Not Use Mail Surveys • When seeking actual behavior: • What people say they do might not be what they actually do • When seeking very specific data from records or files: • People probably won’t remember • People probably won’t want to take their time digging up that information Dr. G. Johnson, www.ResearchDemystified.org

13. Details: Cover Letters • Personally addressed • Identifies who is asking for the survey • States purpose and overview of questions • States how the information will be used • Assures confidentiality • Results will only be reported in aggregate Dr. G. Johnson, www.ResearchDemystified.org

14. Details: Cover Letters • Provides name of contact person • Provides a good estimate of amount of time to complete the survey • Provides instructions for returning the survey • Must have: stamped, self-addressed envelope Dr. G. Johnson, www.ResearchDemystified.org

15. Details: Layout of Survey: • Introduction: who your are, purpose of research, overview of survey, assurance of confidentiality, contact person information. • Questions: logical groupings. • Begin with easy questions, move to the core questions, end with demographics. • Last question is open-ended: “any comments?” Dr. G. Johnson, www.ResearchDemystified.org

16. Key Issue: Response Rates • How many of the people who were asked to participate actually participated? • Response rates are a crucial piece of information needed to assess the credibility of the survey results. • No agreed upon standard but 60% is a good target. Dr. G. Johnson, www.ResearchDemystified.org

17. Key Issue: Response Rates • If 300 surveys were sent to MPA graduates from the past 5 years and 100 were returned, the response rate is 33%. • This is a low response rate but not unusual. • It is really, really, really hard to get a high response rate. • It helps if you have a captive audience: like students in a class setting. Dr. G. Johnson, www.ResearchDemystified.org

18. Low Response Rates Create A Volunteer Sample • Low response rates are a “threat to validity” of the results; the results might be biased • Jargon term: low response rates might also be called a non-sampling error • Tough question: Do the respondents represent all 300 students? • What if they tended to be the most angry students because the satisfied students had no strong motivation to give their opinions? • What if they tended to be the most satisfied students because the angry students were too cynical to even bother? Dr. G. Johnson, www.ResearchDemystified.org

19. Prevent Low Response Rates • Do what you can to encourage participation • General: Identify who you are, explain why and how they were selected, why their views matter, assure confidentiality, make sure survey looks professional, provide an incentive, be prepared to do several follow-ups, make it easy for them to participate. • Make sure you are asking good questions; pre-tests care really help Dr. G. Johnson, www.ResearchDemystified.org

20. Improve Response Rates • Self-addressed, stamped envelope • Incentive • Follow-up reminders • Post cards • Second survey Dr. G. Johnson, www.ResearchDemystified.org

21. Improve Response Rates • Easy to read • White space • Easy to complete • Short • Only ask necessary questions Dr. G. Johnson, www.ResearchDemystified.org

22. Recovery Options • If response rate is low: • Compare demographics of the respondents to the demographics of the graduates • If similar, then more strength that the results are represented of the larger group • Contact a small sample of the non-respondents and survey them. Compare their results • If similar, then more strength that the results are represented of the larger group Dr. G. Johnson, www.ResearchDemystified.org

23. Option: Cyber-surveys • Email: embedded in or attached to an email • Requires access to email address • Useful when surveying an organization with known emails • But might be treated as Spam! Dr. G. Johnson, www.ResearchDemystified.org

24. Option: Cyber-surveys • Web-based • Need to let people know it is there • Often creates the database so researchers do not have to enter any data (avoids errors and saves time) • But very little control over who participates (volunteer sample where response rates are meaningless) Dr. G. Johnson, www.ResearchDemystified.org

25. Cyber-Surveys • These still require the same care and attention to detail that a mail survey does • Writing good questions is essential • If wrong word is used, wrong question is asked, or question is unclear, they are unforgiving of mistakes Dr. G. Johnson, www.ResearchDemystified.org

26. Cyber-Surveys • Requires computer access and literacy: may exclude some members of the community • Plus side: generally cheaper (but depends on cyber-host) because avoids printing and postage, labor-saving: no stuffing and licking stamps Dr. G. Johnson, www.ResearchDemystified.org

27. Cyber-Surveys: Risks • Bias, Bias, Bias • Little control over who participates • May be no way to determine demographics • May be no way to compare to the larger population • These can turn into “American Idol” votes • Not representative of anyone other than those who participated Dr. G. Johnson, www.ResearchDemystified.org

28. Survey Options: Phone • Usually very structured • Simple questions • Takes time to develop • Easy to analyze Dr. G. Johnson, www.ResearchDemystified.org

29. Survey Options: Phone • Moderately expensive • Quick way to gather data • Can set a target for number of participants and keep dialing until it is reached • Interactive (but some danger in how researchers explain survey questions) Dr. G. Johnson, www.ResearchDemystified.org

30. Survey Options: Phone • Develop a scripted introduction • The verbal equivalent of a cover letter • Your name, who is conducting the survey • States purpose and overview of questions. • States how the information will be used. • Assures confidentiality • Provides an accurate time estimate for completion. • Provides name of contact person. Dr. G. Johnson, www.ResearchDemystified.org

31. Survey Options: Phone • Shorter the better: less than 15 minutes • Works best for simple issues, like are you likely to vote for X, Y or Z? • Hard to get people to answer their phones • Also, needs to accommodate cell phones • When developing questions, be mindful how words sound: some words sound the same but mean something different Dr. G. Johnson, www.ResearchDemystified.org

32. Pre-test All Surveys • Select a few typical respondents: • This is not a review: ask them to actually complete the survey • Mail surveys: be present when they complete the survey • Keep track of how long it takes to complete • Observe if they seem to have difficulty or turn to previous pages Dr. G. Johnson, www.ResearchDemystified.org

33. Pre-test All Surveys • For phone: run through as if the real thing • Keep track of time and clarification questions • Cyber survey: check all protocols, links: make sure everything works Dr. G. Johnson, www.ResearchDemystified.org

34. Pre-test Surveys • Debrief: • What was clear, what was not? • What questions are missing? • What questions are unnecessary? • Make changes based on pre-tests. • Pre-test again. Dr. G. Johnson, www.ResearchDemystified.org

35. DiscussionSurvey Options: Phone vs Mail • When might a phone survey be better than a mail survey? • When might a mail survey be better than a phone survey? • Consider your own experiences in participating in phone surveys and completing surveys. • What worked? • What did not? Dr. G. Johnson, www.ResearchDemystified.org

36. Takeaway Lessons • It is hard to avoid surveys—and hopefully, you will not want to. • Managers will want to use employee and customer surveys as a way to get needed information. • The key is make sure surveys contain good questions—especially when using a quantitative approach • How the questions are written matter: • Garbage in, garbage out applies. Dr. G. Johnson, www.ResearchDemystified.org

37. Takeaway Lessons • Pre-tests really matter but often are not done because: • Researchers run out of time. • Researchers believe that everyone will understand the questions. • Make sure contracted research builds in pre-test time. • If you use survey results, you want to know who they surveyed, how many, how they were selected, the response rate, and the exact questions that were asked. Dr. G. Johnson, www.ResearchDemystified.org

38. Creative Commons • This powerpoint is meant to be used and shared with attribution • Please provide feedback • If you make changes, please share freely and send me a copy of changes: • Johnsong62@gmail.com • Visit www.creativecommons.org for more information Dr. G. Johnson, www.ResearchDemystified.org