Tips for Survey Design StudentVoice Tuesday, March 17, 2009

1. Tips for Survey DesignStudentVoiceTuesday, March 17, 2009

2. Overview of session • General introduction • Planning for a survey • Developing the survey • Things to consider/remember • Tips for writing questions • Response formats • Flow of the survey • Administration of the survey

3. Definitions: What is a survey? • As a noun: A detailed inspection or investigation. • As a noun: A gathering of a sample of data or opinions considered to be representative of a whole. • As a verb: To examine or look at comprehensively. • Survey research is the method of gathering data from respondents thought to be representative of some population, using an instrument composed of closed structure or open-ended items (questions). • Surveys are perhaps the dominant form of data collection in the social sciences.

4. Initial questions to ask… • Has someone already collected the information you are looking to gather? • (How) can you access the existing data? • (How) can you use the existing data? • Is there potential for collaboration with another individual, program, or department?

5. A note about methods of assessment • Direct Methods - Any process employed to gather data which requires subjects to display their knowledge, behavior, or thought processes. • Indirect Methods - Any process employed to gather data which asks subjects to reflect upon their knowledge, behaviors, or thought processes.

6. So you think you want to do a survey? • Alternative methodologies: Is there a better way to answer your questions? • What are the objectives of the survey? • What are the critical topics to address or questions to ask? • How will the results be used?

7. More specific questions… • Who will you survey? • Will you use a sample? • How will you select your sample? • How much will it cost? • What is your timeline? • How will you analyze and report the data?

8. General Survey Development Steps • Outline topic(s) and draft items • Choose response formats • Write and edit items • Determine item sequence • Determine physical characteristics of survey • Review and revise survey • Pilot test survey and revise

9. Surveys Musts • Must be well designed • Must be easy to answer • Take no more than 20-30 minutes, preferably much less time • Keep respondents interested • Should be visually appealing and free of error

10. Survey Construction • Use simple and concise language • Use consistent language (January 2007 - May 2007 vs. Spring 2007) • Ask one question at a time • Avoid loaded/biased questions • Format the question for the answer you want (When did you move here? vs. In what year did you move here?) • Write the question around the response categories (don’t have a frequency scale for a yes/no question) • Response categories should be grammatical answers to the survey questions • Use a question and answer format for easier processing (not “I am…”)

11. What makes for a ‘good’ question? • Will respondents interpret the item in the same way on a different day? (reliability) • Will the item mean the same thing to all people in the population? (reliability) • Will the item elicit the information you want? (validity) • Will the item elicit accurate and relevant data? (validity)

12. Obtaining Useable Survey Responses • Response options need to be mutually exclusive and exhaustive • Keep open-ended questions to a minimum • People interpret things differently, particularly when it comes to time, so frame things appropriately • Provide a meaningful scale • Consider a non-response option • The issue of neutral

13. Response Formats • Single response • Dichotomous • Multiple response/Check all that apply • Ranking • Fixed format/scales (e.g., Likert, frequency, quality) • Multiple response-rating scale, multi-item • Open-ended questions

14. Flow of the Survey • First items should be clearly connected to the goals of the survey • Begin with non-threatening but interesting items • Ask objective items before subjective items (e.g., "Have • you participated in .....?" followed by "How would you • rate.....?") • 4. Use a funnel approach asking broad general items first followed by more specific items • 5. Sensitive items should be asked well after the beginning of the survey but also before the end

15. Flow of the Survey • 7. Do not put important items at the end • 8. Put easy-to-answer items like demographic items at the end. If respondents refuse to answer demographic items, their other responses are still usable. Also put open-ended, and potentially ‘irritating’ questions at the end • 9. Group items into logically coherent sections and make smooth transitions between sections • 10. Arrange items: •   a. From most familiar to least familiar •   b. Chronologically (e. g., high school activities, college • activities, post-graduate activities) •   c. In logical order

16. Format/Design of the Survey • Make the survey look professional • Length itself, within reason, does not interfere with response rate • Should be long enough to cover the content but as short as possible •   - Keep items interesting •   - Make items relevant to the goals of the study •   - Limit the survey to essential items • Include contact information if people have questions or are interested in your study

17. Other General Survey Thoughts • Avoid “Wouldn’t it be nice to know” questions • Avoid educating people by way of a survey • Don’t ask what you already know • Look at each survey question and decide who would use the data and how it could be used • Pilot test your survey

18. Pilot your Survey! • Could take place in paper form or using the actual data collection method. • Take the survey yourself as if you were in the population being examined • Seek reviews from colleagues • Ask someone who knows nothing about the topic to review it for complete clarity

19. Questions to ask your pilot group: • How long did it take you to complete the survey? • What questions were confusing? • Were there any terms that you were not familiar with? • What questions would you definitely keep? • What questions would you consider getting rid of?

20. Do a Focus Group • Give group a copy of your survey and lead a discussion in which they critique both the methodology and instrument section by section, item by item • Have a researcher moderate the session and other researchers record comments • Do not publicly explain or defend the instrument -- accept all comments as helpful

21. Pros of Online/Web Surveys • Data collection: may know immediately whether an online survey has arrived via email. • Data collection: can know that email arrived to the intended person (assuming each person has an individual email account) • Data entry benefits: often data is compiled in real-time meaning no data entry. • Accuracy of the data: online removes human data entry error.

22. Cons of Online/Web Surveys • Sample may be unrepresentative of population. For example, studies have shown that there are distinct racial/ethnic differences in response rates to web-based surveys. • Access to Internet may be limited or not possible for your population • Invalid email addresses

23. Data Collection Strategies • Complete in classroom or group setting: • Best response rate because they are a captive audience! • Can control the environment • Mailing strategies: • Karen Bauer strategy: - Preannounce; mail survey; send follow-up announcement; mail second copy of survey.

24. Survey Basics: Cover Letters • Should convey the importance of the study • Should guarantee confidentiality • Should provide information on how long it will take • Should indicate the time period for which the survey will be accepting responses • Include information on incentives

25. Survey Basics: Response Rate • Explanation in cover letter what results are being used for • Create a culture of sharing results with students; over time they will see the value in thoughtful responses • Follow-up email sharing some findings • Cite assessment results in publicizing new programs/decisions • Develop a sampling strategy so each student gets asked to participate in only a certain percentage of surveys • Consider smaller incentives with higher chances of winning (give out three $10 gift cards instead of one $30) which has shown to be more effective • Incentives may be especially important in longitudinal studies

26. Survey Basics: Timing • Avoid busy times or holidays • Send email/ preannouncement 2-3 days prior to survey mailing • According to Bauer, first mailing should arrive on Thursday/Friday • First half of semester/term may be better if you are surveying in an academic environment

27. Become critical consumers of surveys! • What is the response rate? • What are the characteristics of the respondents? • Are differences significant? • Were these the right questions to be asking?

28. Resources • Fink, A. (1995). The survey handbook. Thousand Oaks, CA: Sage. • Fowler, F.J. (1993).Survey research methods. Newbury Park, CA: Sage. • Rea, L.M. (1992). Designing and conducting survey research: A comprehensive guide. San Francisco: Jossey-Bass. • Suskie, L. (1997). Questionnaire survey research: What works (2nd ed.) Tallahassee, FL: AIR.

29. Questions/Discussion