1 / 23

Developing Effective Surveys

Developing Effective Surveys. Steve Culver, Ph.D. Associate Director Office of Academic Assesment 122 Hillcrest (0157) sculver@vt.edu September, 2009. Today’s Agenda. Discuss survey design as a process Provide tips to enhance the quality of the process and therefore the data.

edana
Download Presentation

Developing Effective Surveys

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Developing Effective Surveys Steve Culver, Ph.D. Associate Director Office of Academic Assesment 122 Hillcrest (0157) sculver@vt.edu September, 2009

  2. Today’s Agenda • Discuss survey design as a process • Provide tips to enhance the quality of the process and therefore the data

  3. Definitions • A survey is a data-gathering and analysis approach in which respondents answer questions or respond to statements developed in advance. • IRB – Institutional Review Board must approve “research”

  4. Types of surveys • Face to face (usually most expensive) more control, sometimes can collect more data • Telephone fairly controlled setting, can ask for clarification, but phone screening, fewer hours at home • Mail (takes longer) Respondent can complete at leisure, can include pictures; no control over who completes survey, potential illegible data; bias - literacy • Web (e-mail survey or survey posted on web) more consistent (reliable) than human interviewer, but no human element to identify problems, no control over data errors; respondents give longer answers to open-ended questions.

  5. Steps in the process • Identify research objective • Identify & characterize the target audiences • Design the sampling plan • Design & write the questionnaire • Pilot test the questionnaire • Distribute the questionnaire • Analyze the results & write the report

  6. Identify research objective • Define precisely the information desired • Try to write as few questions as possible to obtain that information • A clear-cut need for every question should be established

  7. Identify & characterize the target audiences • Is it a wide range of people or a niche group? • What is there interest in the topic? • What is their reading/educational level? • Are there other important characteristics to consider that relate to your study? (how would you know?)

  8. Design the sampling plan • Address how a sample of people should be selected to meet the study objectives and to obtain reliable information. • Need to identify the accessible population. • Sampling methods: random, systematic, stratified • Follow up with the nonrespondents.

  9. Improving your response rate • Increase perceived rewards say thank you, ask them for advice, make questionnaire interesting • Decrease perceived costs avoid subordinating language, don’t embarrass the respondent, make it quick and easy to respond • Establishing trust make the questionnaire seem important, provide a sense of legitimate authority, provide a token

  10. Design & write the questionnaire • As simple, as short as possible • Unambiguous items • No leading questions • No double-barreled questions • Questions follow a logical sequence • Put personal or confidential questions at the end.

  11. Layout & Design • Clear introductory statement & contact information • KISS & lessen clutter • Number items and be consistent with wording, font (serif font like Times easier to read than a san serif font). • Use italics and bold judiciously and with a purpose. • Careful with color & limit graphics

  12. Particularly for web surveys… • Design the survey as multiple pages, with a submit button at end of each section (not one long page) • Use a progress button. • Decide if respondents can exit and re-enter survey. • Design so that it takes no longer than 20 minutes to complete the survey.

  13. Question order • Begin with questions that raise interest but are easy to answer • Then ask questions of interest that are harder to answer • Personal questions at end after respondent has committed to answering (though completion of web surveys is an issue, so maybe move some to middle) • Make logical groupings

  14. Types of questions • Open-ended vs. closed-ended • CE = easy to standardize, analyze • OE = easier to write, draw in respondent • Single vs. multiple response • Choose one; choose all that apply • Ranked responses • Indicate order of importance of a list • Rated responses • Likert scales

  15. Writing tips • Avoid double barreled questions • Do you think college students today should eat less and exercise more? • Avoid questions with negatives • Are you against a ban on smoking? • Ask for precise answers • Your age on January 1, 2009

  16. More writing tips – Response scales --allow for maximum variability (no more than 10; less than h.s. educ., 5) --use a balanced scale --“neutral” or “no opinion” vs. “don’t know” (odd/even number of responses) --Order response categories in a logical way

  17. Pilot test the questionnaire • Field pre-testing • Focus group discussions • Individual interviews – “think aloud” • Behavior coding does the respondent ask for clarification? How long does it take to answer each question?

  18. Distribute the questionnaire • After revision from pilot test, distribute the questionnaire • Document the how, where, when • Document response times, nonrespondents • Follow up nonrespondents • So, distribution is seen as a longitudinal process

  19. Analyze the results & write the report • think in terms of useful information directed toward change, rather than data • Clear presentation – not excessive pretty charts and graphs

  20. In summary . . . • Be mindful that you are using someone else’s time for your survey • Think of the survey process as a process, not just an instrument • Be careful of extending your results to make them mean more than they do

  21. References Dillman, D. A. (1978). Mail and telephone surveys: The total design method. NY: Wiley. Porter, S. (2004). Pros and cons of paper and electronic surveys, overcoming survey research problems new Directions for Institutional Research, 121, 91-99. Frary, R. B. (1996). Brief guide to questionnaire development. Washington, DC: ERIC Clearinghouse on Assessment and evaluation.

  22. Helpful websites • Resources for program evaluation and social research methods http://gsociology.icaap.org/methods/ • Guide to the design of questionnaires http://iss.leeds.ac.uk/info/312/surveys/217/guide_to_the_design_of_questionnaires

  23. Questions/Comments? Please take a few minutes to complete the session feedback Form. Thank you!

More Related