Survey Research

1. Survey Research

2. Terms to Know Questionnaire or Survey – A tool commonly used in research to gather data; these may include written questions, focus groups, and both formal and informal interviews. Research– A formal and systematic structure of investigation. Sound research employs the scientific or problem-solving attitude and method. Most research is done for one or more of the following purposes: (1) to describe "what was" in an attempt to determine the historical aspects of change, growth, and development (2) to describe "what is" in making a description involving some type of current phenomena

3. Terms to Know… continued Anonymity – Assurance that the research subjects remain anonymous or nameless; the subject’s identity is protected, and the individual is unknown. Confidentiality – Assurance that information collected during research may have identifiers attached to it, but the researcher holds it in confidence or keeps it secret from the public; the information is not released in a way that permits linking specific individuals to specific responses or information and is publicly presented only in aggregate form.

4. Questionnaires • Mail Surveys • Group Administered Questionnaires • In-person • Electronic Interviews • Personal Interview • Telephone Interview

5. Selecting your Survey Method • Do you have access to the population of interest? • Is the population of interest literate? • Are there language barriers? • Will the population of interest be cooperative? • Are there geographical restrictions?

6. Selecting Your Survey Method • Will you use open- or closed-ended questions? • Will the questions require complex answers? • Will you need to follow-up questions? • Will respondents need to check records?

7. Selecting Your Sampling Method • Will you be able to make contact with respondents? • Can all members of the population of interest be sampled? • Are response rates expected to be high or low?

8. Issues • Cost • Time • Projected return rate • Convenience • Bias

9. Survey Bias • Response Bias • Respondents may be untruthful because they are embarrassed, uncomfortable, or afraid • Non-response Bias • Respondents may refuse to answer questions or neglect to return questionnaires • Question bias • What do you see as the benefits of a tax cut? • What do you see as the disadvantages of eliminating welfare?

10. Survey Bias… Questions • Wording of Questions • Question may lead or mislead the respondent • Questions that are too vague may be misunderstood • Questions that are too specific may result in too large a variety of responses to be useful • Qualifying words such as always, very, may, etc. may be interpreted differently by respondents • Vocabulary may be at an inappropriate level • Respondents may become fatigued or bored if there are too many questions

11. Question Writing… • Example 1- check your own bias! • Poor: During the 1987-88 school year, HCPSS made a poor decision implementing the “No Smoking” policy. • Better: The “No Smoking” policy, which was implemented during the 1987-88 school year, will help students stop smoking. • Example 2 – don’t get too personal! • What is your annual income last year? • In which category does your income last year best fit? (provide groupings) • Example 3 – Just ask one question. • Poor: Should the school system revise its smoking or field-trip policies? • Better: Should the school system revise its smoking policy?

12. Types of Questions • Dichotomous Questions (two choices) • Example: Do you believe that the death penalty is ever justified? __yes __no • Level of Measurement Questions • Nominal (number labels) • Example: Occupation: __1 truck driver, __2 lawyer, __3 teacher • Ordinal (rank) • Rank the candidates in order of preference from best (1) to worst (4): __ Bob Dole, __ Bill Clinton, __ Newt Gingrich, __ Al Gore

13. Types of Questions continued • Level of Measurement Questions continued • Interval (Likert scale rating) • Example: The death penalty is justifiable under some circumstances. __1 Strongly Disagree, __2 Disagree, __3 Neutral, __4 Agree, __5 Strongly Agree • Interval (semantic differential) • Example: Please state your opinion on national health insurance by checking the appropriate box below: very much somewhat neither somewhat very much Interesting boring • Contingency Questions (if yes, …)

14. Types of Questions • Open-ended • Responses created by responded. • Interpretation is subjective. • Closed-ended • Fixed choices • Rules… length, clarity, abbreviations, jargon

15. Types of Questions • Attitudes – what people say they want. • Should ___ be taught in elementary school? • Beliefs – what people think is true. • ___ is one of the most serious problems in our school. • Behavior – what people do. • Have you visited your child’s school in the past year? • Attributes – what people are. • Do you have children enrolled in the public school system? In what grade levels?

16. The Importance of Language • Avoid ambiguity in directions and key terms. • Consider pre-testing the survey to eliminate “bugs.” • Don’t ask “leading” questions.

17. Question Order • Start with easy, nonthreatening questions and put more difficult questions towards the end • Do not start with an open-ended question • Ask about one topic at a time, and use a transition if switching topics • Ease into questions of a personal nature

18. Question Wording • Can a question be misunderstood? • What kind of headache remedy do you use? • Do you want to know the brand name medication, home remedy, or if they prefer pills, capsules, etc.? • Is there a time frame? • Words like “will”, “could”, “might”, or “may” imply time. • Do you think Congress will cut taxes?

19. Question Wording continued • How personal is the wording? • Are working conditions satisfactory or not satisfactory in the plant where you work? (objective) • Are you personally satisfied with the working conditions in the plant where you work? (personal feeling) • Is the question too direct? • How do you feel about being in war? (direct) • How well were new recruits trained? (less direct)

20. Graphability of Responses • Open-ended questions yield narrative answers that are hard to depict graphically. • Closed-ended questions ask respondents to record attitudes and facts by circling a provided choice or marking a location on a “Likert” scale. • Interview questions yield graphable answers only if all interviews are equal.

21. Common Problems • Low response rate • Misjudging survey design or response turn around time • Forgetting to write “over” at the bottom of a two-sided questionnaire • Making a questionnaire too long • Expecting respondents to remember or know what they probably can’t • Example: Do you think Dean Rusk acted correctly in the Bay of Pigs crisis?

22. Common Problems… continued • Intruding on respondents’ privacy • Asking questions that are sufficiently general and specific • Instead of, “How well did you like the book?” ask, “Did you recommend the book to others?” • Partial responses to questions or whole questionnaires • Questions that lead to unanswerable questions • [1]“Do you own your own business?” yes no [2]“How many workers do you employ?” …full time? part time? None, because I don’t own a business…etc.)

23. Common Problems… continued • Lacking the information to insure a response (consider cover letters) • Not taking editing and overall appearance of the questionnaire seriously • Not allowing any open-ended response opportunity

24. Surveys at School • Common Problem… • Not abiding by school system protocols pertaining to surveys: • Not during instructional time • Not about personal behaviors pertaining to drugs, sex, etc. • Not getting approval of principals • Protecting anonymity of students

25. Final Thoughts • Is confidentiality of respondents assured? • How will responses be used? • Will respondents have access to results? • More information? • This could all go in your cover letter!

26. References • Trochim, W. (2006, 10 20). Web Center for Social Research Methods. Retrieved July 21, 2008, from Web Center for Social Research Methods Web site: http://www.socialresearchmethods.net/ • Bock, D, Velleman, P, & De Veaux, R (2007). Stats: Modeling the World. New York: Addison-Wesley.

27. Let’s Critique Some Surveys! • Are the questions written using appropriate language and reading level for the population of interest? • Will respondents have the necessary background information to answer the questions? • Are follow-up questions necessary? • Are response rates likely to be high or low? • Is response bias likely? If so, how could it be lowered? • Is interviewer bias likely? If so, how could it be lowered? • Is non-response bias likely? If so, how could it be lowered? • Are the questions leading? Misleading? • Are the questions too vague? Too specific? • Is the ordering of questions appropriate? • What changes would you recommend to improve the quality of the survey?