Writing Survey Questions

1. Writing Survey Questions Teri Peterson Teri Peterson

2. Introduction • Writing questions for survey instruments • Learn to ask questions effectively • Write a valid and reliable survey instrument • Resist the temptation to ask questions that might be of interest Teri Peterson

3. Valid and Reliable instrument • Well written instrument • Purposeful questions • Questions determined by objectives • Concrete questions • Complete sentences • Review by experts • Pilot with potential respondents Teri Peterson

4. Good Questions • Address the survey’s objectives • Are concrete and specific • Clearly address only one issue per question • Use appropriate language and reading level for the respondents • Do not contain words that promote bias • Do not contain double negatives Teri Peterson

5. Purposeful Questions • The respondent can identify the relationship between the question and the objective of the survey • The connection between demographic questions and survey objectives may be missing • Prefix the demographic question with a statement clarifying the connection Teri Peterson

6. Concrete questions • Precise and unambiguous • Less concrete: How much do you exercise? • More concrete: In the past week how many days did you participate in an activity that made you breath hard for at least 30 minutes? More concrete Teri Peterson

7. Conventional Language • Use complete sentences and proper grammar. • Race? • Which of the following categories best describes your ethnicity? • Caucasian • Native American/American Indian • … Teri Peterson

8. Conventional language • Avoid abbreviations • Avoid slang and colloquial expressions • Only use jargon and technical expressions with appropriate audiences Teri Peterson

9. Open vs. Closed Questions • Open Questions • Use when: • You don’t know what types of answers to expect • Want the respondents view of the world rather than yours • Answers must be cataloged and interpreted • Difficulties • Responses difficult to compare and interpret • Not practical for large sample sizes Teri Peterson

10. Open vs. Closed Questions • Closed questions • Advantages • Easier to interpret • Better for statistical analysis • Good for large samples • Surveyor’s expectations are more clear • Answers tend to be more reliable and consistent • Disadvantages • Researcher must have understanding of most likely responses • Does not always allow respondent to express their interpretations/feelings Teri Peterson

11. Response Choices • Type of data: • Nominal: purely categorical, mutually exclusive, collectively exhaustive • Ordinal: has an inherent order • Strongly agree, agree, neither agree nor disagree, disagree, strongly disagree • Excellent, very good, good, fair, poor • Always, very often, fairly often, sometimes, almost never, never • Numerical Teri Peterson

12. Ordinal Measures • Include a “do not know” if appropriate • Include a neutral response if appropriate • Balance all responses • Use a 5- to 7-point numbered scale • For socially undesirable behaviors put the negative end first. Teri Peterson

13. Measuring Attitudes • Psychometrics • Very complex entities, difficult to define and measure • Consider using a published measure • Is it suitable for your respondents? • Does it truly ask what you need to know? • Get the owner’s permission. Teri Peterson

14. Demographics • Age, race/ethnicity, education, job, gender, marital status, geographic place of residence, size of family, and so on. • Learn the characteristics of your target population. • Important for describing sample. • Important to help explore your findings. Teri Peterson

15. Demographics • Base your response categories on the characteristics of your target population. • Decide on an appropriate level of specificity. • Ask for exact information in an open-ended format. • Use current words and terms. • If you want to compare to other studies, use the same categories. Teri Peterson

16. Validity • Face Validity • Does the survey seem to ask the needed questions? • Content Validity • Does the survey thoroughly and appropriately assess what it intends to? • Criterion-Related Validity • Does your survey correlate with other criteria known to measure what you are measuring? • Construct Validity • Does your survey behave consistent with other theoretically derived constructs Teri Peterson

17. Reliability • Consistency across time • Consistency within the survey • Test-retest: stability • Alternate-form: equivalence • Internal consistency: homogeneity • Inter- and Intra-rater reliability Teri Peterson

18. Context • Know your respondents • Reading level • Language comprehension • Ask only questions they could be expected to know answers to! • Facing unanswerable questions is frustrating! • Consider the time constraint • How much time will the survey take? • Standardize the response format • Consider the social, cultural and economic context. Teri Peterson

19. Lastly • Always allow plenty of time upfront for survey development! • Always pilot test. • Always have others review what you have written. • Always base your survey on your survey objectives. • Always work with the Human Subjects Committee! Teri Peterson

20. Thank You for Coming! • Feel free to contact me at anytime. • Peteteri@isu.edu • Office: 208-282-4861 • Fax: 208-282-4367 Teri Peterson

21. Qualitative Surveys • “Collect information on the meanings that people attach to their experiences and on the ways they express themselves.” • Necessitate small samples, often not generalizable. • Provide depth and uniqueness rather than breadth and representation. • Often requires content analysis. Teri Peterson