NEWMOA Workshop on Survey Instrument Design

1. NEWMOA Workshop on Survey Instrument Design Linda Piccinino

2. The Importance of Questionnaire Design • “Planning the questionnaire is one of the most critical stages in the survey development process.” • American Statistical Association (1997)

3. Purposes of Surveys for the EPA • Measure results of pollution prevention and compliance assistance efforts • Show level of compliance with regulations • Determine assistance needs • Demonstrate results in a quantitative manner

4. Types of Survey Mode • Mail (letter, email) • Telephone • In-person

5. Group Exercise 1 Learning How to Define Survey Content and Scope • Writing survey questions to address research questions

6. Types of Survey Administration • Group interviews • Structured interviews • Semi-structured interviews • Self-administered questionnaires

7. Other Factors to Consider • Questionnaire length • Questionnaire format and appearance, physical size • Question order

8. Question Wording and Structure

9. Comprehension and Clarification • The question should define words in the way the researcher intended • The question wording and structure should be clear about what event is being asked about

10. Comprehension and Clarification Example: • Were you robbed during the last six months?

11. Comprehension and Clarification Revision: • Was something belonging to you stolen, such as – a) Things that you carry, like luggage, a wallet, purse, briefcase, book b) Clothing, jewelry or calculator c) Bicycle or sports equipment d) Things in your home – like a TV, stereo or tools e) Things from a vehicle, such as a package, groceries, camera OR f) Did anyone ATTEMPT to steal anything belonging to you? Source: National Crime Victim Survey Questionnaire

12. Examples: How many hours did you work LAST WEEK at all jobs? Did you lose any time or take any time off LAST WEEK for any reason such as illness, holiday or slack work? Did you you work any overtime or at more than one job LAST WEEK? Source: Schaeffer, 1993 Problems: Respondents include reports about usual hours worked Questions about hours lost and extra hours follow request for total hours Comprehension and Clarification

13. Comprehension and Clarification Revision: • How many hours do you USUALLY work at your (main) job? • LAST WEEK, did you lose or take off any hours from (work/your main job) for any reason such as illness, vacation, holiday, labor dispute or layoff? • LAST WEEK did you work any overtime or extra hours (at your main job) that you don’t usually work? • So, for LAST WEEK, how many hours did you ACTUALLY work at your (main job/other jobs)?

14. Specifying the Reference Period • The reference period should be as clear and concise as possible. • Memory aids such as a calendar or markers such as important dates (holidays, birthdays, etc.) might help respondents to think about the reference period you are asking about.

15. Example: In the last 7 days, not counting today, how many times did you smoke at work? Source: Belson, 1989 Possible Solution: I am going to ask a few questions about things you might have done while at work LAST WEEK. By last week I mean the week beginning on Sunday June 4, 2006, and ending on Saturday June 10, 2006. In the LAST WEEK, how many times did you take a cigarette break during work hours? Specifying the Reference Period

16. Tips • Look at each question and ask yourself “How will this information be used?” • Put yourself in the respondent’s place – how would you answer the question?

17. Stages of the Survey Response Process

18. The Survey Response Process Four steps in generating a response: 1) Comprehension 2) Retrieval of information 3) Response formation 4) Response mapping and reporting

19. The Survey Response Process Extent to which each stage in the response process is difficult depends on both the question and the respondent. • How many bowls of cereal did you consume in the past week? (retrieval process) • Have you gained 5 or more pounds in the past year? (judgment process)

20. Respondent Cognitive Tasks Comprehension: • What does the respondent think the question is asking? • What do specific words or phrases mean? Retrieval: • What type of information does the respondent need to recall? Does the respondent possess the information required? • What are some difficulties associated with the respondent retrieving information? What types of strategies does he/she use (e.g., counting events by recalling each oneindividually or providing an estimate based on usual behavior) Source: Willis, G. Cognitive Interviewing.

21. Respondent Cognitive Tasks • Response Formation: • Does the respondent have the motivation to use the retrieved information to form a response? • Does response formation pose particularly difficult tasks for the respondent? • Does the respondent tell the truth? Does he/she falsify his/her response? Does the question raise privacy concerns? • Response Mapping: • Can the respondent match his/her answer to the response options provided?

22. Tips • We focused on comprehension, but respondents can have difficulties at any stage of the process • If you know the stage that is causing difficulty, you have a better chance of fixing the problem

23. Response Categories • Frequencies: Absolute and Relative • Rating Scales • Dates • Open and Closed Questions

24. Frequencies • How many boats do you shrink wrap a year? • Source: New Hampshire Dept. of Environmental Services Marina Questionnaire • How often do you dispose of unwanted mercury from your dental office?

25. Rating Scales • Bipolar • Interval • Categorical

26. Dates • Exact dates • Ranges: Approximate dates, durations and estimates

27. Open and Closed Questions • Open-ended: • How did you learn about this program? • Closed: • What is your RCRA hazardous waste generator status? • Large Quantity Generator • Small Quantity Generator • Conditionally Exempt Small Quantity Generator • Don’t know

28. Common Problems with Survey Questions

29. Problemswith Survey Questions Common Question Problems: • Complicated questions • Unfamiliar or technical terms • Ambiguous concepts or words

30. Question Problems: Complicated Questions Example: Over the last month, how many times per month, week, or day did you drink 100% fruit juice such as orange, apple, grape or grapefruit juice? Do not count fruit drinks like Kool-Aid, lemonade, HI-C, cranberry juice drink, Tang, and Twister. Include juice you drank at all mealtimes and between meals. (never, 1-3 times last month, 1-2 times per week, 3-4 times per week, 1 time per day, 2 times per day, 3 times per day, 4 times per day, 5 or more times per day). Source: NCI All-Day Screener

31. Question Problems: Unfamiliar Terms Example: Do you think that a compliance assessment is a satisfactory tool to measure results of compliance assistance efforts? Example: I have a sweet tooth. (strongly disagree, disagree, neutral/NA, agree, strongly agree) Source: Eating Behavior Patterns Questionnaire (2003)

32. Question Problems: Ambiguous Concepts or Words • Examples: • In the past 3 months, how often did you eat a vegetarian dinner? (usually/always, sometimes, rarely, never) • Source: Fat and Fiber Behavior Questionnaire (1997) • Do you now… • Spend half an hour or more in moderate or strenuous physical activity at least 3 times a week? (Yes/No) • Source: NMES - SAQ

33. Question Problems • Question assumptions not applicable to respondent • Loaded questions • Double-barreled questions

34. Common Question Problems: Assumptions Examples: About how many gallons of “perc” do you use in one year? Source: EPA Dry Cleaners Survey (2004) Do you have a Non-Contact Cooling Water Discharge of 5,000 or more gallons per day?(yes/no/not applicable/don’t know) Source: Garment Care Business Information (2004) How long have you been limiting the amount of fat in your diet? (less than 30 days, 1-6 months, 7-12 months, over a year) Source: National Food Stamp Program (1996)

35. Common Question Problems: Loaded Questions Examples: Would you say you are now complying with EPA standards? Are you using environmentally preferable products?

36. Common Question Problems: Double-Barreled Examples: What did you do to lose weight and maintain your weight loss? Source: California Dietary Practices Survey (2001) During the winterization process, did you store engines “wet” and “dry?”

37. Question Problems • Overlapping response categories • Recall difficulty • Sensitive questions

38. Common Question Problems: Overlapping Response Options Examples: How many boats do you sand per year? Fewer than 5 boats 5 to 10 boats 10 to 20 boats More than 20 boats Which of the following main dishes do you think is lowest in fat? (fish sticks, pork ribs, barbeque ribs, chicken) Source: National Food Stamp Program Survey (1996)

39. Common Question Problems: Recall Difficulty Examples: How many newspapers did you recycle in the past month? How many glass bottles, cans of paint, cardboard boxes or batteries did you dispose of since you started working at the hospital maintenance division?

40. Common Question Problems: Sensitive Questions Examples: If you are not using EPP (environmentally preferable) or bio-based products, why not? Source: Connecticut Hospital Environmental Roundtable Follow-up Survey (2005) Do you maintain your weight at recommended levels? (Y,N) Source: Health Behaviors Module (2000)

41. Practice Example Identify problems with the following survey question: • Would you describe the amount of butter or margarine you usually spread on breads and muffins as ____? (none, light, moderate, generous) • Source: Diet and Health Knowledge Survey (DHKS) 1994-1996

42. Group Exercise 2 • Critique of Existing Survey Questions • Based on what has been learned so far, identify problems and offer solutions • Revise questions from Exercise 1

43. Tips • Look at each question – Can it be simplified? • Reconsider words that are not recognized by your spellchecker

44. Pretesting

45. Pretesting Methods • Methods • Low cost options • Consequences of not pretesting

46. Debriefing • Final questioning that occurs after the respondent has completed the questionnaire and reviewed all materials • Interviewer may ask open-ended question to probe further into certain comments or observations the respondent made during the interview • Respondents may also be asked to provide an assessment or overall critique of the instrument

47. Sample Debriefing Questions • Think about the questionnaire as a whole. Does anything in particular come to mind that you want to comment on? What are your general feelings about it? Did you feel you were prepared for the types of questions asked? • In general, did you have difficulty remembering what you did?

48. Sample Debriefing Questions • I noticed that you chuckled at several points. Can you tell me what caused you to do so? • Were there any specific questions that you had trouble answering? Why? • Were there any questions that you felt uncomfortable answering?Why? • What questions do you think most needed to be revised? Do you have any suggestions for how they might be better written?

49. Group Exercise 3 • Pretesting of Questionnaire • Debriefing

50. Group Exercise 4 • Revise and retest questions from Exercise 3 • Review of entire questionnaire development process – what are the key points and problems associated with each step?