December 20, 2017

1. December 20, 2017 Catherine R. Messina PhD Dept Family, Population and Preventive Medicine Survey Design and Administration

2. “The scientist is not a person who gives the right answers, he's one who asks the right questions.” ― Claude Lévi-Strauss (1908-2009), French anthropologist

3. What is a survey or questionnaire? …….. a systematic method for collecting information from a sample of individuals for the purpose of quantitatively describing the attributes of the larger population of which the sample are members. • Groves, Fowler, Couper, Lepkowski, Singer and Tourangeau, 2004.

4. Benefits of surveys • Can reach a large number of people relatively easily • Provide quantifiable answers • Relatively easy to analyse Drawbacks of surveys • Provides only limited insight into problem • Information obtained is limited by questions asked • Always possible that wrong questions are asked • Varying response – respondent may misunderstand or misinterpret questions • Need to get it right first time • Can’t chase after missing data or badly collected data – can’t go back and try again

5. What is a good survey or questionnaire? …….. One that yields responses that are reliable (i.e., consistent) and valid (i.e., measures what it’s supposed to measure) measurements of what you are trying to describe

6. Quality aims for survey design • Valid: measures what it is supposed to measure • Reliable: measures what it is supposed to measure in a consistent and reproducible manner • Unbiased: measures what it is supposed to measure in a manner that does not systematically under- or over-estimate the true (population) value • Discriminating: can distinguish between respondents for whom the measured level of a concept or phenomena is different

7. Making inferences from survey data Characteristic(s) (attribute(s)) of Population You infer…. Characteristic(s) (attribute(s)) of Respondent Characteristic(s) (attribute(s)) of Sample You infer…. Respondent answers question(s)

8. Making inferences from survey data • Measurement is represented by – 3 related concepts: • observed value (aka the “measured value”) • true value • and measurement error Observed value = true value + measurement error • Well designed measurement tools (i.e., surveys) minimize measurement error so that the observed value is very close to the true value

9. Creating a survey or questionnaire • Good surveys are difficult to construct • Data obtained with bad surveys are difficult to interpret • Why are surveys difficult to construct??? • Each survey item must provide a measure that is valid and reliable • Each survey item must clearly communicate the research intention to the respondent • Respondents should have access to information needed to answer each survey item accurately • Respondents must be willing to provide the information you are asking for • The set of survey items must be assembled into a logical instrument that flows easily and naturally and will keep the respondent sufficiently engaged to complete it in a thoughtful manner

10. Creating a survey or questionnaire • First, determine clearly what it is you want to measure in order to answer your research question(s) and hypotheses • Go back to the primary and secondary aims of your project! • Be very clear about what you want to know • Consider how you plan to analyze your data • Define your targeted population and your survey sample • Know your respondents! • What are their needs and capabilities regarding survey administration (e.g., literacy levels; access to internet, etc.)

11. Creating a survey or questionnaire • Define the data (variables) you need to collect - decisions about relevant variables are based on the research question (or hypothesis) • List all relevant variables! • Demographics; clinical measures; behavior; attitudes; knowledge; etc….. • Define how these will be measured • Define the role of each variable in your statistical analyses • Dependent variables • Independent variables • Covariates! - A secondary variable (and of interest to the researcher) that is observed and which can affect the relationship between the IVs and the DV • Will gender or age affect outcomes? • What about parent education? Family size? Family income? Insurance? Co-morbid conditions? Gestational age? ETC………

12. Creating a survey or questionnaire • Review the literature for existing measurement instruments • Determine if questions that you want to include have already been developed (and tested) by other researchers • Review primary research articles – if questions not included look for on-line supplemental info

13. Existing measurement instruments • Advantages of existing measurement tools • Saves development time • Measures that have been used by others provide opportunities for valuable comparisons – especially important if publication of evaluation is a goal • Reliability and validity of existing instruments may be known (at the very least, you can get an idea of how well they perform in other settings)

14. Existing measurement instruments • Cautions! • Usually not a good idea to “adapt” or change existing instruments – but if you do, describe this in your Methods section and note this as a potential limitation in Discussion section • Be cautious if using only a subset of questions from an existing survey as this may change the meaning of the summary scores. Check whether a shorter version of the instrument exists that has also been validated

15. What if there are no existing measures on my topic, and I need to develop my own survey or questionnaire? • Advantages of creating new measures: Potential for greater relevance and validity because tailored to your need

16. Developing Surveys – General Approach • Determine the mode of survey administration: self-administered or interviewer administered • Each has its own advantages and disadvantages …… • and will determine your survey format

17. Types of surveys • Self-administered (e.g. “paper and pencil”) • Hand out • Mailed • Email / web-based • Interviewer administered • Face to face • telephone

18. Self-administered surveys • Advantages: • Relatively inexpensive and easy to administer • Preserves confidentiality (can be anonymous) • Can be completed at respondent's convenience • No influence by interviewer • Disadvantages: • Low response rate – especially email surveys which sift to bottom of inbox very quickly • Questions can be misunderstood • No control by interviewer

19. Interviewer-administered surveys • Advantages: • Facilitates participation by people with low literacy • Interviewer can clarify ambiguity • Disadvantages: • May introduce interviewer bias • Needs more resources – more expensive • Difficult for sensitive issues

20. Developing Surveys – General Approach • Decided on operational definitions • How the researcher chooses to measure a particular variable • E.g.: your child’s age – can be operationally defined as” • DOB • How old your child was on her/his last birthday (in years) • What age group best describes your child • Use measure that offers most flexibility

21. Developing Surveys – General Approach • Consider using standard demographic questions • Facilitates comparing your sample to others in the literature • Examples from national health surveys • Look at the way these are asked in similar studies • Only ask for necessary demo information • Race / ethnicity – always should be comprised of two questions (NIH format): • Are you Hispanic or Latino? (yes vs. no) • What is your race? (allow for multiple answers)

22. Developing Surveys – General Approach • Create a pool of new questions or revise existing questions – write more questions than will be included in the final draft • Put questions in sequence • Group questions that are similar • Put them in a logical order • Place demographic questions at the beginning • Put any sensitive or difficult questions at the end • Put any open-ended questions at the end

23. Developing Surveys – General Approach • Format survey draft • Give your questionnaire a topic-related title • Include a brief explanation of the survey’s purpose (short paragraph) • This includes all instructions for filling out survey and possible skip patterns! • Include appropriate white space to get a good estimate of the length

24. Developing Surveys • If possible, ask experts to review for content • Content validity: do the items in a questionnaire adequately represent the universe of items relating to a specific construct? How accurately does a measurement instrument tap into the various aspects of a specific construct?

25. Developing Surveys • Get feedback (e.g., advisor; peers) on your 1st draft • Face validity: is the measurement logical on the “face of it”? The degree to which the purpose of a measurement instrument is obvious to those using it or responding to it, i.e., “looks like” or appears to respondents as a measure of depression. • Advantage of good face validity: respondent really understands the point of the questions and interprets them accurately • Disadvantage of good face validity: respondent really understands the point of the questions and “tailors” their responses to “fake good” or “fake bad”

26. Developing Surveys • Get feedback (e.g., advisor; peers) on your 1st draft – con’t • Which questions measure IV(s)? • Which questions measure DV(s)? • Which questions measure covariate(s)? • Are questions clear and “answerable”? • Are any questions potentially offensive? • Does format make sense? Is it logical? • Length of survey? • REVISE!! • Get more feedback (e.g., advisor; peers) on your revised draft

27. Developing Surveys • Pilot-test on a small sample of respondents who are similar to your study sample • Are questions clear and “answerable”? • Are any questions potentially offensive? Questions should be culturally sensitive!! • Does format make sense? Is it logical? • Length of survey and how long to complete? (consider participant burden) • Cognitive interviewing and “think aloud” • Ask a few people in pilot sample to read the questions out loud to you and then explain to you, what each question means to him/her and how he/she thought about their answer • Test-drive your coding system – prelim analyses • REVISE!

28. Developing Questions • If possible, pilot-test the near final draft of the survey with new pilot sample • Pilot-test all procedures and logistics for administering survey • Self or interviewer administered? • Who will administer? • Who is responsible for collecting completed surveys? • Where will these be stored before pick-up?

29. Developing Surveys – Specifics

30. Characteristics of effective survey questions • Your goal is to write survey questions that : • Consistently understood – i.e., all respondents will interpret the same way (and interpret in the way that you interpret them) • Respondents are able to respond accurately – all respondents have the information to answer the question (except in cases of knowledge questions) • Respondents are willing to answer

31. Writing good survey questions • Only include questions you plan to use • Too many questions (especially if they seem unrelated to the purpose of your survey) can irritate respondents (an IRB issue too) • Avoid lengthy or complex questions – keep it simple • Write at a low reading level – understand the literacy level of your respondents • Use conventional language! • Use words relevant to respondents - terms and concepts should be familiar to respondents and easy for them to understand • Avoid jargon and acronyms

32. Writing good survey questions To ensure that respondents consistently understand your questions: • Provide clear and concrete definitions of key terms • Sugary drinks include soda, sweetened ice teas, all fruit juices and fruit drinks • Provide all needed information so that respondents can answer the question • E.g., Not counting any sugary drinks your child may have been served outside your home, in the past week how many times did you serve sugary drinks at home. • Instructions need to be placed exactly where needed - not at the beginning of the entire survey. • Use italics or bolding or underline to emphasize instructions and directions

33. Writing good survey questions Write questions that respondents are able to respond to accurately: • E.g., Survey examining video game exposure among children: • In a typical weekday, how many hours does your child spend playing video games at home?  0  1- 3  4-6  more than 6 WHAT DOES THE “0” RESPONSE MEAN?

34. Writing good survey questions Write questions that respondents are able to respond to accurately: • DON’T MAKE ASSUMPTIONS! • Do you have a video game player at home?  No (skip next question and go to question X)  Yes • If yes then, in a typical weekday, how many hours does your child spend playing video games at home?  0  1- 3  4-6  more than 6

35. Writing good survey questions Write questions that respondents are able to respond to accurately: • How many hours do you walk each week:  0 hours  1- 3 hours  3-6 hours  6 or more hours CAN THIS QUESTION BE ANSWERED ACCURATELY?

36. Writing good survey questions Write questions that respondents are able to respond to accurately: • MAKE CHOICES EXCLUSIVE • How many hours do you walk each week:  0 hours  1- 3 hours  4 - 6 hours  more than 6 hours

37. Writing good survey questions Write questions that respondents are able to respond to accurately: • When your child is at home, where is she/he most likely to experience second hand cigarette smoke:  Car  Kitchen  Backyard CAN THIS QUESTION BE ANSWERED ACCURATELY?

38. Writing good survey questions Write questions that respondents are able to respond to accurately: • MAKE CHOICES EXHAUSTIVE • When your child is at home, most is she/he most likely to experience second hand cigarette smoke:  Car  Kitchen  Backyard  Playroom  Bedroom  Other(s) ______________ (please specify)

39. Writing good survey questions • Avoid “and” “or” (double-barreled) questions • Trying to respond to a question measuring two or more ideas is confusing to respondent and their response is difficult for the researcher to interpret • E.g., Do you think that increasing physical activity and / or losing weight is beneficial to you child’s health? • E.g., Do you think that eye contact and / or using words you understand improve communication between you and your child’s doctor • E.g., Were you satisfied with the timeliness and quality of the health care service your child received?

40. Writing good survey questions • Avoid leading questions • Do you think that the food in the hospital made your child sick? • Do you agree that the hospital staff were overworked?

41. Writing good survey questions • Avoid bias by using both positive and negative sides in the question stem “To what extent do you agree….” vs. “To what extent do you agree or disagree with this statement…. “

42. Writing good survey questions • Make sure the question and the response options match Has your child had pain in the last week?  Never  Seldom  Often  Very often • Vs……..

43. Writing good survey questions How often has your child had pain in the last week?  Never  Seldom  Often  Very often

44. Writing good survey questions • Avoid vague quantifiers How often did you use aspirin for pain relief in the last month?  Never  Rarely  Occasionally  Regularly • Vs……..

45. Writing good survey questions How often did you use aspirin for pain relief in the last month?  Not at all  About once in the last month  Two or three times in the last month  About once a week  More than once a week

46. Writing good survey questions • Ask questions that people can answer confidently Which of the following do you think increases a child’s chances of developing diabetes the most?  Being overweight  Family history of diabetes  Not getting enough exercise Vs ………….

47. Writing good survey questions Which of the following do you think increases a child’s chances of developing diabetes the most? Being overweight  yes  no  don’t know Family history of diabetes  yes  no  don’t know Not getting enough exercise  yes  no  don’t know

48. Writing good survey questions • Ask questions that people are willing to answer! • Do not be too personal or risky unless absolutely needed • Be culturally sensitive • People are more likely to answer questions about behaviors and health compared to income or education • Using income categories rather than asking for exact $ amounts is a better strategy

49. Writing good survey questions • Ordering questions • Two key considerations: • Questions should be ordered such that the response to one question does not influence the response to the question that follows it • Questions should be ordered such that respondents are more likely to complete the survey • Fact-based questions before opinion based questions • Start with non-threatening or least sensitive questions • Most important questions should be placed first • Vary question format to reduce likelihood that respondents will “auto-pilot” response sets

50. Writing good survey questions • Good questions follow comfortably from the previous question. • Requires good writing skills • Smooth transitions between questions or groups of questions • Group similar questions together • Avoid non-response (i.e., missing data) by including response options for “Don’t Know” and “Refused” or “Prefer not to answer”