1 / 67

Cognitive Interviewing and Questionnaire Design

Cognitive Interviewing and Questionnaire Design. Gordon Willis, Ph.D. National Cancer Institute National Institutes of Health willisg@mail.nih.gov. Preface: Why worry about questionnaire design problems?. On the one hand: A self-report sample survey is a ‘blunt instrument’

meriel
Download Presentation

Cognitive Interviewing and Questionnaire Design

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cognitive Interviewing and Questionnaire Design Gordon Willis, Ph.D.National Cancer Institute National Institutes of Health willisg@mail.nih.gov

  2. Preface: Why worry about questionnairedesign problems? • On the one hand: • A self-report sample survey is a ‘blunt instrument’ • There is bound to be some ‘slop’ • To the extent questionnaire design IS rocket science, we’re content to just hit the moon, somewhere… • On the other hand: • Of the various types of errors afflicting surveys – Sampling error, Non-response error, Processing error…: • Response Error (e.g., bias or unreliability) is often the largest category -> G Willis 2012 Cog Interviewing USUHS

  3. Do small differences in wording…produce differences in data? VERSION 1 (No filter) On a typical day, how much time do you spend doing strenuous physical activities such as lifting, pushing, or pulling? __ None __ Less than 1 hour __ 1-4 hours __ 5 + hours VERSION 2 (Filtered) On a typical day, do you spend any time doing strenuous physical activities such as lifting, pushing, or pulling? IF YES: Read Version 1 G Willis 2012 Cog Interviewing USUHS

  4. Evaluation study: Embed target questions, using three split-sample survey experiments 1) Field Pretest of NCHS National Health Interview Survey: 78 adult household respondents 2) Women’s Health Study: 191 women, 18-41, in contractor offices 3) Random-Digit Dial (RDD) Survey: 989 respondents in Maryland Omnibus telephone survey (Willis & Schechter, 1997) G Willis 2012 Cog Interviewing USUHS

  5. Survey experiment results: Screening for physical activity On a typical day, how much time do you spend doing strenuous physical activities such as lifting, pushing, or pulling? 0 <1 1-4 5+ FIELD PRETEST (n=78) No filter 32% 32% 35% 0% Filtered 72% 18% 10% 0% WOMEN’S HEALTH (n=191) No filter 4% 42% 50% 4% Filtered 49% 16% 27% 8% RDD SURVEY(n=989) No filter - 45% 34% 22% Filtered - 62% 29% 9% G Willis 2012 Cog Interviewing USUHS

  6. Survey experiment results: Screening for physical activity On a typical day, how much time do you spend doing strenuous physical activities such as lifting, pushing, or pulling? 0 <1 1-4 5+ FIELD PRETEST (n=78) No filter 32% 32% 35% 0% Filtered 72% 18% 10% 0% WOMEN’S HEALTH (n=191) No filter 4% 42% 50% 4% Filtered 49% 16% 27% 8% RDD SURVEY(n=989) No filter - 45% 34% 22% Filtered - 62% 29% 9% G Willis 2012 Cog Interviewing USUHS

  7. The cognitive testingprocessin a nutshell • Develop a questionnaire or material (advance letter, etc.) to be evaluated • Recruit (paid) members of the targeted population (e.g., recipients of home loans, people without employment, cancer survivors) • Conduct one-on-one interviews, in “laboratory” or other location: • Home • Homeless shelter • Health clinic • Elderly center G Willis 2012 Cog Interviewing USUHS

  8. The cognitive testing process in a nutshell • THEN: Make informed decisions, with cognitive testing as one source of input – • We also rely on – • a) Expert Review (Topic area and/or questionnaire design); • b) Review of measurement objectives; • c) Consideration of past practice, need for comparability G Willis 2012 Cog Interviewing USUHS

  9. The cognitive testing process, in a nutshell • To conduct the cognitive interview, we use verbal probing techniques to elicit thinking about question • Finding: Note apparent problems related to question wording, ordering, format • Fixing: Suggest modifications that address problems • Cognitive testing is best done as an iterative process (multiple small testing ‘rounds’) G Willis 2012 Cog Interviewing USUHS

  10. Tourangeau (1984) cognitive model • Encoding of question (understanding it) • Have you ever received care from a podiatrist? • Retrieval of information (knowing/remembering) • Do you approve of the Budget Reconciliation Act of 1993? • How many times have you ridden in a passenger airplane? • Decision and judgment processes (truth, adequacy) • How many sex partners have you had in the past 12 months? • Response (matching internal representation to given categories) • “Would you say your health is excellent, good, fair, or poor?” G Willis 2012 Cog Interviewing USUHS

  11. Focus: Cognitive and other problems • Focus is NOT just on “cognitive” problems: (1) Problems with question assumptions Do you own or rent your house? (2) Problems for interviewer Question is tongue-twister, etc. (3)Format issues Where do I go next, for self-administered questionnaire (“skips”) (4) Issues that influence flow of the interview Subject thinks question is over and interrupts with answer • As such, has been called “Intensive interviewing” (Cantril & Fried, 1944(!); Royston, 1989) G Willis 2012 Cog Interviewing USUHS

  12. Think Aloud interviewing • Mainly advocated by Ericsson and Simon (1980, 1984) as a means to study cognitive processes used in problem solving (chess…) • The interviewer requests a verbal “train” as subject completes the task • The interviewer interjects little except to say “Keep talking” or “Tell me what you’re thinking” • In the extreme case, avoid the use of references to “me” the interviewer(!) G Willis 2012 Cog Interviewing USUHS

  13. Example of think-aloud • INTERVIEWER: How many times have you talked to a doctor in the last 12 months? • SUBJECT: I guess that depends on what you mean when you say “talked.” I talk to my neighbor, who is a doctor…I go to my doctor about once a year, for a general check-up, so I would count that. I’ve been to…a specialist a couple of times in the past year - once to get a bad knee diagnosed, and I also saw an ENT about a coughing thing, which I’m pretty sure was in the past year, although I wouldn’t swear to it. I also talked to doctors when I brought my kids to the pediatrician - I assume that you don’t want that, though I’m not sure. Also, I saw a chiropractor, but I don’t know if you consider that to be a doctor…. So, I’m not sure what number to give you… G Willis 2012 Cog Interviewing USUHS

  14. Limitations of “pure” Think-Aloud • Many subjects are not good at this • My experience: male, teenage smokers… • Was not designed for a task that involves social interaction (survey…) • Is very different from the normal question asking-answering sequence • Analysis of think-aloud protocol is not straightforward • We can use a coding system (e.g., Bickart & Felcher, 1996), but this is complex and burdensome G Willis 2012 Cog Interviewing USUHS

  15. The alternative to Think-Aloud:Verbal Probing Techniques • Ask follow-up questions – • About the questions • About the answer • About the general context • Not new: Used by Cantril (1944), Belson (1980)… • Verbal probing is more active than think-aloud • Requires decision by the interviewer about how to probe -> This is what makes the activity interesting! G Willis 2012 Cog Interviewing USUHS

  16. Basic varieties of (“classic”) verbal probes Comprehension probe: What does the term “dental sealant” mean to you? Paraphrase: Can you repeat the question in your own words? Confidence judgment: How sure are you that your health insurance covers… Recall probe: How do you know that you went to the doctor 3 times…? Specific probe: Why do you think that breast cancer is the most serious health problem? ‘General’ probe: How did you arrive at that answer? G Willis 2012 Cog Interviewing USUHS

  17. The combined approach • Cognitive interviews are normally done as combination of Probing and Think-Aloud (DeMaio & Landreth, 2004) • But, the relative mix of these may vary across type of investigation G Willis 2012 Cog Interviewing USUHS

  18. Verbal probing techniques: Concurrent versus retrospective approach Concurrent probing:Probe immediately after the subject has answered each survey question Advantage: Probing when the memory still exists Disadvantage: Measurement process interrupts normal flow Retrospective probing:Wait until after the interview, and then go back to probe Advantage: Mirrors “field” procedures - Is particularly useful for self-administered Q’s Disadvantage:Subject may have forgotten key information G Willis 2012 Cog Interviewing USUHS

  19. Thesis: • C.I. depends on more than just the process of asking probe questions • We need to start with a working knowledge of questionnaire design • We can’t just make up probes haphazardly • We can’t just leave it to the subject to tell us something is wrong!!! • Active probing requires the targeting and recognition of problems • Model: Question Appraisal System -> (Willis & Lessler, 1999; Willis, 2009) G Willis 2012 Cog Interviewing USUHS

  20. G Willis 2012 Cog Interviewing USUHS

  21. More evidence that we shoulddo ‘Expert Review’ G Willis 2012 Cog Interviewing USUHS

  22. Fashioning probes, in depth G Willis 2012 Cog Interviewing USUHS

  23. Tested (“classic”) question: Pain in the abdomen “In the last year have you been bothered by pain in the abdomen?” What (Anticipated) probes make sense here? • What time period are you thinking about, exactly? • What does “bothered by pain” mean to you? • Where is your “abdomen?” G Willis 2012 Cog Interviewing USUHS

  24. G Willis 2012 Cog Interviewing USUHS

  25. G Willis 2012 Cog Interviewing USUHS

  26. Revised question:Pain in abdomen (Please look at this diagram. During the past 12 months, have you had pain in this area…) • Remember to check out S’s interpretation of vague reference periods like “year,” “month,” “week,” or no reference period at all (e.g., “How often do you X?”). • Argument: “Lose this example” – It’s obvious that “Abdomen” is a problem • Response: Yes, but sometimes we need to demonstrate that which we think we know G Willis 2012 Cog Interviewing USUHS

  27. TV, Smoking… During the past 30 days, about how often have you seen anti-smoking commercials on TV? Probe that was used (Paraphrase): What was that question asking you? • Subject: “About smoking commercials – I don’t think there are any” • So, he heard “anti-smoking” and thought “smoking” -- NO GOOD  A straightforward answer can hide a “Silent Misinterpretation” (DeMaio and Rothgeb, 1996) G Willis 2012 Cog Interviewing USUHS

  28. Fixing… potential alternatives • During the past 30 days, have you seen any TV commercials about the dangers of cigarette smoking? • During the past 30 days, have you seen any TV commercials that warned people NOT to smoke? • During the past 30 days, have you seen any messages on TV warning people NOT to smoke? -> Which is best? Depends on objective – may need to consult with sponsor/client -> Cognitive interviewing is often a “vagueness detector” which brings out hidden ambiguity G Willis 2012 Cog Interviewing USUHS

  29. Beyond Proactively searching • Illustrative example of tested question: Interviewer: “Inside your home, are there any walls that have peeling paint?” Subject: No… not on the walls, anyway… I (Probe): Is there any paint that’s peeling? S: Yeah, the window frame… • Problem wasn’t anticipated beforehand – so neither was the probe • Cog Interviewing needs a capacity not only for problem Verification, but also for reaction to the unexpected: Discovery G Willis 2012 Cog Interviewing USUHS

  30. Easy default: “tell me more…” • Example: I: How often is your household trash collected? S: Twice a week. I: Tell me more about your trash pickup. S: The garbage gets picked up on Monday, and the recycling on Wednesday.” • NOT GOOD: The objective concerns trash, not recycling • Probing revealed that the tested version doesn’t work -> Good example of a Silent Misinterpretation G Willis 2012 Cog Interviewing USUHS

  31. How would you probe? What types of smokeless tobacco products have you used in the past year? • Proactive forms: • To you, what’s a smokeless tobacco product? • When did you last use (X)?; How sure are you that this was within the past year? • Reactive forms: • It dependson what we hear -> G Willis 2012 Cog Interviewing USUHS

  32. How would you probe? What types of smokeless tobacco products have you used in the past year? • How about if given this response: “Well, I used to chew tobacco…” • Does he mean in the past year, or longer ago? • I would first follow up on the Reactive part, before Proactively “fishing” - -> Follow problems before searching for them • So:How long ago did you last chew tobacco? • You can then follow up with other (Anticipated) probes:Besides chewing tobacco, what do you consider tobe a “smokeless tobacco product” G Willis 2012 Cog Interviewing USUHS

  33. An alternative to probing… listening When you first thought you were pregnant, how did you feel? Were you - Happy, Unhappy, Both happy and unhappy, or Neither happy nor unhappy Say that we observe a favorite subject ploy: She jumps in at “how did you feel” with “I felt awful!” So, the respondent thinks the question is over (or that it should be…) G Willis 2012 Cog Interviewing USUHS

  34. So, we change the question… When you first thought you were pregnant, were you - Happy, Unhappy, Both happy and unhappy, or Neither happy nor unhappy Subject: “I don’t get all those different happy-unhappy things” -> Again, we need only to listen – no probing may be necessary G Willis 2012 Cog Interviewing USUHS

  35. So, we change the question… When you first thought you were pregnant, were you… Happy, Unhappy, or did you have mixed feelings? Lessons: 1) There may be multiple problems 2) Iterative testing is good 3) We may not need to probe – but, we react differently from a field interviewer (who is motivated to “make it work”) G Willis 2012 Cog Interviewing USUHS

  36. Avoiding Probing Pitfalls G Willis 2012 Cog Interviewing USUHS

  37. Probing “fouls” • I: …what about contamination of the ground water. Would you say it’s a very important problem, a somewhat important problem, a small problem, or not a problem? • S: “I guess… hmmm. I’m not sure how to answer that, really.” • I: “Why don’t you know how to answer it?” Problems with probe? • Kind of accusatory • I’m not sure that a person who can’t answer can further report why they can’t answer • I would prefer “Ok, tell me what you’re thinking” G Willis 2012 Cog Interviewing USUHS

  38. Probing fouls: continued • (I: “Why don’t you know how to answer it?”) • S: “Well, I don’t know how to answer it, because I don’t know really what your mean… um…” • I: OK, whatever it means to you. Again, please indicate whether it’s a very important problem, a somewhat important problem, a small problem, or not a problem” • S: “Oh, it’s a very important problem” Problems with probe? -> G Willis 2012 Cog Interviewing USUHS

  39. Probing fouls: continued (I: OK, whatever it means to you. Again, please indicate whether it’s a very important problem, a somewhat important problem, a small problem, or not a problem” ) (S: “Oh, it’s a very important problem”) PROBLEMS WITH PROBING: • “Whatever it means to you” is a field interviewer activity – designed only to “get through the questionnaire” • Do we know that S is still thinking about the actual topic of the question (what is “it” that’s an important problem?) • Responses to probing are subject to social desirability effects (just as survey questions) G Willis 2012 Cog Interviewing USUHS

  40. Rule of probing flaws: • Probes follow all the usual rules of questionnaire design • They may be too long • They may be vague • They may produce “huh?” response • They can be biasing • They can be irrelevant to the exchange • If so, they must be fixed, or abandoned! G Willis 2012 Cog Interviewing USUHS

  41. Modifying probes to recover from communication failure: Wellens (1994) I: Are you a citizen of the United States? S:No. I: In your own words, what does the term “citizen” mean to you? S: I don’t… I don’t understand I: I just want to know what you think the definition of the word “citizen” is. S: A citizen is a person who belongs to this country. That person has the right to vote or join the government. I: Can you tell me more about why you are not a citizen…? S: A person must live here for five years, take a test, obtain history course, then the person can become citizen.” G Willis 2012 Cog Interviewing USUHS

  42. Adapting to Survey Administration Mode G Willis 2012 Cog Interviewing USUHS

  43. Adapting cognitive interviewing to Face-to-face administration • This is the default – the situation that is most similar to the in-person cognitive interview • So, there’s no not much to adapt to • The main thing to keep in mind is that the cognitive interviewer is not the same as the field interviewer -> So, pay attention to problems for the interviewer as well as for the subject G Willis 2012 Cog Interviewing USUHS

  44. Adapting cognitive interviewing to Telephone administration • Important issue: • Can C.I. be done over the phone? • Should C.I. be done over the phone? - We miss out on body language - But, phone may be the mode of fielded survey administration • NCHS procedure: • Put subject in separate room and call him/her • Conduct interview over phone, but with camera • Go to Subject location, debrief in-person G Willis 2012 Cog Interviewing USUHS

  45. Adapting cognitive interviewing to Telephone administration • Alternative - • Conduct an initial interviewing round face-to-face, and a second round over the telephone (e.g., CDC Behavior Risk Factor Surveillance Survey) • We are likely to find that the telephone produces impediments to question functioning G Willis 2012 Cog Interviewing USUHS

  46. Adapting cognitive interviewing to Self-administration • Another interesting/unresolved area: • The cognitive demands of the S/A task are different from those of face-to-face and phone: • S/A relies on visual rather than auditory processing • S/A sometimes requires navigational activities (find the starting point, follow skip patterns…) • So, for the C.I., it may not make sense to follow the usual procedure of having the interviewer read the questions to the subject G Willis 2012 Cog Interviewing USUHS

  47. Adapting cognitive interviewing to Self-administration • So… It makes sense in the C.I. to have the subject read the form • But, do we then do the cog interview concurrently, or retrospectively? • If concurrent, should we use (a) uninterrupted think-aloud, (b) interruption with probes?, or (c) both • Issue: Which procedure produces more distortion of the question answering process, and therefore produces reactivity effects? G Willis 2012 Cog Interviewing USUHS

  48. Example: Self-administered teen tobacco questionnaire Do you think young people who smoke cigarettes have more friends? [ ] Definitely yes [ ] Probably yes [ ] Probably not [ ] Definitely not • Retrospective probe: What does “young people” mean? S: “13 year old” • Observation: S left the answer blank – objects that “It doesn’t make any difference” - but there’s no way to say that • Expert judgment: Seems classically biased to me (presents one possibility, but not others) G Willis 2012 Cog Interviewing USUHS

  49. Revised question Do you think that people your age who smoke cigarettes have more friends, fewer friends, or the same number of friends as those who don’t smoke? [ ] People who smoke have MORE friends [ ] People who smoke have FEWER friends [ ] People who smoke and those who don’t have the SAME number of friends  “Fixes” may be appropriate for one mode but not another. The current example lengthens the question and seemed to be ok for self-administration, but may be too much for the phone. G Willis 2012 Cog Interviewing USUHS

  50. Adapting C.I. to Web surveys • The Web Usability Testing area is huge, and well entrenched (Mick Couper) • It shares MANY commonalities with cognitiveinterviewing of S/A questionnaires • But there is a major emphasis on use of the system – navigation, problem-solving (e.g., how do I go back and change an answer, “skip around”) • For this reason, it bears similarities to the classic Ericsson-Simon think-aloud task • Cleo Redline (NSF), Jennifer Crafts (Westat): Think-aloud works for usability testing- we observehow the subject handles a complex, non-verbal activity G Willis 2012 Cog Interviewing USUHS

More Related