Issues Related to Disability Measurement: Cognitive testing and mode

1. Issues Related to Disability Measurement: Cognitive testing and mode Jennifer Madans for the Washington Group on Disability Statistics SPECA Regional Wrokshop on Disability Statistics, Dec 13-15, 2006

2. Part 1: Cognitive testing Adapted from work of Kristen Miller, NCHS, USA

3. Cognitive testing • Designed to help us understand how respondents comprehend, retrieve, judge, respond to questions • Illustrate how aspects of respondents’ lives impact their understanding of the question • Through this examination, can identify • Potential response errors • Patterns of interpretation

4. A good question is: Relevant to the research agenda Relevant to each potential respondents’ experience and knowledge Respondents: Are informants Reference aspects of their lives Do not know or understand the research question Do not use scientific or abstract concepts Components of the process

5. Response errors • When questions are not optimal or the respondent does not understand, error is introduced into the data • False positives • False Negatives • An entirely different phenomena is measured than intended by the research agenda

6. Cognitive stages: responding to questions • Comprehension– the respondent interprets the question • Retrieval– the respondent searches memory for relevant information • Judgment– the respondent assesses the completeness and relevance of memories and makes an estimation • Response: Maps judgment onto response category; may need to edit response to fit the category

7. Cognitive response problems • Respondent cannot remember the correct answer • “In the past month, how many times did you eat rice?” • Respondent makes an error when forming an estimate • “What is the total value of all your financial assets?” • Respondent’s response does not match response categories • “When you go to work, do you drive, bicycle or walk?”

8. Socio-cultural factors impact question response • Level of education • Socio-economic status • Cultural beliefs • Racial, ethnic, gender identities • Access to knowledge and resources

9. Translation Problems • Bad literal translations • Words vary by region • Formal words may not be understood • Multiple meanings of words

10. Cognitive Interviews • Semi-structured • Core questions • the questions that will be tested • interview organized by the core questions • Probe questions • the questions that will provide information about respondents’ understanding of the core questions • Probe questions are open-ended, spontaneous, not pre-scripted, based on information that respondent provides

11. Cognitive Interviews • Data are qualitative • Qualitative methodologies used to analyze data • Small samples of roughly 10 – 30 participants are used

12. Probing for story: Why did the respondent answer the question the way that they did? Does this story match with the intent of the question? Common probes: Why did you answer that way? Can you tell me about that? When the question asks about “difficulty walking,” how are you understanding that? Probes

13. Most Common Probes • How so? • In what way? • Can you say more? • What do you mean by that? • [Silence]

14. Issues related to disability • Conceptual confusion around disability • Cognitive testing allows you to gain insight into what respondent is thinking and whether this varies by socio-demographic characteristics such as education, age, and gender

15. Part 2: Mode of data collection • Mail (paper and pencil) • Telephone • In-person • Web-based

16. Issues related to disability • Mail (paper and pencil) • Literacy level • Insure availability of Braille, large print, or assistant if needed • Cannot clarify questions if respondent unclear • Telephone • Presence of phone in the household may bias who responds • TTY/TDD or assistant for persons with hearing impairment • Can clarify questions if needed

17. Issues related to disability • In-person • May require assistant if respondent has communication or cognitive difficulties (allows for including respondents who would usually be excluded) • Sign language interpreter or alternate accommodations for persons with hearing impairments • Can offer additional explanation of question if needed • Web-based • Presence of computer in the household may bias who responds • Bias in educational level of respondents who are able to respond to web-based survey

18. Issues related to disability Levels of interview assistance • Direct personal interview: Respondent participates directly; no interpretation or facilitation is needed • Interpreted interview: Someone interprets the questions to the respondent and interprets the responses back to the interviewer; interpreter is an intermediary • Facilitated interview: Someone else helps the respondent understand the questions and even answers some of the questions on their behalf • Proxy interview: Someone else completes the questionnaire for the respondent because he / she is unable to do so

19. Refusals and non-responses • Disabled people may not be seen as valid respondents by family members or ignored and therefore not enumerated in the household • Not acceptable to have non-response on basis of disability in disability surveys - losing out on target population • Training of interviewers and provision of support is crucial (e.g. sign language interpreters, assistants, Braille versions, etc.)