190 likes | 202 Views
This article discusses the challenges in measuring disability, specifically focusing on cognitive testing and different modes of data collection. It explores the impact of cognitive stages and response errors on data quality, as well as socio-cultural factors and translation issues. Additionally, it highlights the use of cognitive interviews to gain insight into respondents' understanding of questions and the importance of considering different modes of data collection based on accessibility and potential biases.
E N D
Issues Related to Disability Measurement: Cognitive testing and mode Jennifer Madans for the Washington Group on Disability Statistics SPECA Regional Wrokshop on Disability Statistics, Dec 13-15, 2006
Part 1: Cognitive testing Adapted from work of Kristen Miller, NCHS, USA
Cognitive testing • Designed to help us understand how respondents comprehend, retrieve, judge, respond to questions • Illustrate how aspects of respondents’ lives impact their understanding of the question • Through this examination, can identify • Potential response errors • Patterns of interpretation
A good question is: Relevant to the research agenda Relevant to each potential respondents’ experience and knowledge Respondents: Are informants Reference aspects of their lives Do not know or understand the research question Do not use scientific or abstract concepts Components of the process
Response errors • When questions are not optimal or the respondent does not understand, error is introduced into the data • False positives • False Negatives • An entirely different phenomena is measured than intended by the research agenda
Cognitive stages: responding to questions • Comprehension– the respondent interprets the question • Retrieval– the respondent searches memory for relevant information • Judgment– the respondent assesses the completeness and relevance of memories and makes an estimation • Response: Maps judgment onto response category; may need to edit response to fit the category
Cognitive response problems • Respondent cannot remember the correct answer • “In the past month, how many times did you eat rice?” • Respondent makes an error when forming an estimate • “What is the total value of all your financial assets?” • Respondent’s response does not match response categories • “When you go to work, do you drive, bicycle or walk?”
Socio-cultural factors impact question response • Level of education • Socio-economic status • Cultural beliefs • Racial, ethnic, gender identities • Access to knowledge and resources
Translation Problems • Bad literal translations • Words vary by region • Formal words may not be understood • Multiple meanings of words
Cognitive Interviews • Semi-structured • Core questions • the questions that will be tested • interview organized by the core questions • Probe questions • the questions that will provide information about respondents’ understanding of the core questions • Probe questions are open-ended, spontaneous, not pre-scripted, based on information that respondent provides
Cognitive Interviews • Data are qualitative • Qualitative methodologies used to analyze data • Small samples of roughly 10 – 30 participants are used
Probing for story: Why did the respondent answer the question the way that they did? Does this story match with the intent of the question? Common probes: Why did you answer that way? Can you tell me about that? When the question asks about “difficulty walking,” how are you understanding that? Probes
Most Common Probes • How so? • In what way? • Can you say more? • What do you mean by that? • [Silence]
Issues related to disability • Conceptual confusion around disability • Cognitive testing allows you to gain insight into what respondent is thinking and whether this varies by socio-demographic characteristics such as education, age, and gender
Part 2: Mode of data collection • Mail (paper and pencil) • Telephone • In-person • Web-based
Issues related to disability • Mail (paper and pencil) • Literacy level • Insure availability of Braille, large print, or assistant if needed • Cannot clarify questions if respondent unclear • Telephone • Presence of phone in the household may bias who responds • TTY/TDD or assistant for persons with hearing impairment • Can clarify questions if needed
Issues related to disability • In-person • May require assistant if respondent has communication or cognitive difficulties (allows for including respondents who would usually be excluded) • Sign language interpreter or alternate accommodations for persons with hearing impairments • Can offer additional explanation of question if needed • Web-based • Presence of computer in the household may bias who responds • Bias in educational level of respondents who are able to respond to web-based survey
Issues related to disability Levels of interview assistance • Direct personal interview: Respondent participates directly; no interpretation or facilitation is needed • Interpreted interview: Someone interprets the questions to the respondent and interprets the responses back to the interviewer; interpreter is an intermediary • Facilitated interview: Someone else helps the respondent understand the questions and even answers some of the questions on their behalf • Proxy interview: Someone else completes the questionnaire for the respondent because he / she is unable to do so
Refusals and non-responses • Disabled people may not be seen as valid respondents by family members or ignored and therefore not enumerated in the household • Not acceptable to have non-response on basis of disability in disability surveys - losing out on target population • Training of interviewers and provision of support is crucial (e.g. sign language interpreters, assistants, Braille versions, etc.)