Download
1 / 18
180 likes | 198 Views
Understand cognitive testing objectives, obstacles, and protocol requirements. Explore Washington Group testing challenges and findings from pilot studies. Learn how to improve questionnaire design and data analysis through cognitive interviews and structured protocols.
E N D
Cognitive Testing: Washington Group Questions Kristen Miller, Ph.D. Question Design Research Lab National Center for Health Statistics
Cognitive Testing Objectives • Identify potential response errors related to question design • Understand why and how errors might occur • Differing interpretations • Memory difficulty • Assessment errors • Insufficient response categories
Cognitive Testing Objectives • Identify socio-cultural factors that might impact the question response • Level of education • Socio-economic status • Cultural beliefs • Racial, ethnic, gender identities • Access to knowledge and resources
Traditional Cognitive Interviews are Semi-structured • Core Question- interview is organized by the questions that are being tested • Probe Questions- open-ended, spontaneous, not pre-scripted, based on the information that the respondent provides
Traditional Cognitive Interviewing • Data collected are qualitative • Qualitative methodologies are used to analyze data • Small samples: approx. 10 – 30 participants
Washington Group Testing: Obstacles • Little or no training • No understanding of cognitive methods • Few resources (funding, technology) • Multiple languages • Multiple cultures • Multiple levels of development
Protocol Requirements • Structured interview guide • Very simple instructions • Few skip patterns • Quantitative data and analysis
Cognitive Interview Protocol Designed to illustrate: • Administration ease • Interpretations • Factors considered • Degree of consistency with physical abilities
Cognitive Interview Protocol • Demographic Section: Country, gender, age, SES • Question Testing Sections (Self-report and Proxy-report) • Core Question • Interviewer Coding • Open-ended Follow-up Probe • Cognitive Follow-up Probe • Functioning Follow-up Probe • Health Questions • Subjective health • Chronic condition list
Interviewer Code: Did the respondent… • need you to repeat any part of the question? Yes/No • have any difficulty using the response options? Yes/No • ask for clarification or qualify their answer? Yes/No
Open-ended Follow-up Probe • Why did you answer that way?
Cognitive Follow-up Probes: • How often they had difficulty • The amount of effort they needed to put into the activity • The degree of concern or worry they had regarding the difficulty
Functioning Follow-up Probes • Presence of a condition • Use of an assistive device • Degree of difficulty doing related activities
Health Status • General health question • Why did you answer that way? • Health conditions
Interviewer Debriefing • Details about the interview and interview setting • Insight into the quality of the information
Analysis Plans • Patterns in the response process • Interpretation • Judgment • Patterns of response error • Across countries and demographic variables
Design & Testing of Cognitive Protocol • Extensive testing in the United States • Input from QUEST • Pilot test in Mexico
Pilot Findings • Strengths • Relevant findings • Few resources needed • Concerns • Interviewer training (more than expected) • Protocol translations (critical and will differ by country)
