Total Survey Error in Disability Assessments

1. Total Survey Error in Disability Assessments Measuring Physical and Cognitive Capacity in the National Health and Aging Trends Study (NHATS) Brad Edwards and Tamara Bruce, Westat Presented at the International Total Survey Error Workshop Quebec, Canada June 2011

2. Overview • NHATS and capacity measures • Self report and performance • NHATS assessments • Strategies for error reduction • design, training, standardization, parsing out nonresponse • Pretest results • National experience • Future research

3. NHATS • Westat working with a team led by Johns Hopkins Bloomberg School of Public Health • Funded through cooperative agreement from NIA • CAPI panel study with 9,000 Medicare beneficiaries, annual interviews starting 2011 • New and improved measures of disability for producing data on trends and trajectories; self-reports, mental and physical assessments, eventually biomarkers, links with admin data

4. Conceptual Framework • Blend of Nagi’s model and WHO perspective • Distinguish explicitly between • Capacity to do something – basic building blocks • What people actually do in actual environment – activities • Accommodations may fill gap

5. Capacity Is Critical Measures of capacity over time are key elements in understanding individual patterns of progression to activity limitations. They allow us to track trends in function that are independent of environmental changes or accommodations, for understanding the disablement process, and as targets for interventions to prevent or slow disability.

6. Reducing Error in Capacity Measures • Questionnaire design • Importance of performance measures • Challenges for quality management • New to surveys • Complex • Expect high level of interviewer variance

7. Physical Capacity: Upper Extremity Self report Performance Grip strength • Able to • Put book on shelf/reach overhead • Open jar/grasp small object

8. Physical Capacity: Lower Extremity Self Report Performance Walking speed Balance stands Side by side Semi-tandem Full-tandem One leg eyes open One leg eyes closed Chair stands • Able to • Walk 6/3 blocks? • Kneel/bend over? • Lift and carry 20/10 pounds? • Walk up 20/10 stairs?

9. Cognitive Capacity: Memory Self report Performance 10 word recall Immediate Delayed • At present time? • Memory problems interfere with activities? • Memory compared to one year ago?

10. Other Cognitive Performance Measures • Orientation • Day of week, date, naming President and Vice President • Overall cognitive screening/executive function • Clock drawing • Attention & interference/executive function • Stroop test (computerized)

11. Standardizing Administration • Performance measures of capacity can be hard to standardize in large-scale surveys with lay interviewers • NHATS uses multiple methods to improve consistency & accuracy: • Activities Booklet design & use within CAPI framework • Standardized presentation of cognitive tasks using flash displays • Survey design & interviewer feedback using CARI coding • Formal certification process using live respondents • Follow-up web-based recertification

12. Training Approaches Broader scope of demands on NHATS interviewers for performance measures tests require using a variety of equipment navigation of unfamiliar environments potentially uncomfortable instructions to respondents Integration of video components recruitment in-person interviewer training modules such as walking course certification & re-certification processes

13. Walking Course Timing 1

14. Walking Course Timing 2

15. Walking Course Timing 3

16. Training Video Demo

17. Item Nonresponse • Performance measures used in depth set of exclusion criteria to minimize burden • High completion rate for both cognitive & performance tasks for those eligible • Comprehensive list of reasons why test was not conducted allow detailed analyses • Majority of unit nonresponse due to inability to complete easier task & safety concerns • Performance related to age, health rating, care setting, memory

18. Respondent Reactions • Avoiding “test” and “performance” • Training interviewers how to respond to requests (e.g., “How did I do?”) • Physical activities: “7th inning stretch” • Positive feedback at the end • Interest in assessing respondent satisfaction and rapport in relation to nonresponse in later rounds

19. Comparison of Performance and Self Report • High functioning performance group (attempting one-leg balance stands, meeting accuracy thresholds for Stroop) reported better self-reported health and memory and (for the Stroop) had higher performance on two memory tests

20. Comparison with Other Surveys • NLTCS screening questions in NHATS form a bridge from previous trend data to detailed NHATS self reports and assessments • Possibility of future bridge to 6 disability questions developed by NCHS and used on the American Community Survey

21. Conclusions • Reducing error in capacity measures based on performance assessment • sharpens understanding of individual trajectories and accommodations • is expected to improve predictive ability for health outcomes • NHATS protocol • standardizes administration • focuses on reasons for nonresponse

22. Conclusions (2) As a new survey, NHATS has been able to give consideration to the implications of doing these types of assessments from the outset, starting with interviewer recruitment and training. Experience from pretesting has led to refinements of these procedures (e.g., use of videos in recruitment; certification procedures in training) and to the NHATS Activities Booklet. Training materials and data collection instruments will be available later this year at www.nhats.org.

23. Future Research • Interviewer variance study • Examination of • interviewer and respondent conditioning effects • “drift” trajectories • item nonresponse as predictor of unit nonresponse • Error comparison/tradeoffs between self-reports and performance • Analysis of cognitive performance and data quality • More development • distance learning • SPC charts, integration with paradata for management • re-certification via WebEX, 2-way video

24. For more info, contact: • Brad Edwards bradedwards@westat.com • or Tamara Bruce tamarabruce@westat.com