SSA-NIH-BU Collaboration to Improve the Disability Determination Process:

1. SSA-NIH-BU Collaboration to Improve the Disability Determination Process: Feasibility of Claimant- and Provider-Reported Computer Adaptive Tests (CAT) Stephen M. Haley, Ph.D. Alan M. Jette, Ph.D Elizabeth E. Barfield, MPH

2. The BU HDR Team Beth Barfield, Pre-Doctoral Research Fellow Ron Hambleton, IRT consultant Bill Frey, Director of Westat, Inc. • Steve Haley , PI • Alan Jette, Co-PI • Kara Bogusz, Project Director • Mary Slavin, Training • Pengsheng Ni, IRT Analyst and CAT programmer

3. Overview of the Presentation Functional Domains Formative Work- Item Pools Calibration Study Computer Adaptive Testing (CAT) Interpretation of Individual Scores Pilot study

4. Point of Clarification This exploratory work, being conducted by NIH and BU, examines ways in which claimants and their health care providers can quickly and easily provide information about claimants’ function. SSA has not yet adopted or endorsed this approach.

5. 1. Functional Domains • Physical Demands • Interpersonal Interactions

6. Physical Demands Physical Functioning Self-Care Tasks Potential Ability to Work Interpersonal Interactions Learning and Applying Knowledge Cognitive & Mental Functioning Communication General Tasks and Demands

7. Physical Demands Domain Framework Changing & Maintaining Body Position Physical Demands Whole Body Mobility Carrying, Moving, & Handling Objects

8. Kneeling Standing Changing Basic Body Position Whole Body Transfer Lying Down Shifting Center of Gravity Bending Changing & Maintaining Body Position Sitting Squatting Lying Sitting Maintaining Body Position Squatting Kneeling Standing

9. Changing & Maintaining Body Position Physical Demands Whole Body Mobility Carrying, Moving, & Handling Objects

10. Walking Short Distances Walking Walking Long Distances Walking on Different Surfaces Stepping Around/Over Obstacles Whole Body Mobility Climbing Crawling Moving Around Running Device Aided Mobility

11. Changing & Maintaining a Body Position Physical Demands Whole Body Mobility Carrying, Moving, & Handling Objects

12. Unilateral Lift Bilateral Lift Lifting & Carrying Objects Unilateral Carry Bilateral Carry Picking Up Carrying, Moving, & Handling Objects Gripping Hand & Arm Use Holding Turning Pinching Manipulating Objects Pulling Pushing Reaching

13. Changing & Maintaining Body Position Physical Demands Whole Body Mobility Carrying, Moving, & Handling Objects

14. Kneeling ● Sitting Squatting ● Bending Whole Body Transfer ● Lying Down ● Shifting Center of Gravity Changing Basic Body Position Changing & Maintaining Body Position Maintaining Body Position Lying ● Sitting ● Kneeling● Standing ● Squatting Short Distances ● Long Distances ● Different Surfaces ● Stepping Around/Over Obstacles Walking PhysicalDemands Whole Body Mobility Moving Around Climbing ● Crawling Running ● Device Aided Mobility Lifting & Carrying Objects Unilateral Lift ● Bilateral Lift ● Unilateral Carry ● Bilateral Carry Carrying, Moving, & Handling Objects Manipulating Objects Picking up ● Gripping ● Holding● Turning● Pinching Hand & Arm Use Pulling ● Pushing ● Reaching

15. Interpersonal Interactions Domain Framework Temperament & Personality Behavior Modulation Interpersonal Interactions Adaptability Basic Interactions Workplace Behaviors

17. Temperament & Personality Behavior Modulation Interpersonal Interactions Adaptability Basic Interactions Workplace Behaviors

19. Temperament & Personality Behavior Modulation Interpersonal Interactions Adaptability Basic Interactions Workplace Behaviors

21. Temperament & Personality Behavior Modulation Interpersonal Interactions Adaptability Basic Interactions Workplace Behaviors

23. Temperament & Personality Behavior Modulation Interpersonal Interactions Adaptability Basic Interactions Workplace Behaviors

25. Temperament & Personality Behavior Modulation Interpersonal Interactions Adaptability Basic Interactions Workplace Behaviors

26. Extraversion ● Openness to Experience ● Agreeableness ● Conscientiousness ● Psychic Stability ● Optimism ● Confidence ● Trustworthiness ● Trust Others ● Energy & Drive Temperament & Personality Behavior Modulation Aggression Regulation● Impulse Control ● Control of Internal Distracters ● Interpersonal Disruption Interpersonal Interactions Adaptability Response to Change ● Coping With Stress Respect & Warmth ● Tolerance● Ethical Behavior ● Criticism ● Cooperation ● Accept Direction● Appreciation● Social Cues ● Family & Community Relations Basic Interactions Workplace Behaviors Workplace Relations ● Conscientiousness ● Coping with Stress ● Confidence ● Physical Contact

27. 1. Functional Domains Questions??

28. 2. Formative Work

29. The Process of Building CATs Items from Instruments Items from Claimant Groups Items from Provider Groups New Items Initial Item Pool Evaluated by expert and claimant review, cognitive testing        SSA CAT Item Pool administered to target Sample s    Item Response Theory (IRT) Final Item Bank (Calibrated items eval. for Breadth, reliability, validity, and precision) CAT Short Form Instruments

30. 2. Formative Work • Extensive literature review • Focus groups (interviews) • 6 providers • Six content experts • Cognitive Interviews – 7 claimants

31. Cognitive Interviews After establishing critical item content, time frame and response scales the item pools were prepared for cognitive interviews with both claimants and health care providers. Cognitive Interviewing is a technique used to try to get at sources of response error in questionnaires. Cognitive interviewing asks interviewees about question comprehension, decision processes and response processes .

32. Cognitive Interviews This process involves asking the participant a question and then asking a series of probes. Participants were probed with questions like: • ‘In your own words, what do you think this question is asking? • ‘How confident are you in your answer to this question?’ • ‘Can you think of a better way to ask this question?’ • ‘How did you arrive at that answer?’ • ‘Was it easy or hard to answer that question?’

33. Results of Cognitive Interviews Physical Demands Of the 129 items administered in the physical demands group, 10 items were completely re-written based on claimant feedback and 8 were removed due to comprehension difficulty. Interpersonal Interactions Of the 136 items administered in the interpersonal interactions item bank, 7 items were completely re-written based on claimant feedback and 1 item was deleted due to comprehension difficulty.

34. Sample Physical Demands Items • Are you able to lift a 20 lb. object (a toolbox) from table height to a high shelf? • Responses: without difficulty, with a little difficulty, with some difficulty, with a lot of difficulty, unable to do, I don’t know • How far are you able to walk without stopping?   Responses: More than an hour, An hour, 30 minutes, 15 minutes, Unable to do, I don't know • How quickly are you able to walk? Responses: Faster than those around me, At a normal pace compared to those around me, At a slower pace than those around me, Unable to do, I don't know • Are you able to work overhead for 20 minutes (e.g. organizing a high shelf in a closet)?  Responses: without difficulty, with a little difficulty, with some difficulty, with a lot of difficulty, unable to do, I don’t know

35. Sample Interpersonal Interactions Items I get in conflicts with others I have difficulty calming down I can’t stop myself from doing the same thing over and over again [Strongly agree, agree, disagree, Strongly disagree] • I feel good about myself. • I am so tired when I wake up, it's hard to get going. • I can get back on track when I am distracted.

36. OIDAP RecommendationsPhysical Demands Content Areas Addressed Repetitive items Varying force requirements Duration of typical day Balance Reaching levels Unilateral/Bilateral

37. OIDAP RecommendationsInterpersonal Interactions & Emotional Behaviors Content Areas Addressed Content Areas not Addressed Neuro-cognitive • Interpersonal Functioning • Friction, cooperation, working with others, criticism acceptance and response to, expressing oneself • Initiative & Persistence • Task completion, persistence of task performance, leaving one’s home • Information Process & Decision Making • Ability to make a decision • Self Management & Self Monitoring • Disturbing behaviors, social appropriateness, controlling symptoms, managing mood, stress tolerance, expression and controlling emotion

38. Interpersonal Interactions: 361 initial items: 31 other instruments 57 PROMIS/NeuroQOL 273 new ______________________ 155 final items to calibrate 90 provider items Physical Demands: 174 initial items 22 other instruments 31 PROMIS/NeuroQOL 75 new items ______________________ 124 final items to calibrate 97 provider items Item Pools for each Domain:

39. 2. Formative Work Questions???

40. 3. Calibration Study

41. Goals of the Calibration Study Develop a sample of claimants & providers with geographic diversity of SSA regions, offices, and work disability claimants. Target samples for each content domain: 1000 SSA claimants. 500 primary providers 250–500 supplemental providers Create calibrated item banks for each domain Create a CAT instrument for each domain Analyze claimant – provider dyads

42. Plans for Calibration Study • Work closely with SSA to develop sampling procedures • Work with Westat, an outstanding survey research firm with SSA experience • Administer item pools by web and telephone interview • Implement analyses to build item banks 42

43. Data Collection Strategy • Westat to contact each claimant to enroll and secure consent • Calibration survey (Item pools) administered by web or telephone interview • Provider samples identified by claimants • Westat to contact providers by telephone

44. 3. Calibration Study Questions???

45. 4. CAT

46. Building a CAT Instrument .. Score estimation – show it in terms of probability estimates Item selection criteria- maximum item information criteria Stop rule- static or dynamic stopping rules CAT administers a small sample of Items selected from a large IRT calibrated ‘item bank’ The items that are administered are chosen based on how a person responds to previous items CAT reduces number of assessment items needed for an accurate assessment

47. BU example of how CAT can express claimants’ (and providers’) own assessment of their functioning WORSE BETTER

48. PhysDemands A PhysDemands B PhysDemands C BETTER WORSE BU example of how CAT can express claimants’ (and providers’) own assessment of their functioning

49. PhysDemands A PhysDemands B PhysDemands C WORSE BETTER BU example of how CAT can express claimants’ (and providers’) own assessment of their functioning

50. PhysDemands D PhysDemands E PhysDemands F BETTER WORSE BU example of how CAT can express claimants’ (and providers’) own assessment of their functioning