Developing a Context-Sensitive Approach to AT Outcomes Measurement

1. Developing a Context-Sensitive Approach to AT Outcomes Measurement OT 541 Fall 2003 James A. Lenker, MS, OTR/L, ATP

2. What are Outcomes? • Outcomes are the result of an intervention (Scherer, 1998) • Outcomes answer the question, "Does it work?" (Oldridge, 1996) • Outcomes are the impact produced by AT devices in the lives of users and their environments (Fuhrer, 2001)

3. What is Outcomes Measurement? • "The evaluation process in the service delivery systemthat is designed to measure and establish a baseline of what works; how well something works; for which clients it works; and at what level of economic efficiency it works” (DeRuyter, 1997)

4. Purposes of Outcomes Measurement • Funding justification for equipment and services (assessment & training) • Measure individual's change over time • Program evaluation: "Who do we serve? How well do we serve them?"

5. Domains of Outcomes Measurement • Many domains of inquiry can be considered an "outcome" • Medical status • Activity performance • Quality of Life (QOL) impact • Community impact • AT usage & usability

6. AT Outcomes Research – Developmentally Speaking • Blissful ignorance (Lenker, 2000) • Denial ( “ ) • Exhortative (Fuhrer, 1999) • Sober appraisal ( “ ) • Infrastructure building ( “ ) • Model demonstration ( “ ) • Getting on with it ( “ )

7. AT Outcomes Research: Challenges Heterogeneity of…. • End users: age, abilities, needs, support • AT devices • Contexts of use • Reimbursement models • Practitioner skills

8. Limitations of Previous Studies • Incomplete information about reasons for device abandonment • Inconsistent definitions for prevalence of use that have been difficult to compare across studies because use and abandonment are often defined differently • Changes in functional status when evaluating long-term device use and disuse, which bias the results toward high rates of ‘abandonment’.

9. Limitations of Previous Studies • Incompletely defined outcomes indicators (e.g. comfort, safety, and function), hindering replication by others • Incomplete information about device success • Measurement tools that were not sensitive to improvements in function • Low numbers of participants

10. Research Needs • Measurement of impact on social participation and on work (Fuhrer, 1999) • Consideration of ATD usage within context of use service delivery model, and user characteristics (Fuhrer, 1999) • Methods that embrace the user's perspective and assess impact on quality of life (Fuhrer, 1999; Jutai et al., 1996; Scherer, 1998) • Approaches that evaluate underlying factors contributing to overall success/failure of AT interventions (De Ruyter, 1994; Scherer, 1996)

11. Research Needs (con’t.) • Measurement of the hassle experienced by end users, comparing trade-offs against the functional gains (Sprigle & Abdelhamied, 1998) • Measurement for reduced dependence (Fuhrer, 1999) • Description of devices that are achieving desirable outcomes, which would provide the rationale for funding ATD devices and/or training services (Scherer, 1996).

12. The Chain of Outcomes Research (Keith, 1995) • Conceptual Models => relationships among domains • Domains => measurement tools • Measurement tools => data • Research design links treatment to outcome, i.e. interpretation of data, i.e. “outcomes” • Outcomes influence practice and help build theory

13. So, Where Do I Start? • Know the context of AT use • Identify all relevant stakeholders • Develop specific research questions • Identify indicators that will answer your questions • Identify measurement tools

14. Useful Conceptual Perspectives • Quality of Life • Product Usability • Functioning and Health • HAAT: Human, Activity, Assistive Technology • MPT: Matching Person & Technology

15. Quality of Life: The Bottom Line • Definition: “(Quality of Life is) the degree to which the person enjoys the important possibilities of his or her life” (Renwick & Brown, 1996).

16. Quality of Life (Renwick & Brown, 1996)

17. Product Usability Perspective • Effectiveness • Does it achieve the desired goal? • Efficiency • What are the costs associated with achieving the desired goal? Ex: $$, time, speed, errors effort, comfort, pain, hassle • User satisfaction International Standards Organization. (1993). ISO CD 9241-11.2

18. Smith’s Parallel Interventions Model Reduce the Impairment (Therapy) Change technique / Increase Skill Use AT Devices and Services Redesign the Activity Redesign the Environment Use Personal Assistance

19. ICF – AT Model (Lenker & Jutai, 2002)

20. CONTEXT Social Context Family Peers Strangers Setting Home School Employment Community Physical Context Space Light Sound Temperature HAAT: Human, Activity, Assistive Technology (Cook & Hussey, 2002)

21. MPT: Matching Person & Technology (Scherer, 1998)

22. Getting Down to Business • What question(s) do you want to answer? • This the most important consideration when designing your study & picking a tool • Who are the stakeholders? • Consumers, clinicians, administrators, manufacturers, policy makers • What is your conceptual orientation? • What are your resources?

23. Develop Specific Research Question(s) • Must be a match between • Stakeholder interests • Reimbursement model • User characteristics • AT device • Context of use

24. Identify Stakeholders • Who is your target audience? • End users • Practitioners / direct service providers • Program administrators • Manufacturers of AT devices • Funding / policy agencies • State VR agencies • Federal agencies that sponsor rehabilitation R&D programs • State, local departments of Education A

25. Describe Context of Use • Reimbursement Model • Educational, vocational, medical • Practitioners involved in service delivery • User involvement in service process • AT User • Physical, cognitive, learning, sensory • Ages and environments of use • Needs / goals / roles

26. Context of Use (con’t.) • Assistive Technology Devices • Wheelchair mobility, postural support, seating • Adaptive computer software and hardware • Alternative & augmentative communication aids • Job accommodations, often ‘ergonomic’ • Aids for daily living activities • Important!! AT devices are uniformly defined but they should not be uniformly studied

27. What data will answer your research question? • The domain of data may include • Functional performance, QOL impact, pain, usability, user satisfaction, etc. • The perspective of data may be • objective (by observation) • subjective (self-reported) • The nature of data may be • quantitative (data are numeric) • qualitative (data are language-based)

28. Outcomes research requires measurement tools! • The most important consideration when picking a tool? • Knowing what question (s) you want to answer

29. What other considerations matter when choosing a measurement tool? • Does it measure what you want it to measure? • Does it measure what it purports to measure? • Is it easy for participants to answer consistently? • Test-retest reliability • Is it easy for multiple observers to rate consistently? • Inter-rater reliability • What demands does it place on respondents? • What demands does it place on administrators? • Is tool-specific training required? • Is it time consuming to administer? • How difficult is it to score the results? • Accessibility of format for people with disability?

30. A Useful Model: ICF • Presents a functional continuum of human performance in life situations • Provides framework for outcomes research • Interventions • Reimbursement / service delivery models • Assessment tools • Outcomes questions

31. ICIDH-2 Domain Body Functions and Structures (Body level): Activities (Individual level): Participation (Role level): Contextual Factors (Societal level): Domain-relevant examples Functional and structural integrity vs. impairment. Normal physiologic processes and organ systems function: Cognition, sensory, perception, motor, physiologic and psychological processes Performance of action or activity (cognitive, sensory, motoric, and/or behavioral) that facilitates performance in multiple roles: - initiate, organize, sequence, judge, attend - sit, lift, stand, climb, reach, pinch, grasp, hold, release - relate, interact, cope - hear, speak, see, taste move - read, write, learn, understand Involvement in life situations and performance in roles (family, work, and educational; self-care; leisure) that take place in specific contexts (physical, social, cognitive). Features of the physical and social world that affect the individual but are not specific to the individual's ability or disability; e.g. physical barriers in public spaces, employer attitudes; funding for social service benefits; public policy; Domain-relevant Assistive Technology Heart pacemaker, hip/knee replacement, cochlear implant, baclofen pump Mobility aids: wheelchairs, crutches, canes, walkers; UE orthotics; eyeglasses, hearing aid, AAC device Job modifications of-task/device/ environment; home modification; vehicle modification; computer-based AT Accessible public venues: buildings, transportation, communications, recreation; 3rd party reimbursement for AT services & equipment Funding Model Medical Rehabilitation Vocational Rehab Educational Independent Living Employer Independent Living Public Policy ICIDH-2 / WHO Model in the Context of Reimbursement Models and Assistive Technology Interventions Adapted from: Baum, C. M., & Law, M. (1997). Occupational therapy practice: Focusing on occupational performance. American Journal of Occupational Therapy, 51, 286. Gray, D. B., Quatrano, L. A., & Lieberman, M. L. (1998). Moving to the next stage of assistive technology development. In D. B. Gray & L. A. Quatrano & M. L. Lieberman (Eds.), Designing and Using Assistive Technology: The Human Perspective (pp. 299-309). Baltimore, MD: Paul H. Brookes. Moyers, P. A. (1999). The guide to occupational therapy practice. American Journal of Occupational Therapy, 53, 247-322. World Health Organization. (1999). ICIDH-2: International classification of functioning and disability. (Beta-2 draft). Geneva: Author.

32. Example Research Questions • Do computer-based ATDs improve the psychological and emotional well-being of end-users? • Do computer-based ATDs increase end-user participation in work, school, and/or leisure activities? • Do users with clearly defined goals become more successful user of computer-based ATDs? • Does impact of ATD use vary across disability populations?  

33. Research Questions (con’t.) • Are there societal barriers that preclude or interfere with participation in school, work, or leisure activities? • Are there significant technical problems with computer-based ATDs that are limiting performance for users of these products? • Are there any themes or variables that are associated with successful or unsuccessful computer-based ATD use?

34. Indicators of AT “Outcome” • Psychosocial impact on the individual • User satisfaction with device • User satisfaction with services • Achievement of user goals • Prevalence of long-term AT use • Tasks and roles in which AT is used • Environmental influences that contribute to functional AT use/abandonment • Unmet technology needs • Employment / educational success

35. AT Outcome Measurement Tools • PIADS: Psychosocial Impact of Assistive Device Scale • QUEST: Quebec User Evaluation of Satisfaction with Assistive Technology • COPM: Canadian Occupational Performance Measure • MPT: Matching Person & Technology

36. PIADS • Attribute: Quality of Life • 3 sub-scales: Competence, adaptability, self-esteem • 26 item, 7-point, self-report • Strengths • Not device-specific • Identifies potentially useful points of departure for exploration • Quick, easy to administer • Useful retrospectively or pre/post

37. PIADS • Additional considerations • Items are abstract, some users will struggle -- especially kids/adolescents • Better to administer in person • Open-ended probing questions can provide insight about specific scores

38. COPM • Semi-structured interview format, approx. 30-40 minutes • Process • Identify problem areas • Prioritize 5 areas of greatest importance • User rates each of the 5 areas on • Current performance level • Satisfaction with that performance level

39. COPM • Strengths • User centered • Philosophically sound • Effective as intervention approach; increased consumer ‘stake’ in the outcome (?) • Sensitive to clinically significant changes in performance and satisfaction with performance • Useful for variety of devices and contexts of use • Outcome indicators are built-in to the assessment process

40. COPM • Additional considerations • Goals are likely to be interviewer-dependent, which will affect inter-rater reliability • The specificity of goals to the end user's circumstances hinders generalizability of results to other users • Goals may not suit all stakeholder perspectives

41. QUEST • 12 items, 5-point scale • Device satisfaction • Services satisfaction • Quick, straightforward to administer (10-15 mins.) • Suitable for phone or distance surveys • Suitable for wide range of AT products and user demographics

42. QUEST • Additional considerations • Expectations often color the perceptions of satisfaction, therefore expectations are a potential confounding factor • Satisfaction judgments are best supplemented with information that explains underlying predictors • User performance, or function, with use of AT device is not captured

43. MPT • Several instruments • Survey of Technology Use • ATD Predisposition Assessment (PA) • Educational Technology PA • Workplace Technology PA • Health Care Technology PA

44. ATD Predisposition Assessment • Two perspectives are captured • Professional assesses ... • Match between user abilities and ATD requirements • Psychosocial incentives/disincentives to AT use • End user self-rates ... • Sensory & motor capabilities • Life satisfaction • Perceived value of device

45. MPT • Additional considerations • AT ‘benefits’ to user are not explicitly captured • Professional’s ratings of psychosocial incentives and disincentives to AT use require knowing the consumer really well -- beyond the scope of a typical AT evaluation period • Professional’s ratings of psychosocial factors is subjective, subject to bias • Overall, does a fine job of identifying factors that could predispose user to frequent or infrequent use

46. Conclusion • Acknowledge the futility of identifying a single “best” AT outcomes tool that is appropriate for all contexts and research questions. • Accept the challenge of matching research methods and measurement tools to your research area