Developing a Measure for Recovery using Item Response Theory

1. Developing a Measure for Recovery using Item Response Theory Frances M. Yang, Ph.D. Instructor in Medicine, Assistant Scientist I Harvard Medical School, Department of Medicine Beth Israel Deaconess Medical Center, Department of Internal Medicine Institute for Aging Research, Hebrew SeniorLife francesyang@hsl.harvard.edu

2. Collaborators & Acknowledgements Yih-Ing Hser David Huang Libo Li Executive Committee Members Richard N Jones, ScD Doug Tommet, MS UCLA CALDAR Emerging Investigator Pilot Award

3. Overview • Purpose • We used Item Response Theory (IRT) to find the “best” set of items to identify levels of recovery from drug addictions • Significance • Recovery is difficult to measure • Short, efficient, and sensitive recovery screening tool is needed • Innovative Approach • Clinical and theoretical input from an interdisciplinary team of experts. • Modern methodology using IRT for balancing the discrimination and severity levels of each item to measure the underlying latent trait of recovery.

4. Consensus definition of recovery that came out of the 2005 SAMHSA/CSAT • “a process of change through which an individual achieves abstinence and improved health, wellness, and quality of life.” • This broad and vague definition is an opportunity, as Hser and Anglin have pointed out, to develop and operationalize for research purposes.

5. Modern Measurement Methodology • Item Response Theory • 2-PL Model: Normal ogive assumption • Differential Item Functioning • Detecting measurement noninvariance in items of a scale attributable to certain characteristics • Translational Research (T1) • Bench to Bedside • Identifying markers, tests, or items on the basic level for clinical or research screening

7. Structure Research Algorithm for Addiction Severity Index

8. Recovery Assessment Challenges • Clinical:1 • Forms and protocols • Single step, simple responses • Screening, not diagnosing • Tools/aids to assessing attention • Collect symptoms not attributions • Research: • Develop a recovery screening tool that is: • Short • Simple • Structured • Accurate • Demonstrate validity

9. Methods: Sample • Treatment Utilization, and Effectiveness (TUE) Study • All respondents were recruited from the Los Angeles area in 1992–1994. The settings included: • sexually transmitted disease (STD) clinics, • hospital emergency rooms, • and county jails • Participants: N=865

10. Modified Delphi Method • Expert Consensus Panel of 7 Members who are experts in addictions research • Identified domains based on items from the Addiction Severity Index independently • Agreement regarding narrowing of domains for item response theory analysis • Agreement was based on majority consensus if four or more agreed.

11. Statistical Analysis Psychometric Modeling Steps • Preprocessing: • Items were recoded into indicators (based on distributions of responses and combining conditional items) • Multicollinearity checking • Items were removed if there was a lack of covariance with other items. • Dimensionality Testing: • Parallel analysis • Exploratory factor analysis • Drop the indicators below 0.4 based on NIH Patient Reported Outcomes Measurement Information Systems

12. Statistical Analysis Psychometric Modeling Steps (Continued) 3. Dimensionality Assessment: -Comparative Fit Indices > 0.95 -Root Mean Squared Error of Approximation <0.05 4. Item response theory analysis -Item parameter estimation Software: Stata v. 11 and Mplus v. 6.0

13. Flow Diagram for Item Selection

14. Expert Consensus Form

15. Final Hypothesized Domains • Domains identified by experts from the ASI items (# indicators): • Abstinence (50 indicators) • Citizenship (57 indicators) • Social Relationships (9 indicators), • Self-Sufficiency (24 items) • Physical Health (33 items) • Mental Health (15 items)

16. Sample Characteristics (N=865)

17. Results: Recovery Item Parameters

18. Item Characteristic Curves for Indicators Assigned to Abstinence Domain: Factor 1 EFA CFI: 0.973 RMSEA: 0.124 CFA CFI: 0.992 RMSEA: 0.161

19. Item Information Curves for Indicators Assigned to Abstinence Domain: Factor 1

20. Test Information Curves for Indicators Assigned to Abstinence Domain: Factor 1

21. Results: Recovery Item Parameters

22. Item Characteristic Curves for Indicators Assigned to Abstinence Domain: Factor 1 CFA CFI: 0.981 RMSEA: 0.149 EFA CFI: 0.973 RMSEA: 0.124

23. Item Information Curves for Indicators Assigned to Abstinence Domain: Factor 2

24. Test Information Curves for Indicators Assigned to Abstinence Domain: Factor 1

25. Results: Recovery Item Parameters EFA CFI: 0.996 RMSEA: 0.130 CFA CFI: 0.996 RMSEA: 0.130

26. Item Characteristic Curves for Indicators Assigned to Physical Health Domain

27. Item Information Curves for Indicators Assigned to Physical Health Domain

28. Test Information Curve for Indicators Assigned to Physical Health Domain

29. Results: Recovery Item Parameters

30. Item Characteristic Curves for Indicators Assigned to Mental Health Domain

31. Item Information Curves for Indicators Assigned to Mental Health Domain EFA CFI: 0.996 RMSEA: 0.020 CFA CFI: 0.822 RMSEA: 0.099

32. Test Information Curve for Indicators Assigned to Mental Health Domain

33. Summary • We used item response theory (IRT) analysis to identify 3 domains of recovery: Abstinence, Physical Health, and Mental Health • We chose the best fitting models for each domain • The final number of indicators was 23 indicators from 249 items.

34. Strengths and Limitations Strengths • Large and longitudinal sample • Data-rich with item level information Limitations • Assumptions of model: trait(s) may be discontinuous in population • Best fit models do not meet standard criteria used in psychometric literature. • Limited generalizability

35. Conclusion • We used item response theory (IRT) to empirically derive indicators for the future development of a standardized short recovery assessment for both clinical and research studies. • Based on IRT principles, we determined items that reflect both discrimination between and difficulty across varying levels of underlying recovery levels for individuals in the TUE study.

39. Future Work • Examine other item pools in a different data set, more recent and generalizable. • Resubmit R03 with Specific Aims to examine differential item functioning in the recovery scale due to gender differences • And include validity test with independent outcome measures. • Longitudinal IRT • Diagnostic validity relative to structured clinical psychiatric diagnoses