Barth B. Riley, Michael L. Dennis Chestnut Health Systems

1. Distinguishing between Treatment Effects and DIF in a Substance Abuse Outcome Measures Using Multiple Indicator Multiple Causes (MIMIC) Models Barth B. Riley, Michael L. Dennis Chestnut Health Systems Study supported by National Institute on Drug Abuse Grant (NIDA) No. R37 DA11323.

2. Overview • Differential Item Functioning and Its Impact • The Multiple Indicator Multiple Cause (MIMIC) Model • Demographic differences in substance use and substance abuse treatment • Present Study

3. Differential Item Functioning • DIF: Two groups differ in their likelihood of endorsing an item after controlling for differences on the measured construct. • Group differences in the likelihood of endorsing an item may be due to: • Group differences on the latent trait • Differential item functioning (DIF) • Both • DIF can also occur over time

4. Differential Item and Test Functioning • The presence of DIF items can reduce the validity of a measure in making between group comparisons. • If DIF is of sufficient magnitude to cause measurement bias against one group relative to another, efforts to interpret outcomes measures becomes complex.

5. Did the persons change or did the items in the instrument change?

6. Analysis of DIF • Several approaches have been employed for the analysis of DIF: • T tests comparing item parameters between two groups • Mantel-Haenszel contingency tables • Logistic regression • IRT Likelihood ratio tests • Most of these approaches are limited to comparisons of two groups on a single factor. • Do not directly assess impact of DIF on person measures.

7. Multiple Indicator Multiple Causes (MIMIC) Models • Combines aspects of confirmatory factor analysis and structural equation modeling. • The basic MIMIC models consist of the following components: • A latent variable—the construct being measured. • A set of measured indicators—items • Grouping variables such as race and gender

8. Basic IRT Model Item 1 Item 2 Latent Construct Item 3 … … Item n Indicators

9. MIMIC Model, No DIF Assumed Item 1 Indirect effects Item 2 Ethnicity Latent Construct Item 3 … … Gender Latent Variable Item n Indicators

10. MIMIC Model with DIF Effects Direct DIF effect Item 1 Item 2 Ethnicity Latent Construct Item 3 … … Gender Latent Variable Effect of DIF is partialed out of the indirect effects Item n Indicators

11. Study • The purpose of this study was to examine the effect of DIF by time, gender race on the Global Appraisal of Individual Needs (GAIN) Substance Problem Scale • Data were collected from 446 participants as part of a three-year substance abuse early re-intervention study.

12. Participants (N=446) • Recruited from community-based substance abuse treatment in Chicago in 2004. • Participants were randomly assigned to either outcome monitoring or recovery management checkups, designed to help relapsing participants to return to treatment. • Followed quarterly for 3 years. • Participants were predominantly • Male (54.5%) • African American (80.2%) • Average age: 38.4 years (SD=8.3)

13. Primary Drug

14. Substance Problem Scale • The Substance Problem Scale (SPS) measures problems with alcohol/drug use during the past month, including abuse, dependence and substance-abuse health problems. • Consists of 16 dichotomous items • Based on DSM-IV-TR criteria for substance abuse and substance dependence. • Internal consistency: .9 • Test-retest reliability: .73

15. Model • In order to assess treatment effects over time, a multilevel framework was used: • Level 1: Time: random effect • Level 2: Person: fixed effects • Treatment variables: • Random assignment to recovery management • Days in outpatient, intensive outpatient and residential treatment • DIF factors: gender and ethnicity • One and two parameter IRT models were compared.

16. MIMIC Model: Within Level SPS 1 Time SPS SPS 2 SPS 3 Tx Participation … … SPS n Control for DIF

17. MIMIC Model: Between Level Race SPS 1 SPS SPS 2 Gender SPS 3 Opiates … … RMC SPS n Control for DIF

18. Goodness of Fit N Cases = 400, N Observations = 5393

19. Time DIF

20. Time DIF

21. Gender DIF

22. Ethnicity DIF

23. Primary Drug DIF

24. Group Differences on DIF Factors

25. Treatment Effects

26. Conclusions • The MIMIC model is a promising tool for assessing the presence and impact of DIF on at the scale level (DTF). • Controlling for DIF reduced differences in SPS measures as a function of gender and primary drug. • Treatment effects as measured by the SPS were not affected by gender, ethnicity, primary drug or time DIF.

27. MIMIC: Strengths • Assess DIF and DTF on multiple factors • DIF factors can be discrete or continuous variables • Distinguish between treatment and DIF effects • Can be used in conjunction with longitudinal analysis methods (e.g., multilevel modeling).

28. MIMIC: Limitations/Caveats • In order to specify the model, at least one item must be free of DIF (or have minimal DIF). • Can not detect non-uniform DIF—DIF in the discrimination parameter • Obtaining group specific item parameters is not straightforward • Assumes consistent factor structure across groups

29. Useful References • Muthén, B. (1989). Latent variable modeling in heterogeneous populations. Psychometrika, 54, 557-585. • Fleishman, L.A., & Lawrence, W.F. (2003). Demographic variation in SF-12 scores: True differences or differential item functioning? Medical Care, 41(7 Suppl.) III75-III86. • MacIntosh, R. & Hashim, S. (2003). Converting MIMIC model parameters to IRT parameters in DIF analysis. Applied Psychological Measurement, 27, 372-379. • Finch, H. (2005). The MIMIC Model as a method for detecting DIF: Comparison with Mantel-Haenszel, SIBTEST, and the IRT likelihood ratio. Applied Psychological Measurement, 29(4):278-295.

30. Contact Information • A copy of this presentation will be at: www.chestnut.org/li/posters • For information on this method and a paper on it, please contact Barth Riley at bbriley@uic.edu.