1 / 23

Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures

Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures. Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut Health Systems, Bloomington, IL and Richard Lennox, Psychometric Technologies, Hillsborough, NC

amma
Download Presentation

Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Development and Evaluation of the Global Appraisal of Individual Needs (GAIN) Validity Measures Rodney Funk, Michael L. Dennis, Melissa Ives, Chestnut Health Systems, Bloomington, IL and Richard Lennox, Psychometric Technologies, Hillsborough, NC Workshop at the Joint Meeting on Adolescent Treatment Effectiveness (JMATE), Baltimore, MD, March 28, 2006.

  2. Acknowledgement The content of this presentations are based on treatment & research funded by the Center for Substance Abuse Treatment (CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA) under contract 270-2003-00006 using data provided by the CYT grantees (Nos. TI11317, TI11320, TI11321, TI11323, and TI11324). The authors thank Sarah Knecht, Michelle White and the GAIN QA team for helping to identify common inconsistencies, Sandra McGuinness for the code for the inconsistencies and Barth Riley of UI-C for Rasch code. The opinions are those of the author and do not reflect official positions of the consortium or government. Available on line at www.chestnut.org/LI/Posters or by contacting Joan Unsicker at 720 West Chestnut, Bloomington, IL 61701, phone: (309) 827-6026, fax: (309) 829-4661, e-Mail: junsicker@Chestnut.Org

  3. CYT Cannabis Youth Treatment Randomized Field Trial Coordinating Center: Chestnut Health Systems, Bloomington, IL, and Chicago, IL University of Miami, Miami, FL University of Conn. Health Center, Farmington, CT Sites: Univ. of Conn. Health Center, Farmington, CT Operation PAR, St. Petersburg, FL Chestnut Health Systems, Madison County, IL Children’s Hosp. of Philadelphia, Phil. ,PA Sponsored by: Center for Substance Abuse Treatment (CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA), U.S. Department of Health and Human Services

  4. Design • Target Population: Adolescents with marijuana disorders who are appropriate for 1 to 3 months of outpatient treatment. • Inclusion Criteria: 12 to 18 year olds with symptoms of cannabis abuse or dependence, past 90 day use, and meeting ASAM criteria for outpatient treatment • Data Sources: self report on the GAIN (Dennis et al 2003), collateral reports, on-site and laboratory urine testing, therapist alliance and discharge reports, staff service logs, and cost analysis. • Random Assignment: to one of three treatments within site in two research arms and quarterly follow-up interview for 12 months, and long term follow up 30 to 42 months later. Source: Dennis et al 2002, 2004

  5. 85% 78% 80% . 71% 60% % of Admissions 47% 46% 40% 26% 26% 26% 20% 0% First used Dependence Weekly or Prior under age more use at Treatment Adolescent Cannabis Users in CYT were as or More Severe Than Those in TEDS* 100% 15 intake CYT Outpatient(n=600) TEDS Outpatient (n=16,480) * Adolescents with marijuana problems admitted to outpatient treatment Source: Tims et al, 2002

  6. 100% 83% 80% 62% 55% 60% 50% 40% 30% 17% 15% 20% 0% Female Male African Caucasian Under 15 15 to 16 Single American parent family Demographic Characteristics Source: Tims et al, 2002

  7. Institutional Involvement 100% 87% 80% 62% 60% 47% 40% 25% 20% 0% In school Employed Current CJ Coming from Involvement Controlled Environment Source: Tims et al, 2002

  8. Patterns of Substance Use 100% 73% 80% 71% 60% 40% 17% 20% 9% 0% Weekly Alcohol Weekly Weekly Significant Time Tobacco Use Cannabis Use Use in Controlled Environment Source: Tims et al, 2002

  9. Multiple Problems are the NORM Self-Reported in Past Year Source: Dennis et al, 2004

  10. Co-occurring Problems are Higher for those Self-Reporting Past Year Dependence Source: Tims et al., 2002 * p<.05

  11. Validity Measures • Number of Inconsistencies –Count of 49 paired items consistently answered by over 90% of the clients, but that are inconsistent • Denial/Misrepresentation – Sum of staff rating over 8 sections on a scale of 0-no problem, 1-estimating, 2-misunderstanding, 3-denial, 4-misrepresentation • Context Effect – staff report of problems that might effect the interview (e.g.., someone present, interruptions, in juvenile justice setting) • Proportion of Missing Data on 86 Items used in the GAIN’s core 10 Change measures: Substance Frequency Scale, Current Withdrawal Scale, Substance Problem Scale, Health Problem Scale, Emotional Problem Scale, Recovery Environment Risk Index, Social Risk Index, Illegal Activity Scale, Training Activity Scale and the Employment Activity Scale

  12. Residual Atypicalness Randomness Validity Measures (Continued) This last two measures are based on the residual (actual vs expected answers) of On the 123 items of the GAIN’s 4 main psychopathology and psychopathy scales. They are based the outfit and infit statistics under the Rasch (1960) measurement model and are reported in logits • Atypicalness a measure of endorsing high severity items without first endorsing the typical prior items (e.g.., suicide without depression) • Randomness a measure of answers that are more random than expected on the GAIN’s 4 main psychopathology and psychopathy scales Where yni is the observed response of person n to item i and pni is the probability of a correct response for person n on item i.

  13. While there is some overlap, for the most part these measures capture different aspects of validity Correlation of Validity Measures Denial/Misrepr. Inconsistencies %Missing Data Context Effect Atypicalness Randomness -- -- -- -- -- -- Inconsistencies -- 0.07 -- -- -- -- Denial/Misrepr. -- 0.31 0.10 -- -- -- Context Effect -- -0.06 0.03 0.05 -- -- %Missing Data -- 0.20 0.03 0.05 -0.02 -- Atypicalness Randomness 0.24 0.14 0.57 0.00 -0.04 -- Bold indicates p < .05.

  14. Trichotomization of Validity Measures • Like all GAIN scales we trichotomized the validity measures into low/mod/high range to help line staff interpret them. • Because they were close to normally distributed, we divided Inconsistencies, Atypicalness and Randomness into: Low 0-50%, Mod 51-90% and High 91-100% • Because they were sharply right skewed, Missing data, Denial/misrepresentation and Context effects were divided into Low 0%, Mod 1-90%, and High 91-100%

  15. Overview of Validation Test Results False Negatives Relative To Urine at Intake Bias 3 Month outcomes Internal Consistency at Intake Construct Validity at Intake Test-Retest at Intake Validity Measures X X X Denial/Misrep Rating X X X Missing Data X X X X Context X X X X Inconsistencies X X X X Randomness (aka infit) X X X Atypicalness (aka outfit) X: Continuous or trichotomous version of validity measure is a statistically significant (p<.05) predictor of worse values on the criterion in this column

  16. Inconsistencies are the best predictor of low alpha Internal Consistency Results 1.00 0.90 0.80 0.70 0.60 Average Cronbach’s Alpha* 0.50 0.40 0.30 0.20 0.10 0.00 Denial/Misrep Missing Data Context Inconsistecies Randomness Atypicalness Average Rating 0.87 0.87 0.87 0.87 0.87 0.87 0.87 Low 0.88 0.88 0.86 0.88 0.86 0.87 0.87 Mod 0.89 0.84 0.80 0.83 0.82 0.84 High *average alpha across Substance Problem Scale, Internal Mental Distress Scale, Behavior Complexity Scale & Crime/Violence Scale

  17. Randomness is the best predictor of low test-retest Rho Test-Retest Results

  18. Atypicalness predicts higher than expected values Randomness on symptoms generally predicts lower than expected values Correlations with Intake Variables Days of Illegal Activity for $ Days of Arguing/Violence Days of AOD Problems Days of Illegal Activity Past Month Problems Days of Any AOD use Days of Alcohol Use Days of Marj.Use Spearman Rho -0.10 -0.13 Denial/Misrep -0.06 -0.07 -0.04 -0.02 -0.04 0.00 Missing Data 0.00 0.02 -0.01 -0.02 0.03 -0.03 0.05 0.07 -0.09 Context 0.04 -0.01 0.06 -0.02 -0.04 0.00 0.00 -0.14 -0.09 Inconsistencies -0.01 -0.01 -0.04 0.02 -0.02 -0.05 -0.17 -0.12 -0.14 -0.25 -0.23 0.09 Randomness -0.05 0.01 0.11 0.17 0.20 Atypicalness 0.02 0.02 0.03 -0.03 0.02 Bold indicates p < .05.

  19. Staff Ratings of Denial/Misrepresentation Predict False Negatives Relative to Urine Screens But not all Denial/ Misrepresentation is about drug use Only significant predictor of False Negatives, OR=4.0, 95% CI (2.03, 7.96)

  20. Predicting 3-Month Outcomes • Predicted 3-month variables with intake only and in second step, added the validity measures • Dependent Variables: Substance Frequency, Substance Problems, Emotional Problems, Recovery Environment, Social Risk and Illegal Activity • Context and Inconsistencies had a small significant positive relationship with Recovery Environment Risk at 3 months • Randomness had a small significant positive relationship with Substance Frequency and Illegal Activity

  21. Limitations • We had very little missing data, denial/ misrepresentation, and false negatives in the data from this multi-site clinical trial • This data set was for outpatient and limited in severity and diversity • We plan to replicate the analyses with several larger data sets that are more diverse in terms of clinical severity, geography, demographics, level of care and type of service providers

  22. Conclusions • The 6 GAIN validity measures are good markers for predicting problems with internal consistency, reliability, and validity • Even where there were problems, self report was still generally reliable and valid • The small correlations between measures and differences in what they predicted demonstrate that they are measuring different facets of the problem • Having developed metrics for identifying problem cases, the next step is to develop interventions to reduce the likilihood of these problems.

  23. References • Dennis, M. L., Godley, S. H., Diamond, G., Tims, F. M., Babor, T., Donaldson, J., Liddle, H., Titus, J. C., Kaminer, Y., Webb, C., Hamilton, N., & Funk, R. (2004). The Cannabis Youth Treatment (CYT) Study: Main Findings from Two Randomized Trials. Journal of Substance Abuse Treatment, in press • Dennis, M. L., Titus, J. C., Diamond, G., Donaldson, J., Godley, S. H., Tims, F., Webb, C., Kaminer, Y., Babor, T., Roebeck, M. C., Godley, M. D., Hamilton, N., Liddle, H., Scott, C., & CYT Steering Committee. (2002). The Cannabis Youth Treatment (CYT) experiment Rationale, study design, and analysis plans. Addiction, 97, 16-34. • Dennis, M. L., Titus, J. C., White, M. K., Unsicker, J., & Hodgkins, D. (2003). Global Appraisal of Individual Needs: Administration Guide for the GAIN and Related Measures. Bloomington, IL: Chestnut Health Systems. Retrieved from http://www.chestnut.org/li/gain . • Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danmarks Paedogogiske Institut. (Republished Chicago: The University of Chicago Press: 1980). • Tims, F. M., Dennis, M. L., Hamilton, N., Buchan, B. J., Diamond, G. S., Funk, R., & Brantley, L. B. (2002). Characteristics and problems of 600 adolescent cannabis abusers in outpatient treatment . Addiction, 97, 46-57.

More Related