Michelle White, Ph.D. Chestnut Health Systems, Bloomington, IL

1. The Effects of a Quality Assurance (QA) and Certification Program on the Quality and Validity of Adolescent Substance Abuse Treatment Data Michelle White, Ph.D. Chestnut Health Systems, Bloomington, IL Presentation for the Joint Meeting on Adolescent Treatment Effectiveness, Baltimore, MD, March 28, 2006, Maryland E Room. Preparation of this manuscript was supported by funding from the Center for Substance Abuse Treatment (CSAT Contract no. 270-2003-00006). The content of this poster are the opinions of the author and do not reflect the views or policies of the government. Available on line at www.chestnut.org/LI/Posters or by contacting the author at (309) 827-6026 or by e-Mail: mwhite@Chestnut.Org

2. Summary Important judgments about how we treat adolescents for substance use disorders are typically based on data collected by clinicians and research staff. There has been much debate about the quality of self-report data, but little has been said about the impact of a quality assurance and certification protocol for interviewers collecting self-report data on the quality of that data. We now have data that shows that investing in an appropriate training and quality assurance technology can improve the quality of data collected from adolescents receiving substance abuse treatment. The Global Appraisal of Individual Needs (GAIN) Coordinating Center provides a clear and comprehensive training and quality assurance/certification program for the use of its bio-psychosocial assessments. This program has evolved over time and has been collecting data from many adolescent substance abuse treatment providers (81 providers currently submit data) for 9 years. This study examines the impact of our current quality assurance and certification program on the validity of the data being collected. We will compare interviews by levels of certifications achieved (site interviewer, administrator, or local trainer) and with interviews conducted by non-certified staff (interviews by people who eventually became certified, interviews by staff who were supposed to become certified but did not achieve it, and interviews by staff who were trained before the current certification program existed). We will also control for other potentially important factors like prior GAIN interview experience, job role (clinical versus research), and ability to meet certification deadlines. The dependent variables will be assessment duration, data quality (amount of missing/bad data, inconsistencies, randomness of response patterns, and atypical response patterns), and internal consistency (alpha). Implications for workforce development and issues for practical application of assessment certification programs will be discussed.

3. Treatment Planning Decisions are largely based on self-report information • Newer studies show self-report data to be, overall, reliable and valid • Context may affect reliability (e.g. reports of criminal acts by an inmate to a judge less reliable than a treatment study with certificate of confidentiality) • Staff training and supervision are key to ensuring the most reliable and valid self-report data regardless of setting

4. Goals of this presentation • Describe GAIN training and quality assurance and certification program • Evaluate the extent to which this process improves the efficiency, reliability and validity of GAIN data in practice

5. The GAIN has a comprehensive training and certification program • Train the trainer model to have “onsite expert” • Administration certification • Local trainer certification • Local training / Site interviewer certification

6. GAIN Training • National trainings held in Bloomington IL about 4 times/year and is 4 days in length and include breakfasts, lunches, materials, certification to local trainer level, licensing, software purchase (for single agency), and support for 1 year (up to 4 hours). • On-Site trainings held round the year as contracted and are 3 and a half days in length. • Covers administration, quality assurance, clinical use, software use, training issues, and other issues

7. Who attends training? • Day-to-day person(s) who will be training others to administer the GAIN • Person who will training/supervising others to use the GAIN • Clinical supervisor/trainer • Trainees range in education from GED/high school to PhD/MD • Trainees range in technical experience from administrative to clinical to research

8. Administration Certification • Process requires: • Train the trainer coursework certification • Submission of 2 or more taped interviews and participation in a written and oral review process • Quality assurance audiotape reviews to demonstrate mastery of materials (typically 2-4) • First tape must be submitted within 2 weeks and process must be completed within 3 months of coursework

9. Local Trainer Certification • Local trainers can directly train and certify site interviewers on the GAIN within their agency • Must be a certified administrator and deliver local training and conduct QA reviews on staff • Must submit taped assessment of someone they trained AND completed feedback • 2 stages of review to pass (1 with trainee not ready, 1 ready for certification) • Process must be completed within 6 months of train-the-trainer coursework certification

10. Local Trainings/Site Interviewer • Local trainings and QA are more flexible/do not have to follow rigid process but still have accountability for same standards for completing certification like the administration level • A site interviewer is trained by a local trainer and does not have coursework certification • A site interviewer CANNOT go on to local trainer status until completing coursework certification as part of a national training

11. Practice, QA and Certification Reduced Duration by 31% Administration certification shown to reduce GAIN Administration time 160 140 120 100 80 60 40 20 0 Jul-04 Jan-04 Jun-04 Feb-04 Apr-04 Dec-03 Mar-04 Aug-04 May-04 Core (n=428) (118 to 96 min) Core + (n=140) (143 to 119 min) Full (n=415) (127 to 116 min) Total (n=983) (132 to 108 min)

12. What Does a QA Reviewer do? • Listens to the GAIN interview audiotape and paper copy to determine the quality of the administration. • Reports issues using specific behavioral criteria found within four main sections of the feedback: Documentation, Instructions, Items, and Engagement. • Reviewer rates each of 4 sections as: Excellent, Sufficient, Minor Problems, Problems and rates overall administration as pending (not certified) or certified

13. QA Reviewer Ratings • 100% Agreement in certification status rating in blind reviews (CHS reviewers) • Increase in inter-rater reliability from first attempt to last attempt at local trainer certification by trainees with CHS reviewer (Kappas improve): • Documentation (.28 to .66) • Instructions (.25 to .58) • Items (.17 to .65) • Engagement (.14 to .45)

14. Data Source • Used data collected from 7,269 adolescents interviewed with GAIN version 5 as part of 89 CSAT adolescent treatment grants since 2002 • Data are from geographically disperse programs • CSAT adolescent treatment clients are similar to those in the public treatment system as represented by the TEDS public 2003 admissions

15. Geographic Location of Sites NH WA VT ME MT ND MN OR MA NY ID WI SD MI WY RI IA PA CT NE OH NJ NV DC IN UT IL CA CO WV VA DE DC KS MO KY MD NC TN AR AZ OK NM SC GA AL MS Program ART TX LA EAT AK SCY FL TCE YORP HI PR

16. Demographics 100% 10% 30% 60% 80% 20% 40% 50% 70% 90% 0% 30% Female TEDS (n=153,251) 29% CSAT (n=7,226) 19% African American 18% 58% Caucasian 42% 16% Hispanic 17% 6% Mixed/Other 23% 17% 12 to 14 years old 20% 83% 15 to 17 years old 73%

17. Clinical Severity 100% 10% 20% 30% 40% 50% 60% 70% 80% 90% 0% First used under 82% age 15 74% 33% TEDS (n=153,251) Prior Treatment 37% CSAT (n=7,226) Weekly use at 50% intake 61% Past Year 48% Dependence 53% Criminal Justice 53% System 68%

18. Primary, Secondary or Tertiary SUD Problems 100% 10% 50% 60% 90% 20% 30% 40% 70% 80% 0% 57% Alcohol 25% 82% Marijuana/Hash 60% 8% Cocaine/Crack 5% 4% Heroin/Opiates 3% 7% Meth/amphetamines 7% TEDS (n=153,251) 6% CSAT (n=7,226) Any Other 2%

19. Other ASAM Issues (not in TEDS) 100% 10% 20% 30% 40% 50% 60% 70% 80% 90% 0% Any withdrawal symptoms past week 0.31 Severe withdrawal (11+ symptoms) 0.08 Sexually active in past 90 days 0.84 Major health problems 0.27 Any co-occurring psychiatric 0.79 Ever physical, sexual or emotional victimization 0.59 Doesn't acknowledge AOD problem 0.70 Doesn't acknowledges need for treatment 0.76 Regular alcohol use in recovery environment 0.70 Regular drug use in recovery environment 0.81 Any violence or illegal activity 0.82 Any past year violent crime 0.45

20. Outcome Measures • Number of Inconsistencies –Count of 65 paired items consistently answered by over 90% of the clients, that are inconsistent • Duration – time doing the interview (not including breaks) in Minutes • Denial/Misrepresentation – Sum of staff rating over 8 sections on a scale of 0-no problem, 1-estimating, 2-misunderstanding, 3-denial, 4-misrepresentation • Context Effect – staff report of problems that might effect the interview (e.g.., someone present, interruptions, in juvenile justice setting) • Proportion of Missing Data on 99 Items used in the GAIN’s core 10 Change measures. • Atypicalness a measure of endorsing high severity items without first endorsing the typical prior items (e.g.., suicide without depression) on the 123 Items of the GAIN’s 4 main psychopathology and psychopathy scales; This measures is based on the RASCH outfit statistic and reported in Logits • Randomness a measure of answers that are more random than expected on the GAIN’s 4 main psychopathology and psychopathy scales; This measures is based on the RASCH infit statistic and reported in Logits

21. Pre vs. Post Certification \a Cohen's D -0.6 -0.4 -0.2 0 Proportion of -0.39 Inconsistencies (100%)* Duration -0.25 (in Minutes)* Denial/Misrepresentation -0.24 (Staff Rating)* Context Effect -0.10 (Staff Report) Proportion of Missing -0.04 Data (100%) Atypicalness -0.03 (Outfit in Logits) Randomness -0.03 (Infit in Logits) \a Cohen's d (Post Certification - Pre Certification)/Pooled STD * p<.05

22. Level of Certification Cohen's f \a -0.40 -0.20 0.00 0.20 0.40 Proportion of Inconsistencies (100%) * Certified Interviewers generally do better Duration (in Minutes) * Denial/Misrepresentation (Staff Rating) * Context Effect (Staff Report) * Proportion of Missing Data (100%) * Pre-Certification Administrator Cert. Atypicalness This one in the wrong direction (Outfit in Logits) Site Certified Local Trainer Cert. Randomness (Infit in Logits) * p<.05 \a Cohen's f (Group Mean – Pooled Mean/Pooled STD

23. Staff Experience 500 450 400 350 300 GAIN’s complete 250 200 150 100 50 0 0 90 20 30 40 50 60 70 10 80 100 Number of Prior Interviews by Staff

24. Impact of Staff Experience Major improvement over the first 15 interviews Most improvements have occurred by 60 interviews

25. Discussion • Completing certification improves interview: lower administration time, fewer inconsistencies • No significant difference on Atypicalness and randomness, which depend more on individual severity factors of clients • Contrary to common expectations, “trained and certified” clinicians did better than research assistants in terms of our efficiency, reliability and validity measures. • Staff Experience is also clearly a factor in achieving better quality data

26. Qualitative information verifies value of program… • Post-certification quotes from individuals initially resistant to going through process state that clinicians and researchers feel they do a better job interviewing after having gone through the training, QA, and certification program. • For example…

27. Quotes… • [My QA reviewer’s] “comments were clear and directive. She was supportive, tactful, and empowering. I am very impressed and feel quite positive about my experience…” • [My QA reviewer] “was fantastic! I really wish I would have called him sooner in my administration certification process. He helped me feel more relaxed about the process by easing my fears and explaining things in a way that can only be described as supportive and understanding.”

28. Quotes… • “As you know, mid way through my certification [my reviewer] was assigned another project, while that might have been an awkward time, it was not. In no way did I feel hampered by the change and I found [new reviewer] to be as wonderful to work with…the staff have been professional, knowledgeable, and very approachable. They have never seemed unwilling to help, and not only that, they are so positive…They could teach a great many people [in the World] what customer service is. I will look back on this experience with pleasure and good memories – not something I can say about a lot of procedures.”

29. Acknowledgement The content of this presentations are based on treatment & research funded by the Center for Substance Abuse Treatment (CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA) under contract 270-2003-00006 using data provided by CSAT adolescent treatment grantees under the Adolescent Residential Treatment (ART), Effective Adolescent Treatment (EAT), Strengthening Communities for Youth (SCY), Targeted Capacity Expansion (TCE), and Young Offender Re-entry Program (YORP) grants (TI013313, TI013309, TI013344, TI013354, TI013356, TI013305, TI013340, TI130022, TI03345, TI012208, TI013323, TI14376, TI14261, TI14189,TI14252, TI14315, TI14283, TI14267, TI14188, TI14103, TI14272, TI14090, TI14271, TI14355, TI14196, TI14214, TI14254, TI14311, TI15678, TI15670, TI15486, TI15511, TI15433, TI15479, TI15682, TI15483, TI15674, TI15467, TI15686, TI15481, TI15461, TI15475, TI15413, TI15562, TI15514, TI15672, TI15478, TI15447, TI15545, TI15671)). The author would like to thank Lexy Adkins, Rod Funk, Melissa Ives, Melissa Jerse, Sarah Knecht, & Mike Dennis for their help preparing the presentation. The opinions are those of the author and do not reflect official positions of the consortium or government. Available on line at www.chestnut.org/LI/Posters or by contacting Joan Unsicker at 720 West Chestnut, Bloomington, IL 61701, phone: (309) 827-6026, fax: (309) 829-4661, e-Mail: junsicker@Chestnut.Org.