Evaluation of Colorado TASC Melissa Ippolito Colorado TASC Director Jean Denious Director of Research, OMNI Institu

2. Overview of Presentation Introduction to Colorado TASC and OMNI Statewide Evaluation of Colorado TASC Evaluation Implementation Challenges Strategies to Address Implementation Challenges Preliminary Findings Evaluation Next Steps Statewide Evaluation of Colorado TASC Evaluation Plan Development Evaluation Goals Research Questions Methodology Statewide Evaluation of Colorado TASC Evaluation Plan Development Evaluation Goals Research Questions Methodology

3. Colorado TASC Funded by the Colorado Department of Corrections Administered by Peer Assistance Services, Inc. 10 TASC offices organized by region: Mile High, Northeast, Western, and Southeast Staff at various levels Intake Coordinators Monitoring Technicians Case Managers Regional Directors PAS Mission Statement: Dedicated to quality, accessible prevention and intervention services in workplaces and communities, focusing on substance abuse and related issues. (RE: from website) Over 50 staff.  PAS Mission Statement: Dedicated to quality, accessible prevention and intervention services in workplaces and communities, focusing on substance abuse and related issues. (RE: from website) Over 50 staff.

4. Colorado TASC Services provided to: Targeted population of individuals Recently released from incarceration Have a minimum of six months of parole requirements, and Need substance use treatment Over 10,000 parolees yearly Serves > 80% of all parolees in need of substance abuse treatment in CO

5. Colorado TASC Regional Variations Population demographics Treatment needs Wraparound needs Availability of treatment resources Client volume Setting in which clients are seen 

6. OMNI Institute Social Research and Evaluation firm Areas of expertise include criminal justice, behavioral health OMNI�s partnership with PAS Access to Recovery (ATR) Holistic Health Recovery Program (HHRP) Healthy Relationships Program Screening, Brief Intervention, and Referral to Treatment (SBIRT) 

7. Benefits of Evaluation Better or more efficiently manage programmatic efforts Refine/improve a program to get better outcomes with clients Test and disseminate a program that has strong evidence of effectiveness Strengthen grants or funding efforts to enable sustainability 

8. Development of the Evaluation Plan Iterative Multiple Stages Formative Evaluation (Fall of 2008) Pilot Evaluation (Fall of 2009) Statewide Evaluation (January of 2011) Collaborative PAS and TASC staff Colorado Department of Corrections Formative Evaluation Understand the TASC Program in Colorado Develop a program evaluation plan Identify key research questions Outline methods to test TASC effects on substance abuse, substance abuse related offenses, recidivism Identify other useful variables not currently collected and related data collection tools and methods Obtain staff input to guide development of evaluation plan Help promote program alignment across regions Pilot Evaluation Implement & test TASC procedural changes that emerged as a result of work in formative evaluation Standardize existing procedures and test these Implement new procedures and test these Test evaluation procedures with select sites with differences in factors such as population characteristics and/or treatment resources �to surface any potential weaknesses of the plan before implementing statewide Conduct preliminary analyses to answer key research questions Obtain staff input to guide refinement of evaluation plan Statewide Evaluation Expected to continue through the end of 2012 Implement evaluation (and any procedural changes) statewide Answer key research questions Obtain staff input to contextualize and better understand results Collaborative PAS and TASC Staff Developed evaluation plans in close conjunction with input from TASC staff Obtained TASC staff feedback at various points in the evaluation process Colorado Department of Corrections Obtained guidance and support from DOC that is much more familiar with this work and population  Formative Evaluation Understand the TASC Program in Colorado Develop a program evaluation plan Identify key research questions Outline methods to test TASC effects on substance abuse, substance abuse related offenses, recidivism Identify other useful variables not currently collected and related data collection tools and methods Obtain staff input to guide development of evaluation plan Help promote program alignment across regions Pilot Evaluation Implement & test TASC procedural changes that emerged as a result of work in formative evaluation Standardize existing procedures and test these Implement new procedures and test these Test evaluation procedures with select sites with differences in factors such as population characteristics and/or treatment resources �to surface any potential weaknesses of the plan before implementing statewide Conduct preliminary analyses to answer key research questions Obtain staff input to guide refinement of evaluation plan Statewide Evaluation Expected to continue through the end of 2012 Implement evaluation (and any procedural changes) statewide Answer key research questions Obtain staff input to contextualize and better understand results Collaborative PAS and TASC Staff Developed evaluation plans in close conjunction with input from TASC staff Obtained TASC staff feedback at various points in the evaluation process Colorado Department of Corrections Obtained guidance and support from DOC that is much more familiar with this work and population

9. Development of the Evaluation Plan Adherence to ethical standards of research with human subjects and vulnerable populations Institutional Review Board (IRB) review by University of Colorado at Boulder Research review by Colorado Department of Corrections 

10. Evaluation Goals Document the program�s effectiveness in decreasing: Substance abuse Substance abuse related offenses, and, Over the long term, re-admission to Colorado DOC facilities for new offenses Support quality improvement of TASC services to parolees with substance abuse problems

11. Research Questions Pilot Evaluation and Statewide Evaluation How many clients do TASC programs serve annually? What are the background and significant characteristics of individuals admitted to TASC? What are the treatment needs of clients at admission? What services do TASC clients receive (e.g., number of contacts and referrals)?

12. Research Questions Pilot Evaluation and Statewide Evaluation How many clients successfully complete TASC, Parole, or both? Do the background and significant characteristics of successful completers of TASC and parole differ from non-completers of TASC and parole? Do clients report reduced substance use, reduced social impairment due to substance use, increased motivation to seek help for substance use issues, or increased overall functioning relative to substance use, from intake to discharge? Do clients show a reduced risk of reoffending from intake to discharge?

13. Research Questions Statewide Evaluation All of the previous questions, and What are recidivism rates for TASC clients? How do TASC clients differ from a similar group of parolees not receiving TASC services Background characteristics Recidivism rates

14. MethodologyKey Considerations Use existing data where possible Introduce new procedures where necessary Minimize data collection burden for TASC staff TASC Database Great Resource TASC Database Great Resource

15. MethodologyKey Considerations Use existing data where possible TASC Database Demographic/background information Treatment and wraparound service needs at intake/admission to TASC Referrals to treatment TASC assessment scores Standardized Offender Assessment - Revised (SOA-R) battery TASC Database Great Resource Standardized Offender Assessment � Revised (SOA-R) is a battery of tests used to facilitate identification of treatment needs including: Level of Supervision Inventory (LSI) � calculates offender�s risk of re-offending Adult Substance Use Survey (ASUS)� � evaluates offender�s substance history Service/Treatment Recommendations Worksheet (TXRW) � a tool that uses LSI and ASUS scores to determine a recommended level of care Simple Screening Instrument-Revised (SSI-R) � a screening tool for drug and alcohol history TASC Database Great Resource Standardized Offender Assessment � Revised (SOA-R) is a battery of tests used to facilitate identification of treatment needs including: Level of Supervision Inventory (LSI) � calculates offender�s risk of re-offending Adult Substance Use Survey (ASUS)� � evaluates offender�s substance history Service/Treatment Recommendations Worksheet (TXRW) � a tool that uses LSI and ASUS scores to determine a recommended level of care Simple Screening Instrument-Revised (SSI-R) � a screening tool for drug and alcohol history

16. MethodologyKey Considerations Introduce new procedures where necessary Standardize intake procedures Standardize baseline assessments Introduce 6-month re-assessments Minimize data collection burden for TASC staff Consent procedures and re-assessments Data entry Introduce new procedures where necessary Standardize intake procedures Standardize baseline assessments (LSI) 6-month re-assessments Minimize data collection burden for TASC staff TASC staff conduct consent procedures and reassessments, BUT, OMNI conducts data entry for information not already being entered into TASC database Introduce new procedures where necessary Standardize intake procedures Standardize baseline assessments (LSI) 6-month re-assessments Minimize data collection burden for TASC staff TASC staff conduct consent procedures and reassessments, BUT, OMNI conducts data entry for information not already being entered into TASC database

17. Assessments Level of Supervision Inventory (LSI; Andrews, 1982) Case manager report Semi-Structured Interview Risk of reoffending Criminal History Attitude Towards Crime/Authority Others Level of Supervision Inventory (LSI) The LSI was developed in the late 1970s in Canada through the collaboration of probation officers, correctional managers, practitioners, and researchers (Andrews, 1982). Andrews, D. A. (1982). The Level of Supervision Inventory (LSI): The first follow-up. Toronto: Ontario Ministry of Correctional Services. Criminal risk or risk of recidivism 54 items re: criminal history, attitude towards crime/authority, functioning in areas such as education/employment, family, housing, leisure/recreation, emotional/personal well-being. Level of Supervision Inventory (LSI) The LSI was developed in the late 1970s in Canada through the collaboration of probation officers, correctional managers, practitioners, and researchers (Andrews, 1982). Andrews, D. A. (1982). The Level of Supervision Inventory (LSI): The first follow-up. Toronto: Ontario Ministry of Correctional Services. Criminal risk or risk of recidivism 54 items re: criminal history, attitude towards crime/authority, functioning in areas such as education/employment, family, housing, leisure/recreation, emotional/personal well-being.

21. Assessments Adult Substance Use Survey � Revised (ASUS � R; Kenneth Wanberg) Client report Substance use history Lifetime Past 6 months Problems associated with substance use Readiness to seek help for substance use problems Adult Substance Use Survey � 96 items Involvement scale Disruption scale Social impairment due to substance use Motivation scale Adult Substance Use Survey � 96 items Involvement scale Disruption scale Social impairment due to substance use Motivation scale

24. Assessments LSI ASUS-R Data from DOC Re-arrest/re-incarceration data Comparison Sample

25. Challenges Unanticipated delays related to IRB approval process Inconsistent use of database functions Inconsistent submission of assessment forms Invalid data seen on assessment forms Average length of stay with TASC is less than 6 months Inconsistent use of database For example: not indicating consent, mental health diagnosis, wraparound services Inconsistent coding of service level, discharge status, treatment completion Reflects lack of familiarity with database functions, confusion around data fields or programmatic definitions (e.g., when to indicate �successful discharge�) Challenges with updating database to support evaluation needs For example: providing reminders for 6 month re-assessments Technical issues with complexity of database and limitations of software used Issues arising from discrepancy between original intent of the database (i.e., to facilitate case management of clients) and use of database in evaluation � for example, overwriting fields Average length of stay with TASC is less than 6 months (see note in slide) Due to various factors including long time between intake and first case management appointment because of large volume of clients Inconsistent use of database For example: not indicating consent, mental health diagnosis, wraparound services Inconsistent coding of service level, discharge status, treatment completion Reflects lack of familiarity with database functions, confusion around data fields or programmatic definitions (e.g., when to indicate �successful discharge�) Challenges with updating database to support evaluation needs For example: providing reminders for 6 month re-assessments Technical issues with complexity of database and limitations of software used Issues arising from discrepancy between original intent of the database (i.e., to facilitate case management of clients) and use of database in evaluation � for example, overwriting fields Average length of stay with TASC is less than 6 months (see note in slide) Due to various factors including long time between intake and first case management appointment because of large volume of clients

26. Strategies to Address Challenges Additional Staff Training Greater role of Regional Directors in overseeing data quality and evaluation procedures Monthly evaluation leadership meetings with Team Leads Written manual for standard operating procedures Monthly data transfer Monthly review of data quality issues Additional Staff Training Evaluation procedures Impact of missing/invalid data on assessments Interactive activities �to help increase retention of material Assessments Database functions Additional Staff Training Evaluation procedures Impact of missing/invalid data on assessments Interactive activities �to help increase retention of material Assessments Database functions

27. 4 Types of Errors: if someone notes one instance and there are multiple (e.g., 4) point out all the remaining instances) 1. No age last used throughout 2. Item #3, and #7-10, A and E for number of times used in lifetime 3. Item #4: Never used, but use in last six months shows that they�ve used 1-10 times 4. Item# 7: A and E for number of times in last six months. VO NOTES: Okay here we have some more things that are amiss. 4 Types of Errors: if someone notes one instance and there are multiple (e.g., 4) point out all the remaining instances) 1. No age last used throughout As you can see the entire column for age used throughout is not written we could assume that they never used and that is why the column is blank but this is not true for all of the items. 2. Item #3, and #7-10, A and E for number of times used in lifetime Both never used and more than 50 times are circled. This would be another item that would have missing data. But it helps to have your insight with items like these. In these cases how would you interpret the items that have circles in both places? Rephrase this question to the group if necessary. 3. Item #4: Never used, but use in last six months shows that they�ve used 1-10 times In this case we have column (b) circled, but based on what is in column (a) this should not be the case. 4. Item# 7: A and E for number of times in last six months. We have a similar concern with number 7, both a and e are circled. This would be another item that would have missing data. Again it helps to have your insight with items like these. In these cases how would you interpret the items that have circles in both place? How did the case manager come up with a score of 20 based on the items we noted. Can anyone provide insight on how that is calculated and how we should understand any rules associated with what we just reviewed? 4 Types of Errors: if someone notes one instance and there are multiple (e.g., 4) point out all the remaining instances) 1. No age last used throughout 2. Item #3, and #7-10, A and E for number of times used in lifetime 3. Item #4: Never used, but use in last six months shows that they�ve used 1-10 times 4. Item# 7: A and E for number of times in last six months. VO NOTES: Okay here we have some more things that are amiss. 4 Types of Errors: if someone notes one instance and there are multiple (e.g., 4) point out all the remaining instances) 1. No age last used throughout As you can see the entire column for age used throughout is not written we could assume that they never used and that is why the column is blank but this is not true for all of the items. 2. Item #3, and #7-10, A and E for number of times used in lifetime Both never used and more than 50 times are circled. This would be another item that would have missing data. But it helps to have your insight with items like these. In these cases how would you interpret the items that have circles in both places? Rephrase this question to the group if necessary. 3. Item #4: Never used, but use in last six months shows that they�ve used 1-10 times In this case we have column (b) circled, but based on what is in column (a) this should not be the case. 4. Item# 7: A and E for number of times in last six months. We have a similar concern with number 7, both a and e are circled. This would be another item that would have missing data. Again it helps to have your insight with items like these. In these cases how would you interpret the items that have circles in both place? How did the case manager come up with a score of 20 based on the items we noted. Can anyone provide insight on how that is calculated and how we should understand any rules associated with what we just reviewed?

28. Preliminary Findings from Pilot Evaluation Demographics: male, less than 12 years of education, unemployed, single Criminal Risk: high or medium risk Prior SA treatment: one-half to one-third Most frequently used substances: alcohol, marijuana Clients report low levels of substance use in past 6 months SA treatment needs � 40% severe or mod-to-severe Mental health diagnosis: one-half Wraparound Service Needs: transportation, employment Prior SA treatment � report prior outpatient tx; 1/3 report prior inpatient tx. Clients report low levels of substance use in past 6 months � 20% in the high category Potentially due to recently coming out of incarceration or under-reporting because have not yet developed trust/rapport with case manager Criminal History/Criminal Risk Close to 50% of clients had spent 4 or more years in jail or prison in their lifetime. Over two-thirds of clients scored in the high category with regards to their lifetime history of legal-non conforming behavior. Close to half of all clients had been arrested before age 16. At Western TASC, clients were evenly split between medium and high criminal risk categories. However, at Mile High TASC, the majority of clients (69%) fell in the high criminal risk category. Close to half of all clients were participating in Intensive Supervised Parole at the time of admission. Total scores on the LSI were converted to low, medium, or high risk categories based on guidelines established by the Colorado DOC. See �Evidence Based Correctional Practices� � prepared by Colorado Division of Criminal Justice, Office of Research and Statistics. Available at http://www.nga.org/Files/pdf/0805SENTENCERES10.PDF SA Treatment Needs 40% with moderate; 40% with severe or mod-to-severe Wraparound Services about � need transportation or employment about 1/3 need food, housing, medical care/medications services provided: bus passes/bus tokens � 70%; 10% incentives 6 contacts on average; 1-2 referrals on average; majority of referrals to outpatient tx Prior SA treatment � report prior outpatient tx; 1/3 report prior inpatient tx. Clients report low levels of substance use in past 6 months � 20% in the high category Potentially due to recently coming out of incarceration or under-reporting because have not yet developed trust/rapport with case manager Criminal History/Criminal Risk Close to 50% of clients had spent 4 or more years in jail or prison in their lifetime. Over two-thirds of clients scored in the high category with regards to their lifetime history of legal-non conforming behavior. Close to half of all clients had been arrested before age 16. At Western TASC, clients were evenly split between medium and high criminal risk categories. However, at Mile High TASC, the majority of clients (69%) fell in the high criminal risk category. Close to half of all clients were participating in Intensive Supervised Parole at the time of admission. Total scores on the LSI were converted to low, medium, or high risk categories based on guidelines established by the Colorado DOC. See �Evidence Based Correctional Practices� � prepared by Colorado Division of Criminal Justice, Office of Research and Statistics. Available at http://www.nga.org/Files/pdf/0805SENTENCERES10.PDF SA Treatment Needs 40% with moderate; 40% with severe or mod-to-severe Wraparound Services about � need transportation or employment about 1/3 need food, housing, medical care/medications services provided: bus passes/bus tokens � 70%; 10% incentives 6 contacts on average; 1-2 referrals on average; majority of referrals to outpatient tx

29. Next Steps Refine research questions Evaluation of impact on recidivism Comparison to outcomes for similar parolees not receiving services from TASC Three areas not addressed by the current evaluation design Outcomes for clients with co-occurring mental health problems Impact of criminal thinking Effectiveness of treatment providers and programs Refine Research Questions look at Defensiveness, Disruption insteadRefine Research Questions look at Defensiveness, Disruption instead

30. Contact Information Peer Assistance Services/Colorado TASC Melissa Ippolito, BA, CAC III, Colorado TASC Director Peer Assistance Services, Inc. (970) 242-2800 MIppolito.TASC@PeerAssist.org OMNI Institute Pallavi Visvanathan, PhD, & Jean Denious, PhD (303) 839-9422 PVisvanathan@OMNI.org | JDenious@OMNI.org