Systems of Care Outcomes

1. Systems of Care Outcomes Michael Dennis, Ph.D. Chestnut Health Systems, Bloomington, IL Presentation at “UT CAN Local Academy 2006 Celebration, Integration and Painting the Vision”, June 5-7, 2006, Salt Lake City, Utah. Rockville, MD: Center for Mental Health Services (CMHS), Substance Abuse and Mental Health Services Administration (SAMHSA) under contract 270-2003-00006 and several individual grants. The opinions are those of the author and do not reflect official positions of the consortium or government. Available on line at www.chestnut.org/LI/Posters or by contacting Joan Unsicker at 720 West Chestnut, Bloomington, IL 61701, phone: (309) 827-6026, fax: (309) 829-4661, e-Mail: junsicker@Chestnut.Org

2. Goals • Look at the kinds of outcomes that make sense when evaluating a local or state systems of care • Shifting from emphasis from terminal outcomes, to performance measures that help you manage and fine tune the process • Recognize that there can be multiple measures and perspective • Incorporating the common values and principals into the measures • Provide a detailed example

3. 1. Needs Assessment 5. Evaluation 2. Capacity Building 4. Implementation 3. Program Selection Performance Measures & Outcomes for Local or State Systems of Care Systems of Care • Vision • Skills • Incentives • Resources • Action Plan Source: SAMHSA/CSAP Pathways Course Evaluation 101 http://pathwayscourses.samhsa.gov/eval102/eval102_1_pg2.htm

4. Examples of Systems Outcomes Vision • Can staff reliably articulate the local systems “vision”? • Do they actually use it to guide decision making? (Ask how to learn from them) • Percent of staff who strongly agree that they have used their strengths in their job (most) days • Percent of families who strongly agree that staff were trying to help them navigate the system Skills • Has the needs assessment identified specific skill sets or knowledge that is needed? • Have staff been provided skills training related to above? • To what extent are they actually using it at all, with the right people, and to proficiency? • Has this led to a measurable change in performance?

5. Examples of Systems Outcomes Incentives • Have we developed a set of incentives (recognition, training, trips to professional conferences, money) that staff appear to care about? • Are the incentives being used? • Have waivers been sought from state or other agencies to facilitate collaboration and reduce inefficiency? Resources • Have training and supervision resources been redirected to target the strategic areas for addressing skills deficit and/or building on strengths? • Have partnerships with other agencies or local community resources been made to leverage or bring in additional resources? • Have formal grants or other proposals been submitted for funds to address gaps and/or start up costs?

6. Examples of Systems Outcomes Action Plan • Have we translated the theory/logic model into the critical steps for implementation? • Do we have performance measures for each of these step that are reliably measured and available in relatively real time (e.g., weekly, monthly) to help manage the process? • Are the measures being implemented and the management reports being used? • Can we demonstrate that the management reports helped us to identify a specific problem, that something was changed, and that performance on the targeted measure improved?

7. Washington Circle Group Performance Measures Identification Everyone in Plan or Target Population Percent Screened Percent with Substance Diagnosis Initiation Percent Initiating Treatment Engagement Percent Engaged by Treatment System Services • Percent Retained by Treatment System • % with Psychiatry Services • % with Family Services • % stepped down from Residential/IOP Continuing Care Percent Retained or getting checkup/booster 90+ days after intake

8. Some Other Common Initial Issues • Standardize screening so that identification of needs is independent of what door you walk through • Increased matching based need and strengths of individuals/families and staff/programs • Percent targeted for a service who get it? Who come back for more? Who are satisfied with it? • Early working alliance/satisfaction • Actual rates of referral, receipt of referred services, and engagement in referred services • Percent of staff certified on providing a specific procedure • Adherence to a set of specific clinical or customer oriented procedures • Satisfaction with specific components of care, staff, facilities, scheduling, following up etc

9. Example: Assertive Continuing Care (ACC) Experiment 183 adolescents admitted to residential substance abuse treatment Treated for 30-90 days inpatient, then discharged to outpatient treatment Random assignment to usual continuing care (UCC) or “assertive continuing care” (ACC) Over 90% follow-up 3, 6, & 9 months post discharge Source: Godley et al 2002, in press

10. Cont. CareAdmis. Time to Enter Continuing Care and Relapse after Residential Treatment (Age 12-17) 100% 90% 80% 70% Relapse 60% Percent of Clients 50% 40% 30% 20% 10% 0% 0 10 20 30 40 50 60 70 80 90 Days after Residential (capped at 90) Source: Godley et al., 2004 for relapse and 2000 Statewide Illinois DARTS data for CC admissions

11. ACC Enhancements • Continue to participate in UCC • Home Visits • Sessions for adolescent, parents, and together • Sessions based on ACRA manual (Godley, Meyers et al., 2001) • Case Management based on ACC manual (Godley et al, 2001) to assist with other issues (e.g., job finding, medication evaluation)

12. Sustained Abstinence Early Abstinence General Continuing Care Adherence Relative to UCC, ACC will increase General Continuing Care Adherence (GCCA) GCCA (whether due to UCC or ACC) will be associated with higher rates of early abstinence Early abstinence will be associated with higher rates of long term abstinence. Assertive Continuing Care (ACC)Hypotheses Assertive Continuing Care

13. UCC ACC * p<.05 ACC Improved Adherence 100% 20% 30% 10% 40% 50% 60% 70% 80% 90% 0% Weekly Tx Weekly 12 step meetings Relapse prevention* Communication skills training* Problem solving component* Regular urine tests Meet with parents 1-2x month* Weekly telephone contact* Contact w/probation/school Referrals to other services* Follow up on referrals* Discuss probation/school compliance* Adherence: Meets 7/12 criteria* Source: Godley et al 2002, in press

14. 55% 55% 43% High (7-12/12) GCCA * p<.05 GCCA Improved Early (0-3 mon.) Abstinence 100% 90% 80% 70% 60% 50% 38% 36% 40% 30% 24% 20% 10% 0% Any AOD (OR=2.16*) Alcohol (OR=1.94*) Marijuana (OR=1.98*) Low (0-6/12) GCCA Source: Godley et al 2002, in press

15. 73% 69% 59% Early (0-3 mon.) Abstainer * p<.05 Early (0-3 mon.) Abstinence Improved Sustained (4-9 mon.) Abstinence 100% 90% 80% 70% 60% 50% 40% 30% 22% 22% 19% 20% 10% 0% Any AOD (OR=11.16*) Alcohol (OR=5.47*) Marijuana (OR=11.15*) Early(0-3 mon.) Relapse Source: Godley et al 2002, in press

16. Questions for breakout session • Do you have one or two key processes or needs you are trying to impact? • Have you identified capacity and resources to actually deal with them? • Have you picked a specific intervention to try to do? • Can you out line the small steps that would be necessary for this to work at the client level? • What are the performance measures for each step and where will this data come from? • How will the leadership team get the feedback, what will the tables/reports look like?