Achieving Our Mission

1. Achieving Our Mission The Role of Continuous Quality Improvement in Community Corrections and Public Safety Kimberly Gentry Sperber, Ph.D.

2. Efforts To Date • “What Works” Literature • Principles of Effective Interventions • Growing evidence based on individual program evaluations and meta-analyses • Continuing Gap Between Science and Practice • Few programs score as satisfactory on CPAI

3. Why Isn’t “It” Working?Latessa, Cullen, and Gendreau (2002) • Article notes 4 common failures of correctional programs: • Failure to use research in designing programs • Failure to follow appropriate assessment and classification practices • Failure to use effective treatment models • Failure to evaluate what we do

4. CPAI Data as Evidence of Fidelity • Lowenkamp and Latessa (2005) • Examined data from 38 residential correctional programs for adults • Looked at relationship between program fidelity and program effectiveness. • Program fidelity was assessed using the CPAI. • Found significant correlation between fidelity and effectiveness • CPAI scores correlated to reincarceration

5. Lowenkamp and Latessa FindingsContinued • Differences in recidivism rates based on CPAI scores: • Scores of 0-49% demonstrated 1.7% reduction compared to comparison group. • Scores of 50-59% demonstrated 8.1% reduction. • Scores of 60-69% demonstrated 22% reduction.

6. CPAI Data Continued • Holsinger (1999) • Examined data from Adolescent Community Correctional Facilities in Ohio • Looked at relationship between program fidelity and program effectiveness. • Program fidelity was assessed using the CPAI. • Outcome measures examined included any court contact, felony or misdemeanor, felony, personal offense, and commitment to a secure facility

7. CPAI Data Continued • Total composite score significantly correlated with all outcome measures. • Each individual domain of the CPAI also significantly correlated with all of the outcomes • Program Implementation • Client Assessment • Program Characteristics • Staff Quality • Evaluation

8. Recent CPAI Results • Results from more than 550 programs: • 7% HIGHLY EFFECTIVE • 18% EFFECTIVE • 33% NEEDS IMPROVEMENT • 42% INEFFECTIVE

9. More Fidelity Research • Landenberger and Lipsey (2005) • Brand of CBT didn’t matter but quality of implementation did. • Implementation defined as low dropout rate, close monitoring of quality and fidelity, and adequate training for providers. • Schoenwald et al. (2003) • Therapist adherence to the model predicted post-treatment reductions in problem behaviors of the clients. • Henggeler et al. (2002) • Supervisors’ expertise in the model predicted therapist adherence to the model. • Sexton (2001) • Direct linear relationship between staff competence and recidivism reductions.

10. More Fidelity Research Cont’d. • Schoenwald and Chapman (2007) • A 1-unit increase in therapist adherence score predicted 38% lower rate of criminal charges 2 years post-treatment • A 1-unit increase in supervisor adherence score predicted 53% lower rate of criminal charges 2 years post-treatment. • Schoenwald et al. (2007) • When therapist adherence was low, criminal outcomes for substance abusing youth were worse relative to the outcomes of the non-substance abusing youth.

11. Washington State Example(Barnoski, 2004) • For each program (FFT and ART), an equivalent comparison/control group was created • Felony recidivism rates were calculated for each of three groups, for each of the programs • Youth who received services from therapists deemed ‘competent’ • Youth who received services from therapists deemed ‘not competent’ • Youth who did not receive any services (control group)

12. Functional Family Therapy Results: % New Felony Results calculated using multivariate models in order to control for potential differences between groups

13. Washington State Study Continued • When FFT was delivered competently, the program reduced felony recidivism by 38% • When considering how much the program costs, substantial savings in ‘avoided crime’ were observed – particularly for the competent therapists • When ART was competently delivered, felony recidivism was reduced by 24% • Also resulted in substantial savings

14. Project Greenlight • Short-term prison-based reentry program in New York • CBT Skills Training • Employment Services • Housing Services • Drug Education and Awareness • Family Counseling • Practical Skills Training • Community-Based Networks • Familiarity With Parole • Individualized Release Plans

15. Project Greenlight Benefits • Participants received more service referrals • Participants reported more contacts with community services after release • Participants demonstrated significantly more familiarity with parole conditions • Participants were more positive about parole

16. But Did It Work?

17. What Went Wrong? • Violation of the risk principle • Ceased use of risk assessment instrument when staff deemed process too cumbersome • Violation of the need principle • All offenders received same services whether needed or not • Violation of the fidelity principle • Staff modified delivery of the CBT curriculum (shortened the duration, increased frequency, increased class size) • Differential staff competence • Certain case managers produced worse outcomes

18. UC Halfway House/CBCF Study in Ohio: A Look at Fidelity Statewide • Average Treatment Effect was 4% reduction in recidivism • Lowest was a 41% Increase in recidivism • Highest was a 43% reduction in recidivism • Programs that had acceptable termination rates, had been in operation for 3 years or more, had a cognitive behavioral program, targeted criminogenic needs, used role playing in almost every session, and varied treatment and length of supervision by risk had a 39% reduction in recidivism

19. 2010 UC Halfway House/CBCF Study in Ohio:Adherence to CBT in Groups and Changes in Recidivism

20. What Do We Know About Fidelity? • Fidelity is related to successful outcomes (i.e., recidivism reductions). • Poor fidelity can lead to null effects or even iatrogenic effects. • Fidelity can be measured and monitored. • Fidelity cannot be assumed.

21. QA versus CQI CQI – What Is It? Infrastructure Peer Review Indicators Client Satisfaction Action Planning Process Evaluation Outcome Evaluation Benefits Monitoring Fidelity Through a CQI Process

22. What Is Quality? • Services are based on current professional knowledge. • Services produce desirable outcomes.

23. CQI – What Is It? • A method of continuously examining processes and making them better. • Key principles: • Use of data and team approaches to improve decision making • Involvement of entire organization to improve quality • Strong focus on customers • Continuous improvement of all processes and outcomes

24. CQI versus QA • QA: • Retrospective review process • Emphasis on regulatory and contract compliance • Catching people being bad leads to hide and seek behavior

25. CQI versus QA • CQI: • CQI is a prospective process • Holds quality as a central priority within the organization • Focus on customer needs; relies on feedback from internal and external customers • Emphasizes systematic use of data • Not blame-seeking • Trust, respect, and communication • Move toward staff responsibility for quality , problem solving and ownership of services

26. Objectives of CQI • To facilitate the Agency’s mission • To ensure appropriateness of services • To improve efficiency of services/processes • To improve effectiveness of directing services to client needs • To foster a culture of learning • To ensure compliance with funding and regulatory standards

27. Building a CQI Process • Formal infrastructure • Core Elements • Documentation Review • Indicators • Process Versus Outcome • Performance Goals • Action Planning • Customer Satisfaction • Clients, Staff, Stakeholders • Program Evaluation

28. Process Evaluation Sample Projects

29. Process Evaluation • Are we serving our target population? • Are the services being delivered? • Did we implement the program as designed (tx fidelity)? • Are there areas that need improvement?

30. Example 1Review of LSI Scores • Reviewed all open cases at Facility A • Recorded LSI risk category, UC Risk category, and name of interviewer • 77.5% of cases reviewed did not have a match between staff rating and UC rating

31. LSI Scores Post-Training • First 2 weeks after training – 0 matches • 3-6 weeks after training – 46.2% matched • First 2 weeks after training – 50% were off by 2 risk categories • 3-6 weeks after the training – 0% were off by 2 risk categories

32. Example2CBIT Site Assessments • Cognitive Behavioral Implementation Team • Site visits for observation and rating • Standardized assessment process • Standardized reports back to sites • Combination of quantitative data and qualitative data

33. Example 3Review of Core Correctional Practices • Reviewed 6 programs • Live observation of treatment groups for use of core correctional practices: • Anticriminal Modeling • Effective Reinforcement • Effective Disapproval • Problem-Solving • Structured Skill Building • Effective Use of Authority • Cognitive Restructuring • Relationship Skills • Documented strengths and opportunities for improvement

34. Example 4Process Evaluation of 2 Adolescent Residential Programs • Collecting data on 128 youth admitted during FY10 and FY11 across 2 boys’ programs. • Goal – to identify and quantify changes in population and programming in order to identify any necessary changes to service delivery. • Advantages: • Allows the agency to gain an accurate assessment of current target population and treatment needs • Allows the agency to seek out evidence-based practices that best align with current treatment needs of clients • Positions the agency for future outcome evaluation

35. Example 5Assessing Best Practices at 17 Sites • Use of ICCA Treatment Survey to establish baseline • Complete again based on best practice • Perform Gap Analysis • Action Plan • Reassess

36. Outcome Evaluation Sample Projects

37. Outcome Evaluation • Are our services effective? • Do clients benefit (change) from the services? • Intermediate outcomes • Reduction in risk • Reduction in antisocial values • Long-term outcomes • Recidivism • Sobriety

38. Example 1Off-Site Non-Emergency Medical Visits for 6 Months • Goals of the project: • Reduce the overall number of off-site visits for non-emergency medical care. • Reduce the number of staff-escorted visits (impacts dollars and coverage) • Reduce the amount of behavioral treatment missed • Increase the number of referrals for primary care upon discharge

39. Example 1 ContinuedOutcomes Data

40. ER VISIT COSTS $ 400 Average Visit $ 34 Average Staff Cost $$$ Hospital Pharmacy TOTAL $ 434 not including pharmacy savings TALBERT HOUSE $ 62.50 Average Visit $ 0 off-site staff cost Medication savings: samples, patient assistance programs TOTAL $62.50 average cost/visit Savings of $371.50/visit Sample Cost ComparisonER versus On-Site Care

41. Example 2Outcomes Pre/Post TFM ImplementationPassages Program for Girls

42. Example 3Outcome Evaluation of Enhanced Outpatient Services • 3year SAMHSA grant to enhance drug court OP program • Sample of 357 clients • Assessing intermediate outcomes (6 months post-intake) • Assessing long-term outcomes (12 months post discharge) • Comparing to clients receiving services prior to enhancements

43. Benefits of Program Evaluation • Proof of effective services • Maintain or secure funding • Improve staff morale and retention • Educate key stakeholders about services • Highlights opportunities for improvement • Data to inform quality improvement initiatives • Establish/enhance best practices • Monitor/ensure treatment fidelity

44. The Role of QA/QI in Community Corrections(based on UC Halfway House and CBCF study)

45. NPC Research on Drug Courts

46. Conclusions • Many programs are not implementing the principles of effective intervention with strong fidelity. • Result is an ongoing gap between science and practice. • This gap often results in null or even iatrogenic effects. • Correctional organizations have a responsibility to ensure effective services. • Monitoring fidelity is key to success. • Responsibility for EBP needs to be aligned at all levels – administration, management, line staff. • Need to focus on creating formal infrastructure to support and sustain evidence-based practices.