Using CANS Data to Assist in the Creation of a Learning Collaborative to Understand Program Effectiveness

1. Using CANS Data to Assist in the Creation of a Learning Collaborative to Understand Program Effectiveness Representatives of the Maryland Treatment Foster Care Learning Collaborative: Chris Kelly, LCSW-C, Child and Family Services Division, Catholic Charities Steve Herr, PhD, LCSW-C, University of Maryland, Department of Psychiatry, Division of Service Research Paul Brylske, LCSW-C, Kennedy Krieger Family Center Amy Meyerl, LCSW-C, Kennedy Krieger Family Center Maurice Williams, LCSW, Williams Life Center Maisha Davis, LGSW, Progressive Life Center Debbie Marini, LCSW-C, Baptist Family Services 7th Annual National CANS Conference, May 13-15, 2011

2. Outline and “Takeaways” • Introduction and Formation of the MD TFC Learning Collaborative (MDTFCLC) • Implementation Research Parallels • The Learning Collaborative at Work • Using Data for Decision Making: The Innovation Stage----Panel Review of Sample Data • Future Capacity

3. MD TFC Learning Collaborative • Baptist Family and Children Services • Catholic Charities -Center for Family Services • Kennedy Krieger Institute • Kennedy Krieger Family Center • Progressive Life Center • Williams Life Center • Comparable programs for comparison

4. MDTFCLC Demographic Profile (N) = 326

5. MD TFC Learning Collaborative • Forged around a commitment to outcomes management - Maryland TFC Coalition • Belief that outcomes can guide and inform client, program and system level effectiveness • July 2009--- five private Treatment Foster Care providers in partnership with the University Maryland’s Children’s Outcome Management Center (COMC) began focus on the use of data for program and practice improvement in a “learning collaborative” • Use of KIDnet and CANS

6. A Brief Retrospective • A Commitment to Outcomes with Sumone for Kids (MARFY Initiative - 17+ years ago) • Success and challenges of Sumone for Kids and CAFAS • Federal Science to Service Grant- partnership with local universities to bring research and outcomes to fields like child welfare (pilot project)- forged relationship with COMC • Increased emphasis on accountability through outcomes • Strategic Vision (program level and state wide) • State initiatives with CANS in Group Homes • MARFY TFC Outcomes Committee partnering with Innovations to foster discussions on Outcomes • State initiatives with CANS in TFC • Formation of MD TFC Learning Collaborative as TFC programs began adopting the COMC system

7. Implementation Research • MDTFCLC found parallels in the implementation process and work of Fixen, Naoom, Blase, Friedman and Wallace in their 2005 publication, Implementation Research: A Synthesis of the Literature • Implementation - “a specified set of activities designed to put into practice an activity or program” • EBP’s and programs are not effective unless they are fully implemented • “Performance implementation”---that which is effective is often elusive • Goal is to get beyond “paper implementation”--- to create actual change in practices that produces effects benefitting the clients • The MDTFCLC serves as the vehicle for programs to share, challenge and vet best practices in implementing KIDnet

8. Stages of Implementation • Exploration and Adoption: assess match; examine resources; make determination to adopt (or not) • Program Installation: putting policies in place/hiring • Initial Implementation: process begins (often the stage where inertia is lost) • Full Implementation: completely operational with all clients • Innovation: experience is gained; desirable changes that improve program effectiveness are “innovations” and are adopted as standard practice; fidelity adaptations are made to improve implementation. • Sustainability: goal is continued implementation in the face of an ever changing system • Fixen et al

9. Implementation: The Bear Den Challenge

10. Implementation of Health IT

11. The Learning Collaborative and Implementation • Parallels seen in provider experience • Overcoming the Bear Den Challenge: Together, through joint experience, we were able to anticipate and work through barriers to counter resistance • MDTFCLC became the vehicle to review and vet opportunities and challenges in implementation • Through a series of regular monthly meetings, the MDTFCLC began to discuss the challenges and barriers in implementation • Solutions and best practices emerged as the group partnered with COMC • Changes and priorities to the system were made using a priority and consensus matrix

12. MDTFC Best Practices • Initial implementation: Champions; User Groups; Creating buy-in with Leadership; and Development of Training for TFC parents on the CANS and KIDnet tools---created a remarkable culture shift in many organizations • Full Implementation: Webinars and use of the priority and consensus matrix became critical in creating sustainable buy-in • MDTFCLC is currently in the “Innovation Stage”---making fidelity adaptations and examining data for client and program level improvement opportunities

13. The Context of TFC in MD • Focus for youth and families: least restrictive and permanent environment; access to the family’s community; limited movement in placement; reduction of youth entering care; and reduction of youth in residential care. • Implications for TFC: average age of placement increases; increase in range and acuity of behavior; shorter lengths of stay; greater potential for disruption of placement. • Set the stage for examining those youth who exit our service to higher levels of care

14. Review of Aggregate Data: The Starting Point ---ROLES & LOR

15. Review of Aggregate Data:

18. Average Length of Stay

19. EXPLORATORY PROCESS • “Ok---I have good data---now what?” ---common question among providers • Our starting point in the exploration process--(1) What about those youth we are struggling to serve---the 25% that go to more restrictive settings? (2) How do we fair in relationship to state mandated outcome areas? • First question --- “What can we learn when looking at CANS domain level discharge scores?” --- led to further discussions and drill down

20. Drilling Down: CANS Domains

21. Family Living Situation Social Development Recreational Developmental Job Functioning Legal Medical Physical Sexuality School Behavior School Achievement School Attendance Life Functioning Domain

24. Observations at item level: Externalizing behaviors trending toward area of need---item level score of 2 or 3.

27. Suicide Risk Self Mutilation Other Self Harm Danger to Others Sexual Aggression Runaway Delinquency Judgement Fire Setting Social Behavior Child Risk Behavior

30. Observations • Both groups tend to be admitted with similar scoring profiles… • What is happening in the first few updates cycles? Why do scores tend to be the same for both groups until later in service, e.g. 4th update? • Externalizing behaviors: school behavior, oppositional behavior, impulsivity, risk activity are demonstrating areas of need with this “More Restrictive” cohort.

31. Observations continued • ALOS data suggest program decision making may be unique at certain points in the service delivery process. In other words, there is no defined approach to working with MR youth.

32. What’s Next? • Focus on the 25% of youth being discharged to more restrictive settings • More refined benchmarking: MDTFCLC beginning to look at those programs that have lower values in this area---what are they doing differently? • T-Test analysis to identify “means differences” that are truly statistically significant between both groups. • Consider regression analysis to understand the impact of key variables on discharge disposition, e.g. age, gender, race, diagnosis.

33. What’s Next? continued • Consider longitudinal trajectory analysis to understand the expected trajectory of youth with specific conditions. • Is this data a reflection of what is happening at the state level---youth traditionally served in residential setting transitioning to TFC? • Programs with significant wraparound services like PRP, crisis stabilization, mentoring---are they more successful? • Are fundamental changes in program preparation (for example: pre-service training, trauma treatment, substance abuse treatment and targeted recruitment) enough to promote stabilization?

34. Future Capacity and Q&A • MDTFCLC meets the third Thursday of every month from 10:30am - 12:30pm • To join the mailing list please contact meyerl@kennedykrieger.org