Wraparound Fidelity Monitoring in the Massachusetts CBHI Summary of 2011 Results August 12, 2011

1. Wraparound Fidelity Monitoring in the Massachusetts CBHI Summary of 2011 Results August 12, 2011 Eric Bruns and April Sather University of Washington Wraparound Evaluation and Research Team ebruns@uw.edu / sathea@uw.edu With thanks to Andrea Gewirtz and Melissa King

2. Purpose for this Meeting Very quick review of wraparound fidelity assessment and research Review data from Massachusetts fidelity monitoring Discuss what has been found and what it means

3. Wraparound Competence:What do we want to measure? Wraparound Principles Family voice and choice Team-based Natural supports Collaboration Community-based Culturally competent Individualized Strengths based Persistence Outcome-based Walker, Bruns, Adams, Miles, Osher et al., 2004

4. Implementing the practice model:The Four Phases of Wraparound Wraparound Competence Phase1A Engagement and Support Phase1B Team Preparation Phase2 Initial Plan Development Phase3 Implementation Phase4 Transition Time

5. Phase 1 :Engagement and Team Preparation Care Coordinator & Family Partner meets with the family to discuss the wraparound process and listen to the family’s story. Discuss concerns, needs, hopes, dreams, and strengths. Listen to the family’s vision for the future. Assess for safety and make a provisional safety plan if needed Identify people who care about the family as well as people the family have found helpful for each family member. Reach agreement about who will come to a meeting to develop a plan and where we should have that meeting. Phase 1 A and B

6. Phase 2: Initial Plan Development Conduct first Care Planning Team (CPT) meeting with people who are providing services to the family as well as people who are connected to the family in a supportive role. The team will: Review the family vision Develop a Mission Statement about what the team will be working on together Review the family’s needs Come up with several different ways to meet those needs that match up with the family’s strengths Different team members will take on different tasks that have been agreed to. Phase 2

7. Phase 3: Plan Implementation Based on the CPT meetings, the team has created a written plan of care. Action steps have been created, team members are committed to do the work, and our team comes together regularly. When the team meets, it: Reviews Accomplishments (what has been done and what’s been going well); Assesses whether the plan has been working to achieve the family’s goals; Adjusts things that aren’t working within the plan; Assigns new tasks to team members. Phase 3

8. Phase 4: Transition There is a point when the team will no longer need to meet regularly. Transition out of Wraparound may involve a final meeting of the whole team, a small celebration, or simply the family deciding they are ready to move on. The family we will get a record of what work was completed as well as list of what was accomplished. The team will also make a plan for the future, including who the family can call on if they need help or if they need to re-convene their team. Sometimes transition steps include the family and their supports practicing responses to crises or problems that may arise Phase 4

9. Wraparound Fidelity Index, v.4 Items on the principles and core activities, organized by the 4 phases of wraparound Engagement: Did you select the people who would be on your youth and family team? Principle = Team based Planning: Does the plan include strategies for helping your child get involved with activities in the community? Principle = Community based Implementation: Does the team evaluate progress toward the goals of the plan at every team meeting? Principle = Outcome based Transition: Will some members of your team be there to support you when formal wraparound is complete? Principle = Persistence

10. Team Observation Measure Consists of 20 items, with two items dedicated to each of the 10 principles of wraparound. Each item consists of 3-5 indicators of high-quality wraparound practice as expressed during a care planning team meeting. Internal consistency very good Inter-rater reliability found to be adequate (Average 79% agreement for all indicators) Correlates well with WFI scores In previous studies and evaluations…

11. Document Review Measure Mass version consists of 26 items Each wraparound principle linked to several items Scale = 0-3, with criteria for each point on the scale Source material = documentation (electronic or paper) related to youth’s wraparound process Strengths, needs, culture discovery documentation Wraparound plan of care Safety plan Transition plan Progress notes

12. Measures of the WFAS seem to tap into the same constructs Mean WFI-4 and TOM total scores by site in one state’s QI project

13. WFAS Measures seem to be related to what we would expect Families who experience better outcomes have staff who score higher on fidelity tools (Bruns, Rast et al., 2006; Effland, McIntyre, & Walton, 2010) Wraparound initiatives with positive fidelity assessments demonstrate more positive outcomes (Bruns, Leverentz-Brady, & Suter, 2008) Wraparound initiatives with better support to implementation have higher fidelity scores

14. What does it take to get high fidelity scores? Training and coaching: Found to be associated with gains in fidelity and higher fidelity Community, Fiscal, and Service system support: Better developed supports = fidelity scores Sticking with it: Fidelity scores tend to increase over time and with persistence of effort But… they can also backslide…

15. Caregiver WFI Fidelity over time in NV Bruns, Rast, Walker, Peterson, & Bosworth (2006). American Journal of Community Psychology.

16. What can we do with the data? Measurement as communication What are we attempting to do for families? How well are we stacking up against benchmarks? What is holding us back? How well are we doing over time? What is associated with positive outcomes in Massachusetts?

17. Measurement for quality improvement Examine data at the program level to discuss what is and is not happening consistently What can be reinforced in individual and group supervision? What are the team, program, and resource barriers? In what areas do we need additional training? At the community or system level by collaborative groups, along with reports of team, program, and resource barriers reported up from CSAs or programs

18. What kind of data get used? • What kind of data? • Item scores – Ns who endorse each response option (Yes, No, Somewhat) • Members on teams • Specific plan of care elements • Qualitative feedback on open-ended items

19. Part 2: Results from Massachusetts Scores on the WFI, TOM, and Mass-DRM

20. Total scores on the WFAS instruments

21. Correlations between WFI, TOM, and DRM across Massachusetts CSAs 2 0 1 0 • WFI – TOM = .121 • WFI – DRM = -.473 (p<.001) • DRM – TOM = .012 • WFI – TOM = -.256 • WFI – DRM = .326 (p<.05) • DRM – TOM =.168 2 0 1 1

22. WFI-DRM association at a site level 2010 – DRM / WFI - 2011

23. WFI-DRM association at a site level 2011 – DRM/TOM

24. Wraparound Fidelity Index Massachusetts Overall

25. WFI-4 Total Fidelity ScoresAll CSAs 2010 N = 637 / 2011 N = 587

26. WFI Fidelity by PhaseAll CSAs 2010 N=637 / 2011 N = 587

27. WFI Fidelity by PrincipleAll CSAs 2010 N=637 / 2011 N = 587

28. Item Scores: Engagement All CSAs

29. Item Scores: Planning All CSAs

30. Item Scores: Planning All CSAs

31. Item Scores: Implementation All CSAs

32. Item Scores: Implementation All CSAs

33. Item Scores: Transition All CSAs

34. WFI – Summary • Overall: Significantly higher than national mean • Overall, no change from 2010 • Relative strengths: • Engagement, Planning phases • Family voice and choice, team based, collaborative, Individualized • Initial engagement, family selecting team, crisis planning, vision/mission statements, balance of formal and informal supports • Relative weaknesses: • Natural supports, involvement of friends/advocates, involvement in the community • Youth voice, youth friendships, youth involvement in community

35. WFI – Summary of trends from 2010 • Areas of improvement from 2010: • Crisis identification and plans • Transition planning • including prep for transitions from wrap and within wrap • Areas of decline: • Family voice and choice – selecting team, perception of family influence • Youth friends, youth voice, youth problem solving • Perception that wrap could be discontinued before family ready

36. Team Observation Measure Mass CBHI overall

37. TOM Total Fidelity ScoresAll CSAs 2010 N=285 / 2011 N=658

38. TOM Fidelity by PrincipleAll CSAs 2010 N=285 / 2011 N=658

39. TOM Item & Indicator ScoresAll CSAs

40. TOM Item & Indicator ScoresAll CSAs

41. TOM Item & Indicator ScoresAll CSAs

42. TOM Item & Indicator ScoresAll CSAs

43. TOM Item & Indicator ScoresAll CSAs

44. TOM Item & Indicator ScoresAll CSAs

45. TOM Item & Indicator ScoresAll CSAs

46. TOM Item & Indicator ScoresAll CSAs

47. TOM Item & Indicator ScoresAll CSAs

48. TOM Item & Indicator ScoresAll CSAs

49. TOM – Summary • Overall: Significantly higher than national mean • Significant improvement from 2010 • Relative strengths: • Nearly all principles. Particular strengths in: • School/Agency reps present, effective teams and facilitation, facilitator prep, Creative/individualized services, services for all family members, crisis plans • Relative weaknesses: • Youth involvement/presence at meetings • Natural supports present and involved • Flex resources available and used when needed

50. TOM – Summary of trends from 2010 • Areas of improvement from 2010: • Facilitator prep, team process • Flexible resources available • Balance of informal/formal services/supports • Crisis planning and plans • Use of objective measures and following up • Tying strengths to planning • Areas of decline: • Youths being asked their priorities • Youths present