WFI-4 Scale Scores: 10 Principles of Wraparound

1. The Wraparound Fidelity Assessment System Overview and Results of Pilot Testing Eric J. Bruns, April Sather, Kelly Hyde, Janet Walker & the Wraparound Evaluation and Research Team National Wraparound Initiative www.rtc.pdx.edu/nwi Summary and Conclusions The results presented at this poster demonstrate some strengths and weaknesses of the measures of the WFAS as instruments for assessing implementation fidelity for the wraparound process. For the WFI-4, strengths include its overall internal consistency, adequate inter-rater reliability, and ability to discriminate between wraparound and non-wraparound (or pre-wraparound) communities. Weaknesses include moderate to low internal consistency scores for subscales such as the 10 wraparound principle scores (likely resulting from small n of items and heterogeneity of factors being assessed). The WFI-4 also continues to show restriction in its range of scores that is typical of self-report measures. Data from the TOM are preliminary but show promise: Scores demonstrate good psychometrics and internal consistency, as well as adequate inter-rater reliability. TOM data also have been found to correlate significantly with WFI-4 scores (for the CG and Y forms). More complete validation of the TOM will require discriminant validity tests such as those completed for the WFI-4, and additional inter-rater reliability tests. Data from the CSWI indicate it performs well as a method for assessing readiness or support for implementing the wraparound process. An important next step will be to assess the relationship between CSWI scores and fidelity as assessed by the other measures of the WFAS. Finally, though previous studies have found association between wraparound fidelity instruments and outcomes (Bruns et al., in press), all measures of the WFAS still need to be better studied with respect to prediction of outcomes experienced by children, youth, and families served via the wraparound process. Introduction Wraparound is a care coordination process that has evolved over the past 20 years through efforts to help children with complex emotional and behavioral needs function more effectively in home, school, and community settings. As wraparound has become more widely implemented, programs, communities, and states have been increasingly interested in measuring implementation fidelity (Bruns et al., 2004; Bruns et al., in press; Walker & Bruns, 2006). The Wraparound Fidelity Assessment System (WFAS) is a multi-method approach to assessing the quality of individualized care planning and management for children and youth and their families. The instruments of the WFAS can be used individually or in combination with one another as research, evaluation, or quality assurance tools. WFAS instruments include: 1) The Wraparound Fidelity Index, version 4 (WFI-4). 2) The Team Observation Measure (TOM) 3) The Document Review Measure (DRM) 4) The Community Supports for Wraparound Inventory (CSWI) This poster presents a summary of psychometric data collected to date for three of the four instruments that comprise the WFAS. WFI-4: Mean Total Scores TOM: Distribution of Total Scores(N=37) Mean = .63; SD = .18; Range = 22-93 WFI-4 Discriminant validity • TOM: Reliability Results • Cronbach’s alpha (20 items): .866 • Inter-rater reliability (indicators): 79% • Inter-rater reliability (items): 72% Total Scores for Wrap vs. Non-Wrap Sites Method and Sample Data presented was compiled from pilot sites who requested use of WFAS instruments. Local users were provided with the instrument(s), a User’s Manual, and instructions for training data collection staff to criteria. For the WFI-4, data collection staff were provided with pre-recorded WFI-4 interviews and instructed to score these practice interviews per instructions and scoring rules presented in the scoring manual. For the TOM, such training to criteria was completed via video recorded team meetings provided on DVD. For the DRM, training of reviewers to criteria was conducted using redacted case file documents. Thus far, WFI-4 data have been compiled for N=796 youth in 15 sites across 12 states nationally, including 642 completed wraparound facilitator (WF) forms, 652 Caregiver (CG) forms, 308 Youth (Y) forms, and 260 Team Member (TM) forms. In addition, TOM data on N=37 team meetings were collected and compiled from 3 collaborating sites. CSWI data were collected from N=259 across 7 sites. DRM data have been received but remain in a preliminary phase of pilot testing. Results presented in this poster are a summary of descriptive and psychometric analyses that have been completed. Youths in the WFI-4 sample were 14.4 years old on average (SD = 3.5, range 1-22) and were 61% male. Fifty-eight percent of caregivers were biological or adoptive parents, 14% grandparents or other relatives, and 20% foster parents. Over half (57%, n=362) of the youths currently were in or had previously been in state custody. Youths had been enrolled in wraparound 10.23 months on average (SD = 8.53, range 0-60 months). ** ** * ** CSWI: Internal consistency(N=279) **p < .001 *p < .05 WFI-4: Internal Consistency (Cronbach’s alpha) Supporting Technologies Currently, we are working with Accountability Solutions (www.accountabilitysolutions.com) to develop a web-based resource, the WFAS Online Data Entry and Reporting System, that will allow users to enter their data via a web portal that will compile data from the WFAS instruments into one database, regardless of how many people are entering data or where they are located. This system will allow the user sites to create a range of reports or export the data for their own analyses. The system will be available in Spring 2008. * Youth form has fewer items (N=32 total) WFI-4 Scale Scores: 10 Principles of Wraparound Literature cited Bruns, E.J., Suter, J.C., Burchard, J.D., Leverentz-Brady, K. & Force, M. (2004). Assessing fidelity to a community-based treatment for youth: the Wraparound Fidelity Index. Journal of Emotional and Behavioral Disorders, 12, 69-79. Bruns, E.J., Leverentz-Brady, K.M., & Suter, J.C. (in press). Is it wraparound yet? Setting fidelity standards for the wraparound process. Journal of Behavioral Health Services and Research. Walker, J.S. & Bruns, E.J. (2006). Building on Practice-Based Evidence: Using Expert Perspectives to Define the Wraparound Process. Psychiatric Services. CSWI: Inter-rater reliability For more information To learn more about the instruments of the WFAS, download a copy of this poster, or inquire about becoming a collaborating site, please visit the Wraparound Evaluation and Research Team at http://depts.washington.edu/ wrapeval, or contact April Sather at sathea@u.washington.edu / 206-685-2310. Acknowledgments Development of measures of the WFAS is supported by the National Institute of Mental Health (R41MH077356). The authors would like to thank all the communities participating in the pilot testing and validation process for the WFI-4. Development of the WFI-4 was supported through a task order from the Child, Adolescent and Family Branch of the SAMHSA Center for Mental Health Services. To learn more about the National Wraparound Initiative, check out our website at www.rtc.pdx.edu/nwi. * Note: Limited to respondents with no missing data Percent of possible total score