Continuing Development of Wraparound as an Evidence-Based Practice (Workshop A-11)

1. Continuing Development of Wraparound as an Evidence-Based Practice (Workshop A-11) Sixth California Wraparound Institute Tran M. Ly, Ph.D. Ming H. Lee, M.P.H. Christopher J. Jarosz, Ph.D. Department of Children and Family Services County of Los Angeles June 7, 2010

2. Workshop Outline • Introduction • Overview of Wraparound’s evidence base • Los Angeles County’s placement and cost outcomes studies • Program evaluations • Logic model • Methods • Implementing an evaluation • Review and wrap-up • Resources 2

3. Introduction 3

4. Learning Objectives • Be cognizant of the evidence base of Wraparound, including a recent example of a Wraparound evaluation in Los Angeles County. • Learn what is a program evaluation, and why and when to conduct one. • Recognize how the selection of study designs, measurement tools, and sampling strategies affect the types of questions asked and conclusions drawn in an evaluation. • Practice building a basic logic model as an initial step in devel-oping an evaluation plan. 4

5. The Presenters • Tran Ly has graduate training in psychology, education, and child development. She previously planned and conducted several studies involving families of children with various disa-bilities. • Ming Lee has training in biology, and graduate training in public health and biostatistics. He previously planned and conducted studies in perinatal health indicators and their influences on birth outcomes. • Christopher Jarosz has graduate training in the behavioral and biological sciences. He has planned and conducted behavioral, community health, and organizational research in the public and private sectors. 5

6. DCFS Research and Evaluation • Conduct program evaluations and applied research for DCFS. • Provide technical consulting to other DCFS programs. • Plan and conduct departmental surveys. • Serve as the department’s liaison with outside researchers. Website: http://dcfs.co.la.ca.us/rae/index.html 6

7. Overview of Wraparound’s Evidence Base 7

8. Residential Care • “Residential care” and “group home care” are terms sometimes used interchangeably within the research literature, with no consensus on a single definition. • Residential care is a term applied to a diverse array of services that are provided to youth living in such congregate care as apartments, emergency shelters, halfway homes, secure set-tings, and group homes. 8

9. Residential Care (continued) • Within residential care, placement instability is a common occurrence associated with various negative outcomes such as child behavior problems and juvenile delinquency. • Residential care placements in California were approximately 11% of the foster care population in 2000, but more than 50% of foster care expenditures. • Group homes provide continuous staff supervision, which can make them an expensive placement option. • Group home providers are categorized into Rate Classification Levels (RCLs) based on the level of care and services provided, with RCL 14 being the most restrictive. 9

10. Other Research Studies • Bruns and Suter (2010) updated their summary of experimental and quasi-experimental research studies on outcomes of the Wraparound process. • Nine controlled studies have been presented in peer-reviewed publications encompassing child welfare, juvenile justice, and mental health. • Bruns and Suter note that each study has methodological weak-nesses. • The studies considered in total indicate “superior outcomes for youth who receive wraparound compared to similar youth who receive some alternative service.” Eric J. Bruns and Jesse C. Suter. Summary of the Wraparound Evidence Base: April 2010 Update (2010). Resource Guide to Wraparound, Theory and Research, Chapter 3.5, National Wraparound Institute, 1-9. 10

11. Positive Outcomes • Fewer placement changes, and moves to less restrictive environ-ments. • Older youth more likely to be in a permanency plan at the time of follow-up. • Improved ability to externalize problems, especially with boys in the study. • Overall decline in Child and Adolescent Functional Assessment Scale (CFAS) scores. • Increased school attendance, reduction in school disciplinary actions, and increased grade point averages. 11

12. Los Angeles County’s Placement and Cost Outcomes Studies 12

13. Los Angeles County Wraparound Program • Wraparound provides Los Angeles County (LAC) with an alterna-tive to residential care for foster youth consistent with a national review panel report (Cole, 1998). • LAC implemented Wraparound in 1998 as a 10-child pilot study as an alternative to residential care. • The goal of the pilot was to explore the impact of Wraparound on reducing reliance on out-of-home care. • An analysis of the outcomes suggested that Wraparound youth were more likely to return and stay at home compared to youth in residential care. 13

14. LAC Wraparound Program (continued) • With closure of the County’s children’s shelter (McLaren) in 2003, Wraparound was identified as a primary mechanism for returning youth to their communities. • Referrals were initially slow due to the lack of understanding of Wraparound. • Another factor was the lack of research on its potential impact on youth, which at the time consisted of only a handful of controlled studies. 14

15. An Initial Study • A LAC study was conducted in 2004 to compare the outcomes of Wraparound graduates with youth who were discharged from Rate Classification Level (RCL 12 or 14) and went into less re-strictive placements. • RCL 12 and 14 were selected since youth must qualify at these levels for enrollment in Wraparound since they must be in or at imminent risk of placement in a residentially-based institutional setting. • The two groups were followed for 2-1/2 years using administra-tive data. 15

16. An Initial Study (continued) • The Wraparound graduates did substantially better than the youth who were discharged from RCL 12-14. • The Wraparound graduates had fewer subsequent out-of-home placements and therefore less financial cost to the County of Los Angeles. • A greater percentage of Wraparound graduates also exited foster care. 16

17. Wraparound Growth • Residential care in Los Angeles County has been reduced by over 60 percent since the 2004 study. • About 2,400 foster youth were in residential care in 2004, while the number declined to fewer than 1,000 youth in 2009. • Wraparound enrollment was less than 200 youth in 2003, which increased to over 1,200 youth by 2009. • The number of Wraparound providers increased from eight to 34 in 2006. • In 2009, the Wraparound program was expanded to accommo-date up to 4,200 youth. 17

18. Further Study • In 2009, we applied greater methodological rigor in attempting to replicate and expand upon the initial study. • The areas we focused on including using entire populations in the study, providing a more detailed set of selection criteria for the cohorts, and use of a matched-comparison design. • We found that the placement and cost outcomes of Wraparound versus RCL 12-14 were consistent with the findings of the initial study. Michael Rauso, Tran M. Ly, Ming H. Lee, and Christopher J. Jarosz (2009), Improving outcomes for foster care youth with complex emotional and behavioral needs: a comparison of outcomes for wraparound vs. residential care in Los Angeles County. Emotional & Behavioral Disorders in Youth, 9, 53-76. 18

19. Further Study (continued) • This year, we established new one-year cohorts of Wraparound graduates and youth discharged from RCL 12-14 to lower-level placements. • Our principal goal was to determine how consistent the results were with the previous studies. • We compared the placement outcomes and associated costs of the two cohorts as defined in the selection criteria shown on the next slide. • We hypothesized that Wraparound would continue to result in better outcomes (less restrictive placements) and lower costs to the system. 19

20. Selection Criteria 20

21. Outcome Measures • Case closures—did the case close during the 12 months after Wraparound graduation or discharge from RCL 12-14 to a lower placement level? • Placements—how many placements, including placement types, did the youth have during the 12-month period after graduation or discharge? • Costs—based on the placement types and time durations, what were the system costs during the 12-month period? 21

22. Demographics 22

23. Case Closures Percentages of Cases that Closed within 12 Months 23

24. Placement Outcomes Average Number of Days in Out-of-Home Placements Average Number of Out-of-Home Placements 24

25. Placement Outcomes (continued) Children Who Had None Versus at Least One Placement 25

26. Placement Distribution Distribution of Out-of Home Placement Types 26

27. Cost Outcomes Average Out-of-Home Placement Costs Statistically-significant, p < 0.01 27

28. Cost Distribution Distribution of Out-of-Home Placement Costs 28

29. Summary • The findings support previous cost impact analyses that Wrap-around is more cost-effective and has better outcomes compared to traditional residential care. • Relative to RCL 12-14 children, Wraparound children are more likely to have their cases closed within 12 months of graduation. 29

30. Summary (continued) • Wraparound versus RCL 12-14 children in the 12-month period after graduation had: • No or fewer out-of-home placements. • Placements, when they did occur, were often to less restric-tive environments and require fewer number of days. • Financial costs associated with placements were significantly less. • Despite recent improvements in placement and cost outcomes for the RCL children, the outcomes of Wraparound graduates remain significantly better. 30

31. Summary (continued) • By many indications, Wraparound has provided an effective mechanism for reducing the Los Angeles County’s reliance on out-of-home care. • Wraparound has a positive impact in that youth are living in less restrictive and more stable living environments. 31

32. Policy and Practice Implications • Wraparound provides an effective service delivery option for youth with behavioral and emotional problems. • Wraparound, endorsed as a high-priority strategy in Los Angeles County, was recently expanded to accept youth who are not currently in or at imminent risk of placement into higher levels of residential care. • A training model for service delivery is being developed for children’s social workers that shares many of the Wraparound values and principles. • Wraparound principles and practices will soon influence the nature of residential care in Los Angeles County from long-term placements to planned, short-term, and individualized interven-tions. 32

33. Limitations • The study design is a retrospective study, and therefore youth were not randomly assigned to groups. • We examined placement and cost outcomes for a period of one year. • In using administrative data, we do not have information on the behavioral characteristics or functioning of the comparison youth. 33

34. Future Research Directions • We will be conducting a third, one-year cohort study to further assess the year-to-year stability of placement and cost outcomes for Wraparound and RCL 12-14. • We are currently analyzing two-year cohort groups to determine longer-term outcomes for Wraparound and RCL 12-14. • We are refining the selection criteria for equivalence of emotional, behavioral, and mental health difficulties of the Wraparound and RCL 12-14 groups. 34

35. Program Evaluations 35

36. What is a Program Evaluation? An evaluation involves “carefully collecting information about a program or some aspect of a program… to make necessary decisions about the program.” “The type of evaluation… depends on what you want to learn about the program.” “More recently… evaluation has focused on utility, relevance and practicality at least as much as scientific validity.” (McNamara, 2006) 36

37. Possible Goals • Measure the impact of services on clients and other stakeholders. • Improve the effectiveness and reduce the costs of service delivery. • Verify if the program is operating to plan and if the plans need to be changed. • Facilitate meta-thinking about program goals, and if they are being met. • Promote decision-making based on data in responding to organiza-tional and community needs. 37

38. Possible Goals (continued) • Compare programs to determine which best meet the intended needs. • Establish a basis for duplicating or adapting programs for other venues and uses. Adapted from: McNamara, Carter (2006). Field Guide to Nonprofit Program Design, Marketing and Evaluation (4th edition). 252 pages. 38

39. Types of Program Evaluations • Program goals • Processes and process fidelity • Short- and long-term outcomes • Client satisfaction • Time, quality, and financial costs • Combination of two or more types 39

40. Stakeholders • Who are the formal and informal decision-makers? • Who is likely to benefit from the evaluation? • Who can facilitate or possibly hinder the evaluation? • What resources can the stakeholders offer for the effort? • Is there a shared vision and commitment to mutual goals? 40

41. Shared Vision? 41

42. Group Exercise #1 Let’s say you are asked to conduct an evaluation of Wraparound service delivery in your community. Identify the primary and secondary goals that you would want to achieve. What type of evaluation would you perform? Who would be the three most influential stakeholders in supporting or benefiting from the evaluation? 42

43. Logic Model 43

44. What Is a Logic Model? • A logic model is a useful tool for program development and evaluation and is usually part of an evaluation plan. • A logic model: • Describes the logical links between the program’s theories or assumptions, services, and outcomes desired. • Illustrates a sequence of cause-and-effect relationships. • While there many versions, a logic model is often a one-page graphical or schematic representation of what you do, why you do it, what you hope to achieve, and how you will measure that achievement. 44

45. Sample Template Program vision: Population served: Population needs to be addressed by services: * Include service assumptions 45

46. Building a Logic Model • Establish your program vision and goals or objectives. • Describe your target population that will be served, including the needs to be addressed. • Articulate the services and resources needed, including the assumptions you are making. • Identify your program outcomes and indicators. • Determine how you will measure these outcomes and indica-tors. 46

47. Vision Statement • A vision statement is a brief mission statement of what you hope for and want for the families and community to achieve in the future. • In general, vision statements are very broad and ambitious goals—the objective for the program is to be able to contribute towards that vision. • Vision statements are not necessarily measurable—example, all children are “ready” for school by age 6. 47

48. Target Population and Needs • In building a logic model, you will need to identify your target population. • Describe its key characteristics which may include age, gender, ethnicity, educational status, economic status, and personal risk factors. • Be as specific as possible in identifying the people who will or have received the services to ensure that intended outcomes are meaningful and appropriate. • It is important to describe the population needs that your program intends to address—what will your services help them to do? 48

49. Service Needs and Assumptions • In continuing the development of the model, briefly describe the services that your families will receive. • Describe why you think the services will lead to the desired outcomes. • What assumptions are you making? • Are the services informed by evidence-based practice? • When possible, use theory and research to guide the develop-ment of your logic model. 49

50. Outcomes versus Indicators • An outcome is a change that is likely to take place as a result of the target population’s participation in your program. • Three types of outcomes are: long-term, intermediate, and short-term. • Indicators tell you whether an outcome has been achieved. • Indicators are concrete specific descriptions of what you will measure. • Also known as performance targets, indicators can specify a targeted number or percentage of achievement. 50