The Child and Family Services Review Composite Scores: A “Great Start”

1. The Child and Family Services Review Composite Scores:A “Great Start” Barbara Needell, MSW, PhD Center for Social Services Research University of California at Berkeley *Schuerman, J., & Needell, B. (2009). The Child and Family Services Review Composite Scores: Accountability off the Track. Chicago: Chapin Hall at the University of Chicago http://www.chapinhall.org/research/report/child-and-family-services-review-composite-scores-accountability-track The Performance Indicators Project is a collaboration of the California Department of Social Services and the University of California at Berkeley, and is supported by the California Department of Social Services and the Stuart Foundation

2. CFSR Overview • Review and evaluation of state foster care and adoption services federally funded by Titles IV-B and IV-E of the Social Security Act • Carried out by the Children’s Bureau of the Administration on Children and Families (ACF) • GOAL: Evidence-based accountability and performance improvement of state child welfare systems

3. The CFSR Process

4. Flaws in Performance orFlawed Process? • States not in “substantial conformity” must develop a Program Improvement Plan (PIP) • To date, no state has passed the CFSR at the initial stage – every state has had to develop a PIP • Financial penalties may be incurred if the PIP is not successfully completed But there are weaknesses in the CFSR process (we will focus on the National Standards and composite scores)

5. State Demographic Variation

6. Systemic State Differences in Caseload Inclusion Criteria

7. Data Quality Weaknesses • Database (AFCARS) not designed to measure longitudinal performance, but ACF attempts to stitch together files to construct a pseudo-longitudinal database • Prone to record linkage errors and still does not include all data needed to track performance over time • Data is not of adequate quality to warrant the severity of consequences

8. Practice and Policy Conflicts between Measures

9. Equal Count/Weighting of States Despite Differences in Child Population Size CA and VT are weighted equally, although there are 60x as many children served in CA foster care as in VT foster care CA VT

10. Statistical Method Used is Complicated and Not Superior to a Simpler Approach • ACF uses principal components analysis (PCA), which is often used to simplify a large number of similar variables • This statistical manipulation does not necessarily produce a better result than a more straightforward approach, such as a simple average • Because the composite measure is an abstraction, it is not clear what actions should be taken to improve scores and may even encourage gaming of the system

11. Arbitrary and statistically inappropriate decisions in using PCA

12. Arbitrary National Standard and Adjustment • National standard set at 75th percentile with no clear justification • 75th percentile is actually that for the normal curve, a theoretical statistical distribution • State outcome data are not necessarily normally distributed

13. Flawed Method to Develop Minimum Improvement • CFSR2 requires a percentage improvement by states, based on their baseline performance in a given year • Amount of required improvement determined using a complex calculation involving sampling error, which is not justifiable statistically • The national standards largely use “p”-- the percent of time an event happens, which should not be used in this context

14. Example

15. Summary/Conclusion • Flaws and issues with the national standards and composite scores are substantial enough to take a fresh look at “substantial conformity” • Note: We have not discussed other areas of the CFSR at all today—the State Data Profile, the Case Record Reviews, the Stakeholder Interviews, etc—and all of these may also be in need of review/reform • Not clear that reasonable outcome indicators can be developed from the currently available AFCARS

16. Summary/Conclusion • A fully longitudinal national foster care database is necessary to adequately measure performance (e.g., State Data Center Foster Care Database, California Child Welfare Performance Indicators Project Analytic Assistance for Child Welfare…in North Carolina.) • We have the ability NOW to reform the CFSR process, including the performance measures that come from administrative data • The development of a functional CFSR process can encourage and foster true improvement in state child welfare systems

17. Barbara Needell bneedell@berkeley.edu 510 290 6334 CSSR.BERKELEY.EDU/UCB_CHILDWELFARE Presentation Developed by Christine Wei-Mien Lou