Children & Families Services Review (CFSR) Presented By: Alba Quiroz Garcia, CDSS July 31, 2014

1. CFSR 2016 Children & Families Services Review (CFSR)Presented By: Alba Quiroz Garcia, CDSSJuly 31, 2014 Material from the May 2014 Administration for Children and Families, CFSR Briefing for States, Seattle, WA

2. Recap, what is the CFSR? • The federal government created the CFSR in response to the 1997 Adoption and Safe Families Act. It is an assessment of a state’s performance related to child welfare. • Indicators and standards are used along with information from a sample of cases to inform the federal government about determinations of whether a state is in substantial conformity with certain safety and permanency outcomes and to assess each state’s ability to achieve Safety, Permanency, and Well-being for children and families.

3. Why a CFSR information session in 2014? • With Rounds 1 and 2 of the CFSR completed, Round 3 is near and modifications are occurring presently. • This session is to help inform the field about plans for statewide data indicators, national standards, and the case review process that will be used in the next round of the CFSR.

4. Timeframe for California’s CFSR Round 3 • California’s Round 3 CFSR is scheduled for February through September 2016. • Some states will begin Round 3 of the CFSR as early as Fall 2014. This means that changes still pending for the redesign of the indicators need to be finalized rapidly.

5. Why CFSR modifications? • With regard to data indicators & standards, the CB heard feedback from the field that the process could be improved by using different methods and measures that better reflect state practices and improvements. The challenges with prior indicators & measures ranged from methodological issues, ease of interpreting composite numbers, and concerns about comparability of state data. • Also, the CB now has a greater reliance on entry cohorts, fewer & simpler indicators, and they are dropping composites in favor of single item indicators. • Release of information memorandum to states on establishing and maintaining a Continuous Quality Improvement (CQI) system.

6. CQI • California is seeking to submit a letter of intent to the CB to use our case review process as part or the next CFSR. • For the state’s case review process to be considered, the following must be met: • Use the Federal Onsite Review Instrument (ORI). This tool collects and documents outcomes to Safety, Permanency and Well-Being. • Submit a letter of intent by July 2015 to the CB. The letter must be accompanied by a plan that meets all requirements outlined in the federal register (released April 2014). • Be able to replicate a sample from a recent AFCARS period. • Use a sample period of April 1-Sept 30, 2015 . • The CB is not certifying the CQI process, rather they are approving or not approving our state’s case review process. Approval status is expected as early October 2015, but no later than December 2015.

7. CQI Continued… • If the state is approved, there is no limit on how many cases the state can review during the timeframe. • However, if a state chooses or has to do a traditional CFSR, the CB will only review 65 cases total (statewide). • For Round 3, case reviews do not need to be done in pairs as was the case in the previous two CFSR Rounds.

8. Systemic Factors and Statewide Indicators • The seven (7) Systemic Factors and Statewide Indicators remain present for CFSR Round 3. • Both will be utilized to determine if the state is in substantial conformity and meeting federal targets for Safety, Permanency and Well-Being.

9. Systemic Factors • There will be a focus on systemic factor functioning in Round 3. In CFSR Rounds 1 and 2 the focus was on having “requirements in place.” • Seven (7) Systemic Factors: • 1. Statewide Information System • 2. Case Review System • 3. Quality Assurance System • 4. Staff and Provider Training • 5. Service Array and Resources Development • 6. Agency Responsiveness to the Community • 7. Foster/Adoptive Parent Licensing, Recruitment & Retention

10. Systemic Factors Continued… • The CB can make a determination about substantial conformity for 6 of the 7 systemic factors using the data provided in the Statewide Assessment, with the exception of Service Array. Service Array cannot be determined through the statewide assessment and will require stakeholder interviews (there is no way that states can get around this). • Since the CB can make determinations about substantial conformity for most, but not all systemic factor, there is a possibility of reduction in scope and number of stakeholder interviews.

11. Systemic Factors Continued… • At least 95% of applicable cases must meet the target to be determined “In Substantial Conformity.” • For Round 3, there is a reduction in Systemic Factor Items: • In CFSR Round 1 and 2 there were 22, for Round 3, there are 18.

12. Six (6) statewide data indicators will serve as basis for determining whether or not states meet national standards. • Two (2) for Safety • Maltreatment • Re-Report of Maltreatment • Four (4) for Permanency • Percent of the entry cohort that achieves permanency in 12 months. • Percent of those children who subsequently re-enter care. • Percent of children and youth who have been in care for 2 years or longer who achieve permanency in the next 12 months. • A placement stability indicator for children entering care

13. Safety Indicators-Children are protected from abuse and neglect. 1. Maltreatment in Foster Care (FC) This asks, “Of all children in care during a 12-month period, what is the rate of victimization per day of FC?” • The denominator looks at all children in care during a 12-month period and looks at the total number of days these children spent in care as of the end of the year. • The numerator looks at those same children who are in the denominator and counts the total number of substantiated or indicated reports of maltreatment (by any perpetrator) that occurred during a foster care episode. • Includes all perpetrator types and controls for amount of time spent in FC.

14. Safety Indicators-Children are protected from abuse and neglect. • II. Re-Report of Maltreatment • Of all children with a screened-in report of alleged maltreatment in a 12-month period (regardless of disposition type), what percent had another screened-in report within 12 months of their initial report? • Expanded timeframes: from 6 months to 12 months • Rather than limit the indicator to include victims only, the CB includes all screened-in reports of alleged maltreatment that have reached disposition.

15. Safety Indictors Continued… • Rather than limit the indicator to substantiated cases only, the CB is expanding this indicator to count all children with screened-in reports of alleged maltreatment. • Why? The CB believes that looking at repeat reports, regardless of disposition type, is a viable measure of an agency's’ attempts to prevent maltreatment, based on research indicating that families with screened-in but unsubstantiated reports are at a high risk of re-report. • Using all screened-in reports rather than limiting the [indicator] to substantiated cases makes states more comparable to each other and even against their own performance over time, when considering the growing implementation of differential response (DR), where a substantial percentage of cases may not receive a formal investigation.

16. Permanency Indicators-Children have permanency and stability in their living situations. • Rather than have composites made up of multiple measures, each of these indicators will be assessed individually on a particular cohort of children in FC for the initial determinations of compliance. In some cases, for purposes of monitoring during the PIP, the CB will treat another indicator as a companion indicator (this is new).

17. I. Entry Cohort Permanency in 12 Mo. • This first indicator asks, “Of all children who entered care in a 12-month period, what percent discharged to permanency within 12 months?” • The difference here is the attention on permanency, so this means that while the majority of discharges in a short time frame will be reunification, the CB will also count exits to living with a relative, adoption, or guardianship as a success on this indicator. • In the last CFSR, the CB had a similar measure that looked at entry cohorts, but was limited to first-time entries. The CB is now counting all entries during the year and expanded the timeframe from a 6 to12 month cohort. The past CFSRs also only counted reunifications. • By expanding the denominator to a full year, the CB can achieve more stable and reliable estimates, which is important because those children achieving permanency on this indicator will also serve as the denominator for the re-entry indicator.

18. II. Re-Entry Permanency Indicator • The second indicator looks at children in the entry cohort who discharged to permanency within 12 months, and tracks them for another 12 months after exit to see whether they re-enter foster care. • It excludes exits to adoption because they cannot be as reliably tracked. • For all exits to reunification, living with a relative, or guardianship the CB wants to be sure they were not done without adequate planning or services to support a permanent placement, resulting in a greater risk of re-entry to care. • This indicator is limited to the entry cohort, and does not include all exits to reunification that occur during the year, as it did before. The denominator now including exits to guardianship and this represents another change.

19. III. Children in Care 2+ Yrs.Permanency 12 Mo. • Looks at the cohort of children in FC on the 1st day of a year who already have been in FC for 2 years or longer. • The CB is interested in knowing what proportion of this cohort achieves permanency within 12 months of the 1st day? So the focus is on getting these children and youth to permanent homes. Again, the CB will count all discharges to reunification, living with a relative, adoption, or guardianship as a success.

20. IV. Placement Stability • Has been refined to have one indicator instead of three. • The proposed indicator looks at the rate of placement moves in the first year of FC, for the entry cohort. • Similar to the way the CB calculates maltreatment in FC, the denominator for this indicator is the sum of all days children spent in FC, to control for various lengths of time in care. For the numerator, the CB looks at how many times children were moved from one placement to another, while they were in care. • This is actually cumulative across placement episodes in the same year. The CB will NOTcount their initial placement as a move, but any change is counted. While the CFSR Round 2 indicators did not differentiate children who moved twice from those who had moved more times, this indicator counts every move to accurately capture the rate of movement, rather than the number of children affected.

21. National Standard-Methodology • It is set at the national observed performance for each statewide indicator. • Example: For most of the indicators it is a simple formula: the number of children in the nation with the outcome divided by the number of children in the nation eligible for the outcome. • In reviewing the permanency-by-12-months indicator... Assume we have 220,000 children in the nation that achieve permanency by 12 months out of 500,000 children in the nation that were eligible. So quite simply, the national standard for this indicator is 44%. This will often be similar to the national average. • According to CB analysis, most states will need to improve on at least one indicator.

22. Data Sources • There is continued reliance on AFCARS and NCANDSdata. The CB will also link the two using the child ID. • Performance on these indicators will be provided to states via data profiles, to assist in monitoring Program Improvement Plans (PIP) that may address one or more of these indicators.

23. Data Sources Continued • One criticism of the measurement techniques used in prior CFSR rounds was a heavy reliance on exit cohorts and point-in-time cohorts, rather than using prospective entry cohorts. Exit cohorts and cross-sectional cohorts can distort system performance outcomes because they represent biased subsets of all children who are served by the agency in a year. • For example, looking only at children who exit foster care in a given year ignores those children who did not leave. Looking only at those in foster care at the end of the year (or at any point in time) biases the sample to include more children with long lengths of stay in foster care. Entry cohorts can avoid these problems and provide a more fair assessment of overall system performance. Thus, the CB has shifted toward a greater focus on entry cohorts.

24. Foster Care Exclusions • Youth who are already age 18 or older at the start of the indicator (that is, at the age of entry into care or who are already 18 at the outset of an indicator) will be excluded. • Although CA and other states do provide FC to youth age 18 and older, there is no consistent inclusion of this population across states. • Any FC episode lasting less than 8 days will be excluded from these indicators. These very short-term placements are more prevalent in some states than in others, so this is another way to eliminate a bias and make states more comparable in those outcomes.

25. Measuring States’ Performance-Methodology • The goal of risk adjustment is to minimize the differences in outcomes due to factors over which the state has little control. It takes into account the characteristics of each state’s case mix, something that has been a concern for stakeholders and people in the field for many years. • For example, a state’s children may be at greater risk for a poor outcome simply because of their age or history, which might be a characteristic of the child population in that state. • By performing risk adjustment, the CB can take into account factors that differ across the states and can influence the outcomes, regardless of the quality of care a state provides.

26. Measuring States’ Performance-Risk-Adjustment Possible Variables • Age of child, as an example, is a risk adjustment variable that the CB is considering, others being considered by the CB are: child’s gender/sex, number of prior removals, state FC entry rate. • The CB will finalize the variables after they receive and review public comments.

27. Measuring States’ PerformanceRisk-Adjustment Continued… • Risk Adjustment—Maltreatment in Foster Care Example • Use maltreatment in FC, hypothetically the average age of a child in FC across the country is 10 years old. In state A, the average age is 5 years old. State A can perform worse than other states simply because of the age of its children, putting them at more risk. Since we know from field analysis of NCANDS data and from the vast literature on maltreatment, that younger children are more at risk for maltreatment, regardless of the quality of care being provided….So in this case, risk adjustment would control for the age of the children in the state compared to the average age of the children across the nation.

28. Measuring States’ Performance—Confidence Intervals • For example, 45%, plus or minus 2%, would represent the interval estimate. The CB can be 95% confident that a state’s overall performance in the long run lies somewhere between the lower and upper limit of the state’s confidence interval. • Example: State A, their risk-adjusted performance was 45%, but because of some error and uncertainty, the CB can be 95% confident that their true performance is somewhere between 43% and 47%.

29. Measuring States’ Performance-Categorizing States • So the confidence interval is used to help determine where a state is relative to the national standard. The CB will categorize states into three groups: • No different than the nation, • Higher than the nation, or • Lower than the nation • States that fail to meet the national standard will be required to include that indicator in a PIP. • Regardless of which indicators are included in the state’s PIP, the CB will provide each state with a data profile that includes information on the state’s performance on all of the indicators.

30. Companion Indicators • In addition to setting targets for all statewide data indicators that may be included in a PIP, there will also be thresholds established for companion indicators. • The companion indicators are entry cohort permanency in 12 months and the re-entry indicator…because efforts to improve on the entry cohort permanency in 12 months could have the consequence of increasing the rate of re-entry, states should not get worse than the threshold for re-entry.

31. Resources • http://www.childsworld.ca.gov/PG1520.htm • https://training.cfsrportal.org/resources