1 / 57

Module 2 Slide deck: Lessons in using (and misusing) California’s Child Welfare data

Module 2 Slide deck: Lessons in using (and misusing) California’s Child Welfare data. Instructor Notes for Module 2. This module exposes students to data concerning California’s child welfare system, its purpose is to:

paulos
Download Presentation

Module 2 Slide deck: Lessons in using (and misusing) California’s Child Welfare data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Module 2 Slide deck:Lessons in using (and misusing) California’s Child Welfare data

  2. Instructor Notes for Module 2 • This module exposes students to data concerning California’s child welfare system, its purpose is to: • Provide a broad overview of California’s child welfare system through visual displays of data • Introduce state and federal child welfare indicators for tracking agency performance (with a more technical module for optional use) • Promote critical thinking in the context of basic statistical concepts through the review of popular press examples based on actual child welfare data Module 2: Lessons in Using Data

  3. Module 2, Section 1 Understanding California’s Child Welfare System through Data Module 2: Lessons in Using Data

  4. The “big” picture in 2011… • 9,992,333 children under the age of 18 • 471,790 children reported for maltreatment (47.2 per 1,000 children) • 90,472 children with a substantiated allegation (9.1 per 1,000 children) • 31,431 children entered foster care (3.1 per 1,000 children) • On any given day, roughly 59,484 children in foster care (6.0 per 1,000 children) Module 2: Lessons in Using Data

  5. The Iceberg Analogy Maltreated children known to child protective services Maltreated children not knownto child protective services Module 2: Lessons in Using Data

  6. Module 2: Lessons in Using Data

  7. Module 2: Lessons in Using Data

  8. Module 2: Lessons in Using Data

  9. Module 2: Lessons in Using Data

  10. Module 2: Lessons in Using Data

  11. Module 2: Lessons in Using Data

  12. Module 2: Lessons in Using Data

  13. Module 2: Lessons in Using Data

  14. Module 2: Lessons in Using Data

  15. Module 2, Section 2 Tracking child welfare performance through federal and state outcome measures Module 2: Lessons in Using Data

  16. Trends over the last two decades • Increased (and improved) data collection • Increased emphasis on accountability • Observed across government agencies • Shift from measuring processes, to performance outcomes • What matters is where you end-up…promotes innovation • But what “outcomes” should we measure? And how can we best “measure” these outcomes? Module 2: Lessons in Using Data

  17. Lesson #1: Any One Measure Will Not be Enough… rate of referrals/ substantiated referrals home-based services vs. out of home care reentry to care permanency through reunification, adoption, or guardianship counterbalanced indicators of system performance use of least restrictive form of care length of stay positive attachments to family, friends, and neighbors stability of care Slide Source: Usher, C.L., Wildfire, J.B., Gogan, H.C. & Brown, E.L. (2002). Measuring Outcomes in Child Welfare. Chapel Hill:  Jordan Institute for Families,

  18. Federal and State Outcome Measures • Federal • Child and Family Services Review (CFSR) • State • Accountability Act AB 636 • Went into effect in California on January 1, 2004. • This new system holds the state and counties accountable for improving outcomes for children through the establishment of improvement goals, public reporting of outcomes and county-specific improvement plans that must be approved by county boards of supervisors and submitted to the state • No goals or standards. Rather, objective is continuous, quality improvement within each county.

  19. Lesson #2: Measuring Outcomes Can Get Complicated (quickly)… Module 2: Lessons in Using Data

  20. What you will find reported for California Module 2: Lessons in Using Data

  21. Website view example… Reunification composite

  22. Module 2, Section 2.1 Optional/additional performance outcome information for instructor use Module 2: Lessons in Using Data

  23. Children and Families Service Reviews(more details than most will want, but truly useful to understand!) Federal Child and Families Service Reviews (CFSR) • Transition from individual “measures” to safety indicators and composite measures or permanency and stability • National standards for both the indicators and composites are based on the 75th percentile of state performance in 2004 • Although national standards have been set for the composites rather than individual measures… • The goal is to improve State performance on all measures (every improvement reflects a better outcome for children) • Improvement on any given measure will result in an increase in the overall composite score • Analogous to Academic Achievement Test Scoring… Module 2: Lessons in Using Data

  24. Principal Components Analysis (PCA)(the “black box” version) Three components based on related measures! A bunch of measures… Median Time in Care Emancipating from Care Recurrence of Maltreatment black box of fancy statistical tools Component #1 Timeliness of Reunification Abuse in Foster Care Component #2 Permanency of Reunification Placement Stability Component #3 Timeliness of Adoption Module 2: Lessons in Using Data

  25. Z-Scores? • Before dumping all of the measures into the PCA “Black Box”, they were transformed into standard scores (z-scores) • A z-score serves two purposes: Puts measures in the same “range” Sets measures to the same “system”

  26. And an Example… • A researcher interested in measuring “success” in high school. • Collects the following measures for each student: • Athletic Ability • Good Grades • Physical Attractiveness • Interest in Sports • Chess Club Membership • Science Club Membership • Social Life Principal Components Analysis…

  27. Interest in Sports Athletic Ability Good Grades Chess Club Member Science Club Member Physical Attractiveness Active Social Life Explores the contribution of each part to the whole: Structures the data into independent components: Reduces the number of individual measures: Athletic Ability Jock Component = Interest in Sports VERY HIGHLY ASSOCIATED!! Brainiac Component = Good Grades Chess Club Member Popular Kids Component = Physical Attractiveness Active Social Life Module 2: Lessons in Using Data

  28. Measure Contributions to Composites Reentry Following Reunification (Exit Cohort) Reunification Within 12 Months (Entry Cohort) Median Time To Reunification (Exit Cohort) Reunification Within 12 Months (Exit Cohort) Note: Measures may not sum to exactly 100% due to rounding. Module 2: Lessons in Using Data

  29. Measure Contributions to Composites Adoption Within 12 Months (Legally Free) Legally Free Within 6 Months (17 Months In Care) Adoption Within 12 Months (17 Months In Care) Median Time To Adoption (Exit Cohort) Adoption Within 24 Months (Exit Cohort) Note: Measures may not sum to exactly 100% due to rounding. Module 2: Lessons in Using Data

  30. Measure Contributions to Composites In Care 3 Years Or Longer (Emancipated/Age 18) Exits to Permanency (Legally Free At Exit) Exits to Permanency (24 Months In Care) Note: Measures may not sum to exactly 100% due to rounding. Module 2: Lessons in Using Data

  31. Measure Contributions to Composites Placement Stability (At Least 24 Months In Care) Placement Stability (12 To 24 Months In Care) Placement Stability (8 Days To 12 Months In Care) Note: Measures may not sum to exactly 100% due to rounding. Module 2: Lessons in Using Data

  32. Measure Contributions to Composites Note: Measures may not sum to exactly 100% due to rounding. Module 2: Lessons in Using Data

  33. Module 2: Lessons in Using Data

  34. Module 2: Lessons in Using Data

  35. Module 2: Lessons in Using Data

  36. Module 2, Section 3 Popular press examples of data use/misuse (aka, numbers gone wild) Module 2: Lessons in Using Data

  37. Public Data: Putting It All Out There • PROS: • Greater performance accountability • Community awareness and involvement, encourages public-private partnerships • Ability to track improvement over time, identify areas where programmatic adjustments are needed • County/county and county/state collaboration • Transparency • Dialogue Module 2: Lessons in Using Data

  38. Public Data: Putting It All Out There • CONS: • Potential for misuse, misinterpretation, and misrepresentation • Available to those with agendas or looking to create a sensational headline • Misunderstood data can lead to the wrong policy decisions • “Torture numbers, and they’ll confess to anything” (Gregg Easterbrook) Module 2: Lessons in Using Data

  39. ^ Misused Statistics There are three kinds of lies: Lies, Damned Lies and Statistics Module 2: Lessons in Using Data

  40. Five Ways to Misuse Data (without actually lying!): • Compare Apples and Oranges • Use ‘snapshots’ of Small Samples • Rely on Unrepresentative Findings • Logically ‘flip’ Statistics • Falsely Assume an Association to be Causal Module 2: Lessons in Using Data

  41. 1) Compare Apples and Oranges Two doctors in Anytown, CA… Doctor #1Doctor #2 What if the populations served by each doctor were very different? Doctor of the Year? 20/1000 2/1000 Module 2: Lessons in Using Data

  42. “Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.” • SF Chronicle, “Accidents of Geography”, March 8, 2006 Module 2: Lessons in Using Data

  43. “Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.” Different families and children served? Different related outcomes? First entry rates in Fresno are consistently lower Re-entries in Fresno are also lower…

  44. 2) Data Snapshots… Number of Crimes Period 1: 76 Period 2: 51 Period 3: 91 Period 4: 76 Crime in Anytown, CA… No change. Average = 73.5 Crime jumped by 49%!! Crime dropped by 16% Module 2: Lessons in Using Data

  45. “A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay area...” SF Chronicle, “No refuge. For foster youth, it’s a state of chance”, November 15, 2005 Module 2: Lessons in Using Data

  46. “A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay Area…” Abuse in Care Rate Period 1: 1.80% Period 2: 1.64% Period 3: 0.84% Period 4: 0.00% Responsible use of the data prevents us from making any of these claims (positive or negative). The sample is too small; the time frame too limited. = 2/111 = 2/122 100% improvement! = 1/119 = 0 0 Children Abused! Module 2: Lessons in Using Data

  47. 3) Unrepresentative Findings… Survey of people in Anytown, CA… 90% of respondents stated that they support using tax dollars to build a new football stadium. The implication of the above finding is that there is overwhelming support for the stadium… But what if you were then told that respondents had been sampled from a list of season football ticket holders? Module 2: Lessons in Using Data

  48. “Some reports indicate that maltreatment of children in foster care is a serious problem, and in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.” “My Word”, Oakland Tribune, May 25, 2006 Module 2: Lessons in Using Data

  49. “…in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.” Oakland Tribune Factually true? • Yes. Misleading? • Yes. • This was a survey of emancipated foster youth • Emancipated youth represent a distinct subset of the foster care population • This “accurate” statistic misleads the reader to conclude that one-third of foster children have been maltreated in care… Module 2: Lessons in Using Data

  50. 4) Logical “flipping”… Headline in The Anytown Chronicle: 60% of violent crimes are committed by men who did not graduate from high school. “Flip” 60% of male high school drop-outs commit violent crimes? Module 2: Lessons in Using Data

More Related