Data: Telling a Story … A Walk Through Meaningful Data Analysis

1. Data: Telling a Story …A Walk Through Meaningful Data Analysis Judge Nancy SalyersJudicial Executive and Co-Founder, Fostering Court ImprovementCo-Director, Fostering Results Andy BarclayStatistician and Co-Founder, Fostering Court ImprovementBarton Child Law and Policy Clinic, Emory University School of Law Presented To:Fostering Court Improvement, A Data Workshop for Decision MakersDecember 7, 2006

2. Overview • How Knowledge of the Past and Present can Help you Shape the Future • Data, Data, Data Isn’t Yada, Yada, Yada • Court Performance Measures • AFCARS • The Easy Win vs. Sea Change • Getting Real, Getting Local, Getting Context • Results that Matter to Kids & Families FosteringCourtImprovement.org

3. Shaping the Future Why Data, Data, Data Isn’t Yada, Yada, Yada FosteringCourtImprovement.org

4. Shaping the Future • Lessons learned or missteps repeated? • Lunchroom lore or documented results? FosteringCourtImprovement.org

5. Shaping the Future “They always say that time changes things, but you actually have to change them yourself.” Andy Warhol FosteringCourtImprovement.org

6. Shaping the Future • Do you know where your caseload has been? • Do you know where it is? • Do you know where it is going? FosteringCourtImprovement.org

7. Shaping the Future Am I on the road to success or on a “frolic and detour”? (Only the data know.) FosteringCourtImprovement.org

8. Shaping the Future “Georgia Judge leads stampede out of homes.” FosteringCourtImprovement.org

9. Shaping the Future “Illinois Judge ‘doesn’t have time’ to return children to parents.” FosteringCourtImprovement.org

10. Shaping the Future Take off the blinders. Or Load and Learn. FosteringCourtImprovement.org

11. Shaping the Future Where to Start? FosteringCourtImprovement.org

12. Court Performance Measures The three leading legal/ judicial organizations, with a grant from the Packard Foundation, have created and pilot-tested court performance measures that complement those used in the CFSR process so that agencies and courts can collaborate in the measurement of their performance and tracking of their own progress in improving safety, permanence, and timeliness for children coming before them. FosteringCourtImprovement.org

13. Court Performance Measures (Building a Better Court) • Reinforce current federal reform initiatives, including CFSR and PIP’s • Helps to identify areas of court reform essential to better outcomes • Assist the CIP by providing courts with tools to engage in systematic process of court performance measurement FosteringCourtImprovement.org

14. Court Performance Measures “Performance measures are the yardstick by which courts and child welfare agencies can measure the success they are achieving and the progress they are making.” Building a Better Court FosteringCourtImprovement.org

15. Court Performance Measures • Safety: Are there new allegations while in foster care? • Permanency: What was the exit from foster care and how many came back? • Due Process: Is there notice and representation? • Timeliness: How long between hearings? FosteringCourtImprovement.org

16. Already Measurable and Measured (AFCARS) • Safety • Permanency • Timeliness FosteringCourtImprovement.org

17. AFCARS Adoption and Foster Care Analysis and Reporting System FosteringCourtImprovement.org

18. Uses of AFCARS • Award Adoption Incentive bonuses • Allot Chafee Independence $$$ • Produce annual reports to Congress on child welfare outcomes • Conduct IV-E eligibility reviews • Assess state’s compliance with federal CFSR performance standards FosteringCourtImprovement.org

19. Platform for Collaboration • Longitudinal AFCARS provides a platform for rolling out the due process & timeliness measures recommended by Building a Better Court • Promotes working partnership between the courts, child welfare agencies and universities, • Provides 9 of the 24 measures defined in Building a Better Court, and 9 of the 30 in ToolKit. FosteringCourtImprovement.org

20. Fostering Court Improvement What gets measured gets done, If You know it’s being measured. FosteringCourtImprovement.org

21. Stories from Medicine : An Easy Win Bayesian Pre-Screening Tool • Diagnostic Imaging (Cardiac PET/SPECT) Pre-Screen • Very simple questionnaire • Predictive Value of the Test • Surgery: y/n • Angioplasty: y/n • Easy sell to cardiologists • Very easy sell to insurance companies FosteringCourtImprovement.org

22. Stories from Medicine : A Sea Change Automated Reading of Scans • Major resistance, even when shown computer was right more often, lower mortality • Over time (3 years) older docs modified their risk tolerance, less invasive, better outcomes, learned from computer • Tripled business, more patients living longer, much lower costs ... FosteringCourtImprovement.org

23. Court Stakeholders Have a Right to be Skeptical • In medicine, I see examples of better outcomes at lower cost quite often. • Have yet to see one in child welfare. • Many claims, little evidence • Judges, attorneys, CASAs, and caseworkers have good reason to be skeptical: • Complex Business • Spectacular IT Failures • Terabytes of Unused Data FosteringCourtImprovement.org

24. Get Real • Data reports are only meaningful to you in the context of your job, your reality. • We (FCI) have been successful* using data to develop a shared context among local court stakeholder groups. • Very different jobs, very different realities, but we need to have some common understandings to do successful* work together (and occasionally alter some realities). *successfully improve an outcome FosteringCourtImprovement.org

25. Diverse Realities: Dependency Court Caseloads • 3,140 US counties + DC • 70% of counties remove less than 1 child per week • 50% less than 1 per 2 weeks • 7% of counties (234) average more than 1 child removal per business day FosteringCourtImprovement.org

26. Get Local • Much as I’d like to, we probably can’t do this with just a website. • our web reports are not for a general audience. • County by county … • Judges convene • Georgia’s CIP buys lunch • I walk through a Powerpoint of charts & numbers from the website. FosteringCourtImprovement.org

27. Broad Topic Areas of Court Stakeholder Meetings • Removal Rates • Reasons for Removal • Kids in Care • Group Settings • IV-E Reimbursement • Kids Headed for Adoption • Reunification Rate • Time to Adoption FosteringCourtImprovement.org

28. A Simple Temporal Context • Emphasize the Context for every number, nothing without Context. • Temporal: Interpret current measures in the context of the past. FosteringCourtImprovement.org

29. The Context of Comparable Jurisdictions • Emphasize the Context for every number, nothing without Context. • Comparables: Interpret in context of comparable counties. FosteringCourtImprovement.org

30. The Context of Statewide Rank • Emphasize the Context for every number, nothing without Context. • Rank: Interpret in context of state or other ranking. FosteringCourtImprovement.org

31. The Systemic, Process Context Flow of cases through the system FosteringCourtImprovement.org

32. Outcome Framework Context • Safety • Maltreatment does not occur.Measures: Proportion of children reported maltreated. • Maltreatment does not recur.Measures: Proportion of children re-reported within 3, 6, and 12 months; Proportion re-entering foster care. • Permanency • Children remain in their homes.Measures: Proportion removed from their home. • Bonded attachments are maintained.Measures: Proportion of siblings placed together; Frequency of visitation with kin; Proportion in a stable placement. • Permanency is timely.Measures: Proportion reunified with family within 12 months; Proportion adopted within 24 months. • Well Being • Children are healthy.Measures: Proportion with physical health needs met; Proportion with mental health needs met. • Children are educated.Measures: Proportion performing at age-appropriate grade level. Note that these outcomes are from the child’s perspective, not the system’s. FosteringCourtImprovement.org

33. Douglas County, GA: Removals cut 50% Relative placements increased 10-fold Adoption rate doubled Floyd County, GA: Removals cut 30% Pre-teens in group care down 80% Hall County, GA: Adoptions under 24 months tripled Bartow County, GA: Removals cut 50% Most counties unchanged, but actively dissecting their processes Track Record: Real Work Done in a Shared Context FosteringCourtImprovement.org

34. Improving the Product • Additional Reports: • Safety Report from linked NCANDS-AFCARS data • Children Remaining Safe with Family • CFSR 2001, CFSR 2006 • Illinois Signs of Progress Reports • Toolkit (9 Measures) • Discovering Best Practices: • Composite Measures • Causal Linkages from Process Measures to Outcome Measures FosteringCourtImprovement.org

35. Process Measures(caseloads, due process, …) are crucial, but only when put in the context of Outcome Measures. Judges care deeply about outcomes. Need bottom-lines to keep us aligned and on task, I propose: Children safe with family Failing that, timely permanency Shift spending from data acquisition to data analysis. Don’t call GeekSquad™ for data analysis.(Don’t equate technology and data analysis) Loose Ends FosteringCourtImprovement.org

36. Questions, Comments ... Judge Nancy SalyersJudicial Executive and Co-Founder, Fostering Court ImprovementCo-Director, Fostering Results Andy Barclay, Andy@FosteringCourtImprovement.orgStatistician and Co-Founder, Fostering Court ImprovementBarton Child Law and Policy Clinic, Emory University School of Law Presented To:Fostering Court Improvement, A Data Workshop for Decision MakersDecember 7, 2006