1 / 36

Summary of Breakout Session

Triple P Outcomes in California Arizona Child Trauma Summit April 9, 2013 Cricket Mitchell, PhD Senior Associate, CiMH. 1. Summary of Breakout Session. Overview of California’s Triple P Outcome Evaluation Data Counties supported by CiMH

cianna
Download Presentation

Summary of Breakout Session

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Triple P Outcomes in CaliforniaArizona Child Trauma SummitApril 9, 2013Cricket Mitchell, PhDSenior Associate, CiMH 1

  2. Summary of Breakout Session • Overview of California’s Triple P Outcome Evaluation Data • Counties supported by CiMH • Options to consider in developing outcome evaluation protocols • Outcome evaluation for Arizona’s Triple P implementation – facilitated discussion

  3. What is CiMH? And How is It Related to Triple P? • The California Institute for Mental Health (CiMH) is a statewide non-profit that provides training, technical assistance, research, evaluation, and policy support to publicly-funded agencies • Supports the dissemination and implementation of 12 evidence-based practices • Program performance and outcome evaluation is a critical implementation support • Triple P was selected for dissemination by CiMH and promoted to county agencies in 2006 • Some agencies contract with CiMH, and some do not

  4. Triple P Implementation Sites Across California Counties Mendocino Alameda Shasta Nevada Sonoma Marin San Francisco Contra Costa Santa Cruz Santa Clara San Joaquin Ventura Los Angeles Riverside Orange San Diego ______________ Also Tri-cities Area

  5. Overview of California’s Triple P Outcome Evaluation Data • Summer 2012 Triple P Data Submission to CiMH • Four Counties • Los Angeles • Shasta • Sonoma • Ventura • 74 implementation sites • 5,292 unique child clients served

  6. Overview of California’s Triple P Outcome Evaluation Data • Outcome evaluation protocols within each county vary • Data elements collected • Demographics • Service delivery information • Outcome measures used • Applications/software used for data entry

  7. Overview of California’s Triple P Outcome Evaluation Data • CiMH’s Program Performance and Outcome Evaluation Reports • Three primary domains • Characteristics of clients served • Description of services provided • Outcomes achieved • Two-Pronged Approach to Outcome Measurement • Target-specific symptoms • General mental health functioning

  8. Overview of California’s Triple P Outcome Evaluation Data • Today’s presentation will highlight select data elements from the Summer 2012 data submission • Triple P Levels and Types • Child Client Demographics • Age, Gender, Ethnicity, Primary Language Spoken in the Home, and Primary Axis I DSM-IV diagnosis • Triple P Outcomes • Eyberg Child Behavior Inventory (ECBI), Parenting Scale, and Youth Outcome Questionnaire (YOQ)

  9. Overview of California’s Triple P Outcome Evaluation Data – Level and Type of Triple P

  10. Overview of California’s Triple P Outcome Evaluation Data – Age • Range: .01 – 26.05 years • Some counties serve Transition Age Youth (15-26) • Mean: 7.7 • Standard Deviation: 4.1 • Mode: 4.0 • Frequency distribution is positively skewed • 25th percentile: 4.6 • 50th percentile: 7.2 • 75th percentile: 10.7

  11. Overview of California’s Triple P Outcome Evaluation Data – Gender

  12. Overview of California’s Triple P Outcome Evaluation Data – Ethnicity

  13. Overview of California’s Triple P Outcome Evaluation Data – Primary Language

  14. Overview of California’s Triple P Outcome Evaluation Data – Primary Axis I DSM-IV Dx* *Two of the four Counties track mental health dx

  15. CiMH Outcome Indicators • Percent Improvement • Percent improvement from average pre-score to average post-score • Paired t-test conducted to examine whether or not the difference is likely to be due to chance (p<.01); if not, the percent change is asterisked (*) to indicate a statistically significant improvement • Effect Size Estimate: Cohen’s d • A standardized measure that estimates the magnitude, or strength, of the observed change • Conventional interpretation: .8 ≈ “large” effect; .5 ≈ “moderate” effect; and, .2-.3 ≈ “small” effect

  16. CiMH Outcome Indicators • Reliable Change • The amount of change observed in an outcome measure that can be considered an actual change, and not likely to be due to the passage of time or measurement error (p<.05) • Complex statistical formula that takes the measure’s reliability into consideration, as well as the variability observed among scores • Once the formula is applied, clients can be grouped into one of three categories: reliable positive change; reliable negative change; and, no reliable change

  17. Overview of California’s Triple P Outcome Evaluation Data • Target-Specific Outcome Measure Focused on Child Disruptive Behaviors • Eyberg Child Behavior Inventory (ECBI) • Parent/Caregiver Report of the Intensity and Problematic extent of child behavior problems • 36 items • Intensity Score Range 36 – 252 • Clinical cutpoint 131 and higher • Problem Score Range 0 – 36 • Clinical cutpoint 15 and higher • Used by two of the four counties

  18. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: ECBI *A statistically significant improvement, p < .01

  19. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: ECBI Intensity Solid line indicates clinical cutpoint

  20. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: ECBI Problem Solid line indicates clinical cutpoint

  21. Overview of California’s Triple P Outcome Evaluation Data • Target-Specific Outcome Measure Focused on Parenting • Parenting Scale • Parent/Caregiver Report that assesses parenting and disciplinary styles that are found to be related to the development and/or maintenance of child disruptive behavior problems • 30 items • Total Score is a mean item response ranging from 1 – 7 • Clinical cutpoint 2.8 and higher • Used by two of the four counties

  22. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: Parenting Scale *A statistically significant improvement, p < .01

  23. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: Parenting Scale Solid line indicates clinical cutpoint

  24. Overview of California’s Triple P Outcome Evaluation Data • General Outcome Measure of Mental Health Functioning • Youth Outcome Questionnaire • Parent/Caregiver Report that assesses multiple dimensions of child/youth mental health functioning • 64 items • Total Score Range -16 – 240 • Clinical cutpoint 47 and higher • Used by one of the four counties

  25. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: YOQ Total *A statistically significant improvement, p < .01

  26. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: YOQ Total Solid line indicates clinical cutpoint

  27. Overview of California’s Triple P Outcome Evaluation Data – Outcomes: Reliable Change

  28. Overview of California’s Triple P Outcome Evaluation Data • Follow-up analyses of aggregate data indicate no differences in: • change in ECBI Intensity Score outcomes; • change in ECBI Problem Score outcomes; • change in Parenting Scale outcomes; or, • change in YOQ Total Score outcomes by gender or ethnicity

  29. Options to Consider in Developing Outcome Evaluation Protocols Thoughtful and thorough planning is the key! • Data elements to track/collect • Parsimony • Utility • Outcome measures • Relevance to treatment target/goals • Psychometric characteristics (valid, reliable) • Cost • Time (administration, scoring, data entry) • Training and technical assistance

  30. Options to Consider in Developing Outcome Evaluation Protocols • Application/software used for data entry • System already in place that can be modified? (e.g., EHR, county- or state-level information system) • Cost • Skill level to use/employ • Utility of data elements for analysis and reporting • Training and technical assistance

  31. Options to Consider in Developing Outcome Evaluation Protocols • Frequency of analysis and reporting • Multiple stakeholders • Different reports for different audiences • Processes for maximizing utility of data • Clinical utility • Program improvement • Systems-level decisions Feedback is Essential

  32. Additional Considerations for Telling the Whole Story • Collect minimal data on all clients referred • Determine entry rate • Determine additional need (waiting lists) • Collect completion status (yes/no) • Determine dropout rate • May provide the opportunity to examine dose-response relationships • Track clients who are served by more than one Level/Type of Triple P (within and across providers) • Track population-level indicators (substantiated child maltreatment cases, out of home placements, emergency room visits for unexplained child injuries)

  33. Evaluation for Arizona’s Triple P Implementation – Discussion Questions • Who should be included in decision-making? • Is there an overarching evaluation framework? • What data are currently being collected? • What additional data elements are of interest? • What outcome measures will be used? • How will they be obtained, distributed, and used? • Who will provide training and technical assistance? • How will data be tracked/collected from individual Triple P providers?

  34. Evaluation for Arizona’s Triple P Implementation – Discussion Questions • How will population-level indicators be tracked? With what frequency? • Responsibility for and frequency of data analysis and reporting? • How will data be used to inform decisions? • Client-level • Program-level • System-level

  35. Discussion Summary

  36. The End • Contact Information • Cricket Mitchell, PhD • Email: cmitchell@cimh.org • Cell phone: 858-220-6355

More Related