The Role of Client Outcomes in Evaluating a System of Care Model of Mental Health Delivery

1. The Role of Client Outcomes in Evaluating a System of Care Model of Mental Health Delivery Richard Phillips, Ph.D. Director of Research and Evaluation, Idaho Child Welfare Research & Training Center Eastern Washington University Patty Gregory, LCSW Director Idaho Child Welfare Research & Training Center Eastern Washington University

2. Design • On site meeting at each demonstration site for: • information gathering • case staffing observation • training on evaluation issues • conducting focus groups for process data collection • Goal setting process • Council member identification of barriers encountered in reaching the goals. • Critical case review

3. Role of Outcomes in a System of Care • What are outcomes? • Changes in adaptive behaviors • Basic assumption: what people do as a result of professional interaction is the best measure of professional effectiveness • What’s the difference between outcomes and something else? • Outputs: services/activities provided to clients to help (them) improve their condition • Basic assumption: the amount of services is the best measure of professional effectiveness.

4. Kinds of Outcomes • Representative Outcomes • Refers to the health or capacity of a system around specific issues or mandates • Represents how an entity or person is doing on a global scale. • Refers to how individual outcomes relate to a larger system. • Serves to help an entity or person assess how they are doing on a global scale. • Examples include: • State average scores on standardized academic achievement tests • Decreases in CPS referrals or re-referrals • Decreases in CAFAS scores

5. Kinds of Outcomes • Functional Outcomes • Refers to adaptive behaviors of a person or a program or system as they strive to reach their goals; what people actually do to reach their goals. • Serves to enhance the self-reliance of persons in relation to the goals in question • Example: • Client C visited 3 agencies for the first time and received food and clothing for a week • Social Worker B collaborated with the school counselor to create an early referral form • Student K stayed in school and completed his past assignments

6. Functional Outcomes What people do as a result of participating in a relationship Take actions Change attitudes Increase performance Become more able Representative Outcomes What happens as a result of aggregated functional outcomes Increased classroom performance Increased grades or attendance Positive changes in CAFAS scores Participatory Evaluation: Using Client Progress to Help Clients Progress

7. System of Care for Children • Set of shared values and principles • Focus on strengths of family; not on diagnosis of the child • Maintain child in their own home, in their own school, and in their own community • Service array planned with family to meet family identified needs • Non-traditional • Traditional

8. Post Treatment Data • School attendance/conduct for first full quarter after completion of service • # of failing school academic core subject grades out of total possible • CAFAS score • Post Juvenile Probation Assessment score • Disposition of case • Approximate # of hours spent by council in chambers • Approximate # of hours spent by council members in case management • Number of council members involved in direct service on case • Family satisfaction survey results

9. Nothing About Us Without UsParent perspective on functional outcomes • First and foremost, we seek safety • That our children are insured a quality of life as happy productive members of community as measured by: • sharing a ceremony or celebration, • being “in this together” • making it through a school suspension while adjusting to a new medication • We need strengths acknowledged, resources valued and our children treated with love and respect CrossBear, S. (Spring/Summer 2002) Nothing about us without us, Data Matters, National Technical Assistance Center for Children’s Mental Health, Georgetown University Child Development Center

10. Generalized Service Model Needs Assessment & Diagnosis Client Presenting Issues Tx plan and Clinical goals Services Provided Periodic Assessment & Service Plan Review Case Closure when Clinical goals met

11. Developing Community Council Model Council Goals Family Centered Problem Solving Client Referral Tx Goals & Service Plan Interventions & Assessments Collaborative Reviews Case Closure when Council and Family Goals met

12. Conclusion When the probable results of inaction, and the lack of results of current actions, form the most important reasons for referral to a program, then a conclusion about standardized data can be made. Any data that are used in predictive sense to direct what the parent should do are data that are seen as harmful to the process of getting the family to engage as partners with the council in setting out a treatment plan for their child.

13. Council members defensive because of agency history Lack of experience with engaging families in a process where the family leads interventions Lack of understanding of the participation agencies regulations and guidelines Lack of trust – in the family and in the professional process Inability to get key partners to the table – not full participation Lack of available concrete services/programs Funding constraints The Reality of Collaboration

14. The Role of Advocacy • A key measure of council effectiveness is the willingness and ability of council members to integrate council client concerns into their daily agency routines • Advocacy looks like: • making connections with other professionals as council members go about their daily routines • Intervening in one’s own agency to help clients gain access to services • Being supportive by other council members in challenging status quo treatment options • Advocacy does not look like being held accountable for collecting data to show client progress

15. Greatest Challenge:Supporting care providers as they move from an expert, service provider orientation to a facilitating, partnering orientation

16. Recommendations • Ensure that system of care extends into the bureaucracy • Work with local sites to establish data collection routines that celebrate client actions • Provide evaluation support for local councils so volunteer hours are spent in collaborative functions, not recording or documentation functions • Train to a participatory evaluation model that recognizes that client ownership of outcomes may be the single most important function of a mental health intervention • Learn to acknowledge that validity and reliability may be obstructionist constructs for support changes in client behavior

17. Functional Outcome Evaluation Client Progress Professional Client set Goals Goals

18. Bibliography • Burns, B., & Goldman, S., (1998) Promising practices in wraparound for children with serious emotional disturbance and their families. Washington DC: Georgetown University Child Development Center, CASSP technical Assistance Center. Substance Abuse and Mental Health Services Administration. U.S. Department of Health and Human Services • CrossBear, S. (Spring/Summer 2002) Nothing about us without us, Data Matters, National Technical Assistance Center for Children’s Mental Health, Georgetown University Child Development Center • Friesen, B., & Huff, B., (1996) Family perspectives on systems of care. In B. Stroul (Ed.) Children’s Mental Health. Creating Systems of Care in A Changing Society. Baltimore, MD: Brookes Publishing. • Lourie, I.S., & Davis, C. (1999) A Needs Assessment of Idaho’s Children With Serious Emotional Disturbances and Their Families. • Pines, S. (2002) Building Systems of Care: A Primer. National Technology Assistance Center for Children’s Mental Health • State of Idaho (2002). Building on Each Other’s Strengths. Child Mental Health Initiative Grant, GFA: SM-02-002 CFDA 93.104.