Lessons learned from the Evaluation of Baltimore Expanded School Mental Health: Challenges and Solutions

1. Lessons learned from the Evaluation of Baltimore Expanded School Mental Health: Challenges and Solutions Georgetown University National Technical Assistance Center Webinar Series October 21, 2010 Making the Case for Evaluation of School Mental Health Programming: Strengthening Partnerships for Positive Outcomes Bruno J. Anthony, Ph.D. Georgetown University Center for Child and Human Development

2. Collaborators • Baltimore City Public Schools • Louise Fink • Ike Dibor • Shawn Hales • Baltimore Mental Health Systems • Jane Paplinger • David Jones • Denise Wheatley-Rowe • Georgetown Center for Child and Human Development • Bruno Anthony • Joyce Sebian • Courtney Holland • Clare Dunne

3. Overview BESMH • Baltimore ESMH a recognized state and national leader for more than 20 years. • ESMH services have complemented and supplemented the services provided by Baltimore City Public Schools (social workers, psychologists, and guidance counselors) • Currently, prevention and treatment services provided in 106 schools (approximately half of the schools in Baltimore City) through 12 provider agencies and 106 clinicians. • Intended to address barriers to learning to enable students to make better use of educational programs in their schools and to provide an alternative to mental health services within structure of Special Education System • Funding: Braided funding from school system, Department of Human Resources, Baltimore Substance Abuse System, Baltimore Mental Health System.

4. Evaluation of BESMH • Demand for academic accountability heightens the need for studies examining the specific impact of ESMH • Purpose: Assess the impact of a comprehensive ESMH program • Test assumptions that ESMH is beneficial in terms of academic outcomes • Inform programming • Inform ongoing evaluation and sustainability efforts. • Funding • Planning: Internal funding from BESMH budget • Implementation: State of Maryland Mental Health Transformation State Incentive Grant (TSIG)

5. Challenges and Approaches: Developing a Meaningful and Feasible Plan • A participatory approach • Key Informant Input • Focus Groups (administrators, program directors, clinicians/providers, caregivers, youth) • Consultation (advisory committee, monthly program directors meetings, leaders in the field) • School system partners (DREAA, Student Support Services, leaders) • Goals • Identifying important barriers and challenges • Identifying the appropriate and relevant outcomes • Incorporating content expertise in understanding ESMH in Baltimore • How to use the evaluation findings to improve the ESMH program and for advocacy.

6. Challenges and Approaches: Examining Outcomes at Individual Level Risk of connecting private school and ESMH data . • Employed the “zipper” for linking separate confidential data sets that contain personal identifying information while preserving anonymity. • Safe analyst to link and process data who is not part of either the research team, the agency with overall administrative oversight over the ESMH program or the school system. • Providers send service use data set with ID numbers sent to the “safe analyst,” who has no access to student names. • School system sends demographic and academic/school functioning with ID numbers to safe analyst. • Safe analyst uses ID numbers to merge the two data sets then encrypts ID numbers

7. Challenges and Approaches: Capturing Spectrum of ESMH Services • Service logs were not capturing the range of promotion, prevention and treatment work carried out by clinicians. • Electronic data collection • Developed in a participatory process • Individual: For each date of contact, the type(s) of services and, at the end of each month, the focus of treatment and the type of the interventions. • Group: attendance in prevention and treatment groups. • Universal: four types of prevention activities. • Training: direct contact, manual and web-based

8. Extent and Type of Services • ESMH Clinicians touch an enormous number of students, caregivers and school staff, extending the range of clinical service that could not happen in more traditional settings. • Clinicians had direct contact with more 6254 students -1/5th of students attending the 106 schools that support ESMH • A large part of the clinicians work involves indirect, collateral services

9. Percentage of Students by Grade

10. Percentage of Service Types

11. Percent of Direct and Indirect Services

14. Service Focus • Clinicians focus on issues that disrupt a student’s engagement with school • Clinicians for the most part report using approaches with an evidence base • Family/caregiver engagement is challenging

15. Percentage of Focus Areas Coded

17. Percentage of Intervention Types Coded

18. Challenges and Approaches: Comparison Groups • As is often the case with on-going, community-based programs, a full-fledged experimental (random-assignment) design was not practical, feasible or ethical. • Quasi-experimental strategies to support causal influences of ESMH. • Approaches: • Contact Groups • Matched Comparison Groups • Dosage Groups

19. Distribution of Students in Different Dosage Groups E-E: Students receiving ESMH services in an ESMH school NoE-E: Students NOT receiving ESMH services in an ESMH school NoE-NoE: Students NOT receiving ESMH services in a school without ESMH

20. Matched Comparison Group Formation • Propensity scores • Probability of a student in a school with an ESMH clinician receiving ESMH services, given a set of background variables for that individual. • Determined background risk factors that predicted participation in ESMH using Binary Logistic Regression (2007-08 attendance, suspensions, MSA Reading and Math scale scores). • Children receiving ESMH would have high propensity scores based on the set of background variables . Students with similar background characteristics but not receiving ESMH services should also have high propensity scores. • Propensity score of each ESMH student matched with a student drawn randomly from those students not enrolled in ESMH to form Comparison Group.

21. Preliminary Overall Results A greater percentage of students attending a school with ESMH services and accessing these services showed • Improvement over the course of the year in Math and English benchmarks. • Decreases in suspensions

22. Challenges and Approaches: Outcome Measurement Framework • Devising an outcome measurement framework that satisfies different stakeholders without overburdening clinicians • Integrate data needs and system pressures of different stakeholders. • Use available data records, outcome measurements already in place, • Stakeholders recognized the imperative of demonstrating effectiveness in the school setting with selected mediating/moderating contextual variables. • The importance of looking at school outcomes (e.g., academic success, promotion, absences and suspensions) to ensure sustainability

26. Evaluation Contributions to Sustaining ESMH • Building evaluation and research model – participatory process • Building infrastructure for ongoing evaluation and research • Service Trends • Impact on programming decisions • Impact on financial models • Training • Leveraging other initiatives. • Utilizing evaluation data to make the case for ESMH program • Case for Ancillary and Supportive Services • Teacher Consultation • Care Coordination • Family contacts • Case for School-related Benefits • Improvement in academic measures • Positive effects on school attendance and behavior

27. ContactingUs BESMH Evaluation Team: Bruno Anthony, Ph.D. Professor and Director of Research and Evaluation, Email: bja28@georgetown.edu Joyce Sebian MS. Ed. Senior Policy Associate, Email: jks29@georgetown.edu Georgetown University Center for Child and Human Development Department of Pediatrics 3300 Whitehaven Street, NW, #3300 Washington, DC 20007