Developing and Implementing Multilevel Program Evaluation Plans for SAT-ED Grants

1. Developing and Implementing Multilevel Program Evaluation Plans for SAT-ED Grants March 11, 2013 Michael L. Dennis Chestnut Health Systems, Normal, IL Available from www.gaincc.org/presentations Created for: Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Treatment (CSAT) under contract number HHSS283200700003I, Task Order HHSS28300002T

2. Goals for the Presentation?? • Summarize the key problems in our field that SAT-ED is attempting to address • Review objectives, key questions, and sources of data to be addressed in the evaluation • Identify key steps in designing, implementing, and using evaluation to help manage and improve programs • Discuss strategies for reliable, valid, and efficient collection and analysis of state (including commonwealth), site, and client-level data • Provide links to further resources and training

3. Objectives of SAT-ED To improve treatment for adolescents through the • Development of a Learning Laboratorywith collaborating local community-based treatment provider sites. • Improvements in State-level infrastructure through workforce development, financial planning, licensure, and certification • Improvement of site-level infrastructure through implementation of evidence-based practice (EBP) related to assessment and treatment • Assessment, treatment, and monitoring of change at the client level Feds have to be able to describe what was done with the money State and sites have to decide what to try to sustain

4. Typical Components of a Multilevel Evaluation Plan • Needs assessment • Description of program activities, Theory of Change, and/or Logic Model • Approach to stakeholders • Evaluation questions, data sources, and methodology • Performance monitoring and reporting TIP: Labels and the order of components can vary to fit your situation, the point here is really to make sure that you have them covered or that your team makes an informed decision not to address them

5. 1. Needs Assessment • Description of infrastructure and site level needs and what information is still needed • Local system and/or cultural consideration • Articulate the rationale for the selection of the targeted • Infrastructure activities • Site selection • Evidence-based assessment selection • Evidence-based treatment selection

6. Structural Challenges to Delivery of Quality Care in Behavioral Health Systems High-turnover workforce with variable education background related to diagnosis, placement, treatment planning, and referral to other services Heterogeneous needs and severity characterized by multiple problems, chronic relapse, and multiple episodes of care over several years Lack of access to or use of data at the program level to guide immediate clinical decisions, billing, and program planning Missing, bad, or misrepresented data that need to be minimized and incorporated into interpretations Lack of infrastructure that is needed to support implementation and fidelity of evidence-based practices that have been shown to work better on average

7. Substance Use Disorder and Treatment by Age 7 Higher rates of unmet need for adolescents and young adults Higher rates of need for young adults Source: SAMHSA 2009 National Survey on Drug Use and Health

8. Substance Use Disorder and Treatment by Age 8 Completion rates are lower for adolescents and young adults Lengths of stay are shorter for young adults and adults Source: SAMHSA 2009 Treatment Episode Data Set – Discharges (TEDS-D)

9. No Self-Help Group Participation in the First 3 Months of Treatment Age* Higher adolescents and young adults * p<.05 SAMHSA 2011 GAIN SA Data Set subset to has 3m Follow up (n=21,228)

10. Unmet Need for Mental Health Treatment by 3 Months Age* Higher for adolescents and young adults * p<.05 SAMHSA 2011 GAIN SA Data Set subset to has 3m Follow up (n=14,358)

11. Unmet Need for Medical Treatment by 3 Months Age* Higher for Young Adults * p<.05 SAMHSA 2011 GAIN SA Data Set subset to has 3m Follow up (n=8,517)

12. 2. Description of Program Activities, Theory of Change, and/or Logic Model • Describe infrastructure and site-level activities to be conducted and any specific programs or evidence-based practices you plan to use • Theory or logic model for each need and how they will be addressed by the activity and the expected outcome • Discuss relationship between various needs, activities, or components, including how State- and site-level activities support each other

13. Expected State-Level Infrastructure Activity • Interagency workgroup to improve the statewide infrastructure for adolescent substance abuse treatment and recovery • Memoranda of understanding between SAT-ED awardee agency and other child-serving agencies • Multiyear workforce training plan for specialty adolescent behavioral health (substance use disorder/co-occurring substance use and mental disorder) treatment/recovery sector and other child-serving agencies • Comprehensive and integrated continuum of care for adolescents with substance use and mental health disorders in terms of both funding and services

14. Expected State-Level Infrastructure Activity (continued) • Financial mapping to understand current funding and coverage • Coordination of funding to make the system more efficient, expand coverage, and shift towards more effective practices • Facilitation of a learning laboratory to use above to identify target areas of need, attempt change, evaluate the change, and if necessary adjust strategies to improve the quality of care TIP: Can relate to and/or build on activities already under way. You just want to be sure you will be prepared to address each area in your annual and final progress reports

15. Other Allowable State-Level Infrastructure Activity • Workforce mapping to understand qualifications of staff across the continuum of care and the adequacy of the initial training/ continuing education infrastructure already in place • College, university, and continuing education staff and programs/faculty infrastructure improvements/expansions and number of new/existing staff trained • Other statewide events to provide continuing or community education/training • Reviewing/revising PROGRAM standards for licensure, certification, and/or accreditation of programs that provide substance use and co-occurring mental disorders services for adolescents and their families • Reviewing/revising CLINICAN standards for licensure, certification, and/or credentialing of clinicians that provide substance use and co-occurring mental disorders services for adolescents and their families

16. Other Allowable State-Level Activity (continued) • Family/youth support organization creation, expansion, continuation, or enhancement • People newly credentialed/certified to provide substance use and co-occurring substance use and mental health disorders • Policy changes made as a result of the cooperative agreement • Financing policy changes completed as a result of the cooperative agreement TIP: Choose what makes sense for your needs and proposed activities. Invest more in measuring those areas where you are focusing your resources and attention. There is less interest in the average than identifying and understanding one or more areas where grantees that have done something they found useful.

17. Expected Site-Level Infrastructure Activity • Collaborating sites you have contracting with to provide evidence practice practices (EBPs) • EBP related to (a) assessment and (b) treatment for which you have contracting to obtain at training and technical support to implement • EBP training type, date, and number staff attending each • EBP-proficient staff capacity with regard to the number of employed staff who are certified by level and type of EBP • EBP local trainer or supervisory capacity with regard to the number of employed staff who are certified by level and type of EBP train and supervise new staff

18. Other Optional Site-Level Infrastructure Activity • Implementation of EBP related to assessment in terms of the number completed, linkage to medical records, use of clinical decision support, use for program planning (aka meaningful use) • Expansion of coverage based on number and percentage of assessed youth receiving any services billed to insurance (Medicaid, CHIP, other Federal/State, other private) instead of the block grant • Implementation of EBP related to treatment in terms of the number of clients receiving it and receiving target dosage TIP: Be sure to think about how to describe, measure, and demonstrate a relationship between State- and site-level activities related to the chosen EBP. Collaborate with other State using the same EBP.

19. Comparison of Site EBP for Assessment TIP: Most States/sites have other electronic or hardcopy records and have mentioned additional measures in their proposal or preliminary evaluation plans. Most sites are still in the process of deciding whether to conduct followup with EBP or other measures beyond GPRA.

20. Comparison of Site EBP for Treatment TIP: Several States have talked about comparing to other EBP within their State, comparing to the same EBP in other sites, and/or expanding EBP to other sites.

21. 3. Approach to Stakeholders • Identification of State-, site-, community-, and individual- (youth, family) level holders • Coordination with or creation of strategic planning groups or interagency councils • Coordination with electronic medical and billing records • Involvement of program directors, information technology staff, clinical directors, supervisors, line staff • Coordinating with or creation of community, family, and/or youth advisory groups or partnerships

22. Questions for Stakeholders • Key needs or problems with the current system that might be addressed • Critical time lines, measures, and products that would make it more useful to them • Recognizing how they define and measure things and where multiple definitions or measures may be needed across stakeholders • What will it take for them to support sustainability beyond the grant?

23. Identifying and Addressing Key Subgroups That May Have Concerns or Barriers to Accessing Services • Demographic groups (e.g., by gender, race, ethnicity, age, sexual orientation) • Abilities (e.g., hearing, sight, mobility, IQ) • Clinical subgroups such as • Primary substance, • Co-occurring mental health/trauma/suicide • Crime/violence or justice involvement • Degree of family support and use • Insurance, transportation, or economic TIP: Health disparities need to be treated similarly to safety issues―where best practice is to diligently look for them and work toward reducing them wherever possible to improve effectiveness and reduce liability.

24. 4. Evaluation Questions, Data Sources and Methodology • Operationalizing the program objectives/ questions into activities, measures of implementation/outputs, and outcomes, including the frequency of collection and data sources • Working backwards to make sure the above crosswalk maps onto actual contracts, memos of understanding, and/or expectations of all stakeholders (many of which are developed at different points in the proposal and startup process)

25. State-/Site-Level Infrastructure • Often a matter of documenting what has been done, including dates, type and events, number of staff, degree of completion/certification • Dual Diagnosis Capability in Addiction Treatment (DDCAT) and Dual Diagnosis Capability Youth Treatment Tool (DDCYT) measures of availability and quality of co-occurring services • Identifying how things differ from what was expected, including • Unexpected problems and how they were addressed • Unexpected opportunities and how they were seized • Things that still need to be or might be done

26. Common Client-Level Questions • What are the characteristics and needs of those who were served? • What services did they receive? • To what extent are services targeted at the most appropriate for severe clients? • To what extent are services effective? • Are the services cost-effective? TIP: Not every evaluation will address each of these questions or each question equally well. The point here is to think about how and how well you will be able to answer each.

27. Characteristics and Needs of Those Served

28. What Services Did They Receive? TIP: Without GAIN/CASI followup, you will be very dependent on the quality of and access to records. With them, need to cover first 3 months to describe most of treatment.

29. To what extent are services targeted at the most appropriate or severe clients? • Implementation of reliable, valid, and efficient measures of need and severity • Consensus standards on definition of need, link to services, and/or evidence-based practices associated with better outcomes on average • Implementation of clinical decision support and meaningful use to drive actual treatment planning and services • Evaluation of treatment need profiles, gaps, and health disparities and the program level and monitoring of change over time

30. To what extent are services effective? • Improvements in administrative outcomes (e.g., initiation, engagement, continuing care, evidence-based practices*) associated with better outcomes on average • Participation in self-help and recovery support services • Among those in need, receipt of services related to co-occurring mental health and physical health problems* • Pre-post change in percent of past month abstinence, no substance-related problems, no justice involvement, being housed, vocational engagement, and social connectedness • Comparison of the same program over time, across sites, to other programs, national norms, or standards (ideally matched programs or clients)* TIP: * These have to come from records or supplemental data such as followup data.

31. Are the services cost-effective? • Estimate costs of average services and evidence-based practices using accounting data* • Compare costs to statewide, Federal, or published normative costs overall or adjusting for improved retention* • Putting costs in context relative to baseline costs to society of health care utilization or crime and the extent to which the program is targeting a high cost subgroup • Pre-post change in the cost to society of health care utilization or crime * TIP: *These have to come from records, followup, or other supplemental data such as followup data.

32. 5. Performance Monitoring and Reporting • Early indicators of implementation, fidelity, and steps of the theory of change or logic model • Important for infrastructure measures to include necessary steps (e.g., selection, contracting, events, people, evaluations) • Client-level measures related to— • Recruitment and data collection rate/target, being on time • Case mix of those served • Treatment initiation, engagement, continuing care, satisfaction • Fidelity of EBP • Services targeted at needs

33. Implementation Is Essential (Reduction in Recidivism from .50 Control Group Rate) The best is to have a strong program implemented well The effect of a well implemented, weak program is as big as a strong program implemented poorly Thus, one should optimally pick the strongest intervention that one can implement well Source: Adapted from Lipsey, 1997, 2005 meta analysis of 509 juvenile justice programs

34. What gets measured, gets doneWhat gets fed back, gets done betterWhat gets incentivized, gets done more often Average practice based on TEDS *Based on a count of initiation within 14 days, evidence based practice, engagement for at least 6 weeks, and any continuing care. Source: CSAT 2011 AT SA Data Set subset to 1+ Follow ups (n=17,202)

35. Selected NOMS Outcomes Over Time Most effects are in the first 90 days, important to measure outcome and services received by then Variation in outcomes *Interpolated **Past month Source: CSAT 2011 AT SA Data Set subset to 1+ Follow ups

36. NOMS Outcome Status at Last Wave Measure favors people who come in the door without problems *This variable measures the last 30 days. All others measure the past 90 days. **The blue bar represents an increase of 50% or no problem. Source: CSAT 2011 AT SA Data Set subset to 1+ Follow ups

37. *Based on count of a reduction in the following variables: Substance use frequency, Abuse/Dependence Sx (past 30d), Physical Health (past 90d), Mental Health (past 90d), Nights of Psychiatric Inpatient (past 90d), Illegal Activity (past 90d), Arrests (past 90d), Housed in Community (past 90d), Family/Home Problems (past 90d), Vocational Problems (past 30d), Social Support/Engagement (past 90d), Recovery Environment Risk (past 90d), Quarterly Cost to Society (past 90d), In Work/School (past 90d) Minus No problems at intake with these variables NOMS Outcomes: Count of Positive Outcomes* (Status at Last Followup – Status at Intake) 78% have one or more improved areas Source: CSAT 2011 AT SA Data Set subset to 1+ Follow ups (n=17,722)

38. Health Care Utilization Cost 11% of youth consume 76% of health care costs Source: CSAT 2011 AT Summary Analytic Data Set (n=19,148)

39. Cost of Crime 21% of youth consume 97% of health care costs Source: CSAT 2011 AT Summary Analytic Data Set (n=17,878)

40. Reduction in Health Care utilization off set the cost of SUD Treatment within 12 months \a Includes the cost of treatment \b Year after intake (including treatment) minus year before treatment

41. EBP like A-CRA Cost More but Produce Greater Savings Too \a Includes the cost of treatment \b Year after intake (including treatment) minus year before treatment

42. Impact of Reclaiming Futures Infrastructure Enhancements to Juvenile Treatment Drug Court on Cost of Crime to Society \a RF-JTDC is significantly lower at follow-up than JTDC. Source: Dennis et al 2012

43. Other Evaluation Training Resources • ACYF’s The Program Manager's Guide to Evaluation http://www.acf.hhs.gov/programs/opre/research/project/the-program-managers-guide-to-evaluation • American Evaluation Association http://www.eval.org/ • BJA’s Program Evaluation Manual https://www.bja.gov/evaluation/guide/bja-guide-program-evaluation.pdf • CDC’s resource page on program evaluation and logic model development http://www.cdc.gov/eval/resources/index.htm • CSAP Pathways Course Evaluation 101 http://pathwayscourses.samhsa.gov/eval102/eval102_1_pg2.htm • Evaluator’s Institute http://tei.gwu.edu/ • GAO’s Designing Evaluations http://www.gao.gov/products/GAO-12-208G • GAIN Program Management and Evaluation Training (PMET) http://www.gaincc.org/products-services/training/gain-program-management-and-evaluation-training/ • NIAAA’s State-of-the-art methodologies in alcohol-related health services research http://onlinelibrary.wiley.com/doi/10.1111/add.2000.95.issue-11s3/issuetoc • NIDA’s Blue Ribbon Task Force on Health Services Research www.drugabuse.gov/sites/default/files/files/HSRReport.pdf • NSF’s User Friendly Handbook http://www.nsf.gov/pubs/2002/nsf02057/start.htm • SAMHSA Center for Behavioral Health Statistics and Quality (CBHSQ) national data sets with information on need http://www.samhsa.gov/data/ • SAMHSA NREPP’s Non-Researcher's Guide to Evidence-Based Program Evaluation http://nrepp.samhsa.gov/Courses/ProgramEvaluation/NREPP_0401_0010.html • SAMHSA TIP 14: State Outcomes-Monitoring Systems for Alcohol and Other Drug Abuse Treatment http://store.samhsa.gov/product/TIP-14-State-Outcomes- • Monitoring-Systems-for-Alcohol-and-Other-Drug-Abuse-Treatment/BKD162