Developing and Implementing Multi-Level Program Evaluation Plans for SAT-ED Grants

1. Developing and Implementing Multi-Level Program Evaluation Plans for SAT-ED Grants 3/11/2013 Michael L. Dennis, Chestnut Health Systems. Normal, IL Available from www.gaincc.org/presentations Created for: Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Treatment (CSAT) under contract number HHSS283200700003I, Task Order HHSS28300002T

2. Goals for the Presentation?? • Summarize the key problems in our field that SAT-ED is attempting to address • Review objectives, key questions and sources of data to be addressed in the evaluation • Identify key steps in designing, implementing and using evaluation to help manage and improve programs • Discuss strategies for reliable, valid, and efficient collection and analysis of state (including commonwealth), site, and client-level data • Provide links to further resources and training

3. Objectives of SAT-ED To improve treatment for adolescents through the • Development of a Learning Laboratorywith collaborating local community-based treatment provider sites. • Improvements in State level infrastructure through workforce development, financial planning, licensure and certification • Improvement of Site level infrastructure through implementation of Evidence-Based Practice (EBP) related to assessment and treatment • Assessment, treatment and monitoring of change at the client level. Feds have to be able to describe what was done with the money State and sites have to decide what to try and sustain

4. Typical Components of a Multi-Level Evaluation Plan • Needs assessment • Description of program activities, Theory of Change and/or Logic Model • Approach to stakeholders • Evaluation questions, data sources and methodology • Performance monitoring and reporting TIP: Labels and the order of components can vary to fit your situation, the point here is really to make sure that you have them covered or that your team makes an informed decision not to address them

5. 1. Needs Assessment • Description of infrastructure and site level needs and what information is still needed • Local system and/or cultural consideration • Articulate the rationale for the selection of the targeted • Infrastructure activities • Site selection • Evidence-based assessment selection • Evidence-based treatment selection

6. Structural Challenges to Delivery of Quality Care in Behavioral Health Systems High turnover workforce with variable education background related to diagnosis, placement, treatment planning and referral to other services Heterogeneous needs and severity characterized by multiple problems, chronic relapse, and multiple episodes of care over several years Lack of access to or use of data at the program level to guide immediate clinical decisions, billing and program planning Missing, bad or misrepresented data that needs to be minimized and incorporated into interpretations Lack of infrastructure that is needed to support implementation and fidelity of evidence-based practices that have been shown to work better on average

7. Substance Use Disorder & Treatment by Age 7 Higher rates of unmet need for adolescents and young adults Higher rates of need for young adults Source: SAMHSA 2009 National Survey on Drug Use and Health

8. Substance Use Disorder & Treatment by Age 8 Completion rates are lower for adolescents and young adults Lengths of stay are shorter for young adults and adults Source: SAMHSA 2009 Treatment Episode Data Set – Discharges (TEDS-D)

9. No Self-Help Group Participation in the First 3 Months of Treatment Age* Higher adolescents and young adults * p<.05 SAMHSA 2011 GAIN SA Data Set subset to has 3m Follow up (n=21,228)

10. Unmet Need for Mental Health Treatment by 3 Months Age* Higher for adolescents and young adults * p<.05 SAMHSA 2011 GAIN SA Data Set subset to has 3m Follow up (n=14,358)

11. Unmet Need for Medical Treatment by 3 Months Age* Higher for Young Adults * p<.05 SAMHSA 2011 GAIN SA Data Set subset to has 3m Follow up (n=8,517)

12. 2. Description of Program Activities, Theory of Change and/or Logic Model • Describe infrastructure and site level activities to be conducted and any specific programs or evidence-based practices you plan to use • Theory or logic model for each need and how they will be addressed by the activity and the expected outcome • Discuss relationship between various needs, activities or components, including how state and site level activities support each other

13. Expected State-Level Infrastructure Activity • Interagency workgroup to improve the statewide infrastructure for adolescent substance abuse treatment and recovery • Memoranda of understanding between SAT-ED awardee agency and other child-serving agencies • Multi-year workforce training plan for specialty adolescent behavioral health (substance use disorder/co-occurring substance use and mental disorder) treatment/recovery sector and other child-serving agencies • Comprehensive and integrated continuum of care for adolescents with substance use and mental health disorders in terms of both funding and services

14. Expected State-Level Infrastructure Activity (continued) • Financial mapping to understand current funding and coverage • Coordination of funding to make the system more efficient, expand coverage and shift towards more effective practices • Facilitation of a learning laboratory to use above to identify target areas of need, attempt change, evaluate the change, and if necessary adjust strategies to improve the quality of care TIP: Can relate to and/or build on activities already under way. You just want to be sure that you will be prepared to address each area in your annual and final progress reports

15. Other Allowable State-Level Infrastructure Activity • Workforce mapping to understand qualifications of staff across the continuum of care and the adequacy of the initial training/ continuing education infrastructure already in place • College, university and continuing education staff and programs/faculty infrastructure improvements/expansions and number of new/existing staff trained • Other state wide events to provide continuing or community education/training • Reviewing/revising PROGRAM standards for licensure, certification, and/or accreditation of programs that provide substance use and co-occurring mental disorders services for adolescents and their families • Reviewing/revising CLINICAN standards for licensure, certification, and/or credentialing of clinicians that provide substance use and co-occurring mental disorders services for adolescents and their families

16. Other Allowable State-Level Activity (continued) • Family / youth support organization creation, expansion, continuation, or enhancement • People newly credentialed/certified to provide substance use and co-occurring substance use & mental health disorders • Policy changes made as a result of the cooperative agreement • Financing policy changes completed as a result of the cooperative agreement TIP: Choose what makes sense for your needs and proposed activities. Invest more in measuring those areas where you are focusing your resources and attention. There is less interest in the average than identifying and understanding one or more areas where grantees that have done something they found useful.

17. Expected Site-Level Infrastructure Activity • Collaborating sites you have contracting with to provide evidence practice practices (EBP) • EBP related to a) assessment and b) treatment for which you have contracting to obtain at training and technical support to implement • EBP training type, date, and number staff attending each • EBP proficient staff capacity in terms of the number of employed staff who are certified by level and type of EBP • EBP local trainer or supervisory capacity in terms of the number of employed staff who are certified by level and type of EBP train and supervise new staff

18. Other Optional Site-Level Infrastructure Activity • Implementation of EBP related to assessment in terms of the number completed, linkage to medical records, use of clinical decision support, use for program planning (aka meaningful use) • Expansion of coverage based on number and percent of assessed youth receiving any services billed to insurance (Medicaid, CHIP, other federal/state, other private) instead of the block grant • Implementation of EBP related to treatment in terms of the number clients receiving it and receiving target dosage. TIP: Be sure to think about how to describe, measure and demonstrate a relationship between state and site level activities related to the chosen EBP. Collaborate with other state using the same EBP

19. Comparison of Site EBP for Assessment TIP: Most states/sites have other electronic or hard copy records and have mentioned additional measures in their proposal or preliminary evaluation plans; Also, most sites are still the process of deciding whether to conduct follow-up with EBP or other measures beyond GPRA

20. Comparison of Site EBP for Treatment TIP: Several states have talked about comparing to other EBP within their state, comparing to the same EBP in other sites , and/or expanding EBP to other sites.

21. 3. Approach to Stakeholders • Identification of state, site, community and individual (youth, family) level state holders • Coordination with or creation of strategic planning groups or interagency councils • Coordination with electronic medical and billing records • Involvement of program directors, information technology staff, clinical directors, supervisors, line staff • Coordinating with or creation of community, family and/or youth advisory groups or partnerships

22. Questions for Stakeholders • Key needs or problems with the current system that might be addressed • Critical timelines, measures and products that would make it more useful to them • Recognizing how they define & measure things and where multiple definitions or measures may be needed across stakeholders • What will it take for them support sustainability beyond the grant?

23. Identifying & Address Key Subgroups That May Have Concerns or Barriers to Accessing Services • Demographic groups (e.g., by gender, race, ethnicity, age, sexual orientation) • Abilities (e.g., hearing, sight, mobility, IQ) • Clinical subgroups such as • Primary substance, • Co-occurring mental health/trauma/suicide • Crime/violence or justice involvement • Degree of family support and use • Insurance, transportation or economic TIP: Health disparities need to be treated similar to safety issues – where best practice is to diligently look for them and work towards reducing them where ever possible in order to improve effectiveness and reduce liability

24. 4. Evaluation Questions, Data Sources & Methodology • Operationalizing the program objectives/questions into activities, measures of implementation/outputs and outcomes, including the frequency of collection and data sources • Working backwards to make sure that the above cross walk maps onto actual contracts, memos of understanding, and/or expectations of all stakeholders (many of which are developed at different points in the proposal and start up process)

25. State/Site Level Infrastructure • Often a matter of documenting what has been done, including dates, type and events, number of staff, degree of completion/certification • Dual Diagnosis Capability in Addiction Treatment (DDCAT) and Dual Diagnosis Capability Youth Treatment Tool (DDCYT) measures of availability and quality of co-occurring services • Identifying how things differ from what was expected, including • Unexpected problems and how they were addressed • Unexpected opportunities and how they were seized • Things that still need to be or might be done

26. Common Client Level Questions • What are the characteristics and needs of who was served? • What services did they receive? • To what extent are services targeted at the most appropriate for severe clients • To what extent are services effective? • Are the services cost effective? TIP: Not every evaluation will address each of these questions or each question equally well. The point here is to think about how and how well you will be able to answer each.

27. Characteristics and needs of who was served?

28. What services did they receive? TIP: Without GAIN/CASI follow-up, you will be very dependent on the quality of and access to records. With them need to cover first 3 months to describe most of treatment.

29. To what extent are services targeted at the most appropriate or severe clients • Implementation of reliable, valid and efficient measures of need and severity • Consensus standards on definition of need, link to services and/or evidence-based practices associated with better outcomes on average • Implementation of clinical decision support and meaningful use to drive actual treatment planning and services • Evaluation of treatment need profiles, gaps and health disparities and the program level and monitoring of change over time

30. To what extent are services effective? • Improvements in administrative outcomes (e.g., initiation, engagement, continuing care, evidence-based practices*) associated with better outcomes on average • Participation in self help and recovery support services • Among those in need, receipt of services related to co-occurring mental health & physical health problems* • Pre-post change in percent of past month abstinence, no substance related problems, no justice involvement, being housed, vocational engagement and social connectedness • Comparison of the same program over time, across sites, to other programs, national norms, or standards (ideally matched programs or clients)* TIP: * These have to come from records or supplemental data such as follow-up data.

31. Are the services cost effective? • Estimate costs of average services and evidence-based practices using accounting data* • Compare costs to state-wide, federal or published normative costs overall or adjusting for improved retention* • Putting costs in context relative to baseline costs to society of health care utilization or crime and the extent to which the program is targeting a high cost subgroup • Pre-post change in the cost to society of health care utilization or crime * TIP: * These have to come from records, follow-up or other supplemental data such as follow-up data.

32. 5. Performance Monitoring and Reporting • Early indicators of implementation, fidelity and steps of the theory of change or logic model • Important for infrastructure measures to include necessary steps (e.g., selection, contracting, events, people, evaluations) • Client level measures related to • Recruitment and data collection rate/target, being on time • Casemix of who is served • Treatment initiation, engagement, continuing care, satisfaction • Fidelity of EBP • Services targeted at needs

33. Implementation is Essential (Reduction in Recidivism from .50 Control Group Rate) The best is to have a strong program implemented well The effect of a well implemented, weak program is as big as a strong program implemented poorly Thus, one should optimally pick the strongest intervention that one can implement well Source: Adapted from Lipsey, 1997, 2005 meta analysis of 509 juvenile justice programs

34. What gets measured, gets doneWhat gets fed back, gets done betterWhat gets incentivized, gets done more often Average practice based on TEDS *Based on a count of initiation within 14 days, evidence based practice, engagement for at least 6 weeks, and any continuing care. Source: CSAT 2011 AT SA Data Set subset to 1+ Follow ups (n=17,202)

35. Selected NOMS Outcomes Over Time Most effects are in the first 90 days, important to measure outcome and services received by then Variation in outcomes *Interpolated **Past month Source: CSAT 2011 AT SA Data Set subset to 1+ Follow ups

36. NOMS Outcome Status at Last Wave Measure favors people who come in the door without problems *This variable measures the last 30 days. All others measure the past 90 days. **The blue bar represents an increase of 50% or no problem. Source: CSAT 2011 AT SA Data Set subset to 1+ Follow ups

37. *Based on count of a reduction in the following variables: Substance use frequency, Abuse/Dependence Sx (past 30d), Physical Health (past 90d), Mental Health (past 90d), Nights of Psychiatric Inpatient (past 90d), Illegal Activity (past 90d), Arrests (past 90d), Housed in Community (past 90d), Family/Home Problems (past 90d), Vocational Problems (past 30d), Social Support/Engagement (past 90d), Recovery Environment Risk (past 90d), Quarterly Cost to Society (past 90d), In Work/School (past 90d) Minus No problems at intake with these variables NOMS Outcomes: Count of Positive Outcomes* (Status at Last Follow up – Status at Intake) 78% have one or more improved areas Source: CSAT 2011 AT SA Data Set subset to 1+ Follow ups (n=17,722)

38. Health Care Utilization Cost 11% of youth consume 76% of health care costs Source: CSAT 2011 AT Summary Analytic Data Set (n=19,148)

39. Cost of Crime 21% of youth consume 97% of health care costs Source: CSAT 2011 AT Summary Analytic Data Set (n=17,878)

40. Reduction in Health Care utilization off set the cost of SUD Treatment within 12 months \a Includes the cost of treatment \b Year after intake (including treatment) minus year before treatment

41. EBP like A-CRA Cost more but Produce Greater Savings too \a Includes the cost of treatment \b Year after intake (including treatment) minus year before treatment

42. Impact of Reclaiming Futures Infrastructure Enhancements to Juvenile Treatment Drug Court on Cost of Crime to Society \a RF-JTDC is significantly lower at follow-up than JTDC. Source: Dennis et al 2012

43. Other Evaluation Training Resources • ACYF’s The Program Manager's Guide to Evaluation http://www.acf.hhs.gov/programs/opre/research/project/the-program-managers-guide-to-evaluation • American Evaluation Association http://www.eval.org/ • BJA’s Program Evaluation Manual https://www.bja.gov/evaluation/guide/bja-guide-program-evaluation.pdf • CDC’s resource page on program evaluation and logic model development http://www.cdc.gov/eval/resources/index.htm • CSAPPathways Course Evaluation 101 http://pathwayscourses.samhsa.gov/eval102/eval102_1_pg2.htm • Evaluator’s Institute http://tei.gwu.edu/ • GAO’s Designing Evaluations http://www.gao.gov/products/GAO-12-208G • GAIN Program Management and Evaluation Training (PMET) http://www.gaincc.org/products-services/training/gain-program-management-and-evaluation-training/ • NIAAA’s State-of-the-art methodologies in alcohol-related health services research http://onlinelibrary.wiley.com/doi/10.1111/add.2000.95.issue-11s3/issuetoc • NIDA’s Blue Ribbon Task Force on Health Services Research www.drugabuse.gov/sites/default/files/files/HSRReport.pdf • NSF’s User Friendly Handbook http://www.nsf.gov/pubs/2002/nsf02057/start.htm • SAMHSA Center for Behavioral Health Statistics and Quality (CBHSQ) national data sets with information on need http://www.samhsa.gov/data/ • SAMHSA NREPP’s Non-Researcher's Guide to Evidence-Based Program Evaluation http://nrepp.samhsa.gov/Courses/ProgramEvaluation/NREPP_0401_0010.html • SAMHSA TIP 14: State Outcomes-Monitoring Systems for Alcohol and Other Drug Abuse Treatment http://store.samhsa.gov/product/TIP-14-State-Outcomes- Monitoring-Systems-for-Alcohol-and-Other-Drug-Abuse-Treatment/BKD162

44. Key Points for Breakout • For each area of required activity and the allowable ones you have chosen to target, • What is your rationale/evidenced of need? • What activity will address it? • What are the expected outcomes of doing this? • How will you document and monitor implementation in real time? • Prioritize those the areas in terms of those that • Are your focus or key for sustainability • Likely to be difficult or require help • What needs to be done to finalize the plan before it is submitted later in March