Evidence Based Practice What is it? Why do it? Points to consider

1. Evidence Based Practice – An OverviewWebinar for Reclaiming Futures October 23, 2008Randolph Muck, M.Ed.CSAT/SAMHSAContact Info: randy.muck@samhsa.hhs.gov240-276-1576

2. Evidence Based Practice What is it?Why do it?Points to consider

3. Evidence Based Practice The term evidence-based practice (EBP) refers to preferential use of mental and behavioral health interventions for which systematic empirical research has provided evidence of statistically significant effectiveness as treatments for specific problems. Alternate terms with the same meaning are evidence-based treatment (EBT) and empirically-supported treatment (EST).

4. Evidence Based Practice Tested with good outcomesManual exists so it can be replicated/trainedA training program existsSupervision leading to certificationOngoing monitoring Outcomes measurement

5. Ways of Viewing EBP • EBP is a process. EBP is a way of doing practice that integrates the best evidence with clinical expertise and consumer values. (EBP as a verb.) (Sackett et al., 2000) Practitioner Expertise Best Evidence EBP Client Values & Preferences

6. Ways of Viewing EBP • EBP is a product. An evidence-based practice is any practice that has been established as effective through scientific research according to some set of explicit criteria. (EBP as a noun.) (Drake, 2001) • EB Interventions. (A-CRA, MET/CBT5) • EB Skill sets. (CBT, Behavioral Parent Training)

7. Definition of Implementation “…Specified set of activities designed to put into practice an activity or program of known dimensions…such that independent observers can detect its presence and strength.” (Fixsen et al, 2004, p. 5)

8. Definition of Fidelity • Strategies used to monitor the faithful delivery of a manual-guided behavioral intervention • Important dimensions include • adherence (i.e., extent to which intervention procedures were delivered as prescribed in the treatment manual) • competence (i.e., qualitative measure of the skillfulness in which intervention procedures are delivered)

9. Different Types of Manuals • Session Driven • Procedure Driven • Principle Driven

10. Randomized Clinical Trials (RCT) are to Evidence Based Practice (EBP) like Self-reports are to Diagnosis • They are only as good as the questions asked (and then only if done in a reliable/valid way) • They are an efficient and logical place to start • But they can be limited or biased and need to be combined with other information • Just because the person does not know something (or the RCT has not be done), does not mean it is not so • Synthesizing them with other information usually makes them better

11. Context • The field is increasingly facing demands from payers, policymakers, and the public at large for “evidence-based practices (EBP)” which can reliably produce practical and cost-effective interventions, therapies and medications that will • reduce risks for initiating drug use among those not yet using, • reduce substance use and its negative consequences among those who are abusing or dependent, and • reduce the likelihood of relapse for those who are recovering NIDA Blue Ribbon Panel on Health Services Research (see www.nida.nih.gov )

12. So what does it mean to move the field towards Evidence Based Practice (EBP)? • Introducing reliable and valid assessment that can be used • At the individual level to immediately guide clinical judgments about diagnosis/severity, placement, treatment planning, and the response to treatment • At the program level to drive program evaluation, needs assessment, and long term program planning • Introducing explicit intervention protocols that are • Targeted at specific problems/subgroups and outcomes • Having explicit quality assurance procedures to cause adherence at the individual level and implementation at the program level • Having the ability to evaluate performance and outcomes • For the same program over time, • Relative to other interventions

13. The Current Renaissance of Adolescent Treatment Research * Published and publicly available

14. Issues to Consider • Juvenile Justice involved youth increasing presence in the treatment system • Support for funding relies on ability to demonstrate effectiveness • Treatment needs of the youth that we see and the need to incorporate appropriate and effective interventions for these needs • Continuing Care is as or more important than the treatment delivered

15. 114% 115% Other sources of Referral have grown, but less than expected 61% growth 41% 37% 37% 5% 12% Change in Referral Sources: 1993-2003 90,000 140% JJ referrals have doubled, are 53% of 2003 admissions and driving growth 80,000 120% 70,000 100% 60,000 80% 50,000 40,000 60% 30,000 40% 20,000 20% 10,000 - 0% 1993 School Other Self/Family Community 2003 Agency Care Other SA Tx Other Health Employee/EAP Juvenile Justice Change Source: Treatment Episode Data Set (TEDS) 1993-2003.

16. Despite being widely recommended, only 10% step down after intensive treatment 53% Have Unfavorable Discharges Source: Data received through August 4, 2004 from 23 States (CA, CO, GA, HI, IA, IL, KS, MA, MD, ME, MI, MN, MO, MT, NE, NJ, OH, OK, RI, SC, TX, UT, WY) as reported in Office of Applied Studies (OAS; 2005). Treatment Episode Data Set (TEDS): 2002. Discharges from Substance Abuse Treatment Services, DASIS Series: S-25, DHHS Publication No. (SMA) 04-3967, Rockville, MD: Substance Abuse and Mental Health Services Administration. Retrieved from http://wwwdasis.samhsa.gov/teds02/2002_teds_rpt_d.pdf .

17. Median Length of Stay is only 50 days Median Length of Stay Total 50 days (61,153 discharges) Less than 25% stay the 90 days or longer time recommended by NIDA Researchers LTR 49 days (5,476 discharges) STR 21 days (5,152 discharges) Level of Care Detox 3 days (3,185 discharges) IOP 46 days (10,292 discharges) Outpatient 59 days (37,048 discharges) 0 30 60 90 Source: Data received through August 4, 2004 from 23 States (CA, CO, GA, HI, IA, IL, KS, MA, MD, ME, MI, MN, MO, MT, NE, NJ, OH, OK, RI, SC, TX, UT, WY) as reported in Office of Applied Studies (OAS; 2005). Treatment Episode Data Set (TEDS): 2002. Discharges from Substance Abuse Treatment Services, DASIS Series: S-25, DHHS Publication No. (SMA) 04-3967, Rockville, MD: Substance Abuse and Mental Health Services Administration. Retrieved from http://wwwdasis.samhsa.gov/teds02/2002_teds_rpt_d.pdf .

18. Past 90 day HIV Risk Behaviors Source: CSAT AT Outcome Data Set (n=9,276 adolescents)

19. Recovery Environment Source: CSAT AT Outcome Data Set (n=9,276 adolescents)

20. Co-Occurring Psychiatric Problems Source: CSAT AT Outcome Data Set (n=9,276 adolescents)

21. Past Year Violence & Crime *Dealing, manufacturing, prostitution, gambling (does not include simple possession or use) Source: CSAT AT Outcome Data Set (n=9,276 adolescents)

22. Multiple Problems* are the Norm 100% In fact, over half present acknowledging 5+ major problems 90% 80% Five to Twelve 70% 60% 50% 40% Four 30% Few present with just one problem (the focus of traditional research) Three Most acknowledge 1+ problems 20% Two 10% One None 0% * (Alcohol, cannabis, or other drug disorder, depression, anxiety, trauma, suicide, ADHD, CD, victimization, violence/ illegal activity) Source: CSAT AT Common GAIN Data set

23. No. of Problems* by Severity of Victimization 100% Those with high lifetime levels of victimization have 117 times higher odds of having 5+ major problems* 90% 80% 70% 60% 50% Five or More Four 40% Three 30% Two 20% One None 10% * (Alcohol, cannabis, or other drug disorder, depression, anxiety, trauma, suicide, ADHD, CD, victimization, violence/ illegal activity) 0% Low (31%) Moderate (17%) High (51%) GAIN General Victimization Scale Score (Row %) Source: CSAT AT Common GAIN Data set (odds for High over odds for Low)

24. Most Lack of Standardized Assessment for… • Substance use disorders (e.g., abuse, dependence, withdrawal), readiness for change, relapse potential and recovery environment • Common mental health disorders (e.g., conduct, attention deficit-hyperactivity, depression, anxiety, trauma, self-mutilation and suicidality) • Crime and violence (e.g., inter-personal violence, drug related crime, property crime, violent crime) • HIV risk behaviors (needle use, sexual risk, victimization) • Child maltreatment (physical, sexual, emotional)

25. Summary of Problems in the Treatment System • The public systems is changing size, referral source, and focus – often in different directions by state • Major problems are not reliably assessed (if at all) • Less than 50% stay 50 days (~7 weeks) • Less the 25% stay the 3 months recommended by NIDA researchers • Less than half have positive discharges • After intensive treatment, less than 10% step down to outpatient care • While JJ involvement is common, little is known about the rate of initiation after detention

26. EBPs and Treatment for Youth in the Juvenile Justice System

27. Some Programs Have Negative or No Effects on recidivism • “Scared Straight” and similar shock incarceration program • Boot camps mixed – had bad to no effect • Routine practice – had no or little (d=.07 or 6% reduction in recidivism) • Similar effects for minority and white (not enough data to comment on males vs. females) • The common belief that treating anti-social juveniles in groups would lead to more “iatrogenic” effects appears to be false on average (i.e., relapse, violence, recidivism for groups is no worse then individual or family therapy) Source: Adapted from Lipsey, 1997, 2005

28. Meta Analysis of the Effectiveness of Programs for Juvenile Offenders N of Offender Sample Studies Preadjudication (prevention) 178 Probation 216 Institutionalized 90 Aftercare 25 Total 509 Source: Adapted from Lipsey, 1997, 2005

29. Most Programs are actually a mix of components Average of 5.6 components distinguishable in program descriptions from research reports Intensive supervision Prison visit Restitution Community service Wilderness/Boot camp Tutoring Individual counseling Group counseling Family counseling Parent counseling Recreation/sports Interpersonal skills Anger management Mentoring Cognitive behavioral Behavior modification Employment training Vocational counseling Life skills Provider training Casework Drug/alcohol therapy Multimodal/individual Mediation Source: Adapted from Lipsey, 1997, 2005

30. Most programs have small effectsbut those effects are not negligible • The median effect size (.09) represents a reduction of the recidivism rate from .50 to .46 • Above that median, most of the programs reduce recidivism by 10% or more • One-fourth of the studies show recidivism reductions of 30% or more • The “nothing works” claim that rehabilitative programs for juvenile offenders are ineffective is false Source: Adapted from Lipsey, 1997, 2005

31. Major Predictors of Bigger Effects • Chose a strong intervention protocol based on prior evidence • Used quality assurance to ensure protocol adherence and project implementation • Used proactive case supervision of individual • Used triage to focus on the highest severity subgroup

32. Impact of the numbers of Favorable features on Recidivism (509 JJ studies) Usual Practice has little or no effect Source: Adapted from Lipsey, 1997, 2005

33. Program types with average or better effects on recidivism AVERAGE OR BETTERBETTER/BEST Preadjudication Drug/alcohol therapy Interpersonal skills training Parent training Employment/job training Tutoring Group counseling Probation Drug/alcohol therapy Cognitive-behavioral therapy Family counseling Interpersonal skills training Mentoring Parent training Tutoring Institutionalized Family counseling Behavior management Cognitive-behavioral therapy Group counseling Employment/job training Individual counseling Interpersonal skills training Source: Adapted from Lipsey, 1997, 2005

34. Cognitive Behavioral Therapy (CBT) Interventions that Typically do Better than Practice in Reducing Recidivism (29% vs. 40%) • Aggression Replacement Training • Reasoning & Rehabilitation • Moral Reconation Therapy • Thinking for a Change • Interpersonal Social Problem Solving • Multisystemic Therapy • Functional Family Therapy • Multidimensional Family Therapy • Adolescent Community Reinforcement Approach • MET/CBT combinations and Other manualized CBT NOTE: There is generally little or no differences in mean effect size between these brand names Source: Adapted from Lipsey et al 2001, Waldron et al, 2001, Dennis et al, 2004

35. Implementation is Essential (Reduction in Recidivism from .50 Control Group Rate) The best is to have a strong program implemented well The effect of a well implemented weak program is as big as a strong program implemented poorly Thus one should optimally pick the strongest intervention that one can implement well Source: Adapted from Lipsey, 1997, 2005

36. ACRA did better than MET/CBT5, and both did better than MDFT MET/CBT5 and 12 did better than FSN Moderate to large differences in Cost-Effectiveness by Condition Trial 2 Trial 1 $20 $20,000 $16 $16,000 $12 $12,000 Cost per person in recovery at month 12 over 12 months Cost per day of abstinence $8 $8,000 $4 $4,000 $0 $0 MET/ MET/ CBT5 FSN MET/ CBT5 ACRA MDFT CBT12 $4.91 $6.15 $15.13 $9.00 $6.62 $10.38 CPDA* $3,958 $7,377 $15,116 $6,611 $4,460 $11,775 CPPR** * p<.05 effect size f=0.48 * p<.05 effect size f=0.22 ** p<.05, effect size f=0.72 ** p<.05, effect size f=0.78 Source: Dennis et al., 2004

37. Choosing an EBP • Best evidence • Practitioner experience • Youth/Family values and preferences • Readiness for change (buy-in at all levels of agency, but particularly the top) • Cost/Resources • Ability to implement well

38. What are the pitfalls of EBP? • EBP generally causes some staff turnover • EBP often shines a light on staff or work place problems that would otherwise be ignored • EBP often impact a wide range of existing procedures and policies – requiring modification and provoking resistance • EBP (and most organizational changes) will fail without good senior staff leadership • EBP typically require going for more funds from grant or other funders • On-going needs assessment will create demand for more change and more EBP

39. A Fearless Appraisal… • We are entering a renaissance of new knowledge in this area, but are only reaching 1 of 10 in need • Several interventions work, but 2/3 of the adolescents are still having problems 12 months later • Effectiveness is related to severity, intervention strength, implementation/adherence, and how assertive we are in providing treatment • As other therapies have caught up technologically, there is no longer the clear advantage of family therapy found in early literature reviews • While there have been concerns about the potential iatrogenic effects of group therapy, the rates do not appear to be appreciably different from individual or family therapy if it is done well (important since group tx typically costs less) • Effectiveness was not consistently associated with the amount of therapy over a short period of time (6-12 weeks) but was related to longer term continuing care

40. Common Strategies you can do NOW • Standardize assessment and identify most common problems • Pool knowledge about what staff have done in the past, whether it worked, and what the barriers were • Identify system barriers (e.g., criteria to local access case management, mental health) that could be avoided if thought of in advance • Identify existing materials that could help and make sure they are readily available on site • Identify promising strategies for working with the adolescent, parents, or other providers • Develop a 1-2 page checklist of things to do when this problem comes up • Identify a more detailed protocol and trainer to address the problem, then go for a grant to support implementation

41. Evidenced Based Practice - Summary • Achieving reliable outcomes requires reliable measurement, protocol delivery and on-going performance monitoring. • The GAIN is one measure that is being widely used by CSAT grantees and others trying to address gaps in current knowledge and move the field towards evidenced based practice. • Standardized and more specific assessment helps to draw out treatment planning implications of readiness for change, recovery environment, relapse potential, psychopathology, crime/violence, and HIV risks. • Adolescents entering more intensive levels of care typically have higher severity. • Multiple problems and child maltreatment are the norm and are closely related to each other. • There is a growing number of standardized assessment tools, treatment protocols and other resources available to support evidenced based practices.