Leonard Bickman and Manuel Riemer Center for Mental Health Policy Vanderbilt University

1. Reforming Children’s Mental Health Services – Lessons Learned from the Past and Suggestions for the Future Leonard Bickman and Manuel Riemer Center for Mental Health Policy Vanderbilt University

2. Presentation Overview Lessons Learned from the Past: Children’s Mental Health Reform in the US Professionalism System Level Evidence Based Practice Suggestions for the Future: CQI and the Role of Measurement and Feedback Measurement Feedback Intervention

3. Learning from History - A Critical Evaluation of Three Major Approaches to Children’s Mental Health Reform

4. Defining the Problem Two reports, almost 20 years past by, but … J. Knitzer, 1982: Unclaimed Children • Limited access • Little funding • Little coordination across agencies • Very few staff • Inappropriate services • Provides little information about effectiveness of services Surgeon General, 1999: Report on Mental Health • More funding • More coordination • More staff • More and different services • Limited access • Still, we know little about effectiveness of services

5. Purpose of this Analysis Analyze 3 main reformmodels to find leveragepoints for improvement Reform

6. Professionalism System Level Evidence-BasedPractice Three Approaches to Reform Clinical Outcome SYSTEM PROFESSION TREATMENT

7. PROFESSION Professionalism Professionalism Clinical Outcome Professionalism:The conduct, aims, or qualities that characterize or mark a profession or a professional person(Merriam Webster Dictionary, 2001)

8. Professionalism: Approaches 1) Instruction • 3) Selection Accreditation of providers Preservice training Inservice training Licensing of clinicians Supervision 2) Experience Requirement of practice experience

9. PROFESSION Professionalism Is Professionalism Effective in Improving Clinical Outcomes? ? Clinical Outcome

10. Finding the Evidence • Instruction: • Preservice Training: • Christensen & Jacobson (1994) • Stein & Lambert (1995) • Shadish et al. (1993) • Inservice Training: • Beaudry (1989) • Davis (1998); Davis et al. (1992); Davis et al. (1995); • Bickman (1999) • Supervision: • Holloway & Neufeldt (1995) • Steinhelber et al. (1984) • Ellis et al. (1996)

11. Finding the Evidence (cont.) • Experience: • Sechrest, Gallimore, & Hersch (1967) • Dawes (1994) • Luborsky et al. (1980) • Smith & Glass (1977) • Stein & Lambert (1984) • Strupp & Hadley (1979) • Dush, Hirt, & Schroeder (1989) • Lyons & Woods (1991) • Stein & Lambert (1995)

12. Finding the Evidence (cont.) • Selection: • Accreditation: • Hadley & McGurrin (1988) • Bravo et al. (1999) • Licensing: • 173 publications on licensing (1967–1999) but no investigation of association between licensure and quality of professional services

13. PROFESSION Professionalism Results • Currently there is not enough empirical evidence to make reliable conclusion whether professionalism is effective or not ? Clinical Outcome

14. Profes- sionalism What are the Logic and the Underlying Assumptions of Professionalism?

15. Educator/supervisor knows how to teach best practice • Clinicians’ characteristics • Environmental characteristics: • Organization • System • Society Educator/supervisor knows best practice Instruction Transportability Assessment & Diagnosis Clinician receives instructions about best practice and how to apply it Clinician Is in the position to learn best practice Clinician is motivated to use best practice Clinician Is capable of using best practice Clinician uses best practice Clinician can match best practice to client problem Improved clinical outcome Best practice has been empirically identified to be effective as well as efficacious Professional’s and organizational context Evidence base of treatment

16. Clinician • knows outcome of treatment • receives quick, continuous, and correct feedback • is aware of the processes, tactics, or strategies of their treatment • knows the relationships between processes and outcomes • is able to contextualize their knowledge to fit individuals • is able to generalize from individual cases to establish principles • is able to apply the knowledge • Clinician is motivated • to change his/her behavior • organizational barriers Experience Clinician works in a clinical setting Clinician obtains experience Clinician learns Clinician positively changes his/her behavior Improved clinical outcome Professional’s andorganizational context and conditions Process of learning through experience

17. Authority has empirical evidence what kind of organizational or clinician characteristics are ineffective or bad Note: Conditions under which the bad apple strategy is effective are unclear Selection Authority makes selection Ineffective or bad organizations or clinicians are sorted out The remaining group of organizations and clinicians are more effective or better Improved clinical outcome Process of change

18. SYSTEM PROFESSION Professionalism System Level Three Approaches to Reform Clinical Outcome System of care: Delivering coordinated but diverse services on an individualized basis using case management and interdisciplinary treatment teams to integrate and facilitate transition between services.

19. System Level: Approaches • Increasing access • Community-based services • Adding new services • Using the existing services more effectively • Organizing and systematizing • Cross-collaboration and coordination of services • Tailoring • Individualized services “wrapped” around the child and his/her family • Community-based services • Continuum of care • Diversification of services

20. SYSTEM System Level Is System Level Reform Effective in Improving Clinical Outcome? ? Clinical Outcome

21. Finding the Evidence • Fort Bragg Study (1995; Continuum of Care) • Stark County Study (1998; System of Care) • Wraparound Project Study (2001; Wraparound) • Comprehensive Community Mental Health System Reform (1993 – present; System of Care)

22. SYSTEM System Level Results • System changes have been shown to • Increase satisfaction • Increase access • Increase costs • But not improve clinical outcomes ? Clinical Outcome

23. SystemLevel What are the Logic and the Underlying Assumptions of System Level Reform?

24. Increasing Access Increase access to mental health services Services are used by children In need More children will receive services Improved clinical outcome Client Context Effective treatments have been identified Effective treatments are offered and implemented correctly by services Evidence base of treatment Transportability &Dissemination

25. The critical assumption is that saved resources are not invested elsewhere Same assumptions as before need to be met Organizing and Systematizing More children can be served Increased access Services are organized and systematized Services are used more efficiently Improved clinical outcome Costs are reduced Saved resources are invested into additional services Policy context

26. The correct treatment that matches the child need has been empirically identified Tailoring Assessment & Diagnosis Clinician is able to assess child’s need correctly Clinician is able to match the correct treatment with child’s need Services are tailored to child needs Child receives more appropriate care Improved clinical outcome Evidence base of treatment

27. SYSTEM PROFESSION TREATMENT Professionalism System Level Evidence-BasedPractice Three Approaches to Reform Clinical Outcome EBP: Use of scientifically evaluated treatments that havebeen shown to be efficacious and/or effective

28. EBP: Approaches • Setting standards • Finding and publishing empirically supported treatments (EST) • Efficacious studies • Effectiveness studies • Reviews and meta-analyses • Creating a (web-based) data base with ESTs • Developing manuals, guidelines, treatment algorithms

29. Clinical Outcome TREATMENT Evidence-BasedPractice Is EBP Effective in Improving Clinical Outcome? ?

30. Finding the Evidence • Standards: • APA’s Division 12 Task Force, Committee on Science & Practice (1995) • Interdisciplinary committee on Evidence-Based Youth Mental Health Care • FDA • International Psychopharmacology Algorithm Project

31. Finding the Evidence (cont.) • Reviews & Meta-Analyses: • APA’s Division 12 Review of evidence-based therapies (1998) • Kazdin, Psychotherapy for children and adolescents, Oxford (2000) • Weisz & Jensen, Mental Health Services Research (1999) • JAACAP special issue on psychopharmacology (1999) • Rones & Hoagwood, School-based mental health services,Clinical Child and Family Psychology Review (2000) • Burns, Hoagwood, Mrazek; Child Clinical and Family Psychology Review (2000) • Chorpita et al., Hawaii Emperical Basis to Services Task Force (2001) • Surgeon General’s Mental Health Report (1999) • Surgeon General’s Youth Violence Report (2001) • Surgeon General’s Supplement on Culture, Race, & Ethnicity (2001)

32. Finding the Evidence (cont.) • Web-Based Archives: • Cochrane Collaboration • Campbell Collaboration • Center for Evidence-Based Medicine • Center for Evidence-Based Mental Health • FOCUS • Society of Clinical Psychology

33. Evidence-BasedPractice Results ? ? • Limited number of efficacious treatments • Very small number of effective treatments • Almost no research findings on transportability, implementation, & dissemination ? Clinical Outcome ? TREATMENT

34. EBP What are the Logic and the Underlying Assumptions of EBP?

35. Diagnostic approach is the best match Transportability is good Criteria and standards for evidence are defined, valid, and agreed upon Processes and outcomes of therapeutic change are assessable with empirical methods EST are accepted by clinicians EST fit conditions of real world practice (e.g. comorbidity, feasibility) Process of therapeutic change is known EBP Evidence base of ESTs is developed EST’s are implemented in practice Clinicians provide more efficacious treatments Improved clinical outcome Measurement

36. What have We Learned? We need to pay more attention to the underlyingassumptions of our reform models: • What are the real world conditions and context (client, practitioner, organization, policy) • Is it enough to intervene at one level only (e.g. professionals, treatment, system, organization)? • What are the actual processes of change? • How do we improve transportability and dissemination? (e.g. involve practitioners and providers) • What are the processes of practitioners’ behavior change (such as learning through experience) • How can we match the right treatments to individual clients? • How can we assure correct assessment? • How can we develop valid measures for processes and outcomes?

37. Continuous Quality Improvement As a Reform – The Roles of Measurementand Feedback

38. Previous Reform Efforts Had Limitations • They did not make all their assumptions explicit • They did not involve all levels of an organization • Professionalism deals with clinician • System level reform deals with system • EBP focuses mainly on treatment • They are not focused on improving the quality of treatment • They are not sensitive to the real world • They are externally implemented and not internally innovative

39. Another Suggested Reform Implement continuous quality improvement (CQI) with an integrative concurrent consumer measurement system

40. Continuous Quality Improvement (CQI) • Originally developed for industrial and manufacturing applications • CQI involves the use of assessment, feedback, and application of information to improve services • CQI relies on a continuous evaluation of processes and outcomes • CQI involves a dynamic interplay of assessment, feedback, and application of information • Typically requires changes in the organization and thus is multilevel • CQI can empower organizations, clinicians and clients

41. A CQI Change Process Model System Organization DATA SERVICES • process • outcome INTERPRETATION ACTION • guidelines • training • feedback to practitioner • system modification

42. CQI Was Developed for Simple Processes • Highly sensitive process measures • Highly stabilized and replicable processes • Good ability to refine and “tweak” production processes • Relatively closed system - few external influences on processes and outcome • Few unexpected effects • Simple descriptive statistics • Strong causal linkage of inputs, process, outcome

43. CQI Is More Difficult in Human Services • Few sensitive process and outcome measures • No systematic feedback process • Highly variable and ill defined processes • Low service replicability • Clinician behavior difficult to “re-program” • Open system - multiple external influences on structure, process, and outcome • May be detrimental to program goals: • Gaming when a basis for rewards or accountability • Goal displacement

44. Little Experience With CQI • Limited experience in mental health • Exception is the Heidelberg-Stuttgart Model • Not a sufficient number of validated measures of quality exist • Clinician education and resistance are major factors • Resources seen as barrier to implementation • Few if any evaluations of CQI

45. What Does CQI Require? • That we understand the links between process of care and outcome • That we systematically collect data on these links and its elements • That we feedback how processes affect outcomes • That we use information to change clinician behavior • That we create an atmosphere of change that supports treatment based on data

46. A CQI Change Process Model System Organization Measurement DATA SERVICES • process • outcome INTERPRETATION ACTION • guidelines • training • feedback to practitioner • system modification • Feedback intervention

47. Measurementof Processes and Outcomes

48. Mental Health Measurement Needs For CQI • Sensitive, accurate, real time estimates of meaningful clinical change over time • A comprehensive and integrated system that uses consumer outcomes – e.g. symptoms, functioning • Sensitive and informative characterization of process indicators (therapeutic alliance, treatment modality, readiness to change, etc.) • Reduction in the paperwork burden for clinicians, youth and families

49. Components of the Child Adolescent Measurement System (CAMS) • Measures – concurrent and baseline • Functioning - strengths and impairment • Symptom severity • Hopefulness/satisfaction with life • Satisfaction with services • Victimization • Acuity • Substance use • Violent/aggressive behavior

50. Feedback Intervention Model to Change Clinician Behavior X M Y