Outcomes-Informed Care and Performance Management: Implications for Behavioral Healthcare Integration

1. Outcomes-Informed Care and Performance Management: Implications for Behavioral Healthcare Integration G.S. (Jeb) Brown, Ph.D. Center for Clinical Informatics

2. Sources of data • Information drawn from 5 performance management projects • Human Affairs International: 1996-1999 • PacifiCare Behavioral Health: 1999 – 2006 • Resources for Living: 2001-present • Accountable Behavioral Health Care Alliance: 2002 – present • Regence Group: 2007

3. Outcomes informed care • Use of patient self report outcomes questionnaires • Frequent administration in order to monitor patient response to treatment • Use of decision support tools to inform clinical judgment • Performance feedback to clinicians • Analysis of outcomes data to determine sources of variance: practice based evidence rather than evidence based practice • Use of practice based evidence to identify pathways to improved outcomes and to monitor success

4. Outcomes questionnaires • All patient self report outcome questionnaires tend to load on a common factor: “global distress” • Due to the high degree of correlation between items, well constructed questionnaires of 10-15 items can have coefficients of reliability and construct validity comparable to measures of 30 or more items. • Even ultra brief questionnaires of 4-9 items may have adequate reliability and validity for must measurement needs.

5. In the past week or two, how often did you Rarely Hardly Ever Some-times Often Very often 1 …feel unhappy or sad? 2 …have little or no energy? 3 …have a hard time controlling your temper? 4 …feel tense or nervous? 5 …have a hard time getting along with family or friends? Sample items Source: Regence Group outcomes informed care initiative

6. Meta-methods • Outcomes informed carerelies on findings derived from meta-analyses and related statistical analysis. • Use of practice generated data demands sophisticated statistical models for case mix adjustment. • Pursuit of sources of variance in outcomes results in use of hierarchical linear modeling to model variance at the clinician level. • Use of performance feedback to induce changes in clinician practice patterns resulting in improved outcomes across all diagnoses and treatment methods.

7. Common factors • The effectiveness of all treatments is due, in some part, to factors common to all treatments. • Contact with a helping, caring professional fosters hope and expectancy. • We have come to accept the potency of “placebo effects”, and insist that the effectiveness bona fide treatments exceeds that of placebo treatments

8. Randomized double-blind placebo controlled drug trials • Double blind placebo controlled drug studies provide an exemplar for estimating the role of common factors. • Meta-analysis of multiple studies of antidepressants lets us estimate the relative importance of common factors (placebo effects) versus drug effects. • Traditionally, the drug is interpreted as the difference between placebo and the active drug.

9. Meta-analyses and placebo • Meta-analysis involves the use of statistical techniques to combine results from multiple studies in order in an effort to generalize findings. • Meta-analysis of multiple studies of antidepressants let us estimate the relative importance of common factors (placebo effects) versus drug effects. 1-3

10. Drug effect accounted for 25% of measured improvement

11. Evidenced based psychotherapy • For several decades psychotherapy researchers have attempted to design randomly controlled trails (RCT) to investigate the effectiveness of specific methods of psychotherapy. • Study design analogous to pharmacy trials, except that designing credible “placebo treatments” is much more problematic. • Various treatment methods are being touted as “evidenced based” by citing the number of RCTs providing evidence that the treatment exceeded placebo (or some other treatment).

12. Psychotherapy “brands” • The advocacy for the use of specific therapies is analogous to the advertising of brands of antidepressant medication. • Calls for wide spread use of “evidence based treatments” in psychotherapy is analogous to the FDA’s insistence that a drug may not be marketed for the treatment of depression until at least two studies have shown superiority to placebo. • Advocates and practitioners of various “evidence based treatments” have a vested interest in discouraging the use of “unproven” treatments.

13. Brand differentiation • Advocates of psychotherapy brands insist on the uniqueness of their therapy and the need to adhere to specific treatment procedures • Research methodology requires the use of manuals and other techniques to standardize treatments • Treatment effectiveness presumed to be dependent on the correct application of the “active ingredients” in the psychotherapy method.

14. The Dodo Bird Effect • Rosenzweig S. (1936) • Some implicit common factors in diverse methods of psychotherapy:“At last the Dodo said, ‘Everybody has won and all must have prizes.’” • Am J Orthopsychiatry 6:412-5.

15. The Dodo Bird Lives! Wampold BE, Mondin GW, Moody M, et al. (1997). A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychol Bull 122:203-15. Luborsky, L., Rosenthal, R., Diguer, L., et al. 2002 The dodo bird verdict is alive and well--mostly.J. Psychotherapy Integration Vol 12(1) 32-57

16. Meta-analysis & common factors • Over two decades of meta-analytic studies have served to reinforce Rosenzweig’s 1936 observation that different methods of psychotherapy tend to produce comparable outcomes… the “Dodo Bird Effect” • Lack of evidence for specific treatment effects bolster the argument that almost all of the effects of psychotherapy are due to factors common to all psychotherapies. 5-11

17. Real world example • Human Affairs International (HAI) collected outcome data from a large number of clinicians between 1996 and 1998. • Clinicians were asked to specify the primary method of psychotherapy (or medication management only) • Analyses revealed no significant differences in the outcome or mean number of sessions across all treatment methods, including medication management.

18. Treatment method & outcomeHAI data

19. Test scores and medicationPBH data Normal functioning Severe symptoms

20. Which treatment is best? Goldilocks Effect: Clients tend to get the treatment that is just about right for them. Normal functioning Severe symptoms

21. Who prescribes? • Sample: 31, 70 adults receiving medication concurrent with psychotherapy (PBH data) Prescribers’ specialty explains less than .5% of variance; p>.05

22. Therapists effects • Wampold and others argue that researchers have ignored the individual therapist as a source of variance.11, 16-24 • The person of the therapist is necessary to delivery the treatment, and personal characteristics of the therapist modify the effect of the treatment. • Factors contributing to clinician effects may include elements clinical skill and knowledge as well as personality traits.

23. Recommended reading Rigorous review and analysis of controlled studies on psychotherapy outcome. Conclusion: much more variance resides with the clinician than with the treatments.

24. RCT and ANOVA – brief history • Some of the earliest applications of randomized control group design and analysis of variance were in agriculture and education. 12,13 • RCT methodology later adopted by medicine and eventually psychotherapy research. 11,14 • Simple ANOVA is appropriate only if the individual farmer, teacher or clinician has little or no impact on the effectiveness of the farming, teaching or treatment method!

25. HLM & therapist effects • Hierarchical Linear Modeling (HLM) is an advance in statistical methodology that permits us to model variance at the clinicians level and as well as the treatment level. • An rapidly growing body of published research points to the conclusion that therapist effects almost certainly exceed specific treatment effects by a large margin.

26. Variance due to the clinician • Published research making use of HLM points to the conclusion that the clinician accounts for much more of the variance in psychotherapy outcomes that treatment method per se. 11, 17-21 • Analyses of PacifiCare Behavioral Health’s massive database on patient outcomes confirms significant variance in psychotherapy outcomes at the clinician level. 24,25

27. PacifiCare Behavioral Health ALERT System • Initiated an outcomes management program in 1998 using 30 item patient self report questionnaires administered at regular intervals in treatment. • ALERT System used to capture data and monitor patient outcomes in real time. • Over 10,000 clinicians are contributed outcome data on a regular basis. • Probably largest database on mental health outcomes in the world.

28. PBH research collaboration • PBH actively sought the involvement of leading psychotherapy outcomes researchers from leading academic institutions. • External researchers actively involved in design of the measurement system and ongoing analysis of the data. • PBH encouraged publication of findings in academic journals.

29. Where is the variance?

30. The (almost) Bell Curve PBH data Solo clinicians with sample sizes => 20

31. % of variance due to therapists in the real world • Analysis of PacifiCare Behavioral Health data reveals 6% of variance due to therapist. 25 • Patients on medication have a higher % of variance due the therapist than those receiving psychotherapy alone. • Huh??

32. Therapists and meds Outcomes (residualized scores) of 15 therapists for patients with concurrent medication or no medication 25

33. Cross validation analysis • Psychotherapists in PBH network ranked based on all cases from 1999-2002 if sample size =>30; N=116. • If a therapist’s mean residualized final score < 0 then clinician rated “Highly effective”; else clinician rated “Less effective”. • Outcomes evaluated in the 2003-2004 cross validation period for a new sample of cases.

34. Cross validation results

35. Psychiatrist effects • Wampold and colleagues also used HLM to reanalyze the results antidepressant and placebo legs of the NIMH-Treatment of Depression Collaborative Research Project study. 28 • Included the 9 individual psychiatrists as a variable. • Outcome measured by change on patient self report measure (Beck Depression Inventory). • 9.1% of the variance due to the psychiatrist; only 3.4% due to the medication. • Top 3 psychiatrists had a better outcome with placebo than bottom 3 had with the antidepressant.

36. Placebo & therapist effects • Hypothesis: Placebo/common factor effects are mediated by the clinician/patient relationship. • Common factors tend to account for much more of the variance than specific treatment effects. • If the effects of common factors are mediated by the clinician/patient relationship, then we would naturally find much of the variance in outcomes would be due to the clinician. • The human factor matters!

37. What’s a clinician to do? • If a wide variety of treatments appear to be equally efficacious, what can a clinician do to achieve the best outcomes possible for their patients? • A growing body of research supports the use of repeated administrations of patient self report outcome questionnaires to monitor response to treatment. 29-36 • Routine measurement and early identification of patients with a poor response to treatment has been shown to reduce treatment failures.

38. Therapeutic alliance • A large body of evidence suggests that the relationship and working alliance between the clinician and patient is an important factor in the outcome. 39-45 • Routine use of a session rating/therapeutic alliance scale may permit clinicians to identify and repair problems in the working alliance.

39. Can we improve outcomes? • Increasing the percentage of patients treated by highly effective clinicians (as identified through practice based evidence) is the most direct pathway open to a health plan seeking to improving outcomes across a large system of care. • See PBH’s Honors for Outcomes initiative: • http://www.pbhi.com/Providers_public/FAQs/H4OFAQ.asp • Smaller organizations may be able to improve outcomes by fostering outcomes informed care methods within the organization.

40. Resources for Living (RFL) • Provides telephonic EAP services, data collected over the phone at time of service; clinicians receive real time feed back on trajectory of improvement and working alliance (SIGNAL system) • Outcome measures: Outcome Rating Scale (4 items); also utilizes the Session Rating Scale (4 items) to the working alliance

41. RFL Signal System

42. RFL Signal System: results Training and feedback Baseline period

43. Accountable Behavioral Healthcare Alliance (ABHA) • Managed behavioral healthcare organization servicing Oregon Health Plan members in 5 rural county areas • Outcome measure: Oregon Change Index (4 items; based on the Outcome Rating Scale)

44. OCI Feedback • After collecting baseline data throughout 2004 and early 2005. • In mid 2005 ABHA initiated site optional weekly feedback at the clinician and supervisor level. • Excel based Active Case Report contains data on all cases seen within the last 6 weeks. • Report is updated and emailed to clinicians at the start of each week.

45. OCI Active Case Report

46. Trajectory of Change Graph

47. Outcomes trending upwards

48. Impact of clinical reports Approximately half of the clients in 2006 were treated by clinicians that received weekly active case reports and quarterly closed case outcomes reports.

49. Primary barrier… the clinician • Most clinicians believe that their outcomes are above average and their services are of high value, without the need to actually measure this • Many clinicians feel discomfort at the thought that their performance might be evaluated by their patients via self report outcome questionnaires • Clinicians often believe that a simple outcome questionnaire cannot provide useful information about their patients beyond what they obtain by forming their own clinical judgments.

50. Secondary barriers • Faith in treatments (therapy methods, drugs) to deliver consistent and predictable outcomes • Belief that the cost of the services is so low (relative to overall medical costs) that meaningful performance management isn’t cost effective • Belief that meaningful performance management isn’t necessary to retain existing business or acquire new customers. • Lack of organizational commitment to place the patient first and/or desire to avoid conflict with clinicians