Moving From Data To Information: The Critical Role of Research & Evaluation in Mental Health

1. Moving From Data To Information: The Critical Role of Research & Evaluation in Mental Health A Presentation to the Missouri Mental Health Commission R. Paul Thomlinson, Ph.D. Burrell Behavioral Health June 10, 2010

2. Data Never Speak for Themselves The beginning of wisdom is to recognize our own ignorance. We no longer live in the era of Galen: pointing to the opinions of a wise man will not suffice. Numbers do not stand alone—the history of science shows us that scientific knowledge is not absolute, that it all involves uncertainty. The result of scientific research and statistical analysis is not “unadorned and irrefutable fact”– it is an act of interpretation, and then more interpretation. “Numbers do not lie, but they have a propensity to tell the truth with intent to deceive.” (Eric Temple Bell) Source: Ghaemi (2009)

3. On Hitting and Missing the Mark Every truth…is an error that has been corrected (AlexandreKojeve). The expert sharpshooter does not become so in the absence of good data, properly reported and reasonably interpreted. Evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness.

4. Turning Boring Satisfaction Data Into Actionable Information • What is so? Satisfaction with treatment is crucial for both pragmatic and ethical reasons. However, even before the question of how to measure satisfaction (i.e., which instrument to use) is answered, the question of what will be done with the resulting data should be given serious consideration. In other words, what will be done so that satisfaction data are turned into useful, actionable satisfaction information? What is needed is a method by which individual items can be: (1) categorized as high or low in satisfaction, and (2) evaluated as high or low in importance. The confluence of these factors suggests whether a particular dimension of service needs attention or not.

5. Turning Boring Satisfaction Data Into Actionable Information • So what? The “actionable item matrix” on the next slide is a simple way to turn data into information. • So? We continue to use this method to develop reports from inpatient, community-based, and employee satisfaction survey results, and to decide where to locate our improvement efforts. Thomlinson (2004)

6. The “Actionable Item Matrix”

7. Social and Emotional Skills in Early Childhood Question: Do social and emotional skills or cognitive abilities better predict school readiness? Answer: Results of a study of 122 preschoolers in SW MO show that the Devereux Early Childhood Assessment (DECA) provided better prediction than the DIAL of which children did well in Kindergarten. However, Math Skills in combination with DECA Self-Control scores explained more than either factor alone. So what? Mayor’s Commission Summit, Ready to Learn project, ECHO project all focusing on early childhood SEL. Thomlinson, et al. (2008)

8. Psychological Predictors of Cardiac Mortality Question: What measurable factors best predict one-year mortality among CHF patients? Answer: The BSI scales significantly related to outcomes were the anxiety scale with mortality (rpbs=-0.40, p=.03), and the phobia scale with treatment continuation (rpbs=-0.39, p=.03). So what? Significantly more attention paid to screening for psychological problems in cardiac rehab; increased referrals for psychological intervention for CR patients; and ongoing outcomes evaluation for CR by Burrell. Thomlinson (2000)

9. QEEG Predictors of Post-Stroke Depression Question: To what extent can we identify PSD using non-invasive QEEG readings? Answer: Among older CVA patients (with and without depression), significant correlations were found between left frontal QEEG asymmetries (greater power) and both the Beck and Yesavage depression scales. So what? Strong collaboration with neurology department; increased sensitivity to PSD and corresponding referrals to treatment; awareness of availability of electrophysiological method with aphasic stroke patients. Ulam & Thomlinson (1998)

10. Behavioral Health and Primary Care Integration Question: What factors predict/explain which patients keep and which patients miss their BH appointment when referred by their PCP? Answer: Among those who DNKA, financial, transportation, administrative, increased stress, previous experience with BH, and motivation were identified as barriers; those who attended identified a specific problem, had encouragement of another, and linked medication access to appointment. So what? Strong collaboration with family practice; introduced change readiness and improved distress management at time of referral into residency protocol. Reust, Thomlinson & Lattie (1999)

11. New Opportunities to Advance Health (NOAH) What is so? It should be as easy to schedule a patient for an evidence-based health educational program as it is to schedule a lab test or an x-ray. Burrell funded this $100,000 demonstration project. So What? Chronic Disease Self-Management Program participants showed dramatic health status improvements (SF-36), and reduced utilization. So What? Protocols put in place to screen and refer asthma patients to adult asthma education program (Lorig model), with 3 certified educators, for which NOAH won a Missouri Quality Award for improved process and outcome (improved health status, reduced ER utilization).

12. New Opportunities to Advance Health (NOAH) So What? Timely developmental screenings for children were identified as an area in need of improvement, and resources were deployed to provide screening schedules and information on financial help with screening to all new parents with babies delivered in the system. So What? Physician-led discussion groups on a variety of health topics, including a group for parents of children with ADHD. Evaluation showed substantial reductions in parental stress (validated PSI) over 6-months follow up.

13. New Opportunities to Advance Health (NOAH) So What? Patient self-care manuals provided to 1000 providers and 1000 patients through collaboration with Cox Health Plans, with good uptake rates (over 50%) and reduced health care utilization over 6 months. So? Even though our relationship to Cox has changed, the health educator who managed NOAH from the start continues to provide these and a wide array of other educational interventions. Thomlinson, Schaible & Stanley (2000)

14. Issues of Aging Question: What are the significant aging benchmarks for women, and how do these perceptions relate to health behaviors and concerns? Answer: A study of 1000 indicated that there is significant variability in placing aging benchmarks, healthier/more educated women place “aged” benchmark much later in life, and those who do not fear aging reported excellent health and health habits significantly more often than those who fear aging. So What? Established significant research collaboration with prominent local family foundation; informs our MFH proposal for older adult mental health. O’Reilly, Thomlinson & Castrey (2003)

15. Issues of Aging • Question: Are older adults using computers and the Internet more, and is such use related to health? • Answer: A study of 5000 seniors indicated about a third had access to a computer and Internet, and used them for email, health information, finances, socializing, and shopping. Those who are not online cited lack of equipment or interest as the major reasons for this state of affairs. There is a significant positive relationship between use of the Internet and general mental health functioning (r=.18, p=.001 , and between use of email and general mental health functioning (r=.17, p=.006). • So What? Informed Senior Advantage program with respect to planning for services and placement of computers in Primrose Place. Thomlinson (2001)

16. “A Rising Tide Lifts All Boats” Project • Question: How do our community’s youth fare on the risk and protective factors measured by the validated Communities that Care Survey (targeted nearly 2000 6th, 8th, 10th, and 12th graders in Greene County Schools)? • Answer: Strengths • Low past-30-day rates of methamphetamine, cocaine, other drugs. • Low levels of participation in theft and weapons possession. • Lower than national average for lifetime use of inhalants. • High level of awareness of risks of drug use, and perceive that drugs and firearms are not easily available in their community. • Report a motivation to follow society’s standards and expectations.

17. “A Rising Tide Lifts All Boats” Project • Answer: Opportunities • Almost half have used alcohol in their lifetimes, and over one fifth have used within the previous 30 days. Reported past-30-day use peaks at 41.3% for 12th graders. • Lifetime use of smokeless tobacco by 12th graders is higher than national averages. • Overall, more than 1 in 10 reported having attacked someone with the intent of doing serious harm at least once in the past year. Gender effect: 14% of males, 7.2% of females. • Risk factor scales of concern: Low neighborhood attachment, and high personal transitions and mobility.

18. “A Rising Tide Lifts All Boats” Project So What? Led to convening of the “Changing Tomorrow Today” prevention conference in our community, attended by over 200 community leaders. Led to Community Foundation of the Ozarks taking action in the following ways: instituting a biennial Community Focus report to track such “red flags and blue ribbons”; issuing the “Grantmaker’s Challenge for Children,” resulting in $1 million grant to a community collaborative (ECHO Project) involving Burrell, Community Partnership, schools, and Missouri State University.

19. Co-Occurring Disorders and Homeless Persons • Question: Can we help homeless families struggling with COD make the Journey Home? • Answer: Implementation of IDDT and intensive case mgt/supports have helped over 400 such homeless families move from shelters to permanent housing, and significantly improve employment status, mental health symptoms/functioning, and reductions in alcohol/drug abuse. • So What? The Journey Home project was the first portal through which Burrell’s system embraced and integrated the EBP IDDT, for which we have been awarded a SAMHSA Science and Service Award (2008).

20. On Thin Ice: Methamphetamine in the Ozarks • Question: What are the lived experiences of meth users, and how can we use this information to confront meth in the community? • Answers: (1)There is an identifiable pathway to methamphetamine use, with a major predictor of use being involvement with other drugs; (2) Gender is a distinguishing factor in reasons for use, benefits perceived, method of obtaining and paying for methamphetamine; (3) The cycle of use-abuse-addiction drawn from traditional substance abuse models does not apply well to methamphetamine use: there is a much steeper and rapid decline from first use to addiction; and (4)Family dynamics will be related to use, abuse and recovery among methamphetamine users. • So What? Collaborated with many community partners to address the questions; convened a community-wide conference to present the findings; resulted in grant proposals for both treatment intervention and “Drug Endangered Child” task forces. Thomlinson, et al. (2004)

21. Health Care Leadership Development Survey • Question: What factors are most critical to effective leadership in a health care setting, and how best to feed leadership survey information back to the leaders? • Developed a leadership development instrument based on factors identified in literature and through focus groups, pre-tested and factor analyzed initial drafts, implemented the survey with subordinates completing the evaluation of their leaders, and facilitated feedback of information to leaders. Thomlinson (2008)

23. Children’s Day Treatment Evaluation • Question: To what extent do CDT participants improve in social functioning and age-appropriate behaviors? • Answer: Children enter treatment due to problem behaviors and age-inappropriate functioning—when discharged an average of 9 months later, children were functioning at age-appropriate levels or better according to results from the psychometrically sound PBRS, which measures psychomotor, cognitive, and social domains. • So what? Just ask their parents or caregivers, so what? Meriweather & Thomlinson

24. Cultural Competence • Question: To what extent do rural and urban system of care partners differ with respect to cultural competence? • Answer:Statistically significant differences on the Georgetown Cultural Competency Survey emerged as follows: rural counties were lower in four of the 32 areas—display /décor, videos/other media, acceptance of spiritual views, and seeking information on customs. • So what? Used these results to develop cross-training modules across system of care partners, with emphasis on the identified areas of strengths and weaknesses. Thomlinson, Maples & Rimel (2004)

25. Cultural Competence • Question: To what extent do rural and urban system of care partners differ with respect to cultural competence? • Answer:Statistically significant differences on the Georgetown Cultural Competency Survey emerged as follows: rural counties were lower in four of the 32 areas—display /décor, videos/other media, acceptance of spiritual views, and seeking information on customs. • So what? Used these results to develop cross-training modules across system of care partners, with emphasis on the identified areas of strengths and weaknesses. Thomlinson, Maples & Rimel (2004)

26. SASOP • Specialized Adult Services Outcomes Protocol which includes the OHIO Provider and Consumer Forms, the Missouri DMH Adult Status Report, CAGE-AID, Multnomah Community Functioning Scale, and the Substance Abuse and Treatment Scale. • Provides a tool to assist in treatment planning and to objectively measure clients’ progress in treatment. • The SASOP is completed by clients and the CSW at intake, 3-months, 6-months, and annually thereafter. • This report is based on data from 842 clients who completed the SASOP in the past year.

27. Age Distribution

28. Diagnoses • Approximately 88% of all the individuals in services had a primary diagnosis that fell under the categories of “Schizophrenia & Other Psychotic Disorders” or “Mood Disorders.” • Anxiety Disorders (8.6%) were the third most prevalent diagnoses.

29. Where do they live?

30. Employment Status

31. Highest Education Level

32. Outcomes: Quality of Life • These items assess satisfaction with various aspects of consumers’ lives including friendships, finances, family relationships, and living arrangements. • A series of paired sample t-tests found that clients reported significant improvements in quality of life (t [753] =-3.45, p=.001) from intake to last administration and (t [209] =-4.8, p=.000) from intake to 3-months and (t [83] =-5.98, p=.000) from intake to 6-months.

33. Outcomes: Symptom Distress • Measures the amount of distress caused by psychiatric symptoms. • Scale scores range from 15 to 50; higher scores indicate more symptom distress. • A series of paired sample t-tests found that clients reported significant improvements in symptom distress (t [764] =3.84, p=.000) from intake to last administration and (t [211] =4.8, p=.000) from intake to 3-months and (t [84] =4.02, p=.000) from intake to 6-months.

34. Outcomes: Empowerment • Measure of the construct of personal empowerment as defined from a consumer perspective. • A series of paired sample t-tests yielded significant improvement (t [752] =-5.59, p=.000) from intake to last administration and (t [211] =-3.91, p=.000) from intake to 3-months and (t [82] =4.01, p=.000) from intake to 6-months.

35. Outcomes: Community Functioning • Measures functional status, as well as safety and health, including social support, housing stability, performance of day-to-day living activities, and participation in meaningful activities. • A paired sample t-test between intake and last administration demonstrated significant improvement (t [1341] =3.259, p<.001).

36. SASOP • So What? Treatment teams make use of SASOP reports to monitor progress of their clients, and perhaps more importantly, to make data-based decisions about changes in interventions that can help improve symptomatic and functional change.

37. Current, New, and Future Research • RAISE (Recovery after initial schizophrenia episode)—collaboration with Feinstein Institute for Medical Research. • Integration of EBP decision support into EMR—collaboration with MU’s Dept of Social Work. • Improving welcoming and treatment of PTSD among returning veterans—collaboration with Our House Foundation. • NCCBH-OMJ study of treatment adherence with anti-psychotic medication regimens in “real-world” settings. • Evaluation of aftercare programming in Central Missouri—collaboration with MUPC. • Benchmarking physical and psychosocial outcomes of patients completing cardiopulmonary rehabilitation—collaboration with CoxHealth. • Evaluation of Family Dependency Treatment Court and Juvenile Drug Court (Greene County).