Understanding the CiMH Dashboard Reports for LA PEI MAP Webinar

1. Understanding the CiMH Dashboard Reports for LA PEI MAP Webinar June 25, 2014 Cricket Mitchell, PhD Senior Associate, CiMH

2. Some technical issues… • All participants are in a muted, “listen only” mode • Questions can be submitted at any time using the function on your control panel • I will do my best to address all questions • You can also ‘Raise your Hand’ via the control panel • At the end of the webinar, I will attempt to un-mute those with their hands raised, to address questions verbally and/or allow for discussion • Today’s webinar is being recorded and will be posted on CiMH’s website and YouTube channel

3. Purpose of Today’s Webinar • The CiMH Program Performance and Outcome Evaluation Dashboard Reports for MAP can be confusing • We want to ensure that everyone interested in data from the LA PEI MAP implementation can make sense of the reports • This includes county administrators, provider administrators, program managers, MAP supervisors, MAP therapists, data entry personnel, and PracticeWise… just to name a few… • Anyone who is not able to participate in today’s live webinar, can download the materials and review the information in their own time We’re going to cover a lot of EXCITING material today… so let’s get started!!

4. CiMH Dashboard Reports • Burning questions in the minds of MAP practitioners, supervisors, and administrators across the County…. • What the heck do all those tables mean? • So… how are we doin’? • I’ve got my site report… how can I use it?

5. Understanding the CiMH MAP Dashboard Reports • All attendees registered for today’s webinar received this PowerPoint presentation and the most recent Aggregate MAP report (also the final CiMH-generated MAP report) • Data thru December 2013 • During today’s webinar we will go back and forth between the PowerPoint presentation and the most recent Aggregate report

6. Understanding the CiMH MAP Dashboard Reports • Aggregate Report, Page 1 • All agencies that have submitted data are listed • Some agencies submit multiple data shells reflecting MAP services at different sites; and, some receive more than one site report (upon request) • Some agencies submit more data than requested (i.e., clients with service dates that are later than the date specified by the submission interval) • All data received are included in the report (“at minimum, clients served through…” )

7. Understanding the CiMH MAP Dashboard Reports • Table 1: Entry and Dropout Rates • Overall MAP Entry Rate is defined as clients who were referred to MAP and have a first session documented in one of the four treatment targets approved under LA PEI • No Date of First Session, client did not enter MAP • No treatment target approved under LA PEI, client did not enter MAP • Because the data shells do not track the treatment target to which a client is referred, we are not able to determine the Entry Rate for each treatment target

8. Understanding the CiMH MAP Dashboard Reports • Table 1 • Overall MAP Dropout Rate is defined as clients who entered MAP but did not complete the full intervention as determined by the therapist • In the data shell: “No” for “Completed MAP” without an indication that the client has Begun Another Treatment Focus • Each target-specific Dropout Rate is defined as clients who have a first session documented in the treatment focus, but did not complete the full intervention targeted on that treatment focus as determined by the therapist • In the data shell: “No” for “Completed Focus #”

9. Understanding the CiMH MAP Dashboard Reports • Table 1 (continued) • The number of clients who dropped out each treatment focus does not equal the number of clients who dropped out of MAP • It is possible to drop out of a treatment focus and not drop out of MAP • Some clients are identified in the data shell as dropping out of one or more treatment focus, yet are also identified as successfully completing MAP • Some clients have dropped out of one or more treatment focus, and are still receiving services under the MAP model

10. Understanding the CiMH MAP Dashboard Reports • Table 2: Treatment Focus Distribution • It was expected that all clients who entered MAP would have service delivery information for Focus #1 • The number of clients who entered MAP should match the number of clients in the Focus #1 column • At the most recent data submission, some clients had service delivery information at Focus #2 without data entered for Focus #1 • This explains the difference between 19,410 clients that entered MAP (in Table 1) and 19,407 clients with a Focus #1 in Table 2

11. Understanding the CiMH MAP Dashboard Reports • Table 2 (continued) • The vast majority of MAP clients have been or are being served for only one treatment target/focus • 4.9% have a Focus #2, and less than 1% have a Focus #3 or Focus #4 • Most MAP clients are served for Disruptive Behaviors • 43.0% of those with a valid Focus #1 • The smallest group of MAP clients are served for Trauma • 4.0% of those with a Focus #1

12. Understanding the CiMH MAP Dashboard Reports • Table 3: Client Demographics • The title of the table indicates that this information is reported for clients who Entered MAP • The left-hand column includes the number of clients (N) that Entered Overall MAP, and the number of clients that entered each Treatment Target • The number of clients who entered all four treatment targets does not equal the number of clients who entered MAP • It is possible for a client to be served across more than one treatment target approved under LA PEI • Some clients are indicated as having entered the same treatment target at more than one Focus #

13. Understanding the CiMH MAP Dashboard Reports • Table 3 (continued) • We don’t have demographics on all MAP clients • Age is calculated as the difference between the Date of First Session and the client’s Date of Birth • If one or both of these dates are missing or incorrect, we cannot calculate age • This explains the lower n for age than the N for Entered (e.g., we can only calculate age for 19,118 clients, but we have data to indicate that 19,410 clients Entered Overall MAP) • The percentages across Gender (Female and Male) may not total 100 due to missing data • The percentages across Ethnicity (African American, Asian/ Pacific Islander, Caucasian, Hispanic/Latino and Other) may not total 100 due to missing data

14. Understanding the CiMH MAP Dashboard Reports • Table 4: DSM-IV Primary Axis I Diagnoses • As with Table 3, the title indicates that these data are reported for clients who Entered MAP • The N’s for each group are reported in the far left-hand column, as they were in Table 3 • Because there is a “Missing” category, the percentages across each row of diagnostic categories will total 100

15. Understanding the CiMH MAP Dashboard Reports • Table 5: Process Data • The title of the table indicates that these data are reported for clients who Entered MAP; but, the denominator is reduced to only include clients who were served for that treatment focus and are within the valid age range for the measure(s) • Each column includes parent and/or client self-report; in other words, agencies are credited for collecting pre- data from only one informant, even if both are required • Collecting pre- data on the required outcome measures is an important aspect of the MAP process • MAP treatment outcomes cannot be reported if pre-data are not collected

16. Understanding the CiMH MAP Dashboard Reports • Table 5 (continued) • The number of clients who have a valid pre- outcome measure must meet the following criteria: • The client must be within the valid age range for the measure • The score must be valid (i.e., must be within the possible range of scores for the measure) • The measure must match the treatment target for which it is required under LA PEI (e.g., an RCADS is only included for clients treated for Anxiety) • Appendix A includes descriptions of all required outcome measures for MAP under LA PEI

17. Understanding the CiMH MAP Dashboard Reports • Table 6: Service Delivery Data • The title of the table indicates that these data are reported for clients who Completed; however, the number of clients (N) who Entered each group is again reported in the far left-hand column (as in Tables 3 and 4) • The Number of Clients who Completed is reported in the first data column • Almost 8,000 MAP clients are currently in treatment • 19,410 clients Entered MAP; 5,621 have Completed MAP; and, 5,866 have Dropped Out (as reported in Table 1) • This indicates that there were 7,923 open cases at the time of the data submission

18. Understanding the CiMH MAP Dashboard Reports • Table 6 (continued) • Completed Overall MAP is defined as “Yes” for “Completed MAP” at any of the four treatment foci service delivery sections in the data shell • Completed for each Treatment Target is defined as “Yes” for “Completed Focus #” at any of the four treatment foci service delivery sections in the data shell

19. Understanding the CiMH MAP Dashboard Reports • Table 6 (continued) • Length of Treatment is calculated as the difference between the Date of Last Session and Date of First Session within each Treatment Focus; and, for Overall MAP it is summed across treatment foci. • If one or both of these dates are missing or incorrect, we cannot calculate Length of Treatment • Number of Sessions is a field in the data shell • If it is missing, we cannot report number of sessions

20. Understanding the CiMH MAP Dashboard Reports • Table 6 (continued) • The data reported in this table don’t make sense! • The range for number of weeks in treatment is as low as 1 • The range for number of sessions is as low as 1 • BUT… only clients who are indicated as having Completed are included…?????? • In addition, the upper limits for the range of number of weeks in treatment (300) and number of sessions (339) are higher than what we would expect for model-adherent service delivery under MAP

21. Understanding the CiMH MAP Dashboard Reports • Tables 7-11: Outcome Data • The table titles indicate that outcomes are reported for clients who have Completed (Overall MAP or Target-Specific Treatment); and, the number of clients who have Completed treatment within each group is reported in parentheses at the end of the title (these n’s match the n’s for Completed in Table 6) • Three Outcome Indicators are reported in the outcome data tables • Appendix A describes these Outcome Indicators • The minimum n necessary to report outcomes in the aggregate report is 20 • This is decreased to 5 for site reports… Even though outcomes based on a small n are unreliable, we want to give sites as much of their data back in their site reports as possible

22. Understanding the CiMH MAP Dashboard Reports • Outcome Indicator: Percent Improvement • The percent change from the average pre- score to the average post- score is reported for each outcome measure • Change is always reported in terms of improvement • Interpreting percent change can be difficult • It is influenced by the range of scores – a measure with a smaller range of scores is likely to show a greater percent change than a measure with a larger range of scores • It does not translate to clinically meaningful change • It is an overall reflection of treatment success • Do our clients get better after participation in this treatment?

23. Understanding the CiMH MAP Dashboard Reports • Outcome Indicator: Percent Improvement (cont’d) • Two additional pieces of information are provided under each Percent Improvement: the (n), or number of clients, included in the analysis; and, the average [pre- score] on the measure • Only clients who have valid pre- and post- data, and are indicated as having Completed treatment are included • Knowing the average pre- score… • Indicates whether or not clients were, on average, above or below the clinical cutpoint prior to treatment • Allows you to calculate the post- score

24. Understanding the CiMH MAP Dashboard Reports • Outcome Indicator: Percent Improvement (cont’d) • Some percentages have an asterisk (*) next to them • A paired t-test analysis is conducted with each set of pre/post scores; and, when the difference observed is not likely to be due to chance (p<.01), this is indicated with an asterisk and a footnote • In addition to reporting Percent Improvement in the Outcome Data Tables, there is a Graph that represents each pre/post difference • The average pre- score and the average post- score are presented in bar graphs for each measure • If the measure has a clinical cutpoint, it is indicated with a solid horizontal line

25. Understanding the CiMH MAP Dashboard Reports • Outcome Indicator: Effect Size Estimate • Cohen’s d a standardized effect size measure that estimates the magnitude, or strength, of a relationship • A nice addition to reporting Percent Improvement, which is not a standardized measure • For MAP, it is an estimate of the strength of the relationship between the average pre- score and the average post- score, expressed in terms of standard deviations • An effect size of .5 indicates that the average pre- score is .5 standard deviations greater than the average post-score • Think back to those stats classes!

26. Understanding the CiMH MAP Dashboard Reports • Outcome Indicator: Effect Size Estimate (cont’d) • While there is no absolute agreement about what magnitude of an effect size is necessary to establish clinical significance, conventional interpretations of Cohen’s d are • .2 - .3 represents a small effect • .5 reflects a medium effect • .8 or greater represents a large effect • The same clients included in Percent Improvement are included in the Effect Size Estimate

27. Understanding the CiMH MAP Dashboard Reports • Outcome Indicator: Reliable Change • We can look at two scores and see that they are different; but, how do we know that the difference we’re observing isn’t simply due to measurement error? How much difference do we need to observe to know that it is a reliable amount of change? • A fairly complex formula is used that takes into account the variability of the pre- treatment group and measurement error, resulting in our ability to categorize pre/post change for each measure into three groups: • Reliable Positive Change, Reliable Negative Change, and No Reliable Change

28. Understanding the CiMH MAP Dashboard Reports • Outcome Indicator: Reliable Change (cont’d) • The same clients included in Percent Improvement and Effect Size Estimate are included in Reliable Change • In addition to reporting the percent of clients with Reliable Change in each of the three groups, there is a Graph to represent each set of Reliable Change data • However, these data are presented differently than the other outcomes • Reliable Change groups are graphed in 6-month intervals • Date of Last Session is used to determine the interval in which a client is included • The n’s across all 6-month intervals may not equal the n in the outcomes table • All data are included in the outcomes tables, even for clients with service delivery dates that are later than the minimum date required at the data submission interval

29. So… How are We Doing? • We have enough data to report outcomes for all treatment targets! • The data indicate, on average, clients improve after participation in treatment using the MAP model • They improve in overall mental health functioning, and in target-specific areas, such as anxiety, depression, disruptive behavior, and post-traumatic stress symptoms

30. So… How are We Doing? • On average, clients are entering MAP treatment with scores in the clinical range on the measure of overall mental health functioning; and, completing MAP with scores below the clinical cutpoint • According to both parent report and youth self-report • The target-specific outcomes are a bit different, with clients starting out below the clinical cutpoint on most measures; and, staying below the clinical cutpoint after MAP (on average) • The exception is Disruptive Behavior, with clients starting out right at the clinical cutpoint prior to MAP, and dropping below the cutpoint after treatment

31. So… How are We Doing? • Looking at the Reliable Change data… • The majority of clients with Overall MAP outcomes have an amount of change on the YOQs from pre- to post- treatment that is Positive • 58% Positive Change according to parent report • 53% Positive Change according to youth self-reports • The positive change groups are not as large for the target-specific outcome measures; however, clients are not (on average) beginning treatment in the clinical range on these, as they are on the YOQs • Again, the exception is for Disruptive Behavior, with 53-56% of parents reporting Positive Change

32. So… How are We Doing? Completed MAP n=5,621 YOQ Outcome Data n=2,724 Entered MAP N=19,410 (29% of clients who Entered MAP) (48% of clients who Completed MAP) LA PEI MAP outcomes are looking good… However, the number of clients for whom we have outcome data is less than half of those who have Completed MAP

33. How Can I Use My Site Report? • The MAP reports differ from other CiMH dashboards… • The Aggregate data are provided along with your site-specific data, so you can readily compare how your program is performing relative to the average MAP program in the County • We try to fit both sets of data on the same page for ease of viewing; however, sometimes that’s just not possible! • Prompts throughout the report provide guidance on interpreting the data within the context of the MAP model

34. MAP Process Guide Examples of prompts throughout site reports that guide interpretation of the data within the context of the MAP model

35. How Can I Use My Site Report? • Each site report in this final set is comprehensive… • All outcome data tables and graphs are included, even if a site did not have data for that treatment focus and/or measure • This ensures that each site report contains all Aggregate LA PEI MAP data

36. How Can I Use My Site Report? • Do your data look accurate? • The first level of program performance reflected in these reports is the extent to which data are being collected and entered accurately and completely • There might be cause to go back and look at your data shell • If something doesn’t look like its accurately reported, schedule a call with Cricket • Even though there will be no more submissions to CiMH, we are happy to help investigate potential errors in data entry that can be corrected in the PEI OMA

37. How Can I Use My Site Report? • Compare your site-specific data with the aggregate data • Is your program performing on par with the average MAP program in the County? • There may be unique factors within an agency or site’s implementation that provide context for interpreting the data • Are you achieving positive outcomes on par with the average MAP program in the County? • There are still many sites who do not have enough outcome data to report; and, many who only have outcomes for a very small number of clients – BE CAUTIOUS IN INTERPRETING DATA BASED ON SMALL NUMBERS OF CLIENTS

38. How Can I Use My Site Report? Is your entry rate low? (Table 1) Is your dropout rate high? (Table 1) Is the distribution of demographic characteristics reflective of clients in need of services (within your agency and/or Service Area)? (Table 3) What is the distribution of Primary Axis I diagnoses relative to your treatment focus distribution? (Tables 4 and 2) Are you collecting pre- data on the required outcome measures? (Table 5)

39. How Can I Use My Site Report? • For clients who have Completed MAP • Does your service delivery data look appropriate? (Table 6) • What proportion have outcome data? (Table 7 - divide the n for YOQs by the number who have completed; in the most recent aggregate report, this is 2,724 ÷ 5,623) • For clients who have Completed a Treatment Target • Does your service delivery data look appropriate? (Table 6) • What proportion have outcome data? (Tables 8-11, same process as for Table 7 – the n for completed is in the table header, and the n for an outcome measure is in parentheses below the percent improvement)

40. What if I Still Have Questions? • Don’t hesitate to contact Cricket • The MAP reports are dense and can be confusing (hopefully a little less so after today) • Ask questions about the data in your site report… • Ask questions about the outcome measures… • Ask for help finding an error in your data entry… • We’re here to help!!

41. The End • Contact Information • Cricket Mitchell, PhD • Email: cmitchell@cimh.org • Cell phone: 858-220-6355