Documenting Impact and Ensuring Sustainability

1. Documenting Impact and Ensuring Sustainability BJA National Technical Assistance and Training Event Dr. Tony Fabelo Washington, DC July 15-16, 2009

2. Overview • Role of Data in Decision Making • Minimum Level of Effort for Data • Data and Impact on Sustainability

3. Are Data Relevant to Decision-Making? Is the program working? Yes Do we have data to know? No Does the analysis shows that it works? Can we spin it? No Yes Yes Can we use a testimonial from a powerful person and a poster case? Yes No Problem

4. Pitot Tubes

5. Key Role of Data at Different Decision Making Levels Elected Officials Data Driven Compromise Good Politics Good Policy Fiscal or Grant Managers Poor “Performance” Data Driven Administration Good “Performance” Program Administrators Program Recipients as Indirect Users of Data Data Driven Validation Harm Benefits Different incentives for the use of data

6. Data One of Many Factors for Elected Officials Is there an interest group that supports this? Is the interest group important to me? Does the initiative fits my “world view”? Would I get good press coverage? Is there any evidence/data to show that this would work?

7. More Incentives for Data Driven Decision at Administrative Level Can I account for the use of the funds? Is the program well managed? Is there any evidence/data to show that the program is effective? Does this program has enough clients for cost-effectiveness? How do I justify program continuation if I don’t have good data?

8. Validation of Benefits Critical for Program Participants Is there any evidence/data to show positive outcomes for clients? What is the completion rate? Do we have many complaints? Are program recipients satisfied with treatment? Do program administrators see the benefits at a personal level?

9. Usual Challenges to Document Program Performance Target Population Administration Outcomes Presentation Lack of analytical capabilities to identify target populations Data systems that do not capture key data elements Data systems that do not “talk” across agencies Lack of process evaluations Claims of confidentiality used as barriers No agreement on outcomes and how to measure them Lack of interest/expertise in setting mechanisms to measure outcomes Problems conceptualizing results in a policy relevant manner Pennies for Research

10. Overview • Role of Data in Decision Making • Minimum Level of Effort for Data • Data and Impact on Sustainability

11. This is Your Minimum Level of Effort Target Population Administration Outcomes Presentation Identification of population characteristics Tracking of basic aggregate counts Flow chart of the process Documentation of efficiencies for key administrative junctures Agreement on some basic outcome measures at program start Attempt to measure these outcomes Define: Problem being addressed Program strategies How well were those strategies implemented How many people were served What were the outcomes

12. Creek, River, Dam Analogy Sources Compilation Analysis

13. You Are Dealing with One of These Areas in Addressing Mental Health Issues Law Enforcement Contacts Contact and Identification “Frequent Flyers” “First Time Flyers” Pre-Trial “Shortage” Jail or Prison “Inside” Services Assessments “Co-occurring” “Poor” Discharge Planning “Sick” vs. “Bad” “Uncoordinated” Community Care Benefits Crisis Meds Continuity Records Housing

14. You Are Dealing with One of These Actors Jail Sheriff District Attorney Judges LawEnforcement Court Prison Parole Public Defender Probation Pre-Trial Agency MH Provider

15. Who “Owns” the Data? What Elements are Captured? How Can it Be Retrieved for Analysis? What is the Quality of the Data? You Need to Identify Sources of Data Computerized Records or Case Management Systems Statistical Counts (Aggregate) on Paper or Computerized Paper Case Records

16. You Need to Map Program Model

17. You Need to Develop Basic Analytical Picture Goals of the Initiative Target Population Program Implementation StrategiesAgency/Multi-agency Number Assessed for Program Number Referred/Selected for Program Conceptualization of Service Packages Intervention/Service Number served, number of contacts/services,duration of intervention Complete Drop Out Comparison OutcomesQuality of Life Measures Clinical and Criminal Recidivism

18. Examples of Process Measures • Outputs • Production: indicator of the number of goods or services • Number screened and assessed • Number treatment plans or discharge planning developed • Number of applications for Medicaid or SSI/SSDI • Number released on pretrial diversion • Number of case conferencing • Efficiency • Management: indicator of productivity like unit costs, units of time • Cost-per-client • Time between screened and assessment • Medicaid or SSI/SSDI funds collected per client • Adherence rates to psychotropic medications • Rates of authorization of release of information

19. Example of Outcomes Measures (Benefits) Pre-and-post episodes, hospitalization days Percent re-arrested and percent re-incarcerated during a period after intervention Clinical Recidivism Criminal Recidivism Percent needing housing who are in permanent housing Percent eligible for Medicaid receiving Medicaid Percent functioning a higher level than at program entry Program Specific “Quality of Life” Measures

20. Ideas for Other Measures

21. Analysis and Presentation - Simple

22. Analysis and Presentation – More Detailed

23. Analysis and Presentation – Don’t!

24. Overview • Role of Data in Decision Making • Minimum Level of Effort for Data • Data and Impact on Sustainability

25. Data Key to Sustainability Interest Groups Few Years Later Problem to be Addressed Other Interest Groups/Policy Makers Policy Makers Enact Policy/Program We want this Allocate Resources Data/PerformanceKey to Maintain Resources

26. Summary

27. Thank You This material was prepared for the conference by staff of the Council of State Governments Justice Center. Presentations are not externally reviewed for form or content and as such, the statements within reflect the views of the authors and should not be considered the official position of the Justice Center, the members of the Council of State Governments, or funding agencies supporting the work.

28. Documenting Impact & Ensuring Sustainability Ann-Marie Louison CASES, Inc. NYC Washington, DC July 17, 2009

29. Overview • Nathaniel Project – a case study • Data, data, data • Adapting program in response to outcomes to achieve Sustainability

30. Nathaniel Project – case study Can I account for the use of the funds? January 2000 $100,000 City Council funding Serve 8 participants performance-based contract Is there any evidence/data to show that the program is effective? Does this program have enough clients for cost-effectiveness? 2000 funds to serve 8 participants 2002 53 consumers served in 2 year case management services How did CASES justify program continuation?

31. Nathaniel Project – case study • Can I account for the use of the funds? • Characteristics of the consumers Psychiatric diagnosis Co-occurring substance abuse Intake charge Baseline conviction history Health problems Homeless at intake Gender, race, age Baseline utilization history (hospital) 31

32. Nathaniel Project – case study • Data 101 • Educate private funders to increase support for the program • Justify continued funding • Increase support for program services • Define and refinetarget population Non-violent and violent felonies Primary diagnosis and co-occurring disorder Baseline service needs - housing 32

33. Nathaniel Project – case study • Can I account for the use of the funds? • Staffing • November 1999 one full-time social worker & part-time legal director • May 2000 two social workers and part- time legal director • December 2000 four masters level clinicians and part-time psychiatrist 33

34. Nathaniel Project – case study • Simple Data • Program participant characteristics • 88% of participants mentally ill and chemically addicted • 92% homeless at time of release from jail • 48% chronic health conditions • 78% not engaged in treatment at time of arrest • 50% violent instant offense 34

35. Nathaniel Project – case study • Simple Outcomes • Performance-based contract • Retention – 82% of participants retained in program at one year • 79% in permanent housing after one year in the program • Two participants out of 50 re-arrested on felony charges 35

36. Nathaniel Project – case study • Take Data on the Road • 2000 – 2003 City and State mental health officials • 2000 – 2003 City and State criminal justice officials • State budget office staff • City and State elected officials 36

37. 37

38. Nathaniel Project – case study Simple Recidivism data Participants in the Nathaniel Project demonstrated a dramatic decrease in arrests The number of arrests dropped from 101 (35 misdemeanor and 66 felony) arrests in the year prior to and including arrest on the charge that brought them into the program down to 7 (5 misdemeanor and 2 felony) arrests in the year since intake 38

39. 39

40. 40

41. Nathaniel Project – case study Take Data on the Road Nathan Kline Institute for Psychiatric Research and the Center for Health and Public Service Research at the NYU Wagner Graduate School of Public Service Qualitative study was funded through an NIMH Center Grant (The Center for the Study of Issues in Public Mental Health) 41

42. Nathaniel Project – case study Take Data on the Road Nathan Kline Institute for Psychiatric Research and the Center for Health and Public Service Research at the NYU Wagner Graduate School of Public Service Qualitative study was funded through an NIMH Center Grant (The Center for the Study of Issues in Public Mental Health) 42

43. Nathaniel Project – case study Qualitative Study The Nathaniel Project: Manualizing the Committed Work of an Innovative Program and Monitoring that Program’s Transformation Overview The goals of this qualitative study are to describe the nature and characteristics of an innovative program that provides alternatives to incarceration for felony offenders with serious mental illness (the Nathaniel Project at the Center for Alternative Sentencing and Employment Services) and to then use this description of the key operating principles of the program as a baseline to track the program’s evolution over the next year as it transforms itself into an ACT Team 43

44. Nathaniel Project – case study • Take Data on the Road • 2002 Special Achievement Award from the American Psychiatric Association • 2002 American Probation and Parole Association’s President’s Award • 2002 Thomas M. Wernert Award for Innovation in Community Behavioral Healthcare 44

45. Overview • Role of Data in Decision Making • Minimum Level of Effort for Data • Data and Impact on Sustainability

46. Role of Data in Decision Making 46

47. Recidivism Outcomes Recidivism Results • 89 percent decrease in arrests between pre-intake arrest and post-release period • 2.2 average arrests before intake • 0.25 average post release arrests • 97 percent decline in convictions

48. Role of Data in Decision Making June 2003: CASES began to operate the licensed Nathaniel Assertive Community Treatment (ACT) program, two-year ATI with 68 treatment slots CASES responded to statewide RPF for ACT to sustain demonstration Nathaniel Project Received waivers to provide ACT to criminal justice involved population Certified by NYS Office of Mental Health as licensed provider of ACT services Enrolled by NYS Department of Health as Medicaid Provider 48

49. Role of Data in Decision Making Two-year contract NYC Office of the Criminal Justice Coordinator supports intake NYS Division of Probation and Correctional Alternatives (DPCA) supports court liaison services March 2009, supported employment enhancement funded competitive grant from DPCA New York City Council to support ACT transitional housing 49

50. Role of Data in Decision Making Blended funding: mental health, Medicaid, and criminal justice Nathaniel ACT Team operations are funded by Medicaid, NYS Office of Mental Health (OMH), NYS Division of Probation and Correctional Alternatives (DPCA), NYC Criminal Justice Coordinator, and NYC City Council 50