Consumer Involvement in Evaluating a Recovery-based Systems Change Initiative

1. Consumer Involvement in Evaluating a Recovery-based Systems Change Initiative Vicki Cousins, Director Office of Consumer Affairs South Carolina Department of Mental Health Katherine M. Roberts, MPH Coordinator, Consumer-to-Consumer Evaluation Team South Carolina Department of Mental Health

2. Words to Live Byin South Carolina “Nothing About Us Without Us” Consumer Involvement in Four Key Areas • Planning • Policy-making • Service Provision • Program Evaluation

3. Consumer Involvement in Planningand Policy-making • Central Administration Offices • Regional Service Sites (Community Mental Health Centers in 17 locations serving 46 counties)

4. Consumer Involvement in Planning and Policy-making New SCDMH consumer-centered policies/directives being developed as a result of our system’s emphasis on recovery.

5. Consumer Involvement in Service Provision Following Larry Fricks’ Georgia model, South Carolina has developed a Medicaid-billable service description for peer support • 40-hr. certification training for consumers to become credentialed as “CPSS” • Includes oral and written examination • Final approval from DHSS expected July 1, 2003 • 10 graduates to date; trainings to be year twice a year

6. Consumer Involvement in Program Evaluation • SCDMH Consumer-to-Consumer Evaluation Team • Several Years in the Making • 1996 … • … 2002 Found our place in the system’s • recovery initiative

7. The Purpose of Consumer Involvement Surgeon General’s Report 1999 Consumer staff, clients, and the mental health system all benefit from consumer involvement.

8. The Purpose of Consumer Involvement a Mental Health System Surgeon General’s Report 1999 Consumer Staff • meaningful activity or employment • role models of recovery • enhance sensitivity of the service system

9. The Purpose of Consumer Involvement a Mental Health System Service system benefits: • Increases in service satisfaction as related to people’s recovery needs • Reduced hospitalizations • Cost-savings in community program operating costs (Chamberlin & Rogers, 1990; Carpinello & Knight, 1993)

10. SCDMH Consumer-to-Consumer Evaluation Team Project Mission To involve individuals who receive services, family members, independent advocates and citizens-at-large in program evaluation and continuous quality improvement of the South Carolina Department of Mental Health.

11. SCDMH Consumer-to-Consumer Evaluation Team • Office of Consumer Affairs, Division of Health Care Reform • Consumer-driven evaluation survey process • Primary consumer team members • Paid employees of SCDMH • Participates in instrument development • Conducts interviews, compiles data, and produces reports

12. History of Consumer Involvement in Evaluation • For several years in the late 1990s, the Team conducted open-ended surveys with over 900 clients, striving to evaluate individual CMHC programs. • Starting in 2002, a fast evolution began as the emphasis shifted from program evaluation to system evaluation. Survey questions became closed ended.

13. History of Recovery Measurement in South Carolina • March 2002: Formal planning process for recovery system evaluation begins. Expert researchers partner with the CCET. • June 2002: Consumer focus groups, staff surveys • August–October 2002: Surveying to test for validity and reliability

14. Instruments • MHSIP • Consumer-to-Consumer Evaluation Scale (CCET) • Recovery Questionnaire

15. South Carolina Consumers Define Recovery Focus Groups • What does it mean to be in recovery or recovered from mental illness? • What helps a person recover from mental illness? • What prevents a person from recovering from mental illness?

16. Recovery Questionnaire Domains (10) from the Focus Groups • Hope - Self-esteem - Housing • Employment - Stigma - Self-management Skills • Empowerment/Independence/Self Responsibility/Control • Relationships - Social • Relationships - Family • Spirituality/Higher Power/Religion

17. Sample Size • Plan A: Sample consisted of 50 adult clients with open cases in all 17 community mental health centers. Expectation: 25 (425) would complete the initial survey and half of that number would agree to be resurveyed 2 to 4 weeks later. Plan A Results: <10% available to participate. • Plan B: Asked for available adult clients with open cases to volunteer.

18. Method • Informed consent attained. • Initial face-to-face interviews conducted by CCET team members. • Follow-up surveys were conducted either face-to-face or by phone. The process was the choice of the client.

19. Incentives for Client Participation • Each client received a button, a $5 McDonalds gift certificate, and a certificate of appreciation signed by the SC State Mental Health Director, George P. Gintoli, for the initial interview. • On the second interview, clients received 2 more buttons and an additional gift certificate.

20. Process • 400+ clients were surveyed/resurveyed from mid-August through mid-October 2002. • Scannable forms were reviewed and prepared for data entry. • All analysis was conducted in a de-identified manner.

21. Analysis The psychometric properties of the MHSIP, the Recovery Questionnaire, and the Consumer-to-Consumer Evaluation Scale included: • Item-to-Scale Correlations • Test-Retest Reliability

22. Results • Based on the research, changes were made to the Recovery Questionnaire and to the Consumer-to- Consumer Evaluation Scale. • The Recovery Questionnaire is presently being used by SC Share in South Carolina to evaluate their “Recovery for Life” program. • Plans are underway by South Carolina’s Columbia Area MH Center, the largest community mental health center, to utilize the Recovery Questionnaire in June 2003.

23. Lessons Learned • Challenging to go to a full-system survey approach in a limited time period – 2 times in 8 weeks - from individual program site assignments. • Dramatic increase in consumer hours, conflicted with consumers’ Medicaid benefits, stressed Team Members. • But still … state car break-downs, a crash, getting lost, getting sick, losing surveyors … It was a terrific learning experience with positive outcomes. We survived it all.

24. Consumer Involvement in Evaluating a Recovery-based Systems Change Initiative Vicki Cousins, Director Office of Consumer Affairs South Carolina Department of Mental Health Katherine M. Roberts, MPH Coordinator, Consumer-to-Consumer Evaluation Team South Carolina Department of Mental Health