What Did We Say We Were Going to Do?

What Did We Say We Were Going to Do? Afternoon Session: 1:00pm – 3:00 pm • Overview of Measurement and Change Paradigms • 2. Audience Review and Discussion of Full Service Partnership Strategies T. Stephanie Oprendek, Ph.D. Acting Chief, Evaluation, Statistics and Support Branch California Department of Mental Health Full Service Partnerships - January 30 – 31, 2007

Tri-Level Performance Measurement Paradigm

Community Reaction / Evaluation / Satisfaction with regard to mental health system Community Reaction / Evaluation / Satisfaction with regard to mental health system Mental Health System Structure / Capacity in Community Mental Health System Structure / Capacity in Community Large-Scale Community Indicators Large-Scale Community Indicators Mental Health Promotion and Awareness Mental Health Promotion and Awareness TRI-LEVEL PERFORMANCE MEASUREMENT PARADIGM PUBLIC / COMMUNITY- IMPACT LEVEL (Evaluation of Global Impacts and Community-Focused Strategies) MENTAL HEALTH SYSTEM ACCOUNTABILITY LEVEL (Evaluation ofCommunity Integrated Services and Supports – Program/System-Based Measurement) Monitoring / Quality Assurance / Oversight (multi-stakeholder process) Monitoring / Quality Assurance / Oversight (multi-stakeholder process) Staff / Provider Evaluation / Satisfaction with regard to mental health system Staff / Provider Evaluation / Satisfaction with regard to mental health system Client / Family Satisfaction / Evaluation of Services and Supports Client / Family Satisfaction / Evaluation of Services and Supports INDIVIDUAL CLIENT LEVEL (Evaluation of Community Integrated Services and Supports – Individual Client Tracking) Individual Client Outcomes Tracking Individual Client Outcomes Tracking Client and Services Tracking Client and Services Tracking

PERFORMANCE MEASUREMENT • Individual Client Outcomes Tracking (Examples) • Initial and periodic assessments • Ongoing assessments of core outcomes. The following are examples • (State and local information systems interoperability, based on statewide standards, will be the mechanism by which this client outcome information is captured. DMH will work with counties/providers to provide flexible system options with regard to measurement of outcome indicators.) Recovery & Wellness Oriented Client Outcome Indicators : (These are examples; indicators and measures to be determined though stakeholder and committee recommendations.) Hopefulness Wellness Empowerment Self-efficacy, Etc… Housing Functioning Criminal justice system involvement Substance Abuse Employment / Education Quality of Life Hospitalization (acute//long term restrictive levels of care) Illness self-management Culture-specific indicators Income / Entitlements Social / community connectedness Family preservation Individual service plan goals Symptoms / Suffering Physical health Suicide Etc. INDIVIDUAL CLIENT LEVEL (Evaluation of Community Integrated Services and Supports – Individual Client Tracking) • Client and Services Tracking (Examples) • Client-specific information, e.g., contact, demographic information, reason for system disengagement, etc. • Services / supports information, e.g., new services/programs/supports pertinent to the MHSA, evidence-based practices, levels of care, partnering agency/provider services, etc. • (Client and services/supports data capture is envisioned to be achieved through interoperable information systems residing at both the state and local levels. A phased-in approach will be used to achieve this long-term goal of full interoperability.)

PERFORMANCE MEASUREMENT MENTAL HEALTH SYSTEM ACCOUNTABILITY LEVEL (Evaluation of Community Integrated Services and Supports – Program/System-Based Measurement) • Monitoring / Quality Assurance / Oversight (multi-stakeholder process)(Examples) • Local / county plans and performance with respect to: • Cultural competency / no disparities • Recovery / Resilience philosophy and promotion • Full participation of clients / family members in service delivery system processes • Fidelity to evidence-based practice guidelines or model programs • Adherence to budget / timelines • Staff / provider competencies • Adherence to appropriate client-to-staff ratios • Quality (performance) improvement projects • Service partnerships - Comprehensive / inter-agency / coordinated service delivery • Supportive services (e.g., housing, employment, peer-delivered supportive services) • Coordinated services for co-occurring disorders • Costs, cost-effectiveness of services • Etc. • (Measured with standardized review criteria, monitoring tools, electronic data entry / reporting interfaces, etc. Cost information to be associated with client, service, and outcomes tracking information to determine costs per client, cost-effectiveness and cost-benefit analyses of programs, etc.) • Client / Family Satisfaction / Evaluation of Services and Supports (Examples) • Mental Health Statistics Improvement Program (MHSIP) indicators and surveys • Surveys / assessments targeting specific services / supports appraisal by clients / families / caregivers • Focus groups / multiple means of eliciting client / family / caregiver input • Etc. • Staff / Provider Evaluation / Satisfaction with regard to mental health system (Examples) • Perceived effectiveness of the structure of system, inter-agency issues, effectiveness of service models, etc. • Interviews / surveys/ focus groups • Etc.

PERFORMANCE MEASUREMENT PUBLIC / COMMUNITY- IMPACT LEVEL (Evaluation of Global Impacts and Community-Focused Strategies) • Mental Health Promotion and Awareness (Examples) • Outreach services (e.g., homeless, rural communities, ethnic/culture-specific outreach, Tele-health, etc.) • Community Emergency Response Team Services • Community Mental Health / Depression Screenings • Educational Seminars (e.g., general public, primary care settings, schools, etc.) • Anti-Stigma and Anti-Discrimination Campaigns • Prevention and Early Intervention Efforts • Workforce Recruitment and Development (e.g., university, licensing board collaborations, continuing education) • Community Support Groups • Media, public awareness announcements, (e.g., Recovery & Resiliency) • Access and educational enhancements (e.g., Network of Care website, promotion of recovery philosophy) • Etc • (Typically measured by counts of individuals reached, screened, informed, etc.) • Mental Health System Structure / Capacity in Community (Examples) • Inventory of available services & supports (includes cultural competency and language proficiency) • Location of services, including inter-agency, outreach, mobile, natural/community setting, etc (e.g., GIS mapping) • Etc. • Community Reaction / Evaluation / Satisfaction with regard to mental health system (Examples) • Media reviews • Interviews with public officials • Assessment of community members • Etc. • Large-Scale Community Indicators (Examples) • Population prevalence of mental illness • Community mental health need / unmet need • Percents of youth in juvenile justice or Level12-14 group home placements • Etc.

Accountability and Measurement of Outcomes, Objectives and Process

PROCESS / METHODS OBJECTIVES OUTCOMES The other component is the demonstration that the mental health system is performing appropriately in providing services, supports, and activities - that is,doing what it should do & said it would do. One component of accountability is the effectiveness of services, supports and activities as measured by individual client outcomes and community impact. ACCOUNTABILITY • These are the two “arms of accountability”: • They hold the MHSA transformational processes up to the light • They demonstrate that the mental health system is reaching out to both individuals and the community in ways that produce positive results.

OBJECTIVES: We can measure our achievement of particular objectives and interim outcomes on the path toward our ultimate (long term) goals/outcomes. PROCESS / METHODS: We must measure our process, which includes our methods and tracking our implementation, in order to be able to explain how we achieved our goals/outcomes/objectives.

e.g., Expansion Area Quality Improvement Need Identify Issue or Problem Longitudinal Change Paradigm • Measure Progress Toward Ultimate Goals/Outcomes • Progress toward ultimate goals/outcomes (e.g., impacting MH prevalence) is likely to be modest and slow paced. • Engaging multiple strategies and change partners potentially increases pace and magnitude of progress. • Determine Objectives Met, Measure Interim Client & Community Outcomes • Achieve objectives known/believed to lead to desired ultimate goals/outcomes. • A measurement is only an “outcome” because we chose a particular time as “endpoint”. The "What" and the "How" of Implementation. Measure Process: Track Implementation, Fidelity to Plan/Methods/Model Develop Concept/ Implement Intervention or Strategy Start

FSP Strategies Review and Discussion Evaluating and Documenting the Process -------- The “What” and the “How” of FSP Program Implementation

FSP Strategies Review and Discussion • An FSP strategy is not necessarily an EBP or “model”. Counties proposed service strategies reflective of their community planning process. • Not a question of “model adherence” but knowing if process is now consistent with plan and objectives. • Are FSP programs being implemented with respect to the “Key Constructs”? • If so, what is being done, and how?

FSP Strategies Review and Discussion • Review of Exhibit #4s: What was proposed in FSP strategies? • Evaluate the FSP strategies against the Key Constructs. • Review and document the “whats” and the “hows” of implementation to evaluate the strategies.

Areas in which to address the “Hows”: • Operationally • Administratively • Programmatically • Inter-agency • Interpersonally • Etc…

Key Constructs of FSP Programs • Community collaboration • Cultural competence • Client and family driven • Wellness/recovery/resiliency focus • Integrated services for clients and families (which could include health and substance abuse issues)

Some FSP Strategy Specifics that Reflect the Spirit of the MHSA • Use of flexible funding • Interagency collaboration • 24/7 service availability • Single point of responsibility • A “partnership” versus a provision of service to someone • Shared decision making on an individual’s road to recovery. • Linkage to or provision of all needed services • Individualized service plan that is person centered, with individuals and their families given sufficient information to allow them to make informed choices • Peer provided services • Strategies that reflect the CSOC and Wraparound core values and principles have been incorporated for children, youth, young adults and families • And more…..

Report Back to the Group • What is being done differently now in FSP programs under MHSA? (Choose one or two promising areas to share.) • How are these new things being accomplished? • Where is more info/specificity needed to understand the process better? • What areas pose the greatest challenges or have barriers to be overcome? • What are some action steps to address barriers? (Breakouts may provide some guidance.) This process may help counties in providing information for the annual MHSA implementation update.

What Did We Say We Were Going to Do?

What Did We Say We Were Going to Do?

Presentation Transcript