CalMHSA Prevention and Early Intervention (PEI) Program Evaluation Technical Assistance and Capacity Building

1. CalMHSA Prevention and Early Intervention (PEI) Program EvaluationTechnical Assistance and Capacity Building Facilitating Stakeholder Involvement Through Collecting and Reporting Program Data

2. Overview • Part 1: Tool description and rationale • Part 2: How to use the tool • Part 3: Using program data to facilitate stakeholder involvement

3. Part 1: Tool Description and Rationale

4. MHSA Requires Counties Prepare a 3-Year Program and Expenditure Plan • Plan must be developed with “meaningful stakeholder input” • Clients and families must be involved in all aspects of community planning process • Plan must be updated annually

5. What is “meaningful” stakeholder input? • “Knowledge is power” • In order to make informed decisions, stakeholders need accessible information on current program activities • Making data easy to understand builds confidence • Important to have an easy way to collect, analyze and present data in a way that facilitates meaningful stakeholder input

6. MHSOAC May Require Counties to Report Program Data Proposed requirements include: • Types of programs • Number served and age distribution of participants for each program • Cost per participant by program • Other performance outcomes - not yet defined

7. County-funded CalMHSA TTACB Supported RAND/SRI Data Collection and Reporting Tool • Developed tool in 2013 and pilot tested it with counties at TTACB work groups • Revised tool using feedback from work group participants and experience from TTACB individual technical assistance to counties and programs • Designed to display data for stakeholders in easy-to-understand format • Designed to meet current reporting needs established by MHSOAC

8. Tool Consists of Three Parts • Individual PEI Program Description Questionnaire • All PEI Program Description Workbook • Stakeholder PEI Presentation PowerPoint Slide Deck

9. Part 2: How to Use the tool

10. A Three Part Tool to Describe County PEI Programs • Individual PEI Program Description Questionnaire • Can be completed by each PEI program • Two formats, “fillable” and paper/pencil • All PEI Program Description Excel Workbook • County staff enter data from all programs • Results automatically calculate and create graphics • Stakeholder PEI Presentation PowerPoint Slide Deck • Easily drop graphics into slides for presentations

11. Relationship Among Three Parts All Program Description Excel Workbook Individual Program Questionnaire Stakeholder Presentations Slide Deck

12. Part 1 - Individual Program Description Questionnaire • Should be filled out individually for each PEI program in the county • Should probably be completed by program staff, but could be completed by county staff • Includes instructions and definitions • Formatted for easy and accurate data entry • Purpose: Collect accurate data about each PEI program

13. Individual Program Description Questionnaire Data Collection Steps • Fill in Name of County Contact person on the first page of the questionnaires (both versions) • Distribute the questionnaires to an email mailing list of all county direct service and contract PEI program directors • Note returns against the mailing list • After ten days send a reminder to any program directors who have not yet responded • Continue to log in returns against the original mailing list • On day 14 send a deadline reminder to any remaining programs that have not responded

14. Suggested Request Email for PEI Program Description Questionnaire Dear Program Director Completion of this questionnaire is being requested from all contract and direct service county PEI programs. The purpose of the questionnaire is to help our county to describe the PEI services it funds. Please complete it within two weeks of receipt date and return it to the contact below: Name  of County Contact: Email Address: Phone Number:  If you have any questions, not addressed below, please reach out to the County Contact person named above.  Thank you for your cooperation. 

15. Questionnaire Instructions • Should be completed by the program management staff person most familiar with the program • Provided in two formats—paper and pencil or a fillable form that can be completed online • Provide your best estimate when precise information to answer the question is not available • Definitions of program categories and examples are provided on page 7 and 8 of the questionnaire • Please save a copy of your completed questionnaire for your records

16. Overview of Questionnaire Items 1. What is the name of the program you are describing? (Please provide full name and any acronym commonly used)

17. 2. Is this program primarily focused on: (Please check one box) 1Prevention, or 2Early intervention, or 3System change efforts

18. 3. What are the main activities that this program carries out? Many programs do a bit of everything, so try to identify the most important activities for this program, which are directed at the target population for the program. Please refer to Guide below in determining program activities. (Check all that apply) 1Outreach and public awareness campaigns 2Community strengthening 3Gatekeeper education and training (including program staff training) 4Counseling and support 5Screening and referral 6System change efforts 7Clinical services for early intervention 8Other – what activity? __________________________

19. 4. What was the approximate total amount of PEI funding for this program in fiscal year 2012-13? Do not include funding for the program that comes from other sources. $ ____________

20. 5. Please indicate as best you can how the total funding was used across the main activities of this program in fiscal year 2012-13. If an activity is not part of this program, leave the line blank. • ACTIVITIES% OF total program FUNDING • Outreach and public awareness campaigns %_________ • Community strengthening %_________ • Gatekeeper education and training %_________ • Counseling and support %_________ • Screening and referral %_________ • System change efforts %_________ • Clinical services for early intervention %_________ • Other – what activity? %_________ • 100% = TOTAL FUNDING

21. 6. What is the intensity of this program’s main activities taken as a whole? (Please check one box) 1 Single or light touch activities 2 More intensive activities 3 Both levels

22. Duplicated vs. Unduplicated Counts • Duplicated Count: Refers to a program participant who may be counted more than one time in a grant year. This might occur if a client received multiple services in the same reporting period within one program or across multiple programs • Unduplicated Count: is one (1) person/client that is counted only once, no matter how many different services the client is receiving during the funding period– could be within or across programs. 22

23. About how many individuals were served by this program in 2012, based on a duplicated and/or unduplicated count? • Duplicated count – individuals may be counted more than one time if they receive several services within one program within the same reporting period. • Unduplicated count – individuals are counted only once, no matter how many different services a client is receiving within a program in the same reporting period. 7. Please provide an unduplicated count if possible. ______________ Number of unduplicatedindividuals or [ ] Unduplicated count not available

24. 8. Please provide a duplicatedcount if available. ______________ Number of duplicated individuals or [ ] Duplicatedcount not available

25. 9. What are the age groups mainly targeted by this program? (Check all that apply) 1Children – Age 0-15 2TAY – Age 16-25 3Adults – Age 26-59 4Seniors – Age 60+ 5No main group(s) – targets all ages equally

26. 10. What are the race/ethnicities mainly targeted by this program? (Check all that apply) 1Hispanic/Latino 2White 3Black/African American 4Asian 5American Indian, Native American, Alaskan Native 6Native Hawaiian, Pacific Islander 7Other 8No main group(s) – targets all equally

27. 11. Are materials and program activities provided in languages other than English? (Please check one box) 1Yes – most of them 2Yes – some of them 3Materials and activities are in English only

28. 12. What special groups are mainly served by this program? (Check all that apply) 1Veterans 2Military 3Foster care children/youth and/or families 4LBGTQ 5Undocumented immigrants 6Low income, Medicaid eligible 7Underserved 8Program designed to serve everyone, no targeting of any special group 9Other 10NONE OF THESE

29. 13. In the design or operations of this program have there been any specific adaptations to make it more culturally appropriate for the target group(s)? (Please check one box) 1Yes 2No 3Not applicable

30. 14. We are going to ask about the program and public health goals targeted by this program. Programs may have multiple goals so please try to identify the main goals for the program. What are the main goals for this program? By goals we mean the most immediate intended results of this program. (Check all that apply) 1Increased knowledge and more supportive attitudes about mental illness 2Decreased risk behaviors and/or increased coping skills 3Decreased distress and/or fewer and less severe symptoms Could also include preventing crisis situations from escalating 4Improved individual functioning (could be at home, school, and work) and increased resilience and well-being 5Improved family functioning 6Increased help-seeking and utilization of additional assessment and/or treatment services 7Increased community resiliency, such as improved social climate, reduced stigma and discrimination, and more social supports available in the community 8Improved systems of care which could include more coordinated and culturally responsive services, increased capacity and access, improved policies and protocols for how individuals move between systems (e.g.,primary care to specialty care, or juvenile justice to the school system)

31. 15. Programs may or may not directly target public health goals, but may be intended to contribute to reducing mental health related public health problems. What mainpublic healthproblems is this program intended to reduce:(Check all that apply) 1Suicide 2School dropout related to mental health 3Removal of children from the home related to mental health 4Incarceration related to mental health 5Homelessness related to mental health 6Unemployment related to mental health 7Stigma and discrimination related to mental health 8Prolonged suffering related to mental health 9Disparities related to mental health treatment access and/or utilization

32. 16. Are there main program goals that were not covered in these questions? (Please check one box) 1Yes -> What goals? 2NO, ALL WERE COVERED

33. Guide to classifying program activities OUTREACH AND PUBLIC AWARENESS CAMPAIGNS Purpose:To increase public knowledge and awareness. Target:General public - individuals not identified on the basis of a specific risk factor. Examples:Public service announcements; developing and hosting informational websites; creating and distributing pamphlets or other printed material for wide scale distribution; stigma reduction campaigns; hosting health fairs. Note:Do not include outreach intended to recruit program participants or build community partnerships in this category.

34. Guide to classifying program activities COMMUNITY STRENGTHENING • Purpose:To increase community resources and resiliency, or to improve social climate and increase social support. • Target:These are universal prevention activities and are not directed towards individuals who have specific risk factors. • Examples:A resource or drop in center (if the primary purpose of the resource program is community strengthening); a school-based anti-bullying program directed at the entire school or community; an exercise program for the elderly; NAMI Family to Family programs and vocational programs; Parent education programs for all parents; school readiness programs that are targeted at all children.

35. Guide to classifying program activities GATEKEEPER EDUCATION AND TRAINING • Purpose:To train and educate individuals to recognize someone who is in distress and could benefit from further evaluation or treatment and/or to create a safe and non-stigmatizing environment where someone can self-identify as needing help. • Target:Community leaders, school personnel, emergency workers, and other gatekeepers. • Examples:Teacher training programs, training for suicide hotline operators, training for outreach workers and community leaders. • Note:This category includes training of program staff.

36. Guide to classifying program activities COUNSELING AND SUPPORT • Purpose:To reduce risk factors and increase coping skills. • Target:At-risk individuals, rather than individuals with a known diagnosis. These programs do not include universal prevention programs. • Examples:Peer support programs; Resource centers (if the primary purpose is to provide counseling and support); Suicide hot and warm lines; Parent training programs for children with identified needs or risk factors; Violence prevention programs directed at selected children or youth; Promotoras home visiting programs. • Note:These activities are generally conducted by individuals who are not licensed mental health clinicians.

37. Guide to classifying program activities SCREENING AND REFERRAL • Purpose:To identify individuals who might benefit from additional assessment and/or counseling. • Target:At risk individuals such as foster youth, new moms, frail elderly. • Examples:School-based screening; suicide hot and warm lines, some Promotoras programs.

38. Guide to classifying program activities SYSTEM CHANGE EFFORTS • Purpose:To improve system functioning in order to increase appropriate identification of need and access to services. • Target:Public and private systems. • Examples:Efforts to develop and disseminate culturally appropriate policies and procedures and culturally competent services, as well as data systems to track individuals. Development of referral networks and efforts to increase and make more efficient collaboration across systems and programs. Efforts to increase access and capacity by reducing inefficiencies could also be included here.

39. Guide to classifying program activities CLINICAL SERVICES FOR EARLY INTERVENTION • Purpose:To keep mental illnesses from becoming chronic and increasing in severity. • Target:Individuals with either a known or likely diagnosis or symptoms related to a diagnosis. • Examples:First break psychosis programs, PTSD/PTSD symptom treatment programs; primary care integration programs; school-based trauma treatment; family therapy. • Note:These activities are generally conducted by licensed mental health clinicians.

40. Part 3: Using Program Data to Facilitate Stakeholder Involvement

41. All Program Description Excel Workbook • Example worksheets (labeled EX_Entry on the tabs) show how the Excel worksheets should be completed for the questionnaire items shown on the tabs • EX_Outputs shows example tables and graphics calculated from the data • Using the three sheets labeled AC_Entry, use one row for each program to enter the responses to the Individual PEI Program Questionnaire (make sure the program number is same on all sheets) • Check that all entries are accurate • Excel automatically calculates summary statistics and produces the tables and graphics on AC_Outputs_Sheet • Purpose: Enter the responses received on the Individual PEI Program Description Questionnaires and create graphics

42. PLEASE REFERENCE “THE ALL PROGRAM DESCRIPTION EXCEL WORKBOOK “ IN THE DOCUMENTS LIST AT THIS TIME

43. Stakeholder Presentation Slide Deck • The PowerPoint slide deck contains charts that can be populated using the All Program PEI Description Excel Workbook worksheet labeled AC_Outputs_Sheet • Begin by saving a copy of the Slide Deck with the title of your upcoming presentation • You can select some or all of the Excel graphics depending on your audience and purpose • Simply click on a graphic chart in the AC_Outputs_Sheet tab of the Excel Workbook, copy it and paste it into the PowerPoint slide with the same title - Use the “destination theme” Paste function Purpose: Share descriptions of PEI programs with stakeholders

44. Workbook Creates Graphics for Slides • As program data are entered the workbook will automatically generate graphics • Graphics can be “dropped into” slide deck for presentations to stakeholders • Slides summarize the data provided by individual programs into an easily understandable format • Program information will help stakeholders provide meaningful input into the decision-making process

45. Example Charts for “Golden County” • List of programs and characteristics • Cost • Count of participants • Intensity • What activities programs are doing and distribution of dollars across activities • Target participant characteristics • Short term goals • Public health goals

46. List of Programs and Characteristics

47. Distribution of Programs by Primary Focus

48. Types of Activity • Outreach and public awareness campaigns • Community strengthening • Gatekeeper education and training • Counseling and support • Screening and referral • System change efforts • Clinical services for early intervention

49. How Many Programs are Doing Each Type of Activity?

50. How are PEI Funds Distributed Across Activities?