The National Evaluation of the PATH Program

1. The National Evaluation of the PATH Program Pamela J. Fischer, Ph.D. SAMHSA’s Center for Mental Health Services Homeless Programs Branch Presented at the 2010 PATH Grantee Meeting, Washington, DC

2. Findings from the 2005 National Evaluation of the PATH Program—Pamela Fischer, Ph.D. Design and upcoming activities of the current evaluation—Paul Brounstein, Ph.D. Questionsand discussion Today’s Presentation

3. Congressional Mandate for the National Evaluation of the PATH Program • Section 528 of the Public Health Service (PHS) Act requires the SAMHSA Administrator to evaluate the expenditures of PATH grants at least once every 3 years to ensure they are consistent with legislative requirements and to recommend changes to the program design or operations • Evaluation design is determined by the Federal program staff and conducted through a competitively awarded contract 3

4. Outreach is emphasized by PATH providers and is carried out conscientiously and with quality PATH program uses funds appropriately and effectively Flexibility is widely praised High levels of job satisfaction PATH not necessarily recognized as a program as PATH funds represent only one of many small, but important funding streams received by providers Findings from Previous Evaluations

5. National Evaluation of the 2005PATH Program 5 Conducted during 2006-9 by the MANILA Consulting Group, Inc.: Walter Leginski, Ph.D., and Caroline Fernandez, Project Directors

6. Are services appropriate? Are services well-administered? Are outcome and process goals achieved? increases in the number of persons contacted, enrolled, and receiving mental health services clients placed in permanent housing Evaluation Questions

7. Focused Analyses PATH and rural homelessness Effect of local context on PATH performance Housing Poverty Other Federal funding

8. Methods 5 VPGs informed the evaluation approach emphasis on literal homelessness active State management specific guidance on reporting and definitions focus on exemplary practices active support of transition to mainstream services 2005 selected for analysis due to completeness of reports Data sources included administrative data and contextual information Design and interpretation guided by Technical Panel

9. Results Consistent with the findings of prior evaluations, PATH demonstrated that it continues to meet the three objectives of the evaluation

10. Appropriateness of Services Enrolled PATH clients match the eligibility profile and intent of the PATH legislation exceedingly well PATH providers offer a wide range of services from the PATH menu High proportions of enrolled clients participate in these services Engagement of clients into services and their transition to mainstream services are emphasized

11. Quality of Service Administration Program structure and approach Responsiveness of States and providers to information requested in applications Program monitoring States’ efforts toward adoption of Voluntary Performance Goals (VPGs) Variety of administrative approaches adopted by providers featuring collaborations that would expedite clients’ access to mainstream services and housing

12. Achievement of Identified Outcome and Process Goals GPRA goal achievement was similar to past years Housing collaborations developed to assure access to affordable housing for PATH clients

13. PATH Providers in Rural Locations Received smaller awards Challenges to collaboration with providers in resource-poor environments Rural providers perform equally and in some cases better than providers in metropolitan and mixed (urban/rural) areas

14. Correlating PATH Performance with Poverty Rates, Affordable Housing, and Other Targeted Homelessness Funding Levels PATH provider performance was strong, even in locations where high poverty and lack of affordable housing were evident Additional sources of Federal homelessness funding in the same locale as PATH providers did not detract from PATH performance and supported the concept that multiple providers function as a system of homeless services, enhancing PATH performance

15. Summary PATH is a critical funding stream PATH leverages resources and systemic approaches directed to the most vulnerable subpopulations of homeless persons As populations and policies change in State and national environments, the triennial evaluations are ideal opportunities to assess whether adjustments are needed to sustain PATH’s important contributions to the welfare of homeless persons with serious mental illnesses

16. For additional information contact: Pamela J. Fischer, Ph.D.Homeless Programs BranchCenter for Mental Health ServicesSubstance Abuse and Mental Health Services Administration(240) 276-1901Pamela.Fischer@samhsa.hhs.gov

17. The National Evaluation of the PATH Program Paul J. Brounstein, Ph.D. MANILA Consulting Group Project Director Presented at the 2010 PATH Grantee Meeting, Washington, DC

18. PATH Formula Grant to 50 states, D.C., and 5 Territories To enable States and providers to better allocate resources and improve service availability, accessibility and fill gaps in existing service system to decrease homelessness for individuals and families affected by serious mental illness

19. Key Services Community-based outreach Screening and diagnostic services Habilitation and rehabilitation services Community mental health services Alcohol or drug treatment services (for people with mental illnesses and co-occurring substance abuse disorders) Case management services Supervisory services in residential settings Limited housing services and services to help clients use housing resources

20. Most Common Services 88% of grantees offer outreach services 82% of grantees offer case management services

21. Evaluation Congress requires triennial evaluation answering 3 questions: Are services using PATH monies appropriate (Stewart B. McKinney Homeless Assistance Amendments Act of 1990)? Are services well-administered? Have process and outcome goals been achieved?

22. Current Evaluation Plan goes further and assesses the role of: Grant Structure Organizational Structure Level of Collaboration Mix of Services Populations Served/Characteristics Role of Context on Outcomes Urbanicity, Poverty, Labor and Housing Markets

23. Evaluation Process Developed initial evaluation specifications in response to solicitation for proposals Assembled expert panel Multiple telephone conferences to review plan Plan finalized and approved by SAMHSA

24. Data Requirements Answers to the multiple questions and purposes of the evaluation require data from several different sources: Archival/secondary data (e.g., Annual reports, State Applications, IUPs, Census data) Primary data collection (SPC and provider surveys, site visit assessments and focus groups with PATH clients) Data selected to support analysis of Program Model

25. PATH Logic Model

26. Other Measures Identification of PATH clients Sustained engagement of clients Linkage of PATH and other services Accessibility to and use of housing and MH care opportunities Acceptability of services Gaps (training, outreach, assessment, services, linkages, measurement and reporting Trends in homelessness (populations, mix of services) Identification of innovation and use of best practices

27. Data Sources Archival/Secondary Data 2009 State Applications and Annual Reports (context, grant/organizational structure, population and service characteristics, GPRA) Bureau of Census (American Community Survey), HUD (HMIS homelessness count), National Survey of Homeless Assistance Providers (context, community and population characteristics)

28. Primary Data Collection 14 Site Visits 4 site visits between 12/6/10 and 1/30/10 Pre-test in-person interview and online survey data collection instruments and client focus group questions for comprehensiveness and usefulness Interviews w/SPC, at least 2 provider staff and 1-2 clients 10 assessment visits after OMB approval Detailed assessment of grant structure; process; perceived outcomes and service gaps. Recommendations

29. Data Requirements Site Assessments SPC and key state staff 1-8 provider Directors and key staff offering direct service to clients Depends on size and distribution of provider network Focus group with 8-12 clients re: acceptability, quality, gaps.

30. Online Surveys Developed with recommendations from SAMHSA staffers, Expert Panel and pre-test participants Will be approved by OMB Focused on assessing grant implementation process, throughput, outcomes and gaps 56 SPCs 483 Local providers (1/provider agency—staff who provide direct face-to-face services)

31. Online Surveys cont. Survey will require about 30 minutes to complete Unique password assigned to respondent will allow completion over multiple sessions Your cooperation in completing these surveys is ESSENTIAL!!! Please help.

32. Analyses Describe PATH implementations Look for commonalities in structure, populations served and service mix Create a typology of program implementations Relate PATH program models to outcomes to see if there is differential effectiveness Identify best practices Inform SAMHSA about service, training gaps and inform decisions on technical assistance still needed

33. Evaluation Project Status Developed reliable coding and data extraction procedures for State applications and IUPs Currently coding both, expect completion by mid-February Preparing Annual Report Data to merge with coded information extracted from Annual Reports and State Applications Pilot testing of protocols and instruments begun on December 6. Will complete around mid January OMB draft with revised instruments to SAMHSA by the end of January Field site assessments and surveys should begin by July 2011.

34. Questions and Comments Lisa Kleppel or Paul Brounstein MANILA Consulting Group 6707 Old Dominion Drive McLean, VA 22101 571-633-9797