Selected References: Alkin, MC ed. Evaluation Roots: Tracing Theorists’ Views and Influences . Thousand Oaks: SAGE,

1. Lessons Learned from a Qualitative Evaluation of the Massachusetts Department of Public Health F.O.R. Families Program: Methodological Advantages and Challenges Zobeida E. Bonilla, PhD, MPH, Melissa Marlowe, RN, MS, and Karin Downs, RN, MPH Mapping Tools Key Findings Background • All clients interviewed indicated that F.O.R. Families was instrumental in their progress toward a more stable situation. Families reported that help with housing and with finding health care services were the most useful services that they received from program staff. • Other significant benefits received from the program include: counseling, parenting skills, health education, help with self-esteem and self-confidence, advocacy, help navigating the system and interacting with government and community agencies, phone calls to make appointments for health services, and provision of tangible items such as books, school supplies, toys, clothing, and food. • Immediate family, friends, and agencies were the primary sources of support that clients identified in the ecomap. Seventy three percent of clients identified F.O.R. Families as a source of support of either high or medium strength. Government and HAP agencies were also identified as strong sources of support (33% of families interviewed reported F.O.R. Families and other agencies as their only source of support). All clients interviewed reported more sources of support in their ecomaps than they had reported to the home visitors at the time of enrollment in the program. • Responses from families, home visitors, and key informants indicate that F.O.R. Families has assisted clients with the transition from homelessness to self-sufficiency by successfully linking families to needed services and programs. • F.O.R. Families has been instrumental in the families’ progress to a more stable situation in the five domains of need: housing, health, food security and nutrition, family economics, and social, community, and family support. • The relationship built between home visitors and clients appears to be central to the clients’ positive experiences with the program. Identified sources of support, for example, revealed that F.O.R. Families home visitors were among the closest sources (and often the primary or the only source) of support for the family. • Stabilization of health services, improved parenting skills, improved communication with an adolescent son or daughter, and increased self-esteem/self-confidence leading to self advocacy are examples of how the home visitors have encouraged positive, change-oriented actions that have helped families move closer to attaining self-sufficiency or to move out of homelessness. • The program has been a central player in • (1) identifying needs of families, • (2) delineating strategies to address identified needs, • (3) assembling/aligning resources, • (4) advocating for families, and • (5) providing caring and respectful support to the families in their journey towards self-sufficiency at different moments of the homeless continuum of care. • The F.O.R. (Follow-up, Outreach, and Referral) Families program is a home visiting program of the Massachusetts Department of Public Health and the Department of Transitional Assistance to provide services to homeless families. • This qualitative evaluation was conducted to (1) learn from families, program staff, and collaborators from sister agencies about best practices to provide services to homeless families, (2) define success from the perspective of program staff and families served by the program and thus inform the development of outcome measures, and (3) gather recommendations to facilitate the development of program tools such as a logic model and program protocols. • Several challenges emerged during the course of this project, including the validation of benefits and contributions of qualitative methodologies to evaluation research, the ability to interview families who represented extreme or very difficult cases and therefore learn from their experiences, the complexities of working with multiethnic populations, and the difficulties of obtaining data on homeless populations. ECOMAP Clients were asked to identify the sources of support that they had at the time of the interview and during the time they received services from the F.O.R. Families program. Reported sources of support were mapped on an empty page using arrows and circles to create a visual representation of the family’s support network. This tool, known as an ecomap was originally developed as a diagrammatic assessment of family relationships to study social networks (Hartman 1978; Ray & Street 2005). MAP OF BARRIERS AND STRESSORS Using concentric circles, families mapped sources of stress and barriers to self-sufficiency. In the circle closest to the family circle (inner circle) the client identified sources of high stress. In the middle circle clients identified medium-level difficulties, and in the outer circle clients identified low-stress barriers. The varied sources of stress that families identified (e.g., disability, sick child, financial situation, domestic violence, mental health) suggests that barriers to self-sufficiency go beyond family economics and that multiple stressors are perceived as obstructing the families’ progress towards stability and self-sufficiency. Design and Methods Program Tools Program Goal: To assist families in identifying and addressing barriers to attaining and maintaining self-sufficiency. Assumptions (1) Experiences of program beneficiaries missing (2) Descriptive/program implementation evaluation (3) Program aims to develop outcome measures (4) Program highly individualized & tailored to the needs of the family (5) Qualitative; centers on gathering the experiences and perspectives of families who have participated in the program (6) Application: generate recommendations, inform program, identify best practices (7) Participatory: involve stakeholders in data collection and analysis • Risk factors • Family not paying bills • Income levels • Substance abuse present • Domestic violence present • Mental health issues present • Eviction and reasons for eviction • CORI issues • Immigration status • Family left shelter • DSS involvement • Education history • When they had first job • Longest sustaining job • Out of work for 6 months • Money management history • Weak or non-existing social support system • Possible screening settings • DTA offices • Emergency rooms • Community Health Centers across the state • Community action programs • Immigration services • WIC offices • Psychiatric hospitals • Schools • Sample screening/assessment questions • Do you have safe and permanent housing? • Where are you staying? • How long have you been there? • Have there been changes in your housing situation since your last visit? • Do you have any concerns about your housing? • How many times have you moved in the last year? • What keeps you strong? What keeps you going every day Lessons Learned Guiding Questions (1) How has F.O.R. Families assisted clients with the transition from homelessness to self-sufficiency? (2) What are the perspectives and experiences of the families with the program? (3) Which program strategies (provided or created) were successful? Which may need modifications? (4) How has F.O.R. families benefited other agencies working with homeless families? (5) What measures of success could be developed to assess program’s impact? • Qualitative evaluations can contribute to the development of grounded program tools such as logic models, screening methods, and outcome measures. Qualitative methods provide a framework for program evaluation that takes into account the experience of program participants and providers as well as local definitions of success. This process can reveal valid and relevant outcome measures for the families. In this context, home visitors pointed out that the development of outcome measures for homeless families would have to be flexible enough to capture the small building blocks that lead to larger accomplishments and to increased stability in a spectrum of self-sufficiency. Programs, for example, may have a “grandiose idea about accomplishments, and these accomplishments may be very different from what the family wants” and perceives as needed at that particular moment in their lives to move on to the next step on their journey to self-sufficiency. These disjointed perceptions of goals (and means to achieve the goals) may contribute to the difficulties surrounding the delineation of relevant and appropriate measures of success to assess program impact. • Homeless families are under high levels of stress and face significant challenges. Traditional program outcome measures might not capture the smaller efforts and progress that lead to larger changes. In this context, qualitative methods can identify the unique and critical conditions that help homeless families move to self-sufficiency. • The development of outcome measures for homeless populations continues to be a challenge for researchers, stakeholders, and policymakers. The results of this qualitative evaluation indicate that F.O.R. Families home visitors, with their extensive experience and knowledge of the families, are in the best position to define realistic and appropriate measures of success for the families with whom they work. Hence, home visitors should be at the core of the conversations regarding outcome measures for homeless families. • Despite our efforts to address the challenges of conducting evaluation in multicultural settings, we continued to face the limitations of lack of interpreters and multicultural researches who could participate in the data collection. Language and cultural barriers posed significant challenges to understand the experience of immigrant and refugee homeless families from diverse ethnic and linguistic backgrounds. The instruments developed for this evaluation were not properly adapted to the ethnic and linguistic groups served by the program (e.g., Khmer language, refugee populations from Southeast Asia and African countries) and significant understanding of the needs of homeless families from these backgrounds were not appropriately captured. Concepts such as “success” or “barriers to self-sufficiency” were difficult to explore especially when there is – along with a prescribed method to attain self-sufficiency – a preconceived structural and institutional understanding of what such concepts ought to be in the context of homeless families. • In addition, we faced the traditional structural challenges and difficulties of working with homeless families (e.g., multiple phone calls, difficulties scheduling visits, phone disconnected, family not at home after scheduling a visit for the interview, unexpected situations/needs that emerged during the visit and needed immediate attention), and difficulties locating the families with more extreme barriers and challenges for the interviews. • However, this qualitative evaluation helped to identify factors that contribute to successful or unsuccessful delivery of services, identify intended or unintended outcomes linked to the program, and explore the needs of the groups served. It allowed us to explore what is meaningful progress towards stability among homeless families and to highlight that the context and the personal experiences of participants contribute valid and useful results. For whom Dept. of Public Health, Dept. of Transitional Assistance, Other state agencies, Families, Other partners Document inputs, activities, and outputs that lead to desired changes/outcomes Data Collection Assessment Domains INPUTS DOMAINS OF NEED ACTIVITIES DTA (ICM, Referrals, Funding) • Initial assessment of family status • Follow-up, outreach, & referrals • Home visits • Phone calls & letters • Health education • Counseling & advice (legal, health, parenting, etc.) • Identify resources & connect families with services • Reassessment and identification of needs and services • Advocacy • Housing • Health • Food Security and Nutrition • Family Economics • Social, Community, and Family Support Housing Stable housing • Qualitative data: • Narrative from the F.O.R • Families Access database • Qualitative interviews Quantitative data: Use of existing data from the F.O.R Families Access database for the FY 05 on the following selected variables: age, gender, education, family size & composition, race/ethnicity, number & type of referrals DPH (management, leadership, home visitors Other state agencies (services, referrals) Health & well-being Improved access to health care; allocation of needed health care services CBOs/ non-profits (HAP workers, referrals) • Sampling: • Purposive • 14 FOR Families HV • 10 Workers from DTA & other partner agencies • 15 families served by FOR Families in the last fiscal year • Methods • Key Informant Interviews • Focus Groups • Interviews with program home visitors • Interviews with program clients Families (needs, assets) Food security and nutrition Secure source of food & nutrition Outputs # referrals received from DTA # families served # mothers and children served # children with special care needs served # attempts made to reach families (home visits, phone calls, letters) # referrals to services in the five domains: health, housing, income, social, and food # services secured for families # and type of counseling session (budget, self-esteem, child’s behavior) # educational materials provided to the family (e.g., depression booklets, program brochure, list of resources available in the community, others) # children served through child care programs, summer programs, after school programs # jobs # times family paid his/her portion of the rent length of phone service # referrals to the following specific services: mental health, domestic violence, food pantries, other key referrals # of relatives in their map of support (closeness & strength of relationship) # and type of training programs in which family members engaged with help of program staff Economic stability Family economics Data Entry &Analysis • Socio-demographic profile of families served by the program in the last fiscal year; demographics of the sample • Transcriptions/translations; computer assisted qualitative analysis • Thematic analysis, development of categories • Participatory involving the evaluation team and F.O.R. Families home visitors Social, Community, Family Support Establishment of support networks Outcomes Selected References: Alkin, MC ed. Evaluation Roots: Tracing Theorists’ Views and Influences. Thousand Oaks: SAGE, 2004. Fishman, DB. Postmodernism comes to evaluation IV; A review of Denzin and Lincoln’s Handbook of qualitative Research, 2nd Edition. Evaluation and Program Planning. 2003; 26:415-420 Hartman, A. Diagrammatic assessment of family relationships. Social Casework. 1978; 59, 465-476. Patton, M. Q. Qualitative Research and Evaluation Method, 3rd edition. Thousand Oaks: SAGE; 2001. Ray, RA, & Street, AF. Ecomapping: An innovative tool for nurses. Journal of Advance Nursing. 2005; 50:545-552. Shaw, IF. Qualitative Evaluation. Thousand Oaks: Sage, 1999. Spencer, L, Ritchie, J, Lewis, J, Dillon L. Quality in Qualitative Evaluation: A Framework for Assessing Research Evidence. National Centre for Social Research, UK. Available at: http://www.policyhub.gov.uk/docs/qqe_rep.pdf. Accessed March 6, 2006. Short-term Client knowledge & awareness of resources Skills acquired/employed (e.g., job search, self-advocacy skills) Change in attitudes and motivations Intermediate Behavior Change Evidence of decision-making and consistency following through with referrals and maintaining referrals Clients’ engagement Long-term Status Change in the five domains: housing; health; food security and nutrition; family economics; social, community, and family support Increased and sustainable overall household/family stability ABBREVIATIONS CBO = Community-Based Organizations CORI = Criminal Offender Record DTA = Department of Transitional Assistance DPH = Department of Public Health DSS = Department of Social Services HAP = Housing Assistance Provider HV = Home Visitor ICM = Intensive Case Manager Final Report and Recommendations Impact Families attain and maintain optimal well-being and housing stability Authors’ contact information: Zobeida E. Bonilla, bonillaz@marshall.edu, Dept. of Family & Community Health, Marshall University School of Medicine, P.O. Box 67, Huntington, WV 25706; Melissa Marlowe, melissa.marlowe@state.ma.us, Division of Perinatal, Early Childhood, and Special Health Needs, Massachusetts Department of Public Health, 250 Washington Street, 4th Floor, Boston, MA 02108 ; Karin Downs, karin.downs@state.ma.us, Division of Perinatal, Early Childhood, and Special Health Needs, Massachusetts Department of Public Health, 250 Washington Street, 4th Floor, Boston, MA 02108 Thank You We wish to thank staff from the Massachusetts Department of Public Health and the Housing and Homeless Unit of the Department of Transitional Assistance who assisted with this project.  We especially thank the families, F.O.R. Families home visitors, and key informants from various agencies who shared with us their time and experiences with the program.