Eliminating disparities: The challenge of using evaluation to improve and document a community change approach

1. Eliminating disparities: The challenge of using evaluation to improve and document a community change approach Judith Hager Belfiori, MPH, MA1 Jennifer Rienks PhD1 Linda Mack Burch, MPH1 Geraldine Oliva, MD, MPH1 Virginia Smyly, MPH, CHES 2 Sandra Johnson, BS 2 • University of California, San Francisco, Family Health Outcomes Project, • San Francisco Department of Public Health Community Health Education Section SevenPrinciples Project funded by Centers for Disease Control and Prevention (CDC)

2. Background Community Action Teams (CATS) are an intervention of the SevenPrinciples Project (7Ps) in San Francisco. 7Ps is funded by the Centers for Disease Control and Prevention (CDC) Reach 2010 Initiative. The San Francisco Department of Public Health is the lead agency working in partnership with Community Based Organizations (CBOs) and the UCSF Family Health Outcomes Project (FHOP). 7Ps goal is to eliminate the disparity between African American and Caucasian infant mortality rates in San Francisco and to improve the health of African American infants, their families and communities by increasing awareness of problems and needs, increasing community protective factors, and building capacity for change.

3. Community Action Teams (CATS) • The 7Ps project asserts thatpeople must work together and take responsibility to make positive changes happen in the community.CATS are developed to unite community members and strengthen their capacity to achieve desired changes. • 7Ps funds CBOs to form CATS in 4 African American neighborhoods to promote collective action to address community identified root causes of racial and ethnic health disparities, e.g., access to and preparation of nutritious foods. • CATS recruit and train members, identify a local health- related issue they will work on, conduct an assessment, analyze the assessment data, select and implement an action, and evaluate and maintain the action.

4. Community Action Teams (CATS) • CATS use a Community Action Model (CAM) approach to community organizing, based on the work of Paulo Freire, John McKnight and others, emphasizing the assets of the community and its capacity to change. • 7Ps provides CAM trainings and technical assistance to the CATS. The CATS use their training and energy to make a selected change in their community. • The anticipated outcomeis the successful completion of the CAT team’s chosen community action, e.g., a community garden. • Anticipated outcomes for CAT members are increased sense of community unity, sense of self efficacy and leadership skills, and problem identification and solution development skills.

6. Evaluation of CATS • The Family Health Outcomes Project (FHOP) of the University of San Francisco (UCSF) is a Project Partner and the evaluatorof the project’s interventions • The evaluation provides formative, monitoring and outcome data for the purposeof • improving how the CATS function • assessing the effectiveness of the CATS • documenting, for possible replication, the process and results of this approach and lessons learned

7. Evaluation of CATS • Using the CDC Logic Model framework, quantitative and qualitative evaluation methodsinclude: • outcome measures (pre and post questionnaires, documented completed actions) • process measures (staff and team coordinator reports, CAM training/meeting evaluations) • case study of selected CATS (pre and post interviews and evaluator observation of trainings, meetings, events) • input of participative evaluation workgroup

8. Results • Altogether 10 CATS with over 100 members. In Year 2004-2005, there were 4 CAT teams consisting of 4 - 17 members each. Activities and events often affected larger groups of 30 - 100 people. • Each CAT chose a Targeted Action, addressing one of the root causes: Nutrition, Violence, Undoing Racism, Community Unity. • Examples of completed CAT actions: • Members trained in healthy food preparation by a professional chef. Members can now offer community cooking classes. Produced a cookbook • Produced a neighborhood Resource Guide with inserts about financial topics and held community gatherings • Published Declaration of Prenatal Rights • Youth CAT members met with Board of Supervisors to discuss job and educational resources for the “big homies” (older African American youth, ages 18-24)

9. QualitativeEvaluation • Quantitative evaluation alone was inadequate because of small numbers and low follow-up rates (affecting ability to provide statistically significant findings). Also, responses raised questions needing further inquiry • Case studies (interviews, observation, reports) provided description of CAT activities and insight leading to identification of what is working and what to improve: • Example positive findings: • Most team members felt unity and connectedness with other members (insights were gained from interviews to explain why or why not) • Almost all had increased skills in community assessment, public speaking • Almost all would recommend family and neighbors join a CAT

10. QualitativeEvaluation • Example findings of weaknesses in CAT structure or function: • TA needed for Team Coordinators • Bureaucratic delays in providing stipends should be corrected • Teams need assistance in developing shared objectives • TA needs very different for CATs with teen members versus adult members Conclusion: Qualitative evaluation was rich in information used to help improve CATS in the next project year, provide insights to quantitative data and to share with and engage members of staff and the participative evaluation workgroup.

11. Challenges of Evaluating this Innovative Community-based Intervention • Interventions are designed by CAT members • Objectives not quantifiable • Tension between member driven outcomes and need to document process and outcomes • Teams are fluid. Not always able to obtain matched pre and post surveys or interviews • Quantitative data did not capture the experience of the teams • Case studies are time intensive and require special evaluator skills: obtaining data, insight analysis • Allocating adequate time to build relations with community members and project staff

12. Recommendations to Achieve “Meaningful” Evaluation of Community-Based Programs • Collect qualitative data to augment quantitative data • Evaluators should assist community groups to determine their objectives • Test evaluation instruments and methods, get and use feedback, revise instruments • If intervention changes, reassess usefulness / appropriateness of instrument(s) • Do not attempt to collect too much data: overwhelming to users, won’t have time to analyze • Present findings to the planners, staff and participative workgroup (recommended) to increase chances of implementing program improvements; build relationships, obtain additional insights

13. Contact Us Family Health Outcomes Project University of California, San Francisco 3333 California Street, Suite 365 San Francisco, CA 94118 Telephone: (415) 476-5283 Email: fhop@itsa.ucsf.edu Website: http://www.ucsf.edu/fhop