1 / 30

Strengthening home visitation through large-scale program evaluation

Strengthening home visitation through large-scale program evaluation. PA HOME VISITATION SUMMIT September 29, 2014. Overview. Large -scale program evaluation concepts Evaluation objectives Evaluation design Discussion. Large-Scale Evaluation Concepts. Dissemination. Efficacy

yeo-clayton
Download Presentation

Strengthening home visitation through large-scale program evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Strengthening home visitation through large-scale program evaluation PA HOME VISITATION SUMMIT September 29, 2014

  2. Overview Large-scale program evaluation concepts Evaluation objectives Evaluation design Discussion

  3. Large-Scale EvaluationConcepts

  4. Dissemination Efficacy Experimental design in highly controlled setting Effectiveness “Real-world” evaluation after implementation

  5. What can we learn from large-scale evaluation? Has the program maintained effectiveness? Are there site-level differences in effectiveness? How is the program influenced by community norms? Does the program affect community health indicators? What is the relationship of the program to other MCH & social services in the community?

  6. What can large-scale program evaluation strengthen? • Quality Improvement • Has the program maintained effectiveness? • Are there site-level differences in effectiveness? • Benchmarks • How is the program influenced by community norms? • Target audience • Does the program affect community health indicators? • Health system infrastructure • What is the relationship of other MCH & social services in the community?

  7. What is needed for success in evaluation?

  8. PA MIECHV Evaluation Objectives

  9. Collaborators • OCDEL • HRSA • PolicyLab • NFP, EHS, PAT, EHS local home visitation sites

  10. Evaluation Objectives • Understand the effect of the PA MIECHV expansion of NFP, EHS, PAT and HFA on maternal and child indicators • Describe any differential program effects for priority families (dual-language and children with special health care needs) • Explore the influence of site context and professional development activities on programs Evaluation will consider pre-MIECHV and post-MIECHV periods: 2006-2013

  11. PA MIECHV Evaluation Design

  12. Evaluation Design: Mixed Methods Quantitative Assessment of outcomes Client enrollment data Administrative data Qualitative Assessment of program characteristics, operations, and context Site observation Site interviews Site survey

  13. Data Sources • All Programs • Site-level enrollee data • Organizational Culture Survey • Selected Programs • Site observations • Site interviews with administrators, supervisors, home visitors, and clients • State • Vital statistics files • Medicaid files • Welfare eligibility files

  14. PA MIECHV Sites EHS HFA NFP PAT

  15. Quantitative Component: The Details • 2006-2013 • Outcomes: • Health behaviors: prenatal smoking cessation and prenatal care receipt • Birth outcomes: gestational age and birth weight • Service enrollment: Medicaid, TANF, foodstamps • Childhood injury • Child maltreatment • Matched comparison group design • Propensity score matching • Data sources: • Program data • Birth certificates • Welfare eligibility files • Medicaid files

  16. Quantitative Component: Update • 10,000 clients across NFP, EHS, PAT, HFA from 2006-2012 • To date: 8,044 clients matched to 30,263 comparison women • Approx. 1,300 clients per year • Client Demographics: • Mean age at birth of child: 20.8 years • 87% unmarried • 25% black, 20% Hispanic, 14% Asian/Other • 42% less than high school education • 54% rural communities

  17. Site Survey Number of administrators, home visitors, and other staff Case load New employees hired in past year Services provided Client referral agencies Client retention Agency location

  18. Site Survey Results

  19. Qualitative Component: The Details • Site selection • Benchmark data, geography, other site characteristics • 11 selected sites across PA • On-site observations • Location, setup, interactions • Interviews • Administrators • Home visitors • Clients

  20. Qualitative Component: Update

  21. Organizational Culture Survey Assesses 6 dimensions of an organization’s culture Rate organization on a series of questions Add the results to find your organizations “Now” and “Preferred” culture

  22. Activity Break!

  23. Organizational Culture

  24. Discussion

  25. Discussion Prompts How do you think the culture of your agency affects program outcomes for clients? How would it be different if your agency had a different culture?

  26. Preliminary Qualitative Findings

  27. Preliminary Qualitative Findings

  28. Discussion Prompts What are the benefits of a large-scale mixed methods evaluation for local program stakeholders? What are the limitations of this type of evaluation for local program stakeholders?

  29. PolicyLab www.research.chop.edu/policylab

More Related