230 likes | 517 Views
Learning Objectives. Learn how empowerment evaluation, process evaluation and process use in evaluation design can make the evaluation a mutually beneficial exercise for researchers and agency staff.Learn creative ways to engage staff, collect data, maximize participation, and work toward program
E N D
1. Increasing Engagement in the Evaluation: Incorporating Empowerment Evaluation, Process Evaluation and Process Use in Design
Robin Rosell, LMSW-ACP,LMFT, Director, Tandem Collaboration, People’s Community Clinic
Peg Gavin, LCSW, Director of Educational Services, LifeWorks
Amy Pierce, MPH, Director, LifeWorks Teen Parent Services
Dr. Carol M. Lewis, Associate Director, Center for Social Work Research,
University of Texas at Austin School of Social Work
2. Learning Objectives Learn how empowerment evaluation, process evaluation and process use in evaluation design can make the evaluation a mutually beneficial exercise for researchers and agency staff.
Learn creative ways to engage staff, collect data, maximize participation, and work toward program improvement.
Learn how process evaluation can make use of feedback and short term findings for mid-course corrections and program improvements, helping ensure that goals are met and funder requirements are satisfied.
3. LifeWorks Adolescent Family Life Demonstration Project Five-year, federally funded community partnership in Austin, TX
Intensive case management for pregnant and parenting teens provided by LifeWorks Teen Parent Services and Tandem Collaboration of People’s Community Clinic
Evaluation led by Center for Social Work Research at UT Austin
4. TANDEM Teen Prenatal & Parenting Program Austin Child
Guidance Center
Mental Health Assessments
& Services, Individual, Couples, Family & Group Therapy
5. Tandem Teen Prenatal & ParentingProgram
6. 1. Promote the health and well-being of participating young parents and their children.
2. Reduce the risk of subsequent unplanned pregnancies. Primary Goals
7. Tandem Program Measures for Success Utilization of health care services by young parents and their children
Positive birth outcomes
Use of contraceptive method following birth
Rate and timing of subsequent pregnancies
Utilization of mental health services
Presence of positive parenting behaviors
Participation in education or other activities promoting self-sufficiency
Reduction in high risk behaviors (i.e. substance abuse)
Reduction in risk of child abuse
Involvement of young fathers and other family members in parenting children
8. Tandem Program Key Strengths
9. LifeWorks Teen Parent Services Services Provided
Individualized case management
School-based informational presentations
School-based support groups
Access to a comprehensive, coordinated service delivery system through LifeWorks
Information, referral and linkage to community resources
10. LifeWorks Teen Parent Services Primary Goals
Prevention of subsequent pregnancies
Clients continue with education in order to complete high school and/or GED program.
11. LifeWorks Teen Parent Services Key Strengths
12. Program Evaluation Challenges Engagement of program staff, skepticism regarding evaluation, staff turnover
Recruitment and retention of research participants
Vulnerable, highly transient program populations
Patchwork of funding streams and program partnerships, varied programmatic expectations, simultaneous evaluations
Data collection, competition for time, uniform program data
13. Empowerment Evaluation…
…the evaluation aims to strengthen the intervention by engaging staff in its implementation (Fetterman, 2005)
14. Effective process evaluation…
…makes use of feedback and short term findings for mid-course corrections and program improvements, helping ensure that goals are met and funder requirements are satisfied.
15. Evaluation Capacity Building… “The impact of the evaluation [that] comes from application of evaluation thinking and engaging in evaluation processes (in contrast to impacts that come from using specific findings).” (Patton, 1997)
The capacity to build and implement good evaluations.
16. Process use…
…ensures that the evaluation is a meaningful exercise for all parties,
and the time and expense invested in evaluation reap immediate benefits
and transcend the final report
(Patton, 2007).
17. Using the evaluation process can result in useful outcomes for both the intervention and the research…
18. Strategies to Balance Reality & Design
Directly engage case managers in recruitment and data collection
Monthly team meetings between program directors, staff and evaluators
Midcourse corrections via interviews and surveys with case managers, clients and families
19. Resulting Process Use Outcomes a dramatic increase in collaboration among the project’s five partnering agencies
the development of a unified Outcome Tracking Form that integrated client tracking data required by the evaluation and each agency
the strengthening of the intervention’s mental health component
20. Limitations and challenges. . . Biases associated with case managers collecting data from their clients
Obstacles to a true baseline related to time constraints, building rapport with clients and parental consent
Ethical concerns among program staff regarding randomization and some research recommendations
Delays associated with IRB requirements
Staff turnover
21. 20/20 Hindsight Expand proposal process to include case managers, individual agencies
Larger incentives for client participation from the start
More structure and incentives for case managers to recruit and enroll their clients
Realistic understanding of what is required to engage in an intensive research project
22. References Fetterman, D. “A Window into the Heart and Soul of Empowerment Evaluation: Looking Through the Lens of Empowerment Evaluation Principals.” In Ed. D. Fetterman and A. Wandersman (eds.), Empowerment Evaluation Principles in Practice. New York: The Gilford Press, 2005.
Patton, M. Q. “Process Use as a Usefulism.” In J. B. Cousins (ed.), Process use in Theory, Research and Practice. New directions for evaluation, no. 116, 99-112. San Francisco, Calif.: Wiley, 2007.
Patton, M. Q. Utilization-Focused Evaluation (3rd ed.). Thousand Oaks, Calif: Sage, 1997.
Weiss C. H. “Have We Learned Anything New About the Use of Evaluation?” American Journal of Evaluation, Mar 1998; 19: 21 - 33.
23. Contact Information Robin Rosell, Tandem Collaboration, People’s Community Clinic
Robinr@austinpcc.org
Peg Gavin, LifeWorks Educational Services
Peg.gavin@lifeworksweb.org
Amy Pierce, LifeWorks Teen Parent Services
Amy.pierce@lifeowkrsweb.org
Dr. Carol M. Lewis, UT Austin Center for Social Work Research
Carolmarie@mail.utexas.edu