The Linkages Evaluation Coaching Project

1. The Linkages Evaluation Coaching Project Amy D’Andrade San Jose State University School of Social Work May 19, 2015

2. Presentation Outline • Introduction • Evaluation basics • Our process • Results • Implementation • Process investigation • Case Studies • [Outcomes] and Dissemination • What was it like? • Concluding thoughts

3. Introduction

4. Steps in Evaluation

5. What is a Logic Model? “.. a simple diagram that communicates the rationale for why a program will work.” - Linkages Toolkit

6. PROGRAM ACTIVITIES LONG TERM OUTCOME

7. INPUTS OUTPUTS OUTCOMES Participation/ Short Medium Long-term Activities Program resources What we invest What we do Who we reach What results we obtain

8. INPUTS ACTIVITIES OUTCOMES Development of coordinated case plan Services offered not duplicative Linkages Team DPSS staff CWS staff Service providers Families CFPIC Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists DPSS staff attends TDMs Families more likely to receive needed supports Regular communication between DPSS and CWS workers Services offered provide for more family needs Training for CWS staff on Linkages All eligible families receive Linkages services Identification of shared families [To Linkages activities 1,2,3]

9. Two Approaches to Evaluation Outcome Evaluation: Assess the impacts or outcomes of programs

10. INPUTS ACTIVITIES OUTCOMES Development of coordinated case plan Services offered not duplicative Linkages Team DPSS staff CWS staff Service providers Families CFPIC Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists DPSS staff attends TDMs Families more likely to receive needed supports Regular communication between DPSS and CWS workers Services offered provide for more family needs Training for CWS staff on Linkages All eligible families receive Linkages services Identification of shared families [To Linkages activities 1,2,3]

11. Two Approaches to Evaluation Process Evaluation: Assess the delivery of programs

12. INPUTS ACTIVITIES OUTCOMES Development of coordinated case plan Services offered not duplicative Linkages Team DPSS staff CWS staff Service providers Families CFPIC Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists DPSS staff attends TDMs Families more likely to receive needed supports Regular communication between DPSS and CWS workers Services offered provide for more family needs Training for CWS staff on Linkages All eligible families receive Linkages services Identification of shared families [To Linkages activities 1,2,3]

13. Evaluation Components • Implementation Tracking: Be able to track what proportion of Linkages cases receive primary Linkages activities, and how that changes over time • Process Investigation: Understand why CWS/CMS communication falters or doesn’t occur • Case Studies: Illuminate why and how Linkages activities, when received, make a difference for families • Outcomes Description: Identify relevant child welfare and W2W outcomes of cases receiving Linkages • Dissemination: Share results and lessons learned with stakeholders

14. Evaluation Coaching Support Intensive coaching with 3 focus counties: Madera, Los Angeles, Tulare • Initial in-person visit • On-line or phone coaching sessions • On-going email conversations and phone calls • Concluding site visits and presentation of results/report/recommendations

15. So far with Coaching counties: • Identify program activities • Brainstorm areas of concern, interest, and related questions • Prioritize and select evaluation questions • Operationalize service outputs • Identify data sources • Collect data – in progress • Analyze data – in progress • Report results – in progress

16. Results

17. Implementation Tracking: Be able to track what proportion of Linkages cases receive primary Linkages activities, and how that changes over time.

18. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Regular communication (W2W and CW SWs) Services offered provide for more family needs Monthly Linkages committee meeting Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

19. LA Linkages – FP Protocol • Implementation Concern: Are eligible cases being identified? Data collection on 8 of 14 sites: • 30 FP cases randomly selected from each site • Checked CW data forms for eligibility code used to refer to Linkages • Checked eligibility for W2W/CalWORKS per DPSS system

20. % Correctly Identified: Of those DPSSdetermined to be eligible, what % were identified as eligible by CW? 38% 44% 50% 55% 67% 70% 90% 0% 100% 54% Site 1 Site 6

21. % Correctly Identified: Of those DPSSdetermined to be eligible, what % were identified as eligible by CW? 38% 44% 50% 55% 67% 70% 90% 0% 100% 54% Site 1 Site 6 What steps might the county takebased on this data?

22. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Regular communication (W2W and CW SWs) Services offered provide for more family needs Monthly Linkages committee meeting Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

23. Implementation Tracking Tulare and Madera

24. Percentage of eligible active cases receiving Linkages activities - May

25. What might the county do based on this data?

26. Percentage of cases receiving all Linkages activities over time

27. Process Investigation: Why does CWS/CMS communication falter or not occur?

28. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Regular communication (W2W and CW SWs) Services offered provide for more family needs Monthly Linkages committee meeting Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

29. Identified all workers on cases for which Linkages activities had not occurredCreated and sent email survey to these workers

30. When a child welfare case worker receives a new case that is Linkages-eligible, the worker is to reach out and connect with the Welfare-to-Work case manager - invite them to a staffing, or just establish communication. This doesn't seem to have happened on a recent case of yours.

31. Worker 1: Worked for the agency over 5 years Do you remember why it didn't happen? What got in the way for you? “With changes occurring, processes get lost.” What hinders you or gets in the way of your making this connection? “It's difficult to find time when both parties can meet and discuss the case.” What might increase the likelihood that you would take this action? “Establish a process that we can all work with.” Is there anything else you would like to say about this issue? “Having us meet to discuss the case is important and needed.”

32. Worker 2: Worked for the agency 1-5 years Do you remember why it didn't happen? What got in the way for you? “I don't see the need/value of taking that action.” What hinders you or gets in the way of your making this connection? “There are many service providers that need to be contacted and referrals to be completed and it would be helpful if the WTW case manager were to reach out to the SW instead of the other way around.” What might increase the chance that you would take this action? “If there was a more effective way of communicating that there is a link.” Is there anything else you would like to say about this issue? [no response]

33. Worker 3: Worked for the agency less than 1 year Do you remember why it didn't happen? What got in the way for you? “Not sure what you are referring to.” What hinders you or gets in the way of your making this connection? “The Linkages worker does not come to the meetings.” What might increase the likelihood that you would take this action? “?” Is there anything else you would like to say about this issue? “If I knew what you were referring to, I could answer.”

34. What might the county do based on this data?

35. Case Studies: Understand and describe why and how Linkages activities, when received, make a difference for families

36. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Regular communication (W2W and CW SWs) Services offered provide for more family needs Monthly Linkages committee meeting Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

37. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Single mother with 4 children all under the age of 5. Father was in prison. Mother had serious substance abuse problem. Had not worked in many years (if at all). Regular communication (W2W and CW SWs) Services offered provide for more family needs Monthly Linkages committee meeting Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

38. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Two parent family, five children removed when both parents were taken to jail. Mother was pregnant. Once released from jail, the family had great difficulty finding housing due to parents’ low income (father undocumented), multiple prior evictions, and criminal records. Regular communication (W2W and CW SWs) Services offered provide for more family needs Monthly Linkages committee meeting Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

39. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions “One thing for me, anytime that we had a staffing, I always wanted to have the W2W worker involved…because…(he might) be aware of something that I might not be aware of.” Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Regular communication (W2W and CW SWs) Services offered provide for more family needs Monthly Linkages committee meeting Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

40. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Regular communication (W2W and CW SWs) Services offered provide for more family needs “I think Cesar got quite a few emails from me, wanting to find out what’s available to (the mom) and whether it’s an option or not…The fact that we’re in the same building sitting close by each other makes it very easy to ask questions and be able to plan…” Monthly Linkages committee meeting Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

41. INPUTS ACTIVITIES OUTCOMES Case Study 1 W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Regular communication (W2W and CW SWs) “We kind of help back each other up in a way. If I would forget something with the family and Cesar went to see them the next day, he would send me an email – ‘Hey Maria, I went to go visit them and you forgot…’ And I think we were just better able to help the family because we had each other.” [CW SW] Services offered provide for more family needs Monthly Linkages committee meeting Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

42. INPUTS ACTIVITIES OUTCOMES Case Study 1 W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Regular communication (W2W and CW SWs) “…Once she got her children back she wanted to start working – so we looked to see if she could receive childcare through Linkages, through W2W.” [CW SW] “She’s now working at [fast food] about 30 hours a week. Critical to keep that job is child care. So we secured child care with a relative, and we can pay that relative, and that’s what we’re doing.” [W2W SW] Services offered provide for more family needs Monthly Linkages committee meeting Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

43. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Regular communication (W2W and CW SWs) Services offered provide for more family needs Monthly Linkages committee meeting “I think the family was encouraged to see all the people involved in dealing with the different issues.” [W2W SW] “I think it kind of gave them peace of mind that not only I was working on the case, but so was Cesar. That just made them feel more supported from our department.” [CW SW] Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

44. INPUTS ACTIVITIES OUTCOMES Case Study 1 W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Regular communication (W2W and CW SWs) Services offered provide for more family needs Monthly Linkages committee meeting “It’s still in progress. We’re still assisting with child care because she has an open W2W case. She’s not generating enough income to get off aid, but of course she’s in a much better position now because of …the earnings that she generates. She’s able to take care of her needs better. And at the end of this year, she’s be able to claim Earned Income Tax Credit. That’s probably something she hasn’t seen in a while - or never.” Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

45. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Regular communication (W2W and CW SWs) Services offered provide for more family needs Monthly Linkages committee meeting “This past month was the first month that the family paid their own rent. The program was able to give them $800 a month, and that enabled them to save money to get a car…since W2W helped them with essential items…they were able to get other items on their own…a little table with some chairs, a little TV, and now a couch….they’ve felt empowered because now they’re doing it on their own…The work we completed together was far more than just a patch. We enabled them to get a little bit ahead of the game, and they are going to maintain momentum.” Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

46. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) “This particular mom, she was extremely committed and dedicated to being free from child welfare services. She was constantly calling or texting me or Cesar. She kept us as accountable as we kept her, if not more.” Families more likely to receive needed supports Regular communication (W2W and CW SWs) Services offered provide for more family needs Monthly Linkages committee meeting Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

47. INPUTS ACTIVITIES OUTCOMES W2W staff – ETs, EWs CW SWs Supervisors Exec Team Community/Employers Families, Parents, Youth CFPIC Court Child Abuse Council Agencies Fresno Academy Development of coordinated case plan Services offered not duplicative Increased case closure CWS / DPSS Decreased re-entry CWS / renewal DPSS Families better able to complete case plan requirements Case plans and casework draw on knowledge and resources of both divisions “The W2W worker did a lot with this family. As soon as we got them qualified, he was able to secure the housing….Because without him, no one would have rented to this family… he met with the landlord and was able to explain how the program worked – and that accelerated everything.” Greater service availability/ fewer wait-lists Tandem monthly meetings (W2W and CW SWs) Families more likely to receive needed supports Regular communication (W2W and CW SWs) Services offered provide for more family needs Monthly Linkages committee meeting Increased staff knowledge on value of collaboration, Linkages program Training for CWS staff on Linkages More families receive Linkages services [To Linkages activities 1-4] Identification of shared families

48. What might the county do based on this data?

49. Outcomes Description: Identify relevant child welfare and W2W outcomes of cases receiving Linkages

50. Dissemination • All-county Webinars • Convening Presentation • Linkages County Team Meeting • County Stakeholder Presentation • Other Ideas?