Data Management: Let's Talk About Using Data to Evaluate November 6 th 3pm to 4:30pm

1. Data Management: Let's Talk About Using Data to EvaluateNovember 6th 3pm to 4:30pm Linkages Shared Learning Webinar #8 Stuart Oppenheim, CFPIC, Executive Director Danna Fabella, Linkages Project Director

2. Peer County Advisors • John Dufresne, MSW, Program Manager, Fresno • Johnny Alaniz, Social Services Program Manager, Fresno • Sarah Whittington, MSW, SW Practitioner, Fresno on behalf of Madera

3. Agenda • Overview of Evaluation/Research: Stuart Oppenheim • What’s on the Toolkit? Danna Fabella • What are some counties doing? • John Dufresne & Johnny Alaniz: Fresno • Sarah Whittington: Madera • Q & A • Discussion • Next Steps

4. Research/Evaluation • Research is directed toward increasing knowledge, the primary aim being more knowledge or understanding of a particular group, problem or issue. The strict definition of scientific research is to perform a methodical study in order to prove a hypothesis or answer a specific question.

5. Evaluation is concerned with answering questions about issues that arise in everyday practice. A distinguishing characteristic of evaluation is that, unlike traditional forms of academic research, evaluation is grounded in the everyday realities of organizations. Evaluations can be conducted of programs, processes, products, systems, organizations, personnel, and policies.

6. Evaluation therefore answers questions like: ♦ Does it work? ♦ Does it do what we want it to? ♦ How well does it work?

7. Why Evaluate? • Not required by the Statewide Project • To let you know if and how Linkages is being implemented • To understand if you are getting the outcomes that you want • To let Leadership, staff, and stakeholders know how it improves outcomes for families

8. Approaching Evaluation • Know what you mean by Linkages. • What is success? • Create a Logic Model • Inputs (training, , etc) • Outputs (CCP, Family Engagement, TDM, etc) • Outcomes (children remain at home, sanction cured, etc.) • How will you measure it? • What data will you use?

9. How will it be collected? • Automated/case flags • Case Review • Surveys • Who will analyze it? • How will you use it? • Adjust program • Build support

10. What we can find on the Toolkit:http://www.cfpic.org/toolkit/

11. Presenters: John Dufresne Program Manager Johnny Alaniz – Social Services Program Supervisor

17. FRESNO COUNTY LINKAGES DATA COLLECTION PARAMETERS Who and/or where data is located Data that is collected Sample Client data collected

19. Eligibility Worker TDM Datasheet Stats

23. EW TDM Datasheet - 2011 Year in Review 2011 Totals Monthly Average TDM Datasheets87773 Adults1,11192 Adults w/CalWORKs Case61050 Adults no CalWORKs Case 501 41 Adults Active CalWORKs28323 Adults Not Active CalWORKs 327 27 Sanction705 Drug Felon 27 2 SSI 34 2 Time Out 1149 Undocumented 42 3 Ineligible 40 3

24. 2011 TotalMonthly Average WTW Active Adults15913 WTW Non-Participating Adults 124 10 WTW Exempt847 WTW Never Enrolled292 Teen Parent 8 Non Needy Payee 75 6 No Public Assistance 196 16 Active Cal Fresh 50 4 Active Medi-Cal 78 6 Active Combo-CF&MC 142 11 Pending Applications 18 1 AAP/ GR 17 1 #2302 C/WORKs Referrals 236 19 Return To Parent 176 14 Placed w/Relative 60 5 Processed Return to Parent 102 8 Processed Placed w/Relative 34 2 Applications Pending 14 1 No Application Taken 77 6 Courtesy Applications 5 Denied 13 1

25. Lessons Learned • The amount of time that the Linkages data collection takes to compile. • Lack of communication at times between: • JS/EW and SW • JS & OA • Linkages clients going to Intake rather than being referred to Linkages Intake EW • Current Excel collection method is not user friendly and is difficult to access reports in a timely manner.

26. Future Plans • Expand Linkages to Emergency Housing population • Expand Linkages to include Tribal TANF • Development of a comprehensive Database allowing to access reports at any given point in time.

27. Linkages: A Collaborative Effort to Serve the Needs of Clients: An Analysis of the Implementation of the Linkages Program in Madera County, Department of Social Services Sarah Whittington MSW Graduate Student California State University, Fresno In Collaboration with Madera County Department of Social Services

28. Madera County at a GlanceBureau of Labor Statistics & madera-county.com • LOCATION: Madera County is located in the exact center of California, in the heart of the Central Valley and the Central Sierras. Fresno County borders on the south, Mariposa and Merced counties on the north, and Mono County to the east. • GEOGRAPHIC AREA: 1,374,160 acres; 2,147 square miles, stretching from the rich San Joaquin Valley to the crest of the Sierra Nevada, the highest mountains in the contiguous United States. Bordered on the north by the Chowchilla River and on the south by the San Joaquin River, the County includes some of the richest agricultural land in the nation. • CITIES: Chowchilla and Madera. Unincorporated communities: Ahwahnee, Bass Lake, Berenda, Coarsegold, Fairmead, Madera Ranchos, North Fork, Oakhurst, O'Neals, Raymond, and Rolling Hills. • ECONOMY & EMPLOYMENT : Agriculture is the largest industry in the county, accounting for 29.9% of the employment. Government, another significant sector, accounts for 19.5% and services makes up 16.8% of the total. The county’s leading commodities include: almonds, grapes, milk, and pistachios. • UNEMPLOYMENT: The Bureau of Labor Statistics has the unemployment rate in Madera County at 11.6 % in September 2012, down from a revised 13.0% percent in August 2012.

29. Madera County At A Glance • The 2011 Madera County Population as estimated by the US Census Bureau equals 152,925. • Persons under the age of 18 equal 28.2% of the population. (CA = 24.6%) • Hispanic/Latino = 54.5% of the Madera Co. population. (CA = 38.1%) • White = 37.5% of the Madera Co. population. (CA = 39.7%) • American Indian = 4.5% of the Madera Co. population. (CA = 1.7%) • Black = 4.4% of the Madera Co. population. (CA = 6.6%) • Asian = 2.2% of the Madera Co. population. (CA = 13.6%) • Percentage of high school graduates in Madera Co. = 67.9% (CA = 80.7%) • Percentage of Madera County population with a Bachelors degree or higher = 13.5% (CA = 30.1%) • Median household income, 2006-2010 = $46,039 (CA = $60,883) • Percentage of persons in Madera County living below the poverty level, 2006-2010 = 19.3% (CA = 13.7%)

30. Madera County Department of Social Services:2011 Madera County DSS Client Profile • July 2011 (Client Profile) Program and Number of Clients CALWORKS = 8,782 FOOD STAMPS (NON-CALWORKS) = 17,559 MEDI-CAL (NON-CALWORKS) = 33,165 CMSP = 2,437 GENERAL ASSISTANCE = 114 ADULT PROTECTIVE SERVICES = 106 IN-HOME SUPPORTIVE SERVICES =1,426 CHILD WELFARE SERVICES = 330 PUBLIC GUARDIAN = 152 Total Number of Individuals Served by our Department = 64,071

31. Linkages: A Collaborative Effort to Serve the Needs of Clients: An Analysis of the Implementation of the Linkages Program in Madera County, Department of Social Services Research Completed By California State University, Department of Social Work Education Graduate Student: Sarah Whittington in Partnership with Madera County DSS

32. Background and Impetus • The current Linkages Program/Policy that exists in Madera County (pilot Linkages project in 2003, formal Linkages policy approved 2011) is the result of many different progressions of research and a combination of various legislative efforts. • Poverty and Child Maltreatment • Between 2000 and 2009, the number of poor children (children living in families earning 100% or less of the federal poverty level) increased by 33% (Chau et al., 2010). • The correlation between poverty and child maltreatment is a phenomena that has been widely studied and has come to be recognized as one the strongest predictors of child maltreatment. • The Fourth National Study of Child Abuse and Neglect (NIS-4) explains that children from low socioeconomic status households experienced some form of maltreatment at a rate of more than 5 times the rate for children from higher socioeconomic statuses (Sedlak, Mettenburg, Basenaer, Petta, McPherson, Greene, & Li, 2010). • It is of no great surprise that families experiencing more economic hardships, such as unemployment, homelessness, food insecurity and loss of utility services are at greater risk of coming to the attention of Child Welfare Services . • It has been shown that within the state of California, 60% of the children involved in Child Welfare Services, have a history of Aid to Dependent Families (AFDC) and/or Temporary Aid to Needy Families (TANF) use (Garrison, 2011). • Recent federal and state efforts and legislation have moved the Child Welfare System into a more empirical direction, in which specific outcomes based on the effectiveness of the agency in serving clients (safety, permanence and wellbeing) is assessed and reported to the state and federal levels (The 1997 Adoptions and Safe Families Act (ASFA) & The 2011 Child Welfare Systems Outcomes and Accountability Act, also known as AB 636). • AB 429 mandates continued WTW services for families dually involved in CWS and CalWORKs (CA TANF).

33. Hypotheses/ Research Question • Research Question • Is the Madera County Linkages Program/Policy, which includes interagency collaboration between CalWORKs and Child Welfare Services to holistically serve dual clients with a plethora of service needs, being fully implemented by the parties involved (CalWORKS workers, social workers, supervisors, agency administrators) with current Linkages cases? • Hypothesis One • If the written Linkages policy in Madera County Department of Social Services is (or is not) being fully implemented by the designated employees, the quantitative data (secondary data analysis) will show the presence (or absence) of the elements of the policy in the documented work being done for clients. • Hypothesis Two • If the written Linkages policy in Madera County Department of Social Services is (or is not) being fully implemented by the designated employees, the qualitative data (worker surveys) will show the presence (or absence) of the elements of the policy in the work they describe being done for clients.

34. Research Design/Rationale • Pre experimental, focused on one cohort in one agency, without control group. • Study is exploratory and descriptive in nature, due to sparse research on this topic. • Instrumentation • The Linkages Case Audit Survey Using Secondary Data Analysis is a quantitative case audit tool based directly on the Madera County Linkages Program Policy. (Approved by MCDSS leadership) • The Linkages Child Welfare Services Social Worker Survey and the Linkages CalWORKs Worker Survey were developed to collect qualitative data. (pilot studied) • Procedures • The data used for secondary analysis as outlined in the Linkages Case Audit Survey Using Secondary Data Analysis was gathered from CWS/CMS and CIV Databases and Child Welfare Services and CalWORKs hard cases. • The Linkages Child Welfare Services Social Worker Survey and the Linkages CalWORKs Worker Survey, the letters of informed consent and posted envelopes were distributed to the Madera County DSS worker participants via Madera County interdepartmental mail and mailed directly back to the student researcher.

35. Sample-Study Participants • Secondary Data Analysis • A convenience sample of the 15 Linkages cases open between 9/12/2011 and 10/10/2011 were analyzed. • Staff Surveys • 50% response rate (11 of 22 surveyed responded) • Participants included: 6 Child Welfare staff, 4 Welfare to Work Staff and 1 Eligibility Worker (Total of 5 CalWORKs staff).

36. Data Analysis • Secondary Data Analysis • The quantitative data gathered from the Linkages Case Audit Survey Using Secondary Data Analysis was analyzed using the Statistical Package for the Social Services (SPSS) to provide descriptive statistics. • Staff Surveys • Content analysis, specifically, thematic coding was used to assess the Child Welfare Services Social Worker Survey and the Linkages CalWORKs Worker Survey responses. • Elements of grounded theory data analysis

37. ResultsSummary of Quantitative Data Analysis • 8 of 15 cases had no documentation of the ER social workers consultation or tandem visit with WTW case manager at the onset of the Linkages referral. • Of the 10 cases audited that did result in a protective hold on the children, only 1 of the CWS cases had documentation of the ER worker inviting the WTW case manager or the EW worker to the detention staffing. • 14 of the 15 cases analyzed (whether opening to FM or FR), the CWS social worker did arrange for a Linkages Staffing with the family, WTW case manager and EW. However, all but three of these staffing were not held within the designated 7 day period from opening the case. • A MAD 454 Coordinated Case Plan was documented in 13 of the 15 total cases reviewed. • 8 of 15 had documentation of the CWS social workers notification to the parents for their attendance at the Linkages Staffing. • 12 of 15 did not have the Linkages staffing appropriately documented as Family Engagement Efforts (FEE) in the CWS/CMS database. • 7 of the 9 cases that were maintained in FR, all but 2 cases continued to receive CalWORKs services under AB 429 and had the appropriate FR aid code updated after 30 days.

38. Sample of Graphs and Tables Generated from SPSS

39. Sample of Graphs and Tables Generated from SPSS

40. Sample of Graphs and Tables Generated from SPSS

41. Sample of Graphs and Tables Generated from SPSS

42. Sample of Graphs and TablesGenerated from SPSS

45. Sample of Graphs and Tables Generated from SPSS

46. Sample of Graphs and Tables Generated from SPSS

47. ResultsSummary of Qualitative Data Analysis • All but one of the workers of both CWS and CalWORKs indicated that they had attended at least one Linkages training and many had attended several within the last few years. • All but two respondents from both CWS and CalWORKs indicated that they did have collateral contact with their counterparts when managing a Linkages case. However, the frequency and purpose of these collaterals was not consistently practiced or understood. • 7 of the 11 respondents did state that they complete the MAD 454 Coordinated Case Plans and Detention Staffing C-IV Datasheets for all of their Linkages cases. • The majority of the CWS staff respondents reporting that they indeed had failed to document Linkages activities that they had completed. • the majority of both the CalWORKs and CWS staff agree that the coordinated case planning has made it easier for Linkages clients to comply with both the CalWORKs and Child Welfare case plans. • there was mixed responses from both CWS and CalWORKs staff regarding what they saw as the impact of the Linkages Program on poverty and child maltreatment in the Linkages cases. • Staff members from both CWS and CalWORKs indicated that more ongoing training, flexibility of funding, engagement of parents and increased communication across programs is needed.

48. Discussion • Both the Linkages Case Audit Survey Using Secondary Data Analysis and The Linkages Child Welfare Services Social Worker Survey and the Linkages CalWORKs Worker Survey did provide quantitative and qualitative evidence of whether or not the Madera County Linkages Policy/Program is being fully implemented by the designated employees. In turn, the conclusions about these two hypotheses allowed the student researcher to draw conclusions about the overall research question. • The analysis of the data suggests that the core elements of the Linkages Policy/Program, as written in Madera County Department of Social Services, are being carried out, such as the joint Linkages staffings and the coordinated case planning. However, other parts of the written policy, such as collateral contacts, tandem visits, specific Linkages documentation in the data systems, timely staffing with all required parties present, parental notification and communication regarding the detention staffings are not occurring on a consistent basis and or are not being documented properly. • Staff also seemed to indicate that they understand the impact of Linkages on individual clients and cases but do not seem to recognize the philosophy behind the creation of Linkages, which includes the recognition of one of the prevailing root issues of child maltreatment, poverty, indicating more training and coaching may be needed. • Overall, it appears that the core elements of the policy are being carried out; yet, consistent documentation, clear direction and purpose seem to be lacking. Therefore, although great progress has been made in terms of the implementation of the Linkages Program/Policy in Madera County, the quantitative and qualitative data reveals that the Linkages Program/Policy is not being fully implemented at this time.

49. Implications for the Future • The research findings may lead to more valid outcome studies of the Madera County Linkages Program in the future, since this study informs the degree of fidelity to the policy itself. • Also, further study may be done on the financial imperatives of inter-agency/inter-departmental collaboration and the cost savings to counties from reducing duplicate services and providing joint services through more flexible funding streams. • Through the course of this study the need for more consistent and more worker friendly documentation methods for linkages related activities became apparent. • Due to the increasingly important role of collaboration between child welfare agencies and other social service agencies for the provision of multiple and complex needs of the clients, this study helps to inform the departments and the academic community about the status of this collaborative effort between CalWORKs and CWS (Linkages) in Madera County in terms of implementing the Linkages policy and program. • This study also extend the literature on inter-agency and inter-departmental collaboration, a growing filed of study.

50. Follow Up: we’ll email with… • Copy of this PowerPoint • Updated Linkages Coordinators Directory Next Webinar • December 4th 9:00 a.m. to 10:30 a.m. Helping Sanctioned Linkages Families Succeed:   Best Practices From the Field