The Children & Residential Experiences: The CARE program model theory and research

1. The Children & Residential Experiences: The CARE program model theory and research Charles Izzo (cvi2@cornell.edu) Martha Holden (mjh19@cornell.edu_ Elliott Smith (egs1@cornell.edu) Funded through The Duke Endowment & Cornell University

2. Goals of Workshop • Describe an evaluation that aims for both high scientific standards AND local utility • Discuss how we tried to: • a) produce generalizable knowledge, and • b) meet practical needs of agencies • Critically examine our approach and advance our thinking about these issues

3. Overview of Discussion • Provide context for evaluation • Objectives: Scientific and Practical • Program description • Describe Evaluation • Design decisions • Measurement • Implications for internal validity & utility • Data-based feedback to agencies

4. Some Scientific Objectives • Provide reliable evidence about the outcomes produced by CARE • Research design with credible counterfactual • Multi-modal assessment • Reflects the Theory of Change • Qualify for review by California Evidence-base Clearinghouse (CEBC)

5. Some Practical Objectives • Provide agencies with insight about • Youth, and the services they receive • Agency dynamics that affect service quality • Short and long-term effects of CARE training • Facilitate better CARE implementation by • Providing clues for implementation targets • Shaping the internal dialog • Motivating agencies with positive feedback

6. CARE: Children And Residential Experiences PROGRAM SUMMARY

7. Implementing CARE is about changing the entire operating system • The ‘training’ is focused on changing a mindset, not about simply adding new information or developing new technical skills. • The emphasis is on transforming the organization as a whole.

8. Characteristics of the CARE Approach • The agency is the locus of learning. The agency itself becomes the primary learning site. • The agency is the unit of learning, rather than the individual (or even the team). • The facilitation process involves much more than skills training or knowledge transmission

9. The Implementation Process The CARE consultants are engaged in a co-learning and co-creation process alongside the agency staff members; all participants are learners. CARE consultants work to realize the potential of adult learners and to align their mindsets with the needs and experiences of the children.

10. Interlocking “Nested” Elements in Translating the CARE Program Model into Practice Organizational Climate &Culture Staff Development Integrating CARE Model Change Facilitation Beliefs, Attitudes and Assumptions Six Core Principles Best Interests of Children

11. The Three Levels of the CARE Philosophy The first three circles, or levels, represent the foundation of the CARE program model: • The best interests of the child is essentially a commitment or value • The six principles of CARE provide compass points and a focus for thinking and decision-making. • Articulated beliefs, attitudes and assumptions are an additional level of understanding of the principles that direct actions in practice

12. Adaptive Thinking is Essential Putting CARE into practice requires the ability to move beyond technical thinking (“if x, then y”); it requires adaptive thinking (“what is going on here, and how can I be helpful?”) Ultimately, it is about mindset, and the need to think and act in an integrated manner with an appreciation of complexity.

13. Changing Mindsets Changing a mindset involves all aspects of one’s being – cognitive(how we think), affective(what motivates us), social (where we find reinforcement) and spiritual (what gives our life meaning).

14. thinking Cognitive/intellectual motivation reinforcement Social/affiliative MINDSET Affective/emotional meaning Moral/spiritual Staff/Adult Development Domains

15. Re- Commitment to CARE by agency Re- Re- Gaining confidence Embracing 6 principles Re- Re- Experiencing CARE effectiveness Understanding key concepts (beliefs, attitudes, assumptions) Re- Working through applications Integration of the CARE Philosophy

16. Creating the context for change Providing information Validating Sharing illustrative stories Questioning Paraphrasing Self-disclosure Communicating empathy Analyzing case examples Modeling Challenging Observing Affirming Probing Joining in the task of agency evolution and supporting changing mindsets Elements of the Change Facilitation Process

17. Description Of Evaluation

18. Selecting Focal Outcomes To Assess CARE process involves lots of depth and subtlety Our evaluation targets only selected outcomes that Map onto specific curriculum content Reflect key milestones in the CARE process (i.e., that reflect our Theory of Change)

19. Children And Residential Experiences: Measurement Youth Outcomes Staff Outcomes • Youth Perceptions About Staff • Youth survey pre-training, then annually Socio-emotional Well-being Strengths & Difficulties Questionnairepre-training, then annually CARE-Consistent Practices Staff survey pre-training, then semi-annually Intentions / Motivation to apply CARE Principles Staff survey: post-training only Intervention All Staff Training Technical Assistance Quarterly Database to track key program events Ongoing Knowledge of & Belief in CARE principles Staff survey: pre & post-training, then annually Organizational Context Organizational Social Context Survey Glisson & Hemmelgarn, 1998 pre-training, then annually

20. Quasi-Experimental Cohort Design CARE Now Agencies (N=7) CARE Intervention Interval12 months Socio- Emot Adjust (Pre) Socio- Emot Adjust (Post) Wait-List Agencies (N=8) CARE Intervention Interval12 months Interval with No CARE 12 months Socio- Emot Adjust (Post) Socio- Emot Adjust (Pre2) Socio- Emot Adjust (Pre1)

21. Mostly In North Carolina

22. Training Outcomes: • Short Term & Intermediate CARE Training >Knowledge >Beliefs >Curr Practice >Youth Percep >Knowledge >Beliefs >Curr Practice >Youth Perc >Knowledge >Beliefs >Intended Practice >Curr Practice Pre Post 6 mos 12 mos

23. Measurement Instruments

24. Key Features of CARE Assessments • Survey items map directly on to curricular material • Use of “priming” to improve response quality • Corresponding ratings by youth and staff • Matching scales and items across instruments • Validation of new instruments with established instruments • Matching across assessments while maintaining “anonymity”

25. Assessment of Beliefs & Attitudes Several scenarios with challenging situations, followed by possible responses, rated from “Poor” - - - - - - - - - - “Excellent” Ex: Lacey is an 11 year old who consistently seeks attention from you, asking silly questions, clinging to you, and following you around. Below are several ways to respond to a situation like this. > Spend time with her to help her feel more securely attached. > Give her less attention to avoid reinforcing her clingy behavior. > Try to learn about the reasons for her attention-seeking behavior. > Try to re-direct her to focus more on other youth instead of staff.

26. Beliefs & Attitudes- Subscales Listening & Understanding • Attention-Seeker: “Try to learn about the reasons for her attention-seeking behavior.” Inclusion • Chore resister: “He should not be allowed to participate in group activities.” (R) Investment / Engagement • Attention-seeker: “Spend time with her to help her feel more securely attached.”

27. Beliefs & Attitudes - Subscales Flexibility • Homework refuser: “Give him a choice about doing it now or before school.” Competence-Building • Won't eat in group: “See if he can tolerate eating in a different area with just you and another child he knows.”

28. Assessment of Current Practice 34 practices: “How often have you done this in the past month?” Respondents rate from Never - - - - - - - - - - - - - - - - - Several times/day If “Never,” then why not? No opportunity Not my job No Time Disagree with the practice

29. Assessment of Intended Practice Asks about the same 34 practices: “Now that you have finished CARE training, please indicate if you will use these practices . . . ” Less Often - - - - About the Same - - - - More Often

30. Youth Perceptions about Interactions with Staff Several scenarios followed by possible responses from staff. Youth indicate how often they acted that way from “Never” - - - - - - - - - - - - - - - - “Always” Ex: Think about times when you needed help with something that was difficult for you. Maybe you were nervous about a test / had problems with someone who lives here / problem with a family member / just needed staff to do something for you. > They tried to understand what I wanted. > They tried to be helpful. > If felt that they were happy to help. > They helped me learn how I could handle the situation better myself.

31. Trade-offs: Validity vs. Pragmatic Concerns • Instruments: • Established instruments with general constructs vs. Tailored instruments with program-specific constructs • Response choices • General (agree/disagree) vs. Specific (how often in past month) • Youth report on specific shifts vs. all staff • Minimizing perceived threat • Identified vs. Anonymous data: Implications for matching, statistical power, and candidness

32. Data-Based Feedback To Agencies

33. Useful Features of Data Reports • Discussion at both the construct-level (abstract) and the item-level (tangible) • Providing a reference point (other agencies, subgroups, early time points) • Showing variability, not just means.

34. Useful Features of Data Reports • Showing correspondence across measures • Illustrate with respondents’ own words • Situating each finding within the theory of change.

35. OSC Culture Profile 58.39 72nd percentile 56.64 73rd percentile 45.59 32nd percentile

36. Number of Knowledge Questions Answered Correctly (out of 13) Sample: Self-Efficacy is best described as: “Believing you can accomplish something.”

38. Change in Staff Belief Scales

43. Staff Practices & IntentionsSensitivity / Responsiveness Current Practices (Past Month) Direct Care Staff Only Practice Intentions (Future) Direct Care Staff Only Sample: “Avoid situations that trigger a child’s stress response.”

44. Appendix Slides

45. Current Practices - Subscales Listening / Understanding • “Talk with a child who is not completing tasks or responsibilities to try and understand why.” Investment/ Engagement • “Make a point of showing a child that you care about them and/or enjoy being with them.” Inclusion • “Keep youth out of an activity because they did not complete their responsibilities.”

46. Current Practices - Subscales Flexibility • “Help create expectations or routines that are tailored to a child’s individual needs.” Competence-Building • “Use a real life situation to teach a child how to solve a problem.”

47. Youth Perceptions - Subscales Listening & Understanding • Times when you misbehaved: “Tried to understand why I acted that way.” Inclusion • During recreational activities: “Would not let me participate in group activities.” (R) Investment / Engagement • During recreational activities: “I felt that they enjoyed spending time with me.”

48. Youth Perceptions - Subscales Flexibility • Times when I misbehaved: “Let up on the rules to give me a break.” Competence-Building • Times when I was upset: “Showed me how I could calm down or make myself feel better.”

49. Supplementary Slides

50. Youth Perceptions of StaffStaff Sensitivity / Responsiveness Sample: “They were available to help me whenever I needed it.”