Keeping Fidelity in an Evidence-Based Program: A University-Community Agency Research Partnership

1. Keeping Fidelity in an Evidence-Based Program: A University-Community Agency Research Partnership Susan B. Stern Factor-Inwentash Faculty of Social Work University of Toronto Katina Watson YOUTHLINK Child Development Institute Funded by SSHRC Institutional Grant, Royal Bank Fellowship

2. Overview • Context • Evidence-Based Parenting Program • IY Early Learning Centres in Toronto • Treatment fidelity - transporting evidence-based interventions to community practice • Understand barriers/facilitators • Study implementation process as it unfolds

3. Why is Fidelity Important?

4. WHY EFFECTIVE PARENTING PROGRAMS ARE IMPORTANT • In the absence of treatment, child conduct problems typically intensify • The majority of chronic behaviour problems in school age children had an early onset • If the difficulties continue after entry into school, the risk of peer rejection, later substance abuse and delinquency increases. • Parent-child interaction key risk factor for early onset conduct problems (esp. harsh/inconsistent discipline, low nurturance). • Parent training is the most empirically supported intervention strategy for decreasing problem behaviours.

5. Incredible Years Model • Emphasizes building positiveparent-child interactions and breaking coercive cycles • Increase parent self-efficacy/coping, and problem-solving skills • Increase parents’ positive communication skills and reduce anger and the use of criticism and unnecessary commands. • Strengths-based and goal focused • Learning through modeling and practice • Action-orientedwith home activities • Creates support networks and opportunities to share a range of responses and experiences • Flexible and collaborative, tailored to each child’s temperament and family situation. • Focus on key principles behind strategies • Led by skilledand knowledgeable leaders

6. Expert Model Collaborative Model • Leader in charge, dispenses advice >Parents are active and invested • Parents passive recipients of service >Highlight strength and change • Discusses problems and deficits > Reciprocal working relationship • Can increase guilt, self blame >Non-blaming, non-hierarchical • Diagnoses problems, attempts > Mutual problem-solving, to impose a general solution increases self-efficacy > Can increase “resistance” > Climate of trust and openness

7. Incredible Years • 6 randomized control group studies -Webster-Stratton • 5 independent replications • SAMHSA Model Strengthening Families Program • OJJDP Blueprints for Violence Prevention • UK Home Office recommended EBP for antisocial children • Clinic studies • Clinically significant improvement in positive parenting/child behavior • Head Start - universal prevention with disadvantaged high risk families • Significant differences in parenting practices and child behavior/affect • Maintenance at one year follow ups high BASIC Preschool and School Age, 3-10 years

8. Research (cont’d) • Parent Training group shown to be as effective as individual (family) training with high risk parents whose children had conduct problems. 70% maintained improvements at one year. • Many families stated preference for group over individual family treatment. • Children with high levels of baseline conduct problems and mothers with high levels of critical parenting benefited the most. • Participation and engagement rates high as many practical barriers are reduced .

9. IY Key Dissemination Strategies • Three day intensive trainingworkshop for group leaders • Clear, detailed manualwith session by session checklists outlining key principles • Series of video taped vignettes illustrating parent-child interactions • Weekly home activities and summary lists of key concepts for parents • Comprehensive book • Series of integrity checklistsand leader evaluations

10. Research Foci Questions • Barriers to adherence • Reasons for departures • What facilitates adherence to IY protocol

11. Program Context • 3 year project • Year 1-3 data • 29 groups • 18 Group leaders • MSW or Post-secondary education • Range of 4-20 years experience (M=10 yrs.)

12. Parent DemographicsRound 3 and 7 • 93% mothers, 7% both parents (n=44) • Child age range, 6 mo - 15 yrs (n=43)* • 48% <=3yrs. • 39% 4-7 yrs • 7% 8-11 yrs. • Ethnicity (n=41) * • 41% White • 18% Black • 18% South Asian • 4.5 % Aboriginal, 4.5% Filipino, 2.3% Latin American, 4.5% Other • 59% English as first language (n=43)

13. Parent Demographics (cont.) • Income (n=41)* • 34.1% < $15,000 • 18.2% $15,000 – $24,999 • 40.9% >= $25,000 • 50 % on social assistance (n=43) • 71% high school completed/GED or higher (n=44)

14. IY Fidelity Measures • Leader Session x Session Checklist • Group Leaders Written Narratives • Leader Rating Scale (Program Elements & Collaborative Process) • Independent Supervisor Written Observations • Audiotapes of group leaders joint meeting

15. Group Leaders Focus Group • To understand program implementation and adherence issues over the course of multiple IYPP groups • Semi-structured by interview guide • Two focus group facilitators • Trained by one of the lead researchers

16. Focus Group Questions • What barriers did you as group leaders face or perceive the parents to experience in implementing the IYPP. • What facilitated implementing the program and/or overcoming barriers? • Given your perceptions of the parents, what else would have helped in the implementation of the program? • What do you think were the most effective intervention components and processes? • If IY was to be implemented here or in another organization, what modifications do you think the program needs, or parents have suggested, to improve the fit with a diverse and universal prevention population? • What organizational factors or other supports do you think would be necessary for program sustainability?

17. Trustworthiness of the Data • Dependability of the data through audio-taping and transcriptions • Open and axial coding for thematic analysis • Credibility through persistent observation with participants (e.g. group meetings, observations of sessions) • Prolonged engagement • Negative case analysis • Multiple independent coders • Member checking • Confirmability through quotes

18. IY Protocol Adherence All Rounds • Session x Session Checklist • Overall adherence “DID I” Grand mean 83.88 % • Role play Grand mean 59.94 % • Buddy calls Grand mean 52.77 % • Vignette Grand mean 63.51% • Group Leader Rating Scale • Overall adherence across sites 4.48 • Leader group process skills 4.69 • Showing vignettes 4.45 • Practice and role plays 3.8 • 4th session observation 3.83 • 8th session observation 3.71

19. IY Protocol Adherence Cont’d • Vignettes: Videotape modeling with core vignettes occurred in at least 6 (55%) and often all 11 (100%) sessions that called for vignettes (M= 9.7 or 88%). • Role Plays: Role Plays occurred in as few as two (20%) but sometimes all ten (100%) sessions that called for at least one role play (M = 6.34 or 63%)

20. Negative Cases:Program Adherence • Buddy calls • Vignettes • Role plays

21. Balancing Parent Preferences with Structured Program Components • Collaborative vs. Expert role • Parent discussion preference • Desire to keep parental involvement high • Discomfort with new, unfamiliar (“strange”) activities & skills (parents & leaders) • Cultural preference for learning style e.g., lecture, “education”

22. Balancing parent preferences with structured program components “It is very hard to cover the agenda because of the large group and there is a lot of discussion.” “Often that gets into a pretty large story.” “Evaluations show that parents enjoyed group discussions and did not find role-play as helpful.” “Less vignettes were shown to make time for more discussion.”

23. Balancing Parent Needs within the Groups • Various educational levels • Mixed socioeconomic levels • Various learning styles (lecture, culture) • Range of child ages • Different perceived parent needs re: relevance of IY strategies (e.g., -need for play, time-out)

24. Parent Needs • “There were people with different levels of education and sometimes with some people it was pretty basic and others. . .so it was sometimes difficult to maintain a level where they don’t get bored because you are repeating.” • “Having too much of a wide spectrum of parents with children from literally 6 months old to twelve years old in a group that was specifically designed for a specific age group was a bit challenging for me to do.”

25. Time Constraints • Leader workload/preparation time • Amount of content • Balance content/structured activity time with parent talk time • Balancing individual parent needs with collective needs • Time for reviewing & setting up homework • Parent tardiness (time constraints in their own lives to get to group)

26. Language Issues • Language (ESL) • Time to express self in 2nd language • Translation time • Difficulties in understanding vignettes, written materials • Lingo • Homework difficulties – reading/writing ESL issues took precedence over cultural differences.

27. Language (ESL) “Sometimes we have to be patient, as people are struggling with language and having trouble explaining themselves.” (relates to issue of not enough time) “We were worried that we have so much content to begin with this is going to take forever with two interpreters” “Some parents had difficulties understanding the video-tapes because of language.” “Parents who had difficulty with English could not do the written assignments.”

28. Culture • “Spanking discussion was excellent because it became a discussion about cultural values, how people were raised.” • “In many cultures its different, and I think the tendency is to get to the role of education” • “I find people from traditional cultures learn and push us to be more educational”

29. Relevance • Buddy calls • Vignettes • Outdated • Most focus on one child without siblings • Few examples of single parents, father only in some topics • Many parents had younger children-need to address attachment, developmental issues • Did not cover some common concerns, e.g., bullying, sibling conflict

30. Relevance “The vignettes do not match the teaching points of the topics. For example it [time out] should be used as a last resort yet vignettes show minor things to be timed out (e.g. not hanging up coat).” “Vignettes were an issue this week as in some sections there were not mother-child examples and some mothers had more trouble identifying with the father-child vignettes . . . Other sections were difficult because they show two-parent families.” “Some seem redundant, same topic. Although parents comment they enjoy them but they are dated and some seem staged.”

31. Role Play Reluctance • Anxiety • Self-consciousness • Parents did not think of or experience them as useful • Childish • Prefer discussion • Lack of time • Leaders’ own discomfort

32. Role Play Discomfort/Reluctance “ . . . We weren’t so great at role plays and neither were the parents . . . They preferred to have discussion.” “Because of discussion time didn’t have time for role plays.” “We have more trouble fitting in the role plays because they are time consuming.”

33. Concerns with Time out • Confusion between home use and child care policy • Does not work • Values and cultures • Safety issues • Inappropriate, not relevant for their family • Child temperament • Parents own style • Concerns about abandonment, “letting kids scream”

34. Structure of IY materials, manual • Hard to follow at times – not user friendly • Mismatch • Manual instructions to use Time out for serious behaviors/ vignettes show minor behavior (e.g., not hanging up coat) • One session had 2 homework assignments • Mixing 2 topics in one session

35. Disconnect Between IY procedures for selected/ indicated population and universal preventive one • Parent reactions to specific IYPP strategies (e.g., depth of teaching time-out, buddy calls) • Group leader discomfort/boundaries teaching vs. clinician role (e.g., follow up calls) “I think because the program was developed for high risk situations and they did put in all of these things to help and maybe it isn’t appropriate for a parenting group”

36. Organizational Barriers • Communication • Commitment to the program • Outreach to parents • Clear expectations between child care centres and group leaders

37. Organizational Barriers “It’s a three-way relationship, you have us, the parents, and the childcare centre and all parts of it have to be strong in order for it to work.” “I think that if there are expectations and boundaries about what the childcare centre is supposed to do and what we are supposed to do, we need that, for some groups that wasn’t very clear.” “It’s challenging and goes back to the buy-in from the centre and I think we have to screen very carefully who we’re going to be doing [the program] with.”

38. Barriers to Protocol Adherence • Balancing Parent Preferences with Structured Program Components • Time Constraints • Language (ESL) • Relevance • Role Play Reluctance • Confusion/Concerns with Time Out • Structure of the Materials and Manual • Disconnect between High Risk/Clinical and Universal Preventive IY Groups • Lack of Control • Cultural • Balancing Parent Needs • Organizational

39. Facilitators/Strategies to Overcome Barriers • Highlighting and reinforcing fidelity • Validating • Provides rationales [with group leaders & parents] • Collaboration • Problem solving barriers • Focusing on goal

40. Highlighting and Reinforcing Fidelity “It is (role plays) for many intimidating. I remember in training lots of us hesitated. But we also found out we needed them because some skills needed practicing and it was hard to remember everything.” “Reiterating the importance of vignettes in general - (research) shows they are helpful even if parents don’t like them.”

41. Facilitators of Adherence • Supervisor responsiveness, facilitator of organizational support, concrete resources • 1st session preparation • Elicits ongoing feedback • Follow through • Mediator between IY group leader and the CC centre “Definitely [supervisor] makes it easier for us” “What are your thoughts and ideas about what would be helpful right now?” “Is there anything more I could be doing?” • IY structure, materials (manual, vignettes, book)

42. Monitoring Fidelity and Problem Solving Barriers • Accountability for monitoring/commitment • Understanding reasons for departures • Ensuring program’s responsiveness to parent needs • Tenor of problem solving collaboratively “Weekly evaluations [parent] and follow up calls give us a way to make sure we are being responsive to parents, and our own leader checklists will give us feedback about what we are and aren’t doing.” “Please note which [vignettes] you used on leader checklist and why if skipped or added vignettes.” “Let’s make notes of barriers to doing home activities which are core to the group.”

43. Monitoring Fidelity/PS Barriers “When I observe groups, it will be in the same spirit, to learn and understand the needs of each group and how and why the group leaders have made any modifications to meet the needs of their particular parents” “Our motto always was, that any departure was OK, they could trust their judgment or suggest something, as long as they had a rationale or a clinical reason for it.” Attention to adherence & procedures embedded in agency, feasibility for adherence monitoring in social work community settings

44. Group Leader Fit & Belief in IYPP Core Components & Collaborative Process “You might find that people self select, cause I agree with you, when I came to it, I really liked the material, and agreed with most of it – with the foundation, so I choose to continue doing it, where others might not like it , find it to rigid, so you find people that will adhere to it because they like it.” “I think you do things that you believe in, and most of us are philosophically similar.”

45. Key Contributors to Adherence • Overall, adherence was good with the exception of role plays, # of vignettes shown, & buddy calls • Therapists’ belief in the importance of the program’s core components and collaborative process • Emphasis on accountability and ongoing session monitoring • Therapists’ sense of ownership and responsibility for quality control • Supervisor understanding of IY and underlying research, commitment to high quality implementation • Supervisor commitment to supporting group leaders and follow-through • Training/structure of program and materials • Concrete resources

46. Limitations • Fidelity data either self report or supervisor observation – possibility of systematic bias • Supervisor’s dual role – potential conflict of interest • Validation of quantitative adherence measures Triangulation of findings with clear and consistent patterns across different methods & data sources, and stability over time of seven IYPP cycles, increases confidence in the results.

47. Next Steps

48. Upcoming Article • Stern, S.B., Alaggia, R., Watson, K. & Morton, T.R. (in press). Implementing an Evidence-Based Parenting Program With Adherence in the Real World of Community Practice. Research on Social Work Practice.