Training is Not Enough: Intervention and Evaluation Supplements

1. Training is Not Enough:Intervention and Evaluation Supplements Christina J. Groark and Robert B. McCall University of Pittsburgh Office of Child Development

2. Overview—Intervention • Training alone is a minimally effective intervention • Need: • Hands-on technical assistance in situ • Supportive work environment • Continuous monitoring, positive supervision

3. Overview—Evaluation • Assessing trainee’s satisfaction and learning is a minimum evaluation • Need: • Changes in trainee’s on-the-job behavior • Changes in the behavior/outcomes of clients of the trainees • Analyses to show trainees behavior change mediated the intervention's effect on clients’ behavior/outcomes

4. Intervention Training alone is minimally effective Why? Trainees often do not change their behavior Need: • On-the-ground technical assistance • Supportive work environment • Monitoring and positive supervision

5. Intervention On-the-ground technical assistance • Observe (videotape) client on job • Constructive, positive feedback • Modeling with client on job • Daily review

6. Intervention

7. Intervention Work environment • Supports behaviors taught in training • Hand washing—need a sink, waterless soap handy • Early care and education—need small group size, low children:caregiver ratio, a few consistent caregivers

8. Chaotic Child Environment

9. Orderly Child Environment

10. Monitoring and positive supervision • Sets expectations, standards, reminds, rewards • Continuing system, supervisor, staff meetings • Hand washing • Early care and education—teachable moments, responsiveness to child-directed initiatives

11. Illustrative Example I Latin American Orphanage Intervention (birth-6 years) • Training—sensitive, responsive interactions • 6 one-day training sessions • On-the-ground technical assistance • 12 days more than planned because caregivers were not implementing

12. BUT – • Limited environmental change • 4 primary caregivers per ward • 8-10 children per caregiver • No continuing supervision

14. Results—Caregiving behavior/environment (ITERS/ECERS) 7 Mean Total ITERS/ECERS 6 Typical USA Early Care 5 4 3 Younger 2 Older 1 Pre- Post- Intervention

15. Results—Children’s development (Battelle) 100 Typical Parent-Reared 90 Battelle 80 Younger 70 Older 60 Post- Pre- Intervention

16. Conclusion • Needed hands-on technical assistance to promote implementation • Limited supportive work environment • No supervisory system • Some improvement in caregivers, children, but NOT MUCH

17. Illustrative Example II St. Petersburg (Russia) Orphanage Intervention • More intensive, comprehensive • All major components

18. Orphanage Caregiving Before Interventions • Minimum warm, sensitive, contingently-responsive caregiver-child interactions • Perfunctory, business-like; little talking • Caregiver directed • Conformity St. Petersburg-USA Orphanage Research Team, 2005, 2008

20. Orphanage Structure Before Interventions • 12-14 children/ward • 9-12 caregivers/week • Homogeneous age, disability groups • Periodic graduations • 60-100 caregivers before age 19 months St. Petersburg-USA Orphanage Research Team, 2005, 2008

21. Intervention • Training ◦ Train Trainers ◦ 12 sessions, 2-3 hrs. ◦ Warm, sensitive, responsive ◦ “Love these kids…” • Hands-on technical assistance

23. Intervention—Supportive Work Environment • Reduced group size to 6-7 • Assigned primary, secondary caregivers—6 vs 9 cgrs. • Changed work schedules • Integrated groups by age, disabilities • Eliminated periodic graduations to new groups • Family hour

24. Intervention—Monitoring and Positive Supervision • Training for supervisors • Trainer observed, coached both caregivers and supervisors • Supervisors met periodically to problem solve, case reviews, mutual support

25. Evaluation Three orphanages (Baby Homes), birth – 4 years • T+SC/M&S—Training plus Structural Changes, Monitoring, Supervision • TO—Training Only • NoI—No Intervention

26. Evaluation—Trainee Learning

27. Evaluation—Improved Caregiving Longitudinal Sample T+SC

28. Evaluation – Children’s Development 100 90 Typical Parent Reared T+SC Developmental Quotient 80 TO 70 NoI 60 50 40 9+ mos First 4-9 mos Time in Intervention

29. Evaluation--Mediation Analysis • Intervention improved caregiving • Intervention improved children’s development • But was children’s developmental improvement associated with improved caregiving?

30. Evaluation—Mediation Analysis Two-Stage Theory of Change: T= Train Caregivers Better Better SC=Structural Changes Caregiving Children’s Environment Development Independent Variable Mediator Outcome T+SC vsNoI HOME Battelle DQ

31. Evaluation--Mediation Analysis Independent Variable T+SC vs NoI Intervention Outcome Battelle Total Effect Ĉ = 14.14(2.07)*** Effect of Indep. Var. on Mediator Effect of Mediator on Outcome Controlling for Indep. Variable Mediator HOME a = 3.41(.22)*** b = 1.80 (.59)** Direct effect Controlling for Mediator Independent Variable T+SC vs NoI Intervention Outcome Battelle Ĉ = 8.00(2.87)*** Mediator accounts for 43% of Total Effect

32. Conclusion • Intervention ◦Training alone is often minimally effective ◦Has greater effect if accompanied by • On-the-ground technical assistance • Supportive work environment • Monitoring and supervision

33. Conclusion • Evaluation is improved if • Some training vs. No Training comparison • Measures of • Learning • Work-place behavior change • Outcome behavior of clients • Mediation analysis

34. Evaluation Mediation Analysis:MacKinnon, D. P., & Dwyer, J. H. (1993). Estimating mediated effects in prevention studies. Evaluation Review, 17(2), 144-158.