IMPLEMENTATION QUALITY RESEARCH OF PREVENTION PROGRAMS IN CROATIA

1. IMPLEMENTATION QUALITY RESEARCH OF PREVENTION PROGRAMS IN CROATIA MIRANDA NOVAK University of Zagreb, Faculty of Education and Rehabilitation Sciences Josipa Mihic, Clemens Hosman, Celene Domitrovich Health Promotion Research Centre 18th Annual Summer Conference National University of Ireland Galway

2. MOTIVATION FOR RESEARCH • Although the efficacy of various evidence-based interventions has been established through carefully designed trials in control conditions, there is a lack of evidence for its utilization in natural community conditions (Kam, Greenberg and Weiss,2003; Greenberg et al., 2005; Fixsen et al., 2005, Proctor and Rosen, 2008) • A more systematic process is warranted to translate efficacy results into positive participants’ outcomes IMPLEMENTATION RESEARCH SCIENCE TO PRACTICE CONTEXT FACTORS AND SUPPORT SYSTEM CHARACTERISTICS OF PROGRAM

3. DEFINITION OF IMPLEMENTATION • implementation refers to what a program consist of when it is delivered in a particular setting(Durlak and Dupre, 2008) • characteristics of intervention itself and characteristics of the intervention support system • it is important to differentiate between factors affecting implementation quality(Fixsen et al., 2005, 2009) and implementation aspects(Durlak and Dupre, 2008)

4. WHY WE MONITOR IMPLEMENTATION To knowwhatactuallyhappened To provide feedback for quality To strengthentheconclusionsmadeaboutoutcomes To examinethe change process To understandtheinternal dynamics of program To advanceknowledge for replication To strengthenthequalityofevaluation

5. IMPLEMENTATION FACTORS - CAPACITY FOR PROGRAM IMPLEMENTATION INDICATORS OF IMPLEMENTATION QUALITY PERCEIVEDPROGRAM IMPACT

6. Research project »Preffi – Quality assurance in the Region of Istria« Department of Health and Social Care, Region of Istria and Faculty of Education and Rehabilitation Sciences, University of Zagreb Incorporation of science-based principlesin prevention practice in Istria through collaboration of scientists and practitioners

7. ORGANIZATION MANAGERS PROGRAM AUTHORS PROGRAM DELIVERERS TRAINING PROGRAM Principles of science-based practice Logic modelling  Implementatio  Evaluation  Advocacy Interactive group education and Individual consultation

8. The general aim of this research was to study implementation processes and their outcomes in prevention programs in Croatia.

9. To construct valid and reliable measures of implementation quality. • To explore the level and variation of implementation quality • To explore the differences in perception of implementation quality between program managers, implementers and participants. • To explore the relationships of implementation factors and indicators of implementation quality. • To test the impact of the Training for Prevention on the level of implementation quality in experimental group

10. Parenting Positivedevelopmentand SEL Substanceabuse SAMPLE: 24 mental health promotion and prevention programs

11. MEASURES The Preffi 2.0 instrument • Implementation FactorsQuestionnaire for Program Managers • Implementation Factors Questionnaire for Program Implementers • Indicators of Implementation Quality Questionnaire for Program Implementers • Indicators of Implementation Quality Questionnaire for Program Participants

12. June 2010 - December 2010 - January 2011 February 2011 - February 2011 - January 2011 January 2011 March 2011 December 2011 Implementation of 24 programs Focus groups Evaluate Initial set of items 3 assessors independently assess 24 written programs’ proposals Division 24 programs : experimental (N=12) and control condition (N=12) EXPERIMENTAL Training for Prevention to experimental condition Post- Assessment Implementation quality Mid- Assessment implementation quality 19 PROGRAM MANAGERS 50 PROGRAM DELIVERERS 434 PROGRAM PARTICIPANTS CONSTRUCTION OF SCALES FOR IMPLEMENTATION QUALITY ASSESSMENT

13. June 2010 - December 2010 - January 2011 February 2011 - February 2011 - January 2011 January 2011 March 2011 December 2011 Implementation of 24 programs Focus groups Evaluate Initial set of items 3 assessors independently assess 24 written programs’ proposals Division 24 programs : experimental (N=12) and control condition (N=12) EXPERIMENTAL Training for Prevention to experimental condition Post- Assessment Implementation quality Mid- Assessment implementation quality 19 PROGRAM MANAGERS 50 PROGRAM DELIVERERS 434 PROGRAM PARTICIPANTS STUDY OF IMPLEMENTATION QUALITY

14. June 2010 - December 2010 - January 2011 February 2011 - February 2011 - January 2011 January 2011 March 2011 December 2011 Implementation of 24 programs Focus groups Evaluate Initial set of items 3 assessors independently assess 24 written programs’ proposals Division 24 programs : experimental (N=12) and control condition (N=12) 22 PROGRAM MANAGERS 55 PROGRAM DELIVERERS 744 PROGRAM PARTICIPANTS EXPERIMENTAL Training for Prevention to experimental condition Post- Assessment Implementation quality Mid- Assessment implementation quality STUDY ON THE TRAINING FOR PREVENTION IMPACT 19 PROGRAM MANAGERS 50 PROGRAM DELIVERERS 434 PROGRAM PARTICIPANTS

15. RESULTS

16. STUDY OF IMPLEMENTATION QUALITY

17. Reports on implementation factors across 24 programs show that views of managers and programs implementers are rather similar MAN: Lowest ratings for standardization, training and monitoring IMP: Lowest ratings for standardization and training

18. Indicators of implementation quality are really high – group results all over value of 3 Overall group results show that program implementers give similar although higher ratings than participants

19. STUDY OF THE TRAINING FOR PREVENTION IMPACT

20. Managersfromexperimentalconditionsgive more criticalanswersaboutimplementationfactorsthanthosefromcontrol= CHANGE OF CRITERIA Levels of implementation factors reported by managers and the effect of the Training for Prevention at post-intervention

21. Implementersfromexperimentalconditionsgive more criticalanswersaboutimplementationfactorsthanthosefromcontrol= CHANGE OF CRITERIA Levels of implementation factors reported by implementers and the effect of the Training for Prevention at post-intervention

22. At post-test, program implementers from the experimental group did not report improved indicators of implementation quality in comparison with the control group Levels of implementer’s ratings on indicators of implementation quality and the effect of the Training for Prevention at post-intervention

23. At post-test, participants from the experimental group report onhigher levels of implementation quality, two of them being significant - quality of delivery and responsiveness. Levels of participant’s ratings on indicators of implementation quality and the effect of the Training for Prevention at post-intervention

24. Moderator analyses have shown that Training for Prevention is more effective for SHORT PROGRAMS

25. Moderator analyses have shown that Training for Prevention is more effective for PROGRAMS WHERE MANAGER IS NOT ACTIVE

26. CRUCIAL CONDITIONS FOR SUCCESFUL IMPLEMENTATION

27. Investment in implementation factors

28. THANK YOU FOR YOUR ATTENTION mnovak@erf.hr