The Importance of Fidelity as a Means for Enhancing and Assessing Program Quality

1. The Importance of Fidelity as a Means for Enhancing and Assessing Program Quality A Case Study of the RESPECT Program

2. What is fidelity? • Adherence to program design, both in terms of the amount and quality of what is delivered • Whether a program or treatment is being implemented as planned • “Methodological strategies used to monitor and enhance the reliability and validity of behavioral interventions.” (Bellg et al., 2004)

3. Why is fidelity important? • Ability to justify results • An aspect of evidence-based research • If you want to replicate or compare your program (Smith et al., 2007) • A way to measure AND enhance program quality

4. The RESPECT Program • Three-year implementation in four public middle schools in Boston, MA. • Adventure approach to social/emotional skill building • By the end of third year, all teachers and administrators on board and implementing the program • Research and implementation funded primarily though outside sources

5. RESPECT Program Components • Full Value Concepts • Adventure activities • Goal setting • Quick Class • Assessment tools • (Initial teacher/administrative training and ongoing consultation)

6. RESPECT Program outcomes • Increased SEL skills • Decreased behavioral incidents • Increased time on task • Increased academic performance

7. RESPECT Program Measures • Social Skills Rating Scale (SSRS) • Perceived Competency of Functioning Inventory (PCFI) (Self-efficacy) • Grades and state-mandated test scores • Disciplinary incidents • Attendance • Compared with two control schools

8. Status of findings • Still in progress • Generally, second year data shows more promise (6th grade teachers were into their second year/7th grade students in Year 2 had received RESPECT for two years) • Without accurate data showing level of fidelity to the model harder to state the meaning of the data

9. Factors that influence fidelity • Program design • Level of complexity • Level of articulation of program components • Time and materials • Staff • Training • Supervision • Level of “buy in” and perception of effectiveness

10. How to measure fidelity • Direct observation • Develop protocols for what constitutes evidence of program components • Self-report/self-monitor • Develop questionnaires for practitioners to rate themselves on quantity and quality of implementation • Interviews

11. References • Bellg, A. J., B. Borrelli, et al. (2004). Enhancing Treatment Fidelity in Health Behavior Change Studies: Best Practices and Recommendations From the NIH Behavior Change Consortium. Health Psychology 23(5): 443-451. • Gresham, F. M., MacMillan, D. L., Beebe-Frankenberger, M. E., Bocian, K. M. (2000). Treatment Integrity in Learning Disabilities Intervention Research: Do We Really Know How Treatments Are Implemented? Learning Disabilities Research and Practice 15(4), 198-205. • Hill, L. G., K. Maucione, et al. (2007). A focused approach to assessing program fidelity. Prevention Science 8(1): 25-34. • Lane, K. L., K. M. Bocian, et al. (2004). Treatment Integrity: An Essential--But Often Forgotten--Component of School-Based Interventions. Preventing School Failure, 48(3), 36. • Melde, C., F.-A. Esbensen, et al. (2006). Addressing Program Fidelity Using Onsite Observations and Program Provider Descriptions of Program Delivery. Evaluation Review 30(6): 714. • Orwin, R. G. (2000). Assessing program fidelity in substance abuse health services research. Addiction 95: S309-s327. • Rimm-Kaufman, S.E. & Chiu, Y.I. (2007). Promoting Social and Academic Competence in the Classroom: An Intervention Study Examining the Contribution of the Responsive Classroom Approach. Psychology in the Schools, 44(4). • Smith, S. W., A. P. Daunic, et al. (2007). Treatment Fidelity in Applied Educational Research: Expanding the Adoption and Application of Measures to Ensure Evidence-Based Practice. Education and Treatment of Children 30(4ov): 121.