Lessons Learned About Fidelity & Adaptation from the Massachusetts SIG

1. Lessons Learned About Fidelity & Adaptation from the Massachusetts SIG Wayne Harding, Ed.M., Ph.D. Social Science Research & Evaluation, Inc. 21-C Cambridge Street Burlington, Massachusetts 01803 781-273-4206, wayneh@gis.net

2. Topics • Why is fidelity & adaptation an issue? • Massachusetts SIG (MassCALL)* policies and procedures concerning fidelity and adaptation. • Lessons learned. • Fidelity monitoring tool. * Massachusetts Collaboration for Action, Leadership, and Learning.

3. Why Is Fidelity / AdaptationAn Issue? • Replication can be hard. • Limited supply of science-based programs makes adaptation likely. • Adaptation happens.

4. Adaptation Happens • Between 23% and 81% of program activities may be omitted during implementation (Durlak, 1998). • Only 19% of schools implemented research-based curricula with fidelity (Hallfors and Godette, 2002). • Only 75% of students received at least 60% of the Life Skills Training sessions (Botvin et al., 1995).

5. MassCALL Pre-award Policy & Monitoring Procedures • Pre-RFP meetings discussed science-based prevention, importance of selecting appropriate programs, and guidelines for adaptation.

6. MassCALL Pre-award Policy & Monitoring Procedures • The RFP requested information about adaptations planned and rationales for them.

7. Post-award Policy & Monitoring Procedures • Required evaluation plans that identified any adaptations and plans to monitor program implementation. • Site visits to all subrecipients included importance of fidelity, identification of adaptations, alternatives to adaptation.

8. Post-award Policy& Monitoring Procedures • MassCALL implemented an approval process for adaptations. • Biannual reports from subrecipients included information on any adaptations and rationales for them.

9. Lessons Learned: Adaptation Is Likely Types and Frequency of Adaptations Among MassCALL Subrecipients (Formica & Harding 2002)

10. Lessons Learned:Adaptation Is Likely • All 21 program sites made one or more program adaptations. • Adaptations occurred for all program types. • The adaptations applied to all aspects of the programs (content, delivery methods, target population, setting, delivery agent). • Conservatively, 83 adaptations were made.

11. Lessons Learned: • Adaptation may continue. • Important to track adaptation to both original and proposed program. • Important to track adaptation to evaluation procedures.

12. Lessons Learned • Adaptation can be tracked “accurately.” • Asking about the variety of types of adaptations may increase accuracy with which they are reported.

13. Fidelity Tool • Assess fidelity to original or model program and to the proposed (adapted) program. • Assess fidelity at intervals. • Assess multiple aspects of the program (e.g., content, delivery methods, population served, setting, delivery agent). • Assess the rationale for adaptation. • Assess adaptations to evaluation methods (in both original and proposed program). • Permits assessment/comparisons across program types.

14. References Botvin, G.J., Baker, E., Dusenbury, L., Botvin, E.M., & Diaz, T. (1995). Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. Journal of the American Medical Association, 273: 1106-1112. Durlack, J.A. (1998) Why Program Implementation Is Important. Journal of Prevention & Intervention in the Community. 17(2): 5-18. Formica, S. and Harding, W. M. (2001). Documentation of adaptations to science-based programs implemented by MassCALL sub-recipient communities. Massachusetts Collaboration for Action, Leadership, & Learning (MassCALL), Massachusetts Department of Public Health, Bureau of Substance Abuse Services, Boston, MA. Hallfors, D. and Godette, D. (2002) Will the "Principles of Effectiveness' Improve Prevention Practice? Early Findings from a Diffusion Study. Health Education Research. 17(4): 461-470.