Developing Effective Practice Standards for Your Evidence-Based Program

1. Developing Effective Practice Standards for Your Evidence-Based Program Philip McCallion, Ph.D. Lisa A. Ferretti, LMSW Arthritis Grantee Meeting Albuquerque, New Mexico – July 2011

2. Brainstorming Question…. Why is program fidelity important?

3. Program Fidelity and Quality Programming

4. What is Program Fidelity? • At the organizational/agency level, program fidelity refers to how closely staff and others (i.e. peer leaders, trainers, and evaluators) follow the program that the developers provide. This includes consistency of delivery as intended as well as program timing and costs.

5. Quality Assurance/Quality Improvement (QA/QI) Quality assurance, is concerned with assuring that activities that require good quality in delivery are being performed effectively. “Good quality” is established through comparison with an external standard. Quality Improvement is concerned with raising the quality of program delivery. “Improvement” is established primarily by comparing current performance with past performance with a goal of better attaining the “Quality Assurance” set standard.

6. Continuous Quality Improvement (CQI) • A cyclical process • Utilizes performance objectives to monitor program delivery/goals • Data driven effort enhancing outcomes for all stakeholders and utilizing data from multiple sources • Guides corrective actions • Allows the program team to determine best practices and problem-solve barriers and challenges

7. Why is program fidelity a critical component of CQI and sustainability • Program delivery that is not true to the empirical research that supports it decreases the likelihood that the program will deliver the desired outcomes; this may result in harm. • Poor quality fidelity can result in a range of unintended effects for participants, leaders and trainers that decrease program effectiveness and outcomes.

8. Why should we pay attention to Program/Treatment Fidelity? …Despite the importance of fidelity and consequences encountered when it is breached, techniques used in monitoring fidelity have historically been underreported (Bellg et al., 2004; Borrelli et al., 2005; Calsyn, 2000; Hogue, Liddle, Singer, & Leckrone, 2005; Moncher & Prinz, 1991; Resnick, Inquito et al., 2005; Santacroce et al., 2004)….Frank et al., 2008 …a Type III error, this is the mistake of concluding an intervention is ineffective when it was not implemented in full (Basch, Sliepcevich, Gold, Duncan, & Kolbe, 1985; Glasgow, 2002) Frank et al., 2008 And…it uses program resources and helps to assure that program resources are well spent

9. How Program Fidelity Supports Sustainability • Provides data to make the business case • Responds to funder concerns • Tracks ongoing performance of the program and resources • Builds a multi-faceted infrastructure

10. Begin with the end in mind… • What capacity of workshops is needed and can be sustained? • How many and who will the partners be? • Can we rely on quality delivery of programs? • Will workshops be offered frequently enough and in sufficient locations that it is realistic to encourage referrals/recommendations? • Will delivery, training and quality assurance/improvement be sufficiently assured to meet requirements to justify reimbursement? • How will the pool of leaders be renewed and supported?

11. Workshop Objectives: • Participants will acquire skills needed to: • Identify program fidelity and quality assurance strategies for use in evidence-based health promotion programs • Develop a program fidelity plan that includes multiple partners, assures program quality and leads to the sustainability of high quality deliveries of evidence-based programs.

12. What exactly is an evidence-based program?

13. Randomized Control Trials • Known as the gold standard for evaluating efficacy of interventions. • Randomly assigning subjects to an intervention eliminates any bias from unknown characteristics of the sample that might contribute to treatment effects by balancing these characteristics evenly across intervention groups.

14. Mounting an Intervention RCT Study

15. Extra-Organizational Context ( methods of reimbursement, policies) OrganizationalTherapeutic Person FitProcesses Outcomes culture, climate, structureadherence/fidelity symptoms, functioning Intervention Processes type of treatment, training, supervision, engagement, beliefs, attitudes, values Personal Choice and Control Empowerment (knowledge + self-efficacy), engagement, intentions (i.e., beliefs, attitudes, values) Model of Influences on Effective Implementation & Sustainability

16. Research Phase

17. Translation Phase (Wilson & Fridinger, 2008)

18. Translational Research • Type I – development and testing of treatment approaches generated through laboratory and pre-clinical research • Type II – Enhancing the adoption of effective practices in the community

19. What Does Translation Require • Real world application • Effectiveness – Extensiveness – Efficacy – Engagement in quality implementation • Infrastructure – policy, networks and embedding in delivery systems • Guidelines, standards and quality assurance (Adapted from Spoth, 2008)

20. What Evidence? Lonigan, Ebert & Johnson, 1998; Chambless et al., 1998 At least two controlled group design studies or a large series of single-case design studies Minimum of two investigators (to be well-established) Use of a treatment manual Uniform therapist training and adherence Tested with clinical samples Tests of clinical and functional outcomes Long-term outcomes beyond termination of treatment

21. Best Practices • Processes, practices, or systems widely recognized as improving the performance and efficiency of organizations in a target area, such as health promotion. (NCOA, 2005) • Best practices in health promotion are those sets of processes and activities that are consistent with health promotion values/goals/ethics, theories/beliefs, evidence, and understanding of the environment, and that are most likely to achieve health promotion goals in a given situation ( http://www.bestpractices-healthpromotion.com/id12.html )

22. Components of an Evidence-Based Health Promotion Program • Specific target population • Specific, measurable goal(s) • Stated reasoning and proven benefits • Well-defined program structure and timeframe • Specifies staffing needs/skills • Specifies facility and equipment needs • Builds in program evaluation to measure program quality and health outcomes NCOA, 2006

23. NCOA Model Programs Programs include the essential elements for successful behavior change: • goal-setting • problem-solving • action-planning • ongoing support • monitoring *NCOA – National Council on Aging

24. NCOA Model Programs Effective organizational and program planning strategies for: • social marketing • staff recruitment and training • participant recruitment and retention • partnering • program sustainability • evaluation

25. Health Promotion & Evidence-Based Practice • Some are highly rated as evidence-based • Many do NOT meet criteria for evidence-based • For others the quality of the evidence would not be highly rated …Consideration of Program/Treatment Fidelity issues is still important

26. CDC Approved Physical Activity Interventions for Arthritis • Arthritis Foundation Exercise Program • Active Living Every Day • Arthritis Foundation Aquatics Program • EnhanceFitness • Fit and Strong • Walk with Ease

27. CDC Approved Self-Management Interventions for Arthritis • Arthritis Foundation Self-Help Program • Chronic Disease Self-Management Program • Tomando Control de suSalud • Programa de Manejo Personal de la Artritis

28. Criteria Map Activity link

29. Challenges in the Delivery of Evidence-Based Programs • Delivery in original studies was often supervised by the creator • Delivery was often by PhD students, research team members and/or other trained practitioners • The original manual assumed background knowledge and/or a grounding in professional theories and practices • Limited attention paid to treatment fidelity and how it should be measured • Translational efforts to date relied heavily upon paid, trained staff or have been under the supervision of the program creator

30. Translation for Community Implementation • Delivery needs to move beyond dependence on the daily supervision of the program creator(s) • Delivery cannot assume prior knowledge and professional training of staff nor depend on it • Training and delivery manuals should clearly specify every step of delivery and every piece of knowledge required for a quality translation of the program

31. Translation for Community Implementation Training should be standardized: • A specific, approved training program for leaders is necessary. • A specific, approved train the trainer curriculum is ideal. • Both must include guidance on demonstration requirements so that persons may be certified as leaders/trainers.

32. Translation for Community Implementation Clear guidelines on negotiable and non-negotiable components of the program intervention: • What constitutes successful completion of the program/intervention • What are successful measures of program/treatment fidelity

33. Translation and Program/Treatment Fidelity • The “adoption of effective practices in the community” and the reliance on volunteers and community practitioners for their delivery requires the separation of the program intervention from clinical and program creator management with the assurance that the intervention will continue to be delivered as intended. • How can this be delivered? …. With a focus on effective practice…..

34. Tools for Effective Practice • Attention to the RE-AIM framework: implementation and maintenance • A QA/QI or CQI approach • An understanding of and a belief in the value of program/treatment fidelity as a part of program quality and delivery

35. Adoption Building Programs and Policies with a Large Public Health Impact Implementation Efficacy Effectiveness Maintenance Reach RE-AIM BUILDING BLOCKS THAT TOGETHER PRODUCE PUBLIC HEALTH IMPACT www.re-aim.org

36. The RE-AIM Questions Implementation - Can different levels of “staff” implement the program successfully? - Are different components of the intervention delivered as intended? Maintenance - Can organizations sustain a quality program over time?

37. Program/Treatment Fidelity Five components: • design, • training, • delivery, • receipt, • enactment Resnick, et al., 2005

38. Treatment Fidelity Design: • Can a study adequately test hypotheses in relation to the intervention’s underlying theory? • Why and in what ways do the authors think people’s behaviors will change and can the why and how be demonstrated? Training: • Is there a manual that addresses and specifies all aspects of delivery? • Are people implementing the intervention satisfactorily trained to deliver that intervention? • Is adherence to training manual demonstrated? • Is the training of leaders consistent?

39. Treatment Fidelity • Delivery: Is delivery to participants consistent with the manual and is it demonstrated that participants are only receiving in the intervention what the manual says they should be receiving? • Receipt: Have the participants received and understood what it is intended they should receive and understand? • Enactment: Is there carryover into daily life?

40. Treatment (Program) Fidelity Recommendations Adopt a universal definition of fidelity, include adherence, dose, quality of program delivery, participant responsiveness and program differentiation Standardize measures and methods for fidelity assessment to include both self report and observer data Dusenbury, et al., 2003

41. Treatment (Program) Fidelity Recommendations Increase research on factors influencing fidelity • Provider characteristics • Participant characteristics • Match between providers, participants and the program • Administrative, community & environmental characteristics which influence and promote fidelity of implementation Insist that fidelity be assessed and reported. Dusenbury, et al., 2003

42. Three Key Issues • Evidence-based Health Promotion Programs • Quality Assurance and Quality Improvement in community delivered programs • Importance of Program/Treatment Fidelity

43. The Right Partners To help your program delivery, implementation, quality and sustainability

44. Evidence-based Program Challenges: Implementation, Delivery and Sustainability • Resources • If we build it will they come? • Recruitment • When we build it who will come? • Program quality and fidelity • When they come will it matter?

45. Partners are the Key to Success • Advisory Partners • Delivery System Partners • Site Partners • Support Partners • Special Populations Partners

46. Program Delivery • What is needed for program delivery and which types of partners can/will supply it? • Trained Leaders/instructors • Participant Materials • Workshop/program Locations • Incentives, snacks, stipends, transportation, etc.

47. Program Marketing/Recruitment • Who will market the Program? • Market Targets – • The Demand Side – Participants • The Supply Side – Partners and Feeder Organizations

48. Training and Technical Assistance • Who can/will assure the program delivered is faithful to the design intended by the developers and faithful to the evidence-base? • Training – Sustaining Qualified and Certified Trainers • Technical Assistance – Meeting Standards Established by the Program Developer(s)

49. Quality Assurance/Improvement • How will we track quality measures? • Attendance Monitoring • Workshop/program Observations • Participant Satisfaction • Leader/instructor Feedback • Ongoing Support to Leaders/instructors

50. Valued Outcomes/Impact • What are the valued outcomes for each partner? • Increased Physical Activity • Quality of Life • Participant Activation • Health Improvement , Health Care Utilization Outcomes • Others?