Why Quality

1. Why Quality Daniel F. Perkins Penn State University Professor of Family and Youth Resiliency and Policy Assumptions: Sustain an EBI Working in a coalition Exercise -Have them think about an EBI or program that they have been involved in that has sustained itself? --What was sustained? --What factor was most important in it being sustained --What was the largest barrier to sustaining it? -Share it with a partner? -Solicit responses from the audience Assumptions: Sustain an EBI Working in a coalition Exercise -Have them think about an EBI or program that they have been involved in that has sustained itself? --What was sustained? --What factor was most important in it being sustained --What was the largest barrier to sustaining it? -Share it with a partner? -Solicit responses from the audience

3. Evidence-based and Evidence Informed Programs/Practices Theoretically sound innovations evaluated using a well-designed study (randomized controlled trial or quasi-experimental design) and have demonstrated significant improvements in the targeted outcome(s). Evidence-informed is the integration of experience, judgment, and expertise with the best available external evidence from systematic research. Universal: all populations Selected: groups who are at higher risks (children of alcoholics) Indicated: Groups already engaged in the problem behaviorUniversal: all populations Selected: groups who are at higher risks (children of alcoholics) Indicated: Groups already engaged in the problem behavior

5. EBP work, but .. Research has shown that most aren�t being implemented with sufficient quality or fidelity In 500 schools and 14 types of programs, 71% of content was delivered, but only half of the programs followed recommendation implementation practices. (Gottfredson & Gottfredson, 2002) Very few programs measure or monitor implementation quality

6. Positive Program/Practices Outcomes ? Effective Implementation The usability of program or practice has nothing to do with the weight of the evidence regarding it Evidence on effectiveness helps you select what to implement for whom Evidence on outcomes does not help you implement the program

7. Why focus on implementation? Programs will likely show no effect when implemented poorly Quality implementation is linked to better outcomes Quality implementation does increase the probability of sustainability

8. Implement Innovations

9. Evidence-based programs/practices are most effective when they are implemented with Quality High Quality = Fidelity = the practitioners use all the core intervention components skillfully

10. FIDELITY COMPONENTS Adherence: delivered the way it is designed with correct protocols and trained staff Exposure/Dosage # of sessions delivered, length and frequency Quality of Program Delivery ways in which staff deliver it (skills and attitude)

11. FIDELITY COMPONENTS Reach: the proportion of intended partcipants who actually participated in the program Participant Responsiveness: the extent to which participants are engaged in the programme (attendance, + reactions)

12. Life Skills Training Program Botvin, G.J., Baker, E., Dusenbury, L., Botvin, E.M., Diaz, T. (1995). Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. Journal of the American Medical Association, 273, 1106-1112. Linda Dusenbury, NIDA Dissemination Conference (April, 2003), http://www.tanglewood.net/ (Under Conference Presentations) Program Description found at: http://www.lifeskillstraining.com/program.cfm �The LifeSkills program is designed for both:  Elementary school students and Middle/junior high school students It has been evaluated and proven to be effective with: White middle-class students, Ethnic minority students (primarily African-American and Hispanic), Inner-city urban populations, Suburban populations, Rural populations Components The LifeSkills program consists of three major components that cover the critical domains found to promote drug use. Research has shown that students who develop skills in these three domains are far less likely to engage in a wide range of high-risk behaviors. The three components include: Drug Resistance Skills enable young people to recognize and challenge common misconceptions about tobacco, alcohol and other drug use. Through coaching and practice, they learn information and practical ATOD (Alcohol, Tobacco, and Other Drug use) resistance skills for dealing with peers and media pressure to engage in ATOD use. Personal Self-Management Skills teach students how to examine their self-image and its effects on behavior; set goals and keep track of personal progress; identify everyday decisions and how they may be influenced by others; analyze problem situations, and consider the consequences of each alternative solution before making decisions; reduce stress and anxiety, and look at personal challenges in a positive light. General Social Skills teach students the necessary skills to overcome shyness, communicate effectively and avoid misunderstandings, initiate and carry out conversations, handle social requests, utilize both verbal and nonverbal assertiveness skills to make or refuse requests, and recognize that they have choices other than aggression or passivity when faced with tough situations.� Life Skills Training Program Botvin, G.J., Baker, E., Dusenbury, L., Botvin, E.M., Diaz, T. (1995). Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. Journal of the American Medical Association, 273, 1106-1112. Linda Dusenbury, NIDA Dissemination Conference (April, 2003), http://www.tanglewood.net/ (Under Conference Presentations) Program Description found at: http://www.lifeskillstraining.com/program.cfm �The LifeSkills program is designed for both: Elementary school students and Middle/junior high school students It has been evaluated and proven to be effective with: White middle-class students, Ethnic minority students (primarily African-American and Hispanic), Inner-city urban populations, Suburban populations, Rural populations Components The LifeSkills program consists of three major components that cover the critical domains found to promote drug use. Research has shown that students who develop skills in these three domains are far less likely to engage in a wide range of high-risk behaviors. The three components include: Drug Resistance Skills enable young people to recognize and challenge common misconceptions about tobacco, alcohol and other drug use. Through coaching and practice, they learn information and practical ATOD (Alcohol, Tobacco, and Other Drug use) resistance skills for dealing with peers and media pressure to engage in ATOD use. Personal Self-Management Skills teach students how to examine their self-image and its effects on behavior; set goals and keep track of personal progress; identify everyday decisions and how they may be influenced by others; analyze problem situations, and consider the consequences of each alternative solution before making decisions; reduce stress and anxiety, and look at personal challenges in a positive light. General Social Skills teach students the necessary skills to overcome shyness, communicate effectively and avoid misunderstandings, initiate and carry out conversations, handle social requests, utilize both verbal and nonverbal assertiveness skills to make or refuse requests, and recognize that they have choices other than aggression or passivity when faced with tough situations.�

13. Pentz MA, Trebow EA, Hansen WB, MacKinnon DP, Dwyer JH, Johnson CA, Flay BF, Daniels S, Cormack CC. Effects of program implementation on adolescent drug use behavior: The Midwestern Prevention Project (MPP). Evaluation Review, 14, 264-289, 1990. http://www.tanglewood.net/services/knowledgebase/81.htm Pentz MA, Trebow EA, Hansen WB, MacKinnon DP, Dwyer JH, Johnson CA, Flay BF, Daniels S, Cormack CC. Effects of program implementation on adolescent drug use behavior: The Midwestern Prevention Project (MPP). Evaluation Review, 14, 264-289, 1990. http://www.tanglewood.net/services/knowledgebase/81.htm

14. Why does Quality matter? Research has clearly linked quality of implementation with positive outcomes Higher fidelity is associated with better outcomes across a wide range of programs and practices (PATHS, MST, FFT, TND, LST and others) Fidelity enables us to attribute outcomes to the innovation/intervention, and provides information about program/practice feasibility

15. The reality�. Quality takes real effort because fidelity is not a naturally occurring phenomenon � adaptation (more accurately program drift) is the default Most adaptation is reactive rather than proactive thereby weakening rather than strengthening the likelihood of positive outcomes

16. Improving Quality Locally Good pre-implementation planning Improve practitioner knowledge of program/practice theory of change Build a sustainable infrastructure for monitoring implementation fidelity and quality What gets measured matters Build internal capacity AND desire

17. Why Monitor Implementation? To ensure that the program is implemented with quality and fidelity to the original design. To identify and correct implementation problems. To provide �lessons learned� for future implementation efforts. To identify and celebrate early successes.

18. Keys to Quality Implementation Support of site administrators Qualified implementers who support the program Ongoing planning meetings and community of practices A detailed implementation plan(who, what, where, when and how) Ongoing monitoring and technical assistance

19. We are Guilty of many errors and many faults, but our worst crime is abandoning the children, neglecting the fountain of life. Many of the things we need can wait. The child cannot. Right now is the time his bones are being formed, his blood is being developed.To him we cannot answer �Tomorrow.�His name is �Today.�Gabriela Mistral, Nobel Prize-winning Poet

20. References Community Anti-Drug Coalitions of America (2007). Sustainability Primer: Fostering Long Term Change to Create Drug Free Communities. Washington, DC: Office of National Drug Control Policy. Goodman, R. M., & Steckler, A. (1989). A model for the institutionalization of health promotion programs. Family and Community Health, 11, 63-78. th, 11, 63-78. Johnson, K., Hays, C., Center, H., & Daley, C. (2004). Building capacity and sustainable prevention innovations: A sustainability planning model. Evaluation and Program Planning, 27, 135-149. Mancini, J. A., & Marek, L. I. (2004). Sustaining community-based programs for families: Conceptualization and measurement. Family Relations, 53, 339-347. Marek, L., Mancini, J.A., Earthman, G. E., & Brock, D. (2003). Ongoing Community-Based Program Implementation, Successes, and Obstacles: The National Youth at Risk Program Sustainability Study�. Blacksburg, VA: Virginia Cooperative Extension. http://www.ext.vt.edu/pubs/family/350-804/350-804.html Marek, L. I., Mancini, J. A., & Brock, D. J. (1999). Continuity, success, and survival of community-based projects: The national youth at risk program sustainability study (Virginia Cooperative Extension Publication 350-801). Retrieved September 6, 2003, from http://www.ext.vt.edu/pubs/family/350-801/350-801.html Scheirer, M. (2005). Is Sustainability Possible? A Review and Commentary on Empirical Studies of Program Sustainability. Americal Journal of Evaluation, 26, 320-347. Shediac-Rizkallah, M. C., Scheirer, M. A., & Cassady, C. (1997, May). Sustainability of the Coordinated Breast Cancer Screening Program (Final report to the American Cancer Society). Baltimore: Johns Hopkins University School of Hygiene and Public Health. Small M. (2004). Sustainability planning. A presentation made at the annual PROSPER Statewide Meeting University Park: Prevention Research Center, The Pennsylvania State University.