AMCHP-CDC Skills Building Suite: Evidence-based Practice: Moving along the Continuum from Selection to Sustainability

1. AMCHP-CDC Skills Building Suite: Evidence-based Practice: Moving along the Continuum from Selection to Sustainability Friday, September 13, 2013 Starr Banks, MPH Kathi Wilson, PhD, MPH, CHES

2. Quick Overview How to Use Web Technology • Mute your line by using the mute function on your phone or by using *6 to mute/un-mute • Asking a Question • You can type your questions into the chat box (shown right) • Raise your hand. Using the icon at the top of your screen (example shown right) • Lastly active participation will make sure today’s webinar a success!

3. Evaluation and Sustainability of Your Evidence-Based Programs Starr Banks Kathi Wilson Community Guide Branch, Centers for Disease Control and Prevention Division of Epidemiology, Analysis, and Library Services (proposed)

4. Disclaimer Any views or opinions expressed by the speaker do not necessarily represent the views of the CDC, HHS, or any other entity of the United States government. Furthermore, the use of any product names, trade names, images, or commercial sources is for identification purposes only, and does not imply endorsement or government sanction by the U.S. Department of Health and Human Services. 4

5. Acknowledgements This training is brought to you by

6. Objectives • Name your most important outcomes. • Define different types of evaluation. • Match your evaluation methods with your objectives. • Identify components of sustainability. • Assess your program’s sustainability . • Recognize the importance of incorporating a sustainability plan for your evidence-based approach.

7. Where do We Stand? Establishing Goals & Objectives; Planning for Evaluation Community Assessment Finding Evidence Selection Adaptation Sustainability Implementation & Evaluation

8. Evaluation Planning

9. Why collect evaluation data? • Communicate findings to staff, partners and funders • Contribute to the evidence base. Create practice-based evidence • When findings indicate that objectives were not met, explore and address the reasons • When findings indicate objectives were met, celebrate!

10. Causes of Program Success or Failure

11. What outcomes are most important? • In designing an evaluation plan,find balance between • Desired outcomes • Feasible outcomes

12. Outcomes • Are the changes that result from your program • Are linked to either activities or objectives • Can be • Immediate • Short-term • Long term

13. Poll Question #1

14. General Evaluation Procedure

15. Objectives Contribution to Evaluation

16. Poll Question #2

17. Write SMART objectives • Specific: Boy and girls between the ages of 10-13 • Measureable : Decrease obesity rate in adolescents from 35% to 5% • Achievable: Make sure a 30% decrease is possible in the time fame given • Realistic: Adolescents in 3 middle schools in Oconee County, GA • Time-framed: In 2 years

18. Activities Achieve Objectives Activities are the specific things that you do to accomplish your objectives. • Link activities to objectives • Describe actions • Specify timing and intended population BOTTOM LINE: Each activity leads in an obvious way to an objective.

19. Evaluation Planning:Evaluation Phases and Types • Process evaluation • Are all activities being implemented as planned? • Are you reaching the intended audience? • Short-term outcome evaluation • Does behavior change? • Is a policy documented? Is it enforced? • Long-term outcome evaluation • Does health improve? • Do morbidity and mortality decrease?

20. Process Evaluation • Can find problems early • Includes an assessment of the staff, budget and how well the program is doing overall • Includes measurement of adoption, reach, participation, exposure, implementation, satisfaction, barriers and facilitators, and maintenance • Data collection methods may include: • databases • registration sheets • surveys • interviews Windsor et al., Evaluation of health promotion, health education, and disease prevention programs, 2003.

21. Short-term Outcome Evaluation • Assesses short-term effects on awareness, knowledge, attitudes, behavior, setting or environment • Can assess adoption or enforcement of a policy (including in schools, worksites, faith organizations, etc.) • Measures progress in meeting your objectives • Data collection methods may include: • surveys • interviews focus group • observation • program implementation logs

22. Poll Question #3

23. Evaluation Methods & Tools • Tracking logs: Keep an ongoing record of activities or outcomes • Existing data: Use data that has already been collected for another purpose. • Surveys or interviews: mailed, handed, or emailed to clients, patients, providers either one time or both before and after an activity. • Focus groups: Open ended questions asked of a small group of similar people to explore questions more deeply. • Observation: site visits to document extent to which activities are implemented as planned.

24. Assesses if the long-term goals were met Measures sustained program or strategy changes Measures longer term changes in rates of illness, death, or health status of your population Use common data collection methods Long-term Outcome Evaluation

25. Activity #1

26. Sharing Evaluation Results • Evaluation findings need to reach all stakeholders • In language that everyone understands • In time to make decisions • Program maintenance v. adaptation based on evaluation findings • Broader dissemination and application

27. Sample 1: Evaluation Worksheet

28. Sample 1: Evaluation Worksheet Instructions • Program Goal and Objectives column • Write a major objective of the program. • Overall Intervention column • Write at least one activity that will help you achieve your objective. • Evaluation Activities column • Look for possible evaluation activities. They should be based on the goals and objectives as well as planned intervention activities. • Evaluation Results column • Brainstorm about what you hope to learn from the evaluation. • Evidence of Activities and Quality column • Brainstorm about what kinds of materials will be needed. • For example, a focus group would require a question guide and consent forms. You might need instruction materials for the participants. Are they available? • Evidence of Results column • Think about what kind of results would show that an objective has been met.

29. Sample 2- Smoke-free Policy Evaluation Methods

30. Evaluation Resource List • CDC Framework for Program Evaluation • http://www.cdc.gov/eval/framework/index.htm • American Evaluation Association http://www.eval.org/ • CDC Demonstrating your Program’s Worthhttp://www.cdc.gov/ncipc/pub-res/demonstr.htm • Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) http://cancercontrol.cancer.gov/is/reaim/ • The Evaluation Center at Western Michigan University www.wmich.edu/evalctr • University of Wisconsin - Extension: Program Development and Evaluation http://www.uwex.edu/ces/pdande/index.html • W.K. Kellogg Foundation Evaluation Handbook http://www.wkkf.org

31. Glasgow, R.E., Vogt, T.M., & Boles, S.M. (1999) Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89, 1322-7. Audrey, S., Holliday, J., Parry-Langdon, N., & Campbell, R. (2006). Meeting the challenges of implementing process evaluation within randomized controlled trials: The example of ASSIST (A Stop Smoking in Schools Trial). Health Education Research, 21, 366-77. Beresford, S.A., Shannon, J., McLerran, D., & Thompson, B. (2000). Seattle 5-a-Day work-site project: Process evaluation. Health Education & Behavior, 27, 213-22. Grembowski, D. (2001). The practice of health program evaluation. Sage: Thousand Oaks, CA. Evaluation References

32. Sustaining your intervention

33. Where do We Stand? Establishing Goals & Objectives; Planning for Evaluation Community Assessment Finding Evidence Selection Adaptation Sustainability Implementation & Evaluation

34. Planning for Sustainability Sustainability is… … the existence of structures and processes which allow a program to leverage resources to most effectively implement evidence-based policies and activities. Sustainability capacity is… …the ability to maintain programming and its benefits over time. Sustainability requires… … knowledge of factors critical to program success. Luke D, Calhoun A, Herbers S. Building & Managing Sustainability in Public Health Programs. CDC Roundtable Presentation. Feb. 23, 2012

35. Why is planning for sustainability important? • WITHOUT planning for sustainability • Waste $ and other resources • Damage trust between program and community • Limit ability to succeed • WITH planning for sustainability • Lasting outcomes • Healthier communities

36. Sustainability Planning Resources http://ctpr.wustl.edu/sustainability/

37. Sustainability Tool Definitions • Program – refers to the set of formal organized activities that you want to sustain over time. • Organization – encompasses all the parent organizations or agencies in which the program is housed. • Community – refers to the stakeholders who may benefit from or who may guide the program.

38. Program Sustainability Assessment Tool • Purpose – • Assess program’s current capacity for sustainability across a number of factors • Identify strengths and challenges to sustainability • Guide action to improve sustainability as part of program planning

39. Program Sustainability Domains • Funding Stability – making long term plan on stable funding • Political support – internal and external political environments • Partnerships – connection between program and community • Organizational Capacity – resources needed • Program Adaptation – ability to adapt and improve • Program Evaluation – monitoring of processes and outcomes • Communications – strategic distribution of successes • Public Heath Impacts – program’s effect • Strategic Planning – process that defines program directions

40. Sustainability Activities - Examples • Seek additional funding • Maintain or secure a program champion • Continue partnerships developed or build new ones to share costs/implementation • Referrals for further service, if needed • Make intervention a part of organizational services • Surveillance of the impact of the PSE(e.g., tracking of smoking rates, BMI of children)

41. Poll Question #4

42. Homework Challenge • Complete Sustainability Plan - Center TRT

43. Questions & Discussion

44. Thank you! Kathi Wilson Starr Banks Community Guide Branch, Centers for Disease Control and Prevention Kathi Wilson kwilson@cdc.gov Starr Banks smbanks@cdc.gov Division of Epidemiology, Analysis, and Library Services (proposed) Center for Surveillance, Epidemiology, and Laboratory Services (proposed)