Sustainability Sessions: Evaluating Impact & Planning for Success

1. ECHO Sustainability Sessions Reston Hartsell, MS-MPH Synar/Tobacco Prevention Coordinator

2. Session Objectives Session 1: Review sustainability Discuss elements of process, impact, and outcome evaluation Share informatics tools and resources Cover the basics of visual evaluation reports Session 2: What TLDR means for your evaluation reports Evaluation report document development

3. How do we define sustainability? The Center for Disease Control and Prevention (CDC)’s Healthy Communities Program Sustainability Planning Guide defines sustainability as “a community’s ongoing capacity and resolve to work together to establish, advance, and maintain effective strategies that continuously improve health and quality of life for all.” (p. 8) Centers for Disease Control and Prevention’s Healthy Communities Program Sustainability Planning Guide Team (2009). A Sustainability Planning Guide for Healthy Communities. Atlanta (GA): Centers for Disease Control and Prevention: National Center for Chronic Disease Prevention and Health Promotion.

4. Why develop a sustainability plan? Gives you the opportunity to obtain buy-in from decision-makers and key stakeholders1 You and your team can identify short- and long-term policy strategies1 You can strategically plan how to make the most out of human, financial, and other resources to achieve your short- and long-term goals1 Additionally, you can organize the data that you collect (evaluate your findings, justify your program(s), create criteria grids, etc.)1 Centers for Disease Control and Prevention’s Healthy Communities Program Sustainability Planning Guide Team (2009). A Sustainability Planning Guide for Healthy Communities. Atlanta (GA): Centers for Disease Control and Prevention: National Center for Chronic Disease Prevention and Health Promotion.

5. What are key elements of successful sustainability planning? Sustainable public health programs must have: Buy-in and support from key decision-makers1 Sufficient leadership, funding, channels of communications. 1 Procedures in place to monitor policy (e.g., city ordinances) results through enforcement and compliance, and to modify strategies accordingly. 1 Centers for Disease Control and Prevention’s Healthy Communities Program Sustainability Planning Guide Team (2009). A Sustainability Planning Guide for Healthy Communities. Atlanta (GA): Centers for Disease Control and Prevention: National Center for Chronic Disease Prevention and Health Promotion.

6. What are key elements of successful sustainability planning? Sustainable coalition efforts must have: A long-term plan for ensuring the viability of the coalition1 Develop a diverse funding portfolio, collaborative leadership, and marketing/branding strategies. 1 Ensure that all community stakeholders are ready to respond to a changing environment1 Centers for Disease Control and Prevention’s Healthy Communities Program Sustainability Planning Guide Team (2009). A Sustainability Planning Guide for Healthy Communities. Atlanta (GA): Centers for Disease Control and Prevention: National Center for Chronic Disease Prevention and Health Promotion.

8. Why do we evaluate public health programs? • Research tries to provewhile evaluation tries to improve2 • “Effective program evaluation is a systematic way to improve and account for public health actions by involving procedures that are…”3 • “Useful” 3 • “Feasible” 3 • “Ethical” 3 • “Accurate” 3 Patton M.Q. Qualitative Evaluation and Research Methods. 2 ed. Sage; Newbury Park, CA, USA: 1990. p. 532. [Google Scholar] Salabarría-Peña, Y, Apt, B.S., Walsh, C.M. Practical Use of Program Evaluation among Sexually Transmitted Disease (STD) Programs, Atlanta (GA): Centers for Disease Control and Prevention; 2007.

9. What framework can we use to evaluate public health programs? Framework Step 1 Checklist | Program Evaluation | CDC. (2018). Retrieved from https://www.cdc.gov/eval/steps/step1/index.htm

10. What are the steps to engage stakeholders? • Assuming that you already have a narrowed list of stakeholders, you can begin engaging them with a series of questions about your program(s), such as: • “What do you see as the main outcomes of the program?” 4 • “What do you see as the main activities of the program?”4 • “Which of the activities and outcomes are most important to you? That is, to retain your involvement and support, which activities must be effectively implemented and/or which outcomes achieved?”4 • “What do you see as the most important evaluation questions at this time?”4 • “Do you have preferences regarding the types of data that are collected (e.g., quantitative, qualitative)?”4 • “What resources (e.g., time, funds, evaluation expertise, access to respondents, and access to policymakers) might you contribute to this evaluation effort?”4 • “In what parts or steps of this evaluation would you want to be involved? All or just some specific ones? (How) will you use the results of this evaluation?”4 Framework Step 1 Checklist | Program Evaluation | CDC. (2018). Retrieved from https://www.cdc.gov/eval/steps/step1/index.htm

11. How do I describe my program? • A logic model can help you describe your program5 • Logic models typically have various formats and level of specificity. The following are common key terms used (although not all logic models may use this language): • Inputs: “The resources needed to implement the activities” 5 • Activities: “What the program and its staff do with those resources” 5 • Outputs: “Tangible products, capacities, or deliverables that result from the activities” 5 • Outcomes: “Changes that occur in other people or conditions because of the activities and outputs” 5 • Impacts: “The most distal/long-term outcomes” 5 • Moderators: “Contextual factors that are out of control of the program but may help or hinder achievement of the outcomes” 5 Framework Step 2 Checklist | Program Evaluation | CDC. (2018). Retrieved from https://www.cdc.gov/eval/steps/step2/index.htm

12. Sample Logic Model – Outcome Approach https://www.bttop.org/sites/default/files/public/W.K.%20Kellogg%20LogicModel.pdf W.K. Kellogg Foundation (2004). W.K. Kellogg Foundation Logic Model Development Guide. Battle Creek (MI): Centers for Disease Control and Prevention: National Center for Chronic Disease Prevention and Health Promotion. Retrieved from https://www.bttop.org/sites/default/files/public/W.K.%20Kellogg%20LogicModel.pdf

13. Reading Logic Models https://www.bttop.org/sites/default/files/public/W.K.%20Kellogg%20LogicModel.pdf W.K. Kellogg Foundation (2004). W.K. Kellogg Foundation Logic Model Development Guide. Battle Creek (MI): Centers for Disease Control and Prevention: National Center for Chronic Disease Prevention and Health Promotion. Retrieved from https://www.bttop.org/sites/default/files/public/W.K.%20Kellogg%20LogicModel.pdf

14. How do I focus my evaluation? • Your evaluation’s focus is inherently linked to your logic model. Additionally, your stakeholders influence the focus of your evaluation. • Clusters to focus your evaluation: • “Utility:Who needs the information from this evaluation and how will they use it?” 3,7 • “Feasibility: How much money, time, skill, and effort can be devoted to this evaluation?”3,7 • “Propriety: Who needs to be involved in the evaluation to be ethical?” 3,7 • “Accuracy: What design will lead to accurate information?”3,7 Framework Step 3 Checklist | Program Evaluation | CDC. (2018). Retrieved from https://www.cdc.gov/eval/steps/step3/index.htm

15. Types of Evaluation • Formative evaluation“ensures that a program or program activity is feasible, appropriate, and acceptable before it is fully implemented. It is usually conducted when a new program or activity is being developed or when an existing one is being adapted or modified.”8 • Process/implementation “evaluation determines whether program activities have been implemented as intended.” 8 • Outcome/effectiveness “evaluation measures program effects in the target population by assessing the progress in the outcomes or outcome objectives that the program is to achieve.” 8 • Impact evaluation“assesses program effectiveness in achieving its ultimate goals.” 8 Salabarría-Peña, Y, Apt, B.S., Walsh, C.M. Practical Use of Program Evaluation among SexuallyTransmitted Disease (STD) Programs, Atlanta (GA): Centers for Disease Control and Prevention; 2007.

16. Process Evaluation • Process Evaluation determines whether your program activities were implemented as intended with fidelity, which resulted in desired outputs. This type of evaluation can be done throughout your program’s start and stop date. You can begin by examining the activities and outputs from your logic model. 8 • “Results of a process evaluation will strengthen your ability to report on your program and use information to improve future activities. It allows you to track program information related to Who, What, When and Where questions:” • “To whom did you direct program efforts?” 8 • “What has your program done?” 8 • “When did your program activities take place?” 8 • “Where did your program activities take place?” 8 • “What are the barriers/facilitators to implementation of program activities?”8 Salabarría-Peña, Y, Apt, B.S., Walsh, C.M. Practical Use of Program Evaluation among SexuallyTransmitted Disease (STD) Programs, Atlanta (GA): Centers for Disease Control and Prevention; 2007.

17. Outcome Evaluation • Outcome evaluation examines how your program effects the target population. You can design an outcome evaluation by starting with a review of your logic model’s outcomes.8 • “Some questions you may address with an outcome evaluation include:” • “Were medical providers who received intensive STD training more likely to effectively counsel, screen and treat patients than those who did not?” 8 • “Did the implementation of STD counseling in community-based organizations result in changes in knowledge, attitudes, and skills among the members of the target population?” 8 • “Did the program have any unintended (beneficial or adverse) effects on the target population(s)?” 8 • “Do the benefits of the STD activity justify a continued allocation of resources?” 8 Salabarría-Peña, Y, Apt, B.S., Walsh, C.M. Practical Use of Program Evaluation among SexuallyTransmitted Disease (STD) Programs, Atlanta (GA): Centers for Disease Control and Prevention; 2007.

18. Impact Evaluation • Impact evaluation examines the program’s overall effect on participants. 8 Salabarría-Peña, Y, Apt, B.S., Walsh, C.M. Practical Use of Program Evaluation among SexuallyTransmitted Disease (STD) Programs, Atlanta (GA): Centers for Disease Control and Prevention; 2007.

19. Reporting Visually

20. Data Visualization • Data visualization refers to the ways that data can be manipulated visually. • Examples include: • Maps, bar charts, timelines, artistic renderings, etc. • Data visualization usually includes information and scientific visualization. • Information Visualization – covers graphs and charts, as well as other visual metaphors • Scientific Visualization – involves visualizing scientific data with real-world objects (via spatial properties) • Examples include: • Taking MRI scans and creating 3D volumes (Zoss, 2017, “About Data Visualization”) Zoss, A. (2017, August 9). About Data Visualization – Data Visualization. Retrieved from https://guides.library.duke.edu/datavis

21. Visualization Types • Types of data: • 1-Dimensional/Linear (Ex. Organized lists of data) • 2-Dimensional/Planar (Ex. Geospatial maps) • 3-Dimensional/Volumetric (Ex. 3D computer models, volume renderings, and computer simulations) • Temporal (Ex. Timelines, time series, time plots, scatter plots, Gantt charts, etc.) • Multi-dimensional (Ex. Pie charts, histograms, Wordles, unordered bubble charts, bar charts, scatter plots, line charts, box and whisker plots, etc.) • Tree/Hierarchical (Ex. Tree map) • Network (Ex. Matrix, node-link diagram) (Zoss, 2017, “About Data Visualization”)

22. Why care about data visualization? • What makes data visualizations so important is how efficient visuals are in relaying information. • Our visual cortex is extremely efficient at processing information quickly and picking up on trends, patterns, and outliers. • With this in mind, we want to process our data and be able to tailor a data-driven narrative to our target audience. (O’Reilly Media, 2011, “Designing Data Visuals”)

23. Visualization Types • Visualizations are more general, whereas infographics serve a specific purpose. • Infographics use statistics and visuals to depict a narrative. • They are a tool that presents information visually in a creative way. Point of emphasis: • Health informatics is a specific term for healthcare purposes. The US National Library of Medicine defines health informatics as “… the interdisciplinary study of the design, development, adoption, and application of IT-based innovations in healthcare services delivery, management, and planning.”10 (Kosara, 2011, The Difference Between Infographics and Visualization) Kosara, R. (2011, December 23). The Difference Between Infographics and Visualization. [Web log comment]. Retrieved from https://eagereyes.org/blog/2010/the-difference-between-infographics-and-visualization

24. Periodic Table of Visualization

25. Visualization Tools for Evaluation Purposes • List of evaluation resources: • https://www.cdc.gov/eval • http://ctb.ku.edu/ • http://www.gse.harvard.edu/hfrp • http://innonet.org/ • http://www.uwex.edu/ces/pdande/index.html • List of infographic tools: • PowerPoint • Excel • Piktochart • Visme • Easely • Visualize.me • Canva • Infogr.am • Venngage • Example: https://create.piktochart.com/dashboard

26. Infographic Evaluation Report • Always keep in mind these three inputs: • 1) Reader • 2) Designer • 3) The Data • You want to introduce the topic, then the problem and its complexity, followed by the central argument, and finish with the conclusion and take home message.

27. Infographic Evaluation Report – Design Layout

28. Evaluation Report Elements • This document presents the following: • Findings • Conclusions • Recommendations from the evaluation • Future program improvement and decision making If we boil it all down, the evaluation final report should describe the “What,” the “How,” and the “Why It Matters” questions about your program.11 Developing an effective evaluation report: Setting the course for effective program evaluation. Atlanta, Georgia: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health, Division of Nutrition, Physical Activity and Obesity, 2013. Retrieved from https://www.cdc.gov/eval/materials/Developing-An-Effective-Evaluation-Report_TAG508.pdf

29. Sustainability Session Part II

30. Infographic Evaluation Report • Always keep in mind these three inputs: • 1) Reader • 2) Designer • 3) The Data • You want to introduce the topic, then the problem and its complexity, followed by the central argument, and finish with the conclusion and take home message.

31. Infographic Evaluation Report – Design Layout

32. Evaluation Report Elements • This document presents the following: • Findings • Conclusions • Recommendations from the evaluation • Future program improvement and decision making If we boil it all down, the evaluation final report should describe the “What,” the “How,” and the “Why It Matters” questions about your program.11 Developing an effective evaluation report: Setting the course for effective program evaluation. Atlanta, Georgia: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health, Division of Nutrition, Physical Activity and Obesity, 2013. Retrieved from https://www.cdc.gov/eval/materials/Developing-An-Effective-Evaluation-Report_TAG508.pdf

33. What is TLDR? • Too Long Didn’t Read (TLDR) 1-3-25 Model “One-page handout, three-page executive summary, and a 25 page report”12 How to avoid #TLDR: • “Only visualize the important stuff” 12 • “Findings first (start with the bottom line)” 12 • “Work backwards from there (such that your methodology is in the appendix or a sidebar/sidebox)” 12 • “Use graphics to signal the start of new sections” 12 • “Use icons for faster navigation” 12 Evergreen, S. (2018, September 19). Handouts. Retrieved from https://stephanieevergreen.com/handouts/

34. Why are one-page evaluation reports ideal to accompany your longer reports? • Summarizes key information • Audience specific • Engaging and accessible http://www.evalu-ate.org/resources/one-pagers/#grids • Data snapshots are a perfect way to summarize key points and elaborate on how key indicators changed over the course of the project.

35. Example 1 – My Attempt to Avoid #TLDR

36. Example 2 – My Attempt to Avoid #TLDR

37. Example Data Visualization Checklist • Infographic Report Evergreen, S. (2018, September 19). Handouts. Retrieved from https://stephanieevergreen.com/handouts/

38. Creating a One-Pager Steps 1 • Identify the audience • Identify the purpose • Prioritize the information • Choose a grid • Draft the layout • Create an intentional visual path • Create a purposeful hierarchy • Use white space • Get feedback • Triple check consistency http://www.evalu-ate.org/wp-content/uploads/2018/09/OnePager-Worksheet.pdf 2 http://www.evalu-ate.org/resources/one-pagers/#grids/3HDmM5mWTyY

39. Demonstration Data One-Page Data Snapshot Document ECHO – MIS Notes

40. PowerPoint Steps • Click on slide size button -> select “Custom Slide Size” • Select Letter Paper (8.5 X 11 IN) -> Under Orientation -> Select Portrait • New window will appear -> For a document already created, select “Ensure Fit.” If the document is blank, select either. • Document is ready for the GRID. Design to the edge, that way margins do not show. • Insert a grid by clicking on Pictures under the Insert tab. • Select the GRID size, and insert as image. Drag to the edges of margins.

41. References • Centers for Disease Control and Prevention’s Healthy Communities Program Sustainability Planning Guide Team (2009). A Sustainability Planning Guide for Healthy Communities. Atlanta (GA): Centers for Disease Control and Prevention: National Center for Chronic Disease Prevention and Health Promotion. • Salabarría-Peña, Y, Apt, B.S., Walsh, C.M. Practical Use of Program Evaluation among Sexually • Transmitted Disease (STD) Programs, Atlanta (GA): Centers for Disease Control and Prevention; 2007. • Framework Step 1 Checklist | Program Evaluation | CDC. (2018). Retrieved from https://www.cdc.gov/eval/steps/step1/index.htm • Framework Step 2 Checklist | Program Evaluation | CDC. (2018). Retrieved from https://www.cdc.gov/eval/steps/step2/index.htm • Framework Step 3 Checklist | Program Evaluation | CDC. (2018). Retrieved from https://www.cdc.gov/eval/steps/step3/index.htm • Types of Evaluation in Health Promotion and Disease Prevention Programs - RHIhub Toolkit. (n.d.). Retrieved from https://www.ruralhealthinfo.org/toolkits/health-promotion/4/types-of-evaluation • W.K. Kellogg Foundation (2004). W.K. Kellogg Foundation Logic Model Development Guide. Battle Creek (MI): Centers for Disease Control and Prevention: National Center for Chronic Disease Prevention and Health Promotion. Retrieved from https://www.bttop.org/sites/default/files/public/W.K.%20Kellogg%20LogicModel.pdf • Developing an effective evaluation report: Setting the course for effective program evaluation. Atlanta, Georgia: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health, Division of Nutrition, Physical Activity and Obesity, 2013. Retrieved from https://www.cdc.gov/eval/materials/Developing-An-Effective-Evaluation-Report_TAG508.pdf • Zoss, A. (2017, August 9). About Data Visualization – Data Visualization. Retrieved from https://guides.library.duke.edu/datavis • Zoss, A. (2017, August 9). Visualization Types – Data Visualization. Retrieved from https://guides.library.duke.edu/datavis/vis_types • Kosara, R. (2011, December 23). The Difference Between Infographics and Visualization. [Web log comment]. Retrieved from https://eagereyes.org/blog/2010/the-difference-between-infographics-and-visualization • Canva Learn. (2017). Infographic Design [Web log comment]. Retrieved from https://designschool.canva.com/how-to-design-infographics/ • Evergreen, S. (2018, September 19). Handouts. Retrieved from https://stephanieevergreen.com/handouts/

42. Questions?

43. 1801 Main Street, 4th Floor • Columbia, South Carolina 29201 telephone: 803-896-5555 • fax: 803-896-5558 www.daodas.sc.gov