Chapter 8 – evaluating health promotion efforts

1. HLED 411 – Worksite Health Promotion Chapter 8 – evaluating health promotion efforts

3. Why evaluate?

4. To obtain feedback so that you can improve your programming efforts To demonstrate the value of your program To measure change To secure continued funding To establish accountability To compare the efficacy of different interventions Why evaluate?

5. Built into the planning process and into the budget. Process measures - answer many questions about the basic operation of the program. Can be used to modify the program to enhance participation and participant satisfaction. Outcome measures - gauge the extent to which specific program goals have been achieved. Helps to secure continued management support for the program or points to the need for changes. Evaluating whp programs

6. Process Measures are: Participation rates (Number of staff enrolled and participating). Web site hits Observation or counts (ex. track number walking at noon) Participant satisfaction (via survey, focus groups, etc.) Policy or environmental changes/tracking (compare Worksite Wellness Assessment Checklist at 1 year) Sample Evaluation Measures

7. Outcome Evaluation is more difficult & takes longer to show up in your data. Pre/Post test surveys–Measure changes in attitude, knowledge and current wellness habits over a specified time period Quizzes Vending items being chosen (arrange with vendor to track selections) Cafeteria menu options Health Indicators/ reduced risk factors. Comparison of company aggregate screening measures such as blood pressure, cholesterol, body weight, BMI, etc. before and after a specified program or campaign. Corporate costs and return on investment. Sample Evaluation Measures

8. Merit (i.e., quality), worth (i.e., cost-effectiveness), and significance (i.e., importance) What will be evaluated? (That is, what is the program and in what context does it exist?) What aspects of the program will be considered when judging program performance? What standards (i.e., type or level of performance) must be reached for the program to be considered successful? What evidence will be used to indicate how the program has performed? What conclusions regarding program performance are justified by comparing the available evidence to the selected standards? Purpose of program assessment

9. Framework for program assessment • CDC Evaluation Working Group - http://www.cdc.gov/eval/index.htm

10. Step 1: Engage stakeholders. Step 2: Describe the program. Step 3: Focus the evaluation design. Step 4: Gather credible evidence. Step 5: Justify conclusions. Step 6: Ensure use and share lessons learned. Framework for program evaluation

11. Fostering input, participation, and power-sharing among those persons who have an investment in the conduct of the evaluation and the findings; it is especially important to engage primary users of the evaluation. • Helps increase chances that the evaluation will be useful; can improve the evaluation’s credibility, clarify roles and responsibilities, enhance cultural competence, help protect human subjects, and avoid real or perceived conflicts of interest. • Those involved in program operations • Those served or affected by the program • Primary users of the evaluation Engage stakeholders

12. Characterizing the need (or set of needs) addressed by the program; Listing specific expectations as goals, objectives, and criteria for success; Clarifying why program activities are believed to lead to expected changes; Drawing an explicit logic model to illustrate relationships between program elements and expected changes; Assessing the program’s maturity or stage of development; Analyzing the context within which the program operates; Considering how the program is linked to other ongoing efforts; and Avoiding creation of an overly precise description for a program that is under development. Describe the program

13. Meeting with stakeholders to clarify the real intent or purpose of the evaluation Learning which persons are in a position to actually use the findings, then orienting the plan to meet their needs; Understanding how the evaluation results are to be used; Writing explicit evaluation questions to be answered; Describing practical methods for sampling, data collection, data analysis, interpretation, and judgment; Preparing a written protocol or agreement that summarizes the evaluation procedures, with clear roles and responsibilities for all stakeholders; and Revising parts or all of the evaluation plan when critical circumstances change. Focusing the evaluation design

14. Indicators - criteria that will be used to judge the program Sources - persons, documents, or observations that provide information for the inquiry. Quality - the appropriateness and integrity of information used in an evaluation. Logistics - encompasses the methods, timing, and physical infrastructure for gathering and handling evidence. Gather credible evidence

15. Standards - reflect the values held by stakeholders and provide the basis for forming judgments concerning program performance. Analysis and synthesis - methods for examining and summarizing an evaluation’s findings. Interpretation - the effort of figuring out what the findings mean and is part of the overall effort to make sense of the evidence gathered in an evaluation. Judgment - statements concerning the merit, worth, or significance of the program. Recommendations - actions for consideration resulting from the evaluation. Justify Conclusions

16. Design - refers to how the evaluation’s questions, methods, and overall processes are constructed. Preparation - refers to the steps taken to rehearse eventual use of the evaluation findings. Feedback - the communication that occurs among all parties to the evaluation. Follow-up - refers to the technical and emotional support that users need during the evaluation and after they receive evaluation findings. Dissemination - the process of communicating either the procedures or the lessons learned from an evaluation to relevant audiences in a timely, unbiased, and consistent fashion. Ensure Use and Share Lessons Learned

17. 8 primary variables which make excellent evaluation targets: • Participation • Participant Satisfaction • Improvements in Knowledge, Attitudes and • Behaviors • Changes in Biometric Measures • Risk Factors • Physical Environment and Corporate Culture • Productivity • Return On Investment Benchmark #7: Carefully Evaluating Outcomes (Welcoa)

18. Although a basic measure, participation is an excellent barometer of how the wellness program is functioning. • Participation can have a variety of meanings—all of which have some level of basic validity. • Any employee who has participated in a program at least once during the course of the year. • how many people take part in programs more regularly. Participation

19. Specifically, participant satisfaction can be measured through the administration of a simple survey at the completion of a program. Although, high participant satisfaction ratings do not necessarily mean that your program will achieve return on investment, it does indicate that participants feel good about the message that you are communicating. Participant Satisfaction

20. With the advent of technology, measuring changes in knowledge, attitudes, and behaviors is becoming much more straightforward. As health promotion programs continue to gain popularity, more and more standardized instruments have become available to help practitioners measure these important variables. Improvements in Knowledge, Attitudes,Behaviors

21. Biometric measures include such things as cholesterol levels, blood pressure, and BMI (body mass index). When programs begin evaluating changes in biometric measures, they are moving into a more elite level of program documentation. Changes in biometric measures are a highly valued form of evaluative feedback, but careful design of programs and interventions is essential in order to systematically gather this type of information. Changes In Biometric Measures

22. Risk factors • Identify individuals who are at-risk (generally defined as four or more risk factors) and intervene to make sure that the individuals don’t progress further along the disease continuum. • Make sure that low-risk employees (generally defined as zero to one risk factor) don’t migrate from low-risk status toward the at-risk category.

23. Many organizations have begun incorporating evaluation targets that concentrate on measuring and detecting changes in the physical working environment and the culture of the corporation. Measurements of corporate culture and the physical environment are important factors that should be monitored and incorporated into any results oriented wellness program. Physical Environment and Corporate Culture

24. Measuring the impact of wellness initiatives on key productivity indicators such as absenteeism, turnover, morale, etc. This area is a relatively new development in the arena of corporate health promotion initiatives. There are excellent resources and instruments available through such groups as the Institute for Health and Productivity Management (IHPM). Productivity

25. Often viewed as the gold standard of evaluation efforts, return on investment is quickly catching on as an excellent measure by which corporate wellness programs can be evaluated. Most often, this analysis requires outside investment and expert consultation. Return On Investment

26. Program registration sheets Participant satisfaction surveys—mid and post Self-reported behavior surveys HRA results, biometric testing, screening results Productivity questionnaires Medical claims comparison Evaluation methods

27. Ongoing - participation should be monitored and tracked using registration protocols and participant satisfaction should be captured using paper and pencil and/or electronic instruments. Annual - personal health assessments/health risk appraisals should be conducted at least once every 12 months. Develops longitudinal trends that can be monitored over time. Bi-annually - modifiable medical claims analysis and an organizational health and productivity audit should be evaluated every 24 months since they represent much larger data sets that don’t change that frequently and that dramatically from one year to the next. When to evaluate

28. Keep It Simple Use Pictures Tell Stories Never Have More Than One Main Point Invite Feedback Reporting results

29. http://www.youtube.com/watch?v=N-HI4zdfqG0 http://www.youtube.com/watch?v=2wIEwfslQDY http://www.youtube.com/watch?v=jpPh_Tn_R0c http://www.youtube.com/watch?v=WKEwTp78CAo Video clips – review of topics