Evaluation Essentials for Health Promotion in Higher Education: A Primer

1. Evaluation Essentials for Health Promotion in Higher Education: A Primer Gina Baral Abrams, MPH, LSW, CHES Amy Melichar, MEd, CHES American College Health Association Annual Meeting San Francisco, CA May 28, 2009

2. Learning Objectives • Describe the uses and purposes of program evaluation in health promotion. • List conceptual approaches to evaluation. • List the key qualities of measurable evaluation questions. • List qualitative and quantitative data collection methods. • Describe key qualities of an effective evaluation report.

3. Evaluation Defined “Examining the worth of a program, usually measuring it against a set of predetermined objectives or a standard of acceptability.” Green & Kreuter, 1999.

4. Uses and Purposes of Program Evaluation • Determine the rate and level of attainment of program objectives. • Assess the strengths and weaknesses of a program. • Help make decisions. • Monitor standards of performance. • Establish quality assurance and control mechanisms. • Determine the generalizability of an overall program or program elements to other populations. • Meet the demand for public or fiscal accountability. • Improve professional staff skill in program planning, implementation, and evaluation activities. • Promote positive public relations and community awareness. • Fulfill grant or contract requirements. • Contribute to scientific knowledge. • Indentify hypotheses for future study.

5. Conceptual Approaches to Evaluation Objectives-based evaluation Determine extent to which objectives are achieved. Management-oriented evaluation Provide information to aid in making decisions. Consumer-oriented evaluation Provide information about products to aid decisions about purchases or adoptions. Expertise-oriented evaluation Provide professional judgments of quality. Participant-oriented evaluation Understanding the complexities of programmatic activity, responding to an audience’s requirements for information.

6. The Evaluator’s Role Objectivity is key when strategically selecting the evaluator. Possible evaluators include: External evaluator Internal evaluator Internal evaluator with external consultant

7. Basic Types of Evaluation Formative evaluation To provide information for program improvement. What is working? What needs to be improved? How can it be improved? Summative evaluation To provide information to make decisions about the program’s future or adoption. What results occur? With whom? Under what conditions? With what training? At what cost?

8. Levels of Evaluation in Health Promotion Process evaluation: to determine feasibility. “Implementation evaluation” “Quality assurance review” What was delivered and how it was delivered? Did program follow protocol? Applies non-experimental designs. Impact evaluation: to provide information to make decisions about the program’s future or adoption. What were short-term (immediate-24 months) effects of the program on target behaviors and their predisposing, enabling and reinforcing antecedents or on environmental factors? Applies quasi-experimental and experimental designs. Outcome evaluation: to assess changes in overall health status and quality of life indicators. What were long-term changes (1-10 years)? Applies quasi-experimental and experimental designs.

9. Types of Evaluation Designs Non-experimental Does not include random assignment or a control group. Asserts little or no control over confounding factors. Quasi-experimental Includes experimental and control group. Does not include random assignment. Offers some control over confounding factors. Experimental Experimental and control group. Includes random assignment. Asserts greatest degree over confounding factors. Evidence is most interpretable, defensible, and definitive.

10. Evaluation Plan: Key Steps • Gather background information • Identify stakeholders and audiences • Budget for evaluation • Develop evaluation question(s) • Determine sources of information required • Select data collection methods • Conduct evaluation (collect, analyze and interpret data) • Communicate findings -- provide final report

11. Step 1: Gather Background Information • Why is the program needed?  • General background, research on other programs and approaches. • What are the goals and objectives of the program? • What is the program context? • What are the program activities?

12. Step 1: Case Study Example Required orientation program for all first-year students. One-hour workshop/discussion. Delivered by residential advisors in small groups. Conducted concurrently in residential setting. Resident advisors trained in advance to lead the workshops. Program goals: Upon completion of the program, participants will understand the role of drinking motives in decision-making concerning alcohol. Upon completion of the program, participants will be aware of the role of social norms in decision-making concerning alcohol. Upon completion of the program, participants will be able to identify “decision-making” as part of their own drinking behaviors.

13. Step 2: Identify Stakeholders and Audiences Who are the stakeholders and audiences? What are their roles? Stakeholder: anyone who has a stake in the program to be evaluated or the evaluation’s results. Audiences: groups which have an interest in the evaluation and receive its results. Consider stakeholder and audience roles: To make policy To make operational decisions To provide input to evaluation To react to For interest only

14. Step 2: Case Study Example

15. Step 3: Budget for Evaluation What resources are available to dedicate to the evaluation? Estimate 7 – 10% of overall project budget for evaluation. Components may include, but are not limited to: Staff salary with benefits Consultants Travel Communications Printing and duplication Program materials Supplies and equipment Food Incentives Components may be paid in-kind, with operational dollars, external funding, etc. List all regardless of funding source.

16. Step 3: Case Study Example

17. Step 4: Develop Evaluation Questions • What questions need to be answered? • Seek input from stakeholders and audiences. • Consider process, impact and outcome evaluation needs. • Divergent phase: • Create a laundry list of potentially important questions • Convergent phase: • Select most critical questions from laundry list

18. Step 4: Case Study Example Process Evaluation Question: To what fidelity was the program implemented? Does the content of the workshop cover the intended objectives? Are resident advisors sufficiently prepared to lead the workshops? Do first-year students participate in the workshops in the intended manner? Impact Evaluation Question: Did the students who participated in the workshop achieve the desired results? Do participants understand the role of drinking motives in decision-making concerning alcohol? Do participants understand the role of social norms in decision-making concerning alcohol? Do participants identify “decision-making” as part of their own drinking behaviors?

19. Step 5: Determine Sources of Information What information do you need in order to answer the evaluation questions? Adapted from Green & Kreuter, 1999. Process Impact Outcome Health promotion program Predisposing, enabling and reinforcing factors Protective behavior or environment Health Social benefit Peer review, quality control, accreditation, audit, certification, satisfaction Changes in knowledge, attitudes, beliefs, skills, resources, social support, policy Changes in frequency, distribution, timing of behavior, or quality of the environment Changes in mortality, morbidity, disability or risk factors Changes in quality of life

20. Step 6: Select Data Collection Methods What data will you collect and how will you collect them? Strategies may include, but are not limited to: Tests and assessments Written questionnaires Policy audit Interviews with key participants Observations of behavior or environmental conditions Self-report logs or diaries Focus groups Community-level indicators of impact Service utilization rates Case studies Document review

21. Steps 5 & 6: Case Study Example Determine information required and source/data collection method for each question. Selected questions appear below. Note that there would be many more data collection methods if all questions were presented.

22. Step 7: Conduct Evaluation Plan

23. Step 8: Communicate Findings Components of the final report may include: Front matter (front cover, etc.) Executive summary Program background/evaluation description Present the results Discussion, conclusion, recommendations Report in ways that encourage follow-through by stakeholders, to increase the likelihood that the evaluation will be used. Re-visit list of stakeholders and audiences to determine how to communicate findings and to whom.

24. References Fitzpatrick, JL, Sanders, JR & Worthen, BR. (2004). Program evaluation: Alternative approaches and practical guidelines, 3rd ed. Boston, MA: Pearson. Green, LW & Kreuter, MW. (1999). Health promotion planning: An educational and ecological approach, 3rd ed. Mountain View, CA: Mayfield Publishing Company. McDermott, R., & Sarvela, P. (1999). Health education evaluation and measurement, 2nd ed. The McGraw-Hill Companies. Modeste, NN & Tamayose, TS. (2004). Dictionary of public health promotion and education terms and concepts, 2nd ed. San Francisco, CA: Jossey-Bass. WK Kellogg Foundation. (1998). The W.K. Kellogg evaluation handbook. Available at www.wkkf.org. Windsor, R., Baranowski, T., Clark, N., & Cutter, G., (1994). Evaluation of health promotion, health education, and disease prevention programs, 2nd ed. Mountain View, CA: Mayfield Publishing Company.

25. Questions? • Gina Baral Abrams, MPH, LSW, CHES Princeton University gbaral@princeton.edu • Amy Melichar, MEd, CHES Marquette University amy.melichar@marquette.edu