How to Evaluate Your Health Literacy Project

1. How to Evaluate Your Health Literacy Project Jill Lucht, MS Project Director, Center for Health Policy luchtj@missouri.edu 573-882-2300 Presentation developed with Elizabeth Pafford, Institute of Public Policy, University of Missouri 573.882.5835 pafforde@missouri.edu

2. Everyday example H E A D A C H E Getpills Takepills Feel better OUTPUTS INPUTS OUTCOMES Situation University of Wisconsin-Extension, Program Development and Evaluation

3. Program Logic Model

4. Inputs • Staff • Volunteers • Time • Money • Research base • Materials • Equipment • Technology • Partners Human and material resources that support the program

5. Outputs • Conduct trainings, workshops, meetings • Deliver services • Develop products, curriculum, resources • Partner/Collaborate • Reach target population • Target population satisfaction Direct products or activities- what you will do to meet your objectives

6. Outcomes Short term: Change in… • Awareness • Knowledge • Attitudes • Beliefs • Skills • Opinions • Aspirations • Motivations Intermediate: Change in… • Action • Behavior • Practice • Policies Changes or benefits as a result of outputs

7. Outcomes Long Term (Impact): Change in… • Conditions • Environment • Institutionalized policy Changes or benefits as a result of outputs

8. Assumptions Assumptions underlie much of what we do. It is often these underlying assumptions that hinder success or produce less-than-expected results. One benefit of logic modeling is that it helps us make our assumptions explicit. Beliefs about how the activities correspond with the outcomes- how the program will work and why.

9. External Factors • Biophysical environment • Economic structure • Housing patterns • Demographic makeup • Family circumstances • Values • Political environment • Background and experiences • Media • Policies Influences outside the program that cannot be controlled but do/could affect project outcomes

10. ProgramEvaluation Why do evaluation? • Modify current programs or strategies • Identify success and challenges • Justify spending/resources • Prepare grant reports • Write future grant applications • Identify needs • Hold personnel accountable • Improve service delivery • Enhance record keeping • Communicate results to stakeholders Systematic collection, analysis and sharing of information for the purposes of answering specific questions about a program.

11. Types of Evaluation • Process Evaluation: Measuring the inputs, activities and outputs of a program. • Fidelity Evaluation: Measuring how program implementation adheres to the prescribed implementation plan. • Implementation Evaluation: Describing and assessing program delivery with an eye towards recommending efficiency gains. • Outcome Evaluation: Measuring the extent to which the program achieved its desired or intended outcomes. • Impact Evaluation: Measuring the long-term outcomes or community level impact experienced by the target population as a result of the program.

12. Evaluating Your Health Literacy Initiative Project • Keep it simple! • Focus on Process (measuring inputs, activities and outputs of your project) & Outcome (measuring the extent to which your project achieved its desired or intended outcomes) • Get the technical assistance you need (just ask Jill!) • Use existing tools:* • Logic model • Questionnaire measuring patient/client perception of communication with their health care professional *Only use the tools that make sense for you and your project

13. Program Logic Model

14. 5 Questions*You Can Use to Measure Your Outcomes • How often did this provider give you all the information you wanted about your health? • Did this provider encourage you to talk about all your health problems and concerns? • How often did this provider ask you to repeat back or describe how you were going to follow the doctor’s instructions? • How often did this provider give you easy to understand instructions about how to take your medicines? • How often were the results of your blood test, x-ray, or other test easy to understand? *Based on Weidmer, Brach and Hays. 2012. “Development and Evaluation of CAHPS® Survey Items Assessing How Well Healthcare Providers Address Health Literacy.” Medical Care, Vol. 50 (9, Supplement 2): pages S3-S11.

15. Thanks! For more information please contact: Jill Lucht Center for Health Policy luchtj@missouri.edu (573) 882-2300 Presentation developed with Elizabeth Pafford, Institute of Public Policy, University of Missouri 573.882.5835 pafforde@missouri.edu