Plan, Evaluate and Improve Your Health Literacy Interventions and Activities

1. Plan, Evaluate and Improve Your Health Literacy Interventions and Activities Sabrina Kurtz-Rossi, M.Ed. Health Literacy Consultant April 10, 2013

2. Objectives • After attending the session, participants will be able to: • Conceptualize a health literacy program evaluation • Write measurable health literacy outcome objectives • Identify indicators to assess both process and outcomes • Identify user-friendly evaluation resources for future use Olney, C. and Barnes, Susan (2006). Planning and Evaluating Health Information Outreach. NN/LM Outreach Evaluation Resource Center. Seattle, WA.

3. Ten Attributes of Health Literate Health Care Organizations – Brach C, Keller D, Hernandez LM, et al. Ten Attributes of Health Literate Health Care Organizations. Washington DC: National Academy of Sciences. 2012

4. Case Study Providence, RI Western Maine Dorcas Place Adult and Family Learning Center Computer Lab Auburn Public Library Computer Training Class

5. Research vs. Evaluation • Research: To draw causal links between observed phenomena and to add to the knowledge base of the field • Evaluation: To demonstrate effectiveness, and determine what works and what does not for the purpose of program improvement Boulmetis, J and Dutwin, P (2005). The ABCs of Evaluation. Jossey-Bass. San Francisco, CA

6. Health Literacy Program / Evaluation Planning • Step 1: Know Your Audience • Step 2: Identify Goals and Objectives • Step 3: Develop Program (and evaluation) Plan • Step 4: Implement and Evaluate • Step 5: Apply and Share Results

7. Step 1: Know Your Audience • Start with an Assessment • Identify needs in context • Guide your program • Build on already existing capacity • Identify champions within your organization • Methods • Informal – discussion, observation • Formal – Key informant interviews, focused group discussions, survey

8. Attribute 1: In a Health Literate Organization leadership makes health literacy integral to its mission, structure, and operations. –

9. Case Study • Methodology: Leaders at community-based organizations interviewed. Semi-structured interview protocol designed to explore community confidence in using the Internet to find health information and local resources to support access. All interviews conducted in person. Data were coded and analyzed into themes. • Key Findings: • Maine has a growing senior population and community-based organizations support public health efforts. Public schools, adult education, and libraries offer computer access and support (n=9). • Latinos get their health information from sources they know and trust, including the radio. A network of minority health programs offer computers access, training and support (n=10).

10. Step 2: Identify Goals & Objectives • Goal -- Why you are doing what you are doing and for whom • Example: • Increase use of health literacy interventions • Improve community access to reliable online health information

11. Attribute 2: A Health Literate Organization integrates health literacy into planning, evaluation, patient safety, and quality improvement. –

12. Case Study Focused Health Information Outreach Project Goal: Increase access to health information among communities experiencing health disparities National Network of Libraries of Medicine – New England Region (NN/LM NER) http://nnlm.gov/ner/

13. Step 3: Develop Program(and Evaluation) Plan Logic Model: A planning tool connecting activities to outcomes Resources Activities/Outputs Outcomes If you have these resources you will get these outcomes and conduct these activities

14. Why a Logic Model? • Provides a framework for planning • Helps organize your thoughts • Clarifies intended outcomes • Links activities to outcomes • Acts as a communication tool Resources Activities Outcomes Outputs

15. Attribute 3: A Health Literate Organization prepares its workforce.

16. Case Study Goal: Increase access to health information among communities experiencing health disparities Consultant Number of trainings held Increase knowledge of NLM resources Develop training materials Outreach coordinators Number trained Conduct trainings Increase use of NLM resources NLM materials Types of materials developed NLM funding Distribute materials at community events Working student Increase community experts to help others Number of materials distributed Librarians in communities

17. Activity • Choose a goal (or use your own) • To increase staff awareness and use of health literacy interventions • To improve community access to reliable online health information • To improve health information literacy skills among patients and their families • Complete the logic model worksheet - identify outcomes, activities, outputs, and resources

18. Attribute 4: A Health Literate Organization involves the population served in evaluation of health information and services.

19. Case Study • Community Assessment • Key informant interviews • Program • Tailored efforts based on findings • Community-based partnerships • Evaluation • Outcome assessment: pre-post training evaluations, follow-up assessment • Process assessment: story-based evaluation, community partner exit interviews Western Maine (rural) Providence, RI (Latino)

20. Evaluation Plan • Process • To determine what worked and what didn’t • For program improvement • Outcome • To demonstrate effectiveness, value, and influence • For comparison purposes, to determine which programs are more effective

21. Attribute 5: A Health Literate Organization meets the needs of the populations and avoids stigmatization.

22. Process Assessment • Methods • Activity logs • Teacher feedback forms • Focused group discussion • Interviews

23. Outcome Assessment • Methods • Existing records • Pre/post tests • Follow-up surveys • Observation • Interview • Story-based

24. Attribute 6: In a Health Literate Organization providers and staff use health literacy strategies in verbal communication and confirm understanding.

25. Case Study Outcomes & Indicators Will name resources on post-surveys Will report use two weeks later of follow up Will document helping others on story-based evaluation forms

26. Measurable Objectives Four components of a measurable objective 1. Condition • Following the training… • After watching the video… 2. Who • …the learner… • …the patient… • …the caregiver

27. Measurable Objectives 3. Performance (knowledge, attitudes, behavior) • …will name… • … will agree… • … will use… 4. Criterion (quality/quantity) • …5 out of 6… • … at least 2… • … the correct way…

28. Characteristics of a Measurable Objective • Is audience focused, not intervention focused (the participant will…) • Includes action verbs • Targets one expectation • Includes observable criteria for evaluation • Matches goals to activities/strategies

29. Attribute 7: In a Health Literate Organization consumers have easy access to health information and services and navigation assistance.

30. Bloom’s Taxonomy • Cognitive – explain, recall, list, predict, select, describe, define • Affective – agree, avoid, help, join • Psychomotor – use, develop, design, repair, taste, use, move

31. Verbs to Avoid Know Comprehend Understand Appreciate Study Gain knowledge of Learn

32. Attribute 8: A Health Literate Organization develops plain language materials that are easy to understand and act on.

33. Case Study Writing Measurable Objectives NLM resources on post-survey Immediately after training participants will name Two weeks after training NLM resources on follow up interview participants will report using Three months training Describing how they helped others find information on story-based evaluation form participants will write a story

34. Activity • Choose an outcome (or use your own) • Increase confidence in using plain language/teach-back • Increased awareness of available online health information resources • Increased interaction with diabetes educators • Complete the writing measurable objectives worksheet - identify condition, audience, performance, and criterion

35. Step 4: Implement and Evaluate • Expect the Unexpected • Be flexible • Adjust to the environment • Address learners’ needs • Tweak your notion of success

36. Attribute 9: A Health Literate Organization addresses health literacy in high-risk situations, including transitions and about medicines.

37. Case Example Maine: Pre/Post: Ever Used NLM Resources Rhode Island: Pre/Post: Confidence in Ability to Find Health Information on the Internet Pre N=72 Post N=55 Pre N= 520 Post N=503

38. Case Example • Successes • Training participants shared resources with friends and neighbors • Partners will continue to promote and distribute resources • Providers used resources to find accurate information to use in their work • Challenges • Pre- and post-evaluations were hard to administer

39. Step 5: Apply and Share Results • Apply Lessons Learned • Make changes based on what you learned • Replicate with similar or different audiences • Share Your Findings • Partners • Colleagues

40. Attribute 10: At a Health Literate Organization they clearly communicates about health plans and health care costs.

41. Evaluation Resources • Planning and Evaluating Health Information Outreach Projects • Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach http://nnlm.gov/evaluation/guide/index.html

42. Questions & Comments Sabrina Kurtz-Rossi, M.Ed. Principal | Health Literacy Consultant 781-835-6488 sabrina@kurtz-rossi.com Kurtz-Rossi & Associates Health*Literacy*Education*Evaluation www.kurtz-rossi.com Member, Clear Language Group www.clearlanguagegroup.com Health Literacy Leadership Institute, June 10-14, 2013 http://healthliteracyleadreship.com