Process Measures

1. Process Measures Session 3: Collaborative Learning Project

2. Today We Will… • Describe process measuresrelevant to healthy weight program (HWP) evaluations • Identify valid, reliable, and generalizable tools and/or methods to collect process measures   • Prioritize process measures to identify core measures to recommend for inclusion across programs 

3. Process Measures Documenting & Enhancing Engagement • Nancy Sherwood, PhD • University of Minnesota

4. What Are Process Measures? • Process measures are measures that examine how your program activities are delivered. • Examples of process measures include: • Measures of participation • Readiness to change • Mastery of skills

5. Core Process Measures • Program Enrollment • Are you engaging the target audience? • Are enrollment goals being met? • Program Attendance • How many sessions are people attending? • What percent of families who start the program, finish the program? • Program Activities: In-Session & Between Session • What percent of planned session activities are completed? • What percent of “homework” is completed (e.g., food/activity logs)?

6. Core Process Measures • Program Enrollment • Are you engaging the target audience? YES • Are enrollment goals being met? YES • Program Attendance • How many sessions are people attending? 100% • What percent of families who start the program, finish the program? 100% • Program Activities: In-Session & Between Session • What percent of session activities are completed? 100% • What percent of “homework” is completed? 100%

7. Using Process Data to Enhance Engagement: Example Source: French SA, Sherwood NE, Veblen-Mortenson S, et al. Multicomponent Obesity Prevention Intervention in Low-Income Preschoolers: Primary and Subgroup Analyses of the NET-Works Randomized Clinical Trial, 2012-2017. Am J Public Health. 2018 Dec;108(12):1695-1706.

8. Pediatric Primary Care Annual Well Child Visit Pediatric Primary Care Annual Well Child Visit Child Physical Activity & Screen Time Home Environment & Parenting Practices Food, Physical Activity, Media, Parenting Parent/ Primary Caregiver Family Connector Home Visits 12 per year & Check-Ins 4 per year Child BMI Child Dietary Intake Parenting Classes 12 per year Neighborhood Physical Activity & Nutrition Resources

9. 3 Year NET-Works Intervention • Goal: Keep parents actively engaged in all components of the intervention for a three year period. • Two key process measures documented in study database • Dose delivered, or amount and length of intervention sessions offered • Dose received, or participant attendance and completion of the offered sessions • Define “At Risk” families • 30 days, 45 days, 60 days, 90+ days without participating in an intervention activity • Implement re-engagement protocols

11. Summary • Develop tracking systems and use process evaluation data to optimize engagement and participation • Flexibility and responsiveness to the unique needs of families can enhance engagement • Tracking systems & re-engagement strategies need to be tailored to specific contexts

12. A Path to Sustainability Creating and Demonstrating Value • Chris Bolling, MD • American Academy of Pediatrics

13. What Do Stakeholders Want?

14. What Do Stakeholders Want? Before you figure what they want, there are some things to know: • Figure out who they are • Ask what they want • Know that there are different ways to ask • Be aware that the answers will surprise • You will need different messages for different stakeholders, BUT

15. What Do Stakeholders Want? Be true to your mission keeping your big picture in sight

16. What Do Stakeholders Want? • Return on Investment • Financial • Workforce productivity • Healthy community • Improved quality of life • Something you didn’t even think of!

17. What Do Stakeholders Want? And you thought BMI was hard to change

18. What Do Stakeholders Want? Outcomes can look very different to them • Do not underestimate the power of the subjective, especially early in the process • Reasonable deliverables • Patient satisfaction • A timeline for when things will happen • The overarching goal clearly stated • Ongoing follow-up with stakeholders

19. What Does YOUR Community Want?

20. What Does YOUR Community Want? As with stakeholders, ask first • What they tell you, may surprise as well • Who are the key players? • Who are the ones who really “get it” and are prepared for the long haul?

21. What Does YOUR Community Want? • Time expectations • Again, keeping the big picture in view • Being clear about roles • Keeping commitments • Being prepared to be the driver Things to consider with your community

22. Making the Cost-Benefit Case • Difficult to make on a purely monetary basis • Health care systems present special challenges • The depressing lesson of tertiary obesity care centers

23. Making the Cost-Benefit Case • You have to think globally • The value added is system and community wide

24. Making the Cost-Benefit Case • The goal may be “as revenue neutral as possible” • Think as infrastructure • Value to other entities • Speaks to community health where it is recognized as a public health crisis • Breakdown occurs at patient level • Weight bias at work (sorry to editorialize!)

25. Making the Cost-Benefit Case • The importance of being a trusted expert • Picking your battles • Finding your “sugar daddy” • Using your voice • Prioritizing your efforts

26. Process Measures • Measuring Provider Communication Style and Skill • Ken Resnicow, PhDUniversity of Michigan

29. Motivational Interviewing Treatment Integrity Coding Manual 4.2.1Moyers TB, Manuel JK, & Ernst D. Motivational Interviewing Treatment Integrity Coding Manual 4.1. Unpublished manual, 2014.

30. MI Specific Universal

32. Manual for Motivational Interviewing Skill Code (MISC)Version 2.1Miller WR, Moyers TB, Ernst D, & Amrhein P. Manual for Motivational Interviewing Skill CodeVersion 2.1. University of New Mexico, 2008.

33. MISC: Coding Client Speech

34. One Pass • McMaster F & Resnicow K. Validation of the one pass measure for motivational interviewing competence. Patient Educ Couns. 2015 Apr;98(4):499-505.

37. Behaviour Change Counseling Index (BECCI)Lane C, et al. Behaviour Change Counselling Index (BECCI). University of Wales College of Medicine, 2002.

39. MI Coach Rating ScaleNaar, S. & Safren, S. (2017). Motivational Interviewing and Cognitive-Behavioral Interventions. Guilford Press, New York, NY.

40. MI Coach Rating Scale

41. MI Coach Rating Scale: Scoring Rubric

42. The Roter Interaction Analysis System (RIAS): Universal

43. RIAS: Universal • Forty exclusive behavior categories: • Socioemotional communication • Positive, negative, emotional, partnership building, and social exchanges • Task-focused communication • Asking questions, giving instruction and direction, and giving information • Grouped into four primary functions of the medical visit • Data gathering • Patient education and counseling • Responding to patient emotions (rapport) • Partnership building

44. Source: https://doi.org/10.1371/journal.pone.0203970.t002

45. ComOn CheckRadziej K, Loechner J, Engerer C, et al. 2017. How to assess communication skills? Development of the rating scale ComOn Check. Med Educ Online. 22:1.

47. Cognitive Behavioral TherapyHepner KA, Howard S, Paddock SM, Hunter SB, Chan Osilla K, Watkins KE. A Fidelity Coding Guide for a Group Cognitive Behavioral Therapy for Depression. Santa Monica, CA: RAND Corporation, 2011. Sponsored by the National Institute on Alcohol Abuse and Alcoholism and the National Institute on Drug Abuse

48. Measure of CBT Treatment Fidelity

49. CBT Fidelity Checklist Source: Johnson C, L. Handen B, Butter E, et al. Development of a parent training program for children with pervasive developmental disorders. Vol 222007.