POPULATION RESEARCH SEMINAR SERIES

1. POPULATION RESEARCH SEMINAR SERIES Sponsored by the Statistics and Survey Methods Core of the U54 Partnership Administering multisite studies: Pitfalls and pointers Lee Hargraves, PhD Center for Survey ResearchUMass Boston

2. POPULATION RESEARCH SEMINAR SERIES Sponsored by the Statistics and Survey Methods Core of the U54 Partnership Questions? Comments? Type them in or ask over your webcam/microphone Or send an email to u54.ssmc@gmail.com

3. Overview • The advantages of conducting a multi-site study, focusing on behavioral interventions • Key components of administering the study with an eye on ensuring fidelity • Some "lessons learned" and guidance to address the issue of "if I only knew then what I know now”

4. Overview • How analysis of a multisite trial sparked interest • A foray into conducting a randomized experiment in community health centers • 12 health centers, 6 intervention/6 control • Building on these experiences trying it again • 2 health centers, randomizing patients • Brief introduction to a model of treatment fidelity • Q & A

5. First Exposure to Multisite Studies • 8 adult and 8 pediatric primary care practices • Hospitals or health centers • Internal medicine, family medicine, nurse practitioners • Cycles of quality assurance feedback • Measured adherence to guidelines for multiple common conditions • Otitis media, low hematocrit, CA screening, and others

6. Findings: More than did performance improve? yes or no Multiple sites allow for examining change in varying conditions and settings • After QA intervention, full-time practitioners improved more than part-time practitioners • Female practitioners were more likely to perform cancer screening among women than male practitioners • Individual practitioner performance varies from patient-to-patient, from guideline-to-guideline • Practice leaders who participated in QA influenced performance among others in the practice • Practitioners in larger health centers performed better and improved more than those in smaller centers

7. The Pros and Cons of Conducting Interventions in Multiple Sites • The advantages of conducting a multi-site study • Increase in sample size, e.g., sometimes it’s hard to find people with specific characteristics at one place • Organizational Size Varies • Diversity (in both people doing the work and the participants) • Location • The disadvantages of conducting a multi-site study, • Organizational Size Varies • Diversity (in both people doing the work and the participants) • Location

8. Why Do Sites and Organizations Participate? • Not for the money • Generate new knowledge • Satisfy requirements, e.g. accreditation for clinics • Recognition • Guide developments of new models, cutting edge research

9. Funding from Finding Answers: Disparities Research for Change, An Initiative of the Robert Wood Johnson Foundation Example of a Multisite Study:Using Community Health Workers to Reduce Disparities in Diabetes Care

10. Overview of the Project An evaluation of the effects of enhanced training of community health workers (CHWs) to reduce disparities in care in a low-income, racially and ethnically diverse population served by 12 community health centers (CHCs)

11. Collaborative Partners • Community Health Centers in Massachusetts • Massachusetts League of Community Health Centers • Central Massachusetts Area Health Education Center & the Outreach Worker Training Institute • University of Massachusetts Medical School

12.      

13. Community Health Patients • Nearly 40% are insured through Medicaid • Approximately 40% are low-income uninsured and underinsured • More than 60% belong to an ethnic, racial or linguistic minority group • Fewer than 20% are insured by Medicare or commercial insurance

14. An Intervention with Community Health Workers • Curriculum developed for health worker training related to the Care Model and diabetes self-management • Reviewed and customized for divergent cultural, linguistic, and ethnic groups • Training sessions equal to 45 hours

15. Overview • Random selection among health centers, paired for size and patient characteristics • Selected and trained Community Health Workers, employed by 6 health centers • Supervision of CHWs • Structured workbook • Data collection • CHW activities • Ongoing data collection using patient registries • Qualitative interviews and focus groups

16. CHW Study Design & Measurement • Compare sites with trained CHWs pre/post intervention to control community health centers without CHW • Measures included clinical data (HgA1c, BP) and encounter data (patients working on goals, # visits) • Assess qualitative data via focus groups and intensive interviews in all CHCs • Focus groups with CHWs • Focus groups with supervisors

17. Training CHWs • Developed a Community Health Worker curriculum (Outreach Workers Training Institute, Worcester) • Developed curriculum to train Supervisors of Community Health Workers • Convene CHW training advisory group to review materials • Quarterly training sessions, using case studies

18. Multiple Data Sources • PECS—the Patient Electronic Care System • Disease management registry • It was free • Used to enter, store, and report clinical data • Did not test how to export to data file for analysis • CHW Encounter Forms • Patient Survey • Community Health Center Monthly Narrative Reports, and • Site Visits

19. Using Data Collection Tools to Guide Structured Interventions Lemay, C. A., Ferguson, W. J., & Hargraves, J. L. (2012). Community health worker encounter forms: a tool to guide and document patient visits and worker performance. Am J of Public Health, 102(7), e70-e75.

20. Unexpected Events • Health center needed to drop out • Health workers were often pulled in other directions • Monitoring was more necessary than planned • Data collection system, not build for research

21. Lessons Learned • PECS was not designed for research • Disease management registry • Systems on site (e.g., EHRs) are not designed for research • Training was key, needed continuing education • CHWs were part-time on the project • Valuable assets of health centers • Needed to monitor activity on the study • Focus groups: Inadequate preparation of health center, i.e., many didn’t know the study was there • Encounter data was essential

22. The Takeaway • Employing CHC staff was necessary to get buy-in, yet hard to manage • Need to do a better job of working with sites, i.e., meet with staff, work with management • CHWs need more extensive training in how to work with patients • Structured encounters was the bare minimum • Planned activities was helpful, but need specifics for ongoing visits • Test your data entry beyond “getting a report” • Need better planning for unexpected outcomes

23. Why Not Try Another Multisite Study? Community Health Workers Using Patient Stories to Support Hypertension Management Funding from the National Institute on Minority Health and Health Disparities(NIMHD) http://cheir.org/projects/projects-topic-area/community-health-workers

24. CHWs, Trained in Motivational Interviewing Using Storytelling • Two health centers • Enhanced training in motivational interviewing • Delivery of support tools (Videos of patients telling how they manage hypertension) • Better data collection strategies

25. Motivational Interviewing Skills and Techniques An empathic, gentle, and skillful style of counseling that helps practitioners have productive conversations with individuals with co-occurring and other disorders. Essential characteristics of motivational interviewing include: • Expressing empathy through reflective listening. • Noting discrepancies between current and desired behavior. • Avoiding argumentation and rolling with resistance. • Encourage the consumer's belief that he or she has the ability to change. • Communicating respect for and acceptance of people and their feelings. • Establishing a nonjudgmental, collaborative relationship. • Being a supportive and knowledgeable consultant. • Complimenting rather than denigrating. • Listening rather than telling. • Gently persuading, with the understanding that change is up to the person. • Providing support throughout the process. Source: http://www.samhsa.gov/co-occurring/topics/training/skills.aspx

26. Applying Lessons Learned from FindingAnswers CHW Study • Manageable number of sites • Allow for more contact with staff, including providers • More training to work with patients, specifically in motivational interviewing ,was inadequate • Build in monitoring, supervision • Use better technology for data collection and management

27. Motivational Interviewing, Key Intervention Component • Cannot be taught in a couple of hours, e.g., from a website • Is more difficult for some who have extensive training in other techniques • Needs to be learned through doing • Practitioners need feedback, coaching • How do you know if it’s done right?

28. Extensive Training in MI

29. More Intense Training in Motivational Interviewing • Ensuring fidelity in Motivational Interviewing • Training, extensive with role modeling, teaching, observing, and feedback • Evaluation using “standardized patients” • Monitoring, record and observing patient visits • Scoring when health workers employ techniques consistent with MI • Coaching to avoid telling patients what they should do…

30. Tools to Standardize Both Delivery and Collection of Data • REDCap Research Electronic Data Capture • Lives in the internet via a collaborative • Available wherever there’s WiFi • Design forms for data collection • Randomize participants • Implement surveys • Scheduling function • Export data in files that make sense

31. Online Videos and Training Materials

33. How REDCap is Used to… • Randomize patients for immediate or delayed entry • REDCap and patient support tools on tablet PCs in community health centers (video on Vimeo) • Streamlined Electronic Medical Record abstraction • Online Surveys to assess health behaviors, literacy, trust, and demographic information, pre-post • And, provide a timetable for repeat contact with patients • And there’s more…provide a log of user activities

34. Guiding Evaluation of Interventions “Researchers and practitioners are typically most interested in the outcomes or results of their study or program. If they have thoughtfully designed the intervention, it is likely that theory has guided the effort. Conceptual and logic models can be used to depict how intervention activities or “inputs” are expected to produce desired outcomes by way of changes in intermediate factors that are often theoretically linked. For example, changes in colorectal cancer screening behavior are more likely to occur when there are subsequent changes in knowledge, attitudes, perceived susceptibility to colon cancer, and reduced barriers to obtaining screening”. Source: Glasgow R.E. & Linnan L.A. “Evaluation of Theory-Based Interventions” in Glanz, K., Rimer, B. K., & Viswanath, K. (Eds.). (2008). Health behavior and health education: theory, research, and practice. John Wiley & Sons, 487-508.

35. Resource to Guide Evaluation www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide

36. What is Being Evaluated? Source: Kellogg, W. K. (2004). Logic model development guide. Michigan: WK Kellogg Foundation.

37. Example of a Logic Model to GuideDesign, Plan Evaluation Source: Glasgow R.E. & Linnan L.A. “Evaluation of Theory-Based Interventions,” Ch. 21 in Glanz, K., Rimer, B. K., & Viswanath, K. (Eds.). (2008). Health behavior and health education: theory, research, and practice. John Wiley & Sons..

38. Lessons Learned • Use Logic Models to plan evaluation • Building Data Collection into the Fabric • You may not know what you need to measure (e.g., hours, time, etc.). Therefore, within reason, try to measure comprehensively • Use Enrollment charts (e.g., Targeted Enrollment/Week and Actual Enrollment) • Encounter forms are nice but a cloud-based system is better, e.g, REDCap

39. Lessons Learned • Oversight and monitoring, at every step • Periodic timesheets to record activities • Keep logs of work effort, enthusiasm for the study waxes and wanes • Periodic conference calls • Training seminars for staff, especially those delivering interventions • Ensuring Fidelity at each step • Count on personnel turnover

40. On Fidelity Large literature, e.g., • Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., et al.. (2004). Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology, 23(5), 443. Urging fidelity assessment in the … • Design of study • Training • Delivery of treatment • Receipt of treatment • Enactment of skills

41. How to Infuse Fidelity Spillane et al. recommend many varied treatment fidelity procedures to improve the reliability and validity of research including: • Use of manuals and standardized guidelines for intervention providers. • Researcher observations through video/audiotaped observation of the intervention (e.g. a consultation). • Self-report evaluation forms/checklists completed after consultations by those delivering the intervention. • Quality assurance checks completed after the consultations by the researcher. • Researcher field notes regarding the study intervention (e.g. requests for additional assistance with intervention implementation and records of all phone and face-to-face contact between interventionists and researchers). • Case conferences that provide a forum for discussion of the intervention. • Questionnaire surveys/focus groups/interviews with intervention providers and recipients. • Monitoring and testing study participants’ acquisition of treatment skills. Source: Spillane, V., Byrne, M. C., Byrne, M., et al. (2007). Monitoring treatment fidelity in a randomized controlled trial of a complex intervention. Journal of advanced nursing, 60(3), 343-352.

42. Finally, Why Multisite?It Gets Closer to Effectiveness Research Laboratory “the real world” Source: From Greenwald and Cullen (1985) as summarized in Glasgow, R. E., & Linnan, L. A. (2008). Evaluation of theory-based interventions. Health Behavior and Health Education: Theory, Research, and Practice. San Francisco, CA: Jossey-Bass, 487-508.

43. Model of Phases of Research Source: Greenwald, P., & Cullen, J. W. (1985). The new emphasis in cancer control. Journal of the National Cancer Institute, 74(3), 543-551.

44. Good luck • Others willing to share experiences • Q & A.

45. POPULATION RESEARCH SEMINAR SERIES Sponsored by the Statistics and Survey Methods Core of the U54 Partnership Questions? Comments? Type them in or ask over your webcam/microphone Or send an email to u54.ssmc@gmail.com