Jeanne-Marie Guise, MD, MPH Professor Oregon Health & Science University

1. Stakeholder Engagement in the Evidence-based Center (EPC) Practice Program Jeanne-Marie Guise, MD, MPH Professor Oregon Health & Science University

2. Objectives • Describe ways in which the EPC program has engaged stakeholders to improve relevance and applicability of its products • Present methods used for stakeholder engagement • Identify challenges in stakeholder engagement and strategies employed by the EPC program

3. EPCs & SRC Evidence-based Practice Centers (EPCs) • 11 centers throughout the United States and Canada • Systematically and critically appraise existing research and synthesize knowledge • Produce comparative effectiveness and effectiveness reviews Scientific Resource Center (SRC) • Responsible for scientific methodological work of reviews and other research projects for EPCs and DEcIDES

4. EPCs & SRC • Blue Cross and Blue Shield Association, Technology Evaluation Center, Chicago, IL • Brown University, Providence Rhode Island • ECRI Institute, Plymouth Meeting, PA • Johns Hopkins University, Baltimore, MD • Kaiser Foundation Research Institute, Oakland, CA • Oregon Health & Science University, Portland, OR • RTI International – University of North Carolina at Chapel Hill, Chapel Hill, NC • RAND Corporation , Santa Monica, CA • University of Alberta, Edmonton, Edmonton, Alberta • University of Minnesota EPC, Minneapolis, MN • Vanderbilt University Medical Center, Nashville, TN • Scientific Resource Center (SRC) for EPCs and DEcIDES – Portland VA Research Foundation, Portland OR

5. Opportunities for Stakeholder Input

6. EHC Report on Engaging Stakeholders for Development and Prioritization of Future Research Needs • O’Haire C, McPheeters M, Nakamoto E, LaBrant L, Most C , Lee K, Graham E, Cottrell E, Guise J-M. Engaging Stakeholders To Identify and Prioritize Future Research Needs. Rockville (MD): Agency for Healthcare Research and Quality (US); 2011 Jun. (Methods Future Research Needs Reports, No. 4.) Available from: http://www.ncbi.nlm.nih.gov/books/NBK62565/

7. Phase I: Literature Summary (Oregon EPC) Phase II: Key Informant Interviews (Oregon EPC) Phase III: EPC interviews (Vanderbilt EPC) • Literature Search EPC Protocol Review Recruitment and Interview Guide Development • Invitations and EPC Discussion Guide Key Informant Interviews EPC Discussions Literature Review Literature Synthesis Analysis of Discussions Analysis of Interviews Cumulative Synthesis

8. Stakeholder Identification & Recruitment • Research Needs Development • Identify pertinent stakeholder groups (intentional based upon stakeholder groupings) • Ensure a balance of stakeholder perspectives (i.e. consumer, clinician, researcher, research funder, insurer/payer, manufacturer) • The results of EPC FRN pilot projects suggest that engaging 6-10 stakeholders is appropriate • OMB restricts the number of participants in surveys to no more than 9 non-federal employees

9. Stakeholders

10. Methods of Stakeholder Engagement Engagement • Most common: Focus Groups, Symposia/Conferences Prioritization • Most common: Priority Questionnaire, Consensus Engagement & Prioritization Simultaneously • Priority Survey • Delphi

11. – + • GOOD FAIR POOR

12. EHC Program Research Reviews Topics Evidence Review Research D&I

13. Topic Generation Approaches Nominal Group Process • In-person • Idea generating • Intended to promote group participation Our experience: In-person brainstorming with flip chart used to document topics Each state had 8 stickers to vote for priority topics: #1 (2 red dots) #2 (2 green dots) #3 (2 blue dots), and #4 (2 yellow dots) Weighting system applied to priorities: #1 - 4 points for each vote #2 - 3 points #3 – 2 points #4 – 1 point

14. Topic Generation Approaches • Medicaid Medical Directors Analytic Process: Weighting system : #1 - 4 points #2 - 3 points #3 – 2 points #4 – 1 point

15. Topic Generation Approaches Prevention in Women’s Health Modified Process: • Local Key Informants • Three web meetings: • May 11 – Framework & Initial Topics (Extranet & email topic generation) • June 8 – Topic Generation (Evidence Scan) • 13 – Final Prioritization • Prioritization • Survey – survey monkey • Web whiteboard during 3rd meeting

16. Scoping CER Stakeholders Engaged in: • Development of CER key questions • Refinement of topic • Review of Draft Methods used: • Group or individual calls • Web-based forums with survey type format to provide responses

17. Scoping CER Preparation of Stakeholders is critical: • Assume that all stakeholders need orientation to the CER process • Very important for them to know where they fit in and what to expect from their feedback • Familiarity with research methods likely to be very variable • Conflicts of interest are to be expected with stakeholder groups, but must be disclosed.

18. EHC Program Research Reviews • Stakeholder Preparatory Materials

19. Stakeholder Preparatory Materials

20. Future Research Needs Development Purpose • Comparative Effectiveness Reviews (CERs) critically appraise and synthesize bodies of literature and identify areas where research is lacking or insufficient in either quantity or quality • There are limitations in funds available to support research • To fully actualize their potential, CERs should be utilized to inform and guide future research • Future Research Needs (FRN) documents are designed to facilitate this process by engaging stakeholders to prioritize future research topics

21. Future Research Needs Development

22. Research Needs Development Combined Methods: • Semi-structured interviews to identify topics • Modified Delphi process for prioritization Semi-structured interviews • Telephone • Conference calls • Webinars • Delphi • #1 Survey • Topics organized by PICOs • Likert Scale – 6-point scale of no, low, highest priority • #2 Survey • Likert • Ranking • Indicate top 3 in order

23. Research Needs Development

24. Research Continuum • Research Needs Development Stakeholder Involvement: Across Research Phases Topic Generation Study Design Research Prioritization Analysis/ Interpretation Dissemination Implementation Clinicians Consumers Funders Insurers Manufacturers Researchers PayersPolicymakers Clinicians Consumers Funders Insurers Manufacturers Researchers PayersPolicymakers Clinicians Consumers Funders InsurersResearchers Policymakers Clinicians Manufacturers Researchers Policymakers Clinicians Consumers Researchers Policymakers Clinicians Researchers EPC Stakeholder Engagement Expanded Scope of Future Research Needs Beyond CER

25. Summary • Several options for methods to use when engaging stakeholders • Method depends on several factors including • Intent (identifying topics, prioritization) • Resources • Number and variety of stakeholders • What is already known in topic • Experience • Important to outline approach and intent a priori • Relationships with stakeholders are important • Contact potential stakeholders through multiple venues • Consistent communication and follow-up in the form of emails or phone calls is important to ensuring a balanced selection of stakeholders and a high response rate • Preparation of stakeholders is critical to define expectations and allow engagement