Panning for Research Questions: Using systematic reviews and stakeholder input to identify needed research in comparat

1. Panning for Research Questions: Using systematic reviews and stakeholder input to identify needed research in comparative effectiveness • Jeanne-Marie Guise, MD, MPH • Oregon Evidence-based Practice Center • Tim Carey, MD • RTI-UNC Evidence-based Practice Center Elisabeth Kato, MD, MRP Effective Health Care Program, AHRQ

2. Overview • The Future Research Needs Program • Working with Stakeholders • Prioritization Strategies • Exercise • Discussion and Future Directions

3. The Effective Health Care Program The 2003 Medicare Prescription Drug, Improvement, and Modernization Act authorized the Agency for Healthcare Research and Quality (AHRQ) to improve the quality, effectiveness, and efficiency of health care delivered through Medicare, Medicaid, and State Children’s Health Insurance programs. The Effective Healthcare Program carries out this goals through • Evidence Synthesis • Evidence Generation • Dissemination and Translation

4. How do we decide which comparative effectiveness questions to research? Traditional research paradigm: unplanned, researcher driven, inconsistent public input • Decentralized generation of questions • Peer review by researchers • Closed-door funder decisions New CER/PCOR paradigm: inclusive, rapid, and targeted process • Systematic identification of gaps • Involvement of a range of stakeholders, including patients • Open process

5. AHRQ’s Position with Stakeholders “The Agency for Healthcare Research and Quality (AHRQ) firmly believes that involving all stakeholders in the research enterprise from the beginning improves the end product and facilitates the diffusion and implementation of the findings by getting early buy-in from users. Involving stakeholders also helps to ensure that the research reflects the various needs of all diverse users.” Operating Plan for American Recovery and Reinvestment Act Funds for Comparative Effectiveness Research as the Agency for Healthcare Research and Quality (2009)

6. PCORI’s Mission Statement “PCORI helps people make informed healthcare decisions, and improves healthcare delivery and outcomes, by producing and promoting high-integrity, evidenced-based information, that comes from research guided by patients, caregivers, and the broader health care community.” http://www.pcori.org/about/mission-and-vision/

7. Great idea but….. • Where do gaps come from? • Who should be involved? • What is the best way to prioritize needs? • How much time is required? • How much money does it take? Carey et al. (2012)

8. Systematic reviews = evidence gaps EVIDENCE EVIDENCE evidencegaps= futureresearchneeds?

9. Not necessarily…… • The review team may not identify all gaps • Not every evidence gap is equally important • Different goals “More, bigger, longer, better randomized controlled trials are needed”

10. AHRQ Effective Healthcare Program Future Research Needs Projects • 2010: AHRQ piloted 8 Future Research Needs (FRNs) Projects to extract researchgaps from a systematic review, transforming them into prioritized research questions aided by diverse stakeholders. Goal: alert researchers and funders to important evidence gaps affecting healthcare decisions. • 2011: pilot program reviewed, resulting in a common 7 step framework for AHRQ FRN projects. • 2012: 29 “second generation” FRNs have been or are being completed, 8 FRN methods papers published to date.

11. How do you ensure meaningful and representative engagement? Working with Stakeholders

12. What is meaningful engagement? PCORI – “extending well beyond the simple notion of securing input from a variety of stakeholders” – “projects must include stakeholders as partners with significant involvement in all appropriate stages of the research project” – “ true partnership of shared accountability”

13. Who do you need at the table?

14. Stakeholder Groups Defined

15. Research Continuum Stakeholder Involvement Across Research Phases Topic Generation Research Prioritization Analysis/ Interpretation Dissemination Implementation Study Design Clinicians Consumers Funders Insurers Manufacturers Researchers PayersPolicymakers Clinicians Consumers Funders Insurers Manufacturers Researchers PayersPolicymakers Clinicians Consumers Funders InsurersResearchers Policymakers Clinicians Manufacturers Researchers Policymakers Clinicians Consumers Researchers Policymakers Clinicians Researchers EPC Stakeholder engagement expanded scope of future research needs beyond CER

16. Things to Consider in Selecting Stakeholders • Make sure your expectations and those of stakeholders are aligned • Stakeholders can play multiple roles • Categories are broad, need to consider this carefully • Do you want patients, general public, or patient advocates? • When working with organizations make sure you understand their interests • Individual vs. broader interests • Preparedness to understand topic & research

17. How do you ensure meaningful engagement?

18. Preparation of Stakeholders • Assume that all stakeholders need orientation to the systematic review and evidence grading processes. • Familiarity with research methods likely to be very variable • Conflicts of interest are to be expected with stakeholder groups, but must be disclosed. • Patient/consumer/caregivers may need significant orientation and training regarding research the research process

19. Brief Description of Project: Purpose Process Timeline Background Expectations Final Product Potential Formats: Hard Copy Web Conference In-person Consider offering personal sessions Provide Background Materials in Lay Terms

20. Example: Purpose • Goals: To establish an agenda to guide the Agency for Healthcare Research and Quality’s (AHRQ’s) comparative effectiveness review and research activities regarding prevention in women’s health • Identify topics stakeholders struggle with in clinical decisionmaking and health policy, where answers will improve the quality, effectiveness, and efficiency of United States health care

21. Example : The Process

22. Example : Timeline • Three meetings: • May 11 – Framework • June 8 – Topic Generation • July 13 – Final Prioritization • Extranet and email topic generation and participant discussion

23. Example: Background Comparative Effectiveness Reviews • Definition: Comparative Effectiveness Reviews (CERs) systematically and critically appraise existing research comparing the benefits and harms of alternative methods to prevent, diagnose, treat and monitor a clinical condition, or to improve the delivery of care • Purpose: To assist consumers, clinicians, purchasers and policymakers to make informed decisions that will improve health care at both the individual and population levels Sox, HC & Greenfield, S (2009).

24. Example: Expectations You are here • Technical Expert (TEP) Panel • The TEP is constituted to inform the scientific content of the research review and provide guidance to the EPCs throughout the evidence review process. Technical Expert Panels may assist the EPCs by providing input to: • Focus the literature search • Identify inclusion/exclusion criteria • Answer specific technical or clinical questions • Identify specific subgroups for which there may be heterogeneity of treatment effect.  • *Note: The role of the EPC is to synthesize a report of the existing body of evidence. The final EPC report will be published by AHRQ and may be used by other agencies to inform policy or determine clinical guidelines.

25. Recommendations • Key Informants: Optional • Stakeholder Number : 2-3 stakeholders from each SH category (One stakeholder is not representative; Different stakeholders of the same “type” have different perspectives) - OMB • Stakeholder Constituency: Depends on research phase • Most times clinicians, researchers, funders, consumer advocacy representatives • Methods of Engagement: Mixed • In person (telephone or face to face with or without web) – orientation clarification and expectations delineation; also can be used to generate topics clarify questions • Materials – summarizing overview for stakeholders

26. Current dilemmas • Different stakeholders need different levels of training • What extent can stakeholders be reused from project to project? • Preparation and experience may change the stakeholder over time to be more like us

27. When to do it How to do it What method is best? How can we do it better? Prioritization

28. AHRQ Effective Healthcare Program Future Research Needs Projects • 2010: AHRQ piloted 8 Future Research Needs (FRNs) Projects to extract researchgaps from a systematic review, transforming them into prioritized research questions aided by diverse stakeholders. Goal: alert researchers and funders to important evidence gaps affecting healthcare decisions. • 2011: pilot program reviewed, resulting in a common 7 step framework for AHRQ FRN projects. • 2012: 29 “second generation” FRNs have been or are being completed, 8 FRN methods papers published to date.

29. Priority Ranking • Reviews may generate many gaps, need for prioritization • Some organizations use broad internet data gathering • Will the participants understand all of the issues? • Multiple methods currently used • Ranking 1-xx • Likert scale 1-7 • Multi-voting, multiple (but limited) votes per choice • Pair-wise comparisons • Delphi methods • Nominal group process • Consensus conference

30. Transformation of Research Gaps into Needs • Gaps are generally in the form of a declarative sentence. • Needs are questions similar to research questions in a grant proposal. • Most organizations use PICOTS framework: • Population, Intervention, Comparator, Outcome, Timeframe, Setting. • Methods questions may be important, but may not be a fit for PICOTS.

31. Refinement and Re-Ranking • When number of gaps is large, multiple rounds of prioritization may be needed. • Many groups use 2-3 rounds • Gaps may be combined or split depending on the level of granularity that is useful. • Is consensus needed?

33. Example: AHRQ Future research needs on ADHD Key Questions from Comparative Effectiveness Review 20 research gaps from the review mapped to the key questions, presented to a group of 12 stakeholders, including funders, advocates, clinicians, regulators, researchers, and policymakers. After stakeholder input, 29 research gaps. 8 gaps emerged as the top future research needs after two rounds of prioritization using an online prioritization tool. The next two slides show the presentation of one gap from identification to study design.

34. Identify Research Gap: For children less than 6 years of age with disruptive behavior disorder or ADHD, limited data are available about the efficacy and effectiveness of psychosocial treatment programs (e.g., parent training and summer behavior treatment programs), alone or in combination with pharmacological interventions, compared with other psychosocial treatment programs, alone or in combination with pharmacological interventions. (KQ 1) After One Round of Prioritization Apply PICOTS and Develop Research Question: Research Question: For children less than 6 years of age with disruptive behavior disorder or ADHD, what is the comparative efficacy and effectiveness of specific psychosocial treatments alone compared with pharmacological treatments alone or in combination with psychosocial treatments for patient outcomes?

35. After Second Round of Prioritization Develop Study Design Considerations: • Randomized controlled trials • Randomized trials could be designed to test various components in a 2x2 matrix of psychosocial treatment variants (parent training, school-based intervention, combination, or pharmacological). • Advantages of study design for producing a valid result • Allows isolation of causal inferences related to the intervention being tested. Multiple-armed trials would allow testing of several hypotheses regarding relative efficacy of singular or combination treatment components. • Ability to recruit/availability of data • Common condition in this age group with uncertainty regarding treatment choice; all arms receive some treatment. • Resource use, size, and duration • Large sample size (N = 840; n = 210 per treatment arm) needed. Key outcomes such as school achievement will require follow-up of several years. • Ethical, legal, and social issues • Vulnerable population, careful informed consent will need to occur.

36. State of the Science • Multiple groups are currently conducting work in this area • Tendency to emphasize cross-cutting issues across diverse stakeholder groups • Sufficient common aspects to serve as a consensus • Criteria for gaps identification • Broad aspects of stakeholder panel composition • Need to train stakeholders in PCOR • Explicit prioritization method - but multiple methods currently used, no empiric guidance as to optimal method • Decisions regarding study design considerations • Organizations can use existing methods now while refining the approaches

37. Timelines • Work can expand to fill a substantial amount of time • Rapid turnaround important for: • Working from the current systematic review • Dissemination • 4-7 month time frame • Step 1: Identification/Recruitment (1-2 months) • Step 2: Engagement & Prioritization (2-4 months) • Step 3: Analysis/Drafting Report (1-2 months) • After publication: Dissemination- targeted vs broad, role of web vs peer review (may not be the EPC conducting this step)

38. Recommendations • Evaluate different stakeholder panel sizes and compositions in prioritization. • Evaluate the reliability of stakeholder prioritization through replication studies. • Test different methods of prioritization to assess for transparency, reproducibility and efficiency. • Clarify role of gap identification and prioritization with other methods such as value of information modeling. • Collaborate across PCOR programs in improving methods, developing consistent terminology.

39. Current dilemmas • What prioritization works best? • General versus specific research topics or questions • Do you propose study designs? • How many rounds are required?

40. EXERCISEFuture research needs in Attention Deficit Hyperactivity Disorder (ADHD)

41. Key Questions from Comparative Effectiveness Review

42. Analytic Framework

43. Evidence Gaps • How does the use of different data capturing methods (such as child self-rating scales, parent scales, semistructured interviews) affect apparent treatment efficacy in people >= 6 yrs with ADHD. • What are the prescriptive tx or optimal circumstances for adding specific psychosocial and/or pharmacological tx components based on the needs of people >= 6 yrs with ADHD. • What are the prescriptive tx or optimal circumstances for adding specific psychosocial and/or pharmacological tx components based on the needs of preschoolers with disruptive behavior disorder or ADHD. • What is the amount of variation in case identification and prevalence across geographic areas, age groups, settings and cultures. • What is the comparative effectiveness of different preschool interventions on long term outcomes.

44. Nominal Group Exercise

45. Summing it all up • Systematic reviews provide a feasible basis for inclusive, rapid, and targeted identification of future research needs. • Meaningful stakeholder involvement requires careful selection and preparation. • There are many ways to involve stakeholders in prioritization but the optimal approach is not yet clear.

46. Future Directions for Future Research Needs • How should research needs identified through topic specific systematic reviews fit into broader topic identification efforts? • Will the gaps and prioritization resonate with funders and policymakers? • What are the best ways to communicate with the public and with funders? • Who should be involved in refining this area?

47. Questions? Comments? elisabeth.kato@ahrq.hhs.gov guisej@ohsu.edu timothy_carey@med.unc.edu

48. References • Carey T, Yon A, Beadles C, Wines R. Prioritizing Future Research through Examination of Research Gaps in Systematic Reviews. Prepared for the Patient-centered Outcomes Research Institute. Washington DC: March 15, 2012. • Chang SM, Carey T, Kato EU, Guise JM, Sanders GD. • Ann Intern Med. 2012 Jul 31. doi • Kane RL, Guise JM, Hartman K, Rothenberg B, Trikalinos T, Wilt T. Presentation of Future Research Needs. Methods Future Research Needs Report No.9. AHRQ Publication No. 12-EHC053-EF. Rockville, MD: Agency for Healthcare Research and Quality. April 2012. www.effectivehealthcare.ahrq.gov/reports/final.cfm. • Sox, HC & Greenfield, S. Comparative effectiveness research: a report from the Institute of Medicine. Annals of Internal Medicine. 2009; 151(3): 203-205.