The AHRQ Training Modules for the Systematic Reviews Methods Guide: An Introduction

1. The AHRQ Training Modules forthe Systematic Reviews Methods Guide:An Introduction Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide www.ahrq.gov

2. Learning Objectives • To review the goals and activities of the Evidence-based Practice Center (EPC) program in relation to the Effective Health Care (EHC) Program • To describe the roles and responsibilities of the different participants in the EHC Program in relation to the EPC program • To outline the purpose and components of the Training Modules for the Methods Guide for Comparative Effectiveness Reviews (CERs) • To define important terms and concepts used in the Training Modules for the Methods Guide for CERs

3. What Is the Effective Health Care Program? • Section 1013 of the 2003 Medicare Prescription Drug, Improvement, and Modernization Act authorized the Agency for Healthcare Research and Quality (AHRQ) to improve the quality, effectiveness, and efficiency of health caredelivered through Medicare, Medicaid, and State Children’s Health Insurance programs. • AHRQ started the Effective Health Care Program in 2005 upon appropriation of funds. • In 2009, the American Recovery and Reinvestment Act (ARRA) awarded additional funding to AHRQ to: • broaden comparative effectiveness research, • build an infrastructure to support the expanded efforts, and • invest in innovative mechanisms to disseminate findings of research to health care decisionmakers.

4. What Does the Effective Health Care Program Do? • Conducts effectiveness and comparative effectiveness research to inform the health care decisions of patients, providers, and policymakers   • Involves individual researchers, research centers, and academic organizations in the United States and Canada  • Comparative effectiveness research includes: • Reviews and syntheses of published and unpublished scientific evidence • New scientific evidence and analytic tools • Translates research findings into easily understood formats for patients, providers, and policymakers

5. What Is Comparative Effectiveness Research? • The conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat, and monitor health conditions in “real world” settings. • The purpose of this research is to improve health outcomes by developing and disseminating evidence-based information to patients, clinicians, and other decisionmakers and responding to their expressed needs about which interventions are most effective for which patients under specific circumstances. Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. June 30, 2009. Available at: http://www.hhs.gov/recovery/programs/cer/cerannualrpt.pdf.

6. The ABCs of Comparative Effectiveness Research • Answer real questions • Health care decisionmaking • Best evidence • Internal and external validity • Comparative • Real-world comparisons • Effectiveness • Patient-centered outcomes of interest

7. Effective Health Care Program Principles • Relevant and timely • Useful for decisionmakers • Stakeholder input for topic nomination and development • Objective and scientifically rigorous • Identification of methodological standards • Careful consideration of potential conflicts of interest • Ensure balanced approach to topic and research • Open for public participation and transparent • Accountable • Clear documentation of processes • Stakeholder input for topic nomination, development, and review of draft reports Slutsky J, et al. Comparing medical interventions: AHRQ and the Effective Health Care Program. In: Methods guide for comparative effectiveness reviews. Available at: http://effectivehealthcare. ahrq.gov/ehc/products/60/318/2008_1118CompareInterventions.pdf.

8. Horizon Scanning Research Training Evidence Synthesis Evidence Generation Evidence Need Identification Career Development Dissemination and Translation Components of theEffective Health Care Program Stakeholder Input and Involvement

9. How Does the Effective Health Care Program Answer Real-World Questions? • Evidence Synthesis • Research reviewssummarize the scientific literature on comparative effectiveness questions and highlight areas for future research. • Comparative effectiveness and effectiveness reviews outline the effectiveness, benefits, and harms of treatment options. • Technical briefs describe the state of the literature for new or emerging health care tests or treatments. • Evidence Generation • New research reportsanalyze new data to answer comparative effectiveness questions. • Research on health care databases may be supported through the Effective Health Care Program, specifically through the DEcIDE Network or the CERTs. • Clinical comparative effectiveness trials will be supported through extramural grants. • Dissemination and Translation • Summary Guidestranslate findings from research reviews for patients, providers, or policymakers.

10. Who Are the Key Playersin the Effective Health Care Program? • Evidence-based Practice Centers (EPCs) • The John M. Eisenberg Center for Clinical Decisions and Communications Science • The Scientific Research Center (SRC) • Partners for new research: • Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) Network • The Centers for Education and Research on Therapeutics (CERTS) • Extramural grantees and other researchers • Stakeholders: • Formal Stakeholder Group • Key Informants • Technical experts • Public input • Task order officers at the Agency for Healthcare Research and Quality

11. Each Player on the Team Has a Different Role

12. The Game Plan AHRQ Task Order Officers Develop, coordinate, and monitor Scientific Resource Center coordinates topic selection/peer review/public input, provides research support Stakeholders Nominate topics, provide input on questions, reviews, and program Evidence-based Practice Centers Evidence Synthesis and Evidence Need Identification DEcIDE, CERTs, Grantees Evidence Generation Eisenberg Center Evidence Translation and Dissemination Patients, health care providers, and policymakers make better health care decisions

13. Engaging Stakeholders • Evidence-based Practice Center researchers directly engage stakeholders for specific topics or reports • Specific input through focus groups, key informant panels • Topic development and refinement • Determining Research needs • Public input through posting and comment periods • Draft Key Questions • Draft research reviews • Effective Health Care (EHC) Program members engage stakeholders for the Program as a whole • Open Web topic nominations • Citizen’s forum (in development) • Stakeholder groups provide input on EHC Program processes • Volunteer members for a 2-year term • Representatives of various stakeholder groups

14. Points of Stakeholder Engagementfor Systematic Reviews AHRQ = Agency for Healthcare Research and Quality; OCKT = Office of Communications and Knowledge Transfer; SRC = Scientific Resource Center

15. Evidence-based Practice Centers Conduct Research Reviews for a Variety of Purposes • Research reviews completed under the Effective Health Care and Generalist Programs are broad and balanced in scope: • Useful to a broad range of stakeholders • Scope determined by the Evidence-based Practice Center through topic development • Comparative effectiveness reviews • Effectiveness reviews • Technical briefs • Reviews requested by Federal partners may have specific purpose and scope as defined by the partner: • Center for Medicare and Medicaid Services (CMS), through the Technology Assessment Program • U.S. Preventive Services Task Force (USPSTF) • National Institutes of Health (NIH), through the Office of Medical Applications of Research (OMAR) • Other federal partners: AHRQ, Centers for Disease Control and Prevention (CDC), Food and Drug Administration (FDA), and other NIH centers

16. The Current Evidence-based Practice Centers • The Evidence-based Practice Center program was instituted in 1997. The following are the centers that were awarded contracts from the Agency for Healthcare Research and Quality in 2007. The contracts are competitive on a 5-year cycle. • Blue Cross and Blue Shield Association, Technology Evaluation Center, Chicago, IL • Duke University, Durham, NC • ECRI Institute, Plymouth Meeting, PA • Johns Hopkins University, Baltimore, MD • McMaster University, Hamilton, Ontario, Canada • Minnesota Evidence-based Practice Center, Minneapolis, MN • Oregon Evidence-based Practice Center, Portland, OR • RAND Corporation, Santa Monica, CA • RTI International-University of North Carolina, Research Triangle, NC • Tufts Medical Center, Boston, MA • University of Alberta, Edmonton, Alberta, Canada • University of Connecticut, Storrs, CT • University of Ottawa, Ottawa, Ontario, Canada • Vanderbilt University Medical Center, Nashville, TN

17. The Scientific Resource Center • The Scientific Resource Center (SRC) is currently* located at the Oregon Evidence-based Practice Center (EPC) and supports the activities of the Effective Health Care (EHC) Program by: • Communicating with stakeholders and assisting EPC members to ensure that EHC Program products meet the practical needs of providers, patients, and policymakers • Receiving and sorting topic nominations • Facilitating and coordinating the decisionmaking process for topic selection of research reviews • Coordinating peer review and public input for research reviews • Collaborating with EPC members to develop consistent and rigorous scientific methods in conducting research reviews. *Awarded by competitive bid in 2007.

18. The John M. Eisenberg Center for Clinical Decisions and Communications Science • The John M. Eisenberg Center for Clinical Decisions and Communications Science is currently* located at Baylor College of Medicine in Houston, Texas. • The Center works with products developed by Evidence-based Practice Center members to translate research reviews into short, easy-to-read guides that can be used by consumers, clinicians, and policymakers. • Involvement of the Center begins at topic development and continues until the guides are disseminated, generally within 6 months of the final research review. Early involvement helps to: • improve the usefulness of research products, and • optimize fidelity of translation. *Awarded by competitive bid in 2008.

19. Agency for Healthcare Researchand Quality Task Order Officers • Serve as the contracting officer technical representative for all Effective Health Care Program partners • Specific task order officer listed for each task order • Contact the task order officer for questions regarding: • Task order • Collaboration or coordination with other partners in the EHC and Evidence-based Practice Center Programs • Participate in calls and meetings for each research review step, from topic development through publication • Provide input on scientific, protocol, and process issues • Help EPC researchers identify opportunities for collaboration within the EHC Program to address common challenges and communicate best practices

20. American Recovery and Reinvestment Act–Funded Evidence-based Practice Center Activities • In 2009, eight Evidence-based Practice Centers were awarded additional task orders to: • Develop topics for potential research review within a given area of concentration that corresponds to AHRQ's priority conditions or cross-cutting themes • Conduct research reviews within a given area of concentration • Identify important future research that is needed to fill gaps and help patients, physicians, and policymakers make health care decisions within a given area of concentration • Develop methods for rigorous and transparent methods for determining specific study design and prioritizing research needs • Involve stakeholders, which include funders of new research and investigators who conduct new research

21. Areas of Concentration for the ARRA-Funded Evidence-based Practice Centers The eight Evidence-based Practice Centers (EPCs) that were awarded funding for 3 years through the American Recovery and Reinvestment Act (ARRA) in 2009 will focus on the following research areas: • Arthritis, functional limitations, and disability • Minnesota EPC • Cancer and infectious disease • Blue Cross and Blue Shield Association EPC • Cardiovascular and pulmonary disease • Duke University EPC • Mental health • RTI International-University of North Carolina EPC • Pregnancy, developmental delays, and autism • Vanderbilt University EPC • Diabetes, obesity, and peptic ulcer disease • Johns Hopkins University EPC • Health care delivery, prevention, and behavioral intervention • Oregon EPC • Diagnostics and devices • Tufts Medical Center EPC

22. Partners for New Research • Evidence-based Practice Centers work with researchers, funders, and other stakeholders to determine important research needs: • The DEcIDE Network • An AHRQ contract-funded network of research centers, primarily organized by consortia around selected priority conditions • Generates new comparative effectiveness evidence through analysis of existing health care databases and patient registries, and new data collection • Centers for Education and Research on Therapeutics (CERTs) • A network of research centers funded by AHRQ as cooperative agreements, each addressing different themes, including diseases, populations, data systems, and methodologies • Generate, translate, and disseminate new evidence on benefits and harms of therapeutics • AHRQ extramural grantees • AHRQ grant-funded researchers • Conduct clinical comparative effectiveness studies AHRQ = Agency for Healthcare Research and Quality DEcIDE = Developing Evidence to Inform Decisions about Effectiveness

23. The Current DEcIDE Network Centers* • Acumen, LLC, Burlingame, CA • Brigham and Women's Hospital, Boston, MA • Duke University, Durham, NC • Harvard Pilgrim Health Care, Harvard Medical School, Boston, MA • Johns Hopkins University, Baltimore, MD • Outcome Sciences, Cambridge, MA • RTI International, Research Triangle Park, NC • University of Colorado at Denver and Health Sciences Center, Aurora, CO • University of Illinois at Chicago, Chicago, IL • University of Maryland at Baltimore, Baltimore, MD • University of North Carolina at Chapel Hill, Chapel Hill, NC • University of Pennsylvania School of Medicine, Philadelphia, PA • Vanderbilt University Medical Center, Nashville, TN * Awarded by competitive bid in 2005 DEcIDE = Developing Evidence to Inform Decisions about Effectiveness

24. The Current CERTs Centers* • Clinical and economic issues in hospital settings — University of Chicago • Consumer education and patient adherence — University of Texas M.D. Anderson Cancer Center and Baylor College of Medicine • Therapies for heart and blood vessel disorders — Duke University Medical Center • Musculoskeletal disorders — University of Alabama • Drug interactions, women's health — University of Arizona/The Critical Path Institute • Drug use, safety, and effectiveness in managed care — HMO Research Network • Health information technology — Brigham and Women's Hospital • Anti-infective therapeutics use and resistance — University of Pennsylvania School of Medicine • Therapeutic medical devices — Weill Medical College of Cornell University • Mental health therapeutics — Rutgers, The State University of New Jersey • Therapies for older adults and the effects of aging — University of Iowa • Pediatric therapeutics — Cincinnati's Children's Hospital Medical Center • Tools for optimizing prescribing — University of Illinois at Chicago • Therapeutics for vulnerable populations — Vanderbilt University Medical Center • * Grants awarded by competitive bid in 2006 • CERTS = Centers for Education and Research on Therapeutics

25. AHRQ Training Modules forthe Systematic Reviews Methods Guide • The purpose of the training modules is: • To provide practical training on the Methods Guide for Conducting Comparative Effectiveness Reviews • The Methods Guide is: • Intended to improve the transparency, consistency, and scientific rigor of the Effective Health Care (EHC) Program • A collaborative effort of AHRQ, the Scientific Resource Center (SRC), and the Evidence-based Practice Centers (EPCs) • A living document with revisions as new evidence and understanding grows (initial draft posted in 2007) • Posted on the EHC Program Web site • Other materials available to new EPC investigators or staff: • The Methods Guide • Stakeholder training (soon to be available to EPCs) • Program materials, such as guides and templates (available to EPCs on secure site) • Learning Network (soon to be available to EPCs on secure site) • Methods library and weekly e-mail methods article alert (contact task order officer or SRC to be added to the listserv)

26. The Training Modules • Topic Refinement • Analytic Frameworks • Study Eligibility Criteria • Searching for Relevant Studies • When To Select Observational Studies as Evidence • Data Extraction • Assessing the Quality of Individual Studies • Assessing Applicability • Presentation of Findings • Quantitative Synthesis (I) • Quantitative Synthesis (II) • Grading Strength of Evidence • Reporting the Review

27. Systematic Review Process Overview • Analyze and Synthesize Studies • Assess the quality of individual studies • Assess applicability • Present findings • Synthesize quantitative data • Grade strength of evidence • Search for and Select Studies for Inclusion • Identify study eligibility criteria • Search for relevant studies • Select evidence for inclusion • Prepare Topic • Refine topic • Develop analytic framework Report Systematic Review Extract Data from Studies Identify gaps and future research needs

28. Important Terms and Concepts (I) Research Review or Evidence Synthesis Systematic Review Technical Brief Comparative Effectiveness Review Effectiveness Review

29. Important Terms and Concepts (II) • PICO ≈ PICO(TS) ≈ PICOTS ≈ PICOT ≈ PICOS • Population • Intervention • Comparison • Outcome • Timing of outcome assessment • Setting (sometimes study design)

30. Helpful Web Sites forEvidence-based Practice Center Investigators • www.effectivehealthcare.ahrq.gov • Public Web site of the Effective Health Care (EHC) Program: • EHC Program published methods papers, including the Methods Guide • EHC Program published research reviews and summary guides • Opportunities for collaborations through listserv notifications • Key question, draft report, and final report postings • https://www.kpchr.org/ehc/system/login.aspx • Secure site available only to Evidence-based Practice Center (EPC) investigators • Contact task order officer or Scientific Resource Center for access • Enable “remember me” function to improve usability and access • Policy and process documents, including templates and topic development working documents • Learning Network — a forum for asking questions and sharing experiences and ideas • Contact information for EHC Program and partners • http://mc.manuscriptcentral.com/ehc • Scholar One Manuscript Central — peer-review management system (contact task order officer or the Scientific Resource Center for access) • Upload draft and final documents through this site

31. References • Effective Health Care Program. Methods guide for effectiveness and comparative effectiveness reviews. Rockville, MD: Agency for Health Care Research and Quality, 2009. Available at: http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318. • Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. Washington, DC: U.S. Department of Health and Human Services; June 30, 2009. Available at: http://www.hhs.gov/recovery/programs/cer/cerannualrpt.pdf. • Helfand M, Balshem H. Principles in developing and applying guidance. In: Methods Reference Guide for Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality, Posted August 2009. Available at: http://effectivehealthcare.ahrq.gov/healthInfo. cfm?infotype=rr&ProcessID=60.

32. References • Slutsky J, Atkins D, Chang S, et al. Comparing medical interventions: AHRQ and the Effective Health Care Program. In: Methods guide for comparative effectiveness reviews. Rockville, MD: Agency for Health Care Research and Quality, Posted November 2008. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/60/318/2008_1118CompareInterventions.pdf. • Slutsky J, Atkins D, Chang S, et al. AHRQ series paper 1: comparing medical interventions: AHRQ and the Effective Health-Care Program. J Clin Epidemiol 2010;63:481-483.

33. Authors This presentation was prepared by Stephanie Chang, M.D., M.P.H., and Elisabeth Kato, M.D., M.R.P., at the Agency for Healthcare Research and Quality (AHRQ) Center for Outcomes and Evidence.