Prepared for: Agency for Healthcare Research and Quality (AHRQ) ahrq

1. Introduction to the User’s Guide for Developing a Protocol for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) www.ahrq.gov

2. What Is Comparative Effectiveness Research? • Comparative effectiveness research is the “generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve delivery of care.” • Comparative effectiveness research is used to “assist consumers, clinicians, purchasers, and policymakers to make informed decisions that will improve care both at the individual and the population levels.” • Institute of Medicine. Initial national priorities for comparative effectiveness research. Available at www.iom.edu/Reports/2009/ComparativeEffectivenessResearchPriorities.aspx.

3. Background • There is a critical need for comparative effectiveness research in health care. • It is estimated that more than half of treatments lack valid evidence of effectiveness (long-term and patient-centered/quality-of-life outcomes). • Results of randomized controlled trials are not always applicable to real-world settings. • Without patient-relevant and unbiased comparative effectiveness research across the range of potential users, uncertain outcomes occur. • Typically most common in groups with the greatest needs (the elderly, the disabled, and patients with complex conditions) • Can lead to overuse, underuse, and off-label/risky use (unknown balance of benefits/harms) • Data on comparative effectiveness are needed to determine if new therapies perform better, worse, or the same as existing treatment options in real-world settings.

4. Evidence-Based Health Care • In evidence-based health care, decisions are based on the best available scientific evidence for an individual patient and his/her values. • Policy level — for example, decisions focused on Medicare/Medicaid (may include cost/resource availability information) • Gathering knowledge helps inform treatment decisions and policymaking, yet evidence gaps can still occur, and poor-quality research can invalidate conclusions. • To address the need for outcomes research, Section 1013 of the Medicare Prescription Drug, Improvement, and Modernization Act (MMA) authorized comparative effectiveness research studies by the Agency for Healthcare Research and Quality (Effective Health Care Program). • Patient-Centered Outcomes Research Institute (PCORI) • Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) Research Network

5. Aims of the User’s Guide in Designing Observational Comparative Effectiveness Research Protocols • The study protocol is one of the most important components of designing research. • Provides transparency and improves reproducibility, which can increase credibility and validity • There are few standards for elements to be included in observational study protocols, when compared with randomized controlled trials. • Researcher and stakeholder collaborations on protocol design allow relevant, applicable results that may inform stakeholder decisions/actions (and minimize biases). • The User’s Guide aims to identify both minimal standards and best practices for designing observational comparative effectiveness research studies.

6. Goals for the User’s Guide • Support the development of scientifically rigorous observational research that produces valid new knowledge and reduces uncertainty about the effects of interventions on patient health outcomes • Increase the collaboration between researchers, patients, and other decisionmakers in designing valid studies that generate new scientific evidence for informing health care decisions • Increase the transparency of methodologies and study designs that are used in comparative effectiveness and patient-centered outcomes research • Improve the quality and consistency of research by eliminating or reducing inappropriate variation in the design of studies • Stimulate researchers and stakeholders to consider important principles when designing a comparative effectiveness study and writing a study protocol

7. User’s Guide Overview • Study Objectives and Questions • Study Design Considerations • Estimation and Reporting of Heterogeneity of Treatment Effects • Exposure Definition and Measurement • Comparator Selection • Outcome Definition and Measurement • Covariate Selection • Selection of Data Sources • Study Size Planning • Considerations for Statistical Analysis • Sensitivity Analysis

8. Conclusions • The User’s Guide provides principles for designing observational comparative effectiveness research studies that will inform the health care decisions of patients and other stakeholders. • The User’s Guide is intended to serve as a reference for investigators and stakeholders in thinking through the trade-offs of key issues when designing a new research study. • The goal of the User’s Guide is to support development of scientifically rigorous research that provides new knowledge for informing health care decisions and protects against the introduction of bias into the research.