The G oals of G uidance: Maximizing Guideline Value and Benefit

1. The Goals of Guidance:Maximizing Guideline Value and Benefit • J. Sanford (Sandy) Schwartz, MD • Leon Hess Professor of Medicine andHealth Management & Economics • Perelman School of Medicine &Wharton School of Business • University of Pennsylvania

2. Extramural Activity Disclosure (12 month) • Consultant • Association of University Radiologists/American College of Radiology • Bayer • Christiana Care • Intermountain Health Care • Mathematica • Pfizer • Research Support • National Institutes of Health (NIA, NCI, NHLBI, NIDDK) • Pfizer • Robert Wood Johnson Foundation • Board of Directors • ABC House of Lower Merion • ECRI Institute • Jewish Social Policy Action Network (JSPAN) • National Scientific Advisory Committee Member/Consultant • BlueCross/BlueShield Associations (Medical Advisory Panel) • Centers for Medicare and Medicaid Services (Medicare Evidence Development and Coverage Advisory Committee) • Institute of Medicine National Academy of Sciences • National Institutes of Health (Advisory committees clinical practice guidelines, comparative effectiveness research, value of information analysis) • US Preventive Services Task Force

3. Clinical and Policy Guideline DevelopmentPersonal Perspective • • American College of Physicians Clinical Efficacy Assessment Project, Founding Director • • NIH Consensus Development Conferences • • National Heart Attack Alert Program • • Blue Cross/Blue Shield Technology Evaluation Center Medical Advisory Panel (MAP) • • US Preventive Services Task Force • • CMS Medicare Evidence Development Coverage Advisory Committee (MEDCAC) • • NHLBI ATP III, ATP IV, Integrated CVD guideline panels

4. Summary • • Guidelines should be patient centered, focusing on individual patient decision making needs • • While we have made much progress over the past 30+ years, the evidence base for ‘evidence based guidelines’ must be improved • – ‘Fit for Purpose – broader meaningful incorporation of data and information generated by study designs beyond the excessive focus on RCTs • – Outcomes measured (preferences, costs) and how (time horizon, settings, behavior) • – Analytical methods (validated)

5. Clinical Practice Guidelines • “ … statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options.” • – Institute of Medicine, 2011 • Clinical Practice Guidelines We Can Trust

7. What Will Guidelines Do? • • Convert science-based knowledge to clinical action and connect outcomes research to clinical practice • • Allow respected professional judgment to advise the researcher on what needs to be done • • Clarify medical care choices for the consumer and make explicit different standards of care where they exist • • Strengthen link between quality and management of health care AHRQ conference presentation, 1994?

8. Potential Guideline Targets • • Patients • • Clinicians • • Purchasers/payers • • Policy makers • To informedecisions that will improve health care at both the individual and populationlevels

9. Population Paradoxes • Large benefit to population may provide limited benefit to individuals • Large benefit to an individual may have a small population impact • Modest benefit to an individual may be wiped out by a small harm to many • Discounting is counter to behavioral preferences • Social benefits may not be attractive to the individual, even if beneficial on a social/community basis

10. Clinical Practice Guidelines:Patient-Centered Outcomes Research Working Definition Informs patient–focused questions targeted toward individual patient needs: • Given my personal characteristics, conditions and preferences, what should I expect will happen to me? • What are my options and what are the benefits and harms of those options? • What can I do to improve the outcomes that are most important to me? • How can the health care system improve my chances of achieving the outcomes I prefer? DRAFT

11. “I have yet to see any problem, however complicated, which … looked at it in the right way, did not become still more complicated” – Poul Anderson

12. “Absence of evidence of effectiveness” increasingly will be interpreted as “evidence of absence of effectiveness”

13. Clinical Guidelines: Requirements •Patient centered • Evidencebased • Outcomes focused • Cost–effective

14. Clinical Guidelines: Challenges • Information gaps • Too narrow framing – Conditions/Risks – Measures – Methods – Outcomes – Time horizons

15. Evaluation of Medical Care Safety Side effects acceptable? Efficacy Can it work? Effectiveness Does it work?Balance of benefits/harms?Better than alternatives?Better identifiable subgroups? Cost–Effectiveness Is there sufficient value? Efficacy: Net benefit optimalconditions Effectiveness: Net benefit average conditions

17. Assessment of Medical Interventions Assessment of the most effective and efficient care, defined in terms of patient outcome and cost. • How much benefit and value? • In which patients? • Under which conditions?

18. “… it’s important to understand the methods behind the research in order to know whether or not the results are useful to you. If the wrong methods are used or if the right methods are misused, research results won’t lead to better decisions but instead could cause greater confusion.” Sherine Gabriel, MD, MSc; Sharon-Lise Normand, PhD. PCORI Methodology Committee, 2012

19. Randomized Clinical Trials:The “Efficacy” Reference Standard Random Rx allocation protects internal validity • Internal validity • Adjustment for confounders • Data reliability Random treatment allocation protects internal validity, valid estimates of causal effects and etiologic relationships for the sample at hand

20. Randomized Clinical Trial:Clinical Scenario • High validity required • Significant selection bias / confounding • Ethical issues not prevent randomization (e.g., clinical equipoise) • Sufficient resources (subjects; funds) • Clinically relevant outcomes measurable • Time frame adequate RE: clinically relevant outcomes; can await results; will still be relevant

21. Randomized Clinical Trials:Representativeness and Data Limitations • Representativeness/Selection biases • • Eligibility criteria; Clinically–relevant subgroups • • Interventions compared • • Population studied (clinical/demographic characteristics; practice settings) • Data Limitations • • Data quality (informative censoring; crossover; protocol violations; missing data) • • Outcomes assessed (clinical relevance; range) • • Time horizon / Follow–up • • Resource use/ Cost

22. “The paradox of the clinical trial isthat it is the best way to assesswhether an intervention works,but arguably the worst way to assesswho will benefit from it.” Mant. Lancet. 1999;353:743–746.

23. “The benefit or harm of most treatmentsin clinical trials can be misleading andfail to reveal the potentially complex mixture of substantial benefits for some,little benefit for many, andharm for few.” – R Kravitz, Milbank Quarterly, 2004

24. Randomized Clinical Trials:Capacity and Logistical Challenges • • Ethical issues often interfere with randomization • • Insufficient resources • – Too many questions • – Science changes too frequently • – Not enough subjects • – Not enough money • – Time horizons too long

25. What do I think aboutcomparative effectiveness,RCT evidence basedclinical practice guidelines?

26. – Mahatma Gandhi • [when asked what he thought • of Western civilization] “I think it would be a good idea.”

27. “Without major changes in how weconceive, design, conduct, and analyze RCTs,the nation risks spending large sums of money inefficiently to answer the wrong questions— or the right questions too late.” Luce BR, Kramer JM, Goodman SN, Connor JT, Tunis S, Whicher D, Schwartz JS. Rethinking randomized clinical trials for comparative effectiveness research: The need for transformational change. Annals of Internal Medicine. 2009;151:206-209.

28. “Decisions about the use of therapeutic interventions, whether for individuals or entire healthcare systems, should be based on the totality of available evidence. The notion that evidence can be reliably or usefully placed in ‘hierarchies’ is illusory. Rather, decision makers need to exercise judgement about whether (and when) evidence gathered from experimental or observational sources is fit for purpose.” Rawlins M. De Testimonio – On the evidence for decisions about the use of therapeutic interventions. Lancet. 2008;372:2152–2161.

29. Assessment of Medical Interventions:Fit for purpose • Start with how information will be used • Identify gaps / required evidence generation – Clinically relevant patients / subpopulations – Clinically relevant, validated outcomes • Sequence /integrate analytic approach within and across studies • Systematically, explicitly analyze incremental tradeoffs/uncertainties across subgroups, relevant outcomes and studies • Bound summary estimates/CI; Identify conditions/variables that drive decisions

30. Observational Studies • Estimate causal interpretations from data constructed for other purposes • Random probability sample selection protects external validity • Outcomes not able to be adequately addressed in a timely fashion by RCTs – High need for representativeness that does not perturb “routine care” – Confident confounding can be adjusted adequately statistically – Outcomes adequately measured

31. Analysis of Medical Interventions:Required Considerations • Clinically relevant patients and subpopulations (baseline risk; outcomes; preferences) and study settings • Clinically relevant, validated outcomes (mortality/morbidity; PROs; preferences, economic) • Clinically relevant settings, behaviors, time horizons • Explicit assessment incremental tradeoffs/uncertainties for relevant populations/outcomes • Improved methods RE: indirect comparisons; confounding adjustment • Transparent, flexible, clinically meaningful/relevant assumptions and models • Summary estimates; Bounded estimates/CI; Identification of conditions/variables that drive decisions

32. Analysis of Medical Interventions:Required Considerations Broad range of empiric data and methods • Experimental (RCT; pragmatic/practical) • Observational (case–control, cohort, registry, administrative claims, EMR, clinical networks) • Synthesis (meta–analysis; systematic review) • Integration (formal, structured, transparent, expanded, validated, multi–factorial outcomes models)

33. Challenges Developing the Evidence Base • • Serious errors in clinical practice may result from overreliance on narrowly focused RCTs, observational studies, data synthesis or modeling • • Choice of study design and analytic approaches involves tradeoffs among limitations inherent to each and must be optimized for research question • • Consistency across methods and cross–validation confirmatory; disagreement requires understanding

34. Formal Consideration of Patient Preferences and Utilities • Potential Approaches/Opportunities • • Explicit quantification for outcomes about which patients care • • Model to assess sensitivity of recommendations to alternative preferences • Challenges: • • Whose utilities? • • Distribution of utilities • • Dissimilar trade-offs • • Misinformation/ “Popularity paradox”

35. Clinical Decision Making:Threshold Theory Test Treatment Threshold Threshold No Rx TEST Rx 0 Probability of Disease 1 Pauker SG, Kassirer JP. The threshold approach to clinical decision making. N Engl J Med. 1980;3022:1109-1117.

38. Life is full of calculated risks…

39. Occupational Risks

40. Avoidable Risks

41. Unexpected Risks

42. “All policy decisions should be based onabsolute measures of risk;relative risk is strictly for researchers only.” – Geoffrey Rose Professor Epidemiology London School of Hygiene and Tropical Medicine Absolute vs. Relative Risk

43. “… judgment cannot be excised from the process of evidence synthesis and that the variation of this judgment among experts generates uncertainty just as real as the probabilistic uncertainty of statistical calculations.” Goodman SN. The mammography dilemma: A crisis for evidence-based medicine? Ann Intern Med. 2002;137:363-365.

44. Evidence Based Medicine Is Not Value Free • • Harms and benefits involve comparison of dissimilar outcomes, often from different data sources with different populations, biases, ascertainment methods and time horizons • • Subjective expertise – locus of control shifted from physician to methodologist

45. "Not everything that can be counted counts; not everything that counts can be counted.” – Albert Einstein

46. The Pirate’s Code Elizabeth: Wait! You have to take me to shore. According to the Code of the Order of the Brethren... Barbossa: First, your return to shore was not part of our negotiations nor our agreement so I must do nothing. And secondly, you must be a pirate for the pirate's code to apply and you're not. And thirdly, the code is more what you'd call "guidelines" than actual rules.

47. “In the midst of every challenge lies opportunity” – Albert Einstein