Choosing the Important Outcomes for a Systematic Review of a Medical Test

1. Choosing the Important Outcomes for a Systematic Review of a Medical Test Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Medical Test Reviews Methods Guide www.ahrq.gov

2. Overview of a Medical Test Review • Analyze and Synthesize Studies • Assess Risk of Bias as a Domain of Quality • Assess Applicability • Grade the Body of Evidence • Meta-analysis of Test Performance Evidence With a “Gold Standard” — or — • Meta-analysis of Test Performance Evidence With an Imperfect Reference Standard • Decision Modeling Extract Data From Studies • Prepare Topic • Develop the Topic and Structure the Review • Choose the Important Outcomes • Search for and Select Studies for Inclusion • Search for Studies Research Sources Report Medical Test Review Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

3. Learning Objectives • Understand which outcomes are relevant to the process of testing and to the results of the test. • Consider inclusion of outcomes in the five categories of the framework described in this module: clinical management effects; direct health effects; emotional, social, cognitive, and behavioral effects; legal and ethical effects; and costs. • Determine the group to which the testing outcomes are most relevant. • Prioritize the outcomes and select those to include, given resource limitations. Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

4. Overview • Decision-relevant outcomes areoutcomes that may affect the decision to use the test. • Outcomes may affect the balance of test benefits/risks. • A broad range of outcomes will be considered in this module for illustration. • This module discusses outcomes relevant to screening, diagnostic, genetic, and prognostic tests. (Genetic and prognostic tests are discussed further in modules 11 and 12.) Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

5. Challenges Specific to Choosing Important Outcomes Balancing resource limitations (time/money) with the utility of reviewing all possible outcomes: • Missing outcomes can skew the balance of benefits versus harms of testing or affect decisions made based on the results. • Attempting to review too many outcomes can lead to poor-quality reviews due to resource limitations. • A two-step approach is recommended to deal with this challenge. All Possible Outcomes Resource Limitations Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

6. Principles for Addressing the Challenges in Selecting Outcomes of Medical Tests • Catalog outcomes methodically. • Solicit input from stakeholders. Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

7. Principle 1:Catalog Outcomes Methodically (1 of 2) • The following is a useful framework for identifying outcomes of a test used in a clinical setting. • Outcomes are separated into those attributed to: • The actual testing process • Direct effects of the test • Knowledge of the test results • Patient’s response to the test results • How the patient and clinician act upon results • Others Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

8. Principle 1:Catalog Outcomes Methodically (2 of 2) • There are five categories of patient outcomes: • Clinical management • Direct health effects • Emotional, social, cognitive, and behavioral responses • Legal and ethical • Costs • Outcomes apply to both the testing process and the test results. • The relative importance of outcomes may differ based on whether a test is for screening, diagnosis, or prognosis. Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Bossuyt PMM, McCaffery K. Med Decis Making 2009 Sep-Oct;29(5):E30-8. PMID: 19726782.

9. Consequential Outcomes According toType of Medical Test Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

10. Examples of Outcomes Affected by Test Type • Example 1: High false-positive rates in screening tests (e.g., mammography) make adverse emotional effects and legal and ethical outcomes more significant. • Example 2: Clinical management is more important for prognostic and diagnostic testing (e.g., computed tomography for pulmonary emboli) due to therapies applied after a positive result (in this case, anticoagulation therapy). • Clinical management is often far removed from the screening step. • Example 3: Screening tests (e.g., PAP smears) are designed to be less invasive, so direct health effects are less (if at all) important. Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Berlin L, Berlin JW. AJR Am J Roentgenol 1995 Oct;165(4):781-8. PMID: 7676967.

11. Downstream Activities as Outcomes Consider downstream activities from test results: • Further evaluation: Leading to additional health care services and interventions (e.g., adrenal masses found during abdominal imaging.) • Patient anxiety: Leading to emotional and cognitive outcomes (e.g., degenerative changes in the spine seen on chest imaging.) Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

12. Cost as an Outcome Beyond direct costs of the test, consider: • Confirmatory testing of a positive result • Treatment costsif disease is detected • Treatment of adverse effects • From testing • From treatment of the disease • Cost to society (opportunity costs) Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

13. Identify the Population of Interest When Choosing Outcomes Also consider who actually experiences the outcome in addition to the patient: • Family members (e.g., when testing for heritable conditions) • Population affected by a diversion of resources (e.g., widespread newborn screening that diverts resources from smoking cessation activities.) • Society as a whole (e.g., prophylactic antibiotics or quarantine after exposure to an infectious individual.) • Payers that need to pay for treatments Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

14. Summary of Principle 1:Catalog Outcomes Methodically Mapping outcomes to the testing process and to the test results Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

15. Principle 2: Solicit Input From Stakeholders • Reasons to involve stakeholders in the review process: • Too many possible outcomes to include all • No universal guidelines for which outcomes to include • Choice of outcomes depends on stakeholders’ needs and their intended use of the review • Examples of stakeholders and their outcomes needs: • Clinicians and patients: Health effects of testing and emotional, behavioral, social, and cognitive outcomes • Professional societies (e.g., American College of Physicians): How clinical outcomes vary as a result of medical testing • Payers (e.g., the Centers for Medicare and Medicaid Services): Cost-shifting of testing costs and downstream costs Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Chapters available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm.

16. How To Choose Important Outcomes:Illustrative Examples • Four examples of choosing important outcomes in different testing scenarios: • Screening tests • Diagnostic tests • Prognostic tests • Genetic tests Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

17. Screening Tests • Screening tests are used to detect disease in asymptomatic individuals or individuals with unrecognized symptoms. • Screening tests should be employed when: • The test can accurately separate individuals with the disease from those without it. • There is a treatment available. • Early treatment improves outcomes. • The U.S. Preventive Services Task Force develops recommendations for screening tests. Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Wilson JM, Jungner YG. Principles and practice of mass screening for disease. Available at whqlibdoc.who.int/php/WHO_PHP_34.pdf.

18. Screening Test Example: Bacterial Vaginosis(1 of 4) Analytic Framework: Screening Test for Bacterial Vaginosis in Pregnant Women KQ1 • Reduction in adverse pregnancy outcomes: • Endometritis/neonatal sepsis • Low birthweight • Perinatal mortality • Preterm delivery • Preterm, premature rupture of membranes • Spontaneous abortion • Asymptomatic pregnant women:a. Low risk for • preterm delivery • Average risk for preterm delivery • High risk for preterm delivery Antibiotic treatment Screening KQ2 BacterialVaginosis KQ3 Adverse Effects Adverse Effects Nygren P, Fu R, Freeman M, et al. Evidence Synthesis No. 57. Available at www.uspreventiveservicestaskforce.org/uspstf08/bv/bves.pdf.

19. Screening Test Example: Bacterial Vaginosis(2 of 4) • Clinical Management Effects • Included in the systematic review: • Does screening reduce adverse pregnancy outcomes? • What are the effects of antibiotic treatment? • The reviewers could have considered: • Does screening lead to antibiotic treatment? • Direct Test Effects • Not included in the systematic review: • No harm (vaginal swab will not cause an injury) • No benefit (other than contact with clinicians) Nygren P, Fu R, Freeman M, et al. Evidence Synthesis No. 57. Available at www.uspreventiveservicestaskforce.org/uspstf08/bv/bves.pdf.

20. Screening Test Example: Bacterial Vaginosis(3 of 4) • Emotional, social, cognitive, and behavioral effects of testing: • Could have considered outcomes associated with screening but not the result of antibiotic therapy (e.g., emotional responses to positive tests or maternal worry.) • Legal and ethical effects of testing: • None specifically for bacterial vaginosis screening; however, bacterial vaginosis is often screened for with syphilis or HIV. • Therefore, consider whether the test may be administered along with other tests that do raise ethical issues. Nygren P, Fu R, Freeman M, et al. Evidence Synthesis No. 57. Available at www.uspreventiveservicestaskforce.org/uspstf08/bv/bves.pdf.

21. Screening Test Example: Bacterial Vaginosis(4 of 4) • Cost was not considered as an outcome in the analysis. • Directs costs to the patient (e.g., screening, antibiotic treatment) • Opportunity costs to the population • Analysis of affected parties was incomplete. • Effects on the mother and fetus/infant were considered. • Potentially relevant parties were not considered: • The mother’s partner • Society (due to potential antibiotic resistance from widespread testing/treatment) Nygren P, Fu R, Freeman M, et al. Evidence Synthesis No. 57. Available at www.uspreventiveservicestaskforce.org/uspstf08/bv/bves.pdf.

22. Diagnostic Tests • Diagnostic tests are differentiated from screening tests based on the population being tested. • Diagnostic tests are applied to confirm or refute a disease in a symptomatic person. • Screening tests are used in an asymptomatic/ presymptomatic person. • Organizations other than the U.S. Preventive Services Task Force are concerned with ensuring the safe use of diagnostic tests in patient populations. • Payers are also interested in optimizing the use of diagnostic tests, as many are costly. Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

23. Diagnostic Test Example: Computerized Tomography Versus Coronary Angiography (1 of 3) • Key questions: • What is the accuracy of computerized tomography (CT) versus conventional coronary angiography for coronary artery disease (CAD)? • Based on accuracy measures, can CT replace some coronary angiography for CAD diagnosis and assessment? • A broader review with sufficient resources would include: • Effectiveness of CT angiography. • The full range of outcomes for both tests . Mowatt G, Cook JA, Hillis GS, et al. Heart 2008 Nov;94(11):1386-93. PMID: 18669550.

24. Diagnostic Test Example: Computerized Tomography Versus Coronary Angiography (2 of 3) • Clinical management effects • May recommend medications, angioplasty, bypass surgery, or lifestyle changes. • Depend on the number of occluded arteries detected by each test. • Direct test effects*: • Allergic reaction • Renal failure • Radiation exposure *Coronary angiography has comparable or greater risks, including a procedural risk that computerized tomography does not have. Mowatt G, Cook JA, Hillis GS, et al. Heart 2008 Nov;94(11):1386-93. PMID: 18669550.

25. Diagnostic Test Example: Computerized Tomography Versus Coronary Angiography (3 of 3) • Emotional, cognitive, and behavioral effects: • Results of testing may lead to lifestyle changes, depression, or risky behavior. • Similar effects are expected for both tests. • Legal and ethical effects: • There are potential legal consequences if the patient’s profession requires disclosure of health threats (e.g., he or she is a pilot). • Effects are not expected to differ between the tests. • Costs: • Cost of tests to patients and payers • Opportunity costs Mowatt G, Cook JA, Hillis GS, et al. Heart 2008 Nov;94(11):1386-93. PMID: 18669550.

26. Prognostic Tests • Prognostic tests are used in individuals with known disease to predict outcomes. • The procedure may be the same as screening or diagnostic tests, but − • Intent is different. • Population is different. • Application of the results is different. Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

27. Prognostic Test Example: Spirometry for Chronic Obstructive Pulmonary Disease Prognosis (1 of 3) • Spirometry is commonly used to: • Diagnose chronic obstructive pulmonary disease. • Monitor response to treatment. • Can it also be used to predict survival? • A systematic review conducted by the Minnesota Evidence-based Practice Center in 2005 addressed this topic. • Stakeholders included: • American Thoracic Society • American College of Physicians • American Academy of Family Physicians • American Academy of Pediatrics Wilt TJ, Niewoehner D, Kim C-B, et al. Evidence Report/Technology Assessment No. 121. Available at www.ncbi.nlm.nih.gov./books/NBK37773.

28. Prognostic Test Example: Spirometry for Chronic Obstructive Pulmonary Disease Prognosis (2 of 3) • Clinical management effects: • Alter course of treatment, such as to avoid respiratory failure • Other interventions, such as omitting treatments of other less imminent conditions • Direct test effects: negligible, unless patients are challenged with methacholine. • Emotional, social, cognitive, and behavioral effects • Emotional effects of learning prognosis may lead to behavioral effects, such as efforts to alter prognosis. Wilt TJ, Niewoehner D, Kim C-B, et al. Evidence Report/Technology Assessment No. 121. Available at www.ncbi.nlm.nih.gov./books/NBK37773.

29. Prognostic Test Example: Spirometry for Chronic Obstructive Pulmonary Disease Prognosis (3 of 3) • Legal and ethical effects: possible legal effects if the individual’s actions are influenced by concealed information about his or her prognosis. • Costs:costs of the test, relative to costs of collecting information via a history and physical examination, could be of interest to stakeholders. Wilt TJ, Niewoehner D, Kim C-B, et al. Evidence Report/Technology Assessment No. 121. Available at www.ncbi.nlm.nih.gov./books/NBK37773.

30. Genetic Testing Considerations pertaining to genetic tests: • Effect on family members: • Direct bearing on family members who share genes. • May affect emotional, behavioral, and ethical outcomes (e.g., family members may feel pressured to get tested to provide more information for the rest of the family.) • Possible impact on health insurance eligibility: • Recent U.S. legislation prohibits using genetic information to reject health insurance coverage. • This may not be the case in other countries. Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

31. Key Messages (1 of 2) • Medical test reviews must consider outcomes relevant to: • The testing process. • Test results. • Consider outcomes related to: • Clinical management based on test results. • Direct test effects. • Emotional, social, cognitive, and behavioral responses to testing. • Legal and ethical considerations. • Costs. Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

32. Key Messages (2 of 2) • Consider to whom the desired outcomes are most relevant. • Prioritize outcomes for inclusion with engagement of the stakeholders, given resource limitations. Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

33. Practice Question 1 (1 of 2) • Five categories of outcomes were described for Principle 1, Categorize Outcomes Methodically. Which of the following is not one of them? • Clinical management • Direct health effects • Legal and ethical • Patient preference • Costs

34. Practice Question1 (2 of 2) Explanation for Question 1: The correct answer is d. Patient preference was not discussed. The other categories of outcomes that were discussed are emotional, social, cognitive, and behavioral responses.

35. Practice Question 2 (1 of 2) • Screening tests are typically used in asymptomatic individuals or individuals with unrecognized symptoms to detect as many different types of diseases as possible. • True • False

36. Practice Question 2 (2 of 2) Explanation for Question 2: The statement is false. Screening tests should not be employed to detect as many diseases as possible, but rather only when there is a treatment available and early treatment improves outcomes. Additionally, screening tests may not be appropriate when the screening test is associated with high risk, or if the test is expensive, or if the burden of disease is low, or if other conditions exist that render screening potentially more harmful than beneficial.

37. Practice Question 3 (1 of 2) • Typically, input at the early stages of a systematic review is solicited from which of the following groups: • Clinicians • Patients • Professional societies • Manufacturers of tests or treatments • Payers Choices: • i, ii, and iii • i, ii, iii, and iv • i, ii, iii, and v • All of the above

38. Practice Question 3 (2 of 2) Explanation for Question 3: The correct answer is c. Manufacturers of tests or treatments are not typically considered stakeholders whose input is solicited in the early stages (design and conduct) of a systematic review.

39. Practice Question 4 (1 of 2) • Which of the following is/are useful cost considerations, in addition to direct costs of testing? • Costs associated with access to the testing location (e.g., travel costs) • Cost of treating comorbidities • Cost of treating the adverse effects of testing • Cost to manufacture the test • All of the above • None of the above

40. Practice Question 4 (2 of 2) Explanation of Question 4: The correct answer is c. The test itself may result in adverse effects that must be treated and, therefore. generate a cost. Downstream harms resulting from additional testing or treatment may also be incurred, resulting in additional costs. Costs associated with patient travel, treatment of comorbidities, and manufacture of the test are not considered to be part of the testing process in this module.

41. Practice Question 5 (1 of 2) • When determining who experiences the outcome of testing, which of the following groups should be considered? • Family members • Population affected by a diversion of resources • Society as a whole • Payers • All of the above • None of the above

42. Practice Question 5 (2 of 2) Explanation for Question 5: The correct answer is e. The individual being tested is not the only person who experiences testing outcomes. Family members can experience outcomes when a heritable condition is tested for; a population can be affected by resources being diverted towards testing and away from other needed services; society can be affected by public health interventions such as the dissemination of antibiotics to those in an affected area; and payers are affected when they need to pay for a newly diagnosed condition.

43. Authors • This presentation was prepared by Brooke Heidenfelder, Rachael Posey, Lorraine Sease, Remy Coeytaux, Gillian Sanders, and Alex Vaz, members of the Duke University Evidence-based Practice Center. • The module is based on Chapter 3, Choosing the Important Outcomes for a Systematic Review of a Medical Test. In: Methods Guide for Medical Test Reviews. AHRQ Publication No. 12-EC017. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. www.effectivehealthcare.ahrq.govmedtestsguide.cfm

44. References (1 of 3) • Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality; April 2012. AHRQ Publication No. 10(12)-EHC0653-EF. Chapters available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm. • Berlin L, Berlin JW. Malpractice and radiologists in Cook County, IL: trends in 20 years of litigation. AJR Am J Roentgenol. 1995 Oct; 165(4):781-8. PMID: 7676967. • Bossuyt PM, McCaffery K. Additional patient outcomes and pathways in evaluations of testing. Med Decis Making. 2009 Sep-Oct;29(5):E30-8. PMID: 19726782. • Mowatt G, Cook JA, Hillis GS, et al. 64-Slice computed tomography angiography in the diagnosis and assessment of coronary artery disease: systematic review and meta-analysis. Heart. 2008 Nov;94(11):1386-93. PMID: 18669550.

45. References (2 of 3) • Nygren P, Fu R, Freeman M, et al. Screening and treatment for bacterial vaginosis in pregnancy: systematic review to update the 2001 U.S. Preventive Services Task Force recommendation. Evidence Synthesis No. 57 (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-02-0024, Task Order No. 1). Rockville, MD: Agency for Healthcare Research and Quality; January 2008. AHRQ Publication No. 08-05106-EF-1. www.uspreventiveservicestaskforce.org/uspstf08/bv/bves.pdf. • Segal JB. Choosing the important outcomes for a systematic review of a medical test. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 3.1-13. AHRQ Publication No. 12-EHC017. www.effectivehealthcare.ahrq.gov/ medtestsguide.cfm.

46. References (3 of 3) • Wilson JM, Jungner YG. Principles and practice of mass screening for disease. Geneva, Switzerland: World Health Organization; 1968. WHO Public Health Papers No. 34. whqlibdoc.who.int/ php/WHO_PHP_34.pdf. • Wilt TJ, Niewoehner D, Kim C-B, et al. Use of Spirometry for Case Finding, Diagnosis, and Management of Chronic Obstructive Pulmonary Disease (COPD). Evidence Report/Technology Assessment No. 121 (Prepared by the Minnesota Evidence-based Practice Center under Contract No. 290-02-0009). Rockville, MD: Agency for Healthcare Research and Quality; September 2005. AHRQ Publication No. 05-E017-2. www.ncbi.nlm.nih.gov/books/NBK37773.