1 / 37

Introduction to the Methods Guide for Medical Test Reviews

Introduction to the Methods Guide for Medical Test Reviews. Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Medical Test Reviews Methods Guide www.ahrq.gov. Learning Objectives. Describe the purpose of the Methods Guide for Medical Test Reviews.

Faraday
Download Presentation

Introduction to the Methods Guide for Medical Test Reviews

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Introduction to the Methods Guide for Medical Test Reviews Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Medical Test Reviews Methods Guide www.ahrq.gov

  2. Learning Objectives • Describe the purpose of the Methods Guide for Medical Test Reviews. • Understand the unique features of medical test reviews. • Review the structure of the Methods Guide for Medical Test Reviews. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  3. Medical Test Reviews (1 of 4) • Systematic reviews of medical tests help determine which tests can be relied on to improve health outcomes and the circumstances required for the tests to achieve this. • Systematic reviews developed under the auspices of the Evidence-based Practice Center (EPC) Program: • Need to be technically comprehensive and practical • Need to be completed with limited time and resources Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  4. Medical Test Reviews (2 of 4) • The Methods Guide for Medical Test Reviews: • Provides practical guidance for those who prepare and use systematic reviews. • Promotes consistency and rigor in how specific issues are addressed. • Complements the Methods Guide for Effectiveness and Comparative Effectiveness Reviews developed by the Agency for Healthcare Research and Quality. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Agency for Health Care Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm.

  5. Medical Test Reviews (3 of 4) • Medical tests are defined here as any test used in a health care context, for example, screening, diagnosis, or prognosis. • Using medical tests depends on the context. • Pre-existing conditions • Results of other tests • Skill and knowledge of providers • Availability of therapeutic resources • And others Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  6. Medical Test Reviews (4 of 4) • Researchers have tended to focus on the ability of medical tests to: • Conform to technical specifications • Classify patients into diagnostic or prognostic categories • Influence thought or actions by clinicians and patients • Rarely are medical tests evaluated in randomized controlled trials with representative patient populations and comprehensive measures of patient-relevant outcomes. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  7. Development of the Methods Guide for Medical Test Reviews (1 of 3) • Creation of the Methods Guide for Medical Test Reviews was guided by two tenets: • Evaluation of a medical test must be linked to the context of its use • Medical tests are about improving patient outcomes Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Agency for Health Care Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Available at www.effectivehealthcare.ahrq.gov/methodsguide.cfm.

  8. Development of the Methods Guide for Medical Test Reviews (2 of 3) • The first tenet is to consider the context. • Medical tests are not neutral reporters of reality.The context often affects sensitivity, specificity, and the clinical implications of a particular test result. • Therefore, systematic reviews must specify the context in which the test is being used. • The second tenet is to focus on patient outcomes. • The ultimate purpose of medical tests is to improve patient outcomes. • Improvement occurs by guiding the judgments of clinicians concerning disease state or treatment response. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  9. Development of the Methods Guide for Medical Test Reviews (3 of 3) • Most of the published literature: • Fails to address the clinical impact of tests. • Focuses instead on test development and performance characteristics. • The first step in a medical test review must be to establish a link between the use of a test and patient outcomes. Causal Chain Diagram Test  Result  Categorization* Decision  Patient Outcome *For example: high risk, disease present, disease progression Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  10. Medical Tests Overview (1 of 2) • Typical uses of medical tests are to: • Screen for the likelihood of a disorder. • Diagnose the presence of disease. • Assess response to treatment. • Assess probability of desirable/undesirable consequences. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  11. Medical Tests Overview (2 of 2) • Typical types of medical tests: • Laboratory tests • Radiological imaging • Patient history • Physical examinations • Scored questionnaires • For screening • For determining likely prognosis • For assessing the potential response to therapy Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  12. Challenges Specific to Medical Tests (1 of 2) • Assessing the impact of medical tests can be challenging. • Medical tests have several steps between conducting the test and determining the outcome of clinical importance. • Conversely, most treatments lead directly to intended results or adverse effects. • Consequences result from the less direct relationship between medical tests and patient outcomes. • Medical tests are often evaluated in isolation for their ability to detect a substance or a particular anatomic condition. • Medical tests are not evaluated in terms of their impact on overall health outcomes. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Siebert U. Eur J Health Econ 2003;4(3):143-50 Tatsioni A, Zarin DA, Aronson N, et al. Ann Intern Med 2005 Jun 21;142(12 Pt 2):1048-55. PMID: 15968029.

  13. Challenges Specific to Medical Tests (2 of 2) • Randomized controlled trials, generally considered the strongest form of evidence, are rarely performed for medical tests. • Trials are difficult to carry out because of the numerous steps and the variables between the test and patient outcomes. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  14. Using Medical Tests (1 of 2) • Medical tests can be likened to an electronic signal detector for the presence or absence of a clinical condition. • Signal detection theory (e.g., sensitivity, specificity) can be used to calculate disease probabilities for positive or negative test results. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Green DM, Swets JA. Signal detection theory and psychophysics. New York, NY: Wiley; 1966. Reprinted 1988. Ledley RS, Lusted LB. Science 1959 Jul 3;130(3366):9-21. PMID: 13668531. Yerushalmy J. Public Health Rep 1947 Oct 3;62(40):1432-49. PMID: 20340527.

  15. Using Medical Tests Medical test evaluation studies fall along a continuum related to the study objectives. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Ɩ Fryback DG, Thornbury JR. Med Decis Making 1991 Apr-Jun;11(2):88-94. PMID: 1907710.

  16. Analytic Frameworks (1 of 3) • Reviewers need an explicit strategy for summarizing complex literature in a logical way to respond to key questions. • Frameworks can be used to: • Maintain an orderly process. • Clarify questions. • Organize evidence. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Bravata EP, McDonald KM, Shojania KG, et al. Ann Intern Med 2005 Jun 21;142(12 Pt 2): 1056-65. PMID: 15968030. Mulrow C, Langhorne P, Grimshaw J. Ann Intern Med 1997 Dec 1;127(11):989-95. PMID: 9412305. Whitlock EP, Orleans CT, Pender N, et al. Am J Prev Med 2002 May;22(4):267-84. PMID: 11988383.

  17. Analytic Frameworks (2 of 3) • Analytic frameworks: • Depict graphically the linkages that relate interventions and outcomes. • Identify questions to help structure the literature review. • Provide an evidence map after the review for identifying gaps and weakness in the evidence. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Agency for Healthcare Research and Quality. U.S. Preventive Services Task Force Procedure Manual. Available at www.uspreventiveservicestaskforce.org/uspstf08/methods/procmanual.htm.

  18. Analytic Frameworks (3 of 3) • An analytic framework has two key components: • A typology for describing the context in which the test is used: • Fits with the PICOTS (population, intervention, comparator, outcomes, timing, and setting) typology (the current standard approach). • A visual representation of the relationship between the application of the test/treatment and the outcomes of importance to decisionmaking. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  19. PICOTS Typology (1 of 3) • PICOTS stands for population, intervention, comparator, outcome, timing, and setting. • This typology was developed to describe the context in which medical interventions might be used. • It is important fordefining the key questions of a review. • It is important for assessing whether or not a given study is applicable. • Variations in the typology exist, but the standard unchanging elements are PICO. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Chalmers I, Hedges LV, Cooper H. Eval Health Prof 2002 Mar;25(1):12-37. PMID: 11868442.

  20. PICOTS Typology (2 of 3) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  21. PICOTS Typology (3 of 3) Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  22. Organization of the Methods Guide for Medical Test Reviews • Module 1 – Introduction • Module 2 – Topic Development and Structure • Module 3 – Choosing Outcomes • Module 4 – Search Strategies • Module 5 – Risk of Bias • Module 6 – Applicability • Module 7 – Grading Strength of Evidence • Module 8 – Meta-analysis With a “Gold Standard” • Module 9 – Meta-analysis Without a “Gold Standard” • Module 10 – Decision Modeling • Module 11 – Genetic Tests • Module 12 – Prognostic Tests Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  23. Overview of the Flow of Modules • Analyze and Synthesize Studies • 5. Assess Risk of Bias as a Domain of Quality • 6. Assess Applicability • 7. Grade the Body of Evidence • 8. Meta-analysis of Test Performance Evidence With a “Gold Standard” —or — • 9. Meta-analysis of Test Performance Evidence With an Imperfect Reference Standard • 10. Decision Modeling Extract Data From Studies • Prepare Topic • 2. Develop the Topic and Structure the Review • 3. Choose the Important Outcomes • Search for and Select Studies for Inclusion • 4. Search for Studies Research Sources • Encompassing the Entire Process: • 11. Genetic Tests • 12. Prognostic Tests Report Medical Test Review Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  24. Summary • Evaluation of medical tests presents challenges: • There are several steps between a medical test and outcomes of interest (indirect impact). • Context has great importance. • Few randomized controlled trials are aimed at comparing clinical outcomes of different tests and test strategies. • Available guidance provides some suggestions: • Use the PICOTS (population, intervention, comparator, outcomes, timing, and setting) topology to clarify the context. • Use an organizing framework for classifying the types of studies and their relationship to key questions. Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  25. Practice Question 1 (1 of 2) • How are analytic frameworks used in the context of this module? • To graphically depict the specific questions that need to be answered by the literature review. • To list the challenges associated with a review of a medical test. • To demonstrate recurrent themes from the literature. • To display all areas of potential bias.

  26. Practice Question 1 (2 of 2) Explanation for Question 1: The correct answer is a. The analytic framework should be used to clearly present the questions that need to be answered in the literature review. The questions are shown by linkages between interventions and outcomes.

  27. Practice Question 2 (1 of 2) • Test accuracy is included in which part of the PICOTS typology? • P for patient population • I for intervention • C for comparator • O for outcomes

  28. Practice Question 2 (2 of 2) Explanation for Question 2: The correct answer is d. The “O” in the PICOTS typology stands for outcomes, which include patient health outcomes and intermediate outcomes such as test accuracy.

  29. Practice Question 3 (1 of 2) • Most of the published literature on medical tests addresses the clinical impact of tests. • True • False

  30. Practice Question 3 (2 of 2) Explanation for Question 3: The statement is false. Most of the published literature on medical tests focuses on test development and test characteristics and not on the clinical impact of tests. This poses a challenge when reviewing medical tests.

  31. Practice Question 4 (1 of 2) • A causal chain diagram is used to: • Show possible sources of bias • Show the links between the use of a test and the patient outcomes • Help with decisionmaking based on a positive or negative test result • Display the method used for the literature search

  32. Practice Question 4 (2 of 2) Explanation for Question 4: The correct answer is b. A causal chain diagram is used to show visually the links between the use of a test and patient outcomes.

  33. Authors • This presentation was prepared by Brooke Heidenfelder, Rachael Posey, Lorraine Sease, Remy Coeytaux, Gillian Sanders, and Alex Vaz, members of the Duke University Evidence-based Practice Center. • The module is based on Chapter 1, Introduction to the Methods Guide for Medical Test Reviews. In: Methods Guide for Medical Test Reviews. AHRQ Publication No. 12-EC017. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

  34. References (1 of 4) • Agency for Healthcare Research and Quality. U.S. Preventive Services Task Force Procedure Manual. Rockville, MD: Agency for Healthcare Research and Quality; July 2008. 22-4. AHRQ Publication No. 08-05118-EF. www.uspreventive servicestaskforce.org/uspstf08/methods/procmanual.htm. • Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality; April 2012. AHRQ Publication No. 10(12)-EHC063-EF. Chapters available at www.effectivehealthcare.ahrq.gov/methods guide.cfm. • Bravata DM, McDonald KM, Shojania KG, et al. Challenges in systematic reviews: synthesis of topics related to the delivery, organization, and financing of health care. Ann Intern Med. 2005 Jun 21;142(12 Pt 2):1056-65. PMID: 15968030.

  35. References (2 of 4) • Chalmers I, Hedges LV, Cooper H. A brief history of research synthesis. Eval Health Prof 2002 Mar;25(1):12-37. PMID: 11868442. • Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991 Apr-Jun;11(2):88-94. PMID: 1907710. • Green DM, Swets JA. Signal detection theory and psychophysics. New York, NY: Wiley; 1966. Reprinted in 1988 with corrections and an updated topical bibliography (Los Altos, CA: Peninsula Publishing). • Ledley RS, Lusted LB. Reasoning foundations of medical diagnosis; symbolic logic, probability, and value theory aid our understanding of how physicians reason. Science 1959 Jul 3;130(3366):9-21. PMID: 13668531.

  36. References (3 of 4) • Matchar DB. Introduction to the Methods Guide for Medical Test Reviews. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 1.1-1.11. AHRQ Publication No. 12-EHC017. www.effectivehealthcare.ahrq.gov/ medtestsguide.cfm. • Mulrow C, Langhorne P, Grimshaw J. Integrating heterogeneous pieces of evidence in systematic reviews. Ann Intern Med. 1997 Dec 1;127(11):989-95. PMID: 9412305. • Siebert U. When should decision analytic modeling be used in the economic evaluation of health care? Eur J Health Econ. 2003;4(3):143-50. • Tatsioni A, Zarin DA, Aronson N, et al. Challenges in systematic reviews of diagnostic technologies. Ann Intern Med. 2005 Jun 21;142(12 Pt 2):1048-55. PMID: 15968029.

  37. References (4 of 4) • Whitlock EP, Orleans CT, Pender N, et al. Evaluating primary care behavioral counseling interventions: an evidence-based approach. Am J Prev Med. 2002 May;22(4):267-84. PMID: 11988383. • Yerushalmy J. Statistical problems in assessing methods of medical diagnosis, with special reference to X-ray techniques. Public Health Rep. 1947 Oct 3;62(40):1432-49. PMID: 20340527.

More Related