Systematic Review of Prognostic Tests

1. Systematic Review of Prognostic Tests Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Medical Test Reviews Methods Guide www.ahrq.gov

2. Learning Objectives • Develop the topic and structure the systematic review of a prognostic test. • Describe the similarities and differences between the evaluation of diagnostic and prognostic tests for systematic reviews. • Explain the time-dependent characteristics of prognostic tests. • Perform appropriate statistical analyses based on outcome probabilities. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

3. Steps Involved in Conducting a Systematic Review of a Prognostic Tests • Develop the review topic and framework. • Search for studies. • Select studies and assess quality. • Extract statistics to evaluate test performance . • Conduct meta-analyses of estimates of outcome probabilities. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

4. Step 1: Developing the Review Topic and Analytic Framework (1 of 4) • The review topic, analytic framework, and Key Questions can be fundamentally different for diagnostic and prognostic tests • Diagnostic tests determine whether a patient has a disease at the time of the test. • A “gold standard” (i.e., the best available) reference test is often used to determine true disease presence or absence. • Prognostic tests predict a patient’s likelihood of developing a disease or experiencing a future medical event. • The “reference test” is the proportion of study subjects who actually develop the condition predicted by the test. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

5. Step 1: Developing the Review Topic and Analytical Framework (2 of 4) • It is often useful to group test results by implications for decisionmaking when structuring reviews • Example 1: • Structuring based on prognostic test categories (low/intermediate/high risk) when different treatments for each category are evaluated • Example 2: • Structuring based on precision and accuracy of outcome probabilities of categories when a decision model is used as a framework Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

6. Step 1: Developing the Review Topic and Framework (3 of 4) • Considerations for the review summarized in the general PICOTS framework: Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

7. Step 1: Developing the Review Topic and Framework (4 of 4) • In some contexts, it is informative to: • Categorize subjects as those who did/did not experience the predictedoutcome during a specified time interval. • Consider what followup times are informative to patients, clinicians, or policymakers. • Look back to categorize results of the test. • This step permits assessment of accuracy by calculating sensitivity, specificity, and predictive values. • Prognostic tests can be specifically used to predict response to a treatment. • They predict beneficial/adverse responses to treatment. • They are also referred to as predictive tests Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

8. Step 2: Searching for Studies (1 of 5) Studies can relate to one or more categories: • Proof of concept: Is the test result associated with a clinically important outcome? • Prospective clinical validation: How accurately does the test predict outcomes in different patient cohorts, clinical practices, and prognostic groups? • Incremental predictive value: How much does the new test change predictive probabilities and increase the discrimination of patients who did/did not experience the outcome of interest within a specific time period? • Clinical utility: Does the new assessment change predicted probabilities enough to reclassify many patients into different prognostic groups that would be managed differently? • Clinical outcomes: Would use of the prognostic test improve patient outcomes? • Cost-effectiveness: Do the improvements in patient outcomes justify the additional costs of testing and subsequent medical care? Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

9. Step 2: Searching for Studies (2 of 5) • The first four categories are most readily addressed by large cohort studies and secondary analyses of clinical trials. • For the last two categories, randomized control trials (RCTs) are preferred. • RCTs are rare because of barriers including costs and the time involved. • In the event that no RCTs are found, use a decision model to focus on providing best estimates of outcome probabilities. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

10. Step 2: Searching for Studies (3 of 5) • There are no reliable, validated methods for searching the literature for prognostic test information. • Some strategies use key words and index terms in studies meeting selection criteria. • Others use search terms such as “incidence” and word roots such as “prognos*”. • Terms describing the prognostic test and condition/event to be predicted should be included in the search. • “Find similar” or “related article” functions can be useful. • A manual search of references will be needed. • The records of the regulatory agency for submitted tests can be useful. • The Web site of test producer may be informative. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

11. Step 2: Searching for Studies (4 of 5) • Unlike diagnostic tests, many prognostic tests are incorporated into multivariable regression models/ algorithms for prediction. • Many reports support only an independent association of a variable with the patient outcome. • It is difficult to find reports where the test variable did not add significantly to a multivariable regression model. • This introduces potential bias by failing to uncover the lack of association/predictive value. • All studies including the variable should be sought out. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

12. Step 2: Searching for Studies (5 of 5) • When prognostic groups are defined by predicted outcome probabilities: • Search for decision analyses, guidelines, or expert opinions that support outcome probability thresholds used to define clinically meaningful prognostic groups (i.e., groups that would be treated differently in practice because of their predicted outcome) • Ideally use randomized controlled trials of interventions in patients selected on the basis of the prognostic test. • This helps establish the rationale for using the test to classify patients. • It is not always sufficient to evaluate this use of the test. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

13. Step 3: Selecting Studies and Assessing Their Quality (1 of 4) • Prognostic indicators vary substantially in: • Study design • Subject inclusion criteria • Methods of measuring key variables • Methods of analysis (including definition of prognostic groups) • Adjustment for covariates • Presentation of results • Reviewer access to patient-level data would allow more uniform analyses to overcome these difficulties. • When these data are lacking, following certain questions can help. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

14. Questions for Judging the Quality of Individual Studies of Prognostic Tests (1 of 3) • Was the study designed to evaluate the new prognostic test, or was it a secondary analysis of data collected for other purposes? • Were the subjects somehow referred or selected for testing? What was the testing scenario? • Was the clinical population clearly described including the sampling plan, inclusion and exclusion criteria, subject participation, and the spectrum of test results? Did the sample represent patients who would be tested in clinical practice? • Did everyone in the samples have a common starting point for followup with respect to the outcome of interest, including any treatments that could affect the outcome being predicted? Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

15. Questions for Judging the Quality of Individual Studies of Prognostic Tests (2 of 3) • Was the prognostic test clearly described and conducted using a standardized, reliable, and valid method? • Was the test used and interpreted the same way by all sites/studies, including any indeterminate test results? • Were the results ascertained without knowledge of the outcome? • Were investigators blinded to the test results? • How were previously established prognostic indicators or other prognostic assessments included in the study and analyses? • Was the outcome being predicted clearly defined and ascertained using a standardized, reliable, and valid method? • How complete was the followup of subjects, and were losses to followup related to the test results or the outcome being predicted? • Was the duration of followup adequate? Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

16. Questions for Judging the Quality of Individual Studies of Prognostic Tests (3 of 3) • Were the data used to develop the prognostic test? • Were the prognostic groups predefined based on clinically meaningful decision thresholds for predicted outcome probabilities? • Were the results externally validated using an independent sample or internally validated using bootstrap or cross-validation methods? • Were previously established prognostic tests that were used as comparators fitted to the sample data in the same manner as the potential new prognostic test? • Were outcome predictions adjusted for any other factors? Which ones? How? Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

17. Step 3: Selecting Studies and Assessing Their Quality (2 of 4) • Reviewers should explicitly state inclusion/exclusion criteria • Test comparisons should use data from the same cohort of subjects to minimize confounding • Within a study, the tests being compared should be conducted at the same time. • This ensures a common starting point with respect to the patient outcome being predicted. • Reviewers should note the starting point of each study. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

18. Step 3: Selecting Studies and Assessing Their Quality (3 of 4) • Prognostic test results and their interpretation should be ascertained without knowledge of outcomes to avoid ascertainment bias. • Investigators should be blinded to the results of a test to avoid selective changes in treatment that could affect the outcome being predicted. • Be aware of any previously established prognostic indicators that should be included in the comparative analysis of potential new tests. • Note any adjustments for covariates that could make studies more or less comparable. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

19. Step 3: Selecting Studies and Assessing Their Quality (4 of 4) • Fitting a new prognostic test to the sample data (test development sample) by using these data to define cutoffs or model relationships to outcomes and estimate regression coefficients may: • Be overly optimistic for estimated predicted performance • Bias the comparison to an established prognostic method that was not fitted to the sample data Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

20. Step 4: Extracting Statistics To Evaluate Test Performance (1 of 12) • Summary statistics must be appropriate for the review’s Key Questions. • Example: Hazard ratios from Cox regression analyses or odds ratios from logistic regression analyses for associations between tests and outcomes • Address only the early phases in test development • Often do not discriminate between subjects who eventually do or do not experience the outcome of interest • Statistically significant associations (odds/hazard ratios, relative risks) merely indicate that more definitive evaluation of a new predictor is warranted. • Due to these concerns, the questions that a systematic review can answer by summarizing a test’s association with an outcome are limited. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

21. Step 4: Extracting Statistics To Evaluate Test Performance (2 of 12) • Discrimination statistics • Indices of discrimination • Estimates of sensitivity, specificity, and area under the receiver operating characteristic curve are calculated at one particular time. • They can be calculated retrospectivelyand compared when: • A new prognostic indicator is added to a predictive model • A prognostic test is compared to predictions made by other methods • Retrospective indices of discrimination do not: • Summarize predicted outcome probabilities • Directly address questions about the predictions on the basis of a new prognostic test or its impact on patient outcomes Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

22. Step 4: Extracting Statistics To Evaluate Test Performance (3 of 12) • Reclassification tables are better suited for assessments of the clinical impact of prognostic tests. • They are not as common or reported as often as discrimination statistics. • When using discrimination statistics, recognize that they change over time as more patients develop the outcome being predicted. • Time-dependent measures of discrimination statistics have been developed (e.g., Harrell’s C-statistic); however, examples of reviews using these measures were not found Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

23. Step 4: Extracting Statistics To Evaluate Test Performance (4 of 12) • Reclassification tables • For example: • Patients are placed into prognostic groups based on their Framingham cardiovascular risk scores. • Reclassification tables are then used to determine how adding a prognostic test reclassifies the patients into prognostic groups. • Ideally, the classification of outcome probabilities into prognostic groups should be based on outcome probabilities that will lead to different courses of action. • If not, then reclassifications might not have clinical utility; that is, they may not make a difference in patient care. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

24. Step 4: Extracting Statistics To Evaluate Test Performance (5 of 12) Sample Reclassification Table Based on Predicted Outcome Probabilities Adding the new test reclassified 10% of the 1000 people originally in the lower risk group and 25% of the 400 people in the higher risk group Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

25. Step 4: Extracting Statistics To Evaluate Test Performance (6 of 12) • Reclassification tables provide information about: • Observed outcome probabilities in each prognostic group • Predicted probabilities • Drawbacks • Information is often limited to a single follow-up time. • The precision of estimates may not be reported. • Differences between estimated probabilities and observed outcomes for each prognostic group may be analyzed by using the chi-square goodness-of-fit test. • The results will not help determine if differences in predicted and observed probabilities are better when a new prognostic test is added. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

26. Step 4: Extracting Statistics To Evaluate Test Performance (7 of 12) • The net reclassification improvement statisticis a summary statistic of separate reclassification tables. • One table is for those who did experience the outcome event within a particular time period. • The other table is for those who did not experience the outcome event. • In a group experiencing the outcome of interest (e.g., those who died), net improvement = the proportion of patients reclassified into a higher probability group minus the proportion reclassified into a lower probability group. • In a 2 x 2 table, this is an estimated change in test sensitivity. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

27. Step 4: Extracting Statistics To Evaluate Test Performance (8 of 12) • For those who do not experience an outcome (e.g., those who survived), the inverse is used: net improvement = the proportion reclassified into a lower-risk group minus the proportion reclassified into the higher-risk group. • In a 2x2 table, this is the estimated change in test specificity. • The net reclassification improvement (NRI) statistic is the sum of net improvement in classification in patients who did or did not experience the outcome. • The integrated discrimination index (IDI) uses mean changes in individual predicted probabilities instead of net improvement in the calculations above. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

28. Step 4: Extracting Statistics to Evaluate Test Performance (9 of 12) • The “continuous formulation” version of the net reclassification improvement (NRI) statistic calculates the: • Probability of predicted event among those who have an increase in predicted probability after new test • Probability of predicted event among those who have a decrease in predicted probability after new test • Event probability in the overall sample • With this version of the NRI statistic: • The NRI statistic can be estimated by time-to-event analysis, but the three probabilities still represent only a single point of followup. • The NRI statistic does not require clinically meaningful prognostic categories, since not all increases/decreases in probability prompt a change in patient management. • The NRI statistic focuses instead on subjects with a higher or lower predicted outcome probability when a new test is used. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

29. Step 4: Extracting Statistics To Evaluate Test Performance (10 of 12) • When pooling net reclassification improvement (NRI) or integrated discrimination index (IDI) estimates from different studies, the following should not differ substantially: • Characteristics of prognostic groups • Definition of outcome event • Overall probability of event • Followup time Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

30. Step 4: Extracting Statistics To Evaluate Test Performance (11 of 12) • Predictive values • Treatment decisions based on outcome probabilities are often dichotomous. • For example, “treat those at high risk” and “do not treat those at low risk” • If patients would be treated because a test indicates they are at “high risk,” then the observed time-dependent percentages of patients developing the outcome (without treatment) are positive predictive values. • That is, the proportion of those who have a “positive” prognostic test result who end up having the event. • If patients would not be treated because they are at “low risk,” then: 1 – (observed outcome probabilities) = negative predictive values. • That is, the proportion of those who have a “negative” prognostic test result who do not end up having the event. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

31. Step 4: Extracting Statistics To Evaluate Test Performance (12 of 12) • Predictive values (continued) • For a single point of followup, positive and negative predictive values can be compared via methods used for diagnostic tests. • Ratios of positive and negative predictive values of two prognostic tests are often summarized along with confidence intervals. • The regression model of Leisenring et al. (2000) may be used to determine how patient characteristics relate to relative predictive values. • Time-to-event curves comparing predictive values of two prognostic tests are available if necessary. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Leisenring W, Alonzo T, Pepe MS. Biometrics 2000 Jun;56(2):345-51. PMID: 10877288.

32. Step 5: Meta-analysis of Estimates of Outcome Probabilities (1 of 2) • Randomized controlled trials (RCTs) designed to demonstrate net improvement in patient outcomes and cost-effectiveness are the most definitive level of evidence. • Many prognostic test studies do not provide this level. • Systematic reviews can provide estimates of outcome probabilities for decision models instead. • Estimates from either RCTs or observational studies can be used, provided that prognostic groups are well characterized and similar. • Meta-analysis can provide more precise estimates . • Additionally, meta-analysis of estimates of outcome probabilities in a prognostic group extracted from several studies can provide insight into: • Stability of the estimates • Whether variation is related to prognostic group characteristics Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

33. Step 5: Meta-Analysis of Estimates of Outcome Probabilities (2 of 2) • Methods have been developed to combine estimates of outcome probabilities from different studies. • Dear’s method (1994) • Uses a fixed-effects regression model • Arend’s method (2008) • Is similar to a DerSimonian-Laird random-effects model • Is used when there is only one common followup time for all studies/prognostic groups Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm. Arends LR, Hunink MG, Stijnen T. Stat Med 2008 Sep 30;27(22):4381-96. PMID: 18465839. Dear KB. Biometrics 1994 Dec;50(4):989-1002. PMID: 7787011. DerSimonian R, Laird N. Control Clin Trials 1986 Sep;7(3):177-88. PMID: 3802833.

34. Key Messages (1 of 2) • Methods used to conduct systematic reviews of prognostic tests are not well established. • The intended use of the prognostic test should be specified; predicted probabilities need to be classified into clinically meaningful groups that are described in detail (including outcome probabilities). • Many published reports focus on associations between prognostic indicators and patient outcomes (the first stage of development of prognostic tests); such studies have limited clinical value for a review. • Criteria for evaluating the quality of studies of prognostic tests have not been firmly established. • Reviewers can adapt criteria developed for diagnostic tests. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

35. Key Messages (2 of 2) • Given differences between diagnostic tests (current state of disease) and prognostic tests (future state of disease), common evaluation statistics used with diagnostic tests are not as informative for prognostic tests. • These include point estimates of sensitivity/specificity and area under the receiver operating characteristic (ROC) curves. • The most pertinent summary statistics for prognostic tests are: • The time-dependent observed outcome probabilities • The closeness of the prognostic group’s predicted probabilities to observed outcomes • How the use of a new test reclassifies patients into different groups and improves predictive accuracy and overall outcomes • Methods for comparing and summarizing predictive performance of prognostic tests need further development and widespread use. Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Methods guide for medical test reviews. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

36. Practice Question 1 (1 of 2) • What is the relationship between predictive and prognostic tests? • The terms are essentially synonymous. • Predictive tests are a category of prognostic tests. • Prognostic tests are a category of predictive tests.

37. Practice Question 1 (2 of 2) Explanation for Question 1: The correct answer is b. Predictive tests, which are a category of prognostic tests, can be used to predict beneficial or adverse responses to treatment.

38. Practice Question 2 (1 of 2) • Whichof the following four categories of prognostic tests is NOT most readily addressed by large cohort studies and secondary analyses of clinical trials? • Clinical outcomes • Clinical utility • Prospective clinical validation • Proof of concept • None of the above

39. Practice Question 2 (2 of 2) Explanation for Question 2: The correct answer is a. Questions that pertain to clinical outcomes or cost-effectiveness are generally better answered by randomized controlled trials. The other three categories listed may often be addressed by less costly cohort studies or secondary analyses of clinical trials.

40. Practice Question 3 (1 of 2) • There are relatively few reports where the test variable did not add significantly to a multivariable regression model. • True • False

41. Practice Question 3 (2 of 2) Explanation of Question 3: The correct answer is true. The relative paucity of published findings in which a test variable did not add significantly to a multivariable regression model suggests that there may be potential bias by failing to publish lack of association or predictive value.

42. Practice Question 4 (1 of 2) • Prognosticindicators vary substantially in which of the following: • Study design • Subject inclusion criteria • Methods of measuring key variables • Adjustment for covariates • Presentation of results • All of the above

43. Practice Question 4 (2 of 2) Explanation of Question 4: The correct answer is f. Wide variability in one or more of these factors presents a challenge to selecting studies and assessing quality. Reviewer access to patient-level data would allow uniform analyses to overcome these difficulties. When lacking the ability to access patient-level data, suggestions for judging the quality of individual studies of prognostic tests can be followed.

44. Practice Question 5 (1 of 2) • Examples of indices of discrimination do not include: • 95-percent confidence intervals • Estimates of sensitivity and specificity • Area under the receiver operating characteristic (ROC) curve • All of the above

45. Practice Question 5 (2 of 2) Explanation of Question 5: The correct answer is a. 95-percent confidence intervals provide a measure of the precision of a given statistic, but are not, in themselves, a discrimination statistic.

46. Authors • This presentation was prepared by Brooke Heidenfelder, Rachael Posey, Lorraine Sease, Remy Coeytaux, Gillian Sanders, and Alex Vaz, members of the Duke University Evidence-based Practice Center. • The module is based on Rector TS, Taylor BC, Wilt TJ. Systematic review of prognostic tests. In: Chang SM and Matchar DB, eds. Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; June 2012. p. 12.1-13. AHRQ Publication No. 12-EHC017. Available at www.effectivehealthcare.ahrq.gov/medtestsguide.cfm.

47. References (1 of 10) • Arends LR, Hunink MG, Stijnen T. Meta-analysis of summary survival curve data. Stat Med 2008 Sep 30;27(22):4381-96. PMID: 18465839. • Altman DG. Systematic reviews of evaluations of prognostic variables. BMJ 2001 Jul 28;323(7306):224-8. PMID: 11473921. • Altman DG, Riley RD. Primer: an evidence-based approach to prognostic markers. Nat Clin Pract Oncol 2005 Sep;2(9):466-72. PMID: 16265015. • Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007 Feb 20;115(7):928-35. PMID: 17309939. • Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 2008 Jan;54(1):17-23. PMID:18024533. • Cook NR, Ridker PM. Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Ann Intern Med 2009 Jun 2;150(11):795-802. PMID: 19487714.

48. References (2 of 10) • Dear KB. Iterative generalized least squares for meta-analysis of survival data at multiple times. Biometrics 1994 Dec;50(4):989-1002. PMID: 7787011. • DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986 Sep;7(3):177-88. PMID: 3802833. • Earle CC, Pham B, Wells GA. An assessment of methods to combine published survival curves. Med Decis Making 2000 Jan-Mar;20(1):104-11. PMID: 10638543. • Feng Z, Prentice R, Srivastava S. Research issues and strategies for genomic and proteomic biomarker discovery and validation: a statistical perspective. Pharmacogenomics 2004 Sep;5(6):709-19. PMID: 15335291. • Fibrinogen Studies Collaboration. Measures to assess the prognostic ability of the stratified Cox proportional hazards model. Stat Med 2009 Feb 1;28(3):389-411. PMID: 18833567.

49. References (3 of 10) • Fowkes FG, Murray GD, Butcher I, et al. Ankle brachial index combined with Framingham Risk Score to predict cardiovascular events and mortality: a meta-analysis. JAMA 2008 Jul 9;300(2):197-208. PMID: 18612117. • Groenveld HF, Januzzi JL, Damman K, et al. Anemia and mortality in heart failure patients a systematic review and meta-analysis. J Am Coll Cardiol 2008 Sep 2;52(10):818-27. PMID: 18755344. • Grundy SM, Cleeman JI, Merz CN, et al; National Heart, Lung, and Blood Institute; American College of Cardiology Foundation; American Heart Association. Implications of recent clinical trials for the National Cholesterol Education Program Adult Treatment Panel III guidelines. Circulation 2004 Jul 13;110(2):227-39. PMID: 15249516. • Hall PA, Going JJ. Predicting the future: a critical appraisal of cancer prognosis studies. Histopathology 1999 Dec;35(6):489-94. PMID: 10583572.

50. References (4 of 10) • Hlatky MA, Greenland P, Arnett DK, et al. Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation 2009 May 5;119(17):2408-16. PMID: 19364974. • Hayden JA, Cote P, Bombardier C. Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med 2006 Mar 21;144(6):427-37. PMID: 16549855. • Ingelsson E, Schaefer EJ, Contois JH, et al. Clinical utility of different lipid measures for prediction of coronary heart disease in men and women. JAMA 2007 Aug 15;298(7):776-85. PMID: 17699011. • Ingui BJ, Rogers MA. Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc 2001 Jul-Aug;8(4):391-7. PMID: 11418546.