EPI-820 Evidence-Based Medicine

EPI-820 Evidence-Based Medicine LECTURE 4: DIAGNOSIS II Mat Reeves BVSc, PhD

Objectives: • 1. Understand the derivation and use of Bayes’ Theorem. • 2. Understand the “Fundamental Medical Fact #1". • 3. Define the likelihood ratio (LR) and calculate LR+ and LR- from 2 x 2 table. • 4. Understand the Odds-likelihood form of Bayes’ Theorem. • 5. Understand what clinical conditions maximize the value of test results. • 6. Understand the biases and limitations of published test performance measures.

I. Bayes’ Theorem - its Derivation and Use • Bayes’ theorem = a unifying methodology, based on conditional probabilities, for interpreting clinical test results. • PVP = [Se . Prev] [Se . Prev] + [(1 - Sp) . (1 - Prev)] • PVN = [Sp . (1 - Prev)] [Sp . (1 - Prev)] + [(1 - Se) . Prev]

DISEASE PRESENT (D+) ABSENT (D-) [Se. Prev] [Se. Prev] + [1-Sp. 1-Prev] 1-Sp. 1-Prev Se. Prev Test + [Sp. 1 – Prev] [Sp. 1 – Prev] + [1-Se. Prev] Test - Sp. 1-Prev 1-Se. Prev Prev 1 - Prev Derivation of Bayes’ Equations1) 2 x 2 table Approach:

2) First Principles Approach Using Joint and Conditional Probabilities: • Example - PVP: • Step 1: Specify prior prob. of disease = P(D+), non-disease = P(D-). • Step 2: Calculate joint probability of disease and pos. test result: P(T+, D+) = P(T+|D+) . P(D+) N.B. equation derived from definition of joint probability. = cell a, or Se multiplied by prevalence. • Step 3: Calculate joint probability of non-disease and a pos. test result: P(T+, D-) = P(T+|D-) . P(D-) = cell b, OR FP rate multiplied by prob. of non-disease.

Example PVP • Step 4: Calculate probability of a positive test result i.e., P (T+): P (T+) = P(T+, D+) + P(T+, D-) = the sum of cells a and b. • Substituting formulas from steps 2 and 3 we get: P (T+) = P(T+|D+) . P(D+) + P(T+|D-) . P(D-) • Step 5: Calculate probability of disease given a positive test results i.e., PVP: P(D+|T+) = P(T+, D+) P(T+) = cell a divided by cells a and b. • derived from formal definition of joint probability (Step 2)

Example PVP • So: P(D|T+) = P(T+|D+) . P(D+) P(T+|D+) . P(D+) + P(T+|D-) . P(D-) PVP = [Se . Prev] [Se . Prev] + [(1 - Sp) . (1 - Prev)]

Example of the Use of Bayes’ TheoremIP & I-125 FS for DVT (Se = 90%, Sp = 95%), Prevalence of disease = 15%: • PVP = [0.95 . 0.15] [0.95 . 0.15] + [(1 - 0.95) . (1 - 0.15)] • PVP = 0.1425 = 0.77 or 77% 0.1425 + 0.0425 • PVN = [0.95 . (1 - 0.15)] [0.95 . (1 - 0.15)] + [(1 - 0.90) . 0.15] • PVN = 0.8075 = 0.982 or 98.2% 0.8075 + 0.015 • identical to Fig. 9 in lecture 3 (given rounding error of PVP) • equations are cumbersome? difficult to remember?

Prevalence • The importance of prevalence on the interpretation of predictive values and its influence on the whole framework of clinical practice cannot be overstated. • Prevalence represents the clinician's best guess or opinion (expressed as a probability) prior to ordering an actual test.

Table 4.1 Relationship Between History of Chest Pain and Prevalence of Coronary Artery Disease (CAD) (Weiner et al., NEJM, 1979).

Conclusions: • a) probability of CAD depends on patients history • b) probability of CAD after an abnormal test depends on patients history • c) value of the test information also depends on patients history

Fundamental medical fact # 1: • The interpretation of test results depends on the probability of disease before the test was run (= prior probability or prior belief).

The Prior Probability of Disease: • represents what the clinician believes (prior belief or clinical suspicion) • set by considering the practice environment, patients history, physical examination findings, experience and judgment etc • constantly revised in light of new information (= Bayes’ theorem).

Fig 4.1 Use of Bayes’ Theorem to Revise Disease Estimates in Light of New Test Information Use of Bayes’ Formula After - test Before-test After + test 0 0.5 1.0 Probability of Disease

PV+ = P(D+|T+) Max gain (1 – PV-) = P(D+|T-) Figure 4.3 Advantages of a Pre-test Probability of 40 – 60%(Test Has Se= 75%, Sp= 85%, LR+= 5 and LR-= 0.3 %)

II. Generalizability of Published Test Performance Measures (Se, Sp, LRs) • Test performance measures are frequently assumed to be intrinsic characteristics of diagnostic tests independent of underlying prevalence • It is assumed that Se and Sp (or LRs) are fixed and that valid post-test probabilities can be computed by simply varying the prior probability. • Problems?

Potential Problems • # 1. Disease severity affects diagnostic test performance - the more severe the disease the higher the sensitivity (easier to detect). • test performance depends on the spectrum of disease severity in the source (test) population. • #2. Test characteristics are in fact dynamic - they can therefore: • change due to alterations in underlying prevalence. • be different among subgroups defined by age, gender etc.

Potential Problems • #3. Published Se and Sp estimates can be highly biased (Ransohoff and Feinstein, 1986). • Spectrum bias = the difference in both the spectrum and severity of disease between study population and clinically relevant population.

A. Selection Bias During Phase I Evaluations • initial evaluation of a new diagnostic test typically undertaken at referral centers. • determine if test will be positive among patients with severe disease, i.e., the “sickest of the sick”. --- Se is overestimated (advanced disease is easy to detect). • determine if the test is usually negative in “normal” (healthy, young) volunteers, i.e., “ the wellest of the well” --- Sp is overestimated (population unlikely to have diseases that cause FP results). • sometimes test is applied only to the “sickest of the sick”, with the result that Sp is underestimated (FP results are over-inflated, because very sick patients have conditions that tend to make the index test positive).

B. Selection Bias During Phase II Evaluations (Test-Referral Bias) • Test-referral bias occurs when the index test itself is used as a criterion to select which patients receive the definitive (gold standard) diagnostic procedure. • Test negative subjects don’t go on to get the gold standard which results in: • over-estimation of Se (number of FN underestimated) • under-estimation of Sp (number of TN underestimated).

Test-referral bias – Example (from Cox) Index Pop. No disease Index Pop. Disease Study Pop. T+ T+ T+ T+ T- T- T- T- FPR = 0.55 TPR = 0.8 FPR = 0.3 TPR = 0.6

Net Result of Spectrum bias: • Se is over-estimated • Sp?? - depends on relative balance of the possible biases.

Summary – Published Se/Sp/LR Values and Bayes’ Theorem: • Published estimates of Se/Sp or LRs should be considered average values for a particular (sub) population • There’s a great deal to be learnt from mastering Bayes’ Theorem - but its not without potential pitfalls and errors - be careful!!!

III. The Likelihood Ratio (LR) - Definition • Alternative way of describing diagnostic test performance. • For dichotomous test results it summarizes exactly the same information as Se/Sp. • The LR for a particular value of a diagnostic test is defined as the probability of observing the test result (X) in the presence of disease divided by the probability of observing the test result in the absence of disease. • LR (X) = P(X|D+) P(X|D-) • it is the odds that a given test result (X) occurs in a diseased individual compared to a non-diseased individual.

DVT PRESENT (D+) ABSENT (D-) 8 103 Either or both POS (T+) a b TEST c d 11 152 Both NEG (T-) LR+ = a/(a+c) = b/(b+d) LR+ = 18 LR- = c/(a+c) = d/(b+d) LR- = 0.105 Figure 4.2 The Likelihood Ratio for a Dichotomous Test (Positive or Negative)Example: IP and I-125 FS testing.

Interpretation of LRs • LR+ indicates that a positive result (IP and/or I-125 FS) is 18 times more likely to occur in the presence of DVT than in the absence of it. • LR- indicates that a negative result is 0.105 times less likely to occur in the presence of DVT than in the absence of it (or for every negative result in a patient with DVT, expect 9 negative results in patients without DVT).

Note, for dichotomous tests: • LR+ = Sensitivity or True-positive Rate 1 - Specificity False-positive Rate • LR- = 1 - Sensitivity or False-negative Rate Specificity True-negative Rate • Also: • ROC curve = Se versus 1 – Sp • LR+ = the slope of the ROC curve.

Table 4.2 Likelihood Ratios for Common Tests

IV. Advantages of Using LRs • A. Can Apply Bayes’ Theorem Easily Using: • Odds-likelihood Ratio Form of Bayes’ Theorem: Pre-test odds X LR = Post-test odds • where: Pre-test odds = Prevalence 1 – Prevalence Post-test probability = Post-test odds 1 + Post-test odds

What this equation is telling us: • The environment (indicated by the pre-test odds) is as important as the information provided by the test (indicated by the LR). Must know the prevalence or prior probability. • The LR is easily obtained from reference books, the clinician’s job is to provide an estimate of the pre-test odds!!

Example: IP and I-125 FS Tests and DVT Prevalence = 15% • LR+ = Se/(1-Sp) = 0.90/(1 - 0.95) = 18.0 0.15 X 18 = 3.176 (post-test odds) 1 - 0.15 • Post-test probability = 3.176 = 76% 1 + 3.176

Example: IP and I-125 FS Tests and DVT Prevalence = 15% • LR - = (1 - Se)/Sp = 0.10/0.95 = 0.105 0.15 X 0.105 = 0.0186 (post-test odds) 1 - 0.15 • Post-test probability = 0.0186 = 0.0182% 1 + 0.0186 • Note this is P(D+|T-) or the complement of PVN or (1 – P(D-|T-). • PVN is therefore 1 – 0.0186 = 98.2%

B. Can Calculate LRs for Several Levels of Test Results • Disease more likely in the presence of an very abnormal test result than for a marginal one. • LRs can be calculated for any range of test results, thereby preserving clinical information.

Table 4.3 Likelihood Ratios of MI for Six Levels of CK

LRs and Multiple Levels • When CK results were dichotomized, the LR range was 0.07 to 7.6 (108 fold difference) • When data presented for seven levels, the LR range is now 0.01 to 28.3 (a 2830 fold difference) • LRs preserve the natural degree of severity in the original data - DON’T LUMP!!!!.

C. Other Advantages of LRs • i) Robustness • Test performance measures (Se/Sp or LRs) are assumed to be independent of the underlying prevalence of disease. • But, Se and Sp may change in response to changes in prevalence. • LRs are theoretically less susceptible to these changes - because calculated from “smaller slices” of data

ii) Multivariable Modeling • LR is an exponential function of a linear combination of test data: • LR = exp (ao + B1X1 + .....BkXk) • Logistic regression models calculate the probability of a certain event X, given explanatory variables B1X1 + .....+ BkXk. • P(D+|~X) = e B0 + B1X1 + .....+ BkXk 1 + e B0 + B1X1 + .....+ BkXk • With some minor adjustments, the parameters of the logistic regression model can be used to estimate the LR:

Advantages? • Can combine the discriminatory power of several variables (or tests) into a single LR (each variable is now independent). e.g., IP and I-125 FS!! • Can adjust estimates for important covariates (e.g. age or gender). • Examples: Equine colic (Reeves), BreastAid (Osuch)

EPI-820 Evidence-Based Medicine