1 / 57

Materials reviewed by National Johne's Working Group / Johne's Disease Committee / USAHA 2003

Using and Interpreting Johne’s Disease Tests, Sensitivity, Specificity and Predictive Values. Christine A Rossiter Burhans VMD MS NY Animal Health Laboratory Cornell University Currently Poulin Grain Inc. Graphics by B J Tefft.

anitra
Download Presentation

Materials reviewed by National Johne's Working Group / Johne's Disease Committee / USAHA 2003

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using and Interpreting Johne’s Disease Tests, Sensitivity, Specificity and Predictive Values Christine A Rossiter Burhans VMD MS NY Animal Health Laboratory Cornell University Currently Poulin Grain Inc. Graphics by B J Tefft Materials reviewed by National Johne's Working Group / Johne's Disease Committee / USAHA 2003

  2. C A. Rossiter Burhans Poulin Grain, Inc

  3. Diagnosis • Diagnosis is an imperfect process • Series of “rule outs” • Presumptive diagnosis • Usually a “most likely” rather than an absolute answer • We use tests to assist in disease diagnosis C A. Rossiter Burhans Poulin Grain, Inc

  4. Diagnostic Tests • Provide direct evidence • Isolation of MAP • Culture or DNA Probe • Provide indirect evidence • Antibodies to MAP • ELISA or AGID • Provide microscopic evidence • ID Acid-fast microbes • Tissue reaction C A. Rossiter Burhans Poulin Grain, Inc

  5. Johne’s Control is a Process Test Strategy has its place in the plan 3. Test Strategy 1. Prevent Disease Introduction 2. General Risk Management C A Rossiter Burhans/B J Tefft

  6. Editorial • Test and cull without management is like controlling weeds by cutting off the top of the plant • Inefficient: It takes a management plan to change things Control the Urge to test first C A. Rossiter Burhans Poulin Grain, Inc

  7. Goal: get comfortable with the tests and use them to their best advantage • The tests • ELISA and fecal culture are most commonly used • What can the tests do? How accurate are they? • Sensitivity, specificity, cut-offs • How do you interpret results? How right are they? • Need an estimate of the prevalence and a predictive value • You can choose a test strategy for the purpose • Depends on goals, management plan, resources, decisions that will be made, and the benefit if result is true vs. the cost of it is false, etc C A. Rossiter Burhans Poulin Grain, Inc

  8. Johne’s Testing Pre-requisites • Recognize that tests are imperfect, but good tools when properly used • Know what tests do & how to interpret results (varies with situation) • First, determine what you want to accomplish & how you will use results to make decisions: • Who will be tested? • When – timing of test so results are current? • What decisions will be made - high & lower risk positives? C A. Rossiter Burhans Poulin Grain, Inc

  9. Testing Tips • ELISA may not be suitable as an individual cow test unless the animal is exhibiting clinical signs or justified otherwise in that herd • A positive fecal culture test is definitive, but a single negative test on an animal is not C A. Rossiter Burhans Poulin Grain, Inc

  10. Accuracy of Diagnostic Tests Measured by Sensitivity and Specificity C A. Rossiter Burhans Poulin Grain, Inc

  11. What is Sensitivity? • Test’s ability to detect infected animals • Based on test’s performance in a representative population of known infected animals exhibiting all stages of infection • Results are categorized as true positive or false negative compared to the gold standard used to define the infected population • If 25 cows test positive out of a defined population of 100 ‘known’ infected cows: • Sensitivity of the test = 25/100 = 25%, relative to the ‘gold standard’ and population sampled • Average probability that an infected animal will test positive. (Sensitivity estimates should be used and interpreted with caution) C A. Rossiter Burhans Poulin Grain, Inc

  12. The results of any test can be categorized into one of the four yellow cells of this table C.A. Rossiter Burhans VMD MS

  13. Sensitivity is defined as the probability of a positive test result in an infected animal Sensitivity = A / A + C 25 / (25+75) = 25% 25 75 100 INFECTED ANIMALS C.A. Rossiter Burhans VMD MS

  14. What is Specificity? • Test’s ability to detect non-infected animals • Based on test’s performance in a representative population of animals known to be free of infection. • Results are categorized as true negative or false positive compared to a gold standard test • 99 cows test negative out of a defined population of100 ‘known’ infection-free cows: • Specificity of the test = 99/100 = 99%, relative to the ‘gold standard’ and population used. • It is the average probability that a non-infected animal will test negative on the test C A. Rossiter Burhans Poulin Grain, Inc

  15. Specificity is defined as the probability of a negative test result in a non-infected animal Specificity = D / B + D 99 / (1+99) = 99% B 1 D 99 100non-infected animals C A Rossiter Burhans Poulin Grain, Inc

  16. Applying the principles to the Ideal Test • Sensitivity and specificity values are derived from separate and representative populations of known infected and non-infected animals • For the idealtest • When results from the two populations are plotted out, the two distinct populations are discretely separated at one point value • A logical cut-off point is defined that discriminates a ‘positive’ test result from a ‘negative’ result with very little overlap between the two populations C A. Rossiter Burhans Poulin Grain, Inc

  17. The Ideal Test Frequency distribution of hypothetical ELISA values for infected and non-infected populations Critical Cut Off Value %Frequency True Test-Positive population True Test-Negative population Increasing OD values False Test-Negative population False Test-Positive population C A. Rossiter Burhans Poulin Grain, Inc

  18. Applying the principles to the ELISA test • Johne’s serology test distributions, both positive and negative, are not ideal • There is considerable overlap of positive and negative populations, thus making a definitive cut-off (negative/positive) problematic C A. Rossiter Burhans Poulin Grain, Inc

  19. Non-infected animal group Infected animal group Sensitivity Values Specificity Values A Modified ROC Chart showing Sensitivity (Se) and Specificity (Sp) values at different cutoff values Cutoff Values Se 35%Sp 97% 100 80 Se 50%Sp 90% Number of tests X 10 Percent Sensitivity and Specificity 60 40 Se 60%Sp 80% 20 ELISA OD values on a continuous scale Sensitivity and specificity have an inverse trade-off,as sensitivity increases specificity decreases Adapted from RH Jacobsen

  20. Multiple cut-offs have an advantage • Provide more information to refine decisions according to • Relative Sensitivity and Specificity of the cut - off • Relative benefit of decision if result is true • Cost if result is false • Higher cut-off • Less Sensitive, more Specific, less false positive risk • Lower cut-off • More Sensitive, less Specific, more infected cattle considered ‘positive’, and also more non-infected cattle C A. Rossiter Burhans Poulin Grain, Inc

  21. Idexx and Biocor ELISA tests give multiple categories relative to each manufacturer’s established single cut-off value Appropriate interpretation of a result requires additional consideration of herd prevalence and predictive values of a positive and negative test (next section) Idexx s/p ratio single cut off = .25 (calculated differently than score value) Biocor score single cut off = 1.0 (calculated differently than s/p ratio) C A. Rossiter Burhans Poulin Grain, Inc

  22. Getting back to the sensitivity of a Johne’s ELISA test using the single cut-off example C A. Rossiter Burhans Poulin Grain, Inc

  23. Johne’s ELISA Sensitivity:Depends on the Population Tested Population tested ELISA Sensitivity Clinical Signs 87% High Shedders 75% Low Shedders 15% Intermittent Shedders 10% Overall Population 45% (published study, culture positive gold standard) Note: As animals that are test-positive are culled the sensitivity declines for future tests in that herd Sweeney & RH Whitlock, J Vet Diag Inv 7:488, 1995

  24. Stage III 75% + + + + Mixed Stages 39% + + + + + + + + + + + + + + + + + + + + + + + + + StageIV 87% + + + + + + + + + + + + + + + + + Stage I 0% Stage II 15% + + + + + + + + National Johne’s Working Group (NJWG)Estimated Overall ELISA Sensitivity = 25%* The NJWG observed that the overall sensitivity of the ELISA would be lower if Stage I animals are taken into account *Certification Committee, NJWG for Voluntary JD Herd Status Program 1998 C. A. Rossiter Burhans VMD MS / Graphics Barbara J. Tefft Data from Sweeney & RH Whitlock, J Vet Diag Inv 7:488, 1995

  25. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Stage I 0% Estimate of Overall Fecal Culture Sensitivity = 40%* Stage II ~50% Stage III & IV >95% + The NJWG observed that the overall sensitivity of fecal culture would be lower if Stage I animals are taken into account C A Rossiter Burhans Poulin Grain, Inc. Graphics Barbara J Tefft **Certification Committee, NJWG for VJDHSP 1998, Centrifugation Method

  26. What about Specificity of the tests? C A. Rossiter Burhans Poulin Grain, Inc

  27. Specificity of the Johne’s ELISA n % ELISA negative Cull cows at slaughter Uninfected herds A Uninfected herds B Bull stud 3x Negative Overall 263 98 50 100 172 100 87 100 485 99% +/-1.9% Sweeney RW JVDI 7:488, 1995 C A. Rossiter Burhans Poulin Grain, Inc

  28. Estimate of Overall Specificity of ELISA* = 98% Non-infected Cows from Non-infected Herds + + + * Certification Committee, NJWG for VJDHSP 1998 C A. Rossiter Burhans Poulin Grain, Inc Graphics Barbara J Tefft

  29. Specificity of serology in cattle - precaution • Specificity also may vary with: • Herd • Geographic region • Exposure to other microbes • Variability within the laboratory C A. Rossiter Burhans Poulin Grain, Inc

  30. Estimate of Overall Specificity of fecal culture* = 100% Non-infected Cows from Non-infected Herds * Certification Committee, NJWG for VJDHSP 1998 Graphics Barbara J Tefft C A. Rossiter Burhans Poulin Grain, Inc

  31. Recommended estimates of Sensitivity and Specificity* From reference populations: • Johne’s ELISA serology (single cutoff) Se = *25% (published ~45%) Sp = 98% • MAP Fecal Culture** Se = 40% Sp = 100% *NJDWG 1998 ** Most sensitive centrifugation methods C A. Rossiter Burhans Poulin Grain, Inc

  32. Take home message:Diagnostic Se and Sp of current Johne’s tests • ELISA: (NJWG estimate)Se ~ 25%,Sp ~ 98% • Fecal culture: (NJWG estimate)Se ~ 40%,Sp ~100% • Multiple cut-offs • more info • trade off of Se with Sp as move up or down • Sensitivity and specificity alone do not predict the probability of a test result being correct • Se, Sp AND the existing expectation (probability or prevalence) that an animal or a herd is infected, combined, influence how correct a particular test result is likely to be C A. Rossiter Burhans Poulin Grain, Inc

  33. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? We apply tests to real herds or groups of cattle A particular herd has a particular prevalence and distribution of stages of infection C A Rossiter Burhans

  34. Interpretation • Because the tests are not perfect, Johne’s test results must be interpreted in the context of the specific situation and the other information you know • Epidemiology of the disease • Herd history • Clinical impression • Predictive values C A. Rossiter Burhans Poulin Grain, Inc

  35. How much confidence can we have in a test result, in a particular situation? We can calculate a probability for the situation that a positive or negative test result is ‘correct’, called a predictive value C A. Rossiter Burhans Poulin Grain, Inc

  36. Positive Predictive Value (PPV) • True positives / All test-positives, or A/A+B • Probability that a test-positive animal is infected • Value is based on Sensitivity (SE), Specificity (SP) and an expectation of JD in herd or, Prevalence (P) • PPV = (SE*P) / (SE*P)+[(1-SP)*(1-P)] • The proportion of positive test results that are correct (animal is infected) C A. Rossiter Burhans Poulin Grain, Inc

  37. Negative Predictive Value (NPV) • True negatives / All test-negatives or D/C+D • Percent test-negative that are truly negative • Can estimate with Sensitivity (SE), Specificity (SP) and a Prevalence Estimate (P) • NPV =SP*(1-P) ÷ [SP*(1-P)]+[(1-SE)*P] • Proportion of negative test results that are correct (animal is not infected) C A. Rossiter Burhans Poulin Grain, Inc

  38. What is a prevalence estimate? (pre-test probability or pre-existing chance that animal is infected) • The percent infected in a herd or group of interest • i.e. the estimated % of infected cows in the herd • Also, the probability that any individual in the group or herd is infected • i.e. any cow C A. Rossiter Burhans Poulin Grain, Inc

  39. Pretest Estimates of Prevalence – Where do we get them? • Estimated prevalence from history • Presumptive diagnosis (e.g., 1 of 4 other possible causes, P = 25%) • Whole-herd Test Prevalence • Calculated estimate of true prevalence from the herd-test prevalence • True Prevalence = Test Prevalence + (Specificity - 1) Specificity + (Sensitivity - 1) C A. Rossiter Burhans Poulin Grain, Inc

  40. Test prevalence and true prevalence: “Rule of thumb” for Johne’s tests: • Test prevalence gives a crude estimate of true prevalence: • If test prevalence is > 5%, estimated true prevalence is 2-4 times the test prevalence • If < 5%, the false positive rate for the ELISA distort the relationship. True prevalence is likely < 5% C A. Rossiter Burhans Poulin Grain, Inc

  41. + + + + + + + + Not infected Stage 1 Stage 2 Stage 3 Stage 4 Estimated 30%Prevalence – ELISA Test • High chance that a Positive ELISA is correct • Moderate chance that a Negative ELISA is correct • Do these results correlate with clinical impression? C A Rossiter Burhans Poulin Grain, Inc Graphics Barbara J Tefft

  42. + + + - + + + + + Test + - + Not infected Stage 1 Stage 2 Stage 3 Stage 4 Predictive Values for an Estimated 30% Prevalence – ELISA Test Sensitivity = 25%, Specificity = 98% Pre-test estimate of Johne’s Status 30% 8 1 9 + + + 22 69 91 30 100 cow herd 70 PPV 8/9 = 89% NPV 69/91 = 76% Se=25%Sp=98% C A. Rossiter Burhans Poulin Grain, Inc Graphics Barbara J Tefft

  43. + + + + + + + Test + - + Not infected Stage 1 Stage 2 Stage 3 Stage 4 Predictive Values for an Estimated 30% Prevalence – ELISA Test Sensitivity = 45%, Specificity = 98% Pre-test estimate of Johne’s Status 30% + - 13 1 14 + 17 69 86 30 70 100 PPV: 13/14 = 93% NPV: 69/86 = 80% C A. Rossiter Burhans Poulin Grain, Inc Graphics Barbara J Tefft

  44. + + + Not infected Stage 1 Stage 2 Stage 3 Stage 4 Estimated 4%Prevalence - ELISA • Poor chance that a Positive ELISA is correct • High chance that a Negative ELISA is correct C A. Rossiter Burhans Poulin Grain, Inc Graphics Barbara J Tefft

  45. + + + Not infected Stage 1 Stage 2 Stage 3 Stage 4 Predictive Values for an Estimated 4% Prevalence – ELISA Test Sensitivity = 25%, Specificity = 98% Pre-test estimate of Johne’s Status 4% + - + 1 2 3 Test 3 94 97 - 4 96 100 PPV: 1/3 = 30% NPV: 94/97 = 97% C A. Rossiter Burhans Poulin Grain, Inc Graphics Barbara J Tefft

  46. + + + Not infected Stage 1 Stage 2 Stage 3 Stage 4 Predictive Values for an Estimated 4% Prevalence – ELISA Test Sensitivity = 45%, Specificity = 98% Pre-test estimate of Johne’s Status 4% + - 2 2 + 4 Test 2 96 - 94 4 96 100 PPV : 2/4 = 50% NPV: 94/96 = 98% C A. Rossiter Burhans Poulin Grain, Inc Graphics Barbara J Tefft

  47. + + Not infected Stage 1 Stage 2 Stage 3 Stage 4 4% Prevalence - Fecal Culture • 100% chance that a PositiveFecal Culture is correct • High chance that NegativeFecal Culture is correct C A. Rossiter Burhans Poulin Grain, Inc Graphics Barbara J Tefft

  48. + + + Test - Not infected Stage 1 Stage 2 Stage 3 Stage 4 4% Prevalence – Predictive Values for Fecal Culture Sensitivity = 40%, Specificity = 100% Pre-test estimate of Johne’s Status 4% + - 2 0 2 2 96 98 4 96 100 PPV: 2/2 = 100% NPV: 96/98 = 98% C A. Rossiter Burhans Poulin Grain, Inc Graphics Barbara J Tefft

  49. Summary of predictive values • The probability that the test result is correct depends on: • Specificity • Sensitivity (less impact on values than specificity) • Pre-test prevalence or probability of infection in the population being tested Calculate a Predictive Value for a positive and negative test result C A. Rossiter Burhans Poulin Grain, Inc

  50. Positive and Negative Predictive Values, using a single cut-off, are shown for various prevalence or pre-test estimate values ELISA Se and Sp values are based on NJWG and mfg. recommendations. Fecal culture values are based on NJWG recommendations Sensitivity: ELISA25 = 25%, *ELISA60 = 60%, Fecal Culture = 40% Specificity: ELISA25 = 98%, *ELISA60 = 97%, Fecal Culture = 100% * Values recommended by the manufacturer IDEXX C A. Rossiter Burhans Poulin Grain, Inc

More Related