1 / 55

Exercises in Sensitivity, Specificity, PPV, and NPV

Exercises in Sensitivity, Specificity, PPV, and NPV. There are eight questions on Slides 4, 6, 7, 8, 22, 26, 42, and 54. Answer on Word and place in Dropbox by the end of the week. The rest of the material is meant to guide you. Remember. If you were asked; how would you estimate:

rossb
Download Presentation

Exercises in Sensitivity, Specificity, PPV, and NPV

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Exercises in Sensitivity, Specificity, PPV, and NPV There are eight questions on Slides 4, 6, 7, 8, 22, 26, 42, and 54. Answer on Word and place in Dropbox by the end of the week. The rest of the material is meant to guide you.

  2. Remember If you were asked; how would you estimate: • Se: get a group of dzed animals and test them • Sp: geta group of non-dzed animals and test them • PPV: get a group of test positive animals and determine how many are dzed • NPV: get a group of test negative animals and determine how many are non-dzed Remember: dzed means the animal has the dz, or infection or condition of interest (which could be anything).

  3. Remember • Se = # Test + / # dzedInfo about dzed • Sp = # Test - / # non-dzedInfo about non-dzed • PPV = #dzed / # Test + Info about Test+ • NPV = # healthy / # Test – Info about Test- Remember: dzed means the animal has the dz, or infection or condition of interest (which could be anything).

  4. Which type of test produces a lot of false positives? – Q1 • High sensitivity • High specificity • Low sensitivity • Low specificity

  5. Relationship Between Se and Sp • Sensitivity ONLY deals with diseased animals. • Se tells you nothing about Sp, because Se and Sp are independent. • Also, Sp tells you nothing about Se and only deals with non-diseased animals.

  6. Which of these correctly describes a VERY SENSITIVE test? – Q2 • Many true positive test results • Many false positive test results • Few true positive test results • Few false positive test results

  7. Which type of test produces a lot of false negative test results? – Q3 • High sensitivity • High specificity • Low sensitivity • Low specificity

  8. Which of these correctly describes a VERY SPECIFIC test? – Q4 • Many true positive test results • Many true negative test results • Few true positive test results • Few true negative test results

  9. Measuring Sensitivity Produces Two Results: Sensitivity is measured by testing dzed animals • True Positive Results • False Negative Results High Sensitivity test produces: • Many true positive test results • Few false negative test results Low Sensitivity test produces: • Few true positive test results • Many false negative test results

  10. Measuring Specificity Produces Two Results Specificity is measured by testing non-dzed animals • True Negative Results • False Positive Results High Specificity test produces: • Many true negatives test results • Few false positives test results Low Specificity test produces: • Few true negatives test results • Many false positives test results

  11. Two by two tables……MEMORIZE • Memorize and learn how to use them!!!!! • Use them to calculate sensitivity and specificity, PPV and NPV. • In an exam, draw the table and fill in the squares. • Using a two by two table will prevent you from making mistakes, especially with PPV and NPV.

  12. This is a Two by Two TableLearn it, and use it for Se, Sp, PPV, and NPV. TN TN TP TP Sensitivity = × 100% PPV = × 100% TN + FN TN + FP TP + FN TP + FP Specificity = × 100% NPV = × 100%

  13. The columns are the definition of Se and Sp Se Sp TN TP TN + FP TP + FN Sp = Se =

  14. The rows are the definition of PPV and NPV PPV NPV TN TP NPV = TN + FN TP + FP PPV =

  15. Tests for Liver Flukes in Cattle • Necropsy = Gold Standard for Liver Flukes • 10,000 cattle tested with necropsy, 20% (2,000) are positive (truly infected with liver flukes) • Coproscopy • 1,400 of 2,000 dzed cattle test positive with coproscopy • 600 false negatives • all 8,000 healthy cattle test negative with coproscopy • 0 false positives • What is the Se and Sp of coproscopy? Example from: Rapsch, et al. 2006. Int J Parasitol. 36:1153-1158

  16. 1,400 of 2,000 dzed test positive: 600 false negativesAll 8,000 healthy cattle test negative: 0 false positives TN TP TN + FP TP + FN Step One Put in the total number (N) in the corner. The total column must add up to 10,000. The total row must add up to 10,000. Sensitivity = × 100% Specificity = × 100%

  17. 1,400 of 2,000 dzed test positive: 600 false negativesAll 8,000 healthy cattle test negative: 0 false positives TN TP TN + FP TP + FN Step Two Fill in the total number of diseased. Then, fill in the diseased column (true positives and false negatives). Make sure they add up to 2,000. Sensitivity = × 100% Specificity = × 100%

  18. 1,400 of 2,000 dzed test positive: 600 false negativesAll 8,000 healthy cattle test negative: 0 false positives TN TP TN + FP TP + FN Step Three Fill in the total number of healthy animals. Make sure the totals add up to N (10,000). Then, fill in the healthy column (true negatives and false positives). Make sure they add up to 8,000. Sensitivity = × 100% Specificity = × 100%

  19. 1,400 of 2,000 dzed test positive: 600 false negativesAll 8,000 healthy cattle test negative: 0 false positives TN TP TN + FP TP + FN Step Four Sum the test positives and test negatives and add the totals, making sure the totals sum to N (10,000). Sensitivity = × 100% Specificity = × 100%

  20. 1,400 of 2,000 dzed test positive: 600 false negativesAll 8,000 healthy cattle test negative: 0 false positives TN TP TN + FP TP + FN Step Five Fill in the Se and Sp equations from the squares in the two by two table. Sensitivity = × 100% Specificity = × 100%

  21. 1,400 of 2,000 dzed test positive: 600 false negativesAll 8,000 healthy cattle test negative: 0 false positives 8000 1400 TN TP 1400 + 600 8000 + 0 TN + FP TP + FN × 100% = 70% Sensitivity = × 100% = × 100% = 100% = Specificity = × 100%

  22. Tests for Liver Flukes in Cattle – Q5 • Necropsy = Gold Standard for Liver Flukes • 10,000 cattle tested with necropsy, 20% (2,000) are positive (truly infected with liver flukes) • Serology (ELISA) • 1,880 of 2,000 dzedcattle test positive with the ELISA • 120 false negatives • 7,680 healthy cattle test negative with the ELISA • 320 false positives • Sp of the ELISA? • Calculate this showing the table Example from: Rapsch, et al. 2006. Int J Parasitol. 36:1153-1158

  23. Tests for Liver Flukes in Cattle • Necropsy = Gold Standard for Liver Flukes • 10,000 cattle tested with necropsy, 20% (2,000) are positive (truly infected with liver flukes) • Coproscopy • 1,400 of 2,000 dzed cattle test positive with coproscopy • 600 false negatives • all 8,000 healthy cattle test negative with coproscopy • 0 false positives • What is the PPV and NPV of coproscopy? Example from: Rapsch, et al. 2006. Int J Parasitol. 36:1153-1158

  24. 1,400 of 2,000 dzed test positive: 600 false negativesAll 8,000 healthy cattle test negative: 0 false positives TN TP TN + FN TP + FP Steps One to Four Are already completed for the Se, Sp calculation PPV = × 100% Step Five Fill in the PPV and NPV equations from the squares in the two by two table. NPV = × 100%

  25. 1,400 of 2,000 dzed test positive: 600 false negativesAll 8,000 healthy cattle test negative: 0 false positives 1400 8000 TN TP 8000 + 600 1400 + 0 TN + FN TP + FP × 100% = 100% PPV = × 100% = × 100% = 93% = NPV = × 100%

  26. Tests for Liver Flukes in Cattle – Q6 • Necropsy = Gold Standard for Liver Flukes • 10,000 cattle tested with necropsy, 20% (2,000) are positive (truly infected with liver flukes) • Serology (ELISA) • 1,880 of 2,000 dzedcattle test positive with the ELISA • 120 false negatives • 7,680 healthy cattle test negative with the ELISA • 320 false positives • What is the PPV and NPV of the ELISA? • Calculate these showing the table Example from: Rapsch, et al. 2006. Int J Parasitol. 36:1153-1158

  27. Let’s say…… • You move to a new practice area where the prevalence of liver flukes is much lower (5%). • What will be the PPV and NPV of coproscopy (Se = 70%, Sp = 100%) in your new practice area?

  28. Coproscopy Se = 70%. Sp = 100 %5% of the population infected TN TP TN + FN TP + FP Step One Start with an empty two by two table. Arbitrarily select an N, but make it fairly large; in this case I selected 1,000. It’s easier using a large number because there will be no decimals. PPV = × 100% NPV = × 100%

  29. Coproscopy Se = 70%. Sp = 100 %5% of the population infected TN TP TN + FN TP + FP Step Two 5% of the population is infected with liver flukes, so, fill in the total number infected. 5% of 1000 (0.05 X 1000 = 50) PPV = × 100% NPV = × 100%

  30. Coproscopy Se = 70%. Sp = 100 %5% of the population infected TN TP TN + FN TP + FP Step Three Fill in the total number of healthy (1,000-50 = 950) animals PPV = × 100% NPV = × 100%

  31. Coproscopy Se = 70%. Sp = 100 %5% of the population infected TN TP TN + FN TP + FP Step Four Fill in the TP by multiplying the total number diseased by the Se (50 x .70 = 35). PPV = × 100% NPV = × 100%

  32. Coproscopy Se = 70%. Sp = 100 %5% of the population infected TN TP TN + FN TP + FP Step Five Fill in the FN by subtracting the TP from the total number diseased (50 – 35 = 15). PPV = × 100% NPV = × 100%

  33. Coproscopy Se = 70%. Sp = 100 %5% of the population infected TN TP TN + FN TP + FP Step Six Fill in the TN by multiplying the total number of healthy animals by the Sp (950 x 100 = 950). PPV = × 100% NPV = × 100%

  34. Coproscopy Se = 70%. Sp = 100 %5% of the population infected TN TP TN + FN TP + FP Step Seven Fill in the FP by subtracting the TN from the total number of healthy animals (950 - 950 = 0). PPV = × 100% NPV = × 100%

  35. Coproscopy Se = 70%. Sp = 100 %5% of the population infected TN TP TN + FN TP + FP Step Eight Fill the row totals by adding the rows. Total test positives = 35 + 0 = 35. Total test negatives = 15 + 950 = 965. Make sure the totals add up to N. PPV = × 100% NPV = × 100%

  36. Coproscopy Se = 70%. Sp = 100 %5% of the population infected TN TP TN + FN TP + FP Step Nine Fill in the PPV and NPV equations from the cells in the two by two table. PPV = × 100% NPV = × 100%

  37. Coproscopy Se = 70%. Sp = 100 %5% of the population infected 950 TN TP 35 950 + 15 TN + FN TP + FP 35 + 0 × 100% = 100% PPV = × 100% = × 100% = 98% = NPV = × 100%

  38. Let’s say…… • You move to a new practice area where the prevalence of liver flukes is much lower (5%). • What will be the PPV and NPV of the ELISA in your new practice area? • Try this on your own, and compare to the next slide. • NOTE: use the Se and SP that you calculated for the ELISA.

  39. Problem Set The SNAP 4Dx canine test kit produced by IDEXX detects heartworm antigen, Ehrlichia canis antibodies, Anaplasma phagocytophilum antibodies, and Borrelia burgdorferi (Lyme disease) antibodies. This test does not detect anti-vaccine antibodies, but is specific for natural infection.

  40. Sensitivity & Specificity • When independent researchers compared the Idexx SNAP to the gold standard of IFA + Western blotting, the Idexx SNAP assay detected 238 of 252 positive dogs • What is the sensitivity? • When other researchers tested 987 dogs without Lyme disease with the Idexx SNAP assay, 982 of them tested negative • What is the specificity?

  41. According to the Manufacturer • Idexx performed their own testing in-house, and they advertise a sensitivity of 95.0% and a specificity of 99.7%, slightly better than that obtained by outside researchers. • Lyme disease is most common in the northeastern USA, where the tick vector is often found outside during the summertime.

  42. Scenario 1: An Endemic area – Q7 • Lyme disease is most common in the northeastern USA, with 20.63% prevalence of disease in dogs in Massachusetts. • Let us say you test 5,000 dogs from MA. • Given the prevalence of disease, • How many would have Lyme disease? • How many would be healthy? Calculate showing how you got the number. Calculate showing how you got the number.

  43. Filling in the Grid

  44. Scenario 1, Continued • Using the sensitivity and specificity data provided by Idexx, predict how many of the Lyme disease dogs would test positive. • Sensitivity = 95.0% • So, 95.0% of the dogs with Lyme disease will test positive using the SNAP test. • How many diseased dogs will test negative? 95.0% of 1,032 = 980 dogs 5.0% of 1,032 (all the other infected dogs) = 52 dogs

  45. Filling in the Grid

  46. Scenario 1, Continued • Using the sensitivity and specificity data provided by Idexx, predict how many of the healthy dogs would test negative? • Specificity = 99.7% • So, 99.7% of the healthy dogs will test negative using the SNAP test. • How many healthy dogs will test positive? 99.7% of 3,968 = 3,956 dogs 0.3% of 3,968 (all the other healthy dogs) = 12 dogs

  47. The Finished Grid What is the PPV? = 980 ÷ (992) = 98.8% What is the NPV? = 3,956 ÷ (4,008) = 98.7%

  48. Scenario 2: Rare Disease • The prevalence in Florida is much lower, estimated to be 0.17%. • If you test 5,000 dogs from Florida, given the sensitivity (95.0%) and specificity (99.7%) provided by IDEXX, • How many dogs would have Lyme disease? • How many dogs would be healthy? 0.17% of 5,000 = 8 dogs All the others! = 4,992 dogs

  49. Scenario 2, Continued • How many of the Lyme disease dogs would test positive? • How many diseased dogs will test negative? • How many healthy dogs would test negative? • How many healthy dogs would test positive? 95.0% of 8 = 8 dogs 5.0% of 8 = 0 dogs 99.7% of 4,992 = 4,977 dogs 0.3% of 4,992 = 15 dogs

  50. Scenario 2 What is the PPV? = 8 ÷ (23) = 35.6% What is the NPV? = 4,977 ÷ (4,977) = 100.0%

More Related