1 / 29

Approaches to test evaluation

Evan Sergeant AusVet Animal Health Services. Approaches to test evaluation. Comparing tests. Kappa – how well tests agree McNemar’s chi-sq – are tests significantly different?. Kappa. Expected no. both +ve = (157 x 155)/1122 = 21.7 Expected no. both -ve = (965 x 967)/1122 = 831.6

lane-mendez
Download Presentation

Approaches to test evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evan Sergeant AusVet Animal Health Services Approaches to test evaluation

  2. Comparing tests • Kappa – how well tests agree • McNemar’s chi-sq – are tests significantly different?

  3. Kappa • Expected no. both +ve = (157 x 155)/1122 = 21.7 • Expected no. both -ve = (965 x 967)/1122 = 831.6 • Total Agreement = 1052 • Chance Agreement = 853.4 • K=(1052-853.4)/(1122-853.4) = 0.739

  4. McNemar Chi-Squared McNemar's Chi-squared test with continuity correction McNemar's chi-squared = 22.881, df = 1, p-value = 1.724e-06

  5. OJD AGID and ELISA • Enter data into epitools • Application of diagnostic tests > compare 2 tests • see kappa, McNemar’s and level of agreement

  6. Gold Standard Tests • Use tests with perfect sensitivity and/or specificity to identify the true disease status of the individual from which the samples were taken. • What are the advantages and disadvantages of this approach?

  7. Gold Standards Tests • Advantages • Known disease status, • Relatively simple calculations • Disadvantages • May not exist, or be prohibitively expensive • Rare diseases may only allow small sample size • Disease may not be present in the country? • Difficult to get representative (or even comparable) samples of diseased/non-diseased individuals

  8. Exercises • Calculate Se and Sp for OJD AGID using data provided in OJD_AGID_Data.xls • Calculate confidence limits using epitools

  9. Non-gold standard methods • Do not depend on determining true infection status of individual. • Rely on statistical approaches to calculate best fit values for Se and Sp. • Tests must satisfy some important assumptions.

  10. Comparison with a knownreference test • Assumptions • Independence of tests • Se/Sp of reference test is known. • For ~100% specific reference test, • Se(new test) = Number positive both tests / Total number positive to the reference test

  11. Culture vs Serology • Estimate sensitivity of culture and serology (as flock tests) • Serology followed-up by histopathology to confirm flock status • Both tests 100% specificity (as flock tests) • How would you estimate sensitivity for these test(s) • Which test has better Se? Is the difference significant?

  12. Example • Se (PFC) = 58/63 = 92% (83% - 97%) • Se (Serology) = 58/95 = 61% (51% - 70%)

  13. Estimation from routine testing data • test-positives are subject to follow-up and truly infected animals are identified and removed from the population • Can be used to estimate specificity when the disease is rare in the population of interest. • Sp = 1 – (Number of reactors / Total number tested)

  14. Se and Sp of equine influenza ELISA • During the equine influenza outbreak in Australia, horses were tested by PCR and serology: • to confirm infection; • to demonstrate seroconversion and/or absence of infection >30 days later; • As part of random and targeted surveillance for case detection, to confirm area status and for zone progression in presumed “EI free” areas. • How could you use the resulting data to estimate sensitivity and specificity of the ELISA?

  15. Equine influenza ELISA • 475 PCR-positive horses, 471 also positive on ELISA • 1323 horses from properties in areas with no infection, 1280 ELISA negative • Analyse in Epitools • Application of diagnostic tests> test evaluation against gold standard • Sergeant, E. S. G., Kirkland, P. D. & Cowled, B. D. 2009. Field Evaluation of an equine influenza ELISA used in New South Wales during the 2007 Australian outbreak response. Preventive Veterinary Medicine, 92, 382-385.

  16. Mixture modelling • Assumptions • observed distribution of test results (for a test with a continuous outcome reading such as an ELISA) is actually a mixture of two frequency distributions, one for infected individuals and one for uninfected individuals • Opsteegh, M., Teunis, P., Mensink, M., Zuchner, L., Titilincu, A., Langelaar, M. & van der Giessen, J. 2010. Evaluation of ELISA test characteristics and estimation of Toxoplasma gondii seroprevalence in Dutch sheep using mixture models. Preventive Veterinary Medicine.

  17. Latent Class Analysis • What is Latent Class Analysis? • Maximum Likelihood • Bayesian

  18. Maximum likelihood estimation • Assumptions • The tests are independent conditional on disease status (the sensitivity [specificity] of one test is the same, regardless of the result of the other test); • The tests are compared in two or more populations with different prevalence between populations; • Test sensitivity and specificity are constant across populations; and • There are at least as many populations as there are tests being evaluated. • TAGS software • Hui, S. L. & Walter, S. D. 1980. Estimating the error rates of diagnostic tests. Biometrics, 36, 167-171.

  19. TAGS • Open R – shortcut in root directory of stick • Open tags.R in text editor or word • Select all and copy/paste into R console • Type TAGS() and <Enter> to run • Hui Walter example • 2 tests for TB • Test 1 = Mantoux • Test 2 = Tine test

  20. Follow the prompts to enter data: • Data set = new • Name = test • Number of tests = 2, Number of populations = 2 • Reference population? = No (0) • Enter results for each population from table below • Best guesses use defaults • Bootstrap CI = Yes (1000 iterations) Data

  21. $Estimations pre1 pre2 Sp1 Sp2 Se1 Se2 Est 0.0268 0.7168 0.9933 0.9841 0.9661 0.9688 CIinf0.0159 0.6911 0.9797 0.9684 0.9495 0.9540 CIsup0.0450 0.7412 0.9978 0.9921 0.9774 0.9790

  22. Bayesian estimation • What is Bayesian estimation? • Combines prior knowledge/belief (what you think you know) with data to give best estimate • Incorporates existing knowledge on parameters (Se, Sp, prevalence) • “Priors” entered as probability (usually Beta) distributions • Uses Monte Carlo simulation to solve • Outputs also as probability distributions • Can get very complex • Assumptions • Independence of the tests • Appropriate prior distributions chosen. • Need information on prior probabilities • Some methods can adjust for correlated tests • Multiple tests in multiple populations

  23. Methods • EpiTools (only allows one population so must have good information on one or more test characteristics) • WinBUGS models

  24. Bayesian analysis surra data

  25. EpiTools • Run EpiTools > Estimating true prevalence > Bayesian estimation with two tests • Enter parameters: • Data from 2x2 table: 0, 39, 0, 251 • Prevalence = Beta(1,1) (uniform = don’t know) • Test 1 (CATT): Se = Beta(82, 20), Sp = Beta(160, 2) • Test 2 (ELISA): Se = Beta(76, 26), Sp = Beta(118, 4) • Starting values: 0, 38, 0, 245 • Other values as defaults and click submit

More Related