Reliability of Screening Tests

1. Reliability of Screening Tests RELIABILITY: The extent to which the screening test will produce the same or very similar results each time it is administered. --- A test must be reliable before it can be valid. --- However, an invalid test can demonstrate high reliability.

2. Reliability of Screening Tests • Sources of variability that can affect the reproducibility of results of a screening test: • 1. Biological variation (e.g. blood pressure) • 2. Reliability of the instrument itself • 3. Intra-observer variability (differences in repeated measurement by the same screener) • 4. Inter-observer variability (inconsistency in the way different screeners apply or interpret test results)

3. Use of Multiple Screening Tests Sequential (two-stage) testing: A less expensive, less invasive, or less uncomfortable test is performed first… those who screen positive are referred for further testing using a test which may have greater sensitivity and specificity… reduces false positives, hence an increase in net specificity.

4. Use of Multiple Screening Tests Simultaneous testing: Multiple tests are used simultaneously… Person tests “positive” if there is a positive result on any of the tests employed… reduces false negatives, hence an increase in sensitivity … but at the expense of decreased specificity.

5. Measuring the performance (yield) of a screening test

6. Performance Yield • People with positive screening test results will also test positive on the diagnostic test: Predictive Value Positive (PV+) • People with negative screening test results are actually free of disease Predictive Value Negative (PV-)

7. Performance Yield True Disease Status - + Results of Screening Test a b + - c d Predictive value positive (PV+): The probability that a person actually has the disease given that he or she tests positive. PV+ = a / (a + b)

8. Performance Yield True Disease Status - + Results of Screening Test a b + - c d Predictive value negative (PV-): The probability that a person is truly disease free given that he or she tests negative. PV- = d / (c + d)

9. Performance Yield True Disease Status - + Results of Screening Test 400 995 + - 98905 100 Sensitivity: a / (a + c) = 400 / (400 + 100) = 80% Specificity: d / (b + d) = 98905 / (995 + 98905) = 99% PV+: a / (a + b) = 400 / (400 + 995) = 29% PV-: d / (c + d) = 98905 / (100 + 98905) = 99%

10. Performance Yield True Disease Status - + Results of Screening Test 400 995 + - 98905 100 PV+: a / (a + b) = 400 / (400 + 995) = 29% Among persons who screen positive, 29% are found to have the disease.

11. Performance Yield True Disease Status - + Results of Screening Test 400 995 + - 98905 100 PV-: d / (c + d) = 98905 / (100 + 98905) = 99.9% Among persons who screen negative, 99.9% are found to be disease free.

12. Performance Yield Factors that influence PV+ and PV- 1. The more specific the test, the higher the PV+ 2. The higher the prevalence of preclinical disease in the screened population, the higher the PV+ 3. The more sensitive the test, the higher the PV-

13. Performance Yield Prevalence (%) Sensitivity Specificity PV+ 0.1 90% 95% 1.8% 1.0 90% 95% 15.4% 5.0 90% 95% 48.6% 50.0 90% 95% 94.7%

14. Performance Yield Thus, the PV+ is maximized when used in “high risk” populations since the prevalence of pre-clinical disease is higher than in the general population…. screening a total population for a relatively infrequent disease can be very wasteful of resources and may yield few previously undetected cases.

15. Effectiveness of Screening Evaluating if the screening program reduced morbidity and mortality from the disease: 1. Overall shift in severity of disease at the time of diagnosis. 2. Compare cause-specific mortality from among those whose disease was picked up by screening versus those with a diagnosis related to symptoms.

16. Effectiveness of Screening Evaluating if the screening program reduced morbidity and mortality from the disease: 3. Reduction in disease-related complications. 4. Improvement of quality of life in screened individuals.

17. Effectiveness of Screening In reality, establishing the sensitivity and specificity of screening tests may be difficult… often times, data are only available on persons who screen positive and are referred for further testing. Data are available for cells “a” and “b” only. Permits calculation of PV+ only

18. Effectiveness of Screening Sources of bias in evaluating screening programs: 1. Self-selection bias (volunteer bias) 2. Lead time bias 3. Length bias 4. Over-diagnosis bias

19. Effectiveness of Screening 1. Self-selection bias (volunteer bias): --- Volunteers for screening programs may be healthier, on average, than persons who do not participate in screening programs. On the other hand…. --- The “worried well” may be more likely to participate and may be at overall higher risk due to family history or lifestyle characteristics.

20. Effectiveness of Screening 2. Lead time bias: Lead time: The interval between “diagnosis” of disease at screening and when it would have been detected from clinical symptoms. --- Survival may appear to be increased among screen-detected cases simply because diagnosis was made earlier in the course of the disease.

21. Effectiveness of Screening 3. Length bias (prognostic selection): ---- The overrepresentation among screen- detected cases of those with a long pre- clinical phase, and thus a more favorable prognosis. --- Those with a long pre-clinical phase are more readily detectable by screening than more rapidly progressing cases with a short pre-clinical phase.

22. Effectiveness of Screening 4. Over-diagnosis bias: ---- Persons who screen positive and are truly disease free (false positives), yet are erroneously diagnosed as having the disease. --- Since these persons are truly disease free, we expect a more favorable long- term outcome – giving the appearance of an effective screening program.

23. Effectiveness of Screening Possible interpretations for null results in a screening program evaluation include: ---- The disease has an extremely short detectable pre-clinical phase. --- Current therapeutic intervention is no more effective when provided earlier than at usual time of diagnosis. --- Inadequacies of care provided to those who screen positive.

24. Effectiveness of Screening Study Designs Used to Evaluate Screening Programs: 1. Ecological Studies 2. Observational Analytic Studies 3. Randomized Trials (infrequent and difficult to carry out)