Study validity, bias and misclassification

1. Summer Course:Introduction to Epidemiology August 28, 0845-1015 Study validity, bias and misclassification Dr. N. Birkett, Department of Epidemiology & Community Medicine, University of Ottawa

2. Consider a precise number: the normal body temperature of 98.6°F. Recent investigations involving millions of measurements have shown that this number is wrong: normal body temperature is actually 98.2°F. The fault lies not with the original measurements - they were averaged and sensibly rounded to the nearest degree: 37°C. When this was converted to Fahrenheit, however, the rounding was forgotten and 98.6°F was taken as accurate to the nearest tenth of a degree.

3. Session Overview • Review what is meant by ‘study validity’. • Explore the main categories of biases which affect study validity and explore some important examples. • Understand misclassification and its role in producing bias.

4. Scenario #1 • Ottawa Citizen headline: • ‘Scientific study shows smoking doesn’t cause lung cancer’. • But, we know there are hundreds of studies showing the opposite. • Why did this study disagree?

5. Scenario #2 • US Cholesterol treatment guidelines: • Mr. Smith had a total cholesterol test done as part of a work-place annual examination and the level was 244. • Follow-up test with family doctor was 198. • Why the difference? • What should Mr. Smith do?

6. Validity (1) • Any measurement is subject to error. • Labs make mistakes. • Poor machine calibration. • Biological variation in subject • Fasting vs. non-fasting • External stress • Diet • Bad luck

7. Validity (2) • Actually, two key concepts are covered under my general title ‘validity’ • Reliability: • Do you get the same result if you repeat a study or test more than once? • If a subject tests +ve for HIV today, will the same test/lab be +ve tomorrow? • Validity: • Does the test or study give the ‘right’ answer? • Does a test for depression actually identify depressed people as opposed to people with anxiety or who are just ‘sad’?

8. Validity (3) DValidity low; reliability high

9. Validity (4) BValidity high; reliability low

10. Validity (5) • Concepts can be applied to: • Individual tests and • Research studies. • We’ve already discussed individual tests • Screening and diagnostic tests. • Focus for rest of the session is on validity issues in research studies • ‘Texas Sharpshooter’

11. Validity (6) • Four possible explanations for a result from an epidemiological study: • Chance • Bias • Confounding • A third factor explains the apparent result (more later) • The TRUTH • Must always consider other explanations before concluding result is true.

12. Validity (7) • For studies, reliability is mainly related to chance factors (random error) • The domain of statistics • Statistical methods attempt to quantify amount of chance effect and aid interpretation of study despite this. • Approaches can get very complex. Sometimes, the results can reflect the models and not the data (a problem, but not discussed here).

13. Validity (8) • Bigger studies have less element of chance

14. Validity (9) • Random chance in selection of subjects An Aside: Mean of the 3 samples is 20%. Quite close to the ‘correct’ prevalence.

15. Validity (10) Before moving to consider bias, let’s take a side-trip. • External vs. internal validity of a study. • External validity (generalizability) • Relates to the degree to which the results of the study reflect the underlying population • A study of the average BMI for Ottawa which only looked at men would have poor external validity for the general population.

16. Validity (11) • External validity (cont.) • Main issue concerns use of volunteers • Volunteers differ from the general population • BUT all studies are done in volunteers • informed consent and ethical issues. • Eligibility criteria • Limiting study to sub-groups of population • Most studies of CHD etiology and treatment have been in men. Do they apply to women? • Harder to study CHD in women due to lower incidence. • Relationships generalize better than means.

17. Validity (12) • Internal validity • Does the study produce a valid estimate of the effect under study? • Largely addressed by study design • Avoid (more to come on these) • Selection bias • Measurement bias • Confounding • Include an appropriate comparison group.

18. Bias (1) • A systematic error in a study which leads to a distortion of the results. • Can be deliberate (fraud) • Much more commonly, due to design weaknesses or problems with study execution. • A more serious issue with observational studies than RCT’s. • Can cause the true RR or OR to be distorted away from, or towards, the null

19. Bias (2)

20. Bias (3)

21. Bias (4) • Two main sources for bias • Categories have over-lap (gray zones) • Selection bias • The process of selecting or retaining study subjects distorts the relationship under study • Informationbias • The process of collecting information on the study participants distorts the relationship under study. • Confounding is sometimes included here (discussed in later session)

22. Selection Bias (1) • Can arise from: • Problems with which people enter a study • Problems with retaining people in the study once they are enrolled. • Can affect • External validity • Internal validity

23. Selection Bias (2)

24. Selection Bias (3) Volunteer bias • People who volunteer are generally different from the general population • Younger • Healthier • Higher SES and education level • Results can potentially be adjusted using statistical models • Requires knowledge of distribution of key factors in general population • A risky approach

25. Selection Bias (4) Healthy worker effect • Do work place exposures have adverse health effects? • Common design is to compare outcomes in workers to a general population (SMR) • Generally, workers will have less disease than general population • Workers are pre-selected to be healthier • Workers who get ill often will retire or quit. • Note: this is a different effect from loosing track of workers who get the target illness (loss-to-follow-up)

26. Selection Bias (5) Clinic/referral bias • One of the causes of hypertension is renal artery stenosis. • In the 1960’s, research showed that about 10% of hypertension was caused by renal artery stenosis • Routine patient evaluation included Intravenous Pyleogram • Expensive • Risk of severe adverse reactions • In the 1970’s, new research showed that less than 0.5% of hypertension had renal artery stenosis. • Why the discrepancy?

27. Selection Bias (6) Clinic/referral bias (cont.) • Early studies were based in specialty hypertension treatment clinics. • They do the research • They got referred the ‘difficult’ or ‘interesting’ cases. • Later studies were done in general practice or with the general public. • They got ‘everyone’. • Referral bias is at work.

28. Selection Bias (7) Loss-to-follow-up • Most serious selection bias problem with RCT’s and cohort studies. • Losses occur at different points of time in the study and for different reasons. • Random losses produce reduced power but no bias • Losses related to probability of getting outcome can produce a serious bias.

29. Selection Bias (8) Loss-to-follow-up (cont.) • Consider this situation • RCT of new medication to treat cancer. In truth, the drug doesn’t work. However, it does have more side effects than the control treatment. Patients can tolerate the side effects when healthy but, as they become terminally ill, the side effects interfere with their quality of life. As a result, patients on the new treatment who are about to die drop out of the study. • Impact: • Mortality rate in the group with the new treatment will be very low, making the new drug look ‘better’ than standard therapy.

30. Selection Bias (9) Control selection in case-control studies • The biggest threat to the internal validity of a case-control study. • Consider this example: • Question: • Does smoking cause lung cancer? • Cases: • All men treated at the Ottawa hospital with lung cancer. • Controls: • Men admitted to the respiratory ward with advanced emphysema • OR = 1.00 • WHY?

31. Selection Bias (10) Control selection in case-control studies (cont.) • Smoking causes emphysema as well as lung cancer. • Hence, the controls will be mostly smokers just like the cases. • Selection bias has distorted the smoking rate in the underlying target population and produced a serious bias.

32. Information Bias (1) • Also called Observation & Measurement Bias • For categorical variables, is essentially the same as misclassification • Arises when the measurement procedure doesn’t give the right answer. • Measuring height using a misassembled rule • Most commonly a problem with questionnaires

33. Information Bias (2) Recall bias • A serious problem for case-control studies • Not a problem for cohort studies • People who have been diagnosed with a disease are more likely to remember past exposure than controls. • A mother of a child born with a serious developmental problem will tend to ruminate on the pregnancy and what might have gone wrong

34. Information Bias (3) Interviewer bias • If interviewer knows the study hypothesis and which subjects are cases and controls, they might probe harder for exposure in one group. • Also, interviewer attitude might influence response patterns. • In an HIV study, cases are homosexual while controls aren’t. If an interviewer were biased against homosexuality, cases may be less forthcoming with sensitive information. • Prevent by blinding interviewers

35. Misclassification (1) Misclassification • Occurs when subject is placed into the wrong category • Classed as a case when didn’t have disease • Classed as a control when did have disease (more common) • Classed as exposed when subject was not exposed. • Classed as not exposed when subject was exposed. • Concept can be extended to multi-level categories and to continuous variables

36. Misclassification (2) • Consider a case-control study. • Subjects who are truly exposed have a 10% chance of being classed as unexposed • Subjects who are truly unexposed have a 20% chance of being classed as exposed. Misclassification table

37. Misclassification (3) The TRUTH ORtrue = 3.5

38. Misclassification (4) Observed The TRUTH 90% are exposed 20% are classed as exposed by error ORtrue = 2.35 ORtrue = 3.5

39. Misclassification (5) • In this example, the OR is biased towards the null. • Non-differential misclassification • The misclassification rates don’t differ between cases and controls. • ‘always’ biases towards null • As long as test isn’t really bad • Ignores random properties • ‘good’ type of misclassification since OR isn’t over-estimated.

40. Misclassification (6) • Differential misclassification • The misclassification rates differ between cases and controls. • Recall bias is a good example. • Cases have better (less misclassified) exposure estimates than do controls. • The bias can be in any direction and (almost) to any extreme desired. • Re-do previous example but now, cases have no misclassification while controls have 20% of exposed subjects misclassified and 5% of the non-exposed subjects.

41. Misclassification (7) Misclassification table Cases Controls

42. Misclassification (3) Observed The TRUTH 80% are exposed 5% are classed as exposed by error ORtrue = 3.95 ORtrue = 3.5

43. Summary • What can we do about bias? • Prevention is the key approach • Good design • Careful attention to issues in the field work of the study • Good strategies to retain study participants. • Once you get to the study analysis, there are very few options to handle bias.