Retrospective Studies of Clinical Outcomes A Primer for Clinicians

1. Retrospective Studies of Clinical OutcomesA Primer for Clinicians Marc D. Silverstein, MD FACP

2. Overview • Role of retrospective studies in a research program • “Anatomy” & “physiology” of research • Observational research designs • Cross-sectional studies • Cohort studies • Case-control studies • Human Subjects & IRB Review

3. Objectives • Describe 4 research designs • Describe 3 threats to validity • List advantages of retrospective studies • List disadvantages of retrospective studies • Understand requirements and processes for IRB review of retrospective studies

4. Research • Definition • Types of Research

5. Definition of Clinical ResearchNIH Director’s Panel, 1997 • Patient-oriented research Research conducted with human subjects (or on material of human origin such as tissues, specimens and cognitive phenomena) for which an investigator (or colleague) directly interacts with human subjects • Mechanisms of human disease • Therapeutic interventions • Clinical trials • Development of new technologies • Epidemiologic and behavioral studies • Outcomes research and health services research

6. Health Services Research • Health services research is the multidisciplinary field of scientific investigation that studies how social factors, financing systems, organizational structures and processes, health technologies, and personal behaviors affect access to health care, the quality and cost of health care, and ultimately our health and well-being. • Its research domains are individuals, families, organizations, institutions, communities, and populations. AcademyHealth

7. Outcomes Research • Research on measures of changes in patient outcomes - patient health status and satisfaction - resulting from specific medical and health interventions • Attributing changes in outcomes to medical care requires distinguishing the effects of care from the effects of the many other factors that influence patients’ health and satisfaction AcademyHealth

8. PatientCare Formulate the research question Translate research into practice OutcomesResearch Patient Care and Outcomes Research

9. Physiology of Research • Design the study • Implement the study • Make valid (causal) inferences

10. Research Question Truth in the Universe The Goal

11. Design & Implementation Design Implement Research Question Actual Study Study Plan

12. Design Implement Research Question Truth in the Universe Actual Study Findings in the Study Study Plan Truth in the Study Infer Infer Causal Inferences

13. Research Question Truth in the Universe Study Plan Truth in the Study Actual Study Findings in the study Target Population Phenomenon of Interest Intended Sample Intended Variables Actual Subjects Actual Measurements Errors May Occur in Design & Implementation of Research Error Error Design Implement Infer Infer

14. Research Question Truth in the Universe Study Plan Truth in the Study Actual Study Findings in the study Target Population Phenomenon of Interest Intended Sample Intended Variables Actual Subjects Actual Measurements Errors May Occur in Making Inferences about Internal or External Validity Design Implement Error Error Infer Infer

15. Research Question Truth in the Universe Study Plan Truth in the Study Actual Study Findings in the study Target Population Phenomenon of Interest Intended Sample Intended Variables Actual Subjects Actual Measurements Errors May Occur Anywhere … Error Error Design Implement Error Error Infer Infer

16. Threats to Validity • Chance • Bias • Confounding

17. Threats to Validity • Chance – random error due to unknown sources of variation that distort sample and measurements in either directions • Bias – systematic error that distorts sample and measurements in one direction • Confounding – an external factor that is associated with a predictor variable and an outcome variable

18. Reducing Random Error • Increase sample size • Statistical analyses

19. Reducing Systematic Error (Bias) • Population-based studies • Inclusion and exclusion data • Standardize measurement • Train and certify observer • Refine instrument • Automate instruments • Blinded measurements

20. Reducing Confounding • Anticipate potential confounders • Measure potential confounders • Matching, restriction, stratification • Multivariate analysis

21. “I cannot give any scientist of any age better advice than this: the intensity of the conviction that a hypothesis is true has no bearing on whether it is true or not.” P.B. Medawar

22. Anatomy of Research • Research Question • Significance • Methods

23. Research Question

24. Characteristics of a Good Research Question • Feasible • Interesting • Novel • Ethical • Relevant

25. “It can be said with complete confidence that any scientist of any age who wants to make important discoveries must study important problems.” P.B. Medawar

26. Hypotheses • Simple (versus complex) • Specific (versus vague) • In advance (versus after-the-fact)

27. Estimation • In clinical studies the goal is often to estimate a risk of an outcome or the magnitude of impact on a clinical measurement

28. Strengths Intervention is not feasible or ethical Rapid & efficient Existing data Less time Expenses are lower Case-control studies for rare events Limitations Do not permit true assessment of time sequence of factors and outcomes Subject to bias and confounding Limited power to study rare risk factors or rare outcomes (surveys & cohort studies) Observational Studies

29. Study Designs • Observational studies • Cross sectional studies • Cohort studies • Case Control studies • Experiments

30. Cross-Sectional Study Design

31. Risk Factor Disease Risk Factor No Disease No Risk Factor Disease No Risk Factor No Disease Cross Sectional Study Population Sample

32. Cross Sectional Study Example The Probability of Malignancy in Solitary Pulmonary Nodules Swensen, Silverstein, Ilstrup et al Arch Intern Med 1997; 157: 849

33. The Probability of Malignancy in Solitary Pulmonary Nodules • Can clinical and radiological SPN characteristics predict malignancy in SPNs • Retrospective cohort at multi-specialty group practice • New diagnosis of 4mm – 30 mm solitary pulmonary nodule • Radiological indeterminate SPN with no calcification on thin section CT • Exclude patients with primary lung cancer or other cancer within 5 years

34. The Probability of Malignancy in Solitary Pulmonary Nodules • Outcomes determined by radiological follow-up for 2 or more years, surgical diagnosis, transthoracic needle biopsy, bronchoscopy biopsy or washings • Clinical characteristics (age, smoking, history of other cancer) and radiological characteristics (size, location, edge characteristics – lobulation, spiculation, shagginess) • Multivariate analysis with logistic regression • Predictors developed in 2/3 random sample and validated in remaining 1/3 sample

35. DiscriminationROC Curve Area (0.83, 0.80)

36. Calibration – Observed vs Predicted Probability

37. Threats to ValidityMalignancy in SPN, 1 • Large number of SPN’s (629) • Referral population • Clinically relevant SPNs (5-30 mm) • Excludes low risk (< 4mm) • Excludes high risk (> 30 mm) • No calcification on thin section CT (benign) • Cancer diagnosis • Cohort study for outcomes after 2 years • Some SPN’s indeterminate classification

38. Threats to ValidityMalignancy in SPN, 2 • Independent review single radiologist • Swenson, Silverstein, Edell et al. SPN: Clinical Prediction vs Physicians, Mayo Clin Proc, 1999 • Analysis • Discrimination and calibration • Independent sample for validation • Distance to assess referral bias • Included all SPNs • Malignant vs (indeterminate + benign) • (Malignant + indeterminate) vs benign

39. Cohort Study Design

40. Population Sample Risk Factor Disease No Disease No Risk Factor Disease No Disease Cohort Study Design

41. Population Sample Risk Factor Disease No Disease No Risk Factor Disease No Disease Prospective Cohort Study Design The Present The Future

42. Population Sample Risk Factor Disease No Disease No Risk Factor Disease No Disease Retrospective Cohort Study Design The Past The Present

43. Cohort Study Example Long-term Survival of a Cohort of Community Residents with Asthma Silverstein, Reed, O’Connell et al N Eng J Med 1994;331: 1537

44. Long-term Survival of a Cohort of Community Residents with Asthma • Asthma mortality based on general US population death certificates with asthma listed as underlying cause of death • Residents of Rochester, MN with first asthma diagnosis 1/1/1964-12/31/1983 • Explicit pre-defined criteria, review of all medical records from all providers of care • Medical records and autopsy reports used to classify deaths as due to asthma or other conditions

45. Long-term Survival of a Cohort of Community Residents with Asthma • 2499 patients with definite or probable asthma • Mean duration follow-up 14 years (range 0-29 years) • 140 deaths in 32,605 person-years of follow-up • Survival not significantly different form expected • Survival worse in asthmatics with other lung disease • 4% of deaths in persons with asthma were due to asthma

46. Observed vs. Expected Survival

47. Survival in Asthma only vs Asthma and Other Lung Disease

48. Threats to ValidityLong-term Survival in Asthma, 1 • Large population-based cohort (2499) • Yunginger, Reed O’Connell, A Community based study of the Epidemiology of asthma, Am Rev Resp Dis, 1992 • Asthma diagnosis • Beard, Yunginger, Reed et al, Interobserver Variability in Medical Record Review: An Epidemiological Study of Asthma, J Clin Epid, 1992

49. Threats to ValidityLong-term Survival in Asthma, 2 • Asthma deaths • 14 years of follow-up • Small number of deaths (140) • Classification of deaths • Hunt, Silverstein, Reed et al. Accuracy of Death Certificate in a Population-Based Study of asthmatic Patients, JAMA, 1993 • Review of all death certificates & autopsy reports • 13 out of state

50. Case-Control Study Design