How to Design and Interpret Observational Outcomes Studies in Cardiovascular Disease

1. How to Design and Interpret Observational Outcomes Studies in Cardiovascular Disease Nathan D. Wong, PhD, FACC Professor and Director Heart Disease Prevention Program Division of Cardiology, UC Irvine Adjunct Professor of Epidemiology, UCLA and UC Irvine President, American Society for Preventive Cardiology

2. Why are papers rejected for publication? (The Top 11 Reasons) • The study did not address an important scientific issue • The study was not original • The study did not actually test the authors’ hypothesis • A different type of study should have been done • Practical difficulties led the authors to compromise on the original study protocol (e.g., recruitment, procedures) Greenhalgh T, BMJ 1997; 15: 243-6

3. Reasons 6-11 for Paper Rejection • The sample size was too small • The study was uncontrolled or inadequately controlled • The statistical analysis was incorrect or inappropriate • The authors drew unjustified conclusions from the data • There is a significant conflict of interest among authors • The paper is so badly written that it is incomprehensible

4. Critical Appraisal • Why was the study done, and what clinical question is being asked? (a brief background, review of the literature, and aim / hypothesis should be stated) • What type of study was done? (experiment, clinical trial, observational cohort or cross-sectional study, or survey)

5. Critical Appraisal (cont.) 3. Was the design appropriate for the research? • Clinical trial preferred to test efficacy of treatments • Cross-sectional study preferred for testing validity of diagnostic/screening tests or risk factor associations • Longitudinal cohort study preferred for prognostic studies • Case-control study best to examine effects of a given agent in relation to occurrence of an illness, esp. rare illnesses (e.g., cancer)

6. Outline • Elements of Designing a Research Protocol • Concepts of Study Design: Observational cross-sectional, case-control, cohort studies • Advantages and Disadvantages of Different Study Designs – which is right for you? • Analysis of Observational Studies

7. Nine Key Elements of a Research Study Protocol • Background • Hypotheses • Clinical Relevance • Specific Aims / Objectives • Methodology • Power / Sample Size • Measures and Outcomes • Data Management • Statistical Methodology (UCI School of Medicine Scientific Review Committee)

8. Background • A brief review of the problem to be studied and of related studies that generated the rationale and the central idea of the proposed study. Several pertinent references should be provided.

9. Was the study original? • Few studies break entirely new ground • Many studies add to the evidence base of earlier studies which may have had other or more limitations • Meta-analyses depend on literature containing multiple studies addressing a question in a similar manner

10. Features Distinguishing New vs. Previous Studies • Sample size • Length of follow-up • More rigorous methodology • Different population studied different from that of previous studies (ages, gender, ethnic groups)? • Does the new study address a clinical issue of sufficient importance? Greenhalgh T, BMJ 1997; 315: 305-8

11. Specific Aims / Objectives • What the study is intended to study or demonstrate; includes mention of predictor and outcome (or endpoint) variables. • For example: "The primary aim of the study is to examine whether treatment A is more effective than treatment B in reducing levels of C", or "in finding out whether X is associated with Y", etc. • There may both principal and secondary aims

12. Elements of a Formulated Question • Patient or Population: Who is the question about? (e.g., pts with diabetes mellitus) • Intervention or Exposure: What is being done or what is happening to the patient/population? (e.g., tight control) • Outcome(s): How does the intervention affect the patient/population (mortality, CHD incidence) • Comparison(s): What could be done instead of the intervention? (e.g., standard management)

13. Hypotheses • The problem/s stated in the Background may generate a primary hypothesis and possibly one or two secondary hypotheses. • A hypothesis is often stated in the null – e.g., "No difference between treatments A and B" is anticipated, or "No association between X and Y exists". • Alternatively, it can be stated according to what one expects e.g., “A will be more effective than B in reducing levels or symptoms of C", or “X will be associated with Y".

14. Clinical / Community Relevance • In the case of clinical studies, the potential value in the understanding, diagnosis, or management of a clinical condition or pathological state should be stated. • Funding agencies often now require a statement of community relevance – e.g., how will the results be translated and disseminated to the target population or community.

15. Methodology • Methodology should validate or not validate the hypothesis and specific aims using procedures consistent with sound scientific study design including: • the size and nature of the subjects studied • recruitment, screening, and enrollment procedures • inclusion and exclusion criteria • treatment schedules, and follow-up procedures, if applicable. A chart of the studies to be performed at each visit and the time of each visit and test is needed.

16. Study Population Issues • How were the subjects recruited? Is there potential recruitment bias (e.g., from taking respondents of advertisements), or is survey done in a random (e.g., random digit-dialing) or consecutive sample? • Who was included? Many trials exclude those who have co-morbidities, do not speak English, or take other medications—may provide scientifically clean results, but may not be representative of disease in question.

17. Study Population (cont.) • Who was excluded? Study may exclude those with more severe forms of disease, therefore limiting generalizibility • Were subjects studied in “real-life” circumstances? Is the consenting process describing the benefits/risks, access to study staff, equipment available, etc. be similar to that in an ordinary practice situation?

18. Power / Sample Size • A power/sample size analysis should include an estimate of minimum effect or difference expected at a given level of power when the sample size is fixed, or a projection of the number of subjects needed to achieve a clinically important difference in what is being examined in the hypotheses and the specific aims.

19. Measures and Outcomes • Includes both independent (predictor) and dependent (outcome) variables. • Outcomes include what the investigator is trying to predict, e.g., new or recurrent onset of a disease state, survival, or lowering of cholesterol. • The independent or predictor variables should always include treatment status (e.g., active vs. placebo) in the case of a clinical trial, or primary variables of interest (such as age, gender, levels of X at baseline) for other studies. • The measures and outcomes should expect to answer the proposed question and the importance of the knowledge expected from the research.

20. Data Management • Data Management includes how data is captured for analysis and the tools that will be utilized while capturing the data. This includes: • Case report forms for clinical trials • Surveys, questionnaires, or interview instruments • Computerized spreadsheets or entry forms • Methods for data entry, error checking, and maintenance of study databases

21. Statistical Methods of Analysis • Statistical analysis includes a description of the statistical tests planned to perform to examine the results obtained, e.g., • Student’s t-test will be used to compare levels of A and B between treatment and placebo groups • Multiple logistic regression analysis will be used to examine an independent treatment effect on the likelihood of recurrent disease.

22. Hierarchy of Evidence (for making decisions about clinical interventions or proving causation) • Systematic reviews and meta-analyses • Randomized controlled trials with definitive and clinically significant effects • Randomized controlled trials with non-definitive results • Cohort studies • Case-control studies • Cross-sectional surveys • Case reports

23. Features Affecting Strength and Generalizability of Study • sample size • selection of comparison group (control or placebo) • selection of study sample (is it representative of population the study results are intended to apply to?) • length of time of follow-up • outcome assessed (e.g., hard vs. soft or surrogate endpoint) • Measurement and ability to control for potential confounders

24. Case Reports and Series • Provides “anectdotal” evidence about a treatment or adverse reaction • Often with significant detail not available in other study designs • May generate hypotheses, help in designing a clinical trial. • Several reports forming a “case series” can help establish efficacy of a drug, or thru adverse reports, cause its demise (example: Cerivastatin fatal cases of rhabdomyolysis).

25. Observational Studies • Cross-sectional, prospective, and case-control studies seldom can identify two groups of subjects (exposed vs. unexposed or cases vs. controls) that are similar (e.g., in demographic or other risk factors). • Much of the controlling for baseline and/or follow-up differences in subject characteristics occurs in the analysis stage (e.g., multivariable analysis as in Framingham)

26. Observational Studies (cont.) • While statistical procedures may be done correctly, have we considered all possible confounders? • Some covariates may not have been measured as accurately as possible, and more often, may not be even known or measured.

27. Observational, cross-sectional • Examines association between two factors (e.g, an exposure and a disease state) assessed at a single point in time, or when temporal relation is unknown • Example: Prevalence of a known condition, association of risk factors with prevalent disease. • Conclusions: Associations found may suggest hypotheses to be further tested, but are far from conclusive in proving causation

28. Cross-Sectional Studies and Surveys • Examples: NHANES III, CHIS (telephone), chart-review studies • Surveys should include a representative, ideally randomly-chosen (rather than a small sample of approached subjects who actually agree to be surveyed) sample. • Data collected cannot assume any directionality in exposure / disease. • Can statistically adjust for confounders, but difficult to establish the temporal nature of exposure and disease.

29. Prevalence of CHD by the Metabolic Syndrome and Diabetes in the NHANES Population Age 50+ 19.2% 13.9% CHD Prevalence 8.7% 7.5% No MS/No DM MS/No DM DM/No MS DM/MS % of Population = 28.7% 2.3% 14.8% 54.2% Alexander CM et al. Diabetes 2003;52:1210-1214..

30. Prospective (Cohort) Studies • Cohort studies begin with identification of a population, assessment of exposure (e.g., lipid or BP levels) • Follow-up to the occurrence of outcomes (CHD events)-- temporal sequence (e.g, follow-up time) to events is known

31. Cohort Studies (cont.) • Difficult to ascertain effect of exposure because of many differences between exposed and unexposed groups (confounding factors). • Statistical adjustment for known risk factor differences can help, but unknown factors that may differ between exposed and unexposed groups will never be adjusted for.

32. Duration of Follow-up • Is the planned follow-up reasonable and practical for the study question and sample size utilized? • effect of a new painkiller on degree of pain relief may only require 48 hours • effect of a cholesterol medication on mortality may require 5 years

33. Prospective cohort studies • Examples: • Framingham Heart Study • Cardiovascular Health Study (CHS) • Multiethnic Study of Atherosclerosis (MESA) • Nurses Health Study • Advantages: • large sample size • ability to follow persons from healthy to diseased states • temporal relation between risk factor measures and development of disease

34. Prospective Studies (cont.) • Disadvantages: • expensive due to large sample size often needed to accrue enough events • many years to development of disease • possible attrition • causal inference not definitive as difficult to consider all potential confounders

35. Framingham Heart Study • Longest running study of cardiovascular disease in the world • Began in 1948 with original cohort of 5,209 subjects aged 30-62 at baseline • Biennial examinations, still ongoing, most of original cohort deceased • Offspring cohort of 5,124 of children of original cohort enrolled in 1971, and more recently and still being enrolled to better understand genetic components of CVD risk are up to 3,500 grandchildren of the original cohort. • Routine surveillance of cardiovascular disease events adjudicated by panel of physicians

36. Framingham Most Significant Milestones • 1960 Cigarette smoking found to increase the risk of heart disease • 1961 Cholesterol level, blood pressure, and electrocardiogram abnormalities found to increase the risk of heart disease • 1967 Physical activity found to reduce the risk of heart disease and obesity to increase the risk of heart disease • 1970 High blood pressure found to increase the risk of stroke • 1976 Menopause found to increase the risk of heart disease • 1978 Psychosocial factors found to affect heart disease • 1988 High levels of HDL cholesterol found to reduce risk of death • 1994 Enlarged left ventricle (one of two lower chambers of the heart) shown to increase the risk of stroke • 1996 Progression from hypertension to heart failure described

37. Low HDL-C Levels Increase CHD Risk Even When Total-C Is Normal (Framingham) 12.50 11.91 11.91 14 9.05 10.7 11.24 12 6.6 10 5.53 3.83 6.56 8 14-y incidence rates (%) for CHD 4.85 6 4.67  260 2.06 4.15 3.77 4 2.78 230–259 2 200–229 Total-C (mg/dL) 0 < 200 < 40 40–49 50–59  60 HDL-C (mg/dL) Risk of CHD by HDL-C and Total-C levels; aged 48–83 y Castelli WP et al. JAMA 1986;256:2835–2838

38. Cardiovascular Health Study • 5,201 Medicare eligible individuals aged 65-102 at baseline enrolled beginning 1992 at six field centers. • Assessment of newer and older risk factors. • Ongoing follow-up of cardiovascular events and mortality • Subclinical disease measures included: • carotid B-mode ultrasound for carotid IMT at Year 2, Year 7, and Year 11 • m-mode echocardiographic measures of left ventricular mass and dimensions, left atrial dimension done at baseline (Year 2) (at UC Irvine) and follow-up (Year 7) examinations. • Ankle brachial index (ABI) for measurement of PAD • Pulmonary function (FVC and FEV1)

40. Cardiovascular Health Study: Combined intimal-medial thickness predicts total MI and stroke Cardiovascular Health Study (CHS) (aged 65+): MI or stroke rate 25% over 7 years in those at highest quintile of combined IMT (O’Leary et al. 1999)

41. Case-control Studies • Most frequent type of epidemiologic study, can be carried out in a shorter time and require a smaller sample size, so are less expensive • Only practical approach for identifying risk factors for rare diseases (where follow-up of a large sample for occurrence of the condition would be impractical) • Selection of appropriately matched control group (e.g., hospital vs. healthy community controls) and consideration of possible confounders crucial • Relies on historical information to obtain exposure status (and information on confounders)

42. Case-Control Studies (cont.) • Cannot determine for sure whether exposure preceded development of disease • Also difficult to identify all differences between cases and controls that can be statistically adjusted for

43. Example of case-control study: Folate and B6 intake and risk of MI (Tavani et al. Eur J Clin Nutr 2004) • Cases were 507 patients with a first episode of nonfatal AMI, and controls were 478 patients admitted to hospital for acute conditions • Information was collected by interviewer-administered questionnaires • Compared to patients in the lowest tertile of intake, the ORs for those in the highest tertile were 0.56 (95% CI 0.35-0.88) for folate and 0.34 (95% CI 0.19-0.60) for vitamin B6. • Author conclusion: A high intake of folates, vitamin B6 and their combination is inversely associated with AMI risk

44. Potential sources of bias and error in case control studies • Information on the potential risk factor or confounding variables may not be available from records or subjects’ memories • Cases may search for a cause of their disease and be more likely to report an exposure than controls (recall bias) • Uncertainty as to whether agent caused disease or whether occurrence of the disease caused the person to be exposed to the agent • Difficulty in assembling a case group representative of all cases, and/or assembling an appropriate control group

45. Prospective, observational: nested case-control • In this design, one takes incident cases (e.g., incident CVD) and a matched set of controls to examine the association of a risk factor measured sometime before development of the outcome of interest • Less costly than a true prospective design where all subjects are included in analysis; may not provide equivalent estimates

46. Prospective study of CRP and risk of future CVD events among apparently healthy women (Ridker et al., Circulation 1998) – a nested case control study • 122 female pts who suffered a first CVD event and 244 age and smoking-matched controls free of CVD • Logistic regression estimated relative risks and 95% CI’s, adjusted for BMI, diabetes, HTN, hypercholesterolemia, exercise, family hx, and trt • Those who developed CVD events had higher baseline CRP than controls; those in the highest quartile of CRP had a 4.8-fold (4.1 adjusted) increased risk of any vascular event. For MI or stroke, RR=7.3 (5.5 adjusted)

47. hs-CRP Adds to Predictive Value of TC:HDL Ratio in Determining Risk of First MI Relative Risk hs-CRP Total Cholesterol:HDL Ratio Ridker et al, Circulation. 1998;97:2007–2011.

48. Examples where observational studies have taken us down the wrong path…… • Meta-analysis of observational studies have shown a 50% lower risk of CHD among estrogen users vs. non-users (which may have had many unknown differences that were not adjusted for), but recently randomized trials (HERS, WHI) show no benefit • Numerous prospective studies show a 25-50% lower risk of CHD among those taking vitamin E and other antoxidants vs. placebo– recent randomized trials (e.g., HOPE, HPS) show no benefit.

49. Randomized Clinical Trial • Considered the gold standard in proving causation– e.g., by “reducing” putative risk factor of interest • Randomization “equalizes” known and unknown confounders/covariates so that results can be attributed to treatment with reasonable confidence • Inclusion and exclusion criteria can often be strict (to maximize success of trial) and may require screening numerous patients for each patient randomized

50. Randomized Clinical Trials (2) • Expensive, labor intensive, attrition from loss to follow-up or poor compliance can jeopardize results, esp. if more than outcome difference between groups • Conditions are highly controlled and may not reflect clinical practice or the real world • Funding source of study and commercial interests of investigators can raise questions about conclusions of study