1 / 54

The Role of Non-randomized Studies in a Randomized World

The Role of Non-randomized Studies in a Randomized World. Intervention Studies in Health Research (Module 2) Department of Epidemiology and Community Medicine University of Ottawa Ottawa, Ontario, Canada. Effectiveness of healthcare interventions Evaluation of effectiveness Methods

cher
Download Presentation

The Role of Non-randomized Studies in a Randomized World

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Role of Non-randomized Studies in a Randomized World Intervention Studies in Health Research (Module 2) Department of Epidemiology and Community Medicine University of Ottawa Ottawa, Ontario, Canada

  2. Effectiveness of healthcare interventions • Evaluation of effectiveness • Methods • Randomized vs Non-randomized studies • Contribution of Randomized Controlled Trials (RCT) and Prospective Cohort Studies

  3. Fact 1: a large, well-designed, properly conducted RCT yields comparison groups similar in every way except for the intervention attribute differences in outcome between groups to differences in effect of the interventions

  4. Fact 2: a non-randomized study may yield comparison groups with dissimilar characteristics differences in outcome between groups may be due to differences in characteristics of the groups rather then the effect of the interventions

  5. Levels of Evidence I Evidence from at least 1 properly randomized controlled trial (RCT) II-1 Evidence from well-designed controlled trials without randomization II-2 Evidence from well-designed cohort or case-control analytic studies preferably from more than one centre or research group II-3 Evidence from comparisons between times and places with or without the intervention; dramatic results in uncontrolled experiments could also be included here III Opinions of respected authorities, based on clinical experience, descriptive studies or reports of expert committees

  6. Levels of Evidence I Results from a single RCT I+ Results from a meta-analysis of RCTs with consistent effects I- Results from a meta-analysis of RCTs with disparate effects Sources of Heterogeneity in Meta-analysis - clinical differences in studies included (Thompson, SG BMJ 1994;309:1351-1355)

  7. Levels of Evidence and Grades of Recommendations for Therapy Level I Grade A Level I Single RCT, LCL > MCIB Level I+ MA of RCTs, consistent, LCL > MCIB Level I- MA of RCTs, disparate, LCL > MCIB Level II Grade B Level II Single RCT, CI overlaps MCIB Level II+ MA of RCTs, consistent, CI overlaps MCIB Level II- MA of RCTs, disparate, CI overlaps MCIB Grade C Level III Non-randomized concurrent cohort studies Level IV Non-randomized historic cohort studies Level V Case series

  8. Current Approach to Grades of Recommendation Grade Clarity of Methodologic Strength Implications Risk/Benefit of Supporting Evidence (recommendation) _______________________________________________________ 1A Clear RCT no important limitations Strong / no reservarion 1B Clear RCT with important limitations Strong / likely 1C+ Clear Extrapolate from RCT or Strong overwhelming evidence from observational studies 1C Clear Observational studies Intermediate ________________________________________________________ (Guyatt & Rennie, User’s Guide to the Medical Literature, 2002)

  9. One View Non-randomized studies tend to report larger estimates of intervention effects than RCTs [ Often expressed view. We should review the evidence whether the size of intervention effect for non-randomized studies are systematically larger (or smaller) than for RCTs ]

  10. Statistical Perspective 'just say no' to non-randomized studies because of 'inherent bias' (Stat Med 1994;13:557-67)

  11. Internal Validity R O X O O X O R O O O O _____________________________________________________ History + + Maturation + + Testing + + Instrumentation + + Regression + + Selection + - Loss + + Interaction + + _____________________________________________________ (Campbell & Stanley, Experimental and Quasi-Experimental Designs for Research, 1963)

  12. Internal Validity Non-randomized Studies: Allocation bias Intervention Outcome A vs B Important Prognostic Characteristics

  13. 1. Design - match - stratified sampling - restrict inclusion 2. Analysis - standardize - stratify (subgroups) - adjust (statistical model) 3. Randomize Can allocation bias be identified? Can allocation bias be accommodated for in the analysis? [ Major criticism of non-randomized studies. We should review whether analysis adjustments for baseline prognostic characteristics in non-randomized studies are useful.]

  14. Internal Validity Randomized Controlled Trials: Patient Preference If intervention allocated randomly then patient (and practitioner) deprived of preference If choice and control are of therapeutic benefit than randomized comparison may reduce estimate of effectiveness 1. Preference studies 2. Adjustment for preference

  15. External Validity Randomized Controlled Trials: Patient ineligible Practitioner non-participant Practitioner preference Patient preference [ Major criticism of randomized studies. We should review whether exclusions in RCTs are justified and generalizing results are appropriate. ]

  16. The Issues (1) Review the evidence whether the size of intervention effect for non-randomized studies are systematically larger (or smaller) than for RCTs. (2) Review whether exclusions in RCTs are justified and generalizing results are appropriate. (3) Review whether analysis adjustments for baseline prognostic characteristics in non- randomized studies are useful.

  17. First Issue Review the evidence whether the size of Intervention effect for non-randomized studies are systematically larger (or smaller) than for RCTs. Chalmers TC et al, NEJM 1983; 309:1358-61 - reviewed comparison groups in MI studies - non-randomized studies tended to have patients with good prognosis in the new treatment groups

  18. Sacks H et al, Am J Med 1982;72:233-40 - compared RCTs with studies using historical controls - non-randomized studies produced larger effects

  19. Observational Studies versus Randomized Controlled Trials: Revisiting the Comparison with Current Data Benson,K. and Hartz,A. (abstract, VIth Cochrane Colloquium, Roma, 1999) - MEDLINE search 1985-98 identifying observational studies evaluating treatment effects - MEDLINE search 1966-98 for RCTs and observational studies examining same treatments and outcomes identified in first search - found 136 articles in 19 treatment areas

  20. Graft Survival After Kidney Transplant Calcium Channel Blockers vs Control Infection Rate after Appendectomy Laparoscopic vs Open Area of Cardiology Nifedipine vs Control in CAD CABG vs PTCA CABG vs Medical Beta-Blockers vs Control

  21. Other Areas Endarteretomy-Local vs General Anaesthetic Geriatric Unit vs Medical Ward Ratinopexy vs Medical Ward Ratinopexy vs Scleral Buckle Intensive Insulin vs Conventional Lithotripsy vs Nephrolithotomy Laser vs Electrosurg Salpingostomy CVS vs Amniocentesis Breast Cancer Chemo+Surg vs Surg Adenoldectomy vs No for Otitis Media Eder-Puestow vs Balloon Dilation

  22. Conclusion: Of the 19 areas, only 2 had observational study results outside the RCT 95% Confidence Interval Little evidence that treatment effects in recent observational studies were either consistently larger or qualitatively different than in RCTs

  23. Comparing the Outcome in RCTs and Non-randomized Studies Britton et al, 1998 - inclusion criteria: RCT compared to non- randomized study, intervention same and setting similar, control arms similar therapy, comparable outcomes - MEDLINE, EMBASE, Science and Social Science Citation Index, Cochrane Library (1996) - found 18 articles

  24. Conclusion: No consistent patterns Does not support suggestion that non- randomized studies produce larger estimates of treatment effects than RCTs

  25. Estimated Treatment Effects (a) RCTs > Non-randomized Studies: - patients receive higher quality care - patients selected have greater ability to benefit (b) RCTs < Non-randomized Studies: - practitioner selection bias in deciding whom to treat - enhanced response among those with a strong preference for a particular treatment - publication bias (c) RCTs = Non-randomized Studies: - same eligibility criteria used - prognostic factors understood, collected and adjusted for

  26. Second Issue Review whether exclusions in RCTs are justified and generalizing results are appropriate RCTs often criticized for: - excluding types of patients on whom treatment advice is sought - using study settings not typical of those in which most patients are treated

  27. Patients Enrolled in A Study Ideally: include all patients with condition the intervention is intended for Could: exclude patients on medical or ethical grounds - risk of adverse event - benefit established - benefit believed unlikely Further: exclude patients on scientific or administrative grounds - increase precision - avoid bias Result: only a small proportion of patients with condition enrolled

  28. Choice in Study Design • Include one very homogeneous subset of patients • - provide precise result • - generalization left to clinical judgement • Versus • Exclude few patients • - wide CIs if response not homogeneous • - estimates more directly generalizable

  29. 'Blanket' Exclusions • Elderly/Children • Women • Ethnic minorities

  30. Proportion of patients with relevant condition excluded: • - few RCTs report • - indirect evidence • Strategies to determine extent of generalizing study results: • (Davies CE, Controlled Clin Trials 1994;15:11-14)

  31. BHAT Study: Beta-Blocker Therapy after MI (Horwitz RI et al, Am J Med 1990;89:630-38) Compared BHAT study patients with two other cohorts of patients - using Acute MI patients admitted to Yale- New Haven Hospital in 1979-82 derived - 'Expanded' Group: - 'Restricted' Group: (applied exclusion criteria)

  32. Participation of Practitioners and Patients Practitioners who participate may not be typical - teaching/specialist centre (different resources, case-mix) - special interest, experience, skill Patients who participate may not be typical - not invited (participant preference) - refuse participation (patient preference)

  33. Third Issue Review whether analysis adjustments for baseline prognostic characteristics in non-randomized studies are useful.

  34. Statistical Models Multiple regression analysis Analysis of variance Analysis of covariance Logistic regression analysis Cox proportional hazards model Discriminant analysis

  35. Example: Nifedipine in Patients with Cardiovascular Disease ( Risk of Mortality) Nifedipine is a calcium antagonist Recent controversy (associated with increased risk of mortality) 9 RCTs with dosages 30-60 mg/day (n > 8000) 1 large, high quality non-randomized study with detailed risk adjustment (n=11,575) with dosages 30-60 mg/day

  36. Example: Stroke Units versus General Medical Wards ( Risk of Mortality) Example of an organizational intervention 11 RCTs with 1 year survival information 2 non-randomized studies - Copenhagen study: no baseline differences except hypertension (concurrent control) - Edinburgh study: adjustments made (before/after)

  37. Problems with Statistical Models Under- or over-fitting Violation of proportional hazards Interactions Selection of variables Multicollinearity Influential outliers Validation

  38. Quality Assessment Instruments for Non-randomized Studies Clemens ’83 Powe ’94 Zola ’89 Ogilvie-Harris ’95 Horwitz ’90 Black & Downs ‘96 Loevinsohn ’90 Hadorn ’96 Stieb ’90 Heneghan ‘96 Ter Riet ’90 Talley ‘96 Boers ’91 Downs & Black ’98 Bass ’93 Maziak ‘98 MacMillan ’94 Smeenk ‘98

  39. Newcastle-Ottawa Quality Assessment Scale: Cohort Studies • Selection • Comparability • Outcome

  40. Selection 1. Representativeness of the intervention cohort a) truly representative of ______________________ in the community  b) somewhat representative of ___________________ in the community  c) selected group of users eg volunteers d) no description of the derivation of the cohort 2. Selection of the control cohort a) drawn from the same community as the intervention cohort  b) drawn from a different source c) no description of the derivation of the control 3. Ascertainment of intervention a) secure record eg medical records  b) structured interview  c) written self report d) no description 4. Demonstration that outcome of interest was not present at start of study a) yes  b) no

More Related