300 likes | 418 Views
What is evidence? Quality registers: use and misuse Jonas Ranstam PhD jonas.ranstam@med.lu.se Dept of Clinical Sciences, Orthopedics, Lund University, Sweden. The truth, the whole truth, and nothing but the truth? Competent statistical analysis is crucial
E N D
What is evidence? Quality registers: use and misuse Jonas Ranstam PhD jonas.ranstam@med.lu.se Dept of Clinical Sciences, Orthopedics, Lund University, Sweden
The truth, the whole truth, and nothing but the truth? Competent statistical analysis is crucial Registry data need to be interpreted with respect to several statistical phenomena, not least - sampling uncertainty (hypothesis tests, interval estimation) - confounding bias (case-mix)
Registries and randomised controlled trials Efficacy: The capacity of a treatment effect - Efficacy can in practice only be studied in an experiment (a RCT) - The efficacy of many medical devices is obvious
Registries and randomised controlled trials Effectiveness: How well a treatment works in practice - Data quality is important - Sample size is not crucial - Follow up is often not crucial - RCTs is one alternative (with the right trial design) - An observational study may be another alternative
Registries and randomised controlled trials Safety: How safe a treatment1 is - Risks are low (the larger sample size the better) - Events occur late (the longer follow up the better) - RCTs are insufficient (with regard to sample size and follow up) - A continuously ongoing register is the only alternative 1. Joint replacement
A fundamental methodological issue Randomisation “converts” systematic effects to random errors Their effect on the outcome is included in p-values and confidence intervals Randomisation works for all factors, known, unknown, measured and unmeasured
A fundamental methodological issue Observational studies rely on confounding adjustment This, can only be performed for known and measured factors Knowledge about the studied mechanisms are necessary, data driven methods such as stepwise regression are not useful for confounding adjustment
Randomization and confounding A classical confounding factor: Adjustment reduces bias Covariate Exposure Effect
Alzheimer's Disease Confounding bias Females r ≈ 0, p = 0.9 Crude [biased] correlation r ≈ 0.5, p < 0.001 Males r ≈ 0, p = 0.9 Adjusted [true] correlation r ≈ 0, p = 0.9 (Aluminium)
Alzheimer's Disease Confounding bias Females r ≈ 0.6, p < 0.001 Crude [biased] correlation r ≈ 0, p = 0.9 Males r ≈ 0.6, p < 0.001 Adjusted [true] correlation r ≈ 0.6, p < 0.001 (A specific toxin)
Randomization and confounding A covariate on the pathway between exposure and effect: Adjustment creates over-adjustment bias Covariate Exposure Effect
ᄃ Over-adjustment bias Heart Disease Yellow fingers r ≈ 0.3, p < 0.001 Crude [true] correlation r ≈ 0.7, p < 0.001 Adjusted [biased] correlation r ≈ 0.3, p < 0.001 No yellow fingers r ≈ 0.3, p < 0.001 (Smoking)
Randomization and confounding A collider: Adjustment creates collider stratification bias Covariate Exposure Effect
Collider stratification bias Breast cancer Screening non-participants r ≈ 0.5, p < 0.001 Crude [true] correlation r ≈ 0, p = 0.9 Screening participants r ≈ 0.5, p < 0.001 Adjusted [biased] correlation r ≈ 0.5, p < 0.001 (Education)
Randomization and confounding A directed acyclic graph (DAG) for confounding adjustment of bias when estimating the causal effect of warming up on the risk of injury.
Propensity scores (PS) and causal inference “[T]he use of PS methods … may actually increase, not decrease, bias. Shrier … provides a simple example; the crude estimate is bias-free, while PS methods introduce new bias.” Pearl J. Remarks on the method of propensity score. Statistics in Medicine 2009;28:1415–1424. See also: Rubin D. The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Statistics in Medicine 2007; 26:20–36. Shrier I. Letter to the editor. Statistics in Medicine 2008; 27:2740–2741. Rubin D. Author’s reply (to Ian Shrier’s Letter to the Editor). Statistics in Medicine 2008; 27:2741–2742.
Any claim coming from an observational study is most likely to be wrong 12 randomised trials have tested 52 observational claims (about the effects of vitamin B6, B12, C, D, E, beta carotene, hormone replace- ment therapy, folic acid and selenium). “They all confirmed no claims in the direction of the observational claim. We repeat that figure: 0 out of 52. To put it in another way, 100% of the observational claims failed to replicate. In fact, five claims (9.6%) are statistically significant in the opposite direction to the observational claim.” Stanley Young and Allan Karr, Significance, September 2011
Registries and misuse of data An important principle in scientific research The weaknesses, limitations, and uncertainty of presented findings must be clearly disclosed. Ignoring this because of ignorance, incompetence, or a wish to mislead represents data misuse.
Threats to proper use of registry data Possible reasons for misusing data - Incompetence - Political or administrative purposes - Commercial conflicts of interest - Academic career in a publish-or-perish environment
Ranked revision risks Swedish Knee Arthroplasty Register Annual Report 2013
Avoiding misuse of registry data Challenges Resist external pressure to use registry data in irresponsible or misleading fashions Find ways to avoid distorted analyses and conclusions of open registry data
Summary - Registries are the best alternative for safety surveillance - Registries are a complement to randomised controlled trials - Registry data require competent statistical analysis - Registry data can easily be misused - Independence and integrity important for the future
Thank you for your attention! The author declares that the research for and communication of this independent body of work does not constitute any financial or other conflict of interest.