Covariate selection strategies, Negative controls, and Empirical calibration

1. Covariate selection strategies, Negative controls, and Empirical calibration Martijn Schuemie Janssen R&D OHDSI UCLA

2. Trouble with observational research

3. Residual study bias Rush et al., 2018

4. How to choose covariates to adjust for? Manual selection by experts Automated procedure using the data

5. Arguments against manual selection • Non-reproducible • Reality is not that simple • Human body is collection of feedback and feedforward loops • The healthcare system reacts to what is known

6. Arguments against automated approach To be continued (if time permits) … Accidentally including Instrumental variables Colliders “There is a strong consensus within the causal inference field that confounders should be selected by using subject matter knowledge rather than automatized procedures, to avoid, for instance, collider-stratification bias “ – Anonymous reviewer

7. Channeling adjustment Kitchen-sink: • Demographics • Conditions • Drugs • Lab values • Procedures • … Typically between 10,000 and 100,000 variables • Comparing paracetamol to ibuprofen • CPRD database • Propensity score matching • 37 ‘publication covariates’ • ‘Kitchen-sink’ + LASSO

8. Covariate balance: standardized difference of means Shown: Kitchen sink PS: Publication covariates Shown: Kitchen sink PS: Kitchen sink Shown: Publication covariates PS: Publication covariates

9. Covariate balance: standardized difference of means Shown: Kitchen sink PS: Publication covariates Shown: Kitchen sink PS: Kitchen sink Shown: Publication covariates PS: Publication covariates

10. Covariate balance: standardized difference of means Shown: Kitchen sink PS: Publication covariates Shown: Kitchen sink PS: Kitchen sink Shown: Publication covariates PS: Publication covariates • Not adjusted for in manual approach: • paracetamol users are less like to have a diagnose of pain recorded in their data • paracetamol users are more likely to be on cough suppressants and/or opioids Automated approach balances on all covariates, including manually selected ones

11. Covariate selection should be automated Include all possible covariates in propensity model Use LASSO to fit propensity model Match / stratify on propensity score Check that covariate balance is achieve on all observed variables How do we know that is enough? Negative controls

12. Examples of negative controls Infectious mononucleosis Multiple sclerosis ? Rubella ? Measles ?

13. Example of a negative control Odds ratio: Infectious mononucleosis Multiple sclerosis 2.22 * Rubella 1.31 * Measles 1.42 * * P < .05

14. Example of a negative control Odds ratio: Infectious mononucleosis Multiple sclerosis 2.22 * Rubella 1.31 * Measles 1.42 * Negative controls: A broken arm 1.10 Concussion 1.23 * Tonsillectomy 1.25 * * P < .05

15. Negative controls: exposure-outcome pair without causal relationship Outcome controls: same exposure, different outcome Exposure controls: different exposure, same outcome

16. Negative controls demonstrated to detect 3 primary sources of systematic error: • Confounding • Selection bias • Measurement bias

17. Even for randomized trails!

18. Pick negative controls with similar bias as exposure / outcome of interest, but without direct causal relationship

19. Selection of negative controls • No causal relationship • Exchangeability: same confounding • True confounding structure is unknowable • Our solution: pick a sample of negative controls (50 < n < 100) • Assumption: distribution of confounding from which exposure-outcome of interest is sampled

20. How to interpret results from negative controls: Example study Case-control study Clopidogrel  UGIB IBM MarketScan CCAE database 67 negative controls for UGIB

21. P value calculation SE (Log scale)

22. P value calculation Some drug (Log scale)

23. P value calculation clopidogrel – GI bleed: p < .001 clopidogrel (Log scale)

24. P value calculation clopidogrel

25. Estimated effect sizes for negative controls 55% of negative controls have p < .05 (Expected: 5%)

26. Negative controls & the null distribution We can use the negative controls to estimate the real null distribution

27. Negative controls & the null distribution Under this empirical null distribution, we cannot reject the null hypothesis for clopidogrel: p > 0.05

28. Different visualization Showing same negative controls, with their respective RR and SE

29. Different visualization Estimates below the dashed line have p < .05 (theoretical null)

30. Different visualization Estimates in the orange area have p < .05 (empirical null)

31. Different visualization clopidogrel

32. Internal validation Using leave-one-out cross validation

33. Other example study Canagliflozin vs non-SGLT2i (excluding metformin & insulin) Outcome: Heart failure Propensity score matching using the kitchen sink & LASSO IBM MarketScan CCAE database

34. Other example study Method appears generally robust to confounding: exchangeability assumption less important?

35. Criticism by Gruber & TchetgenTchetgen • P-values are not useful • Reducing type-I error increases type-II error • Not robust to strong violation of exchangeability assumption • Simulation: all negative controls have +2 bias (relative risk = 3), outcome of interest has no bias (relative risk = 1)

36. How to pick > 50 negative controls? Exposure or outcome of interest Remove known effects Final list Manual review Candidates rank-ordered by prevalence All outcomes / exposures observed in data Literature Product labels Spontaneous reports

37. Negative controls & p-value calibration Negative controls can provide an additional diagnostic for our study design We use large numbers of negative controls to prevent ‘picking the wrong one’ P-value calibration allows interpretation of results from negative controls But P-value often is not what we are interested in Negative controls can’t detect bias towards the null

38. Confidence interval calibration Simply assume bias is not a function of the true effect size or Include positive controls Real positive controls are problematic: Scarce Effect size not know with great precision Known to physicians

39. Positive control synthesis Prediction model Simulated outcome Subject 1 Covariate capture Exposure Exposure Covariate capture Subject 2 Real outcome Exposure Subject 3 Covariate capture Time Start with negative control Add simulated additional outcomes during exposure Outcomes are sampled from predicted probability based on outcome model to preserve measured confounding

40. Confidence interval calibration

41. Confidence interval calibration

42. Summary • Observational research has a bad reputation • Just saying your design is better is not good enough • (Negative) controls provide an additional diagnostic • Confounding structure is unknowable, so cannot pick ‘perfect’ negative control • Instead, we pick a sample of controls • Empirical calibration of p-values and confidence intervals allows interpretation of control estimates • Assumptions: • Negative controls are really negative • (weak) exchangeability (less so if no bias observed?)

43. Instrumental variables • Do they really exist? • Would show as strong predictor in propensity model • If correcting for IV • Still leads to balance on all kitchen sink covariates • Does not lead to bias as observed using negative controls Should I then worry about it?

44. Collider bias Depression (unobserved) Smoking (unobserved) Cardiovascular disease ? Antidepressant use Lung cancer Example study: we want to know if antidepressants cause lung cancer. We can’t observe depression status or smoking status Conditioning on CV disease introduces a correlation between antidepressant use and lung cancer, because people with CV disease are either depressed or are smoking. Those who use antidepressants are thus less likely to be smokers, and will have less lung cancer.

45. Collider bias Depression (unobserved) Smoking (unobserved) Cardiovascular disease COPD T2DM ? Antidepressant use Lung cancer But we have many other covariates besides CV disease. We balance on all of those, giving us confidence we balance on smoking status.

46. Collider bias Depression (unobserved) Smoking (unobserved) Cardiovascular disease ? Antidepressant use Lung cancer Gallstone Ingrown nail We also have negative controls, some of which are also caused by smoking. We don’t observe any bias for those.

47. Current practice Research question Write protocol Protocol: Vague description of what will be done Execute study Results & diagnostics Change execution with vague description limits Register protocol Amend protocol

48. Proposed practice Research question Register protocol & study package Implement study Write protocol Study package: Full (machine-readable) implementation Protocol: Full (human-readable) description Full diagnostics Execute study Execute diagnostics Results