Andreas Verras SLAS Barcelona 09/29/19

1. Multiple Mechanisms, Consensus Models: Machine Learning on Phenotypic Data Andreas Verras SLAS Barcelona 09/29/19

2. Overview • Generation of consensus machine learning models from multiple collaborators • Data sharing and security • Prospective study • Model sharing

3. About Me Basel, Switzerland Rahway, NJ New York, NY

4. Richard T. Clark Fellowship • Designed to leverage skills and talents at Merck to support global humanitarian efforts in human health. • Medicines for Malaria Ventures in Geneva developed in 1999 with a mission to discover safe, effective, and affordable malaria drugs.

5. Background & Overview • Background • A consultant at MMV (Chris Waller) began the project by building machine learning models using MMV proprietary data to predict blood stage malaria activity. • Collaborators from pharma who have tropical disease research centers (Astra Zeneca, Novartis, GSK) built models on their internal data sets and shared those models. • All models were used to predict activity of a St. Jude blood stage malaria inhibition screen. • Project Goal • To generate a consensus model using all the MMV and pharma models which can be shared with the malaria screening community to aid in the identification of active compounds and reduce resource barrier to screening. Verras et al. Shared Consensus Machine Learning Models for Predicting Blood Stage Malaria Inhibition. JCIM v53 2017.

6. Model Sharing MMV Data Novartis Data AZ Data GSK Data St. Jude & Commercial Test Set MMV Models Novartis Model AZ Models GSK Model Consensus Model Prediction of Blood Stage Malaria Activity

7. Model Building • Model Building • Naïve Bayesian models were built using Pipeline Pilot (Biovia). • There was no standardization of descriptors, activity cutoffs, or data types. • Model Sharing • All model files were shared between organizations in an xml format proprietary to Biovia. • Model Validation • Raw mean scores were calculated for all combinations of models and evaluated by enrichment of active compounds in the St. Jude Commercial and Screening Sets.

8. Data Sets and Models

9. Naïve Bayesian Active Fingerprints • Fingerprints are only shared for the entirety of the training set unassociated with single models. • Low information finger prints are excluded. • Probable Active Fingerprint • Probable Noncontributing Fingerprint • Probable Inactive Fingerprint Inactive

10. Model Security • While deconvoluting the identity of the molecules is impossible one can “fish” for similarity. • To prevent this only consensus predictions are returned to the user, not individual model predictions. Similarity to Training Set Molecule Active Molecules Inactive Molecules Prediction Probability

11. Model Performance • Model validation done on the withheld data sets from St. Jude. • All possible 256 combinations of 8 models were generated by mean prediction. • Performance was superior on SJ Commercial set possibly due to incorporation of commercial compounds into participant’s screening decks. • Consensus models significantly outperformed single models. SJ Screening Set Commercial Set Single Models

12. Domain of Applicability We do not have the identity of the molecules in each training set, but we have all the fingerprints used by the model. Model Fingerprints New Molecule to Predict Molecule Fingerprints Calculate fingerprints and compare how much are represented in the model. Return fingerprint coverage per molecule. Molecules with low fingerprint coverage may be poorly predicted.. 50% coverage

13. Domain of Applicability • For the SJ Screening validation set we see a clear trend suggesting fingerprint coverage largely predicts enrichment. • For the Commercial set we find a strange relationship where for some consensus models where enrichement is not correlated with coverage. • In the Commercial validation we actually find the quartile with the lowest fingerprint coverage to be the most enriched. SJ Screening Set Commercial Set

14. Domain of Applicability • One of the models has significant higher FP coverage of the SJ Commercial validation set data sets. • This is a large model which results in larger prediction magnitudes. • The mean prediction may be skewed because of these predictions from the GSK set on the Commercial validation set.

15. Prospective Predictions • We were awarded funding from the Bill and Melinda Gates foundation to purchase and screen about 6K compounds from Emolecules. • We used the St. Jude validation sets to establish cutoff criteria for predicted actives vs. inactives of 0 and -20 respectively. PredACT PredINACT

16. Prospective Screen Consensus model Drug-like property filters/diversity eMolecules#23k predicted actives (stricter criteria) eMolecules#13k predicted actives/3k predicted inactives 4Mcompounds MW ≤ 400 XLogP ≤ 4 TPSA ≤ 140 A Rot. bonds ≤ 10 Filters: DDU/PAINS Reformat ABS assay

17. Screening Filters • We ran the best performing consensus model over 12M compounds in the Emolecules database and generated PredACT and PredINACT sets. • For both sets we put very strict substructure definitions in place to avoid selecting many known malaria active chemotypes. • Strict physical property criteria were put in place to insure lead-like hits including: • logD 0-4 • MW 150-400 • PSA 40-120 • All PAINS compounds removed • Several additional “bad” chemical features • Both sets were then clustered by ECFP4 descriptors and a diversity set was taken from each. Some of the removed substructures. There were 48 total.

18. Prospective Screen Results • Compounds tested by a collaborator at UCSD in blood stage malaria assay by monitoring DNA dye. • We saw a significant (3-fold) enrichment in the hit rate in the PredACT set over the diverse PredINACT set. • Well location, plate order, and region did not account for enrichment. • Titration studies currently under way.

19. Example Chemical Matter

20. Conclusions • We were able to generate a consensus model with over 6M data records in the training set from five different organizations including proprietary and public data. • The consensus models outperformed even the largest single models. • Participants agreed to host the models publicly in the ChEMBL malaria portal. • Prospective studies showed significant enrichment and we are optimistic about finding new chemical matter.

21. Publicly Shared Models 1. Wells, T. N. C. et al. NatureReviews Drug Discovery, 15, 661-662 (2016) 2. Van Voorhis, W. C. et al. PLOS Pathogens, 12, e1005763 (2016) 3. Williamson, A. E. et al. ACS Cent. Sci., 2, 687−701 (2016)

22. Publicly Shared New Efforts • MMV partnered with AMG consulting to reengineer shared models. • All model building software to be open source (python, RDKit) • Currently all models have been rebuilt and awaiting release by collaborators. • To be hosted by EMBL hopefully by the end of the year.

23. Acknowledgements Malaria Machine Learning Jeremy Burrows - MMV Chris Waller – Merck Peter Gedeck – Novartis AnandRaicharkur – AZ Mano Panda – AZ Darren Green – GSK Kip Guy – St. Jude Anang Shelat – St. Jude George Papadatos – ChEMBL John Overington – ChEMBL Merck Chris Waller Bob Sheridan Sung Sao So

25. Model Performance Consensus model ROC AUC for St. Jude Sample Collection Consensus model ROC AUC for St. Jude Commerical Screen Single Models

26. Domain of Applicability We do not have the identity of the molecules in each training set, but we have all the fingerprints used by the model. Model Fingerprints New Molecule to Predict Molecule Fingerprints Calculate fingerprints and compare how much are represented in the model. Return fingerprint coverage per molecule. Molecules with low fingerprint coverage may be poorly predicted.. 50% coverage

27. Domain of Applicability • For the SJ Screening validation set we see a clear trend suggesting fingerprint coverage largely predicts enrichment. • For the Commercial set we find a strange relationship where for some consensus models where enrichement is not correlated with coverage. • In the Commercial validation we actually find the quartile with the lowest fingerprint coverage to be the most enriched. SJ Screening Set Commercial Set

28. Generic selection criteria for antimalarial hits • The chemical structure of a hit should be confirmed by identification (for natural products), re-synthesis or re-purification. • Primary screening data should be validated on a selection of hit compounds (>90% pure). • A hit should have an acceptable in vitro response (typically, a sigmoidal concentration–growth inhibition curve reaching a maximal 100% efficacy, with a Hill coefficient ideally between 0.5 and 1.8). • Hits should have an EC50 <1 μM for sensitive and multiple resistant strains of Plasmodium spp. • Preliminary knowledge of the SAR of a hit is desirable. • A hit should have a tractable chemotype: it should have no highly reactive or unstable moieties in the pharmacophore and be amenable to structural variation by chemical (or biochemical) synthesis. Hits should pass basic drug-like filters, such as pan-assay interference filters (PAINS) to eliminate promiscuous hits that lack target specificity. • Conformity to the ‘rule of five’ is preferred. • There should be a greater than 10-fold selectivity between the CC50 for a mammalian cell line(for example, HepG2 or Vero cells) and the EC50 for Plasmodium spp. • A hit requires adequate selectivity in a biochemical counter-assay (for example, a homologous mammalian target) where relevant. However, most infectious disease hit-to-lead projects are not target-based screens but phenotypic. • No serious intellectual property conflicts should exist (that is, a ‘freedom to operate’ is needed). • No major synthesis or formulation issues should be anticipated (compounds should ideally be synthesized in ≤5 steps with an acceptable yield and acceptable solubility). Katsuno, K. et al.Nat. Rev. Drug Discov. 1–8 (2015).