1 / 23

Humanity v The Machines

Can human experts predict solubility better than computers? This study compares the accuracy of human predictions and machine learning algorithms in determining the solubility of drug-like molecules, highlighting the value of computational models in pharmaceutical research.

geter
Download Presentation

Humanity v The Machines

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Humanity v The Machines An AI Challenge Dr John Mitchell, St Andrews Sam Boobier, Anne Osborn & John Mitchell, Can human experts predict solubility better than computers? J Cheminformatics, 9:63 (2017) Image: scmp.com

  2. The Challenge: Computing Aqueous Solubility

  3. Which would you Prefer ... or ? Solubility in water (and other biological fluids) is highly desirable for pharmaceuticals!

  4. Solubility is an important issue in drug discovery and a major cause of failure of drug development projects Expensive for the pharma industry Patients suffer lack of available treatments A good computational model for predicting the solubility of druglike molecules would be very valuable.

  5. Solubility You might think that “How much solid compound dissolves in 1 litre of water” is a simple question to answer. However, experiments are prone to large errors. Solution takes time to reach equilibrium, and results depend on pH, temperature, ionic strength, solid form, impurities etc.

  6. Humankind vs The Machines Challenge is to predict solubilities of 25 molecules given 75 as training data.

  7. Video explanation of wisdom of crowds

  8. Wisdom of Crowds Francis Galton (1907) described a competition at a country fair, where participants were asked to estimate the mass of a cow. Individual entries were not particularly reliable, but Galton realised that by combining these guesses a much more reliable estimate could be obtained.

  9. Wisdom of Crowds Guess the mass of the cow: Median of individual guesses is a good estimator: Francis Galton, Voxpopuli, Nature, 75, 450-451 (1907).

  10. Wisdom of Crowds This is an ensemble predictor which works by aggregating individual independent estimates, and generates a result that is more reliable than the individual guesses and more accurate than the large majority of them.

  11. Humankind vs The Machines Sent 229 emailed invitations to subject experts and students. Obtained 22 anonymous responses, of those 17 made full sets of predictions.

  12. Humankind vs The Machines 10 machine learning algorithms were given the same training & test sets as the human panel.

  13. ML Methods for Solubility • Expt data: errors unknown (0.5-0.7 logS0 units?) but limit possible accuracy of models; • Differences in dataset size and composition often hinder comparisons of methods; • ML numerically better than first principles (but FP not widely validated), at the cost of less insight.

  14. 0.99 0.94 Difference not significant Sam Boobier, Anne Osborn & John Mitchell, Can human experts predict solubility better than computers? J Cheminformatics, 9:63 (2017)

  15. Machine Learning Algorithms Ranked 2nd 1st

  16. Another Layer of Wisdom of Crowds • We don’t know in advance which predictors will be good and which will be poor. • However, we can make an algorithm that will allow us to generate a good (consensus) prediction without prior knowledge of results.

  17. Wisdom of Crowds: Human Consensus Predictor Guess for the solubility of the molecule: Median of all (between 17 & 21) individual human guesses of logS0 for a given compound.

  18. Wisdom of Crowds: Machine Consensus Predictor Guess for the solubility of the molecule: Median of all 10 individual machine guesses of logS for a given compound.

  19. 1.09 Sam Boobier, Anne Osborn & John Mitchell, Can human experts predict solubility better than computers? J Cheminformatics, 9:63 (2017)

  20. 1.14 1.09 Difference not significant Sam Boobier, Anne Osborn & John Mitchell, Can human experts predict solubility better than computers? J Cheminformatics, 9:63 (2017)

  21. Conclusions: Humans v ML • Best humans and best algorithms perform almost equally; • Consensus of humans and consensus of algorithms perform almost equally; • Less effective individual human predictors are notably weaker. • Both humans and ML numerically clearly better than a physics-based first principles theory approach.* * On a similar but non-identical dataset; David Palmer, James McDonagh, John Mitchell, Tanja van Mourik & Maxim Fedorov, First-Principles Calculation of the Intrinsic Aqueous Solubility of Crystalline Druglike Molecules, J Chem Theory Comput, 8, 3322-3337 (2012)

  22. Thanks • Tanja van Mourik (St Andrews), Neetika Nath, James McDonagh (now IBM), Rachael Skyner (now Diamond, Oxford), Sam Boobier (now Leeds), Will Kew (now Edinburgh) • Maxim Fedorov, Dave Palmer (Strathclyde) • Laura Hughes (now Stanford), Toni Llinas (AZ), Anne Osbourn (JIC, Norwich)

More Related