The reception of ML approaches for the drug discovery pipeline, especially when focused on the hit to lead optimization process, has been rather skeptical by the medchem community. One of the main drivers for that is the way many ML publications benchmark their models: Historic datasets are split into two parts, with the larger part used to train and the smaller to test ML models. In order to standardize that validation process, computational chemists have constructed widely used benchmark datasets such as the DUD-E set, which is commonly used as a standard for protein-ligand binding classification tasks. Common criticism from medicinal chemists centers on the main problem associated with benchmark datasets: the absence of direct lab validation.

Just last week, Stokes et al. published this paper in Cell that seemed to impress even the more cautious (when it comes to AI) medicinal chemists like Derek Lowe, who in the past has criticized drug discovery AI publications for their “hyperventilating” style. So what’s different about this paper?

The picture below shows the summary of their methodology with the goal of finding new antibiotics. Thei ML approach seems fairly standard: train a classifier ML algorithm and test it on a set of interesting compounds to try and find active compounds. However, the special part about it is where the training data comes from as well as the validation.

The most impressive part about this paper is the incredible effort that went into carrying out wet-lab experiments to validate and test their machine learning model. This paper has more wet-lab assay results than some purely wet-lab based publications I’ve seen. What stands out is that instead of training the machine learning model on an aggregate of publicly available data, the group just made their own dataset by testing over 2000 compounds in the lab. This ensures consistent quality of the data and eliminates potential problems with inconsistent experimental methods between different labs. They went on to characterize hit compounds extensively in the lab and even went into mouse models.

This is a powerful example of what AI in drug discovery is actually capable of at the moment and shows how important the lab based validation is to make your models credible. Check out the paper, its worth a read. More of that please.