Cool ideas in Deep Learning and where to find more about them

I was planning on doing a blog post about some cool random deep learning paper that I have read in the last year or so. However, I keep finding that someone else has already written a way better blog post than what I could write. Instead I have decided to write a very brief summary of some hot ideas and then provide a link to some other page where someone describes it way better than me.

The Lottery Ticket Hypothesis

This idea has to do with pruning a model, which is when you remove a parts of your model to make it more computationally efficient while barely loosing accuracy. The lottery ticket hypothesis also has to do with how weight are initialized in neural networks and why larger models often achieve better performance.

Anyways, the hypothesis says the following: “Dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that—when trained in isolation—reach test accuracy comparable to the original network in a similar number of iterations.” In their analogy, the random initialization of a models weights is treated like a lottery, where some combination of a subset of these weight is already pretty close to the network you want to train (winning ticket). For a better description and a summary of advances in this field I would recommend this blog post.

SAM: Sharpness aware minimization

The key idea here has to do with finding the best optimizer to train a model capable of generalization. According to this paper, a model that has converged to a sharp minima will be less likely to generalize than one that has converged to a flatter minima. They show the following plot to provide an intuition of why this may be the case.

In the SAM paper (and ASAM for adaptive) the authors implement an optimizer that is more likely to converge to a flat minima. I found this blog post by the authors of ASAM gives a very good description of the field.

Continue reading →

Monoclonal antibody PRNP100 therapy for Creutzfeldt–Jakob disease

Recently, University College London Hospitals (UCLH) received a “Specials License” to allow the treatment of six patients suffering from Creutzfeldt–Jakob Disease (CJD), by way of a novel antibody known as PRN100. The results of this treatment have now been published in The Lancet.

There is currently no cure for CJD, yet over 100 people per year develop it either spontaneously or through external means including (but not limited to) growth hormones, cataract surgery or infected neurosurgical implements [1]. “There is no UK legislation which implements a compassionate use programme as set out in Article 83 of the relevant EU regulation. But the UK has implemented an exemption process known as the “Specials” in light of the requirement to be able to deal with special needs.” [2]

As there is no known cure, the request for use of PRN100 was put before the court as in Law “Some treatment decisions are so serious that the court has to make them.”

Continue reading →

Le Tour de Farce v9.0

With many tours (Farcical and otherwise) restricted due to Covid, 2022 celebrated the resurrection of OPIG’s glorious Tour de Farce. This year’s route was nine miles and an unusually conservative four pubs.

After listening to Lewis’ conference prep talk, we left the Statistics Department around 5pm for a leisurely trundle through Mesopotamia, The Oxford Psychopath, Old Marston and out to our first rest stop, The Victoria.

Continue reading →

Exploring topological fingerprints in RDKit

Finding a way to express the similarity of irregular and discrete molecular graphs to enable quantitative algorithmic reasoning in chemical space is a fundamental problem in data-driven small molecule drug discovery.

Virtually all algorithms that are widely and successfully used in this setting boil down to extracting and comparing (multi-)sets of subgraphs, differing only in the space of substructures they consider and the extent to which they are able to adapt to specific downstream applications.

A large body of recent work has explored approaches centred around graph neural networks (GNNs), which can often maximise both of these considerations. However, the subgraph-derived embeddings learned by these algorithms may not always perform well beyond the specific datasets they are trained on and for many generic or resource-constrained applications more traditional “non-parametric” topological fingerprints may still be a viable and often preferable choice .

This blog post gives an overview of the topological fingerprint algorithms implemented in RDKit. In general, they count the occurrences of a certain family of subgraphs in a given molecule and then represent this set/multiset as a bit/count vector, which can be compared to other fingerprints with the Jaccard/Dice similarity metric or further processed by other algorithms.

Continue reading →

Tackling horizontal and vertical limitations

A blog post about reviewing papers and preparing papers for publication.

We start with the following premise: all papers have limitations. There is not a single paper without limitations. A method may not be generally applicable, a result may not be completely justified by the data or a theory may make restrictive assumptions. To cover all limitations would make a paper infinitely long, so we must stop somewhere.

A lot of limitations fall into the following scenario. The results or methods are presented but they could have extended them in some way. Suppose, we obtain results on a particular cell type using an immortalized cell-line. Are the results still true, if we performed the experiments on primary or patient-derived cells? If the signal from the original cells was sufficiently robust then we would hope so. However, we can not be one hundred percent sure. A similar example is a method that can be applied to a certain type of data. It may be possible to extend the method to be applied to other data types. However, this may require some new methodology. I call this flavor of limitations vertical limitations. They are vertical in the sense that they build upon an already developed result in the manuscript. For certain journals, they will require that you tackle vertical limitations by adapting the original idea or method to demonstrate broad appeal or that idea could permeate multiple fields. Most of the time, however, the premise of an approach is not to keep extending it. It works. Leave it alone. Do not ask for more. An idea done well does not need more.

Continue reading →

Oxford MRC DTP Symposium 2022

The Oxford Medical Research Council Doctoral Training Partnership (MRC DTP), the program through which my DPhil is funded, hosts an annual Symposium to highlight research being conducted by DTP students and offer insights into the career paths of external speakers.

This year, I was on the committee organising the Symposium and was involved in selecting student presenters, as well as deciding on and inviting external speakers. It was a great experience!

Panel on careers in biotech featuring Loïc Roux, Ochre Bio (centre); Helena Meyer-Berg, Sirion Biotech (centre right); and Claire Shingler, Oxford BioEscalator (right).

Here are my key takeaways from the Symposium:

Continue reading →

Entering a Stable Relationship with your Neural Network

Over the past year, I have been working on building a graph-based paratope (antibody binding site) prediction tool – Paragraph. Fortunately, I have had moderate success with this and you can now check out the preprint of this work here.

However, for a long time, I struggled with a highly unstable network, where different random seeds yielded very different results. I believe this instability was largely due to the high class imbalance in my data – only ~10% of all residues in the Fv (variable region of the antibody) belong to the paratope.

I tried many different things in an attempt to stabilise my training, most of which failed. I will share all of these ideas with you though – successful or not – as what works for one person/network is never guaranteed to work for another. I hope that the below may provide some ideas to try out for others facing similar issues. Where possible, I also provide some example hyperparameter values that could act as sensible starting points.

Continue reading →

AIRR Community Meeting VI May 17-19 

Eve, Brennan and I were delighted to attend the sixth AIRR (adaptive immune receptor repertoire) Community Meeting: Exploring New Frontiers in San Diego. Eve and I had been awaiting this meeting for a mere 3 years, since it was announced during the last in-person AIRR Community Meeting back in 2019. Fortunately, San Diego did not disappoint. 

After a rocky start (featuring many hours stuck in traffic on the M40, one missed flight and one delayed flight), we made it to California! The three day conference had ~230 participants (remote and in-person) and featured great talks from academia and industry. We particularly enjoyed keynote talks from Dennis Burton on rational vaccine design using broadly neutralising antibodies, Gunilla Karlsson Hedestam on functional consequences of allelic variation, Shane Crotty on covid and HIV vaccine design, and Atul Butte on uses of electronic health record data and how we should all found start-ups.

We had fun delivering a tutorial on OPIG antibody tools and, most importantly, we all won AIRR t-shirts in the raffle (potentially we were the only people who noticed how to enter on the conference app). Highlights outside of the conference included paddle boarding and seeing hummingbirds, pelicans, sealions, seals, ‘Garibaldi’ the state fish, and meeting Bob the golden retriever at a surfing shop. We’re now off to find jobs on the West Coast so we can live at the beach….

 The AIRR community has many webinars and talks available on their youtube channel https://www.youtube.com/c/AIRRCommunity

Sarah, Eve & Brennan

Visualise with Weight and Biases

Understanding what’s going on when you’ve started training your shiny new ML model is hard enough. Will it work? Have I got the right parameters? Is it the data? Probably.  Any tool that can help with that process is a Godsend. Weights and biases is a great tool to help you visualise and track your model throughout your production cycle. In this blog post, I’m going to detail some basics on how you can initialise and use it to visualise your next project.

Installation

To use weights and biases (wandb), you need to make an account. For individuals it is free, however, for team-oriented features, you will have to pay. Wandb can then be installed using pip or conda.

$ 	conda install -c conda-forge wandb

or 

$   pip install wandb

To initialise your project, import the package, sign in, and then use the following command using your chosen project name and username (if you want):

import wandb

wandb.login()

wandb.init(project='project1')

In addition to your project, you can also initialise a config dictionary with starting parameter values:

Continue reading →

Sharing Data Responsibly: The FAIR Principles

So you’ve submitted your paper, made your code publicly available, and maybe even provided documentation to ensure somebody can reproduce your work. But what about the data your work is based on? Is that readily available to your readers, too?

Maybe it’s too large to put on GitHub alongside your code. Maybe it’s sensitive, or subject to GDPR restrictions, so you can’t just stick a download link on your website. Maybe it’s in a proprietary format that needs non-open software to read. There are many reasons sharing data can be less straightforward than sharing code, and often it’s not entirely clear what ‘best practices’ are for a given situation. Data management is a complicated topic, and to do it justice would require far more than a quick blog post. Instead, I’d like to focus on a single source of guidance that serves as a useful starting point for thinking about responsible data management: the FAIR principles.

Continue reading →