Suppose we need to do some interactive analysis in a Jupyter notebook, but our local machine lacks the power. We have access to a slurm cluster, but we can’t SSH from the head node to the worker node; we can only SSH from the worker node to the head node. Can we still interact with a Jupyter notebook running on the worker node? As it happens, the answer is “yes” – we just need to do some reverse SSH tunnelling.
Continue readingAuthor Archives: Fergus Boyles
Easy Python job queues with RQ
Job queueing is an important consideration for a web application, especially one that needs to play nice and share resources with other web applications. There are lots of options out there with varying levels of complexity and power, but for a simple pure Python job queue that just works, RQ is quick and easy to get up and running.
RQ is a Python job queueing package designed to work out of the box, using a Redis database as a message broker (the bit that allows the app and workers to exchange information about jobs). To use it, you just need a redis-server installation and the rq module in your python environment.
Continue readingMounting a remote file system with SSHFS
If you’re working with data stored on a remote server, you might not want to (or even have the space to) copy data to your local file system when you work on it. Instead, we can use SSHFS to mount a remote file system via SSH, allowing us to read and write data on the remote file system without manually copying files.
Continue readingDeploying a Flask app part II: using an Apache reverse proxy
I recently wrote about serving a Flask web application on localhost using gunicorn. This is sufficient to get an app up and running locally using a production-ready WSGI server, but we still need to add a HTTP proxy server in front to securely handle HTTP requests coming from external clients. Here we’ll cover configuring a simple reverse proxy using the Apache web server, though of course you could do the same with another HTTP server such as nginx.
Continue readingDeploying a Flask app part I: the gunicorn WSGI server
Last year I wrote a post about deploying Flask apps with Apache/mod_wsgi when your app’s dependencies are installed in a conda environment. The year before, in the dark times, I wrote a post about the black magic invocations required to get multiple apps running stably using mod_wsgi. I’ve since moved away from mod_wsgi entirely and switched to running Flask apps from containers using the gunicorn WSGI server behind an Apache reverse proxy, which has made life immeasurably easier. In this post we’ll cover running a Flask app on localhost using gunicorn; in Part II we’ll run our app as a service using Singularity and deploy it to production using Apache as a HTTP proxy server.
Continue readingSAbBox in 2023: ImmuneBuilder and more!
For several years now, we have distributed the SAbDab database and SAbPred tools as a virtual machine, SAbBox, via Oxford University Innovation. This virtual machine allows a user to utilise the tools and database locally, allowing for high-throughput analysis and keeping confidential data within a local network. Initially distributed under a commercial licence, the platform proved popular and, in 2020, we introduced a free academic licence to enable our academic colleagues to use our tools and database locally.
Following requests from users, in 2021 we released a new version of the platform packaged as a Singularity container. This included all of the features of SAbBox, allowing Linux users to take advantage of the near bare-metal performance of Singularity when running SAbPred tools. Over the past year, we have made lots of improvements to both SAbBox platforms, and have more work planned for the coming year. I’ll briefly outline these developments below.
Continue readingUsing Conda environments with Flask and Apache
With the advent of ABlooper, we’ve recently introduced OpenMM as a new dependency for the SAbDab-SAbPred antibody modelling platform. By far the easiest way to install the OpenMM Python API is via Conda, so we’ve moved to Conda environments for the entire platform. This has made installation of the platform much easier, but introduces complications when it comes to running its web applications under Apache. In this post, I’ll briefly explain the reason for this, and provide a basic guide for running Flask apps using Conda environments under Apache.
Continue readingSharing Data Responsibly: The FAIR Principles
So you’ve submitted your paper, made your code publicly available, and maybe even provided documentation to ensure somebody can reproduce your work. But what about the data your work is based on? Is that readily available to your readers, too?
Maybe it’s too large to put on GitHub alongside your code. Maybe it’s sensitive, or subject to GDPR restrictions, so you can’t just stick a download link on your website. Maybe it’s in a proprietary format that needs non-open software to read. There are many reasons sharing data can be less straightforward than sharing code, and often it’s not entirely clear what ‘best practices’ are for a given situation. Data management is a complicated topic, and to do it justice would require far more than a quick blog post. Instead, I’d like to focus on a single source of guidance that serves as a useful starting point for thinking about responsible data management: the FAIR principles.
Continue readingFeeding a drove of hungry OPIGlets
In preparing for battle I have always found that plans are useless, but planning is indispensable.
Dwight D. Eisenhower
Following the previous post about OPIG retreat 2022, and having received numerous requests for recipes, I thought I’d document the process of ensuring that 24 people are kept fed and happy. Recipes at the foot of the post.
Disclaimer – these recipes are entirely my own interpretations, adapted where necessary to suit a range of dietary requirements. They are in no way authentic to any national cuisines and are not intended to be.
Disclaimer II: The Disclaiming – all measurements are approximate. I rarely write down recipes or use precise measurements. Taste as you go, and don’t be afraid to add more salt.
Continue readingUsing Singularity on Windows with WSL2
Previously on this blog, my colleagues Carlos and Eoin have extolled the many virtues of Singularity, which I will not repeat here. Instead, I’d like to talk about a rather interesting subject that was unexpectedly thrust upon me when my faithful Linux laptop started to show the early warning signs of critical existence failure: is there a good way to run a Singularity container on a pure Windows machine? It turns out that, with version 2 of the Windows Subsystem for Linux (WSL), there is.
Continue reading