Traceback (most recent call last):
ZeroDivisionError: integer division or modulo by 0

We’ve all been there. You’ve written your code, tested it out on some toy data and then when you make the move to the real data, there was something you didn’t expect.

Maybe some samples have been truncated to zero. Maybe the input arrays are the wrong shape. Suddenly your code comes crashing down around you, and you’re left thinking: well how could I have known that was going to happen? I can’t test everything

You can however, get pretty close. Stateful testing is a strategy made popular by the Haskell library Quickcheck.

Here is an example, shamelessly stolen from the Hypothesis page. Lets say we have an encoder and a decoder.

def encode(input_string):
    count = 1
    prev = ''
    lst = []
    for character in input_string:
        if character != prev:
            if prev:
                entry = (prev, count)
                lst.append(entry)
            count = 1
            prev = character
        else:
            count += 1
    entry = (character, count)
    lst.append(entry)
    return lst

def decode(lst):
    q = ''
    for character, count in lst:
        q += character * count
    return q

Now we probably want the decoder to invert the encoder’s function no matter what the input is. But we don’t want to go and test every possible string. What we need is a smart strategy for the space of legal data and finding anything that might break our code. This is where hypothesis comes in. Lets write a simple py.test test and add some Hypothesis decorators to it.

from hypothesis import given
from hypothesis.strategies import text

@given(text())
def test_decode_inverts_encode(s):
    assert decode(encode(s)) == s

What is happening here is that the @given decorator is fundamentally changing the test behavior. Normally py.test will run this test one, with no arguments. Instead, with the @given decorator, the test will be run many times with strings generated from the strategy text() (which as you might expect generates strings).

But this is where things get clever. Hypothesis doesn’t just randomly draw strings, instead it has an adaptive strategy of generating legal strings and then paring the data down to minimal falsifying examples. That is to say, it searches the space of strings for anything that might break the test efficiently.

Anyhow this immediately exposes a problem in our code:

Falsifying example: test_decode_inverts_encode(s='')

UnboundLocalError: local variable 'character' referenced before assignment

It just doesn’t make sense when an empty string is supplied. But, for arguments sake, lets say you knew that was never going to happen. How could we tell Hypothesis not to search for data of certain types? This is where the assume statement comes in. Lets make a little change to our code.

from hypothesis import given, assume
from hypothesis.strategies import text

@given(text())
def test_decode_inverts_encode(s):
    assume(s != '')
    assert decode(encode(s)) == s

Now Hyptothesis will not run the test with the empty sting. But out assume could be more general: any condition can be checked!

Hopefully you can see how between assume, given and the inbuilt strategies you can efficiently do property based testing to your hearts content, and find all those niggling bugs you never would have thought to look for.

That said, there is even more exciting features in the hypothesis packacge, and if you are interested in messing about with them I highly encourage you to take a look: https://hypothesis.readthedocs.io/en/latest/index.html.

Conor of OPIG