Testing with Python (part 1): the basics

Tautology, the masterclass

Mar 24, 2024

Summary

This intro is about dumb test writing, as it's the necessary foundation to learn what comes next.

We'll therefore start with unittest, the default, verbose, but always available test toolkit in Python stdlib.

In summary, it looks like this:

import unittest

# The code to test
def add(a, b):
    return a + b

# The tests
class TestAddFunction(unittest.TestCase):

    def setUp(self):
        ... # This is run before each test

    def tearDown(self):
        ... # This is run after each test

    def test_add_integers(self):
        result = add(1, 2)
        self.assertEqual(result, 3)

    def test_add_floats(self):
        result = add(0.1, 0.2)
        self.assertAlmostEqual(result, 0.3)

    def test_add_mixed_types(self):
        with self.assertRaises(TypeError):
            add(1, "2")


if __name__ == "__main__":
    unittest.main()

And when you run it, you get a report of what worked or not:

python the_tests.py
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

Boredom alert

Testing is a tough topic to teach, because you quickly need tooling for it not to be painful, but you can't start with tooling. Also, writing the test code without understanding why we write it this way, how and what to test, is frustrating. Yet you can't explain the deeper concept without having a common language to express it.

So you'll have to bear with me a little on this one.

We will start with dumb test writing. You will learn libraries, syntax, and API first. It will serve as a base to build a better understanding of the whole concept of testing later on. It will be a bit boring at first, so hold on to it.

Unittest

You know how we love giving confusing names to stuff, like the whole venv thing that makes people go O_o ?

Well, it's the same with tests.

You may have heard of an unit test as a concept, but there is also the standard library unittest. Which incidentally can be used to create any kind of test, not just unit tests.

We will explain the different kinds of tests in a few articles from now, so you can ignore the whole nuance: today will test things, what kind of tests and if those tests are good is of no importance for your current understanding.

In the context of this article, unittest is only the standard library module dedicated to testing, and that's what we are going to learn to use.

I'll make a confession: I never use unittest to write tests professionally, as there are much better alternatives. But it's always there, and since one of the appeals of Python is that it comes batteries included, it's wise to learn it first. That's why we start with it.

Your first test

In the context of tests, imports cause a bit of a trouble to beginners. So for this hello world, make sure you are exactly in the same directory, with the same files, at the same place.

This is what our starting point looks like, a directory named “basic_project”, and 2 python files:

basic_project
├── the_code_to_test.py
└── the_tests.py

We are going to work in basic_project, it contains all our code. You'll notice there are no __init__.py file anywhere. That's on purpose.

The two files are empty right now, but I think you can guess what will go inside.

Now, let's put the code of our future one billion dollars unicorn in "the_code_to_test.py":

def add(a, b):
    return a + b

Addition as a service, that's Y combinator material, right there.

And here is what we are going to put in "the_tests.py" (we’ll explain what it does after):

import unittest

from the_code_to_test import add

class TestAddFunction(unittest.TestCase):
    def test_add_integers(self):
        result = add(1, 2)
        self.assertEqual(result, 3)


if __name__ == "__main__":
    unittest.main()

To run this test, open your terminal (if you are on windows and not familiar with it, we have an article on that), make sure you are in basic_project directory, then run the "the_tests.py".

For me it looks like this:

python the_tests.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

What did we just do?

We wrote a test named "test_add_integers" that runs add(1, 2), store the result, and check if the result is indeed equal to 3.

Doing this with unittest requires several steps:

# Make sure we have imported the unittest module
import unittest

# Import the code we need to test
from the_code_to_test import add

# Create a class that inherits from TestCase. This is necessary for
# unittest to detect it as a test, and to provide methods like
# self.assertEqual
class TestAddFunction(unittest.TestCase):

    # Write the test. It must be a method of a TestCase type
    # and the name MUST start with "test" to be detected
    def test_add_integers(self):

        # Write an assertion. Assertions are lines of code
        # that state what we expect to happen if everything
        # goes as planned.
        result = add(1, 2)
        self.assertEqual(result, 3)

# If this file is run (but not if it's imported),
# collect the tests and execute them.
if __name__ == "__main__":
    unittest.main()

The important part of the test is the assertion:

result = add(1, 2)
self.assertEqual(result, 3)

That's what we test: that the function can add integers and that it produces what we think it should.

At this stage, something probably hit you: wait, that's it?

Like, tests are just calling the code?

Yes.

Tests are mundane, boring, very often even primitive.

In fact, the less interesting your tests are, the better it is.

And now you understand why most people don't test: it's as fun as doing the dishes.

It's also as necessary.

So what does self.assertEqual(result, 3) do? Well, it checks that the result is what you think it is. If it is, the test passes, if not, the failure is recorded by unittest, and at the end of the run, it will show you that it failed.

Let's introduce a bug in our function:

def add(a, b):
    return a - b # woops

Imagine we copy/pasted it as usual from ChatGPT, and with our growing confidence formed by habit, we didn't verify it’s correct.

Let's now see the report:

python the_tests.py
F
======================================================================
FAIL: test_add_integers (__main__.TestAddFunction)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "the_tests.py", line 9, in test_add_integers
    self.assertEqual(result, 3)
AssertionError: -1 != 3

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

The system reports that one of the tests didn't pass. We'll see later on how to interpret those results.

Making several tests

Since it's unlikely you will have only one test, let's add more.

First, we revert our function to sane code:

def add(a, b):
    return a + b

Then let's add one more assertion to check for negative numbers:

import unittest

from the_code_to_test import add

class TestAddFunction(unittest.TestCase):
    def test_add_integers(self):
        result = add(1, 2)
        self.assertEqual(result, 3)

        result = add(1, -2)
        self.assertEqual(result, -1)


if __name__ == "__main__":
    unittest.main()

Our test now covers more use cases.

If we run it, it still reports a single test.

python the_tests.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

One test can indeed contain several assertions, but it will succeed or fail as a whole in the report.

Now let's add a new method to test adding strings, since the + operator works with strings too:

import unittest

from the_code_to_test import add

class TestAddFunction(unittest.TestCase):

    def test_add_integers(self):
        result = add(1, 2)
        self.assertEqual(result, 3)

        result = add(1, -2)
        self.assertEqual(result, -1)

    def test_add_strings(self):
        result = add("1", "2")
        self.assertEqual(result, "12")



if __name__ == "__main__":
    unittest.main()

Running it, we now can see two tests in the report:

python the_tests.py
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

It says "2 tests", and it also shows two dots near the top, one for each passing test.

Let's make it fail by adding a 3rd test so you can see what happens to the report. We will attempt to add floats, which cannot be safely compared with equality:

import unittest

from the_code_to_test import add


class TestAddFunction(unittest.TestCase):

    def test_add_integers(self):
        result = add(1, 2)
        self.assertEqual(result, 3)

        result = add(1, -2)
        self.assertEqual(result, -1)

    def test_add_strings(self):
        result = add("1", "2")
        self.assertEqual(result, "12")

    def test_add_floats(self):
        result = add(0.1, 0.2)
        self.assertEqual(result, 0.3)

if __name__ == "__main__":
    unittest.main()

Running the tests now gives us:

python the_tests.py
F..
======================================================================
FAIL: test_add_floats (__main__.TestAddFunction)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "the_tests.py", line 21, in test_add_floats
    self.assertEqual(result, 0.3)
AssertionError: 0.30000000000000004 != 0.3

----------------------------------------------------------------------
Ran 3 tests in 0.000s

FAILED (failures=1)

We now have 3 tests detected and executed. One of them failed, so instead of 3 dots at the top, we get an "F", and 2 dots.

This failure, however, is not due to a bug in our code, but because we wrote our test incorrectly. This is one of the annoying things with tests, you have more opportunities to make mistakes, and it will be frustrating until you get more experienced with it.

Keep at it, this is a "practice make perfect" kind of situation. You'll need experience to be productive with writing code. And use ChatGPT, it's great at it.

Also, not gonna lie, I still fight with tests regularly, just much less than before.

Reading the report

The first reflex you need to have with test failing, is to read the report, so let's explain what it contains.

I'll add one more failing test:

import unittest

from the_code_to_test import add


class TestAddFunction(unittest.TestCase):

    def test_add_integers(self):
        result = add(1, 2)
        self.assertEqual(result, 3)

        result = add(1, -2)
        self.assertEqual(result, -1)

    def test_add_strings(self):
        result = add("1", "2")
        self.assertEqual(result, "12")

    def test_add_floats(self):
        result = add(0.1, 0.2)
        self.assertEqual(result, 0.3)

    def test_add_mixed_types(self):
        add(1, "2")


if __name__ == "__main__":
    unittest.main()

Now the report will show us 4 tests, 2 are failing:

python the_tests.py
F.E.
======================================================================
ERROR: test_add_mixed_types (__main__.TestAddFunction)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "the_tests.py", line 24, in test_add_mixed_types
    add(1, "2")
  File "the_code_to_test.py", line 2, in add
    return a + b
TypeError: unsupported operand type(s) for +: 'int' and 'str'

======================================================================
FAIL: test_add_floats (__main__.TestAddFunction)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "the_tests.py", line 21, in test_add_floats
    self.assertEqual(result, 0.3)
AssertionError: 0.30000000000000004 != 0.3

----------------------------------------------------------------------
Ran 4 tests in 0.001s

FAILED (failures=1, errors=1)

Let's label all that:

In red, you can see the errors. The information is repeated 3 times. Once in the summary at the top, with the letter "E", once with the label "ERROR" for each error (we have only one here) during the run, and one at the end in the final statistics.

Errors are tests that didn't succeed because the code crashed.

In orange, you can see the failures. The information is also repeated 3 times. Once in the summary at the top, with the letter "F", once with the label "FAIL" for each occurrence (we have only one here) during the run, and one at the end of the final statistics.

Failures are tests that didn't succeed because one assertion returned False, meaning one of your expectations about how the code worked proved wrong.

The green dots at the top are tests that passed, so there is no more mention of them in the reports.

Next to the labels "ERROR" and "FAIL", you can see the test names that didn't succeed (in purple). This lets you locate where the problem occurred.

Finally, for each issue, you have a regular Python stack trace, which are the blue blocks starting with "traceback". If you don't know how to read traceback, lo and behold, we have an article on that.

In an ERROR, the stack trace will be showing you what part of the code exploded.

In a FAIL block, the stack trace will show you the assertion that returned False.

The various assertions

assertEqual is not the only assertion we can make, there are many of them: assertIs, assertIn, assertWarns...

We are going to leverage that to fix our tests.

Our "FAIL" in test_add_floats is because we made an assumption that 0.1 + 0.2 == 0.3 would return True. It's not so much as a bug in our code, as the fact our test was incorrect.

This will happen regularly when testing: you'll have to fix the tests, not just the tested code.

Because floating-point arithmetic in most programming languages use the IEEE 754 standard, we have some precision limitations:

>>> 0.1 + 0.2
0.30000000000000004

We cannot use equality for this test, but fortunately the unittest module comes with assertAlmostEqual that will let you check those two numbers are very close together:

def test_add_floats(self):
    result = add(0.1, 0.2)
    self.assertAlmostEqual(result, 0.3)

For the test_add_mixed_types, the error is warranted. We want the code to fail on a TypeError when the user pass mixed type. So let's rewrite the test to consider that an exception here is actually a success:

def test_add_mixed_types(self):
    # The test now passes if the function raises a TypeError
    with self.assertRaises(TypeError):
        add(1, "2")

Our test suite is getting happier:

python the_tests.py
....
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

Setup and tear down

There are groups of tests that will require you to prepare something before each test, such as a database, a file, or simply precalculate things. And others that will require to delete, clean up or close something after each test.

This can be done by declaring setup up and tear down methods on the test class:

import unittest

from the_code_to_test import add


class TestAddFunction(unittest.TestCase):

    def setUp(self):
        # Anything you attach to self here is available
        # in other tests
        print('This is run before each test')

    def tearDown(self):
        print('This is run after each test')

    def test_add_integers(self):
        result = add(1, 2)
        self.assertEqual(result, 3)

        result = add(1, -2)
        self.assertEqual(result, -1)

    def test_add_strings(self):
        result = add("1", "2")
        self.assertEqual(result, "12")

    def test_add_floats(self):
        result = add(0.1, 0.2)
        self.assertAlmostEqual(result, 0.3)

    def test_add_mixed_types(self):
        with self.assertRaises(TypeError):
            add(1, "2")


if __name__ == "__main__":
    unittest.main()

You can see those methods being automatically called:

python the_tests.py
This is run before each test
This is run after each test
.This is run before each test
This is run after each test
.This is run before each test
This is run after each test
.This is run before each test
This is run after each test
.
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

Using the test runner

Having:

if __name__ == "__main__":
    unittest.main()

at the end of each test file is, in fact, not mandatory. We can remove it, and use a test runner instead.

Test runners are programs that detect, collect and run all tests according to certain rules, such as the names of files, their place in the directory tree, etc.

Unittest comes with its own test runner nowadays, so we can also run all the tests calling it that way:

python -m unittest the_tests.py

Which gives us:

This is run before each test
This is run after each test
.This is run before each test
This is run after each test
.This is run before each test
This is run after each test
.This is run before each test
This is run after each test
.
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

Test runners work on a directory, not just a file, so as soon as your project grows, you will likely prefer to use one. There are other test runners in Python, such as nose, pytest, tox, nox, etc. And they can even come with a lot of tooling, as we will see with pytest.

Indeed, writing tests is a chore, and therefore we should make it as easy as possible to write them otherwise the motivation to do so drops fast.

pytest makes it much less painful to write tests, and comes with a lot of features you quickly learn to appreciate once you get serious about testing.

For all these reasons, we will not spend more time on unittest, and in the next part of this series, we will continue with pytest.

Also, tests are one of those areas where generative AI shines, do not hesitate to quiz your favorite LLM, be it copilot, chatgpt, claude or whatever, to create them for you.

Give them the function to test, and tell them to write a test for it. You'll often have to fix a few things, but it will be much faster than writing them by hand, at least for the simple ones.

And most tests are not that complicated.

Bite code!

Discussion about this post