Make Django tests rebulid the database if it exists

Have you ever seen this log?

Got an error creating the test database: database 'db_name' already exists

There are plenty of tickets for this issue hanging there for over 10 years. The problem with resolving it is that it’s actually not a bug. It’s all just a matter of a proper configuration of your IDE.

It shouldn’t be attributed to Django, as if you run tests from a standalone Django app you probably won’t bump into this issue. I noticed this problem occurs only when I’m using PyCharm.

How Django handles testing

Adam Johnson already exhausted this topic in his blog post What happens when you run manage.py test? so go there if you’re looking for a more detailed explanation.

For this post all you need to know, is that

• test is a Django command that runs the test suite using Django test runner
• --noinput is a test command flag that suppresses user input
• noinput flag is translated to interactive variable and then is passed to setup_databases function
• create_test_db is invoked with autoclobber=not interactive

Here’s the snippet of setup_databases where Django asks you if you want to delete the DB

    self.log('Got an error creating the test database: %s' % e)
    if not autoclobber:
        confirm = input(
            "Type 'yes' if you would like to try deleting the test "
            "database '%s', or 'no' to cancel: " % test_database_name)
    if autoclobber or confirm == 'yes':
        try:
            if verbosity >= 1:
                self.log('Destroying old test database for alias %s...' % (
                    self._get_database_display_str(verbosity, test_database_name),
                ))
            cursor.execute('DROP DATABASE %(dbname)s' % test_db_params)
            self._execute_create_test_db(cursor, test_db_params, keepdb)

What PyCharm has to do with it

When you run test command you can specify if you want to run it interactive or not. The problem is that if you’re using docker as I do, pressing this green triangle near your test will result in running this command

/usr/local/bin/docker-compose -f /Users/tom/PycharmProjects/project/docker-compose.yml -f /Users/tom/Library/Caches/JetBrains/PyCharm2020.2/tmp/docker-compose.override.969.yml up --exit-code-from backend --abort-on-container-exit backend

As you can see, it runs docker-compose up and the stack will be removed on exit code from backend (Django app). I added a long time.sleep in my tests so I could see what command is actually run in this container.

python -u /opt/.pycharm_helpers/pycharm/django_test_manage.py test project.tests_file.TestsClass.test /opt/project/backend

So what’s wrong with that exactly? The problem is that it runs Django test command without the flags that I want.

Though the docker-compose up command can be configured in Run/Debug Configuration, I have no idea how to adjust PyCharm’s test command and I think it’s safer not to rely on PyCharm’s configuration.

Custom test runner

When I run my tests I want them to be run in the exact same way everywhere. That’s why I came up with my own TestRunner which will always suppress user input. That way, whenever Django asks me if I want to drop the DB, it will automatically select yes.

Pros:

• portable, replicable testing

Cons:

• will “accept” everything, not only “do you want to rm the DB?”, so might be unpredictable
• I might not want this to happen in the future

As I don’t care about the cons, I’m fine with this solution.

DiscoverRunner is the default Django test runner.

# app.test_runner.py

from django.test.runner import DiscoverRunner


class NonInteractiveTestRunner(DiscoverRunner):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.interactive = False

    def teardown_databases(self, old_config, **kwargs):
        try:
            super().teardown_databases(old_config, **kwargs)
        except RuntimeError:
            pass

and install it in settings

TEST_RUNNER = "app.test_runner.NonInteractiveTestRunner"

Why do I silence RuntimeError, you ask?

I often kill my tests while they’re running so I thought Django didn’t have enough time to do the cleanup. To my surprise when I let Django finish the test suite, the RuntimeError was raised when running teardown_databases. Apparently it was the root cause of the issue all along. Somehow Django was silencing this exception when running with the default runner.

So why the DB can’t be teardown? When docker-compose up is executed Django app is waiting for the DB to be up and running but once all tests are finished the DB service is stopped before Django does the cleanup. Or at least I think that’s what is happening here.

Result?

Creating test database for alias 'default'...
Got an error creating the test database: database "db_name" already exists

Destroying old test database for alias 'default'...

If you want to be more specific you can always handle it in your custom setup_databases and teardown_databases though the way I handle it is very simple and it just works and that’s all I need.