personal web log written by izabeera and dryobates

testing CI tox

Tox - testing in multiply versions

by dryobates

In my work I use BuildBot for testing project in different python and Django versions. At home running BuildBot seems like a little overkill. Tox is an answer for build in home CI system.

For a long time I have used-home made set of Paver [1] recipes to test libraries in different python and/or Django versions. I still use Paver for automation, but part of my system has been replaced with Tox [2].

Before I send my library to pypi I want to:

  1. check code for “kwalitee” [3]
  2. test code under different python and Django versions
  3. check code coverage
  4. build package
  5. test package install
  6. sign and send package to pypi
  7. build and publish documentation

In that second requirement Tox comes with help. Creating different test environments in Tox was as simple as putting in tox.ini (sample for django-redisdb):

envlist = py27-django{12,13,14,15,16,17},py{32,33,34}-django{15,16,17}

commands = {envpython} test redisdb
deps =
    py27-django12: Django>=1.2,<1.3
    py27-django13: Django>=1.3,<1.4
    py27-django14: Django>=1.4,<1.5
    py{27,32,33,34}-django15: Django>=1.5,<1.6
    py{27,32,33,34}-django16: Django>=1.6,<1.7
    py{27,32,33,34}-django17: Django>=1.7,<1.8

It has created 6+3*3=15 environments and ran test in it. When I used only Paver I created virtualenvs for every platform combination, installed package in every virtualenv and run tests on installed version. The good think about it was that installed version was checked, so any missing files in package would make tests fail. The bad thing was maintaining all of this. Tox runs the same source code for all environments. Not installed package. So if I do mistake in packaging then tox won’t help me. But ease of maintenance has won. The other good part is that every project gets it’s own virtualenv and doesn’t conflict with other projects. Thanks to that I can run the same tox configuration on BuildBot [4] and different projects won’t break each other if they run in the same time. My list of development virtualenvs can also be kept short :)

With above configuration is one problem. It doesn’t check code coverage. When I used paver-only scripts I was merging coverage files created by each test in each virtualenv and then merge it using custom written rules for merging. With tox I have used solution found on the net [5]. So now my tox.ini looks like that:

Now my whole has simplified. Test task has been shortened to:

def test_install(options):
    """ Test installation of packages. """
    distfiles = glob.glob('dist/*')
    venv_bin_dir = '.tox/status/bin'
    pip_path = os.path.join(venv_bin_dir, 'pip')
    python_path = os.path.join(venv_bin_dir, 'python')
    for idx, distfile in enumerate(distfiles):
            # install package
            sh('%s install -q %s' % (pip_path, distfile))
            # test package installation
            sh('%s -c "import %s"' % (python_path,
            # uninstall
            sh('%s uninstall -q -y %s' % (pip_path, options.package_name), ignore_error=True)

Is that correct way to do it? Well, as you see “test_install” task is not quite beautiful. It has hard coded “status” environment that I use for testing installation. But maintainability wins. If you know some tool that makes running tests, building packages and testing packages easy to maintain, run fast and can be run on quite old home computer I’ll be more then happy to hear about it. For now I’ll stick with Paver + Tox in home and of course BuildBot at work.

[5]tox and coverage
Jakub Stolarski. Software engineer. I work professionally as programmer since 2005. Speeding up software development with Test Driven Development, task automation and optimization for performance are things that focus my mind from my early career up to now. If you ask me for my religion: Python, Vim and FreeBSD are my trinity ;) Email: