personal web log written by izabeera and dryobates

python celery debugging

Debugging python code

by dryobates

“Using debugger is the most primitive form of debugging” - once said my lecturer. Although it was said with pejorative intention sometimes we have to fall back to checking code with debbuger.

All of us probably know and from time to time use python debugger [1]. I won’t describe commands that you can use inside it. It’s extensively described in python’s documentation and with a lot of examples in PyMOTW [2]. For me the most annoying problem with debugging with pdb (except wanted bug) was how to start it in right moment.

Starting debugger

The easiest method to start pdb is from command line:

python -m pdb

For me working a lot with Django it would mean starting django project:

python -m pdb runserver

And stepping over a lot of code (or setting breakpoint far in the code).

More common I start it directly from code:

import pdb; pdb.set_trace()

That method has one caveat. Let us consider this code:

for i in range(100):
     import pdb; pdb.set_trace()

Even if you’ll give up debugging that loop and would like to continue after the loop you’ll have to type ‘c’ one hundred times to leave a loop. My co-workers often ask me if there’s a possibility to jump over that kind of “breakpoint”. Well I don’t know about any such built in method. What I do is using function to start debugger:

def set_trace():
    if 'pdb_running' not in globals():
        import pdb;
        global pdb_running
        pdb_running = True

for i in range(10):

Post mortem

Sometimes a bug appears in random places so you have three choices:

  • start debugger in on of places reported in traceback and count that it appear there again
  • step over lines in code from beginning or almost beginning
  • use post mortem debugging

In cases like that I use post mortem debugging. To start debugger in post mortem I attach my except hook:

import pdb
import sys

def post_mortem_hook(type, value, tb):
sys.excepthook = post_mortem_hook

With that code snippet instead of program exit on exception and beautiful traceback debugger will start exactly in the place where exception was raised so that you can check stack and all variables. That code should be run before exception is raised. It can be even at the beginning of the first loaded module.

Debugging hanging processes

I don’t remember where I’ve seen that code snippet for the first time. Long time ago I’ve noted it in my knowledge base as I thought it’s a brilliant idea. To be honest I don’t use it very often. Code I have to debug mostly run fast ;) or doesn’t run at all or if it run too long is killed by external monitoring tool and traceback appears in sentry [3]. Nevertheless here is code that after you send SIGINT to hanging program (Ctrl-C) will start debugger instead of killing it permanently.

import pdb, signal
signal.signal(signal.SIGINT, lambda signal, frame: pdb.Pdb().set_trace(frame))

Remote debugging

Some types of software are harder to debug then others. As most of my programming is connected with developing for web I often have to debug remotely running code e.g. started from web server that haven’t attached console.


In case like that in the past I used rpdb [4]:

pip install rpdb
import rpdb; rpdb.set_trace()

You can start it like regular pdb except that you type “rpdb” instead of “pdb”. Then on port 4444 is started server waiting for connection. You can connect to it through telnet on fixed port 4444:

telnet localhost 4444
Connected to localhost.
Escape character is '^]'.
> /usr/home/jstolarski/tmp/
-> print 'ok'

The problem with rpdb is that both program starting rpdb and telnet client can freeze in some cases. E.g. while “set_trace” was run twice. Moreover default telnet connection sometimes have problems with “unusual” keys like “tab”.


Debugger that has no problems with freezing is winpdb [5]:

pip install --allow-unverified winpdb --allow-external winpdb winpdb

It’s approach is opposite to that of rpdb. Rpdb start listening server in debugged script and allows for connections. Winpdb starts server in debugging console and from debugged script connection is made. If you have wxPython installed you can use GUI debugger. I like console tools so I sticked to rpdb2 - console debugger comming with winpdb. First I start debugger and set password for connections:

$ rpdb2
RPDB2 - The Remote Python Debugger, version RPDB_2_4_8,
Copyright (C) 2005-2009 Nir Aides.
Type "help", "copyright", "license", "credits" for more information.

> password "test"
Password is set to: "test"


Then in code I start connection to debugger:

import rpdb2; rpdb2.start_embedded_debugger("test")

In debugger console I can attach to one of waiting scripts:

> attach
Connecting to 'localhost'...
Scripts to debug on 'localhost':

   pid    name
   9323   /usr/home/jstolarski/tmp/

> attach 9323
> *** Attaching to debuggee...
> *** Debug Channel is NOT encrypted.
> *** Successfully attached to '/usr/home/jstolarski/tmp/'.
> *** Debuggee is waiting at break point for further commands.

> l
Source lines for thread 679489664 from file '/usr/home/jstolarski/tmp/':

Winpdb is better suited for threading or forking code then rpdb. E.g. It has some special commands to handle forking.

The method of running rpdb2 shown above isn’t the only one. We can save some typing and console switching. Debugged script by default waits 5 minutes for debugger server. So I it is possible to run this way:

import rpdb2; rpdb2.start_embedded_debugger("test")
$ rpdb2 --attach
A password should be set to secure debugger client-server communication.
Please type a password:test
Password has been set.
RPDB2 - The Remote Python Debugger, version RPDB_2_4_8,
Copyright (C) 2005-2009 Nir Aides.
Type "help", "copyright", "license", "credits" for more information.

> *** Attaching to debuggee...
> *** Debug Channel is NOT encrypted.
> *** Successfully attached to '/usr/home/jstolarski/tmp/'.
> *** Debuggee is waiting at break point for further commands.

The most annoying things in winpdb/rpdb2 is that it’s command set isn’t exactly similar to those in regular pdb.

Celery’s rdb

The last one remote debugger I would like to mention is “celery.contrib.rdb” [6].

It works similar to rpdb but it listens on port 6900 and if it’s in use it tries 6901, 6902 and so on.

from celery.contrib import rdb; rdb.set_trace()
telnet localhost 6900
Connected to localhost.
Escape character is '^]'.
> /usr/home/jstolarski/tmp/
-> print 'ok'

It’s well suited for debugging many parallel tasks that run with celery. You don’t need to use celery to use that debugger. It can successfully replace rpdb. Celery has more dependencies then rpdb so I wouldn’t install it only for debugger.

Which debugger

Which debugger do I use the most? None of presented here! ;) The most comfortable I feel with ipdb [7]. It’s a wrapper around IPython [8] debugger. I use ipython interactive console as my main python console. I didn’t find remote debugger that behaves like ipdb, so I try as much as I can not to debug remotely. Well, I try not to debug with debugger at all. Maybe I’ll write some day about methods I think are better then debugging with debugger.

I’m still waiting for a debugger with ipython console and remote capabilities. Do you know any?


And small bonus. I wouldn’t be myself if I didn’t give a try for debugger integrated with vim :) Not exactly remote debugging I’m looking for but if you have vim compiled with “+clientserver” you can give a try for vimpdb [9].

After quick configuration:

$cat ~/.vimpdbrc
vim_client_script = vim
vim_server_script = vim
server_name = VIM
port = 6666

Start vim with servername VIM:

$ vim --servername VIM

And then put in your code:

import vimpdb; vimpdb.set_trace()

And run script. In vim you’ll see loaded code and you’ll be able to debug from vim. Interesting idea but I feel that it’s a little to slow.

Jakub Stolarski. Software engineer. I work professionally as programmer since 2005. Speeding up software development with Test Driven Development, task automation and optimization for performance are things that focus my mind from my early career up to now. If you ask me for my religion: Python, Vim and FreeBSD are my trinity ;) Email: