Page Operations
Browse Space
 |
This page describes common errors when using Python twisted and its asynchronous programming patterns. |
Background
Twisted provides a reactor. If you start twisted via twistd, the reactor starts and calls your code. The reactor will always keep calling you back until it ends (assuming some event happens). When your code is called back, it runs until it either returns from a function call, or it hits a yield (see below).
Twisted introduces the concept of "deferreds". A deferred is an object with a promise to call back with a result later. Asynchronous functions return a deferred immediately and continue on for a while concurrently. When they are done, they trigger the deferred, which then calls you back at a function that you have specified. If an error occurred, an error handling function that you specified is called back. So far so simple.
1 2 3 4 5 | d = some_async_func(args) # d deferred is returned but not the actual result def _cb(res): print "Completed finally, moving on. Result=", res d.addErrback(logging.error) d.addCallback(_cb) |
This is clunky code. And sometimes you want to just wait for the result to finish before moving on, but still not block the entire reactor. I.e. you want your code to give control to the reactor until the deferred that you got calls you back with a result. The way to do this is by using Twisted's inlineCallbacks function.
1 2 3 4 | def my_sync_function(): d = some_async_func(args) res = yield d # Now I got the final result d1 = defer.inlineCallbacks(my_sync_function) # is again a deferred |
Note that the yield in the above code makes a generator out of a function. A generator is quite different from a function in that it does not return one value, but several values over time. In this case, it returns the deferred d as first result. You also call it in a different way (see Python generators). The inlineCallbacks function expect a generator and calls it back as often as necessary and waits for each deferred it receives. It itself returns a deferred that calls back when the generator is done.
There is more polished way to call inlineCallbacks by using a decorator:
1 2 3 4 5 | @defer.inlineCallbacks def my_sync_function(): d = some_async_func(args) res = yield d d1 = my_sync_function # is again a deferred |
Well, the problem is that you always end up with a deferred.
Pitfalls
Since @inlineCallbacks are decorators, odd errors pop up if you misuse them.
@inlineCallbacks but don't call yield:
You'll see this misleading error at runtime: exceptions.AttributeError: 'NoneType' object has no attribute 'send' as in
2010-01-28 16:32:55-0800 [-] [user_notification] File "/Users/hubbard/code/ve/dx/lib/python2.6/site-packages/twisted/internet/defer.py", line 746, in _inlineCallbacks 2010-01-28 16:32:55-0800 [-] [user_notification] result = g.send(result) 2010-01-28 16:32:55-0800 [-] [user_notification] exceptions.AttributeError: 'NoneType' object has no attribute 'send'
and wonder why, since you might not be sending anything. Yep, just remove the decorator or check for a missing yield statement.
yield but no @inlineCallbacks
A yield turns a function into a generator. If you call a generator, you get a generator object back, not a the result of a function.
If you have a yield, you cannot have a return. If you do, you immediately get the error (Komodo will catch this in the editor, thankfully):
SyntaxError: 'return' with argument inside generator
If you call a function that actually is a generator and don't do anything with the result (such as print it or access the value), then you will not notice any error. However the code in your function will never execute, because the generator is never activated.
If you expected a deferred and got a generator back it gets tricky. At some point you notice your code is not executed. Add the @inlineCallback
return from an @inlineCallbacks function
A yield turns a function into a generator (which provides values via yield). You cannot return a value using return. Use the following solution to still get a value back to the caller.
1 2 3 | from twisted.internet import defer ... defer.returnValue(result) |
(Komodo will catch if you use a return here)
returnValue from non-decorated function
returnValue actually raises an Exception to get out of a generator (typically), which is not what you would expect
1 | twisted.internet.defer._DefGen_Return |
You probably don't catch it so your end up with a high level Exception.
Functions just appear to hang
Despite the @defer.inlineCallbacks being set and properly yielding happening, sometimes function calls just seem to hang. This may be because the yield is offering up an unserializable object across an AMQP messaging interface. To avoid this, make sure that all yielded items that are destined for an AMQP message are serializable. Consider yielding a error or ok message if you are looking for something to return.