Wednesday, August 25, 2010

suds and appengine

suds is an incredibly useful and easy to use client library for SOAP.  Many times better than using SOAPy or ZSI (for fun client stuff, anyway).

However, suds has a couple of problems working with appengine.  This is from versions 0.3.9 and 0.4.

(complete file for what follows:

1. Its default client constructor has a hard initialization of self.options.cache = ObjectCache(days=1), and ObjectCache by default is a FileCache, and FileCache tries to use a temporary file which app engine doesn't like.  Maybe we can use memcache instead?

2. For some reason the suds.transport.http.HttpTransport.u2open() function breaks here:

        if self.u2ver() < 2.6:
            return, timeout=tm)

at least in app engine, that is.

I wanted to make it work with app engine without actually changing any of the suds code itself.  Here's how I did it.  Now I realize I could have overridden ObjectCache(days=1) with a similar effect, but then it would have had to ignore days which seems dirty.

First, I overrode the u2open() function in transport.http.HttpTransport:

def new_u2open(self, u2request):
    tm = self.options.timeout
    url = self.u2opener()
    if self.u2ver() < 2.6:
        return, timeout=tm)

transport.http.HttpTransport.u2open = new_u2open

Then, I had to override the entire Client constructor because of the way suds initializes options.cache in the original.  And, I'm not sure if it's possible to just override pieces of a constructor in Python without executing it (I imagine it isn't).

class Client(client.Client):
    def __init__(self, url, **kwargs):
        options = Options()
        options.transport = HttpAuthenticated()
        self.options = options
        options.cache = MemCache()
        reader = DefinitionsReader(options, Definitions)
        self.wsdl =
        plugins = PluginContainer(options.plugins)
        self.factory = Factory(self.wsdl)
        self.service = ServiceSelector(self, = []
        for s in
            sd = ServiceDefinition(self.wsdl, s)
        self.messages = dict(tx=None, rx=None)

And then I threw together a memcache implementation of suds.cache.Cache.  I copied the definitions from suds's suds.cache.Cache abstract class, so it has extra "getf" and "putf" methods, and I'm not sure what they do, so I just overrode them too:

class MemCache(cache.Cache):
    attempt to implement a memcache cache in suds
    def __init__(self, duration=3600):
        self.duration = duration
        self.client = memcache.Client()
    def get(self, id):
        thing = self.client.get(id)
        return thing
    def getf(self, id):
        return self.get(id)
    def put(self, id, object):
        self.client.set(id, object, self.duration)
    def putf(self, id, fp):
        self.put(id, fp)
    def purge(self, id):
    def clear(self):

I'm not sure how well the MemCache() Cache works, but I know it doesn't crash appengine on my local instance, and my app runs fine with an WSDL URL loaded.  Here's how you want to call it in your appengine client.  Imagine the file you put all this junk into is named, suds_memcache:

from suds_memcache import Client
    url = ('      WSDL')
    c = Client(url)
    print c

Well, hopefully you get the idea, anyway.  Here's the complete file:

Note:  If MemCache() doesn't work for you, or you just don't want to use it, you can use suds.cache.NoCache() instead (which comes with suds).  But you still need to put it in the constructor override.

I just noticed that I had a few problems using this with 0.3.9 (need to remove the Plugin stuff because it's only in 0.4).  Also, when loading a wsdl off the disk, I noticed that the MemCache() implementation fails when reading or writing to app engine's memcache.  This is because suds passes an ObjectId object to the caching functions (at least in 0.3.9), but app engine expects a string and throws a TypeError otherwise.  I fixed this by overriding suds.reader.ObjectId with a __str__ method (it didn't have one), and then wrapping the id variables in the MemCache() implementation with str().  I'll post the new version later tonight.  Maybe I'll put it on my github or something, too.  UPDATE:  Uploaded the newer file.

I found out you don't actually have to override the Client constructor.  Chalk it up to not fully understanding the suds code.  You can just pass in cache=None or cache=suds_memcache.MemCache() (or you should be able to, anyway.  I've had success so far).  e.g.
client = suds.client.Client(url, cache=None)
I'm still having to use the u2open() override, though.


l8rs said...

This was really helpful. Awesome post!!!

Gwyn Howell said...

Excellent post - thanks!