A Twisted Mind

The Talks I Saw At Pycon

2013-03-22T00:00:00-07:00

PyCon 2013 was amazing. Here are video links to the talks I saw whose videos are currently available. I liked them all.

In no particular order:

Of course, these are just a few of the talks at the conference, you can find more at pyvideo.org.

Deferreds Are A Dataflow Abstraction

2012-03-30T00:00:00-07:00

This was originally concieved as a response to A Conversation with Guido about Callbacks over at Duncan McGreggor’s blog.

First, a jQuery-ish example of using some callbacks:

def handleResponse(resp):
    def handleParsed(xmlObj):
          firstStyle = xmlObj.xpath('//link[rel="stylesheet"]')[0]

        def handleAnotherResponse(anotherResp):
              print anotherResp

        getPage(firstStyle['href'], handleAnotherResponse)

    asyncParser(resp.body, onComplete)

getPage("http://example.com", handleResponse)

Now this is an absurd example of callback soup. Or is it? I’ve read a lot of JavaScript, and it isn’t really uncommon to see multiple levels of calls which take a callback. The JavaScript actually usually looks cleaner and is easier to read than our contrived Python because JavaScript has multi-statement anonymous functions that can be used to inline the callback definition into the function call. (That feature has other readability side-effects though, such as the pattern of defining a single argument anonymous function to invoke a single, single argument function.)

Now, when you write callbacks like this, it’s no wonder people tend to dislike them. Especially when contrasted with the more straightforward non-callback example.

resp = getPage('http://example.com')
xmlObj = parser(resp.body)
firstStyle = xmlObj.xpath('…')[0]
print getPage(firstStyle['href'])

4 lines of code vs 8 lines of code (11 including almost mandatory blank lines.) Is it any wonder that Guido prefers this approach? No, absolutely not. That first example is an absolutely terrible but perfectly normal piece of callback using code. Let’s enumerate some of the ways in which it is bad.

Everything is out of order.
Excessive indentation. Flat is better than nested, but close is better than far, so nested wins.
The desire for locality causes function definitions to be scattered among statements.
You’re defining a lot of non-reusable, non-testable functions.

I’m sure you’ve come up with plenty more of complaints (including non-PEP8 camelCase names) but these are the 4 big points I’d like to discuss further.

Order matters.

Most of us read left to right and top to bottom. Even if your primary language doesn’t read left to right and top to bottom your primary programming language almost certainly does. When we make todo lists we give them an order, items at the top are more important. Shopping lists are sometimes ordered starting with vegetables and moving towards more time fragile things like ice cream, because that’s the way the stores are laid out. Computer programs are written line by line in the order we want instructions executed, because that is how computers work.

Order matters. On first encounters with code order matters a lot.

Related statements should be close.

Locality is the topic of both issues 2 & 3. And it can best be summed up as, related things should be close. It’s the most basic element of organization for everything from kitchens to code.

Anytime you’re reading anything you’ll find yourself glancing back to reread a few lines, or sentences, or even words as soon as you’ve encountered something that you don’t understand or doesn’t quite make sense. The further you have to look the longer it’s going to take to get your answer.

Reduce, Reuse, Retest.

We write functions to enhance the readability, and testability of our code. We reduce the size of large functions by breaking them up into higher level discrete steps and creating functions to encapsulate those actions. We reuse common code by creating more functions. We retest, ok well retest doesn’t make much sense, but we increase testability by concentrating functionality into smaller more well defined units.

You would never inline all the socket calls necessary to make an HTTP request. Maintainable software is built out of well defined reusable and testable abstractions, and there is no reason that callbacks should not also be reusable and testable as much as possible.

There is a better way.

I’ll admit, I have a love/hate relationship with callbacks. I view them as a necessary evil to do event-driven programming. There is of course a better way to think about callbacks and you may have already guessed it. Deferreds and Dataflow programming. A lot of people use deferreds for the first time much the same way they use plain callbacks.

d = getPage(url)
def handleResponse(resp):
    def handleParsed(xmlObj):
        firstStyle = xmlObj.xpath(pattern)
        def handleOtherResponse(otherResponse):
              print otherResponse

        d3 = getPage(firstStyle['href'])
        d3.addCallback(handleOtherResponse)

    d2 = asyncParser(resp.body)
    d2.addCallback(handleParsed)

d.addCallback(handleResponse)

Now, this example, though still contrived is not unrealistic, it is in fact perfectly functional and probably exists in more than a few Twisted using code bases. Of course it’s terrible for all the same reasons our plain callback example was terrible.

So clearly it is not the better way. So what is?

Deferreds as a dataflow abstraction.

Here you have a acyclic directed graph of data through the callback chain. The individual operations being performed is less important than the flow of information through the structure. An event causes a result to be available, it is put into the deferred and moves along the directed graph through the callback chain until an error is encountered. Then we move to the errback chain will proceed until the error has been handled at which point we can go back to the callback chain, or until we run out of errbacks.

The result from one operation flows directly into the next.

getPage(getFirstStyle(parse(getPage("http://example.com")))

The above is a dataflow program. Our imperative example from earlier is not, because the grouping of operations is a side effect of style, any number of operations could be interspersed which may or may not have side effects that influence future operations.

d = getPage("http://example.com")
d.addCallback(parse)
d.addCallback(getFirstStyle)
d.addCallback(getPage)
d.addCallback(printResult)

Now, everything is order, related operations are close, and the structure of higher level operation is broken up into discrete and independently testable units.

By this point I hope I have made it obvious about why “thinking in Deferreds” is better than “thinking in callbacks” but if I have not imagine this example from a fictional streaming API and maybe you’ll understand why “thinking in dataflows” is a good thing.

urls.flowTo(getPage)
    .flowTo(parse)
    .flowTo(getFirstStyle)
    .flowTo(getPage)
    .flowTo(stdout)

urls.put("http://example.com")
urls.put("http://google.com")

Now we have encapsulated not only how we handle one result, but rather a potentially infinite number of events. Now we have the potential for building some realtime distributed systems.

That is all.

Go read about Storm and Orc

Klein A Twisted.web Microframework

2012-03-28T00:00:00-07:00

What is Klein?

klein is a micro-framework built out of widely used components that aims to be production ready out of the box.

Inspired by frameworks such as flask and bottle it features:

A minimalist API.
A fast, event-driven, production-ready web server.
Complete support for a wide range of asynchronous client APIs.

It’s routing is provided by werkzeug and it’s webserver & event-loop by Twisted.

An Example

from klein import run, route

@route('/')
def hello(request):
    return 'Hello, world!'

run('localhost', 8080)

This is the most simple example possible. It’s actually simpler than the first bottle example. This is the example you want to use in the “how fast can I serve a hard coded static string” benchmark you post to Hacker News

This post will not contain one of those benchmarks. Lets break it down.

from klein import run, route

run and route are the most basic APIs and it should be pretty clear what they are for. In this example we’ve imported the top-level functions which actually operate on a global instance of a the class klein.Klein.

This class may be used directly to facilitate a more flask-like usage. For example:

from klein import Klein

app = Klein()

@app.route('/')
def hello(request):
    return 'Hello, world!'

app.run('localhost', 8080)

Next we have the actual handler definition:

@route('/')
def hello(request):
    return 'Hello, world!'

The route decorator is built on top of Werkzeug and passes all unknown arguments directly to werkzeug.routing.Rule.

You may have noticed that unlike both flask and bottle handler functions in klein take an explicit request argument, which is an IRequest provider. This is primarily because “Explicit is better than Implicit”, but also because anything else would greatly complicate the implementation.

At this point we have our feet firmly planted in the world of Twisted. Though our example does not do anything asynchronous, or Twisted specific the full power of Twisted is available to us. You may return a str, unicode, twisted.web.template.Element, or twisted.web.resource.IResource, and klein will handle all of them for you. You may of course also return a twisted.internet.defer.Deferred which fires with any of the above as it’s result and we will handle that as well. The README of course has examples covering the full range of return values.

Finally we are ready to run the server.

run('localhost', 8080)

This starts up a twisted web server listening on port 8080 and only accessible to localhost. You can listen any specific IP address, or ‘0.0.0.0’ for all IP addresses. As a bonus logging to stdout is set up for you.

Work in Progress

klein is of course not complete, as no software is ever complete. It does however have a comprehensive test suite and builds on Twisted’s own heroicly extensive tests. We have lots of ideas about how to improve it and don’t plan to stop developing it. It is not currently used in production anywhere but it’s only a matter of time.

If you try it give us feedback in #twisted.web on freenode. And of course fork it on github.

Why I still trust Tor, WikiLeaks, and Jacob Appelbaum.

2010-12-29T00:00:00-08:00

I feel compelled to comment on Zed Shaw’s recent “analysis” of an apparent conflict of interest concerning Tor developer and WikiLeaks supporter Jacob Appelbaum.

If you don’t know what I’m talking about you should start by reading Why I Don't Use Tor

I’m going to try to not be too “logical” in my argument because apparently that will cause my argument to be inherently flawed. So instead I’m going to try to mix emotion with logic in what I hope will not be too volatile a cocktail for this blog post to contain.

I’ve met Jacob Appelbaum multiple times, mostly through our mutual friend Zooko of “Zooko’s Triangle” and Tahoe-LAFS fame. He seems like a great guy, nice, smart, funny, and passionate about personal privacy. It was his demo of Tor on Android over cheap Korean BBQ which convinced me to give up my iPhone.

I’ve also met Zed Shaw. He was abrasive, and seemed offended that I did not recognize him as “Zed ‘Mother-Fucking’ Shaw” but who knows, maybe that was just his schtick.

The above linked post is not what I would consider a classic example of trolling. I believe that Zed Shaw probably believes the things he’s written there and that’s fine. I’m not one to judge people’s other personal beliefs, however his argument seems flawed not only because he bases partly on a well known-rumor that has been repeatedly denied while no evidence has ever been offered for it. But also because it makes certain assumptions which I’m not convinced are valid. 3 to be exact.

1) Jacob Appelbaum is in a position to drastically impact the security of Tor.

Yes, and no. He is certainly qualified, and what little I know of the project’s organization he seems to be in a position where he could make decisions which could impact Tor’s security. However Tor is an open code base, freely available for anyone to audit and not the work of one lone programmer. This does not mean that everyone is qualified to audit the code, I’m certainly not, however I think enough people care about personal privacy and anonymity that qualified individuals are looking at it.

2) WikiLeaks’ job would be easier if an insecure Tor were in widespread use.

I don’t think it would be. As far as I know, there is no evidence to suggest that information obtained by WikiLeaks has ever come from a source other than a conventional whistleblower. A person with some level of access to certain information decides that it is right, just, or worth 15 minutes of fame to leak some information. If Bradley Manning leaked those diplomatic cables, and that collateral murder video he probably didn’t have legitimate access to that information, but he also didn’t obtain it by setting up a rogue Tor exit node. He had legitimate access to military networks.

On a second point, in a future where Tor usage is widespread, I would also expect that you would see Tor Hidden Services also be widespread and any organization hoping to insure anonymity and privacy to their users would be better off running one of those than a normal website that users could access by Tor.

3) WikiLeaks’ job would be harder if a secure Tor were in widespread use.

I hope so, but probably not sufficiently so that it would be worth risking both Tor and WikiLeaks’ reputations to do so. On the off chance that an oppressive regime or a major corporation used Tor for all their most dirty little secrets it’s still more likely that the heavily bearded sysadmin in Reeboks and a Fedora would take that information and leak it than some hacker or even enemy government or corporation would steal it.

Of course, these 3 points only matter if you’re willing to completely ignore the fact that Tor and WikiLeaks’ interests do not actually conflict. WikiLeaks does not appear to be interested in posting the latest celebrity sex tape or pictures from Miley Cyrus’ Facebook page.

In conclusion I think it’s far more likely that Zed Shaw wanted to get some hackernews juice by blogging about WikiLeaks than it is that Jacob Appelbaum and Julian Assange want to read your email.

Mimicing source virtualenv/bin/activate in Emacs.

2010-02-03T00:00:00-08:00

For a while now I’ve had a problem where at work we rely heavily on virtualenv and I occasionally want to do things like run pyflakes or run trial from the Twisted installed in that virtualenv. This following activate-virtualenv function appears to be all that is necessary to mimic source virtualenv/bin/activate for Emacs.

Enjoy.

Twisted + Django + Pinax: Hey, yeah that works.

2009-03-30T18:44:00-07:00

Ok, so yeah. As of about 2 hours ago Pinax runs on top of Twisted trunk. These seems to indicate that Twisted is a pretty capable and correct WSGI implementation. I suggest you start serving stuff. It’s great! Here is a screenshot.

Twisted + Django: It won't burn down your house.

2009-03-30T09:34:00-07:00

It’s actually really easy to make these two things work together. Since ~8.2.0 Twisted has had a WSGI container runnable from the command line. So here is how you run Django on a supported version of Twisted.web.

easy_install Twisted
easy_install Django
Profit!
django-admin.py startproject foo

Create a myapp.py with the following code:


from django.core.handlers.wsgi import
WSGIHandler

application = WSGIHandler()

export DJANGO_SETTINGS_MODULE=foo.settings
twistd -no web --wsgi=myapp.application

And there you have it. There are probably a few compatibility problems but this mostly seems to work just fine. I’ll be working with some Django weenies to figure out what, if anything is broken.

So look, this is really AWESOME.

Updated (3/30/2009 4:09pm): Twisted tickets #3585, #3721 are problems on Twisted 8.2.0 but are fixed or should be fixed in Twisted trunk soon.

Updated (3/30/2009 6:34pm): Both of the above tickets are done.