Cross posted from my work blog.


django.utils.datastructures is intentionally not documented by the django core devs because it is an internal API and is liable to change without any notice. This file is not governed by django's lenient backwards-compatible policy. You have been sufficiently warned!

With the note out of the way, let's look at the interesting datastructures in this file. You may ask why we should learn about those when we shouldn't be using them? Reading code is the best way of learning and this file has some beautiful code.

MergeDict is the first of the datastructures in the file. It provides a dictionary like interface but can look up from multiple dictionaries provided during the initialization.

Here's an example:

>>> md = MergeDict({"foo": "bar", "moo": "cow"}, {"abc": "def"})
>>> md["foo"]
>>> md["abc"]
>>> md.get("abc")
>>> md["xyz"]
>>> md.items()
[('foo', 'bar'), ('moo', 'cow'), ('abc', 'def')]
>>> md.keys()
['foo', 'moo', 'abc']
>>> md.values()
['bar', 'cow', 'def']

The MergeDict is used within django in attaching values with a form widget and in request.REQUEST.

The built-in dictionary does not maintain the order of the items but the SortedDict is a subclass of the built-in dictionary that maintains the keys in exactly the same order they were inserted.

Here's an example:

>>> dd = {"foo": "bar", "moo": "cow", "abc": "def"}
{"abc": "def", "foo": "bar", "moo": "cow"}
>>> sd = SortedDict((("foo", "bar"), ("moo", "cow"), ("abc", "def")))
{"foo": "bar", "moo": "cow", "abc": "def"}
>>> dd["xyz"] = "pqr"
>>> dd
{'abc': 'def', 'foo': 'bar', 'moo': 'cow', 'xyz': 'pqr'}
>>> dd["lmn"] = "ghi"
>>> dd
{'abc': 'def', 'foo': 'bar', 'lmn': 'ghi', 'moo': 'cow', 'xyz': 'pqr'}
>>> sd["xyz"] = "pqr"
>>> sd
{'foo': 'bar', 'moo': 'cow', 'abc': 'def', 'xyz': 'pqr'}
>>> sd["lmn"] = "ghi"
{'foo': 'bar', 'moo': 'cow', 'abc': 'def', 'xyz': 'pqr', 'lmn': 'ghi'}

The SortedDict is fairly widely used inside of django generally to build a hierarchy (like models and it's parents), maintaining the order of the form fields while iterating etc.

In python 2.7, a new datastructure was introduced that mimics the SortedDict in the collections module and is called OrderedDict.

MultiValueDict is a dictionary subclass that can handle multiple values assigned to a key.

Here's an example:

>>> dd = {"abc": "def", "foo": ["bar1", "bar2"]}
>>> dd["foo"]
['bar1', 'bar2']
>>> mvd = MultiValueDict({"abc": "def", "foo": ["bar1", "bar2"]})
>>> mvd["foo"]
>>> mvd.getlist("foo")
['bar1', 'bar2']
>>> mvd.getlist('blah')
>>> mvd.getlist('abc')
>>> mvd.setlist('xyz', ['pqr', 'ghi'])
>>> mvd
<MultiValueDict: {'xyz': ['pqr', 'ghi'], 'abc': 'def', 'foo': ['foo1', 'foo2']}>
>>> mvd.appendlist('xyz', 'ijk')
>>> mvd
<MultiValueDict: {'xyz': ['pqr', 'ghi', 'ijk'], 'abc': 'def', 'foo': ['foo1', 'foo2']}>
>>> mvd.update({'xyz': 'lmn'})
>>> mvd
<MultiValueDict: {'xyz': ['pqr', 'ghi', 'ijk', 'lmn'], 'abc': 'def', 'foo': ['foo1', 'foo2']}>

The MultiValueDict is used in binding data to request.POST, the files to request.FILES and in the get parameter parsing.

The ImmutableList is an immutable datastructure that raises errors when it is attempted to be mutated.

Here's an example:

>>> il = ImmutableList(['foo', 'bar', 'abc'])
>>> il += 'lmn'
AttributeError: ImmutableList object is immutable.
>>> il = ImmutableList(['foo', 'bar', 'abc'], warning='Custom warning')
>>> il[1] = 123
AttributeError: Custom warning

The ImmutableList is used in request.upload_handlers to prevent modification after the request.POST or request.FILES have been accessed.

The DictWrapper is a subclass of the built-in dictionary that prefixes the keys. It takes a dictionary, a function and a prefix as arguments. If a specific key lookup begins with the prefix then the value is passed through the function before it is returned.

Here's an example:

>>> dw = DictWrapper({'foo': 'bar', 'moo': 'cow'}, lambda x: x, 'abc_')
>>> dw['foo']
>>> dw['abc_foo']
>>> dw['xyz_foo']
KeyError: 'xyz_foo'
>>> def post_process_value(value):
...     return "The value is " + value
>>> dw = DictWrapper({'foo': 'bar', 'moo': 'cow'}, post_process_value, 'abc_')
>>> dw['foo']
>>> dw['abc_foo']
'The value is bar'

The DictWrapper is used in quoting names for SQL queries with the key prefix.

Hope you enjoyed learning about these hidden gems and how django works under the hood but take the note on the top into consideration.

Dropbox file upload handler for django

Tue 17 July 2012 by Thejaswi Puthraya

Cross posted from work blog.

Dropbox announced new pro plans last week and some accounts have had their storage size doubled. Wouldn't it be wonderful if we could upload all our files to dropbox from our django webapp?

In this post, I write a custom file upload handler that will …

read more

Behind the Scenes: From request to response

Mon 02 January 2012 by Thejaswi Puthraya

Cross posted from my work blog.

In the previous installment of "Behind the Scenes", we saw how the control flows from Form to File Storage. Today, we are going to see how the application reacts from request to response.

In this post, we are going to assume that we are …

read more

Behind the Scenes: From HTML Form to Storage

Wed 21 September 2011 by Thejaswi Puthraya

Cross posted from my work blog post

In this post, we are going to see what happens behind the scenes when a file is uploaded to a django powered web application.

File Upload Flow

An HTML form with a file input (atleast one) and encoding set to multipart/form-data is submitted. The MultiPartParser …

read more

Released django-check-constraints

Mon 15 March 2010 by Thejaswi Puthraya

Ever since model validation hit the django trunk, I had been having thoughts of reviving my Google Summer of Code 2007 project. So nearly a couple of weeks back, I started resurrecting django-check-constraints. Today, I released the code into wild here.

The feature of the project is that it not …

read more

Confista: A conference management app

Tue 24 November 2009 by Thejaswi Puthraya

I can't believe I had been living under a rock all this while! How could I have missed Pinax? I knew it existed but was under a wrong impression that it was only for the social stuff.

Recently, I forced myself to look at Pinax for a project and after …

read more

RFC: Comments framework usage documentation

Thu 29 October 2009 by Thejaswi Puthraya

This is my attempt at ticket #9819 which aims for better documentation to the comments framework. This is my second attempt at writing documentation and I have uploaded a patch to the ticket but wanted some feedback here without spamming the timeline.

Example of using the in-built comments app

  1. Follow …
read more

A wiki with a couchdb backend and django

Fri 28 August 2009 by Thejaswi Puthraya

Recently, I had a requirement for a personal simple wiki. I checked out various options like TiddlyWiki, MoinMoin and MediaWiki but wanted something very simple and something that didn't use a database server or flat files and was portable (and hostable). So I quickly wrote up a wiki that used …

read more

Reusable App: Authenticated Comments

Tue 04 August 2009 by Thejaswi Puthraya

This app had been on my todo list for a long time. Today, no matter how much I tried to procrastinate, I could not ignore it and hence decided to scratch my itch and get it done. I swore I wouldn't even get up for a glass of water until …

read more

Yet another redesign at

Wed 08 July 2009 by Thejaswi Puthraya

Oh yes!!! You are seeing it right! There's been another redesign of my site.

Technology fascinates me and I cannot get enough of it. This time I wanted my blog powered by the latest and coolest technology. Want to know more?

CouchDB powers the heart of the blogging engine as …

read more