Stephen McDonald

Topological Sorting Acyclic Directed Graphs

2012-04-06T00:00:00-07:00

The actual code written when developing a typical web application, ranks pretty lowly on the complexity scale, when compared to the rest of the software engineering industry. Of course, web development has its own set of interesting complexities around architecture and scaling, but the application code itself is relatively simple. Glue one library to another, update a row in a database, print some more rows out, send out an email, and so on.

As a web developer, I always enjoy the chance to dig deeper into more formal data structures and algorithms when the opportunity arises. At Fairfax I’ve recently had two distinct situations come up where I’ve needed to work through sets of data, with each of the items in the set containing references to other items that they depends on. The desired outcome was to sort the items so that I could process each item, knowing that each time I reached an item with dependencies, those dependent items would already have been processed.

The first scenario was one with serialized Django models. I had a number of model instances that would be instantiated and serialized, with the aim of persisting them to the database at a later point. These models contained relationships with other models that were also being serialized. Temporary primary keys were used in order to build their relationships, allowing all of the models to then be serialized with their relationships intact.

The problem then arose after deserializing these instances, and persisting them to the database. Each model with a foreign key to another couldn’t be processed, until the model it was related to had been persisted to the database first, as real primary keys are required for foreign key relationships to be defined. My first solution was a quick hack that manually sorted the objects correctly, knowing in advance which classes of models I was dealing with. This of course meant the code couldn’t be reused in a general manner, and as more classes of models were added to the mix down the track, this would need to be addressed.

The second case like this came up in an entirely different situation. At Fairfax we’re building an impressive distributed system built on many RESTful APIs. Each of these APIs contains its own schema, and we set out to build a tool that could introspect these schemas, and generate client objects for interacting with them. These schemas also contain relationships between resources, and these resource relationships are also modelled in our client objects. Again this situation was one where I needed to process an unordered set of resources in the correct order, so that their relationships could correctly reference resources already handled.

Shyly fool my bitten shame twice, or something like that. When this problem came around the second time, I didn’t have the luxury of knowing up front what the exact classes of data I’d be dealing with were, as they were due to change quite quickly as we iterated. I then set out to implement the correct solution for the problem, that could be applied to both of these situations.

A quick Google search for “dependency resolution” brought me to the Wikipedia page for topological sorting, which was the solution I was looking for. In both cases, we have graphs. Think of a graph as points on a map called nodes, with each node connected by lines called edges, to other nodes in the graph.

A simple graph

A directed graph is one where each of the edges contain a direction, so each of the lines contain an arrow pointing one way or the other. An acyclic directed graph is where each of the edges only point in one direction, so that it’s not possible to follow the edges from one node to another, returning back to the original node. If this last condition is not satisfied, then the graph is said to contain directed cycles, that is, a path can be followed from one node to others, and back to the original node again.

An acyclic directed graph

In both scenarios I faced, my graph structure was different from those commonly found in examples of topological sorting. I had the outgoing edges defined, rather than incoming edges. The gist of the topological sort I needed, is to repeatedly go through all of the nodes in the graph, moving each of the nodes that has all of its edges resolved, onto a sequence that forms our sorted graph. A node has all of its edges resolved and can be moved, once all the nodes its edges point to, have been moved from the unsorted graph onto the sorted one.

Consider the graph above from left to right, as pairs of nodes and their outgoing edges:

graph_unsorted = [(2, []),
                  (5, [11]),
                  (11, [2, 9, 10]),
                  (7, [11, 8]),
                  (9, []),
                  (10, []),
                  (8, [9]),
                  (3, [10, 8])]

Our expected output from a topological sort function would be as follows, with no nodes containing edges pointing to nodes before themselves:

>>> from pprint import pprint
>>> pprint(topolgical_sort(graph_unsorted))
[(2, []),
 (9, []),
 (10, []),
 (11, [2, 9, 10]),
 (5, [11]),
 (8, [9]),
 (3, [10, 8]),
 (7, [11, 8])]

Note that the ordering need not be precisely the same each time. In the result above, node 9 could come before node 2, as neither of these contain edges, so they equally belong first in the sorted graph.

Here’s an implementation of my topological sort in Python:

def topolgical_sort(graph_unsorted):
    """
    Repeatedly go through all of the nodes in the graph, moving each of
    the nodes that has all its edges resolved, onto a sequence that
    forms our sorted graph. A node has all of its edges resolved and
    can be moved once all the nodes its edges point to, have been moved
    from the unsorted graph onto the sorted one.
    """

    # This is the list we'll return, that stores each node/edges pair
    # in topological order.
    graph_sorted = []

    # Convert the unsorted graph into a hash table. This gives us
    # constant-time lookup for checking if edges are unresolved, and
    # for removing nodes from the unsorted graph.
    graph_unsorted = dict(graph_unsorted)

    # Run until the unsorted graph is empty.
    while graph_unsorted:

        # Go through each of the node/edges pairs in the unsorted
        # graph. If a set of edges doesn't contain any nodes that
        # haven't been resolved, that is, that are still in the
        # unsorted graph, remove the pair from the unsorted graph,
        # and append it to the sorted graph. Note here that by using
        # using the items() method for iterating, a copy of the
        # unsorted graph is used, allowing us to modify the unsorted
        # graph as we move through it. We also keep a flag for
        # checking that that graph is acyclic, which is true if any
        # nodes are resolved during each pass through the graph. If
        # not, we need to bail out as the graph therefore can't be
        # sorted.
        acyclic = False
        for node, edges in graph_unsorted.items():
            for edge in edges:
                if edge in graph_unsorted:
                    break
            else:
                acyclic = True
                del graph_unsorted[node]
                graph_sorted.append((node, edges))

        if not acyclic:
            # Uh oh, we've passed through all the unsorted nodes and
            # weren't able to resolve any of them, which means there
            # are nodes with cyclic edges that will never be resolved,
            # so we bail out with an error.
            raise RuntimeError("A cyclic dependency occurred")

    return graph_sorted

I imagine some other uses for topological sorting would be task queues, where certain tasks are dependent on other tasks being completed first. It could also be used by package managers that install software libraries, to ensure each library has its dependencies met before it’s installed.

Mezzanine 1.0: The Aftermath

2012-03-05T00:00:00-08:00

This weekend I finally launched Mezzanine 1.0 after two years in development.

Rather than talk about Mezzanine itself and the lead up to 1.0, I thought it would be fun to look at the online reach of the release announcement, as well as some of the growth that has occurred over the last 18 months since I first made Mezzanine available. For more detail on Mezzanine and the release itself, check out the 1.0 announcement to the wider Django community, as well as the discussion leading up to the 1.0 release.

Active Growth

At the end of 2010 I wrote my annual year in review post, where I talked about Mezzanine and the activity that had occurred in its first 6 months. With the new 1.0 release, I thought it would be interesting to take a fresh look at some of the stats I talked about back then. We can see below that growth has continued all the way through at a steady pace.

	Version 0.9.16 months	Version 1.0.02 years
Project followers	120	430
Project forks	30	130
Project contributors	10	40
Mailing list subscribers	60	190
Mailing list messages	300	1,800
PyPI downloads	4,000	38,000
Homepage visitors	9,000	47,000

Combined follower/fork count for GitHub and Bitbucket brought to you by One True Repo

The Launch Party

The launch party we held for the release was a raging success. Just to be clear, by we I mean me, and by party I mean sitting at my computer all day with several browser tabs open, feverishly refreshing them with the hope of some good exposure and interest around the release. And there was plenty!

After first announcing the release to the django-users mailing list, I then posted it to a handful of popular channels in the programming community. All around the responses were positive, ranging from congratulation and praise, to all sorts of questions from curious people in the Django space who hadn’t heard of Mezzanine before.

django-users

As mentioned, I made the initial announcement to the django-users mailing list. Here I gave a general overview of Mezzanine and Cartridge, and went through their core features. The follow-up responses were positive, and included some questions about how Cartridge compares to other more popular ecommerce Django apps. I then gave a short version of my original post from when I first made Cartridge available, which covers that area in more detail.

Hacker News

I’ve posted a few articles to Hacker News before, but they’ve never been promoted to the front page, which occurs after receiving enough votes from the community. So I was delighted when 10 minutes or so after posting the announcement to Hacker News, it had reached enough votes to hit the front page. After that the votes came pouring through, with the announcement eventually making its way to 4th place on the front page, where it remained for most of the day.

When I woke up the next day, the article had reached over 90 votes, and a good range of questions had been posted, from looking for help getting started, to comparisons against other CMS projects. By then the news of the latest Rails exploit had flooded the front page, and the Mezzanine release had been pushed down into oblivion.

At the same time, I posted the announcement to Reddit. It also reached the front page of proggit, the programming sub-reddit I posted it to. Again there were a good number of comments with praise and questions. What was interesting about Reddit is that the number of votes both for and against the article are visibly displayed.

Downvotes, really!!?

It was amusing to think about what would cause people to vote it down. Perhaps they were fanatical supporters of another language, framework or CMS. I’ll never know! I think it’s a safe bet though that it had nothing to do with Mezzanine itself, based on the actual comments posted there and elsewhere.

Twitter

Of course I tweeted the announcement once it was made. After a handful of direct and indirect retweets, the tweets started coming through with a lot of lovely praise, particularly for the responsive layout of Mezzanine’s new project homepage, something that all credit goes to the Bootstrap team for. The most humbling moment though was when a tweet came through from Antonio Rodriguez, the former CTO of HP of all people!

This looks like it may be Django’s killer app

Very cool.

After the announcement had hit the front pages of Hacker News and proggit, an army of Twitter bots that are connected to those sites then tweeted links to the announcement, resulting in hundreds of tweets and thousands of new visitors. The final assault was then triggered by Smashing Magazine, who tweeted Mezzanine to their half a million followers.

GitHub

While GitHub wasn’t somewhere I explicitly made the announcement to, like the channels above, the reaction on there was probably the most important. Mezzanine and Cartridge received over 100 new developers following the projects.

This resulted in Mezzanine being the most watched Python project on GitHub for both the day and the week! For the first time it also entered into the top 100 Python projects on GitHub of all time.

A quick burst of small contributions followed, from spelling corrections in the documentation, to patches for getting things running smoothly on Windows. I then released version 1.0.1, and 1.0 became history.

Final Toast

It’s been a hard slog over the last two years. With a full-time job and a family to take care of, the biggest challenge has always been finding the time to build new features and work with the community to ensure Mezzanine stays on the right track for its users. In that regard, Mezzanine wouldn’t be what it is today without the contribution of all the developers who have written features, fixed bugs, and most importantly, helped out new-comers on the mailing list. A special thanks goes out to all of you:

Lex Hider	Van Lindberg	Timur Bobrus
Toby White	Eric Floehr	Tom von Schwerdtner
Brad Montgomery	Andrew Fisher	Carlos David Marrero
Lee Matos	Josh de Blank	Dominique Guardiola Falco
Michał Oleniec	John Campbell	Andrew Grigorev
Audrey Roy	Josh Cartmell	Osiloke Emoekpere
Eduardo Gutierrez	Rich Atkinson	Brett Clouser
Brent Hoover	Owen Nelson	Zeke Harris
Ken Bolton	Eli Spizzichino	Michael Delaney
David Prusaczyk	Alexey Makarenya	Sebastián Magrí
Kevin Levenstein	Josh Batchelor	John Barham
Luke Plant	Zdeněk Softič	Alvin Mites
Jason Kowaleski	Nicola Larosa	Anders Hofstee
Chris Trengove	Chris Smith	Tommy Wolber

Here’s looking forward to the next two years, and keeping Mezzanine and Cartridge a lean, mean, site building machine.

My Baby Sinatra Apps

2012-02-27T00:00:00-08:00

It was almost a year ago that I took up a new role using Ruby on Rails. I’ve previously talked about my thoughts on Rails, and given my experience with Django I probably wouldn’t consider using Rails for my own projects. What I did explore in my time with Ruby was another framework called Sinatra which I used to build several apps. Firstly I’ll go over Sinatra and some of the related pieces in the stack, and then I’ll cover the apps I built.

Sinatra

Sinatra is a micro-framework, which differs from mega-frameworks like Rails and Django, in that Sinatra is bare-bones. It mostly deals with mapping URLs to request handlers, and not much more beyond that. No templating, no ORM, no middleware. All of these features can be slotted in using third party libraries where required. This makes for a very pleasant development experience with smaller sized apps - instead of having to do everything the Django/Rails way, you’re free to pick and choose the parts you need, and weave them together in the best way you see fit. You’re working at a relatively lower level, with much less scaffolding, and a lot more flexibility and control.

Python has its own counterparts in this space as well, such as Bottle and Flask. However at the time I was looking to dive further into Ruby, and Sinatra seemed like a great way to lean into it.

If you’re new to web development, or an experienced developer coming to Ruby or Python from older stacks like ASP.NET or PHP, I’d highly recommend starting out with a micro framework like Sinatra or Flask, before moving onto their bigger siblings Rails and Django. You’ll get a great feel for their respective languages, without getting bogged down in the frameworks themselves.

DataMapper

You can’t go very far these days developing a web application, without needing some form of persistent storage such as a database, and a library to work with it that goes beyond hand-written SQL. Django has its own ORM which is very powerful, but suffers from lacking a blessed, seamless migration tool. Rails has Active Record, which has grown into the defacto ORM in the Ruby eco-system, and has its own set of problems. The main issue I had with Active Record was that there was no clear definition of what fields a particular model implemented, aside from diving directly into the database itself. It coincides clearly with the notion that Rails contains too much magic. Compare this to Django’s declarative ORM, where each model’s class contains an explicit blueprint of which fields and methods the model implements. The value of this in quickly picking up a new code base is highly under-stated, if the popularity of Active Record is anything to go by.

I began looking for an ORM solution for Sinatra and quickly came across a project called DataMapper, and was incredibly pleased. Not only does it provide the same declarative style that Django’s ORM does, it goes above and beyond that with features that blow both Django’s ORM and Active Record out of the water.

Firstly it provides the ability to automatically migrate models. Arguments against magic aside, this is an amazing feature. Simply change the model, and the changes are migrated to the database when next instantiated.

Secondly, DataMapper completely eliminates the N+1 query problem. When iterating through data and accessing other models via related fields, DataMapper will query the database when the first relationship is accessed, retrieving all required data pertaining to the outer loop, building up all of the instance relationships on the fly prior to accessing them. Yes, it sees into the future and protects you from obliterating your database, a mistake all too common in web development. It’s worth noting that Django introduced this feature in the yet to be released 1.4 with the prefetch_related method. This is another great example of the explicitness of Python compared to the implicitness of Ruby.

Whether you’re building a web application or not, if you’re accessing a database from Ruby, consider using DataMapper. It’s a great piece of software.

Heroku

The explosion in Ruby and Python development frameworks over the last half decade has been a boon to web development. Security, modularity, shelf life, and time to market have all dramatically improved thanks to dynamic languages, the frameworks that have developed around them, and the open source communities that make them possible. It’s not all fun and games however. Deployment of these applications has grown considerably more complex. Gone are the days of using FTP to upload some PHP scripts to a server, and hitting refresh on the web page to test your changes. We now have to deal with a wide variety of deployment tasks, from reloading application processes, database migrations, dependency management and much more.

Naturally the community has risen to solve these problems, a movement sometimes referred to as DevOps, with tools in place such as Ruby’s Capistrano, Python’s Fabric, and configuration management tools to map complex deployments such as Chef and Puppet. While the learning curve is steep, with enough time invested up front, deployments can become as simple as pushing a button, and are more robust and integrated with quality assurance than ever before.

Modern deployments such as these require experts. This is where Platform as a Service (PaaS) offerings come in. PaaS providers are modern hosting companies that take care of all the dirty work in configuring servers and automating deployments for you. Typically they’ll expose a distributed version control server using Git or Mercurial that you can push code commits to. The process of pushing commits then triggers deployment, automatically performing all of the required tasks. PaaS providers will also provide all of the related services required, such as databases, message queues, caching servers and so forth.

Are PaaS providers a magic bullet? Absolutely not. Every application will have a tipping point where it grows beyond the “one size fits all” approach provided by PaaS offerings. You’re also at the mercy of the provider when it comes to uptime, so mission critical applications with strict service level agreements may require much more fine grained control in their hosting environments. Never forget that you are the only one responsible for your service’s availability. However for smaller, stock, non-critical applications, PaaS providers are a dream come true, that remove most of the complexities around service provisioning, configuration and application deployment.

The most well known of these providers is Heroku, who were the first to popularise the PaaS architecture. Not only does Heroku offer a rich variety of add-on services that I was looking for, like Varnish, Memcached, and PostgreSQL, they also provide free hosting for low capacity sites - ideal for the types of applications I ended up building with Sinatra.

So with Sinatra, DataMapper and Heroku combined, I developed several small applications that scratched particular itches for me, in order to build up a good working knowledge of Ruby.

Linked Out

I don’t keep an up to date CV anymore. If I pick up a new skill, or start a new role, I’ll update my LinkedIn profile. It’s the quickest and easiest way to keep my professional information up to date. For better or worse though, over the last few years LinkedIn has turned into a mass hunting ground for recruiters. I used take the time to enter into a dialogue with each and every recruiter that contacted me, after all anything less would be rude, but over time I realised the futility in this, as the practice by recruiters to blast out boilerplate introductions to anyone who matched a keyword search, became more and more common. But I digress. The state of recruitment aside, these conversations would inevitably lead to recruiters asking for a CV they could present to their clients. LinkedIn profiles contain a “download as PDF” feature, which I would always refer recruiters to, but LinkedIn embeds their logo within the PDF, and over time as they’ve added new profile features, the PDF download hasn’t picked these up. What I’d always wanted was an easy way to export my profile as a clean PDF, containing only the information relevant to a CV. I also wanted to be able to share the tools with anyone else who wanted to use it, so a baby Sinatra app seemed like the perfect fit.

There’s a great Ruby library for interacting with LinkedIn’s API, and PDFKit for converting HTML to PDF, which meant that I could format the CV using HTML and CSS, and all I then needed to do was convert that directly to PDF.

The app I ended up building is called Linked Out. It has a very simple flow to it. You first authenticate via LinkedIn’s oAuth service, and then you’re redirected back to Linked Out, where you can create a PDF version of your profile. As an added bonus I hooked into LinkedIn’s connections API, so you also get the option of creating a CV for any of your LinkedIn connections - no need for recruiters to bother people with CV requests, they can create the CVs themselves.

LinkedIn is somewhat lacking when it comes to formatting large blobs of text in profiles. People tend to create all sorts of formatting themselves, typically to create bulleted lists. So Linked Out contains some smarts to look for these different types of free-text formats, converting them into proper lists and headings where appropriate.

Need to fend off pesky recruiters with a nice looking CV? Go and update your LinkedIn profile, and give Linked Out a try.

Klout Feed

If you’re unfamiliar with Klout, it’s a reputation measurement system that gives you a daily score based on your online interactions. It looks at your Twitter account as well as other social media services, and applies an algorithm based on the number of mentions you receive, retweets, and favourites, also taking into account the Klout score for each of the people who trigger these. It then assigns you a score out of 100 which you can measure on a daily basis to gauge how effectively you’re using Twitter. According to Klout at least.

Klout has been described as many things, from a revolutionary game-changer in social media, to a vapid and narcissistic waste of time. Personally I found it to be an entertaining distraction. It did keep me coming back each day to check my score.

Now I’m one of a dying breed that still uses RSS exclusively to keep track of everything going on online. Updates from my LinkedIn connections, contributions and issues for my projects on GitHub, Google Groups mailing lists, and of course various news sites and blogs. They all contain RSS feeds which I can catch up on, in one single interface. But not Klout. You need to log into their site each time you want to check you score. There’s an interesting point here. If you want to keep users coming back to your site and piss them off at the same time, create a great service but make sure you don’t include an RSS feed for the data your users are interested in.

To solve this I put together an app called Klout Feed. It uses Klout’s API to provide an RSS feed for each user, publishing their score and its change each day. The flow isn’t as seamless as Linked Out’s. Klout doesn’t provide any form of application integration the same way LinkedIn, Twitter and Facebook do. Just a per-user API key with daily limits assigned to it. So with Klout Feed you first need to head on over to Klout and grab an API key, then bring that back to Klout Feed to get the URL for your score’s RSS feed.

Ian Anderson has since gone ahead and written a how-to article on combining Klout Feed with the If This Then That service. The result is that you can get email or SMS notifications each time your Klout score changes.

One True Repo

Most open source projects use Git or Mercurial for version control, and are hosted on either GitHub or Bitbucket respectively. Some people like myself host their projects on both sites. I’ve talked about my setup for hosting on GitHub and Bitbucket before. Both sites provide totals for the number of interested developers following the project, and the number who have forked the project. A fork is when someone creates a copy of a project, usually with the intent of adding some news features or fixes, and pushing them back to the original source.

What I’ve always wanted is a combined API for totaling followers and forks across both services for a single project hosted on both sites, and also for all projects for a given user on both sites. This is what I tackled for my next Ruby project, which I called One True Repo (OTR).

OTR’s original form was as a library that other developers could embed in their project, so I built it as a Ruby gem that you can both include in your own project, or simply run from the command line and pipe the data it returns into other programs. The next step was to build a baby Sinatra app that provided a hosted version of the API that people could query. The project itself contains everything for all three of these forms - the library, the command-line tool, and the Sinatra web app.

Querying the GitHub API was trivial and all the information I wanted was provided by it very easily. The Bitbucket API wasn’t quite up to scratch for this however. Remarkably it doesn’t expose follower and fork count on each project. Some screen-scraping was therefore required to get these totals for each of a user’s projects.

Do you mirror your open source projects across both GitHub and Bitbucket? Ever wonder how many people are following all your projects on both services? Give One True Repo a try.

Conclusion

As you can see from some of these apps, the Sinatra on Heroku combination is the perfect fit for small mashups that act as glue between other popular APIs. Free of charge, rapid development, and a great pool of libraries to choose from in the Ruby eco-system.

2011: My Year in Review

2012-01-10T00:00:00-08:00

Another year has gone by and another obligatory year in review post is due. 2011 was a year full of change for me. I changed jobs twice, spent half the year coding in a new language, moved interstate, and switched my primary operating system.

Ruby

After working at Citrus for almost nine years, I was well overdue for a change of scenery, so when I was approached by Impact Data to come and work for them using Ruby on Rails, I welcomed the opportunity. My time there was short and sweet however, as after six months I decided to move back home to Sydney, but it was a really rewarding experience working with an incredibly smart team on a technology stack that was new to me. I had a great time learning Ruby, forming a strong appreciation for its elegance, and it was very interesting along the way making plenty of comparisons between Rails and Django. Naturally I started using Ruby in my own projects, getting to know Sinatra for a handful of apps that I built. I’ll be writing a more detailed post about those soon, so stay tuned!

Django Dash

For the second year in a row I entered the Django Dash hackathon. This time around I wanted to do something that really pushed Django outside of its typical usage patterns. I’d recently read Cody Soyland’s introductory blog post on using WebSockets with Django, and so I came up with an idea I called Drawn By, a collaborative drawing app where people could create sketches together in real-time, save them to an image they can download, and rate others’ sketches in the gallery.

I got to use a variety of technology I hadn’t used before which was really fun. I used gevent as the evented web server for maintaining open socket connections with the browser, the NoSQL database Redis for queuing events and storing temporary pixel data, and the browser’s Canvas API for front-end drawing interaction and rendering. I set a relatively high goal for ourselves this year with what we wanted to achieve, but we pulled it off nicely in the end.

This year we came 3rd place out of around 50 entries, which was a great improvement on the previous year’s result of 8th. The most important result however was the creation of django-socketio, which was extracted from Drawn By and released as open source. It brings together all of the scaffolding for using WebSockets with Django, and implements an events and channels system for building your own applications around it. I previously wrote about django-socketio right after releasing it, and since then it has gained quite a lot of traction, with a handful of developers contributing back fixes and new features.

Open Source

I actually spent less time this year contributing to open source than I did the previous year, but I steamed ahead nonetheless with a lot of new projects, as well as continued development and support for my major works, Mezzanine and Cartridge. Both these projects have reached a very mature level over the course of 2011, thanks to tons of contributions from the Mezzanine and Cartridge community, which continues to grow steadily. Here’s a list of the projects I released as open source over the year:

Drawn By: Collaborative real-time sketching. (Django / Python)
django-socketio: WebSockets for Django. (Django / Python)
Virtualboxing: Comparison utilities for Riak and MongoDB. (Ruby)
Grillode: Multi-purpose chat server. (Node / CoffeeScript)
Linked Out: Create PDF resumes for LinkedIn. (Sinatra / Ruby)
Klout Feed: Daily Klout scores via RSS. (Sinatra / Ruby)
Babbler: A Twitter bot. (Python)
One True Repo: Combined GitHub and Bitbucket API. (Sinatra / Ruby)
hg-github: A Mercurial extension for GitHub. (Mercurial / Python)
sphinx-me: A Sphinx documentation generator. (Python)

OSX

As I mentioned, towards the end of the year I moved back to Sydney. An opportunity came up to work with Fairfax, the largest media organisation in Australia. Fairfax is building a new publishing platform using Django, so my experience with content management and Django was a natural fit. Mostly though, it was a chance for me to move back home and be closer to my family, after being away from them in Melbourne for a decade.

When I started at Fairfax, I was surprised to find the entire team running OSX. I wasn’t surprised so much by the choice itself, as OSX is very popular in the Django and wider development community, but more so by my own lack of experience with it, having solely used Linux for the last half decade. I decided to give it a go and found it to be on par with Linux as a development environment.

My biggest gripe was breaking down the mental muscle I’d built up around the shortcut keys for wrangling text. At first I was fumbling on OSX, but after a few days of using OSX during the day and Linux at night, I found that I wasn’t efficient with either of them - I needed consistency. I was also long overdue for a new machine. I originally had my eye on some of the MacBook Air clones like the Acer UltraBook and Asus ZenBook, but I couldn’t find any information about running Linux on these, and I wasn’t prepared to go through the pain of working it out if anything went wrong. So I bit the bullet and picked up a 13 inch MacBook Air.

All I need at the software level is visible application shortcuts, keyboard-driven application switching, a decent editor, terminal, web browser and package manager, and I’m good to go. In that regard, OSX and Linux are equivalent for my use, each with their own minor flaws. What I am really loving is the hardware. The keyboard seems laid out in a way that lets me type faster than ever, and the solid state drive means everything is instantaneous - that paired with the best battery life I’ve ever experienced, and it’s a dream machine.

Announcing hg-github

2011-12-31T00:00:00-08:00

The two front-runners in source code management these days are undoubtedly Git and Mercurial. Distributed version control has clearly proven itself as the superior model over older centralised systems like SVN, particularly in the context of open source development, where the ability to fork repositories and push and pull branches between them facilitates a much more efficient and streamlined work-flow.

My personal opinion about the difference between the two is similar to my take on Django and Rails. Relative to their alternatives, the two are far more similar than different, essentially providing the same concepts and features, but with quite different underlying philosophies in their implementations. For this reason I choose the less popular Mercurial over Git to manage all of my projects. Firstly I feel it has a more natural and intuitive UI (I’m not referring to a graphical interface here, but the actual commands it implements and their arguments). Another key factor is that Mercurial is written in Python, which means hacking on it and building extensions for it is a breeze.

What makes Git more popular than Mercurial? One major factor is their respective online hosting services, GitHub for Git and Bitbucket for Mercurial. If you’re unfamiliar with these sites, they’re like Facebook for programmers who share projects and collaborate together on code rather than post photos and videos. While both of these sites mostly implement the same core features, Github has always been several steps ahead of Bitbucket in terms of overall polish, and being first to market with new features. This difference has led GitHub to become far more popular than its Mercurial counterpart, and as a result, Git far more popular than Mercurial.

With the majority of open source activity occurring on GitHub, what then is a Mercurial user to do? Limiting your projects to the audience of Bitbucket means missing out on a lot of potential collaboration. Fortunately some time ago, the team at GitHub developed a Mercurial extension called hg-git. It allows you to transfer code back and forth between a Mercurial repository on your machine to a Git repository on another machine, like GitHub for example, taking care of all the translation required between Git and Mercurial. After several years of using hg-git, it’s one of the only pieces of software that continues to amaze me. Think of Google’s language translator, which on a good day can provide some very quirky translations when converting text from one language to another. hg-git performs the same task, but has no room for error when translating a source code repository from Mercurial to Git, and back again. Admittedly hg-git has a much easier job to do than translating human languages with all their ambiguities. Still, I am constantly impressed by the task it performs.

So hg-git allows me to develop my projects using Mercurial and have them shared on both Github and Bitbucket, allowing for maximum collaboration which is fantastic. It’s not entirely seamless however. I still need to perform a couple of manual steps such as adding GitHub paths to my repo configuration, and creating Mercurial tags that map to the Git branches I want to work with. I need to do this each time I set up a new repository, be it for starting a new project, or forking a project of my own or someone else’s. Recently I had to do this about half a dozen times in the space of an hour while working on a few different projects, and I thought to myself that I should be able to automate it. The result is a Mercurial extension I’ve called hg-github which automatically takes care of these manual steps required. It also wraps hg-git, so you don’t need to install both extensions, as hg-github pulls in and takes care of all the hard work that hg-git does.

Overview

Once hg-github is installed, assuming the default remote location of your repository is on Bitbucket, the GitHub path is automatically added and given the name github, so you can push to it with the following command:

$ hg push github

For other named Bitbucket locations, the name github-NAME is given, where NAME is the name of the path located on BitBucket. For example consider the following .hg/hgrc repo config:

[paths]
default = ssh://hg@bitbucket.org/stephenmcd/hg-github
somefork = ssh://hg@bitbucket.org/stephenmcd/hg-github-temp

hg-git will add entries to the config file as follows. Note that the config file isn’t actually written to:

[paths]
default = ssh://hg@bitbucket.org/stephenmcd/hg-github
somefork = ssh://hg@bitbucket.org/stephenmcd/hg-github-temp

github = git+ssh://git@github.com/stephenmcd/hg-github.git
github-somefork = git+ssh://git@github.com/stephenmcd/hg-github-temp.git

hg-github assumes you have the same username on GitHub and Bitbucket. If you have a different GitHub username, you can specify it by adding the following section to your global .hgrc file. For example my GitHub username is stephenmcd:

[github]
username = stephenmcd

Django Model Field Injection

2011-11-10T00:00:00-08:00

As a creator and maintainer of several popular reusable Django applications, one of the most commonly requested features I’m asked for is the ability to customise the fields that a model implements. This topic comes up often on the Mezzanine mailing list, and during this particular thread we researched ways that fields could be dynamically injected into models at run-time.

Other Approaches

It’s worth taking a look at other approaches to the general problem, and what their drawbacks are, in order to provide context for what the final solution needs to achieve.

One approach is to implement as many of the model classes as possible as abstract base classes, so that users can subclass these with their own models. This approach makes sense for certain types of customisation, and it’s what I’ve done with django-forms-builder for example. Some caveats exist with this approach however. Firstly, relationship fields can’t be defined on the abstract models, so these need to be implemented in concrete models either within the same app, or by the user implementing their own subclasses. Secondly, any functionality that references your models, such as views or middleware, needs to either have configurable settings for choosing which models to use, or be reimplemented entirely by the user to make use of their custom fields.

Another approach is to simply recommend that users subclass the models that the app provides using multi-table inheritance. Unfortunately this will introduce unnecessary overhead with the extra database queries required when accessing the instances of the subclasses. Best case is that this amounts to an extra query or two in a view dealing with a single instance. Worst case is that when this approach is used with a queryset in a template, an extra query is performed for each instance returned - the classic N+1 query problem.

Dynamic Injection

The ideal approach would allow users to directly modify models with their own code, outside of the models’ apps, without the models themselves having to implement any special hooks for customisation. The end result being an optimal database design, with no extra API requirements for the relevant models. It just so happens that this is possible by using several features that Django exposes, and combining them together in a particular way.

The approach boils down to three concepts:

Dynamically adding fields to model classes
Ensuring Django’s model system respects the new fields
Getting the load ordering correct for the above to work

Django’s model fields provide an undocumented contribute_to_class method. This method serves as a fancy version of setattr and takes a value and attribute name to use as arguments. Internally it then takes care of all the house-keeping required for a field to be added to a model.

The other feature of Django we’ll use is the class_prepared signal. This signal is emitted each time a model class is declared and loaded for the first time by Django’s model system.

from django.db.models import CharField
from django.db.models.signals import class_prepared

def add_field(sender, **kwargs):
    """
    class_prepared signal handler that checks for the model named
    MyModel as the sender, and adds a CharField
    to it.
    """
    if sender.__name__ == "MyModel":
        field = CharField("New field", max_length=100)
        field.contribute_to_class(sender, "new_field")

class_prepared.connect(add_field)

The final consideration is connecting the class_prepared signal at the correct time. It needs to occur prior to the relevant model class being declared, otherwise the signal will never be triggered when we want it to. A general way of achieving this is to connect the signal from within an app that is listed before the app containing the relevant model, in the INSTALLED_APPS setting. Note that in the above code, we don’t explicitly import the model to use it as the signal’s sender, instead checking for the model’s class name, as importing it would break these load ordering requirements.

Caveats

Like the previously described approaches, dynamic injection also comes with a set of drawbacks. These drawbacks stem from the fact that the apps containing the models being customised don’t contain a definition for the fields being injected. This means that migration tools likes South are unable to detect the new fields, and workarounds such as creating manual migrations are required.

Another related problem is when new admin classes containing references to the custom fields are registered and the fields haven’t yet been injected. A typical requirement for injected fields is to expose them via Django’s admin interface, which can be achieved by unregistering existing admin classes for the models that fields are being injected into, subclassing these admin classes with new references to the injected fields, and registering the new admin classes. Unfortunately if this unregister/register dance occurs in an admin module, the fields may not have yet been injected. A quick work-around for this is to perform the unregister/register calls inside your project’s urlconf.

Mezzanine Support

Drawbacks aside, the field injection technique described above has characteristics that make it incredibly useful. As such the approach has first-class support in Mezzanine by way of the EXTRA_MODEL_FIELDS setting. This setting allows you to define a sequence of all the custom fields you’d like to inject. Each item in the sequence contains four items: the dotted Python path to the model (including the field name to use for injection), the dotted Python path to the field class to use for the injected field, a sequence of the field’s position arguments, and finally a dict of its keyword arguments.

EXTRA_MODEL_FIELDS = (
    # Add a custom image field from the fictitious somelib.fields module
    # to Mezzanine's BlogPost model:
    (
        # Dotted path to field.
        "mezzanine.blog.models.BlogPost.image",
        # Dotted path to field class.
        "somelib.fields.ImageField",
        # Positional args for field class.
        ("Image",),
        # Keyword args for field class.
        {"blank": True, "upload_to": "blog"},
    ),
    # Example of adding a field to *all* of Mezzanine's content types:
    (
        "mezzanine.pages.models.Page.another_field",
        "IntegerField", # 'django.db.models.' is implied if path is omitted.
        ("Another name",),
        {"blank": True, "default": 1},
    ),
)

Mezzanine then uses this setting to inject all of the fields defined, using class_prepared and contribute_to_class as described earlier. It handles getting load order correct by performing the injection within the mezzanine.boot app, which is forced to the front of all apps defined in INSTALLED_APPS. Django’s admin is also patched in the boot app, to defer certain calls to unregister and register, to correct the ordering issues described earlier.

Open Source for You

2011-09-12T00:00:00-07:00

Update, Feb 14: @rkJun has translated this post into Korean.

I recently gave a presentational talk to the development team at my work, on what it’s like contributing to open source. A common perception by people not involved is that the open source community seems like one big socialist hippie commune, whose motivations are entirely altruistic with little reward for the individual. The approach I took was to dispel this myth and show that for many who participate, their motivations are mostly self serving, with many personal gains to be made. Despite some of the awkward truths I focused on, the talk was received really well.

Your Day Job Sucks, Make Programming Fun Again

If you have a regular 9 to 5 job as a software engineer at a typical company, chances are likely that your job sucks in several ways. You work on the same types of projects every day. You work with the same technology every day, and most likely it’s dated. There are more modern technology stacks out there that are much more elegant, yet for a variety of non-technical reasons, you’re not allowed to use them at work. You also have deadlines. Deadlines lead to knocking out features as quickly as possible, which directly conflicts with the quality of your work, so you’re often forced to take ugly shortcuts, leaving little to be proud of.

If this is your only exposure to programming, it gets boring pretty quickly and isn’t much fun at all. Contrary to this, programming can actually be incredibly fun! Imagine your company had hundreds of different projects to choose from, and let you choose whichever one you wanted to work on. Then on top of that, they told you that you could use whichever technologies you wanted to, and that you could take as long as you needed to build it correctly - so long as the end result was the most well designed and beautifully written software you’ve ever produced. As a professional programmer, that sounds like a dream, but that’s exactly how it is when you work on open source projects. Choose your problem domain. Choose your technology stack. Choose your pace. Choose your quality.

Become a Better Developer through Collaboration

There are two ways to become a better developer: writing code and reading code. This clearly doesn’t make sense in a silo on your own. There’s little to learn from reading your own code beyond reminiscent value, and you could write code on your own for years and the only lessons learned would be those you teach yourself. Naturally the real learning is to be had from interacting with other developers, writing code that they can review and give you feedback on, and reading code written by other developers with more experience and different approaches than you have.

Drawing back to your day job, how many developers do you collaborate directly with. A few? A dozen? A small pool that will give you much to learn from, but is still relatively limited in scope. The open source community is made up of thousands of developers, who over many years have produced millions of lines of code. Code of quality much greater than you’d ever be exposed to without stepping outside of your workplace.

Acquire Cash Money

One of the biggest misconceptions abut open source development is that it’s a financially fruitless labour. While true to a certain extent, there are actually several ways in which it can pay a monetary return, both indirectly and directly.

You may work a steady, full-time job, or you’re looking for one, and think of yourself as an employee with a fixed salary, in total contrast to those wild freelancers and entrepreneurs who don’t know where their next meal will come from. Wrong. You are a business, just like your employer is. You have one customer, your employer. Sometimes you might feel as though you’re stuck in your day job, when in actual fact you’re simply stuck with one client.

So how can you gain more clients? How can you gain a better client? By promoting your business of course. Open source provides a great way for you to showcase you and your ability as a developer. Your open source contributions can distinguish your CV from other developers, namely, your business’s competition. Graphic designers have always had the luxury of being able to display their portfolios, and developers can too. I’ve been on both sides of the hiring table, and when choosing between potential hires, someone with significant open source contributions has already demonstrated a clear passion for what they do, and therefore has a major advantage over those who are tasked with convincing people of the same, without having anything to show for it.

Getting your open source projects to the level of quality and usefulness where people are actually building production systems with them is huge achievement in itself. Stemming from that are direct opportunities for paid work. Your corporate users may need your software customised for their use cases. You may even be lucky enough that they’d like features developed into the project that are great ideas - things you might not have thought of, and will take the project to the next level. Who better to develop these than the creator of the software themselves? I’ve had several instances where this has occurred, and companies have sponsored development on features that, had I had the insight to think of, would have developed freely on my own anyway.

Broaden Non-Technical Experience

Hands on coding is only one small aspect of software development. There are many other disciplines that make up the process of shipping a successful product. Project management, community building, formal writing, product development and marketing. While freelancers and one-man commercial studios may get the benefit of dipping their toes into these different areas, as a developer at a typical company your role often doesn’t take you outside of the day to day coding.

Running an open source project properly will allow you to skill up in broad range of non-technical areas. I spend a lot of my time working on my most popular project, however these days only a small portion of this time is actually spent coding. Most of the time is spent in discussion with other developers on their feature ideas, coordinating the inclusion or exclusion of these, and providing support for new users. It’s hard work, but it’s a great and relevant experience.

Corollary to this, many non-technical people who’d like to contribute to open source in some way, believe that they don’t have anything worthwhile to offer. With all that’s involved, some of the most beneficial contributions are those of a non-technical nature. If this describes you, then please don’t hesitate to dive in. Your help means more than you can imagine.

Feels Good Man

The most obvious yet least tangible benefit in contributing to open source, is the idea I first alluded to around altruism and giving to others. As I mentioned, coordinating a non-trivial open source project can be hard work. It has its highs and lows, and sometimes it feels like there’s more of the latter. However every now and then, I’ll get a personal email of praise and gratitude. Not in any public forum where the soapbox can be a conduit for ulterior motives, just a private and genuine expression of thanks. I would never have thought of this as being particularly rewarding, until I actually experienced it. It really fills you with a true sense of pride and satisfaction, which makes it all worth while in the end.

Get Started Now

The Chinese philosopher Lao Tzu said:

A journey of a thousand miles begins with a single step

Getting started might seem like a daunting task. Start small. Ignore the inner voice that whispers in your ear, arguing that your open source idea wouldn’t be of any use to anyone. That might even be true, but it doesn’t matter. Do it anyway. A lot of my projects are only a few dozen lines of code. People have found them useful in ways I couldn’t have imagined.

Ideas can be hard to come by. Most of mine were itches I wanted to scratch. Product gaps in the technology stacks I’m interested in, or mundane processes I went through regularly, upon realising they could be automated and productised.

Perhaps you’re an end user of open source software, and it isn’t perfect. A feature works differently than how you think it should, or something is undocumented and you struggled to find it. These days it seems that the modus operandi is to throw up your arms, and complain as loudly as possible about how the software you’re using for free, isn’t exactly the way you want it to be. Don’t. You have the choice to do something about it. Dive into the source code. Contribute something back. Improve yourself, and be a part of something.

Real-time Web Apps with Django and WebSockets

2011-08-13T00:00:00-07:00

I recently took part in Django Dash, the annual hackathon where teams of up to three compete to build the best Django project they can within 48 hours. This year I worked with Travis White and Josh de Blank to build DrawnBy - a collaborative drawing application that allows people to sketch together in real-time.

DrawnBy makes extensive use of WebSockets which are mostly unheard of in web stacks like Django and Rails, or even antiquated stacks like PHP and ASP.NET, which are all designed around accepting a request and returning a response to the browser as quickly as possible. This is in contrast to WebSockets which allow full duplex communication between the browser and server, and therefore require long running requests per user.

A variety of patterns for dealing with WebSockets in Django emerged while developing DrawnBy, and since the Dash I’ve been working on abstracting these into a reusable Django application called django-socketio which I’ve released today. It’s available on Github, Bitbucket and PyPI.

Here’s an overview of the features.

Features

Installation of required packages from PyPI
A management command for running gevent’s pywsgi server with auto-reloading capabilities
A channel subscription and broadcast system that extends Socket.IO allowing WebSockets and events to be partitioned into separate concerns
A signals-like event system that abstracts away the various stages of a Socket.IO request
The required views, urlpatterns, templatetags and tests for all the above

Channels

The WebSocket implemented by gevent-websocket provides two methods for sending data to other clients, socket.send which sends data to the given socket instance, and socket.broadcast which sends data to all socket instances other than itself.

A common requirement for WebSocket based applications is to divide communications up into separate channels. For example a chat site may have multiple chat rooms and rather than using broadcast which would send a chat message to all chat rooms, each room would need a reference to each of the connected sockets so that send can be called on each socket when a new message arrives for that room.

django-socketio extends Socket.IO both on the client and server to provide channels that can be subscribed and broadcast to.

To subscribe to a channel client-side in JavaScript use the socket.subscribe method:

var socket = new io.Socket();
socket.connect();
socket.on(connect, function() {
    socket.subscribe(my channel);
});

Once the socket is subscribed to a channel, you can then broadcast to the channel server-side in Python using the socket.broadcast_channel method:

socket.broadcast_channel("my message")

Events

The django_socketio.events module provides a handful of events that can be subscribed to, very much like connecting receiver functions to Django signals. Each of these events are raised throughout the relevant stages of a Socket.IO request.

Events are subscribed to by applying each event as a decorator to your event handler functions:

from django_socketio.events import on_message

@on_message
def my_message_handler(request, socket, context, message):
    ...

Each event handler takes at least three arguments: the current Django request, the Socket.IO socket the event occurred for, and a context, which is simply a dictionary that can be used to persist variables across all events throughout the life-cycle of a single WebSocket connection.

on_connect - occurs once when the WebSocket connection is first established.
on_message - occurs every time data is sent to the WebSocket. Takes an extra message argument which contains the data sent.
on_subscribe - occurs when a channel is subscribed to. Takes an extra channel argument which contains the channel subscribed to.
on_unsubscribe - occurs when a channel is unsubscribed from. Takes an extra channel argument which contains the channel unsubscribed from.
on_error - occurs when an error is raised. Takes an extra exception argument which contains the exception for the error.
on_disconnect - occurs once when the WebSocket disconnects.
on_finish - occurs once when the Socket.IO request is finished.

Like Django signals, event handlers can be defined anywhere so long as they end up being imported. Consider adding them to their own module that gets imported by your urlconf, or even adding them to your views module since they’re conceptually similar to views.

Binding Events to Channels

All events other than the on_connect event can also be bound to particular channels by passing a channel argument to the event decorator. The channel argument can contain a regular expression pattern used to match again multiple channels of similar function.

For example, suppose you implemented a chat site with multiple rooms. WebSockets would be the basis for users communicating within each chat room, however you may want to use them elsewhere throughout the site for different purposes, perhaps for a real-time admin dashboard. In this case there would be two distinct WebSocket uses, with the chat rooms each requiring their own individual channels.

Suppose each chat room user subscribes to a channel client-side using the room’s ID:

var socket = new io.Socket();
var roomID = 42;
socket.connect();
socket.on(connect, function() {
    socket.subscribe(room- + roomID);
});

Then server-side the different message handlers are bound to each type of channel:

@on_message(channel="dashboard")
def my_dashboard_handler(request, socket, context, message):
    ...

@on_message(channel="^room-")
def my_chat_handler(request, socket, context, message):
    ...

Chat Demo

The “hello world” of WebSocket applications is naturally the chat room. As such django-socketio comes with a demo chat application that provides examples of the different events and channel features available.

Rails Quick Start for Djangonauts

2011-07-30T00:00:00-07:00

I recently started a new role at a Ruby on Rails shop, which as a long time Django specialist was a really interesting opportunity. There’s a lot of competition between the two frameworks’ communities, ranging from friendly rivalry and respectful admiration at the mature end of the scale, to all out fanboy fuelled flame-wars at the other.

After you wade through the rivalry, the common wisdom voiced is that they’re conceptually the same. If you know Python, go with Django and if you know Ruby, go with Rails. After spending several months with Rails I can attest to this being true. At a bird’s-eye view both frameworks contain almost identical concepts, implemented with different philosophies stemming from the ideals expressed by the languages they’re written in. Both frameworks provide a vastly superior approach to security, modularity and rapid development than their predecessors do.

An interesting question to ask would be: which would be the best framework to choose, not knowing either language? It would be naive of me to believe I am unbiased, but I would certainly recommend Django over Rails. The relative strictness of Python and explicitness of each component in Django, compared to the implicit magic in Rails, is simply much more geared towards creating large-scale systems in a sane and transparent way. To Ruby’s credit though, I have developed a real admiration for the language itself, and have continued using it in my own projects - but that’s a topic for another post.

Considering how similar the two frameworks are component-wise, one thing I did miss was a side-by-side cheat sheet for working out what each of the concepts were in Rails that I already knew well in Django. I’ve put one together below to help out anyone who might be picking up either framework while already knowing the other. For clarity, I’ve also included descriptions of each type of component, for those who haven’t used either framework.

Django	Ruby on Rails
URL patterns	Routes	Regular expression definitions for each type of URL and what part of the web site they map to.
Views	Controllers	The units of code that the above regular expressions map to, that perform application logic and pass data to a rendering layer.
Templates	Views	The rendering layer that is given data from the code described above, and performs display logic typically wrapped around HTML code.
Template tags (built-in)	Embedded Ruby	The flow-control language that can be used in the rendering layer.
Template tags (custom)	Helpers	The system for defining custom functions that can be used in the rendering layer.
Models	Models	The data-definition layer that maps classes to database tables - the ORM.
Managers	Scopes	The way to extend the ORM to define custom database queries.
Management commands	Rake tasks	Scripts for performing administrative tasks via the command line.
Project	App	An entire application built with the framework.
App	Plugin	The way in which all components in the framework can grouped together in separate areas of functionality.
South (third-party)	Migrations	The system used for automatically applying changes in the ORM definition to the underlying database tables, such as adding and removing columns.
Admin	RailsAdmin (third-party)	A web-based interface for authenticating administrative users and providing CRUD tools for managing data.
The following table lists software that aren't part of Django or Rails, but are core parts of the Python and Ruby eco-systems, and go hand-in-hand with using either framework.
Python	Ruby
Virtualenv	RVM	The system used for running isolated environments bound to a particular language version, combined with a set of install libraries.
PIP	Bundler	The package manager for installing libraries.
WSGI	Rack	A standard specification for applications to interface with HTTP, allowing for a single application entry point and middleware to be implemented.
Fabric	Capistrano	A system for automating tasks on remote servers from a local machine, typically as part of a deployment process.

Not all of these pairings are a perfect one-to-one match conceptually, but should be good enough to get an overall view of what each concept is within both frameworks.

Does Mezzanine Scale?

2011-06-16T00:00:00-07:00

The question of scalability with regard to Mezzanine and Django recently came on up on the mezzanine-users mailing list, to which I offered the following reply.

Mezzanine and Django itself are fantastic choices for someone concerned with scaling for high traffic.

The Mozilla add-ons site that hosts all Firefox plugins, and Disqus which is currently the world’s highest traffic commenting system, both run on Django. Each of these have been quoted at 500 million hits per day and 1 billion per month respectively.

One of the keys to scaling sites like these is the wealth of options available for caching in the Django ecosystem, and the ability to then scale out the number of both application servers running Django, and caching servers typically running memcached, with very little modification to your application code. Django comes out-of-the-box with the ability to switch on site-wide caching, per page caching and template fragment caching. Beyond that there are also third-party Django applications that implement object level caching such as Django Cache Machine and Johnny Cache, for even finer-grained control with little modification to your code.

Both Mezzanine and Cartridge have been designed from the ground up with scalability in mind. Particular care has been taken to avoid any n+1 queries, for example rendering out multiple instances of Mezzanine’s navigation tree containing any number of nested levels of navigation will only ever run a single database query. Same with Cartridge’s products. What this means is that you can go very far traffic- wise, using a single server without even thinking about caching. Once you do then the ability to add application and cache servers is trivial, and will take you incredibly far using a single database. Once you start needing multiple database servers Django also comes with the built-in ability to route models across different databases, so there are extra options there beyond your typical master/slave database replication scenario.

I’ve only touched on the topic to provide an overview of what’s available, and you should certainly research it further for your particular scenario, but as you can see scalability is a core concern baked into Django and Mezzanine, so you can choose these with confidence.

Mezzanine and Cartridge Hit the Mainstream

2011-03-21T00:00:00-07:00

Mezzanine and Cartridge Hit the Mainstream

This week I had the fantastic opportunity to be interviewed by Rodney Gedda from IDG for their TechWorld and CIO magazines. The interview focused on my open source work with the Mezzanine and Cartridge projects.

Melbourne hacker releases open source Django apps

These projects have been gaining traction at an incredible pace, with a solid community forming around them over the last year. All of this has happened in spite of having no formal promotion outside of the community itself, which makes this type of exposure all the more exciting.

It’s also no secret that I think the Australian web development market is many years behind the United States, so it’s a great step forward to see modern technology getting covered by the mainstream local tech media.

How I Now Know Node

2011-02-01T00:00:00-08:00

Having remained utterly faithful to Python and Django over the last year, and dedicating all of my available time to open source projects like Mezzanine and Cartridge, with the new year at hand I though it was about time to take a break and add some new technology to my repertoire. During his keynote speech at Djangocon.eu 2010, the creator of Django Jacob Kaplan-Moss states “it will challenge what you think you know about web/server architecture” when referring to Node.js. Since then it has been sitting in the back of my mind as something I definitely needed to check out, so I decided to dive in head first.

Node.js is a general purpose JavaScript development environment geared towards writing network servers. It uses an event-based, non-blocking architecture which allows your web application to scale to thousands of concurrent connections without needing a finite pool of threads or processes. JavaScript is executed using Google’s V8 engine which ranks very highly in speed compared to other dynamic languages, so not only does Node.js scale elegantly, it’s damn fast.

Having recently built Grillo, a console based chat server, I’d been considering what it would be like to put together a web-based version. In fact I had achieved something similar in the past using Python’s BaseHttpServer module, and while functional for a few hundred connections, my approach would never scale, as either a separate thread or process would be required for each open connection. The event driven architecture of scaling a web server for an increasing number of open connections is mostly a solved problem, especially in the Python community with projects like Twisted and Tornado. However Node.js is different in that its non-blocking evented model is a first class citizen by design.

With a useful project at hand to try Node.js on, I set about creating what I’ve named Grillode (yes you guessed it: Grillo + Node). It’s a web-based chat server with a set of configuration options that lets you run it in various modes, such as a customer support queue, or with Chatroulette style random match- ups. I’ve released the source onto Github and Bitbucket, and also have a demo up and running.

The process of putting Grillode together led me through many parts of the ecosystem that has developed around Node.js - following is an overview of the pieces I ended up working with.

Node Package Manager (NPM)

NPM is a command line utility that gives you access to a central online repository of packages built for Node.js. It works wonderfully when installed correctly, but on my machine I encountered a handful of issues where it ended up recreating various system directories all throughout my home directory. After setting up various symlinks by hand, I did get it to work after many failed attempts at installing it. This issue was definitely specific to my machine as I was then able to go ahead and install NPM seamlessly on several different servers.

Once everything was working correctly it made deployment of Grillode a breeze. By specifying all of its dependencies in a package.json file, NPM was able to installed everything required in a single step.

Express

Express provides basic URL routing to functions that will typically perform some application logic and hand off data to a template to be rendered. It contains integration points for a handful of different templating libraries and it also contains a simple middleware system. I’d definitely consider it to be a micro framework, but it’s a great start at whipping your Node.js application into a well defined structure.

Socket.io

Socket.io takes all the leg work out of maintaining an open connection to the browser. You start by attaching it to your Node.js server which then automatically makes available the client-side JavaScript required. This provides the communication channel between the client and the server, which attempts to use web sockets when available, and transparently falls back to Flash sockets or even old-school AJAX polling if the former options aren’t supported by the browser.

It then provides all of the methods and event handlers for connecting and sending data over the connection. The beauty behind how this is implemented is that it exposes these methods and events almost identically to both the Node.js server, and the browser client - instantly you have available two-way communication between the browser and the server via an open connection, without requiring any new requests to the server.

CoffeeScript

As many others have done, I’ve often compared JavaScript to Python in that they both share an object model defined by a hash table of names and object members, which can be introspected and dynamically modified. While this is a very elegant model, JavaScript boasts syntax reminiscant of the turn of the century, cluttered with semicolons and braces, and missing a handful of features found in modern languages such as list comprehensions and much more. Well the war on semicolons is over with the explosion in popularity of languages such as Python and Ruby showing this to be true.

CoffeeScript is a language inspired by Python and Ruby which gets compiled directly into JavaScript. It therefore retains all the properties of JavaScript such as its data types, objects and methods, but provides a much more modern and clean syntax with some fantastic sugar, such as list comprehensions, class-based object construction, string interpolation and more.

Experimentally, CoffeeScript can be run directly in the browser in place of JavaScript by including the compiler JavaScript file itself, but much more interestingly it can be used as an execution environment for Node.js, which will perform the compilation to JavaScript when the Node.js application is first started. I found my experience to match reports of up to 30% in reduction of the amount of code required.

Coffeekup

When looking into what was available for templating, I started out with EJS which basically gives you executable JavaScript within your template files. I found this to provide a poor separation between display and application code, something that Django gets right by providing a limited template language that has basic flow control, but makes writing non-trivial logic difficult to do.

I then discovered a Node.js template library called Coffeekup, that tied in very closely with the time I had already spent with CoffeeScript. Coffeekup allows you to define your HTML entirely in CoffeeScript. I’m still undecided on whether this is a thing of beauty or horror. It’s very surreal to work with web page markup expressed entirely in programming code. I guess there’s somewhat of an undeserved feeling of the presentation being too closely tied to programming logic. There is a magical feeling in having your server code, client code and presentation code all in the exact same language, coupled with the given language being CoffeeScript which is incredibly clean.

Conclusion

I really enjoyed working with Node.js and the young ecosystem surrounding it. At this point in time, I wouldn’t consider it for a typical project over a full stack framework like Django with the elegance of Python, however it definitely serves as a fantastic choice for a very specific criteria - scalable, real time web applications.

Grillo, a Terminal Based Chat Server and Client

2011-01-03T00:00:00-08:00

I recently came across the Rosetta Code Project. It’s a community contributed wiki that contains hundreds of solutions to programming problems, implemented in hundreds of different programming languages, which is great source of entertainment for a programming languages enthusiast such as myself. The main focus of the project isn’t to demonstrate individual solutions on their own, but to provide comparisons between different programming languages and how they approach the same task.

The wiki also contains various dynamic reports, such as which tasks have yet to be implemented for each particular programming language. I took a look at the Python page to see if there were any interesting tasks remaining that I could potentially provide a solution to, and as I expected there were only a few relatively obscure tasks that were yet to have solutions provided.

One task remaining for Python that did catch my eye was to demonstrate a simple chat server using sockets. I’ve always been especially fond of network programming, from web crawlers to XMLRPC to lower level sockets, I seem to really enjoy writing code that runs over the Internet without necessarily being related to web development, so I went ahead and added a Python solution for the chat server task.

I honestly had so much fun working on this that I decided to extend it even further. Firstly I refactored it into a more object oriented approach which allowed for creating both server and client tools which could be run concurrently using separate threads of control. I then added several commands that could be run directly in chat by any user, for example listing the current users who were logged in.

I then decided to publish the code for my chat server and client onto both GitHub and Bitbucket, after giving it the name “Grillo”. It’s named after the Italian phone of the same name, developed in 1965. They both share the common theme of being a very small communications device for their class, while being implemented with relatively basic technology.

Given that there are many richer and more powerful applications available for implementing the features Grillo provides, at the least it serves as a good example of how to do basic socket programming in Python, as well as a demonstrating some simple tricks for controlling threads. At the best case someone will pick up the code base and extend it further in ways I haven’t anticipated - here’s hoping!

2010: My Year in Review

2011-01-01T00:00:00-08:00

2010 was an amazing year for me professionally. I learnt and achieved much more than I’ve historically packed into a single year, so like many other I’ve decided to jump on the year in review bandwagon and put this post together.

Open Source

Prior to 2010 I’d certainly used open source quite heavily, in fact I’d based my entire career as a developer around it, focusing on Python, Django and Linux and the open source ecosystem surrounding this amazing platform. However as far as giving back to the community went, I had only ever contributed small bug fixes and enhancements to a handful of open source projects over the years, and had never been heavily involved in any single project. So in 2010 I dived in head first launching a range of open source projects. Firstly some smaller utilities such as django-forms-builder and gunicorn-console, and then the Django content management platform Mezzanine, along with its shopping cart plug-in, Cartridge.

The reception Mezzanine has received has been nothing short of amazing and well beyond anything I had anticipated. It’s been an incredibly rewarding learning experience, managing product development and working closely with its contributors towards growing its community. Here are a few stats for the project only 6 months since it was launched:

120 followers and 30 forks across GitHub and Bitbucket
Code contributions from 10 developers
A mailing list with over 60 members and over 300 messages
Over 4,000 downloads from the Python Package Index
Almost 9,000 visitors to the website with over a 50% return rate

Team Leadership

Shortly prior to 2010 I moved into the role of development team lead and 2010 provided me with a wealth of new experience in this regard. Being responsible for a development team producing rock solid work has always been an aspiration of mine and it was incredibly fulfilling to have that come to fruition. It’s been a fairly painless experience due to working with some of the best developers I’ve met in over a decade, and it’s been nothing short of amazing.

Tooling

I added a ton of new software tools to my arsenal, being largely responsible for building out our development and deployment processes. I spent a lot of time getting up to speed with some amazing software such as South, Fabric, NGINX and gunicorn. I’ve recently been putting together a development and deployment guide that covers combining all of these and other parts of our stack which I’ll also publish in the near future, so stay tuned for that.

Django Dash

This was the first year I decided to put my Django skills to the test by entering Django Dash - the 48 hour hackathon where teams of 3 build a Django project from scratch. Teaming up with Josh de Blank and Andrew Fisher, the final result saw us launch Rate My Flight which earned us 8th place out of 40 teams internationally. It was a ton of fun getting to interact with some of the wider Django community and I even picked up a few new Django tricks along the way.

So that was 2010 - what a blast! I can’t imagine what 2011 will bring if it involves as much change and initiative as the previous year has. Bring it on!

On Modern Web Development

2010-09-28T00:00:00-07:00

Web development technology has come an incredibly long way over the last decade. Unfortunately in local markets like Australia where I reside, the tech sector seems to languish years behind the United States and Europe. We saw this happen with broadband adoption at the turn of the century where most Australians were still on dial-up for years after the rest of the world enjoyed widely available cable and ADSL.

Fast forward to 2010; and the same situation has occurred with web development. Anyone who follows the startup scene focused around San Francisco and NYC will be familiar with the most popular development technologies available - Ruby, Python, Scala and many more. Contrast this to the Australian space where the majority of development houses have been sitting for many years on technologies such as ASP.NET and PHP, which reached their peak in popularity over half a decade ago.

Why are the majority of shops in markets like mine so complacent when it comes to their technology stacks? Is it CTOs that haven’t written a line of code in years and their fear of the unknown? Is it a lack of willingness to invest in keeping their developers’ skill-sets up to date and marketable? On these things I can only speculate. What I can speak about knowledgeably however are some of the reasons these latest technologies far outshine their predecessors, and if only a single technology manager reads this post and decides to act on it then it was well worth the time spent writing it.

We chose Django several years ago so that’s what I’ll be making reference to in the following comparisons, however you could just as easily swap it out with Ruby on Rails or any other modern platform and the points would be more or less equivalent.

	ASP.NET	PHP	Java	Django
Efficiency	✘	✘	✘	✔
Security	✘	✘	✔	✔
Freedom	✘	✔	✘	✔
Developers	✔	✔	✔	✘
Mature Applications	✔	✔	✔	✘

Efficiency

I recently came across this fantastic quote:

“IDEs: a form of automation needed when the environment in question erects artificial barriers.”

Have you ever tried writing C# or Java in a plain text editor? It is an exercise in futility as these languages sport incredibly verbose syntax and deeply nested libraries which require specialised tools simply to write code. What about the developer that needs to work with all of these different technologies each day, can they be expected to be experts in several different IDEs and switch between them freely? If only these different programming environments could be used from the same editor - what a joy it would be as a practicing polyglot!

On the plus side languages like C# and Java are relatively clean and consistant when compared to abominations such as PHP, which truly is a disorganized mess - functions named using both verb_noun and noun_verb, lots of similar functions with no apparent naming convention (Eg: sort(), arsort(), asort(), ksort(), natsort(), natcasesort(), rsort(), usort(), array_multisort(), uksort()) and a weak type system that can lead to bugs which are difficult to discover.

These languages stand in great contrast to modern dynamic languages such as Python and Ruby. Ask any Python or Ruby developer which IDE they use and the majority of them will tell you they don’t need one. These languages are terse, use flat heirarchies in their libraries and are incredibly expressive.

Security

Aside from the recent Padding Oracle Exploit, ASP.NET has remained fairly secure over the years. Unfortunately the other parts of the stack, IIS and SQL Server, that it’s exclusively tied to have been the punching bag of the network security world throughout the last decade with viruses such as Code Red, SQL Slammer and more, leaving countless websites either defaced or knocked entirely offline. With a track record like this it truly is a wonder how anyone would knowingly choose to build public facing Internet services based on a Windows stack.

Inversely we have PHP that while typically deployed on a LAMP stack built with security in mind, the language itself makes writing secure applications an extemely disciplined task. One need look no further than the ongoing range of security issues that have plagued applications such as Wordpress over the years: SQL injection, cross site scripting (XSS), remote code execution - it’s like an all-you-can-eat smorgasbord of web application exploits.

Django in contrast runs on top of a secure LAMP stack and is designed from the ground up with security in mind. It’s protected by default against SQL injection, XSS and cross site request forgeries. A developer would actually have to make a concerted effort to create an exploitable Django application. Also like many open source projects a security issue in Django isn’t dealt with because a corporation deems it to be the most cost effective decision. In the very few and far between occasions when security issues have inevitably been discovered, turn-around time for resolving these has been within a 24 hour period - not weeks or even months as is often the case with corporate vendors that lack the agility and motivation to act responsibly.

Freedom

A common misconception about open source software is that it lacks the reliability of support that comes with choosing a commercial vendor. This is a short sighted view now plaguing many businesses. When Microsoft introduced .NET it made the skill-sets of thousands of VB developers redundant. What happens when Microsoft announces that .NET is to be deprecated in their next technical adventure? The problem here is that a public company with an obligation to generate as much profit as it can controls the technology path of billions of dollars of software. Sometimes it’s in their best financial interest to create fantastic technology, and sometimes it’s in their best financial interest to tear it all down again.

There is then the issue of acquisition. Companies like Microsoft and Oracle have a long and successful history of acquiring their competitors simply to discontinue their competing technology - let’s hope the vendor you’re in bed with isn’t too good.

Even Java, which for all intents and purposes is an open source technology has recently shown that it isn’t immune to the flaw of corporate ownership with Oracle suing Google over its use on Android phones. Will Android developers find that their time and effort invested in this platform will all be for naught?

Python and Django are both licensed under Permissive Free Software Licenses, which allow anyone to go ahead and do whatever the hell they like with them. They are owned by the Python Software Foundation and the Django Software Foundation respectively. These are non-profit bodies that for the most part exist to enforce the IP rights of each technology. These technologies are community driven with one goal in mind: to create best of breed technology. There is no financial motivation here and so we thus find ultimate reliability with this software stack being impervious to the risks described above - it cannot be made redundant by any financially driven corporate strategy as the licensing and foundations have been specifically put in place to prevent this from being possible. An interesting corollary to this is that these technologies go relatively unheard of without the backing of large corporations promoting them. Next time you’re the target of a technology sales pitch consider the high likelihood that you’re not looking at the best technology in its given application domain - best of breed doesn’t need to be sold at all, it sells itself.

Developers

ASP.NET, Java and PHP developers outnumber Python and Ruby developers by the hundreds if not thousands. This is a great selling point to technical decision makers - the ability to quickly and easily hire developers when the need arises is critical. But what of the quality of these developers? An interesting question to pose is why a developer who specializes in .NET or PHP chose their particular technology. Answer: because that’s what everyone else uses. You certainly won’t find a developer who has gone out and thoroughly investigated a broad range of different languages coming back and choosing .NET over everything else. Those who have done so have chosen their languages on its technical merits. These are the passionate developers with a love for their craft, not those who are merely in it for a paycheck and whose workmanship will reflect as much. Paul Graham referred to this in 2004 in his incredibly insightful essay The Python Paradox.

Mature Applications

The final point I’d like to cover is the maturity of applications developed on top of any given platform. This is where modern languages fall short as by definition they simply haven’t gained enough penetration for mature applications to have been developed yet. This is where there is opportunity. This is where the next generation of mature web applications will be built using web application frameworks like Django and Rails that are designed from the ground up for rapid customization over long software life-cycles while maintaining the original design integrity of your application - something only a very disciplined developer can maintain with something like PHP, which in almost all cases will eventually end up as spaghetti code.

Applications like Wordpress and Magento may work fine off the shelf for an end user, but what type of path have you created for your customer by implementing technology that gets closer and closer to its end of life the more it’s customized?

In conclusion - developers and CEOs, challenge your technical decision makers to overcome their complacency and invest in technology of the future. Push your employer, your peers and most importantly yourself forward. Invest your time in efficient, secure and unencumbered technology.

Plugging In Cartridge

2010-09-21T00:00:00-07:00

I’m happy to announce the first release of Cartridge - a Django shopping cart application I started working on back in 2009. The development path that Cartridge has taken has been a strange one. I stopped working on it throughout 2010 in order to get the ball rolling with a project called Mezzanine that I’ve blogged about previously. Many parts of Mezzanine actually originated in Cartridge and once development of Mezzanine was well under way it made the most sense for continued development of Cartridge to occur as a plug-in for Mezzanine which has now come to fruition.

Beyond creating a kick-ass shopping cart application for Django, the main design goal I originally had for Cartridge was to address some of the mistakes I felt existed in other offerings available both within and outside of the Django community. These areas I’ve aimed to address are as follows:

Performance

The Django ORM is a double-edged sword that while saving you a lot of time can also do a lot of damage when used without regard to the underlying SQL queries being performed. I’ve come across examples in templates where queries are being performed inside loops nested inside more loops resulting in abysmal performance. Fixing these problems wasn’t simply a case of refactoring template logic as these issues were core to the design of how prices and variations were modelled. The only solution was to throw a ton of fine-grained memcached usage at the problem, but this should be a optional approach to scalability - not a minimum requirement for keeping the site from falling offline. Cartridge has been designed with performance in mind from the start with a range of denormalized data structures providing O(n) performance as the number of products and categories grow.

Intuitive Interfaces

An end user should be able to use an admin system for the first time and discover an interface that is logical and intuitive. Having to go through a handful of screens to set up a single product requires users to mentally train themselves to remember a work-flow that isn’t entirely natural. If the former is achievable then the latter is definitely unacceptable. The number of forms and fields in an interface can be described as a conversation between the system and its user, and this conversation should be as quick and painless as possible. Cartridge provides single interfaces for creating products, categories, discounts and other types of shop data, with only applicable fields making up these forms.

Bloated Code

Having a system that implements every single feature that might ever be required in a shopping cart implementation certainly makes for an easy sell, however as this list grows these features become more obscure and less likely to be required in an average implementation. This can result in a convoluted code base that is very difficult to apply customizations to - an inevitable requirement given the unique nature of shopping cart implementations. Cartridge addresses this issue by implementing only the features typically required by all shopping cart implementations, leaving custom features up to the developer who will find that due to a tight feature list that the code base and data models remain very clean and easy to work with.

Aside from these features that distinguish it from other shopping cart applications, Cartridge comes with a standard set of features that you’d expect to find:

Hierarchical categories (via Mezzanine pages)
Easily configurable product options (colours, sizes, etc)
Hooks for shipping calculations and payment gateway
Sale pricing and discount codes
Stock management
Product popularity
Thumbnail generation
Built-in test suite
Separation of presentation (no embedded markup)
Smart categories (by price range, colour, etc)
Configurable number of checkout steps

The live demo of Mezzanine now includes Cartridge so go ahead and try it out! If you’re interested in hacking on Cartridge then the source code is freely available under a BSD license at both github and bitbucket.

An Open Source Brief

2010-07-22T00:00:00-07:00

You’d be forgiven for reading the title of this post and thinking it’s about a crazy approach to project briefing that somehow mimics open source development - as interesting as that sounds, it isn’t the case and my motives are much more simplistic and sinister. What I’d like to do here is put a brief together for an open source project called Mezzanine. This brief isn’t specifically geared towards programmers so if you think this isn’t for you then please continue reading and let me prove you wrong.

What is Mezzanine?

Anyone who follows my updates will know it’s an open source CMS framework I’ve been working on over the last couple of months. It now has a concrete feature set having come remarkably far in a very short amount of time. This might lead you to believe an entire team of people have been working on it but in fact it’s mostly been myself alone - it’s thanks to the incredibly rapid development that using Django brings you that so much has been done so quickly. For those readers who aren’t familiar with it please go ahead and check out the overview in the documentation, play around with the live demo and have a read of my previous blog post that talks about why I started Mezzanine and what I hope to achieve.

Why would I want to help?

Perhaps you’re an end user of a poorly designed CMS and you’ve often wished you could do something about it. Perhaps you’re a developer that’s had the unfortunate experience of trying to extend a seemingly user-friendly CMS that’s built using archaic technology, and wished you could be working with something that’s much more cutting edge and elegantly designed. Perhaps you’re someone who “gets” open source at a deeper level but always felt as someone who isn’t a coder that you couldn’t contribute. Perhaps you’re in business development and you’re tired of trying to sell “enterprise” crap with completely absurd price tags. If you have anything to do with web development then there’s something in this for you.

What do I get out of it?

As much as you put in of course. The experience of contributing to open source software on paper can often be a competitive advantage over other candidates for a job interview or even development contracts for your business. There’s also the chance of notoriety - imagine being responsible for the user interface or branding of the next Wordpress. Imagine your staff are core contributors to one of the web’s leading development tools. Again the success of the project will only match its contributions so it’s ultimately up to you.

What can I do to help?

A common misconception about open source software is that it’s something that only coders can participate in. Unfortunately the result of this is that the majority of open source software ends up being only contributed to by coders and is incredibly lacking in a variety of areas. I’m talking about visual branding, copy-writing, UI development - all these areas that fall outside of coding but are equally crucial in successfully shipping a professional piece of software. Mezzanine has now reached a point where it can only continue to move forward at a consistent pace by bringing in these skills that I don’t specialise in. So without further ado, here are the specific roles I think need filling and what the focus of each would be.

Graphic Designer

The entire project is desperately in need of some visual love. At the simplest level it could really use some basic branding such as a logo and “powered by” buttons. Then there’s the Mezzanine website, documentation and default site that are all currently quite spartan looking.

Interface Developer

So far the template mark-up for the default site is as minimal as can be. While this is intentional to some extent in order to best serve those that would come along and customise it for their projects, I think this idea could be improved upon with a greater level of modularity. I’m also keen to introduce a CSS framework like Blueprint into the default site. Once that’s all in order then I’d like to address what theming would look like. Is this simply a matter of packaging up copies of the templates as separate themes? A great milestone for Mezzanine would be to have a handful of built-in themes created, as well as documenting the process for creating your own.

UX Designer

I’ve introduced a handful of user interface elements into Mezzanine that could definitely use some ironing out from a usability and accessibility perspective. The main contender is the navigation tree in the admin that’s used for managing the hierarchy of the entire site as well as being the entry point for accessing most of the content management. There’s the dashboard interface for the admin area which is in a very early stage. There’s the overall layout for both the project’s own site and the default site. Lastly and of great importance, there’s the entire system for in-line editing which is featured in the default site - making this feature as user-friendly as possible is critical.

Technical Writer

Mezzanine currently has a good start on documentation but at the moment it’s mostly focused on developers. I’d eventually like to have a lot more material aimed at both end users of Mezzanine as well as marketing material geared towards business decision makers.

Product Evangelist

This is probably the easiest task of all. We simply need the word to be spread. Learn about Mezzanine and use whatever medium you like to let the world know how great it is, be it Twitter, mailing lists or blog posts.

This list isn’t entirely complete and some of the tasks certainly overlap. If you think you fit the bill or know anyone else who would get a kick out of working on Mezzanine then there’s no time like the present to get started. There aren’t any obligations with this so contributions of any size are welcome. If you’d like to get involved but don’t know where to start just post a message to the mailing list and let’s talk!

Mezzanine: Just Another Django CMS?

2010-06-11T00:00:00-07:00

Ask any developer that has put together a Django admin interface and they’ll tell you that it’s an amazing piece of technology that allows you to whip up an admin system for your web application in a number of minutes rather than days. Unfortunately this power can be a double-edged sword as without enough time and thought up front, a developer can easily end up creating an interface that’s almost impossible for its intended audience to work with.

I’ve seen this issue manifest itself in two ways. The first is the bare-bones case where the options available for creating the admin interface are simply left out, resulting in a spartan admin that does little to guide the user on how to use it. The second could be described as the opposite end of the scale where there is actually far too much going on in the admin interface at the cost of simplicity and intuitiveness. This can often be the result of gluing together a range of resuable apps that each have their own approach to providing admin interfaces with the end result looking like a Rube Goldberg machine. These scenarios are bad for customers and bad for Django. When end-users think of Django, they think of the admin interface - that’s what Django is to them so it’s critical to get this component right.

I recently had Wordpress suggested to me as a solution to this problem and it’s easy to see why. The Wordpress install base alone speaks huge volumes while its admin interface is incredibly user- friendly. It also benefits from not requiring technical expertise to get a simple website with pages and a blog up and running. However I felt this idea overlooked the underlying issue of poorly configured Django admin interfaces, while taking a step backwards by investing in PHP - a relatively inelegant technology with a very limited application scope.

My solution to the problem was to tackle the underlying issue more directly by creating a Django application which I’ve called Mezzanine. The approach I’ve taken is to have functionality on par with Wordpress that can be used as a starting point when developing basic websites. This meant putting a lot of thought into the admin options used, as well as including a custom version of the django- grappelli admin skin to come up with a modern looking and intuitive admin interface. The other key approach I’ve taken is to include as much functionality as possible directly in the application itself for the sake of a consistent and lightweight code base that can easily be hacked on. It’s worth noting that this is in total contrast to other Django website applications such as Mingus and Pinax, and that this difference really comes down to a question of scope. Pinax for example is capable of a much wider range of functionality than what I’m aiming for with Mezzanine out of the box which is to cater for basic websites with the following features:

Hierarchical page navigation
Save as draft and preview on site
Drag-n-drop page ordering
WYSIWYG editing
SEO friendly URLs and meta data
Mobile device detection and templates
Blogging engine
Tagging
Custom templates per page or blog post
Gravatar integration
Google Analytics integration
Twitter feed integration
bit.ly integration
Sharing via Facebook or Twitter
Built-in threaded comments, or:
Disqus integration

The Mezzanine admin dashboard

I’ve open sourced the initial version of Mezzanine with a BSD license on both github and bitbucket - it still has a long way to go so jump right in and fork away.

Announcing gunicorn-console

2010-05-29T00:00:00-07:00

Like a lot of Django shops our software stack consists of two layers up front: a public facing web/proxy server and an application server sitting behind it. For a long time we’ve enjoyed success using nginx and Apache to fill these roles respectively, but as an application server the 800 pound gorilla that is Apache can really be overkill, which over time we’ve found can have quite a cost around lack of granular control. So we recently decided to try out the up and coming gunicorn which is currently gaining in popularity throughout the Django community and so far it’s been very smooth.

One of the interesting features it provides is the ability to handle various kill signals which map to functions such as adding and removing worker processes as well as reloading the master process, all on the fly without dropping a single client connection. So after a brief honeymoon period I then came up with the following list of questions that mightn’t be apparent when serving a single application, but really come into play when serving dozens of applications this way on a single server:

How can we deal with the signals interface without knowledge of process IDs?
How can we gain visiblity around the ports being used?
How can we gain visiblity around the number of worker procesess being used?
How can we gain visiblity around the amount of memory being used per application?

All of these can be answered with a small amount of command-line-fu, however I wanted this process to be ridiculously easy for our entire team. For quite some time I’ve wanted to put together a console application using the curses library so a simple management console for gunicorn seemed like the perfect opportunity to do so and as such, gunicorn-console was born.

As pictured above, after firing up a few gunicorn instances with varying parameters gunicorn-console gives you the following interface in all its 8bit glory:

If you’re hosting multiple applications served up via gunicorn then gunicorn- console should make managing them easier. I’ve released it with a BSD license on both github and bitbucket using the amazing hg-git extension, so go ahead and make it better!

Update, May 30: I ended this post with a request for others to contribute and after only a day someone already has. Adam Vandenberg went ahead and forked the project with some patches to get it running on OSX, so a big thanks goes to him.

Optional Django Apps

2010-05-10T00:00:00-07:00

A project of mine contains a number of third-party apps that are development related and potentially not available on every machine the project will run on. My general approach to dealing with these was to try and import the app in my settings module and if successful, add it to the INSTALLED_APPS setting. However as the number of these apps grew it became a wart within the settings module so I put together this snippet for managing them.

We first create a sequence of dictionaries, each containing information about an installed app such as the module to try and import, an extra potential condition for checking and then the sequences of names to add to INSTALLED_APPS, MIDDLEWARE_CLASSES and TEMPLATE_CONTEXT_PROCESSORS. Let’s start with the settings for optionally including the apps django- command-extensions, django-debug-toolbar and south.

# Define any settings specific to each of the optional apps.
import sys
USE_SOUTH = not (len(sys.argv) > 1 and sys.argv[1] == "test")
DEBUG_TOOLBAR_CONFIG = {"INTERCEPT_REDIRECTS": False}

# Sequence for each optional app as a dict containing info about the app.
OPTIONAL_APPS = (
    {"import": "django_extensions", "apps": ("django_extensions",)},
    {"import": "debug_toolbar", "apps": ("debug_toolbar",), 
        "middleware": ("debug_toolbar.middleware.DebugToolbarMiddleware",)},
    {"import": "south", "apps": ("south",), "condition": USE_SOUTH},
)

Next we simply iterate through the sequence of optional apps and set them up.

# Set up each optional app if available.
for app in OPTIONAL_APPS:
    if app.get("condition", True):
        try:
            __import__(app["import"])
        except ImportError:
            pass
        else:
            INSTALLED_APPS += app.get("apps", ())
            MIDDLEWARE_CLASSES += app.get("middleware", ())

Where is PEP 8 for Technical Documentation?

2010-03-18T00:00:00-07:00

Anyone who has programmed in Python for a considerable length of time will at least have some passing familiarity with PEP 8 - the document that goes into an incredible level of detail in dictating precisely how code should be written. While its primary goal is to ensure that Python code is written in a consistant fashion throughout the community, therefore making it as easy as possible to read, it also provides one of many aspects that makes programming in Python an incredibly efficient process - it negates the need for a lot of decision making around any of the choices one might come across that are already covered in PEP 8.

I’ve recently been spending a lot of time writing technical documentation. While it’s been interesting doing something different for a change, the perfectionist in me is constantly frustated with finding myself using inconsistant language across different sections when faced with the exact same context, for example:

contains / has / includes / provides
discussed / referred to / described / mentioned
above / earlier / previously
below / later / next

Are these types of ambiguities in technical writing something that professional editors typically deal with? What I’d love to see is something like PEP 8 for technical documentation.

Hasta la Vista, Windows

2010-02-13T00:00:00-08:00

Earlier this week I had the pleasure of removing my final Windows install after wiping my machine at work and installing Ubuntu on it. It was during the late 90s that I first tried out Linux after getting my hands on a Redhat 6.1 CD from the cover of a magazine I’d bought. I didn’t keep it installed for very long and after a few more tries over the years with Mandrake (now Mandrivia) and Damn Small Linux, it wasn’t until 2005 when I installed Slackware 10.2 as my primary operating system at home and really cut my teeth on it in order to test how cross-platform my Python projects were. It was a great experience learning about all the various sub-systems, compiling software and libraries from source, embracing the command line and modifying some of the internal scripts to get things working the way I wanted.

Fast forward to 2010 and in my workplace the migration from a Microsoft development shop to a Linux/Python shop after several years is finally complete, paving the way for this latest install. I did experience a couple of hiccups that hadn’t happened before. Firstly I have dual wide-screen monitors at work and I rotate one of them 90 degrees in order to maximize the amount of visible code on my screen. The display properties in Ubuntu only gave me the ability to flip the display 180 degrees which seemed quite odd so I then tried to manually rotate the display with the xrandr command which reported my overall virtual screen size as being too small for the rotation. I resolved this with the update below to my x.org configuration to use a virtual screen size large enough to handle the rotation while including the second monitor.

Section "Screen"
    Identifier "Configured Screen Device"
    Device "Configured Video Device"
    SubSection "Display"
        Virtual 2880 1440
    EndSubSection
EndSection

The second issue was more a lack of foresight on my part than a problem with the new install itself. After a vanilla install of any modern operating system you’ll undoubtedly be required to download a series of updates that have occurred since the version you’ve installed was initially released. The difference with most Linux distributions is that almost all of your software is managed in this way from installing to updating, it all goes through the same service known as a package manager - one of the many things with Linux that once you’re used to using you won’t know how you ever worked without it. So away I went with the initial round of updates which left the package manager busy for several hours, during which time some issues arose with a project that immediately required my attention. Unfortunately I needed to install a handful of libraries to get up and running and with the package manager busy I was left in a real bind. Fortunately I was able to use one of our test servers remotely to resolve the issue but the lesson learnt here is that for a new development system it’s best to leave the initial system update until after your development environment is completely set up.

Linear Traversal of Adjacency List Trees

2010-01-26T00:00:00-08:00

My current project has the common requirement of storing and rendering a hierarchical tree of categories. This project is geared towards potentially junior developers with the expectation of it being hacked at every time it’s used - a set of scaffolding where simplicity isn’t just a loose goal but a fundamental requirement.

Two popular approaches to the hierarchical tree are the Adjacency List (AL) and Modified Preorder Tree Traversal (MPTT) models. The advantage of AL is that it only stores the exact data required for representing the tree while MPTT stores extraneous data for assisting in traversing the tree in an optimal fashion. The simplicity of the AL model makes it much better suited to the requirements I mentioned, however the problem with AL is the recursive nature in which you traverse it.

def show_branch(parent, depth=0):
    # iterating the entire tree for each branch gives quadratic performance
    for node in nodes:
        if node.parent == parent:
            print (" " * depth) + node
            show_branch(node, depth + 1)

Worst case here is O(n²) performance but thanks to Python’s lightning fast hashtable implementation we can create a copy of the tree as a dictionary of branches giving us O(n) overall performance when traversing the entire tree.

# copy the tree into a dict of branches
branches = {}
for node in nodes:
    parent = node.parent
    if parent not in branches:
        branches[parent] = []
    branches[parent].append(node)

def show_branch(parent, depth=0):
    # iterating only the nodes in the branch gives linear performance
    for node in branches[parent]:
        print (" " * depth) + node
        if node in branches:
            show_branch(node, depth + 1)

When rendering the entire tree, using this technique will greatly increase performance as the tree grows in size. Be aware though that if your application only ever deals with a single branch in any given view, this technique won’t perform as well as directly querying the database for the nodes in a single branch.

Technology Shows Style of Things to Come

2009-11-30T00:00:00-08:00

Technology shows style of things to come

The Sydney Morning Herald has an article that talks about some of the work I did for Sportsgirl including their community portal running on Django and a Facebook app written in C# - nice to get recognized!

FTP Browser for gedit

2009-11-03T00:00:00-08:00

Many major Linux distributions such as Ubuntu ship with gedit as the default text editor. It has all the standard features you’d expect in an editor such as syntax highlighting, a tabbed interface and the ability to integrate external tools. Most importantly it’s highly extensible with the ability to create plug- ins for it written in Python or C.

One great plug-in that’s been written is gedit-ftp- browser, an FTP client that embeds itself into the editor giving you the ability to remotely edit files over FTP. I’ve just been accepted as a contributor to the project which I’m really excited about. I’ve implemented a “Save As” feature allowing the user to create and upload new files over FTP. Next up I’ll be working on the ability to manage multiple FTP servers via profiles.

Dynamic Django Deployment

2009-10-20T00:00:00-07:00

Lately I’ve noticed people posting various different takes on making the default Django settings a lot more dynamic. The development and deployment requirements for the projects I work on tend to be far from straight-forward and over time I’ve come up with my own approach to Django settings, so here it is.

The simplest approach typically involves importing all the names from a custom settings module at the end of the project’s standard settings module, providing the ability to override settings on a per machine basis.

try:
    from local_settings import *
except ImportError:
    pass

This still requires modifying local_settings.py on a per machine basis. Another approach that builds on this is to import a custom settings module from a host_settings package that has a unique name derived from the current machine the site is running on - this gives the advantage of being able to specify custom settings per machine and have each of these settings modules reside in the version control system, without the same file having to be modified on a per machine basis.

from socket import gethostname
try:
    exec ("from host_settings.%s import *" % 
        gethostname().replace("-", "_").replace(".", "_"))
except ImportError:
    pass

This simple version of the host_settings approach I’ve seen attempts to deal with the differences between a valid hostname and a valid module name with the two calls to replace, but ignores the fact a hostname could begin with a number which would be an invalid module name. Other versions of this approach handle this correctly and involve calling the __import__ built-in, iterating over and updating each name in the settings module individually, but once we look at some further requirements below and how to deal with them we’ll find this isn’t necessary.

Let’s take a step back for a moment and talk about some of the requirements I mentioned before. Where I work a project can end up deployed in a dozen different locations - a handful of developer machines and various different servers managing the project life cycle. Due to a variety of non-technical reasons it’s often required that various versions of a project run side by side in the same location, so with a project called project_x we end up with project_x_feature_a and project_x_feature_b sitting in the same location - all of a sudden all of our references to project_x are broken. So we end up taking the approach in our code that the actual name of a project’s directory is a moving target and should never be referenced - we never import from package_x and anything in our settings module that would typically reference this we set dynamically.

import os
project_path = os.path.dirname(os.path.abspath(__file__))
project_dir = project_path.split(os.sep)[-1]
MEDIA_URL = "/site_media/"
MEDIA_ROOT = os.path.join(project_path, MEDIA_URL.strip("/"))
TEMPLATE_DIRS = (os.path.join(project_path, "templates"),)
ADMIN_MEDIA_PREFIX = "/media/"
ROOT_URLCONF = "%s.urls" % project_dir

So that removes any hard-coded reference to the project’s directory name, however we sometimes have to go as far as having host specific settings that vary across these different project versions residing on the same machine, such as a different database for example. The ultimate goal here is to not have any files in the project’s version control repository that are manually edited for a specific instance of the project. So using the host_settings approach from earlier on, we continue on from the code above by using the project_dir variable when referencing the machine specific host_settings module so that each of the host_settings modules are named not only after the machine they exist for, but the project directory as well.

from socket import gethostname
host_settings_module = "%s_%s" % (project_dir, 
    gethostname().replace("-", "_").replace(".", "_").lower())
host_settings_path = os.path.join(project_path, "host_settings", 
    "%s.py" % host_settings_module)
if not os.path.exists(host_settings_path):
    try:
        f = open(host_settings_path, "w")
        f.close()
    except IOError:
        print "couldnt create host_settings module: %s " % host_settings_path
try:
    exec "from host_settings.%s import *" % host_settings_module
except ImportError:
    pass
TEMPLATE_DEBUG = DEBUG

As an added bonus, we try to create the host_settings module if it’s missing and warn if we’re unable to create it.

Nokia E63

2009-09-26T00:00:00-07:00

After living the last few years without a mobile phone it finally became a problem for me so recently I decided to get a new one. I only needed something simple for receiving calls, not making them so any kind of plan or contract was out of the question since I could spend a few bucks on pre-paid and theoretically not pay anything after that. I didn’t consider any of the latest smart-phones either like the HTC or iPhone (I wouldn’t buy a crippled device anyway) since they’re so ridiculously priced compared to lower end models that aren’t even a year old.

I ended up getting a Nokia E63 that has a full qwerty keyboard and wireless LAN access which were the main selling points for me. It cost $299 when it was being sold for over $500 at other places around town so it felt like a smart purchase. Apart from those features I looked at, I really knew nothing about the phone and Symbian OS which powers it and after taking a closer look at the software available for it I’ve been really surprised.

Google Apps for Symbian OS Putty for Symbian OS Python (PyS60) on Symbian OS

Google has created a handful of Symbian apps for things like gmail, reader, youtube and maps which all work great. The other day I found an app called fring which is very similar to Pidgin in that it wraps up all the various IM clients into one, even including Skype with voice working! So it has essentially turned my phone into a Skype handset which is amazing. I’ve also found that there’s a version of putty for Symbian so I can actually SSH onto any of our Linux servers or desktops and access the shell from my phone!

The most incredible thing I’ve found for Symbian however has been a project called PyS60 - Nokia has actually ported the Python run-time to the Symbian OS. I was amazed once I had this installed and was sitting there typing out Python code into an interactive console directly on my phone. The standard library is even included and it’s very interesting - certain pieces aren’t fully ported but it comes with a handful of modules specifically for PyS60 which handle things like locating wireless networks and working with the user interface. It even includes OpenGL bindings which is unbelievable - that’s right, you can develop 3D games in Python for your phone.

I’m well into developing my first app which is a small RPC server for the phone controlled by a wxWidgets client. The idea is to be able to traverse the phone’s file system and create, edit and execute Python apps on the phone from a remote machine. The SimpleXMLRPCServer module isn’t included with PyS60 and broke when I tried to copy it onto the phone and import it manually. I’ve since been able to patch it and get it working which I’ve submitted to Nokia, hopefully they’ll include it in their next release.

CHM Files on Linux

2009-09-21T00:00:00-07:00

I think CHM files are great and my main Python reference is the CHM version of the main documentation. I’ve used GnoCHM which is the default CHM viewer for Gnome for quite a while and it’s really poor - slow as hell on startup and segfaults half the time you click a link. I finally gave up and looked for an alternative which I found in KchmViewer which appears to be the default CHM viewer for KDE. As usual the KDE counterpart of a given Gnome app is much easier on the eyes and in this case the problems I had are solved - lightning fast and stable.