Stephen McDonald

You’d be forgiven for reading the title of this post and thinking it’s about a crazy approach to project briefing that somehow mimics open source development - as interesting as that sounds, it isn’t the case and my motives are much more simplistic and sinister. What I’d like to do here is put a brief together for an open source project called Mezzanine. This brief isn’t specifically geared towards programmers so if you think this isn’t for you then please continue reading and let me prove you wrong.

What is Mezzanine?

Anyone who follows my updates will know it’s an open source CMS framework I’ve been working on over the last couple of months. It now has a concrete feature set having come remarkably far in a very short amount of time. This might lead you to believe an entire team of people have been working on it but in fact it’s mostly been myself alone - it’s thanks to the incredibly rapid development that using Django brings you that so much has been done so quickly. For those readers who aren’t familiar with it please go ahead and check out the overview in the documentation, play around with the live demo and have a read of my previous blog post that talks about why I started Mezzanine and what I hope to achieve.

Why would I want to help?

Perhaps you’re an end user of a poorly designed CMS and you’ve often wished you could do something about it. Perhaps you’re a developer that’s had the unfortunate experience of trying to extend a seemingly user-friendly CMS that’s built using archaic technology, and wished you could be working with something that’s much more cutting edge and elegantly designed. Perhaps you’re someone who “gets” open source at a deeper level but always felt as someone who isn’t a coder that you couldn’t contribute. Perhaps you’re in business development and you’re tired of trying to sell “enterprise” crap with completely absurd price tags. If you have anything to do with web development then there’s something in this for you.

What do I get out of it?

As much as you put in of course. The experience of contributing to open source software on paper can often be a competitive advantage over other candidates for a job interview or even development contracts for your business. There’s also the chance of notoriety - imagine being responsible for the user interface or branding of the next Wordpress. Imagine your staff are core contributors to one of the web’s leading development tools. Again the success of the project will only match its contributions so it’s ultimately up to you.

What can I do to help?

A common misconception about open source software is that it’s something that only coders can participate in. Unfortunately the result of this is that the majority of open source software ends up being only contributed to by coders and is incredibly lacking in a variety of areas. I’m talking about visual branding, copy-writing, UI development - all these areas that fall outside of coding but are equally crucial in successfully shipping a professional piece of software. Mezzanine has now reached a point where it can only continue to move forward at a consistent pace by bringing in these skills that I don’t specialise in. So without further ado, here are the specific roles I think need filling and what the focus of each would be.

Graphic Designer
The entire project is desperately in need of some visual love. At the simplest level it could really use some basic branding such as a logo and “powered by” buttons. Then there’s the Mezzanine website, documentation and default site that are all currently quite spartan looking.

Interface Developer
So far the template mark-up for the default site is as minimal as can be. While this is intentional to some extent in order to best serve those that would come along and customise it for their projects, I think this idea could be improved upon with a greater level of modularity. I’m also keen to introduce a CSS framework like Blueprint into the default site. Once that’s all in order then I’d like to address what theming would look like. Is this simply a matter of packaging up copies of the templates as separate themes? A great milestone for Mezzanine would be to have a handful of built-in themes created, as well as documenting the process for creating your own.

UX Designer
I’ve introduced a handful of user interface elements into Mezzanine that could definitely use some ironing out from a usability and accessibility perspective. The main contender is the navigation tree in the admin that’s used for managing the hierarchy of the entire site as well as being the entry point for accessing most of the content management. There’s the dashboard interface for the admin area which is in a very early stage. There’s the overall layout for both the project’s own site and the default site. Lastly and of great importance, there’s the entire system for in-line editing which is featured in the default site - making this feature as user-friendly as possible is critical.

Technical Writer
Mezzanine currently has a good start on documentation but at the moment it’s mostly focused on developers. I’d eventually like to have a lot more material aimed at both end users of Mezzanine as well as marketing material geared towards business decision makers.

Product Evangelist
This is probably the easiest task of all. We simply need the word to be spread. Learn about Mezzanine and use whatever medium you like to let the world know how great it is, be it Twitter, mailing lists or blog posts.

This list isn’t entirely complete and some of the tasks certainly overlap. If you think you fit the bill or know anyone else who would get a kick out of working on Mezzanine then there’s no time like the present to get started. There aren’t any obligations with this so contributions of any size are welcome. If you’d like to get involved but don’t know where to start just post a message to the mailing list and let’s talk!

Ask any developer that has put together a Django admin interface and they’ll tell you that it’s an amazing piece of technology that allows you to whip up an admin system for your web application in a number of minutes rather than days. Unfortunately this power can be a double-edged sword as without enough time and thought up front, a developer can easily end up creating an interface that’s almost impossible for its intended audience to work with.

I’ve seen this issue manifest itself in two ways. The first is the bare-bones case where the options available for creating the admin interface are simply left out, resulting in a spartan admin that does little to guide the user on how to use it. The second could be described as the opposite end of the scale where there is actually far too much going on in the admin interface at the cost of simplicity and intuitiveness. This can often be the result of gluing together a range of resuable apps that each have their own approach to providing admin interfaces with the end result looking like a Rube Goldberg machine. These scenarios are bad for customers and bad for Django. When end-users think of Django, they think of the admin interface - that’s what Django is to them so it’s critical to get this component right.

I recently had Wordpress suggested to me as a solution to this problem and it’s easy to see why. The Wordpress install base alone speaks huge volumes while its admin interface is incredibly user-friendly. It also benefits from not requiring technical expertise to get a simple website with pages and a blog up and running. However I felt this idea overlooked the underlying issue of poorly configured Django admin interfaces, while taking a step backwards by investing in PHP - a relatively inelegant technology with a very limited application scope.

My solution to the problem was to tackle the underlying issue more directly by creating a Django application which I’ve called Mezzanine. The approach I’ve taken is to have functionality on par with Wordpress that can be used as a starting point when developing basic websites. This meant putting a lot of thought into the admin options used, as well as including a custom version of the django-grappelli admin skin to come up with a modern looking and intuitive admin interface. The other key approach I’ve taken is to include as much functionality as possible directly in the application itself for the sake of a consistent and lightweight code base that can easily be hacked on. It’s worth noting that this is in total contrast to other Django website applications such as Mingus and Pinax, and that this difference really comes down to a question of scope. Pinax for example is capable of a much wider range of functionality than what I’m aiming for with Mezzanine out of the box which is to cater for basic websites with the following features:

• Hierarchical page navigation
• Save as draft and preview on site
• Drag-n-drop page ordering
• WYSIWYG editing
• SEO friendly URLs and meta data
• Mobile device detection and templates
• Blogging engine
• Tagging
• Custom templates per page or blog post
• Gravatar integration
• Google Analytics integration
• Twitter feed integration
• bit.ly integration
• Sharing via Facebook or Twitter
• Built-in threaded comments, or:
• Disqus integration

The Mezzanine admin dashboard

I’ve open sourced the initial version of Mezzanine with a BSD license on both github and bitbucket - it still has a long way to go so jump right in and fork away.

Like a lot of Django shops our software stack consists of two layers up front: a public facing web/proxy server and an application server sitting behind it. For a long time we’ve enjoyed success using nginx and Apache to fill these roles respectively, but as an application server the 800 pound gorilla that is Apache can really be overkill, which over time we’ve found can have quite a cost around lack of granular control. So we recently decided to try out the up and coming gunicorn which is currently gaining in popularity throughout the Django community and so far it’s been very smooth.

One of the interesting features it provides is the ability to handle various kill signals which map to functions such as adding and removing worker processes as well as reloading the master process, all on the fly without dropping a single client connection. So after a brief honeymoon period I then came up with the following list of questions that mightn’t be apparent when serving a single application, but really come into play when serving dozens of applications this way on a single server:

• How can we deal with the signals interface without knowledge of process IDs?
• How can we gain visiblity around the ports being used?
• How can we gain visiblity around the number of worker procesess being used?
• How can we gain visiblity around the amount of memory being used per application?

All of these can be answered with a small amount of command-line-fu, however I wanted this process to be ridiculously easy for our entire team. For quite some time I’ve wanted to put together a console application using the curses library so a simple management console for gunicorn seemed like the perfect opportunity to do so and as such, gunicorn-console was born.

As pictured above, after firing up a few gunicorn instances with varying parameters gunicorn-console gives you the following interface in all its 8bit glory:

If you’re hosting multiple applications served up via gunicorn then gunicorn-console should make managing them easier. I’ve released it with a BSD license on both github and bitbucket using the amazing hg-git extension, so go ahead and make it better!

Update, May 30: I ended this post with a request for others to contribute and after only a day someone already has. Adam Vandenberg went ahead and forked the project with some patches to get it running on OSX, so a big thanks goes to him.

A project of mine contains a number of third-party apps that are development related and potentially not available on every machine the project will run on. My general approach to dealing with these was to try and import the app in my settings module and if successful, add it to the INSTALLED_APPS setting. However as the number of these apps grew it became a wart within the settings module so I put together this snippet for managing them.

We first create a sequence of dictionaries, each containing information about an installed app such as the module to try and import, an extra potential condition for checking and then the sequences of names to add to INSTALLED_APPS, MIDDLEWARE_CLASSES and TEMPLATE_CONTEXT_PROCESSORS. Let’s start with the settings for optionally including the apps django-command-extensions, django-debug-toolbar and south.

# Define any settings specific to each of the optional apps.
import sys
USE_SOUTH = not (len(sys.argv) > 1 and sys.argv[1] == "test")
DEBUG_TOOLBAR_CONFIG = {"INTERCEPT_REDIRECTS": False}

# Sequence for each optional app as a dict containing info about the app.
OPTIONAL_APPS = (
    {"import": "django_extensions", "apps": ("django_extensions",)},
    {"import": "debug_toolbar", "apps": ("debug_toolbar",), 
        "middleware": ("debug_toolbar.middleware.DebugToolbarMiddleware",)},
    {"import": "south", "apps": ("south",), "condition": USE_SOUTH},
)

Next we simply iterate through the sequence of optional apps and set them up.

# Set up each optional app if available.
for app in OPTIONAL_APPS:
    if app.get("condition", True):
        try:
            __import__(app["import"])
        except ImportError:
            pass
        else:
            INSTALLED_APPS += app.get("apps", ())
            MIDDLEWARE_CLASSES += app.get("middleware", ())
            TEMPLATE_CONTEXT_PROCESSORS += app.get("context_processors", ())

Anyone who has programmed in Python for a considerable length of time will at least have some passing familiarity with PEP 8 - the document that goes into an incredible level of detail in dictating precisely how code should be written. While its primary goal is to ensure that Python code is written in a consistant fashion throughout the community, therefore making it as easy as possible to read, it also provides one of many aspects that makes programming in Python an incredibly efficient process - it negates the need for a lot of decision making around any of the choices one might come across that are already covered in PEP 8.

I’ve recently been spending a lot of time writing technical documentation. While it’s been interesting doing something different for a change, the perfectionist in me is constantly frustated with finding myself using inconsistant language across different sections when faced with the exact same context, for example:

• contains / has / includes / provides
• discussed / referred to / described / mentioned
• above / earlier / previously
• below / later / next

Are these types of ambiguities in technical writing something that professional editors typically deal with? What I’d love to see is something like PEP 8 for technical documentation.

Earlier this week I had the pleasure of removing my final Windows install after wiping my machine at work and installing Ubuntu on it. It was during the late 90s that I first tried out Linux after getting my hands on a Redhat 6.1 CD from the cover of a magazine I’d bought. I didn’t keep it installed for very long and after a few more tries over the years with Mandrake (now Mandrivia) and Damn Small Linux, it wasn’t until 2005 when I installed Slackware 10.2 as my primary operating system at home and really cut my teeth on it in order to test how cross-platform my Python projects were. It was a great experience learning about all the various sub-systems, compiling software and libraries from source, embracing the command line and modifying some of the internal scripts to get things working the way I wanted.

Fast forward to 2010 and in my workplace the migration from a Microsoft development shop to a Linux/Python shop after several years is finally complete, paving the way for this latest install. I did experience a couple of hiccups that hadn’t happened before. Firstly I have dual wide-screen monitors at work and I rotate one of them 90 degrees in order to maximize the amount of visible code on my screen. The display properties in Ubuntu only gave me the ability to flip the display 180 degrees which seemed quite odd so I then tried to manually rotate the display with the xrandr command which reported my overall virtual screen size as being too small for the rotation. I resolved this with the update below to my x.org configuration to use a virtual screen size large enough to handle the rotation while including the second monitor.

Section "Screen"
    Identifier "Configured Screen Device"
    Device "Configured Video Device"
    SubSection "Display"
        Virtual 2880 1440
    EndSubSection
EndSection

The second issue was more a lack of foresight on my part than a problem with the new install itself. After a vanilla install of any modern operating system you’ll undoubtedly be required to download a series of updates that have occurred since the version you’ve installed was initially released. The difference with most Linux distributions is that almost all of your software is managed in this way from installing to updating, it all goes through the same service known as a package manager - one of the many things with Linux that once you’re used to using you won’t know how you ever worked without it. So away I went with the initial round of updates which left the package manager busy for several hours, during which time some issues arose with a project that immediately required my attention. Unfortunately I needed to install a handful of libraries to get up and running and with the package manager busy I was left in a real bind. Fortunately I was able to use one of our test servers remotely to resolve the issue but the lesson learnt here is that for a new development system it’s best to leave the initial system update until after your development environment is completely set up.

My current project has the common requirement of storing and rendering a hierarchical tree of categories. This project is geared towards potentially junior developers with the expectation of it being hacked at every time it’s used - a set of scaffolding where simplicity isn’t just a loose goal but a fundamental requirement.

Two popular approaches to the hierarchical tree are the Adjacency List (AL) and Modified Preorder Tree Traversal (MPTT) models. The advantage of AL is that it only stores the exact data required for representing the tree while MPTT stores extraneous data for assisting in traversing the tree in an optimal fashion. The simplicity of the AL model makes it much better suited to the requirements I mentioned, however the problem with AL is the recursive nature in which you traverse it.

def show_branch(parent, depth=0):
    # iterating the entire tree for each branch gives quadratic performance
    for node in nodes:
        if node.parent == parent:
            print (" " * depth) + node
            show_branch(node, depth + 1)

Worst case here is O(n²) performance but thanks to Python’s lightning fast hashtable implementation we can create a copy of the tree as a dictionary of branches giving us O(n) overall performance when traversing the entire tree.

# copy the tree into a dict of branches
branches = {}
for node in nodes:
    parent = node.parent
    if parent not in branches:
        branches[parent] = []
    branches[parent].append(node)

def show_branch(parent, depth=0):
    # iterating only the nodes in the branch gives linear performance
    for node in branches[parent]:
        print (" " * depth) + node
        if node in branches:
            show_branch(node, depth + 1)

When rendering the entire tree, using this technique will greatly increase performance as the tree grows in size. Be aware though that if your application only ever deals with a single branch in any given view, this technique won’t perform as well as directly querying the database for the nodes in a single branch.

Many major Linux distributions such as Ubuntu ship with gedit as the default text editor. It has all the standard features you’d expect in an editor such as syntax highlighting, a tabbed interface and the ability to integrate external tools. Most importantly it’s highly extensible with the ability to create plug-ins for it written in Python or C.

One great plug-in that’s been written is gedit-ftp-browser, an FTP client that embeds itself into the editor giving you the ability to remotely edit files over FTP. I’ve just been accepted as a contributor to the project which I’m really excited about. I’ve implemented a “Save As” feature allowing the user to create and upload new files over FTP. Next up I’ll be working on the ability to manage multiple FTP servers via profiles.

Lately I’ve noticed people posting various different takes on making the default Django settings a lot more dynamic. The development and deployment requirements for the projects I work on tend to be far from straight-forward and over time I’ve come up with my own approach to Django settings, so here it is.

The simplest approach typically involves importing all the names from a custom settings module at the end of the project’s standard settings module, providing the ability to override settings on a per machine basis.

try:
    from local_settings import *
except ImportError:
    pass

This still requires modifying local_settings.py on a per machine basis. Another approach that builds on this is to import a custom settings module from a host_settings package that has a unique name derived from the current machine the site is running on - this gives the advantage of being able to specify custom settings per machine and have each of these settings modules reside in the version control system, without the same file having to be modified on a per machine basis.

from socket import gethostname
try:
    exec ("from host_settings.%s import *" % 
        gethostname().replace("-", "_").replace(".", "_"))
except ImportError:
    pass

This simple version of the host_settings approach I’ve seen attempts to deal with the differences between a valid hostname and a valid module name with the two calls to replace, but ignores the fact a hostname could begin with a number which would be an invalid module name. Other versions of this approach handle this correctly and involve calling the __import__ built-in, iterating over and updating each name in the settings module individually, but once we look at some further requirements below and how to deal with them we’ll find this isn’t necessary.

Let’s take a step back for a moment and talk about some of the requirements I mentioned before. Where I work a project can end up deployed in a dozen different locations - a handful of developer machines and various different servers managing the project life cycle. Due to a variety of non-technical reasons it’s often required that various versions of a project run side by side in the same location, so with a project called project_x we end up with project_x_feature_a and project_x_feature_b sitting in the same location - all of a sudden all of our references to project_x are broken. So we end up taking the approach in our code that the actual name of a project’s directory is a moving target and should never be referenced - we never import from package_x and anything in our settings module that would typically reference this we set dynamically.

import os
project_path = os.path.dirname(os.path.abspath(__file__))
project_dir = project_path.split(os.sep)[-1]
MEDIA_URL = "/site_media/"
MEDIA_ROOT = os.path.join(project_path, MEDIA_URL.strip("/"))
TEMPLATE_DIRS = (os.path.join(project_path, "templates"),)
ADMIN_MEDIA_PREFIX = "/media/"
ROOT_URLCONF = "%s.urls" % project_dir

So that removes any hard-coded reference to the project’s directory name, however we sometimes have to go as far as having host specific settings that vary across these different project versions residing on the same machine, such as a different database for example. The ultimate goal here is to not have any files in the project’s version control repository that are manually edited for a specific instance of the project. So using the host_settings approach from earlier on, we continue on from the code above by using the project_dir variable when referencing the machine specific host_settings module so that each of the host_settings modules are named not only after the machine they exist for, but the project directory as well.

from socket import gethostname
host_settings_module = "%s_%s" % (project_dir, 
    gethostname().replace("-", "_").replace(".", "_").lower())
host_settings_path = os.path.join(project_path, "host_settings", 
    "%s.py" % host_settings_module)
if not os.path.exists(host_settings_path):
    try:
        f = open(host_settings_path, "w")
        f.close()
    except IOError:
        print "couldn't create host_settings module: %s " % host_settings_path
try:
    exec "from host_settings.%s import *" % host_settings_module
except ImportError:
    pass
TEMPLATE_DEBUG = DEBUG

As an added bonus, we try to create the host_settings module if it’s missing and warn if we’re unable to create it.

After living the last few years without a mobile phone it finally became a problem for me so recently I decided to get a new one. I only needed something simple for receiving calls, not making them so any kind of plan or contract was out of the question since I could spend a few bucks on pre-paid and theoretically not pay anything after that. I didn’t consider any of the latest smart-phones either like the HTC or iPhone (I wouldn’t buy a crippled device anyway) since they’re so ridiculously priced compared to lower end models that aren’t even a year old.

I ended up getting a Nokia E63 that has a full qwerty keyboard and wireless LAN access which were the main selling points for me. It cost $299 when it was being sold for over $500 at other places around town so it felt like a smart purchase. Apart from those features I looked at, I really knew nothing about the phone and Symbian OS which powers it and after taking a closer look at the software available for it I’ve been really surprised.

Google Apps for Symbian OS Putty for Symbian OS Python (PyS60) on Symbian OS

Google has created a handful of Symbian apps for things like gmail, reader, youtube and maps which all work great. The other day I found an app called fring which is very similar to Pidgin in that it wraps up all the various IM clients into one, even including Skype with voice working! So it has essentially turned my phone into a Skype handset which is amazing. I’ve also found that there’s a version of putty for Symbian so I can actually SSH onto any of our Linux servers or desktops and access the shell from my phone!

The most incredible thing I’ve found for Symbian however has been a project called PyS60 - Nokia has actually ported the Python run-time to the Symbian OS. I was amazed once I had this installed and was sitting there typing out Python code into an interactive console directly on my phone. The standard library is even included and it’s very interesting - certain pieces aren’t fully ported but it comes with a handful of modules specifically for PyS60 which handle things like locating wireless networks and working with the user interface. It even includes OpenGL bindings which is unbelievable - that’s right, you can develop 3D games in Python for your phone.

I’m well into developing my first app which is a small RPC server for the phone controlled by a wxWidgets client. The idea is to be able to traverse the phone’s file system and create, edit and execute Python apps on the phone from a remote machine. The SimpleXMLRPCServer module isn’t included with PyS60 and broke when I tried to copy it onto the phone and import it manually. I’ve since been able to patch it and get it working which I’ve submitted to Nokia, hopefully they’ll include it in their next release.

I think CHM files are great and my main Python reference is the CHM version of the main documentation. I’ve used GnoCHM which is the default CHM viewer for Gnome for quite a while and it’s really poor - slow as hell on startup and segfaults half the time you click a link. I finally gave up and looked for an alternative which I found in KchmViewer which appears to be the default CHM viewer for KDE. As usual the KDE counterpart of a given Gnome app is much easier on the eyes and in this case the problems I had are solved - lightning fast and stable.