Higher Order Types for Redis

October 23, 2013  /  Home

Several months ago I wrote about an approach I came up with for embedding Lua libraries in Redis, in my post Bitwise Lua Operations in Redis. At the time I was developing a library that would allow me to use regular Python types such as lists and sets, which would then be backed by Redis each time they were modified. The end goal was to be able to compose these types together, to form the structures I needed for Kouio, the RSS reader that Adam O’Byrne and I have built together. Things like a FIFO queue that would only accept unique items, while supporting blocking operations with timeouts, multisets for storing unread counts, and more.

I wanted a library that would let me build these components in a natural way, without having to deal with each of the underlying Redis commands. The end result I came up with is called HOT Redis, which originally stood for Higher Order Types for Redis, specifically, types backed by Redis that could be used to build richer composite types. Some may argue that what I’ve built doesn’t meet the strict definition of higher order types, such as those found in languages like Scala and Haskell, in which case the recursive acronym HOT Object Toolkit for Redis can be used, which should appease the most luscious of bearded necks.

Usage

Each of the types provided by HOT Redis strive to implement the same method signatures and return values as their Python built-in and standard library counterparts. The main difference is each type’s __init__ method. Every HOT Redis type’s __init__ method will optionally accept initial and key keyword arguments, which are used for defining an initial value to be stored in Redis for the object, and the key that should be used, respectively. If no key is provided, a key will be generated, which can then be accessed via the key attribute:

>>> from hot_redis import List
>>> my_list = List()
>>> my_list.key
'93366bdb-90b2-4226-a52a-556f678af40e'
>>> my_list_with_key = List(key="foo")
>>> my_list_with_key.key
'foo'

Once you’ve determined a strategy for naming keys, you can then create HOT Redis objects and interact with them over the network, for example here is a List created on a computer we’ll refer to as computer A:

>>> list_on_computer_a = List(key="foo", initial=["a", "b", "c"])

then on another computer we’ll creatively refer to as computer B:

>>> list_on_computer_b = List(key="foo")
>>> list_on_computer_b[:]  # Performs: LRANGE foo 0 -1
['a', 'b', 'c']
>>> list_on_computer_b += ['d', 'e', 'f']  # Performs: RPUSH foo d e f

and back to computer A:

>>> list_on_computer_a[:]  # Performs: LRANGE foo 0 -1
['a', 'b', 'c', 'd', 'e', 'f']
>>> 'c' in list_on_computer_a  # Works like Python lists where expected
True
>>> list_on_computer_a.reverse()
>>> list_on_computer_a[:]
['f', 'e', 'd', 'c', 'b', 'a']

The last interaction here is an interesting one. Python’s list.reverse() is an in-place reversal of the list, that is, it modifies the existing list, rather than returning a reversed copy. If we were to implement this naively, we would first read the list from Redis, reverse it locally, then store the reversed list back in Redis again. But what if another client were to modify the list at approximately the same time? One computer’s modification to the list would certainly overwrite the other’s. In this scenario, and many others, HOT Redis provides its own Lua routine specifically for reversing the list in-place, within Redis atomically.

Data Types

The following table is the complete list of types provided by HOT Redis, mapped to their Python counterparts and underlying Redis types, along with any special considerations worth noting.

HOT Redis Python Redis Notes
List list list  
Set set set  
Dict dict hash  
String string string Mutable — string methods that normally create a new string object in Python will mutate the string stored in Redis
ImmutableString string string Immutable — behaves like a regular Python string
Int int int  
Float float float  
Queue Queue.Queue list  
LifoQueue Queue.LifoQueue list  
SetQueue N/A list + set Extension of Queue with unique members
LifoSetQueue N/A list + set Extension of LifoQueue with unique members
BoundedSemaphore threading.BoundedSemaphore list Extension of Queue leveraging Redis' blocking list pop operations with timeouts, while using Queue's maxsize arg to provide BoundedSemaphore's value arg
Semaphore threading.Semaphore list Extension of BoundedSemaphore without a queue size
Lock threading.Lock list Extension of BoundedSemaphore with a queue size of 1
RLock threading.RLock list Extension of Lock allowing multiple acquire calls
DefaultDict collections.DefaultDict hash  
MultiSet collections.Counter hash  

Conclusion

That’s all for now, but I’m really keen to grow the range of data structures provided by HOT Redis. One of the nice things about such a narrowly defined library like this is the ability to craft a very thorough test suite, simply by executing every method on every type, and doing the same for each of the Python built-in and standard library counterparts, finally comparing the results. This should really ease contributing new types — so if you have any ideas for additions, go right ahead and dive in.

Home