Several months ago I wrote about an approach I came up with for embedding Lua libraries in Redis, in my post Bitwise Lua Operations in Redis. At the time I was developing a library that would allow me to use regular Python types such as lists and sets, which would then be backed by Redis each time they were modified. The end goal was to be able to compose these types together, to form the structures I needed for Kouio, the RSS reader that Adam O’Byrne and I have built together. Things like a FIFO queue that would only accept unique items, while supporting blocking operations with timeouts, multisets for storing unread counts, and more.
I wanted a library that would let me build these components in a natural way, without having to deal with each of the underlying Redis commands. The end result I came up with is called HOT Redis, which originally stood for Higher Order Types for Redis, specifically, types backed by Redis that could be used to build richer composite types. Some may argue that what I’ve built doesn’t meet the strict definition of higher order types, such as those found in languages like Scala and Haskell, in which case the recursive acronym HOT Object Toolkit for Redis can be used, which should appease the most luscious of bearded necks.
Each of the types provided by HOT Redis strive to implement the same method signatures and return values as their Python built-in and standard library counterparts. The main difference is each type’s
__init__ method. Every HOT Redis type’s
__init__ method will optionally accept
key keyword arguments, which are used for defining an initial value to be stored in Redis for the object, and the key that should be used, respectively. If no key is provided, a key will be generated, which can then be accessed via the
Once you’ve determined a strategy for naming keys, you can then create HOT Redis objects and interact with them over the network, for example here is a
List created on a computer we’ll refer to as computer A:
then on another computer we’ll creatively refer to as computer B:
and back to computer A:
The last interaction here is an interesting one. Python’s
list.reverse() is an in-place reversal of the list, that is, it modifies the existing list, rather than returning a reversed copy. If we were to implement this naively, we would first read the list from Redis, reverse it locally, then store the reversed list back in Redis again. But what if another client were to modify the list at approximately the same time? One computer’s modification to the list would certainly overwrite the other’s. In this scenario, and many others, HOT Redis provides its own Lua routine specifically for reversing the list in-place, within Redis atomically.
The following table is the complete list of types provided by HOT Redis, mapped to their Python counterparts and underlying Redis types, along with any special considerations worth noting.
|String||string||string||Mutable — string methods that normally create a new string object in Python will mutate the string stored in Redis|
|ImmutableString||string||string||Immutable — behaves like a regular Python string|
|SetQueue||N/A||list + set||Extension of
|LifoSetQueue||N/A||list + set||Extension of
That’s all for now, but I’m really keen to grow the range of data structures provided by HOT Redis. One of the nice things about such a narrowly defined library like this is the ability to craft a very thorough test suite, simply by executing every method on every type, and doing the same for each of the Python built-in and standard library counterparts, finally comparing the results. This should really ease contributing new types — so if you have any ideas for additions, go right ahead and dive in.