Last month I unveiled CurioDB, a distributed and persistent Redis clone built with Scala and Akka. Since then the project has moved forward at a steady pace, with a couple of contributors jumping on board (and fixing my bugs), as well as some really cool new features, such as the addition of a Lua scripting API that mimics the one found in Redis, which I’d like to write about.
I was reading over the list of features that CurioDB lacks compared to Redis, that I’d previously documented. It contained the item “no Lua scripting”, which I’d written confidently at the time, certain that I wouldn’t possibly be able to implement something so serious in my toy project. But then I thought to myself, “why not?”, and after a bit of research, I discovered the fantastic LuaJ library, which transformed the task from a significant engineering feat, into yet another case of merely gluing some libraries together.
LuaJ provides a complete API for compiling and running Lua scripts, and exposing JVM code to them as Lua functions. LuaJ also makes a complete range of Lua data types available to your JVM code. I purposely refer to the JVM (Java Virtual Machine) here rather than Java the language, as in my case, it’s actually Scala being used to code against LuaJ, which makes the overall accomplishment seem even more death-defying.
Let’s take a look. First up, we’ll compile and run a snippet of Lua (sans imports and class scaffolding, which you can pick up from the final result in CurioDB if you wish):
As simple as that. Now let’s look at passing Lua variables from JVM land into a Lua script, and pulling Lua variables back out as JVM objects:
There you have it — JVM objects representing Lua variables, moving into and back out of Lua code. Now let’s look at the reverse, calling JVM code from inside Lua. To do this, we implement a class representing a Lua function:
That’s it, the full round trip — Lua calling JVM code, and vice versa. With that working, the rest is up to your imagination. I’ve only scratched the surface of what LuaJ provides — all of the data types found in Lua, its standard library, and much more.
The final result was a little more involved than the above implies. CurioDB is carefully designed to be non-blocking, using tell rather than ask, where actors only ever send messages forwards, without expectation of a reply. The challenge here was introducing the synchronous
redis.call API into a fundamentally asynchronous system. The solution involved modelling the scripting API as a client actor, with a controlled amount of blocking, much like the way TCP and HTTP connections are managed in the system.
A really fun and whacky side effect of implementing this possibly a little too correctly (for lack of a better term), is that it unintentionally allows the Lua API to recursively call itself:
Is this a feature, or a bug? The scripting API in Redis specifically disallows this, most likely for good reason.