Kue: Under the hood. Using the Job Que Externally.

Kue: Under the hood. Using the Job Que Externally.

Kue is a priority job queue backed by Redis, built for Node.js. http://automattic.github.io/kue

Kue

I started using it in my projects where I was supposed to convert the results of computer vision algorithms into GeoJSON. These algorithms generate millions of segments and are run on very large images. Hence the conversion process takes a lot of time! My goal was to provide REST endpoints to users to upload their results and view them in GeoJSON. Kue fits perfectly in this setting since my application server was in Node.js. The users submitted their results which triggered the conversion job that kue tracked and showed me the status. Users could poll that status to get the latest state of the job: inactive, active, complete or failed.

We decided to use Kue for some of our other projects when we quickly ran into problems pertaining to Kue's heavy dependence on Node.js and lack of documentation for the job queue to be accessed outside of Node. I dug around the source code and found the redis messages, data structures etc used in Kue. We decided that external programs that wanted to access Kue would mimic these messages.

Kue: Under the hood

Kue works great when there's a Node.js master process running it. Often though other applications would want to subscribe to the messages that Kue is providing. Kue provides little documentation about what it's doing under the hood. In this post I explain some of the redis data structures and messages used by Kue. Other applications can subscribe to Redis with the help of this.

Subscribing to Jobs

Creation of new jobs

Kue uses Redis's pubsub functionality. Other applications can either poll Kue's APIs constantly or subscribe to the messages on Redis.

127.0.0.1:6379> subscribe q:events
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "q:events"
3) (integer) 1
1) "message"
2) "q:events"

On receiving a new job

3) "{\"id\":6,\"event\":\"enqueue\",\"args\":[\"enqueue\",\"order\"]}"

Everytime a new JOB is created a message is published on
q:events channel in Redis with the job ID.

Job completion

On job completion Kue publishes a similar message on Redis's q:events channel.

"{\"id\":\"42\",\"event\":\"complete\",\"args\":[\"complete\",null,null]}"

You can subscribe to this channel from some other program that depends on this job to finish before executing something else. Or you could use this to publish messages from some other system that you're developing and would like to use Kue as your queueing system.

 

Job MetaData

All of Kue's jobs are stored in Redis HashMaps with their key as: q:job:<id>. This <id> is the same as you received in the q:events channel.

 

Getting Jobs to show up nicely on the dashboard

So you've updated the Job type, published the message on q:events and you want this to be updated on the dashboard as well. Unfortunately your job would still be under the inactive section in the dashboard. For this you'll update the zlist(sorted list) named:
q:jobs:inactive
q:jobs:active
q:jobs:complete
and so on. So whenever a job moves from inactive to active: the corresponding entry in
q:jobs:inactive must be deleted and an entry must be added to
q:jobs:active.

With this information in play you can use Kue with other programs that you're writing. You can allow other programs to subscribe to your job que and even use other programs to programmatically manage your job queue.

Disque

A new interesting project Disque(https://github.com/antirez/disque) has garnered my interest now. It's fairly new and unstable but promises to be a more general purpose job queue. Would be interesting to see how it turns up.

 

 

 

About Ganesh

Page with Comments

  1. Great for pointing to Disque, I was thinking to write the next generation Kue on top of disque before 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *