I started using it in my projects where I was supposed to convert the results of computer vision algorithms into GeoJSON. These algorithms generate millions of segments and are run on very large images. Hence the conversion process takes a lot of time! My goal was to provide REST endpoints to users to upload their results and view them in GeoJSON. Kue fits perfectly in this setting since my application server was in Node.js. The users submitted their results which triggered the conversion job that kue tracked and showed me the status. Users could poll that status to get the latest state of the job: inactive, active, complete or failed.
We decided to use Kue for some of our other projects when we quickly ran into problems pertaining to Kue's heavy dependence on Node.js and lack of documentation for the job queue to be accessed outside of Node. I dug around the source code and found the redis messages, data structures etc used in Kue. We decided that external programs that wanted to access Kue would mimic these messages.
Kue: Under the hood
Kue works great when there's a Node.js master process running it. Often though other applications would want to subscribe to the messages that Kue is providing. Kue provides little documentation about what it's doing under the hood. In this post I explain some of the redis data structures and messages used by Kue. Other applications can subscribe to Redis with the help of this.
Subscribing to Jobs
Creation of new jobs
Kue uses Redis's pubsub functionality. Other applications can either poll Kue's APIs constantly or subscribe to the messages on Redis.
127.0.0.1:6379> subscribe q:events Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "q:events" 3) (integer) 1 1) "message" 2) "q:events"
On receiving a new job
Everytime a new JOB is created a message is published on
q:events channel in Redis with the job ID.
On job completion Kue publishes a similar message on Redis's
You can subscribe to this channel from some other program that depends on this job to finish before executing something else. Or you could use this to publish messages from some other system that you're developing and would like to use Kue as your queueing system.
All of Kue's jobs are stored in Redis HashMaps with their key as:
<id> is the same as you received in the
Getting Jobs to show up nicely on the dashboard
So you've updated the Job type, published the message on
q:events and you want this to be updated on the dashboard as well. Unfortunately your job would still be under the
inactive section in the dashboard. For this you'll update the zlist(sorted list) named:
and so on. So whenever a job moves from inactive to active: the corresponding entry in
q:jobs:inactive must be deleted and an entry must be added to
With this information in play you can use Kue with other programs that you're writing. You can allow other programs to subscribe to your job que and even use other programs to programmatically manage your job queue.
A new interesting project Disque(https://github.com/antirez/disque) has garnered my interest now. It's fairly new and unstable but promises to be a more general purpose job queue. Would be interesting to see how it turns up.