RestMQ

About

RestMQ is a message queue which uses HTTP as transport, JSON to format a minimalist protocol and is organized as REST resources. It stands on the shoulder of giants, built over Python, Twisted, Cyclone (a Tornado implementation over twisted) and Redis.
Message queues are created on the fly, as a message is sent to them. They are simple to use as a curl request can be.

There is a simple JSON-based protocol for those looking for a more formal syntax, but it is not mandatory.

The core idea is to use Redis's LIST type to provide the message ordering, and SETs to index queues. Also each queue has a UUID generator to provide atomic and unique ids for each queue item.

This basic protocol can be implemented in any language. Python and Cyclone were used due to the maturity and robustness it offered but there are work started using Ruby and Erlang.

The fixed set of operations provides that even different brokers can interoperate in the simplest level (put and take objects out of the queue), thanks to Redis atomic operations.

There are HTTP clients for pratically all languages. That's all that it takes to use RestMQ within your application. No special protocols or strategies, just dynamically created queues.

Example

A http client (curl) post to /queue:

Point your browser to http://localhost:8888/c/test

Run:

$ curl -X POST -d "value=foobar" http://localhost:8888/q/test

Your browser is acting as a consumer to the queue. Using json encoded data it's easy to fit the data into a js based app.

Download

Check out from the github repository: http://github.com/gleicon/restmq.

Status and Dashboard

Point your browser at http://localhost:8888/ (or your configured site) to check the dashboard. It has pause/continue controls for each queue. If you deploy RestMQ into a server with a valid IP address to the internet, check the proper access control by restricting access to these routes or proxying thru nginx. For a json based endpoint, check http://localhost:8888/stats or http://localhost:8888/stats/

A minimal ruby/sinatra based RestMQ protocol implementation

COMET consumer

There is a COMET based consumer, which will bind even if the queue doesn't already exists.

The main route is thru /c/<queuename>, It can be tested using curl: $ curl http://localhost:8888/c/test

In another terminal, run:

$ curl -X POST -d "value=foobar" http://localhost:8888/q/test

See below on how to purge and disconnect all consumers from a queue, using DELETE.

WebSocket consumer

Check README.websocket to test it. You can consume messages as the COMET endpoint using websockets enabled browsers.

REST services

A queue can be accessed as /q/<queuename>.

GET requests will dequeue an object.

POST requests inserts an object in the queue

DELETE requests will purgue the queue.

The usual pattern is listen in the COMET consumer (/c/<queuename>) and insert new stuff at the REST route (POST /q/<queuename>).

Distribution strategies

Both COMET and Websockets endpoints can use two different strategies to distribute messages: broadcast and round robin. Broadcast means that all connected clients will receive every messages, and Round Robin will select one consumer a time.

Start/Pause

A QUEUE can be paused/resumed without disconnecting its clients.

curl http://localhost:8888/control/test
curl -X POST -d "status=stop" http://localhost:8888/control/test
curl http://localhost:8888/control/test
curl -X POST -d "status=start" http://localhost:8888/control/test

Map/Reduce and Job Scheduling

The persistend consumers (COMET and Websockets) can be used to deploy ad-hoc map/reduce and job scheduling. Check examples/mapreduce for the cannonical word count example.

JSON Protocol

The HTTP route /queue/<queuename> uses the JSON protocol. Its the same protocol I've implemented for jsonqueue

{
  "cmd": "add",
  "queue": "genesis",
  "value": "abacab"
}

Creates the queue named "genesis" and inserts the string "abacab" as the message.

If we want to take that out of the queue, the payload would be like that:

{
  "cmd": "take",
  "queue": "genesis"
}

The message can be formatted as a json object, so more complex data can be sent. It was inspired by Amazon SQS.

Other goodies

Check the docs and examples for more:

Select, EPoll or KQueue concurrency (depends on twisted)
Persistent queueing using Redis
Can work on pools, N daemons consuming from the same queues.
Cute ?
Small codebase

Dependencies

python, twisted
redis client: git clone git://github.com/fiorix/txredisapi.git
cyclone: git clone git://github.com/fiorix/cyclone.git

Running

Development environment:

$ ./restmq_server

Production environment (needs epoll and proper user/group): uncomment the proper lines on restmq_server script. Note that there are a collectd_restmq_server script and a syslogd_server_script in the project, which uses the restmq core to run different kind of protocols (collectd and syslogd). You can do pretty much the same for any other protocol.

Quickstart

examples/test_rest.sh
examples/test_xmlrpc.py
python examples/test_comet.py
python examples/twitter_trends.py
python examples/test_comet_curl.py
python restmq_engine.py -h

Related projects

Credits

Thanks to (in no particular order):
Salvatore Sanfilippo for redis and for NoSQL patterns discussion. Alexandre Fiori for the redis client enhancement and patches. Giovani Generali for this website

Redis based message queue