Simple job queues for Python
Go to file
Vincent Driessen e992d65a30 Add RPM dependencies. 2012-01-25 10:46:11 +01:00
bin Allow users to specify the import path manually, if the default (current 2011-12-13 17:09:58 +01:00
examples Change semantics of work(). Add work_burst(). 2011-11-17 22:39:53 +01:00
rq Silently pass when trying to kill child that is already dead. 2011-11-28 15:15:18 +01:00
tests Run class initializations only once. 2011-11-27 08:00:41 +01:00
.gitignore Add some project meta stuff. 2011-11-14 12:11:58 +01:00
LICENSE Add BSD license, copied from git-flow. 2011-11-17 15:22:47 +01:00
README.md Getting the facts right here. 2011-11-26 13:46:22 +01:00
calcsize.sh Fix output. 2011-11-25 00:39:36 +01:00
run_tests Add some project meta stuff. 2011-11-14 12:11:58 +01:00
setup.cfg Add RPM dependencies. 2012-01-25 10:46:11 +01:00
setup.py Make procname a hard dependency. 2011-11-28 14:17:42 +01:00

README.md

RQ (Redis Queue) is a lightweight* Python library for queueing jobs and processing them in the background with workers. It is backed by Redis and it is extremely simple to use.

* It is under 20 kB in size and just over 500 lines of code.

Getting started

First, run a Redis server, of course:

$ redis-server

To put jobs on queues, you don't have to do anything special, just define your typically lengthy or blocking function:

import urllib2

def count_words_at_url(url):
    f = urllib2.urlopen(url)
    count = 0
    while True:
        line = f.readline()
        if not line:
            break
        count += len(line.split())
    return count

Then, create a RQ queue:

import rq import *
use_redis()
q = Queue()

And enqueue the function call:

from my_module import count_words_at_url
result = q.enqueue(count_words_at_url, 'http://nvie.com')

For a more complete example, refer to the docs. But this is the essence.

The worker

To start executing enqueued function calls in the background, start a worker from your project's directory:

$ rqworker
*** Listening for work on default
Got count_words_at_url('http://nvie.com') from default
Job result = 818
*** Listening for work on default

That's about it.

Installation

Simply use the following command to install the latest released version:

pip install rq

If you want the cutting edge version (that may well be broken), use this:

pip install -e git+git@github.com:nvie/rq.git@master#egg=rq

Project history

This project has been inspired by the good parts of Celery, Resque and this snippet, and has been created as a lightweight alternative to the heaviness of Celery or other AMQP-based queueing implementations.