Rooba
/
kombu
mirror of https://github.com/celery/kombu.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update README

This commit is contained in:
Ask Solem 2010-06-29 21:12:29 +02:00
parent ddb17c7cc6
commit 470a269396
1 changed files with 83 additions and 107 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

190
README.rst
View File

@ -58,33 +58,32 @@ Exchanges/Bindings can be bound to a channel::
Introduction
------------
`carrot` is an `AMQP`_ messaging queue framework. AMQP is the Advanced Message
``kombu`` is an `AMQP`_ messaging queue framework. AMQP is the Advanced Message
Queuing Protocol, an open standard protocol for message orientation, queuing,
routing, reliability and security.
The aim of `carrot` is to make messaging in Python as easy as possible by
The aim of ``kombu`` is to make messaging in Python as easy as possible by
providing a high-level interface for producing and consuming messages. At the
same time it is a goal to re-use what is already available as much as possible.
`carrot` has pluggable messaging back-ends, so it is possible to support
`kombu` has pluggable messaging back-ends, so it is possible to support
several messaging systems. Currently, there is support for `AMQP`_
(`py-amqplib`_, `pika`_), `STOMP`_ (`python-stomp`_). There's also an
(`py-amqplib`_, `pika`_), `STOMP`_ (`stompy`_). There's also an
in-memory backend for testing purposes, using the `Python queue module`_.
Several AMQP message broker implementations exists, including `RabbitMQ`_,
`ZeroMQ`_ and `Apache ActiveMQ`_. You'll need to have one of these installed,
`Apache ActiveMQ`_. You'll need to have one of these installed,
personally we've been using `RabbitMQ`_.
Before you start playing with ``carrot``, you should probably read up on
Before you start playing with ``kombu``, you should probably read up on
AMQP, and you could start with the excellent article about using RabbitMQ
under Python, `Rabbits and warrens`_. For more detailed information, you can
refer to the `Wikipedia article about AMQP`_.
.. _`RabbitMQ`: http://www.rabbitmq.com/
.. _`ZeroMQ`: http://www.zeromq.org/
.. _`AMQP`: http://amqp.org
.. _`STOMP`: http://stomp.codehaus.org
.. _`python-stomp`: http://bitbucket.org/asksol/python-stomp
.. _`stompy`: http://pypi.python.org/stompy
.. _`Python Queue module`: http://docs.python.org/library/queue.html
.. _`Apache ActiveMQ`: http://activemq.apache.org/
.. _`Django`: http://www.djangoproject.com/
@ -96,24 +95,24 @@ refer to the `Wikipedia article about AMQP`_.
Documentation
-------------
Carrot is using Sphinx, and the latest documentation is available at GitHub:
Kombu is using Sphinx, and the latest documentation is available at GitHub:
http://github.com/ask/carrot/
http://ask.github.com/kombu
Installation
============
You can install ``carrot`` either via the Python Package Index (PyPI)
You can install ``kombu`` either via the Python Package Index (PyPI)
or from source.
To install using ``pip``,::
$ pip install carrot
$ pip install kombu
To install using ``easy_install``,::
$ easy_install carrot
$ easy_install kombu
If you have downloaded a source tarball you can install it
@ -128,9 +127,9 @@ Terminology
There are some concepts you should be familiar with before starting:
* Publishers
* Producers
Publishers sends messages to an exchange.
Producers sends messages to an exchange.
* Exchanges
@ -188,30 +187,13 @@ Creating a connection
---------------------
You can set up a connection by creating an instance of
``carrot.messaging.BrokerConnection``, with the appropriate options for
``kombu.BrokerConnection``, with the appropriate options for
your broker:
>>> from carrot.connection import BrokerConnection
>>> from kombu import BrokerConnection
>>> conn = BrokerConnection(hostname="localhost", port=5672,
... userid="test", password="test",
... virtual_host="test")
If you're using Django you can use the
``carrot.connection.DjangoBrokerConnection`` class instead, which loads
the connection settings from your ``settings.py``::
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "test"
BROKER_PASSWORD = "secret"
BROKER_VHOST = "/test"
Then create a connection by doing:
>>> from carrot.connection import DjangoBrokerConnection
>>> conn = DjangoBrokerConnection()
... userid="guest", password="guest",
... virtual_host="/")
Receiving messages using a Consumer
@ -222,36 +204,44 @@ First we open up a Python shell and start a message consumer.
This consumer declares a queue named ``"feed"``, receiving messages with
the routing key ``"importer"`` from the ``"feed"`` exchange.
The example then uses the consumers ``wait()`` method to go into consume
mode, where it continuously polls the queue for new messages, and when a
message is received it passes the message to all registered callbacks.
>>> from kombu import Exchange, Binding, Consumer
>>> feed_exchange = Exchange("feed", type="direct")
>>> feed_binding = Binding("feed", feed_exchange, "importer")
>>> channel = connection.channel()
>>> consumer = Consumer(channel, [feed_binding])
>>> from carrot.messaging import Consumer
>>> consumer = Consumer(connection=conn, queue="feed",
... exchange="feed", routing_key="importer")
>>> def import_feed_callback(message_data, message)
... feed_url = message_data["import_feed"]
... print("Got feed import message for: %s" % feed_url)
... # something importing this feed url
... # import_feed(feed_url)
... message.ack()
>>> consumer.register_callback(import_feed_callback)
>>> consumer.wait() # Go into the consumer loop.
Sending messages using a Publisher
----------------------------------
>>> consumer.register_callback(import_feed_callback)
>>> # Consume messages in a loop
>>> while True:
... connection.drain_events(timeout=...)
Sending messages using a Producer
---------------------------------
Then we open up another Python shell to send some messages to the consumer
defined in the last section.
>>> from carrot.messaging import Publisher
>>> publisher = Publisher(connection=conn,
... exchange="feed", routing_key="importer")
>>> publisher.send({"import_feed": "http://cnn.com/rss/edition.rss"})
>>> publisher.close()
>>> from kombu import Exchange, Producer
>>> feed_exchange = Exchange("feed", type="direct")
>>> channel = connection.channel()
>>> producer = Producer(channel, feed_exchange)
>>> producer.publish({"import_feed": "http://cnn.com/rss/edition.rss"},
... routing_key="importer")
>>> producer.close()
Look in the first Python shell again (where ``consumer.wait()`` is running),
Look in the first Python shell again (where consumer loop is running),
where the following text has been printed to the screen::
Got feed import message for: http://cnn.com/rss/edition.rss
@ -276,19 +266,19 @@ Each option has its advantages and disadvantages.
a standard part of Python (since 2.6), and is fairly fast to
decode using the modern Python libraries such as ``cjson or
``simplejson``.
The primary disadvantage to ``JSON`` is that it limits you to
the following data types: strings, unicode, floats, boolean,
dictionaries, and lists. Decimals and dates are notably missing.
Also, binary data will be transferred using base64 encoding, which
will cause the transferred data to be around 34% larger than an
encoding which supports native binary types.
However, if your data fits inside the above constraints and
you need cross-language support, the default setting of ``JSON``
is probably your best choice.
``pickle`` -- If you have no desire to support any language other than
Python, then using the ``pickle`` encoding will gain you
the support of all built-in Python data types (except class instances),
@ -298,10 +288,10 @@ Each option has its advantages and disadvantages.
``yaml`` -- YAML has many of the same characteristics as ``json``,
except that it natively supports more data types (including dates,
recursive references, etc.)
However, the Python libraries for YAML are a good bit slower
than the libraries for JSON.
If you need a more expressive set of data types and need to maintain
cross-language compatibility, then ``YAML`` may be a better fit
than the above.
@ -309,26 +299,20 @@ Each option has its advantages and disadvantages.
To instruct carrot to use an alternate serialization method,
use one of the following options.
1. Set the serialization option on a per-Publisher basis:
>>> from carrot.messaging import Publisher
>>> publisher = Publisher(connection=conn,
... exchange="feed", routing_key="importer",
... serializer="yaml")
1. Set the serialization option on a per-producer basis::
2. Set the serialization option on a per-call basis
>>> producer = Producer(channel,
... exchange=exchange,
... serializer="yaml")
>>> from carrot.messaging import Publisher
>>> publisher = Publisher(connection=conn,
... exchange="feed", routing_key="importer")
>>> publisher.send({"import_feed": "http://cnn.com/rss/edition.rss"},
... serializer="pickle")
>>> publisher.close()
2. Set the serialization option per message::
Note that ``Consumer``s do not need the serialization method specified in
their code. They can auto-detect the serialization method since we supply
the ``Content-type`` header as part of the AMQP message.
>>> producer.publish(message, routing_key=rkey,
... serializer="pickle")
Note that a ``Consumer`` do not need the serialization method specified.
They can auto-detect the serialization method as the
content-type is sent as a message header.
Sending raw data without Serialization
---------------------------------------
@ -340,30 +324,25 @@ not waste cycles serializing/deserializing the data.
You can optionally specify a ``content_type`` and ``content_encoding``
for the raw data:
>>> from carrot.messaging import Publisher
>>> publisher = Publisher(connection=conn,
... exchange="feed",
routing_key="import_pictures")
>>> publisher.send(open('~/my_picture.jpg','rb').read(),
content_type="image/jpeg",
content_encoding="binary")
>>> publisher.close()
The ``message`` object returned by the ``Consumer`` class will have a
``content_type`` and ``content_encoding`` attribute.
>>> producer.send(open('~/my_picture.jpg','rb').read(),
content_type="image/jpeg",
content_encoding="binary",
routing_key=rkey)
The ``Message`` object returned by the ``Consumer`` class will have a
``content_type`` and ``content_encoding`` attribute.
Receiving messages without a callback
--------------------------------------
You can also poll the queue manually, by using the ``fetch`` method.
You can also poll the queue manually, by using the ``get`` method.
This method returns a ``Message`` object, from where you can get the
message body, de-serialize the body to get the data, acknowledge, reject or
re-queue the message.
>>> consumer = Consumer(connection=conn, queue="feed",
... exchange="feed", routing_key="importer")
>>> message = consumer.fetch()
>>> consumer = Consumer(channel, bindings)
>>> message = consumer.get()
>>> if message:
... message_data = message.payload
... message.ack()
@ -374,23 +353,19 @@ re-queue the message.
Sub-classing the messaging classes
----------------------------------
The ``Consumer``, and ``Publisher`` classes can also be sub classed. Thus you
can define the above publisher and consumer like so:
The ``Consumer``, and ``Producer`` classes can also be sub classed. Thus you
can define the above producer and consumer like so:
>>> from carrot.messaging import Publisher, Consumer
>>> class FeedPublisher(Publisher):
... exchange = "feed"
>>> class FeedProducer(Producer):
... exchange = exchange
... routing_key = "importer"
...
... def import_feed(self, feed_url):
... return self.send({"action": "import_feed",
... "feed_url": feed_url})
... return self.publish({"action": "import_feed",
... "feed_url": feed_url})
>>> class FeedConsumer(Consumer):
... queue = "feed"
... exchange = "feed"
... routing_key = "importer"
... bindings = bindings
...
... def receive(self, message_data, message):
... action = message_data["action"]
@ -401,12 +376,13 @@ can define the above publisher and consumer like so:
... else:
... raise Exception("Unknown action: %s" % action)
>>> publisher = FeedPublisher(connection=conn)
>>> publisher.import_feed("http://cnn.com/rss/edition.rss")
>>> publisher.close()
>>> producer = FeedProducer(channel)
>>> producer.import_feed("http://cnn.com/rss/edition.rss")
>>> producer.close()
>>> consumer = FeedConsumer(connection=conn)
>>> consumer.wait() # Go into the consumer loop.
>>> consumer = FeedConsumer(channel)
>>> while True:
... connection.drain_events()
Getting Help
============
@ -422,12 +398,12 @@ Bug tracker
===========
If you have any suggestions, bug reports or annoyances please report them
to our issue tracker at http://github.com/ask/carrot/issues/
to our issue tracker at http://github.com/ask/kombu/issues/
Contributing
============
Development of ``carrot`` happens at Github: http://github.com/ask/carrot
Development of ``kombu`` happens at Github: http://github.com/ask/kombu
You are highly encouraged to participate in the development. If you don't
like Github (for some reason) you're welcome to send regular patches.