27 lines
1.3 KiB
ReStructuredText
27 lines
1.3 KiB
ReStructuredText
:class:`~tornado.queues.Queue` example - a concurrent web spider
|
|
================================================================
|
|
|
|
.. currentmodule:: tornado.queues
|
|
|
|
Tornado's `tornado.queues` module (and the very similar ``Queue`` classes in
|
|
`asyncio`) implements an asynchronous producer / consumer pattern for
|
|
coroutines, analogous to the pattern implemented for threads by the Python
|
|
standard library's `queue` module.
|
|
|
|
A coroutine that yields `Queue.get` pauses until there is an item in the queue.
|
|
If the queue has a maximum size set, a coroutine that yields `Queue.put` pauses
|
|
until there is room for another item.
|
|
|
|
A `~Queue` maintains a count of unfinished tasks, which begins at zero.
|
|
`~Queue.put` increments the count; `~Queue.task_done` decrements it.
|
|
|
|
In the web-spider example here, the queue begins containing only base_url. When
|
|
a worker fetches a page it parses the links and puts new ones in the queue,
|
|
then calls `~Queue.task_done` to decrement the counter once. Eventually, a
|
|
worker fetches a page whose URLs have all been seen before, and there is also
|
|
no work left in the queue. Thus that worker's call to `~Queue.task_done`
|
|
decrements the counter to zero. The main coroutine, which is waiting for
|
|
`~Queue.join`, is unpaused and finishes.
|
|
|
|
.. literalinclude:: ../../demos/webspider/webspider.py
|