* Add an exception handler for the pubsub thread
After a connection error to Redis the pubsub thread was stopped and the
worker could not receive commands anymore.
This commit adds an exception handler for the thread that adds a log
message and ignores `redis.exceptions.ConnectionError`. Any other
exception is re-raised.
redis-py internal mechanism allows the pubsub thread to recover its
connection and reinstall the pubsub channel subscription to allow the
worker to receive commands again after connection errors.
It tries to behave the same as the main worker loop retry mechanism but
without the backoff wait factor.
Fixes#1836Fixes#2070
* Add test for untested line, improve logging and comments
Add *args & **kwargs to tests.fixtures.raise_exc to pass tests with Python 3.7
Prior to this commit it was not possible to get a number of jobs in a
registry or list job IDs without triggering a cleanup.
Calling cleanup caused issues for:
* Monitoring tools (see https://github.com/rq/rq/pull/2104) and
* Cases with high number of jobs in the registries (see
https://github.com/rq/rq/pull/2003).
* Cases where a failure callback is registered and but the
`get_job_ids` is called not from the main thread.
In this commit:
1. A new `BaseRegistry.get_job_count` method is added. It is a side
effect free version of `BaseRegistry.count` and runs in O(1).
2. A `cleanup` parameter added to `get_job_ids` that allows to avoid
clean up if it is set to `False`.
Resolves https://github.com/rq/rq/pull/2104.
Resolves https://github.com/rq/rq/pull/2003.
* Execution objects now return UTC timestamps
* execution.job should return a full Job instance
* worker.maintain_heartbeats() should extend execution TTL
* Don't use cached_property since it's not supported on Python 3.7
* Fixed type hint of execution._job
* add info about running docs in README
* update jekyll config to fix error
* update docs to clarify misc timeout values
* update readme
* update argument from default_worker_ttl -> worker_ttl
* Fix black formatting
* argument should still be respected until it's deprecated
* Updated CHANGES.md
* Updated CHANGES.md
---------
Co-authored-by: Sean Villars <sean@setpoint.io>
This change checks for Redis connection socket timeouts that are too
short for operations such as BLPOP, and adjusts them to at least be the
expected timeout.
* Add new deferred_ttl attribute to jobs
* Add DeferredJobRegistry to cleanup
* Use normal ttl for deferred jobs as well
* Test that jobs landing in deferred queue get a TTL
* Pass pipeline in job cleanup
* Remove cleanup call
We pass a ttl of -1 which does not do anything
* Pass exc_info to add
The add implementation overwrites it so it won't get lost this way
* Remove extraneous save call
The add function already saves the job for us. So no need to save it
twice.
* Tune cleanup function description
* Test cleanup also works for deleted jobs
* Replace testconn with connection
---------
Co-authored-by: Raymond Guo <raymond.guo@databricks.com>
* Add test to list queue with only deferred jobs
Signed-off-by: Simó Albert i Beltran <sim6@probeta.net>
* Register queue also for deferred job
Without this change queues initialized with only deferred jobs are not
listed by `rq.Queue.all()`.
Signed-off-by: Simó Albert i Beltran <sim6@probeta.net>
---------
Signed-off-by: Simó Albert i Beltran <sim6@probeta.net>
* fix docs: use delay to enqueue and use declared in separate file
Signed-off-by: Mehdi Nassim KHODJA <18899702+naskio@users.noreply.github.com>
* fix docs: use delay only
Signed-off-by: Mehdi Nassim KHODJA <18899702+naskio@users.noreply.github.com>
---------
Signed-off-by: Mehdi Nassim KHODJA <18899702+naskio@users.noreply.github.com>
* Fix the "Fork me on GitHub" ribbon
The image URL is broken. Replace it with the "official" URL found in
<https://github.blog/2008-12-19-github-ribbons/>.
* Replace the orange ribbon with a red one
Selwin Ong