Commit Graph

29 Commits

Author SHA1 Message Date
Josh Cohen a3fba1ca1f
Add missing functionality for CanceledJobRegistry (#1560) 2021-09-13 18:48:57 +07:00
Josh Cohen b80045d615
Respect serializer (#1538)
* Add serializer where missing in code

* Fix cli

* Pass option to command

* Add tests for serializer option

* Merge branch 'master' into respect-serializer
- Update enqueue cli to resp. serializer

* Address @selwin's review
2021-08-24 07:40:29 +07:00
Selwin Ong 246d52b977
job.cancel() puts job into CanceledJobRegistry. (#1546)
* job.cancel() puts job into CanceledJobRegistry.

* Improve test coverage
2021-08-22 07:19:15 +07:00
Selwin Ong f9ddf7dca5
Requeueing job should clear job.exc_info (#1543) 2021-08-21 17:07:29 +07:00
rpkak 9737ffa95f
Make unstable tests more stable (#1523)
Co-authored-by: rpkak <rpkak@users.noreply.github.com>
2021-08-03 08:00:49 +07:00
Selwin Ong 5b5cfdf9ab
Jobs that get cleaned up should also be retried (#1467) 2021-05-15 20:04:52 +07:00
Selwin Ong 49b156ecc7
Job retry feature. Docs WIP (#1299)
* Initial implementation of Retry class

* Fixes job.refresh() under Python 3.5

* Remove the use of text_type in job.py

* Retry can be scheduled

* monitor_work_horse() should call handle_job_failure() with queue argument.

* Flake8 fixes

* Added docs for job retries
2020-07-23 19:58:22 +07:00
Selwin Ong cfe389bd65
FailedJobRegistry.requeue() resets job.started_at and job.ended_at (#1227) 2020-04-01 07:42:34 +07:00
Selwin Ong 636d6d2f54
registry.cleanup() now writes information to job.exc_info (#1226) 2020-03-31 21:14:10 +07:00
Selwin Ong ccfd4a02cb
Failed jobs will now auto expire (#1182) 2020-01-21 19:50:13 +07:00
Selwin Ong baa0cc268a
Job scheduling (#1163)
* First RQScheduler prototype

* WIP job scheduling

* Fixed Python 2.7 tests

* Added ScheduledJobRegistry.get_scheduled_time(job)

* WIP on scheduler's threading mechanism

* Fixed test errors

* Changed scheduler.acquire_locks() to instance method

* Added scheduler.prepare_registries()

* Somewhat working implementation of RQ scheduler

* Only call stop_scheduler if there's a scheduler present

* Use OSError rather than ProcessLookupError for PyPy compatibility

* Added `auto_start` argument to scheduler.acquire_locks()

* Make RQScheduler play better with timezone

* Fixed test error

* Added --with-scheduler flag to rq worker CLI

* Fix tests on Python 2.x

* More Python 2 fixes

* Only call `scheduler.start` if worker is run in non burst mode

* Fixed an issue where running worker with scheduler would fail sometimes

* Make `worker.stop_scheduler()` more resilient to errors

* worker.dequeue_job_and_maintain_ttl() should also periodically run maintenance tasks

* Scheduler can now work with worker in both burst and non burst mode

* Fixed scheduler logging message

* Always log scheduler errors when running

* Improve scheduler error logging message

* Removed testing code

* Scheduler should periodically try to acquire locks for other queues it doesn't have

* Added tests for scheduler.should_reacquire_locks

* Added queue.enqueue_in()

* Fixes queue.enqueue_in() in Python 2.7

* First stab at documenting job scheduling

* Remove unused methods

* Remove Python 2.6 logging compatibility code

* Remove more unused imports

* Added convenience methods to access job registries from queue

* Added test for worker.run_maintenance_tasks()

* Simplify worker.queue_names() and worker.queue_keys()

* Updated changelog to mention RQ's new job scheduling mechanism.
2020-01-04 10:14:52 +07:00
Selwin Ong af678243e1
Added `delete_job` argument to registry.remove()` (#1161) 2019-11-23 20:23:40 +07:00
Selwin Ong c4cbb3af2f
RQ v1.0! (#1059)
* Added FailedJobRegistry.

* Added job.failure_ttl.

* queue.enqueue() now supports failure_ttl

* Added registry.get_queue().

* FailedJobRegistry.add() now assigns DEFAULT_FAILURE_TTL.

* StartedJobRegistry.cleanup() now moves expired jobs to FailedJobRegistry.

* Failed jobs are now added to FailedJobRegistry.

* Added FailedJobRegistry.requeue()

* Document the new `FailedJobRegistry` and changes in custom exception handler behavior.

* Added worker.disable_default_exception_handler.

* Document --disable-default-exception-handler option.

* Deleted worker.failed_queue.

* Deleted "move_to_failed_queue" exception handler.

* StartedJobRegistry should no longer move jobs to FailedQueue.

* Deleted requeue_job

* Fixed test error.

* Make requeue cli command work with FailedJobRegistry

* Added .pytest_cache to gitignore.

* Custom exception handlers are no longer run in reverse

* Restored requeue_job function

* Removed get_failed_queue

* Deleted FailedQueue

* Updated changelog.

* Document `failure_ttl`

* Updated docs.

* Remove job.status

* Fixed typo in test_registry.py

* Replaced _pipeline() with pipeline()

* FailedJobRegistry no longer fails on redis-py>=3

* Fixes test_clean_registries

* Worker names are now randomized

* Added a note about random worker names in CHANGES.md

* Worker will now stop working when encountering an unhandled exception.

* Worker should reraise SystemExit on cold shutdowns

* Added anchor.js to docs

* Support for Sentry-SDK (#1045)

* Updated RQ to support sentry-sdk

* Document Sentry integration

* Install sentry-sdk before running tests

* Improved rq info CLI command to be more efficient when displaying lar… (#1046)

* Improved rq info CLI command to be more efficient when displaying large number of workers

* Fixed an rq info --by-queue bug

* Fixed worker.total_working_time bug (#1047)

* queue.enqueue() no longer accepts `timeout` argument (#1055)

* Clean worker registry (#1056)

* queue.enqueue() no longer accepts `timeout` argument

* Added clean_worker_registry()

* Show worker hostname and PID on cli (#1058)

* Show worker hostname and PID on cli

* Improve test coverage

* Remove Redis version check when SSL is used

* Bump version to 1.0

* Removed pytest_cache/README.md

* Changed worker logging to use exc_info=True

* Removed unused queue.dequeue()

* Fixed typo in CHANGES.md

* setup_loghandlers() should always call logger.setLevel() if specified
2019-03-30 09:13:56 +07:00
Darshan Rai ada2ad03ca modify zadd calls for redis-py 3.0 (#1016)
* modify zadd calls for redis-py 3.0

redis-py 3.0 changes the zadd interface that accepts a single
mapping argument that is expected to be a dict.
https://github.com/andymccurdy/redis-py#mset-msetnx-and-zadd

* change FailedQueue.push_job_id to always push a str

redis-py 3.0 does not attempt to cast values to str and is left
to the user.

* remove Redis connection patching

Since in redis-py 3.0, Redis == StrictRedis class, we no longer
need to patch _zadd and other methods.
Ref: https://github.com/rq/rq/pull/1016#issuecomment-441010847
2018-12-03 07:28:36 +07:00
Theofanis Despoudis d6b12c2402 Issue 872 (#954)
* Fixes #872 - Use -1 to indicate infinite ttl

* Fixes #872 Restored comma

* #872 Code review fix
2018-05-25 07:51:30 +07:00
Selwin Ong c639018fb9 Registry objects can be instantiated by passing a queue object. 2018-04-21 19:39:59 +07:00
Selwin Ong f760fcb20f job.delete() should cleans itself from FailedQueue and various registries. 2017-04-01 18:12:41 +07:00
Jannis Leidel c019662430
Allow passing backend classes (job, queue, worker, connection) from CLI and other APIs
This includes:

- a partial refactor of the CLI to organize the shared options
- extends the tests in areas where passing custom backend classes makes sense
- allow setting the core CLI options as env vars
- minor cosmetic changes here and there
2017-01-25 09:43:36 +01:00
Jannis Leidel 27e4f3a768
Small refactor to simplify registry key setup. 2017-01-25 09:42:39 +01:00
Selwin Ong e9d227c3df Dependent jobs are now correctly enqueued even if their parent jobs have result_ttl=0. 2016-05-06 11:41:01 +08:00
ahxxm b06f112cb0 fix tests
syntax: assertEquals -> assertEqual, assertNotEquals -> assertNotEqual
usage: status of worker and job now will use get/set method instead of property method
2015-11-09 12:34:27 +08:00
Selwin Ong faf9d3e668 Added clean_registries(queue) function to clean job registries related to that queue. 2015-05-23 08:46:00 +07:00
Selwin Ong 70d5f971bd Jobs from FinishedJobRegistry that are moved to FailedQueue should have "failed" as status. 2015-05-12 17:24:05 +08:00
Selwin Ong 105b95e9b8 Fixed an error in super call. 2015-01-26 08:37:03 +07:00
Selwin Ong de1cd8a83c Fixed test error in Python 3. 2015-01-26 08:31:58 +07:00
Selwin Ong dac0be6cc7 Added DeferredJobsRegistry to keep track of deferred jobs. 2015-01-23 17:53:27 +07:00
Nic Cope 09cab7a90d Allow maximum job age to be specified when cleaning up or getting expired jobs from a registry.
This ensures that all jobs cleaned from a started registry end up in the failed queue.
2014-12-02 16:37:20 -05:00
Nic Cope 8fa184b86b Ensure the FinishedJobRegistry honors an 'infinite' result_ttl of -1 2014-11-25 20:30:05 -05:00
Selwin Ong 2307bc8253 Added FinishedJobRegistry to monitor finished jobs. 2014-09-14 10:29:17 +07:00