Commit Graph

35 Commits

Author SHA1 Message Date
Selwin Ong baa0cc268a
Job scheduling (#1163)
* First RQScheduler prototype

* WIP job scheduling

* Fixed Python 2.7 tests

* Added ScheduledJobRegistry.get_scheduled_time(job)

* WIP on scheduler's threading mechanism

* Fixed test errors

* Changed scheduler.acquire_locks() to instance method

* Added scheduler.prepare_registries()

* Somewhat working implementation of RQ scheduler

* Only call stop_scheduler if there's a scheduler present

* Use OSError rather than ProcessLookupError for PyPy compatibility

* Added `auto_start` argument to scheduler.acquire_locks()

* Make RQScheduler play better with timezone

* Fixed test error

* Added --with-scheduler flag to rq worker CLI

* Fix tests on Python 2.x

* More Python 2 fixes

* Only call `scheduler.start` if worker is run in non burst mode

* Fixed an issue where running worker with scheduler would fail sometimes

* Make `worker.stop_scheduler()` more resilient to errors

* worker.dequeue_job_and_maintain_ttl() should also periodically run maintenance tasks

* Scheduler can now work with worker in both burst and non burst mode

* Fixed scheduler logging message

* Always log scheduler errors when running

* Improve scheduler error logging message

* Removed testing code

* Scheduler should periodically try to acquire locks for other queues it doesn't have

* Added tests for scheduler.should_reacquire_locks

* Added queue.enqueue_in()

* Fixes queue.enqueue_in() in Python 2.7

* First stab at documenting job scheduling

* Remove unused methods

* Remove Python 2.6 logging compatibility code

* Remove more unused imports

* Added convenience methods to access job registries from queue

* Added test for worker.run_maintenance_tasks()

* Simplify worker.queue_names() and worker.queue_keys()

* Updated changelog to mention RQ's new job scheduling mechanism.
2020-01-04 10:14:52 +07:00
Selwin Ong c4cbb3af2f
RQ v1.0! (#1059)
* Added FailedJobRegistry.

* Added job.failure_ttl.

* queue.enqueue() now supports failure_ttl

* Added registry.get_queue().

* FailedJobRegistry.add() now assigns DEFAULT_FAILURE_TTL.

* StartedJobRegistry.cleanup() now moves expired jobs to FailedJobRegistry.

* Failed jobs are now added to FailedJobRegistry.

* Added FailedJobRegistry.requeue()

* Document the new `FailedJobRegistry` and changes in custom exception handler behavior.

* Added worker.disable_default_exception_handler.

* Document --disable-default-exception-handler option.

* Deleted worker.failed_queue.

* Deleted "move_to_failed_queue" exception handler.

* StartedJobRegistry should no longer move jobs to FailedQueue.

* Deleted requeue_job

* Fixed test error.

* Make requeue cli command work with FailedJobRegistry

* Added .pytest_cache to gitignore.

* Custom exception handlers are no longer run in reverse

* Restored requeue_job function

* Removed get_failed_queue

* Deleted FailedQueue

* Updated changelog.

* Document `failure_ttl`

* Updated docs.

* Remove job.status

* Fixed typo in test_registry.py

* Replaced _pipeline() with pipeline()

* FailedJobRegistry no longer fails on redis-py>=3

* Fixes test_clean_registries

* Worker names are now randomized

* Added a note about random worker names in CHANGES.md

* Worker will now stop working when encountering an unhandled exception.

* Worker should reraise SystemExit on cold shutdowns

* Added anchor.js to docs

* Support for Sentry-SDK (#1045)

* Updated RQ to support sentry-sdk

* Document Sentry integration

* Install sentry-sdk before running tests

* Improved rq info CLI command to be more efficient when displaying lar… (#1046)

* Improved rq info CLI command to be more efficient when displaying large number of workers

* Fixed an rq info --by-queue bug

* Fixed worker.total_working_time bug (#1047)

* queue.enqueue() no longer accepts `timeout` argument (#1055)

* Clean worker registry (#1056)

* queue.enqueue() no longer accepts `timeout` argument

* Added clean_worker_registry()

* Show worker hostname and PID on cli (#1058)

* Show worker hostname and PID on cli

* Improve test coverage

* Remove Redis version check when SSL is used

* Bump version to 1.0

* Removed pytest_cache/README.md

* Changed worker logging to use exc_info=True

* Removed unused queue.dequeue()

* Fixed typo in CHANGES.md

* setup_loghandlers() should always call logger.setLevel() if specified
2019-03-30 09:13:56 +07:00
Selwin Ong ad66d872f0 Fixed a unicode test. 2018-10-26 20:27:36 +07:00
Selwin Ong 47d291771f
SimpleWorker's ttl must always be longer than jobs. (#1002) 2018-10-20 11:49:14 +07:00
Theo 096c5ad3c2 Fixed #866 - Flak8 errors 2017-09-08 10:28:10 +01:00
Theo ee64114e6e Fixed #870 Improved test coverage for connections.py and utils.py 2017-08-31 12:23:17 +01:00
Peng Liu b7d4b4ec1b Solve the UnicodeDecodeError while decode literal things. (#817)
* Solve the UnicodeDecodeError while decode literal things.

* Add test case for when worker result is a unicode or str object that other than
pure ascii content.
2017-04-19 11:16:24 +07:00
Benjamin Root 30a7ab4899 Add similar test for when the job fails 2017-01-27 13:30:51 -05:00
Benjamin Root f68aeff481 Added integration test for the metadata persistence feature 2017-01-27 13:30:51 -05:00
Samuel Colvin bbf50f4893 use compat for PY2 test and add tests 2016-07-25 12:08:45 +01:00
Samuel Colvin 0e26db9e08 correct wording in docstring and tests 2016-06-19 12:46:23 +01:00
Samuel Colvin 2b544e5b17 add tests to HerokuWorker 2016-06-19 12:46:23 +01:00
Arnold Krille df22f127eb Test the worker in its own subprocess
- run with an empty queue
- schedule one job (which uses get_current_connection and get_current_job) and
run `rqworker`
- schedule a job that itself schedules `access_self` and run `rqworker`
- Make sure the job didn't fail by assuring the failed queue is still empty
  afterwards.
- Install this package locally when running in travis.
  This actually unifies the behaviour of tox and travis as tox also builds the
  package and then installs it into each test environment.
- fix flake8 (as run by tox)
2016-06-15 21:35:02 +02:00
Selwin Ong 3c3646cf91 Merge branch 'master' into exception_handling
Conflicts:
	tests/fixtures.py
2015-07-03 17:06:20 +07:00
Robert Brownstein 3d8faa0e5d Added proper conditional behavior to unicode fixture for python 3 2015-05-28 17:23:04 -04:00
Robert Brownstein 303f4ed47c Added test coverage for unicode keyword argument support in method signatures (#536) 2015-05-28 17:15:18 -04:00
Vincent Driessen df4d4c8d5d Make test cases more explicit 2015-04-12 11:15:55 +02:00
glaslos 3af0e95ce5 adding two tests to ensure ttl behaviour 2015-01-29 14:19:50 +01:00
Bradley Young 5caccaabfe Adding optional list handling to the exc_handler option in Worker.
Adding command line --exception_handler option (with multiple entries allowed) to `rq worker`
Added tests for command line options.
2014-11-12 09:04:40 -05:00
Vincent Driessen 4b7c59d68d Fix PEP8 complaints. 2014-09-08 20:29:24 +02:00
foxx 16ee71f26d Use class for test worker, and performs PID check in unit tests, #412 2014-09-08 11:37:14 +01:00
Vincent Driessen 79db282879 Support enqueueing callable classes.
Fixes #388.
2014-07-26 09:34:01 +02:00
Vincent Driessen 38ec259b6e Enable the most modern Python syntax. 2014-05-05 10:50:02 +02:00
Selwin Ong 4f918041e3 Rearranged how explicit kwargs are passed into queue.enqueue(). Fixes #322 2014-04-11 16:47:31 +07:00
Vincent Driessen a5dff6659c Replace the Calculator fixture by a Number fixture.
This makes the tests a little more realistic, since I want to add a test
for class methods.
2013-04-19 21:18:25 +02:00
Vincent Driessen 372de4b45a Implement the get_current_job() function.
This fixes #125.
2012-09-02 22:38:46 +02:00
Selwin Ong a4f1de358f Raise a StopException when Control+C is pressed. 2012-07-26 09:36:50 +07:00
Vincent Driessen d697ddb93a Resolve connections early.
Fixes #101.
2012-07-24 12:37:59 +02:00
Vincent Driessen c7225ba257 Minor bug in test case. 2012-07-24 12:30:37 +02:00
Selwin Ong 8c3292d35b Make "queue" argument in job decorator required.
job decorator now uses Queue's "enqueue_call" method.
2012-07-24 16:03:49 +07:00
Selwin Ong bdc1af28d1 Added a job decorator. 2012-07-23 13:25:31 +07:00
Omar Khan 03bd49511d Allow instance methods to be enqueued
Only works for picklable instances
2012-07-17 11:48:41 +01:00
Vincent Driessen e807748ee6 Test the timing out of jobs.
Really looking for a way to speed up this test.  It takes up a whole
second doing nothing now, really.
2012-02-22 18:01:14 +01:00
Vincent Driessen 9ac9c23412 Flake8. 2012-02-22 17:39:14 +01:00
Vincent Driessen f07d28db86 Organize test fixtures into a separate file. 2012-02-15 21:55:06 +01:00