Rooba
/
rq
mirror of https://github.com/rq/rq.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,629 Commits 18 Branches 81 Tags 12 MiB
Commit Graph

479 Commits

Author SHA1 Message Date
Selwin Ong 6559b0ffd7
Replace "timeout" argument in queue.enqueue() with "job_timeout" (#1010) 2018-11-22 19:09:25 +07:00
Selwin Ong ad66d872f0 Fixed a unicode test. 2018-10-26 20:27:36 +07:00
Selwin Ong 47d291771f
SimpleWorker's ttl must always be longer than jobs. (#1002) 2018-10-20 11:49:14 +07:00
Paul Robertson e86fb57366 add is_async property to queue (#982) 2018-08-08 06:41:15 +07:00
chevell c2b939d2df Replace 'async' keyword with 'is_async' for Queue objects (#977) 
* Replaced async keyword with is_async in the Queue class to fix reserved keyword syntax errors in Python 3.7

* Updated tests to use is_async keyword when instantiating Queue objects

* Updated docs to reference is_async keyword for Queue objects

* Updated tox.ini, setup.py and .travis.yml with references to Python 3.7
 2018-07-07 08:50:33 +07:00
Theofanis Despoudis 875cc27c2f #908 Using a timeout string value for job works (#955) 
Fixes https://github.com/rq/rq/issues/908
 2018-05-27 15:27:56 +07:00
Theofanis Despoudis d6b12c2402 Issue 872 (#954) 
* Fixes #872 - Use -1 to indicate infinite ttl

* Fixes #872 Restored comma

* #872 Code review fix
 2018-05-25 07:51:30 +07:00
Selwin Ong 531fde8e3c worker.main_work_horse should always return 0 2018-05-07 18:18:45 +08:00
Thomas Kriechbaumer 3133d94b58 add periodic worker heartbeats (#945) 
* add periodic worker heartbeats

fixes #944

* improve worker default option handling
 2018-05-06 09:08:19 +08:00
Selwin Ong 63a04d275e Use dbsize() to test for empty Redis database 2018-05-01 07:20:45 +07:00
Selwin Ong c639018fb9 Registry objects can be instantiated by passing a queue object. 2018-04-21 19:39:59 +07:00
stj 487ef72f21 Define redis key prefix as class variable (#939) 
* Define redis key prefix as class variable

Some prefixes were hardcoded in several places. This made it hard to
use custom prefixes via subclasses.

Resolves #920

* fixup! Define redis key prefix as class variable
 2018-04-06 07:16:42 +07:00
Christophe Olinger a6eb5d37ee Delete dependents of job explicitely (#916) 
* Initial take on delete_dependents

* Add tests including corner cases

* No need to canel dependents since they are not in a queue yet anyway

* The dependents keys can be deleted in all cases

* Update tests to included saved jobs in the deletion tests

* Correctly use pipeline in cancel method

* Unused connection

* Include dependents into dict format of job

* Add TODO

* Address comments from selwin

* Delete dependents key in redis if delete_dependents is called on its own

* Address recent comments from selwin

* Small change to trigger travis

* Remove TODO referring to canceled job state

* Remove dependent_ids from to_dict

* Address recent comments from selwin
 2018-01-24 06:40:15 +07:00
Selwin Ong 7a3c85f185
Added the ability to fetch workers by queue (#911) 
* job.exc_info is now compressed.

* job.data is now stored in compressed format.

* Added worker_registration.unregister.

* Added worker_registration.get_keys().

* Modified Worker.all(), Worker.all_keys() and Worker.count() to accept "connection" and "queue" arguments.
 2017-12-18 16:23:28 +07:00
John Lucas 34c403ec8d Add meta to decorator, move depends_on + at_front to decorator (#892) 2017-12-06 21:34:26 +07:00
Samuel Colvin df571e14fd improve logging in worker.py (#902) 
* improve logging in worker

* tests for log_result_lifespan
 2017-11-23 12:57:15 +07:00
Selwin Ong f500186f3d
Job compression (#907) 
job.exc_info and job.data is now stored in compressed format in Redis.

* job.data is now stored in compressed format.
 2017-11-23 12:55:54 +07:00
vanife ff36e0656e Fixed an issue where `birth` not present in Redis (#901) 
* Fixed an issue where `birth` not present in Redis

Fixed an issue where worker.refresh() may fail if `birth` is not present in Redis

* added test coverage
 2017-11-17 09:46:47 +07:00
Selwin Ong 7b9c3b6b66 Fixed an issue where worker.refresh() may fail if last_heartbeat is not present in Redis. 2017-11-03 12:59:28 +07:00
Selwin Ong 1d7b5e834b
Worker statistics (#897) 
* First stab at implementing worker statistics.

* Moved worker data restoration logic to worker.refresh().

* Failed and successfull job counts are now properly incremented.

* Worker now keeps track of total_working_time

* Ensure job.ended_at is set in the case of unhandled job failure.

* handle_job_failure shouldn't crash if job.started_at is not present.
 2017-10-28 17:45:49 +07:00
Selwin Ong 92c88d3f4d Merge pull request #878 from theodesp/Issue-731 
Fixed #731 - Support for deleting Queues
 2017-10-24 08:39:50 +07:00
Theo c095fe1825 Fixed #731 - Code review issues. Added delete_jobs parameter and pipelining. 2017-10-04 11:29:33 +01:00
Samuel Colvin 260fd84f51 add milliseconds into timestamps, fix #721 2017-09-09 20:28:53 +01:00
Selwin Ong 19bc288378 Merge pull request #877 from theodesp/Issue-809 
Fixed #809 - Added tests for various cli config parameters
 2017-09-09 20:42:31 +07:00
Theo 160fe99323 Fixed #731 - Support for deleting Queues 2017-09-08 12:59:36 +01:00
Theo 0fab93d683 Fixed #809 - Added tests for various cli config parameters 2017-09-08 12:21:30 +01:00
Theo 261f4ac3d5 Fixed #866 - Flak8 errors 2017-09-08 10:30:25 +01:00
Theo 096c5ad3c2 Fixed #866 - Flak8 errors 2017-09-08 10:28:10 +01:00
Samuel Colvin 423da3683c remove python 2.6 support 2017-09-05 21:07:40 +01:00
Theo ee64114e6e Fixed #870 Improved test coverage for connections.py and utils.py 2017-08-31 12:23:17 +01:00
Selwin Ong 0efb87a46b Fixed test error in Python 3. 2017-07-28 13:49:38 +07:00
Selwin Ong 54bc04bb45 job.save() shouldn't crash on unpickleable return value. 2017-07-28 13:36:30 +07:00
Alexey Katichev 09697e567f revert back job.cleanup changes 2017-05-22 14:31:22 +03:00
Alexey Katichev 3596449cc0 remove implicit cleanup call from job.save 2017-05-07 16:57:12 +03:00
Alexey Katichev a0113c83cf introduce job.update_meta() to store updated meta to Redis (#823) 
* introduce job.update_meta() to store updated meta to Redis

This closes nvie/rq#811

* rename update_meta to save_meta
 2017-04-30 20:43:31 +07:00
Selwin Ong dc45ab8799 Worker.find_by_key should use hmget instead of repeated hget calls. (#826) 2017-04-29 20:36:17 +07:00
luojiebin cd529d0ce1 Fixed issue#72 (#818) 
* Added a custom exception for timeout transfer

* Added a util to transfer timeout to a united format

* Transfer timeout format when creating a queue or enqueue jobs

* Fixed typos

* Fixed bug in transfer_timeout function

* Added test for function transfer_timeout

* Updated transfer_timeout to allow uppercase unit

* Renamed function in utils
 2017-04-28 10:13:42 +07:00
Peng Liu b7d4b4ec1b Solve the UnicodeDecodeError while decode literal things. (#817) 
* Solve the UnicodeDecodeError while decode literal things.

* Add test case for when worker result is a unicode or str object that other than
pure ascii content.
 2017-04-19 11:16:24 +07:00
Felipe Lacerda cab89254b5 Make `Queue.enqueue_job()` execute immediately if `async=False` (#798) 
Currently, the job is being performed inside `enqueue_call()`, which
means that `async=False` has no effect if `enqueue_job()` is called
directly. This commit fixes that.
 2017-04-15 18:02:48 +07:00
Selwin Ong f6b4c286c9 Merge pull request #757 from jaywink/fix-unicode-decode-error 
Fix UnicodeDecodeError when failing jobs
 2017-04-04 05:59:07 +07:00
Selwin Ong f760fcb20f job.delete() should cleans itself from FailedQueue and various registries. 2017-04-01 18:12:41 +07:00
Samuel Colvin fd9babe8ce correct heroku worker exit logic 
as per @Chronial's comment on b4b99f3
 2017-03-09 11:11:19 +00:00
Jason Robinson 213969742e Fix UnicodeDecodeError when failing jobs 
Worker handle_exception and move_to_failed_queue couldn't handle a situation where the exception raised had non-ascii characters. This caused a UnicodeDecodeError when trying to format the exception strings.

If on Python 2, ensure strings get decoded before building the exception string.

Closes #482
 2017-02-10 11:59:25 +02:00
Selwin Ong 83007b2074 Merge pull request #786 from jezdez/backend-class-overrides 
Allow passing backend classes from CLI and other APIs
 2017-02-03 15:56:19 +07:00
Benjamin Root 30a7ab4899 Add similar test for when the job fails 2017-01-27 13:30:51 -05:00
Benjamin Root f68aeff481 Added integration test for the metadata persistence feature 2017-01-27 13:30:51 -05:00
Jannis Leidel c019662430
Allow passing backend classes (job, queue, worker, connection) from CLI and other APIs 
This includes:

- a partial refactor of the CLI to organize the shared options
- extends the tests in areas where passing custom backend classes makes sense
- allow setting the core CLI options as env vars
- minor cosmetic changes here and there
 2017-01-25 09:43:36 +01:00
Jannis Leidel 27e4f3a768
Small refactor to simplify registry key setup. 2017-01-25 09:42:39 +01:00
Samuel Colvin 763a304ec6 cope with python 2.6 yawwwwwwn 2016-12-20 16:47:10 +00:00
Samuel Colvin f50bdb31bf skip test_run_scheduled_access_self on pypy 2016-12-20 16:34:13 +00:00
Samuel Colvin b4b99f30b0 allow for already dead horse on heroku worker termination 2016-12-20 16:03:21 +00:00
Julien Surloppe dc3bba9362 Another check on failed status and test 2016-12-03 22:32:12 +01:00
Selwin Ong 2428370dfd Merge pull request #763 from jezdez/skip-heroku-macos 
Stop running some Heroku tests on macOS since it’s missing the appropriate signals
 2016-11-30 09:06:12 +07:00
Jannis Leidel fcec17c8e1
Stop running some Heroku tests on macOS since it’s missing the appropriate signals 
Refs issue #584.
 2016-11-29 09:31:10 +01:00
Stefan Hammer f9bff3d12b Merge branch 'master' into 739_fix_race_condition 2016-11-19 12:44:13 +01:00
Vincent Driessen af6ce54ea4 Fix PEP8 complaints 2016-11-18 08:42:43 +01:00
Stefan Hammer a0cee2d2a0 refactored worker code 
Moved code into a new handle_job_success() method and reduced context of used
pipelines.
 2016-09-22 21:41:51 +02:00
Samuel Colvin afc7469c27 fetch_job - check correct queue, fix #728 2016-09-20 11:09:14 +01:00
Stefan Hammer 301e5c927b Raise an exception if a given dependency does not exist 
Adapted some tests to the change: the dependency has to be saved first.
 2016-08-25 20:10:17 +02:00
Stefan Hammer 44f98693c7 added a test for the race condition 2016-08-13 20:33:53 +02:00
Samuel Colvin bbf50f4893 use compat for PY2 test and add tests 2016-07-25 12:08:45 +01:00
Yannis Spiliopoulos 4a8aa0921f Set test timeout to original value 2016-07-24 12:46:14 -04:00
Yannis Spiliopoulos fbb29ec333 Investigating timeouts 2016-07-24 09:58:44 -04:00
Yannis Spiliopoulos 94d5caed15 Make clearer the puprose of test 2016-07-24 09:22:29 -04:00
Yannis Spiliopoulos 3362fe2ba5 Test killing work_horse after we start monitoring 2016-06-30 13:32:48 -04:00
Yannis Spiliopoulos c00d3681f9 Failing test to demonstrate issue #702 
Test that demonstrates that if a work-horse process is terminated unexpectedly
the job being processed could be stuck at the "Started" state
(https://github.com/nvie/rq/issues/702)
 2016-06-30 13:32:48 -04:00
Samuel Colvin 0e26db9e08 correct wording in docstring and tests 2016-06-19 12:46:23 +01:00
Samuel Colvin 9f9c887645 better function names and process double SIGRTMIN 2016-06-19 12:46:23 +01:00
Samuel Colvin 2b544e5b17 add tests to HerokuWorker 2016-06-19 12:46:23 +01:00
Arnold Krille 8e99706b16 run python 3.5 on travis, adopt timeouts 
Also
- Report the five slowest tests to watch for risk of timeout
- Double timeouts.
  Maybe that helps pypy on travis to finish successfully.
 2016-06-15 21:35:02 +02:00
Arnold Krille df22f127eb Test the worker in its own subprocess 
- run with an empty queue
- schedule one job (which uses get_current_connection and get_current_job) and
run `rqworker`
- schedule a job that itself schedules `access_self` and run `rqworker`
- Make sure the job didn't fail by assuring the failed queue is still empty
  afterwards.
- Install this package locally when running in travis.
  This actually unifies the behaviour of tox and travis as tox also builds the
  package and then installs it into each test environment.
- fix flake8 (as run by tox)
 2016-06-15 21:35:02 +02:00
Javier Lopez e92b57d128 test_job.py: Add cancel test on failed queue 2016-05-16 12:12:03 +02:00
Selwin Ong 766bb60006 Merge branch 'dependent-jobs-with-result-ttl' 2016-05-06 11:42:50 +08:00
Selwin Ong e9d227c3df Dependent jobs are now correctly enqueued even if their parent jobs have result_ttl=0. 2016-05-06 11:41:01 +08:00
Selwin Ong 8b7b0e5584 Merge pull request #701 from jlopex/javi/fix_issue_700 
Fixes issue #700
 2016-05-05 06:35:32 +07:00
Javier Lopez 091c2568c9 test_job.py: Add test for cancel_job function 2016-05-03 15:17:46 +02:00
amyangfei 724c844378 Don't call job.cancel if job has finished 2016-04-28 10:29:31 +08:00
Arnold Krille 9df0a853d8 Fix indentation and newlines according to flake8 2016-03-13 18:43:32 +01:00
Selwin Ong e1e4c00a67 Merge pull request #659 from javimb/javimb/decorator-ttl 
Add ttl argument to decorator
 2016-02-24 08:22:57 +07:00
Antoine Leclair 81679a35d9 Merge branch 'master' into custom-queue-class 
Conflicts:
	tests/test_worker.py
 2016-02-22 09:20:38 -05:00
Javi Imbernon 1d6c2b98b7 Add ttl argument to decorator 2016-02-16 15:15:49 +01:00
Antoine Leclair cc1eb9c52b Remove test that failed on Travis CI 2016-02-15 23:10:57 -05:00
Antoine Leclair 173417f337 Remove test that failed on Travis CI 2016-02-15 23:10:01 -05:00
Antoine Leclair 05ed85804c Worker accepts custom queue class 2016-02-15 22:42:24 -05:00
Antoine Leclair 7275f62737 Add tests for custom job class in worker 2016-02-15 22:32:06 -05:00
Antoine Leclair 5c72417cda Do not use deprecated assertEquals 2016-02-15 22:31:02 -05:00
Antoine Leclair 2f36cedd50 Typo in test docstring 2016-02-15 22:30:07 -05:00
Selwin Ong 640b2e31a0 Merge pull request #633 from olingerc/warmshutdown 
Save date on which a busy worker receives a warm shutdown request
 2016-02-15 20:44:30 +07:00
orangain e8165fdddf Accept byte strings as the first argument of Worker() in Python 2 
This make it easy to write Python 2/3 compatible code.
e.g.

    Worker(sys.argv[1:])

    # Without from __future__ import unicode_literals
    Worker(['high', 'normal', 'low'])
 2016-02-04 22:05:38 +09:00
Eduard Carreras 89facfada6 Test to ensure job is finished in not async mode 2015-12-28 11:02:27 +01:00
Christophe Olinger b47c8efe27 Add test cases for shutdown_requested_date 2015-12-16 15:25:03 +01:00
Christophe Olinger 3e586bd6d6 Add remove current_job based on the correct upstream master branch 2015-12-15 08:25:05 +01:00
Selwin Ong 2485334100 Merge pull request #609 from tornstrom/master 
Allow meta when enqueing
 2015-12-06 15:13:46 +07:00
Tornstrom 50a114a0a8 Allow meta when enqueing 2015-11-30 09:42:49 -06:00
Selwin Ong 5afd1a90e5 Merge pull request #589 from samuelcolvin/job-started_at 
add job.started_at
 2015-11-28 11:26:24 +07:00
Selwin Ong 8bbd833855 Merge pull request #600 from glaslos/cancel_remove 
Cancel and Delete differences
 2015-11-23 06:13:05 +07:00
ahxxm b06f112cb0 fix tests 
syntax: assertEquals -> assertEqual, assertNotEquals -> assertNotEqual
usage: status of worker and job now will use get/set method instead of property method
 2015-11-09 12:34:27 +08:00
glaslos 0a6df13d9d delete dependents and delete in cleanup. Fixed tests. 2015-11-06 11:32:43 +01:00
Samuel Colvin 3aa7aebf87 remove assert_dt_between from worker tests 2015-10-27 15:56:00 +00:00
Samuel Colvin 1349e90ba2 test docstrings, and sentinel test 2015-10-14 21:22:36 +01:00
Samuel Colvin 082e61a212 add job.started_at 2015-10-14 21:10:01 +01:00
Samuel Colvin fa48751fed correct SLOW env var, run slow tests on ci 2015-10-14 20:08:55 +01:00
Samuel Colvin aada162a4d worker shutdown tests 2015-10-14 19:54:31 +01:00
Selwin Ong e538512c79 Merge branch 'master' of github.com:nvie/rq 
Conflicts:
	tests/test_job.py
 2015-09-22 18:18:29 +07:00
Selwin Ong 0e4112da7a Modified patch_connection to allow Redis connection to be easily mocked. 2015-09-22 18:17:02 +07:00
Selwin Ong 1a089887c0 Merge pull request #564 from jlopex/javi/fix_dependent_jobs_on_different_queues 
support multiple queues on dependent jobs
 2015-09-01 14:33:16 +07:00
Vincent Driessen c1a4780d10 Fix PEP8 complaints 2015-08-25 09:12:34 +02:00
Javier Lopez a2d0e4f933 Clarify test_enqueue_dependents_on_multiple_queues 2015-08-24 17:15:29 +02:00
Javier Lopez c76c59133b Fix ValueError exception on relative import 
test_job raised an exception:

ValueError: Attempted relative import in non-package
 2015-08-24 17:15:29 +02:00
Javier Lopez c136209804 Add test_enqueue_dependents_on_mulitple_queues test 2015-08-24 17:15:29 +02:00
Selwin Ong e6a499ada8 Merge pull request #561 from selwin/async-bug 
Sync jobs should be cleaned up after execution.
 2015-07-10 09:00:28 +07:00
Selwin Ong edd139d86f Sync jobs should be cleaned up after execution. 2015-07-10 08:58:38 +07:00
Selwin Ong 3c3646cf91 Merge branch 'master' into exception_handling 
Conflicts:
	tests/fixtures.py
 2015-07-03 17:06:20 +07:00
Selwin Ong 8bfbeac67d Running rqworker without specifiying --exception-handler shouldn't override default behavior. 2015-07-03 17:04:01 +07:00
Vincent Driessen 3b67894489 Rename variable in test 2015-06-03 10:49:28 +02:00
Vincent Driessen ec0e04727b Rename `maintenance_date` -> `last_cleaned_at` 2015-06-03 10:13:38 +02:00
Vincent Driessen 4d0ae5da98 Clean up type checking of Worker constructor 2015-06-03 09:52:26 +02:00
Marcus Martins ab6c129833 Fix broken tests 2015-05-28 23:31:43 -03:00
Marcus Martins 5b8726ad2d Fixes #502 
Fixes some broken tests and misbehaviour with ttls. There was a temporal
coupling between saving the job and setting its expires parameter.
 2015-05-28 23:31:43 -03:00
Robert Brownstein 3d8faa0e5d Added proper conditional behavior to unicode fixture for python 3 2015-05-28 17:23:04 -04:00
Robert Brownstein 303f4ed47c Added test coverage for unicode keyword argument support in method signatures (#536) 2015-05-28 17:15:18 -04:00
Selwin Ong f370f79819 Merge pull request #534 from selwin/registry-maintenance 
Worker automatically cleans job registries every hour
 2015-05-28 08:44:10 +08:00
Selwin Ong c3767e28e2 Worker now runs maintenance tasks every hour and on startup. 2015-05-23 10:08:04 +07:00
Selwin Ong 5782ac10c4 Added worker.clean_registries(). 2015-05-23 09:01:25 +07:00
Selwin Ong faf9d3e668 Added clean_registries(queue) function to clean job registries related to that queue. 2015-05-23 08:46:00 +07:00
RyanMTB 9a00b0eca6 Updated Worker API 2015-05-20 21:48:13 -07:00
Selwin Ong 70d5f971bd Jobs from FinishedJobRegistry that are moved to FailedQueue should have "failed" as status. 2015-05-12 17:24:05 +08:00
Vincent Driessen f651a72a50 Make clearer what are fixture functions 2015-04-12 11:27:58 +02:00
Vincent Driessen df4d4c8d5d Make test cases more explicit 2015-04-12 11:15:55 +02:00
Vincent Driessen 82df2ee689 Fix PEP8 issue 2015-04-12 10:29:20 +02:00
Selwin Ong cff6fbff41 Merge pull request #477 from glaslos/ttl_tests_fixes 
TTL tests and fixes
 2015-03-25 08:51:30 +07:00
glaslos 071c47dacb hard-coded TTL to -1 on job.perform() 
Removed merge artifacts
 2015-03-19 15:32:01 +01:00
Serhii Maltsev 91f263d8e0 change try/except in python2/3 compatibility to to_text() 2015-03-19 09:48:04 +02:00
Tamir Duberstein 985a2664a4 Prevent `Queue#dequeue` from blowing the stack 
In the case of many sequential jobs having been deleted, a recursive
implementation of `Queue#dequeue` is prone to blowing the stack in the
absence of tail-recursion support. Change the implementation from
recursive to iterative to work around this issue in CPython.
 2015-03-16 23:36:49 -07:00
glaslos 3d4d6a86d5 persist the job right before execution 2015-02-27 16:07:44 +01:00
glaslos 24d5e08992 Merge branch 'master' into ttl_tests_fixes 
Conflicts:
	tests/test_job.py
 2015-02-27 15:38:45 +01:00
Selwin Ong 30b21ef06e Merge pull request #465 from marklap/workerbirthdeath 
add birth_date and death_date properties to Worker
 2015-02-27 08:51:55 +07:00
Mark LaPerriere 636a537fa7 updates addressing @selwin comments for PR #465 2015-02-26 09:41:20 -05:00
Bradley Young d56b5424c8 merging master and fixing conflicts 2015-02-24 10:36:18 -05:00
alternativshik 219f21b637 Allow non-ASCII characters in arguments 
get_call_string() failed if any arguments contained non-ASCII strings.

Fixes #406
 2015-02-24 11:53:47 +00:00
Selwin Ong 1005222371 Restored the ability to specify connection params in config 2015-02-23 17:28:51 +07:00
Mark LaPerriere 450c5969aa call register birth in the register birth test 2015-02-05 23:12:33 -05:00
Mark LaPerriere af5a8624a6 add tests for birth and death worker methods 2015-02-05 23:05:10 -05:00
Vincent Driessen b8d425b318 Various PEP8 fixes. 2015-01-30 09:24:51 +01:00
Vincent Driessen fac2b10309 Auto-sort imports using isort. 2015-01-30 09:24:51 +01:00
glaslos 646f1cda22 Merge branch 'master' into ttl_tests_fixes 
Conflicts:
	rq/queue.py
 2015-01-29 14:24:18 +01:00
glaslos 3af0e95ce5 adding two tests to ensure ttl behaviour 2015-01-29 14:19:50 +01:00
Selwin Ong 105b95e9b8 Fixed an error in super call. 2015-01-26 08:37:03 +07:00
Selwin Ong de1cd8a83c Fixed test error in Python 3. 2015-01-26 08:31:58 +07:00
Bradley Young d3e4fb567b merge && updates to add multiple exception handlers. 2015-01-24 17:22:20 -05:00