* Add support for dependent jobs in enqueue_many
* Add module to register dependencies for multiple jobs
The get_ready_jobs function will process dependencies for an array of
jobs passed in. If any jobs' dependencies are already met, those jobs
are returned so they can be enqueued.
* Add check for jobs without dependencies
* Remove extra colon in dependencies key
This seems like a bug, but if I'm mistaken please let me know.
* Add bulk deferred jobs to Redis
Need to call queue.enqueue_job to create the job hash in redis. Since all of
these jobs are deferred, they won't be added to the queue and processed
by a worker.
* Revert "Remove extra colon in dependencies key"
This reverts commit 5ebf7a3500.
* Enqueue jobs without dependencies separately
Any jobs without dependencies will be enqueued before handling
* Fix enqueue_many return value
* Rename ready_jobs function
* Fix enqueue_many return value
* Instantiate job category arrays before if statement
* Execute pipe to enqueue jobs with met dependencies
* Add tests for enqueue_many with dependencies
* Change dependency sorting function name
* Use common kwargs dict to create jobs
* Remove redundant tests for dependent jobs
* Alphebetize imports
* Test job with met dependencies using enqueue_many
* Fix typo
* Format with black
* Sort imports
* Update linting configuration
This removes flake8 in favor of ruff, which also provides isort support, and
updates all files to be black, isort, and ruff compliant. This also adds black
and ruff checks to the tox and Github linting workflow.
* Tweak the code coverage config and calls
* test: check dependencies when enqueue via Queue.enqueue_job()
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
* fix: check dependencies when enqueue via Queue.enqueue_job()
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
Co-authored-by: Selwin Ong <selwin.ong@gmail.com>
---------
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
Co-authored-by: Selwin Ong <selwin.ong@gmail.com>
* test: Dependency list in depends_on
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
* fix: Dependency list in depends_on
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
---------
Signed-off-by: Simó Albert i Beltran <sim6@bona.gent>
* Add feature to enqueue dependents at the front of queues
* Add documentation for the Dependency(enqueue_at_front=...) parameter
* docs: Add `enqueue_at_front` to list of parameters for Dependency
* test: Update dependency test to not rely on Redis ordering
* refactor: Save enqueue_at_front boolean in job.meta instead of separate instance attr
* fix: Made enqueue_at_front an instance attribute instead of putting it inside meta
* Fix job.dependencies_are_met() if dependency is canceled
* Slightly better test coverage on dependencies_are_met()
* Fixed job.cancel(enqueue_dependent=True)
* added Dependency class with allow_failures
* Requested changes
* Check type before setting `job.dependency_allow_fail` within `Job.create`
* Set `job.dependency_allow_fail` within `Job.create`
* Added test to ensure persistence of `dependency_allow_fail`
* Removed typing and allow mixed list of ints and Job objects
* Convert dependency_allow_fail boolean to integer during serialization to avoid redis DataError
* Updated `test_multiple_dependencies_are_accepted_and_persisted` test to include `Dependency` cases
* Adding placeholder test to test actual behavior of new `Dependency` usage in `depends_on`
* Updated `test_job_dependency` to include cases using `Dependency`
* Added dependency_allow_fail logic to `Job.restore`
* Renamed `dependency_allow_fail` to a simpler `allow_failure`
* Update docs to add section about the new `Dependency` class and use-case
* Updated `Job.dependencies_are_met` logic to take `FAILED` and `STOPPED` jobs into account when `allow_failure=True`
* Updated `test_job_dependency` test. Still failing with `Dependency` case.
* Fix `allow_failure` type coercion in `Job.restore`
* Re-arrange tests, so default `Dependency.allow_failure` is before explicit `allow_failure=True`
* Fixed Dependency, so it works correctly when allow_failure=True
* Attempt to execute pipeline prior to queueing a failed job's dependents. test_create_and_cancel_job_enqueue_dependents_in_registry test now passes.
* Added `Depedency` test utilizing multiple dependencies
* Removed irrelevant on_success and on_failure keyword arguments in example
* Replaced use of long_running_job
* Add test to verify `Dependency.jobs` contraints
* Suppress connection error in handle_job_failure
* test_dependencies have passed
* All tests pass if enqueue_dependents called without pipeline.watch()
* All tests now pass
* Removed print statements
* Cleanup Dependency implementation
* Renamed job.allow_failure to job.allow_dependency_failures
Co-authored-by: mattchan <mattchan@tencent.com>
Co-authored-by: Mike Hill <mhilluniversal@gmail.com>
Ethan Wolinsky