* [fix#1726] Use boto3 for SQS async requests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The redis-py 4.5.2 changes the UnixDomainSocketConnection class so now
it inherits from AbstractConnection:
https://github.com/redis/redis-py/releases/tag/v4.5.2
This patch makes sure that the health_check_interval parameter is
checked for the __init__ method of the main class and also the bases, so
it doesn't fail with the newer version of redis-py.
* Fix#1618: avoid re-fetching queue URL when we already have it
`_get_from_sqs` was unnecessarily calling `get_queue_url` every
time even though the only place which calls `_get_from_sqs`
(that is `_get_async`) actually already knows the queue URL.
This change avoids hundreds of `GetQueueUrl` AWS API calls per hour
when using this SQS backend with celery.
Also `connection` is set by the one-and-only caller (and `queue` is
actually the queue name string now anyway so couldn't ever have
`.connection`) so remove the None default and unused fallback code.
* Clarify that `_new_queue` returns the queue URL
It seems that prior to 129a9e4ed0 it returned a queue
object but this is no longer the case so update comments
variable names accordingly to make it clearer.
Also remove the incorrect fallback which cannot
be correct any more given the return value has to
be the queue URL which must be a string.
* Unit test coverage for SQS async codepath
This key code path (which as far as I can see is
the main route when using celery with SQS) was
missing test coverage.
This test adds coverage for:
`_get_bulk_async` -> `_get_async` -> `_get_from_sqs`
* Main
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix
* Trigger Build
* Fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix
* Fix
* noqas
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove unused noqa
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* re-add import
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fixes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* use pytest-freezer (#1683)
* Main
* Trigger Build
* Fixes
* remove
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* lint
* Lint
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
* Added unit test: test_ensure_retry_errors_is_not_looping_infinitely()
* Added unit test: test_ensure_retry_errors_is_limited_by_max_retries()
* Added retry_errors arg to Connection.ensure() to allow applying retry policy for specific errors additionally
* Parse credential as a dict when using Azurite emulator
This more flexible credential allows the use of Azurite for integration testing in local docker-compose configurations.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix some lint errors
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add separate transport option for retry loop timeout
This only applies when using `Connect.default_channel`.
Before this change, the retry loop timeout was set equal to TCP connect timeout (`connect_timeout`), meaning when first connection attempt timeouted, no retry would be attempted.
Now if a new transport option `connect_total_timeout` is provided, this overrides `connect_timeout` for the retry loop (but not for TCP connect).
* Add tests
* Fix isort
* Rename to connect_retry_timeout
* Reformat
* connect_retry_timeout -> connect_retries_timeout
* Fix flake8
* Solve Kombu filesystem transport not thread safe
fix: #398
Currently only write lock used in msg/exchange file written. Cause
reading in other thread got some incomplete result.
1. Add timeout for the lock acquire.
2. Add Share locks when reading message from filesystem.
3. Add a unit test for the `lock` and `unlock`
4. Add a unit test to test the lock during message processing.
* Replace deprecated function.
* hub: tick delay fix
todo and timer callbacks can perform actions that
require a tick callback to be executed right away
without polling.
the current order can cause issues
when using single worker with no prefetch (acks late).
related issue in celery:
https://github.com/celery/celery/issues/7718
* add unit test for hub delay fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Avoid losing type of UUID when serializing/deserializing
Serializing UUIDs as strs and deserializing them as strs can lead to
somewhat obscure bugs such as the one that led to the creation of this
PR in django-cacheback
https://github.com/codeinthehole/django-cacheback/pull/100 which some
would call unexpected behaviour.
After all, an UUID is altogether a different type from strs and if
bytes got their own serializer/deserializer for UUID the same logic
should apply.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update documentation for JSON serialization
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add DelaySeconds to kwargs
* Add test
* add default value for DelaySeconds
* Fix tests and add check for properties
* Fix flake8 style issue
Co-authored-by: Edmund Lam <2623895+edmundlam@users.noreply.github.com>
* Datetime serialization and deserialization fixed
* Unit test fixed
* Unit test fixed
* Fixed pylint
* Added Undocumented Autodoc Modules
* Update kombu/utils/json.py
Co-authored-by: Omer Katz <omer.katz@omerkatz.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clean and freeze now
* Clean and freeze now
* Clean and freeze now
* Clean and freeze now
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
Co-authored-by: Omer Katz <omer.katz@omerkatz.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Annotate `abstract.py`
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* apply pre-commit
* Use quotes
* Add typing_extensions as requirement
* Add quotes
* Add quotes
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* updated azurestoragequeues transport for azure-storage-queues v12 + added basic tests
* fixed flake8 issues
* pinned azure-storage-queue lib to >= v12.0.0
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* azure-storage-queue>=12.2.0
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
* Support pymongo 4.x
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix problems detected by CI
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Added possibility to serialize and deserialize binary messages in json
* Flake8 fixed
* Hypothesis added to improve test range. Fixed issue b'\x80' serialization.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Added docstring
* Fixed pylint
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
When there is no locking then there is a possibility that multiple threads
manipulate with the same object at the same time.
The issue is manifested as:
```
RuntimeError: Set changed size during iteration
```
See: https://github.com/celery/celery/issues/7162
* Create a folder for each queue when using filesystem transport and add fanout support
* clean up unused variables
* Add fanout support to filesystem transport
filesystem transport lacks of fanout support.
1. Add fanout support to filesystem transport.
2. Add a unit test for it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove the refactoring work and make the test passed
1. Remove all of refactoring work
2. make the test pass
* Use pathlib for some Path operation
* Some reviewed changes
Co-authored-by: Yuriy Halytskyy <y.halytskyy@auckland.ac.nz>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Warn about missing hostname only when default one is available
The `No hostname was supplied` warning is affecting projects that use
AWS SQS (as detailed in #1357), as a hostname is not required when
setting up the broker URL. Instead, the official documentation [0] specifies
that the valid broker URL formats are:
* `sqs://`
* `sqs://aws_access_key_id:aws_secret_access_key@`
With these formats, the `kombu.utils.url.parse_url` util doesn't return
a hostname, and workers end up triggering the following warning:
> No hostname was supplied. Reverting to default 'None'
As the SQS transport doesn't provide a default value for hostname, this
diff changes the behavior to only warn the user when the hostname hasn't
been supplied but a default one is being set by the default connection
parameters for the defined transport.
Fixes#1357.
[0] https://docs.celeryproject.org/en/stable/getting-started/backends-and-brokers/sqs.html#configuration
* Use caplog default logging level of WARNING
* Remove use of OrderedDict in Connection.info.
* Remove remnant use of collections.OrderedDict
* Undo QoS._delivered move to Python standard dictionary.
This requires more work to convert, due to a hack in how the dictionary
is used.
* Undo LRUCache.data to dict conversion.
It's also problematic, and caused tests to fail.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert QoS._delivered comment chagne.
* Update comment
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Set redelivered property for Celery with Redis
Fixed setting `redelivered` value for Celery when Redis broker is used.
* Add `test_do_restore_message_celery` test
* Fix long line
* (Issue #789) Don't reset the proxy config in the async curl based http request building unless that is specified by the caller
* Test curl proxy only set when explicitly specified by caller
Co-authored-by: Adam Aljets <aljets@users.noreply.github.com>
* allow getting recoverable_connection_errors without an active transport
* move redis transport errors to class
* move consul transport errors to class
* move etcd transport errors to class
* remove redis.Transport._get_errors and references in tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix flake8 errors
* add integration test for redis ConnectionError
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
flolas