* updated azurestoragequeues transport for azure-storage-queues v12 + added basic tests
* fixed flake8 issues
* pinned azure-storage-queue lib to >= v12.0.0
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* azure-storage-queue>=12.2.0
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
* Support pymongo 4.x
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix problems detected by CI
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Added possibility to serialize and deserialize binary messages in json
* Flake8 fixed
* Hypothesis added to improve test range. Fixed issue b'\x80' serialization.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Added docstring
* Fixed pylint
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
When there is no locking then there is a possibility that multiple threads
manipulate with the same object at the same time.
The issue is manifested as:
```
RuntimeError: Set changed size during iteration
```
See: https://github.com/celery/celery/issues/7162
* Create a folder for each queue when using filesystem transport and add fanout support
* clean up unused variables
* Add fanout support to filesystem transport
filesystem transport lacks of fanout support.
1. Add fanout support to filesystem transport.
2. Add a unit test for it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove the refactoring work and make the test passed
1. Remove all of refactoring work
2. make the test pass
* Use pathlib for some Path operation
* Some reviewed changes
Co-authored-by: Yuriy Halytskyy <y.halytskyy@auckland.ac.nz>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Warn about missing hostname only when default one is available
The `No hostname was supplied` warning is affecting projects that use
AWS SQS (as detailed in #1357), as a hostname is not required when
setting up the broker URL. Instead, the official documentation [0] specifies
that the valid broker URL formats are:
* `sqs://`
* `sqs://aws_access_key_id:aws_secret_access_key@`
With these formats, the `kombu.utils.url.parse_url` util doesn't return
a hostname, and workers end up triggering the following warning:
> No hostname was supplied. Reverting to default 'None'
As the SQS transport doesn't provide a default value for hostname, this
diff changes the behavior to only warn the user when the hostname hasn't
been supplied but a default one is being set by the default connection
parameters for the defined transport.
Fixes#1357.
[0] https://docs.celeryproject.org/en/stable/getting-started/backends-and-brokers/sqs.html#configuration
* Use caplog default logging level of WARNING
* Remove use of OrderedDict in Connection.info.
* Remove remnant use of collections.OrderedDict
* Undo QoS._delivered move to Python standard dictionary.
This requires more work to convert, due to a hack in how the dictionary
is used.
* Undo LRUCache.data to dict conversion.
It's also problematic, and caused tests to fail.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert QoS._delivered comment chagne.
* Update comment
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Set redelivered property for Celery with Redis
Fixed setting `redelivered` value for Celery when Redis broker is used.
* Add `test_do_restore_message_celery` test
* Fix long line
* (Issue #789) Don't reset the proxy config in the async curl based http request building unless that is specified by the caller
* Test curl proxy only set when explicitly specified by caller
Co-authored-by: Adam Aljets <aljets@users.noreply.github.com>
* allow getting recoverable_connection_errors without an active transport
* move redis transport errors to class
* move consul transport errors to class
* move etcd transport errors to class
* remove redis.Transport._get_errors and references in tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix flake8 errors
* add integration test for redis ConnectionError
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Ensure that restore is atomic in redis transport
* Fix wrong variable
* Added integration tests
* Update unittests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix integration tests
* Fix flake8
* Fix queue name in int. tests
* Cosmetic clean up of integration tests
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Missing redis key containg set of queues bound to queue is caused
by removal all queues from the exchange. Hence, we should not raise an
exception but just return empty list of queues instead. This commit
fixes e.g. case publishing message against empty exchange:
import kombu
conn = kombu.Connection('redis://')
exchange = kombu.Exchange('name', type='direct')
exchange.declare(channel=conn.default_channel)
producer = conn.Producer()
producer.publish(
{'hello': 'world'},
exchange=exchange,
routing_key='queue1'
)
But it also fixes the case when last queue is unbound from exchange and
after publishing to this exchange:
import kombu
conn = kombu.Connection('redis://')
exchange = kombu.Exchange('name', type='direct')
queue1 = kombu.Queue('queue1', exchange=exchange, routing_key='queue1')
exchange.declare(channel=conn.default_channel)
queue1 = queue1.bind(channel=conn.default_channel)
queue1.declare()
producer = conn.Producer()
producer.publish(
{'hello': 'world'},
exchange=exchange,
routing_key='queue1'
)
queue1.delete()
producer.publish(
{'hello': 'world'},
exchange=exchange,
routing_key='queue1'
)
* fix flake8 in kombu/asynchronous/aws/connection.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] pre-commit autoupdate
updates:
- [github.com/asottile/pyupgrade: v2.26.0 → v2.28.0](https://github.com/asottile/pyupgrade/compare/v2.26.0...v2.28.0)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Remove dependency to case
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix flake8 errors
* Remove unused code
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Add missing DEL, RPUSH, RPOP and SREM commands to the list of commands to prefix.
Also, this commit refactors the prefixing logic a bit to make it simpler.
* Introduce global key prefix for redis transport
Co-authored-by: Matus Valo <matusvalo@users.noreply.github.com>
* refactor: use a custom redis client
As per the suggestions, refactor the redis key prefixing to use a custom
redis client that prefixes the keys it uses.
The custom client implementation does not prefix every key by default as
the way of prefixing keys may differ for some redis commands, instead it
lists those keys that will be prefixed. In case of commands, where
multiple keys can be passed as an argument, the custom client defines
where the arg positions are starting and ending for the given command.
* test: fix unit tests by moving import statement
* fix: wrap redis.parse_response to remove key prefixes
Co-authored-by: Matus Valo <matusvalo@users.noreply.github.com>
* fix: typo
* fix: lint
Co-authored-by: Antonin Delpeuch <antonin@delpeuch.eu>
Co-authored-by: Matus Valo <matusvalo@users.noreply.github.com>
Co-authored-by: Jillian Vogel <jill@opencraft.com>
Previously, calling `reject` when `predefined_queues` was not configured would cause `AttributeError` to be raised from `_extract_backoff_policy_configuration_and_message`. That exception could crash the whole Celery worker and force it to exit early because `AttributeError` is not excepted in the nearby call stack.
* enable pre-commit
* use extend-ignore for flake8
* manual flake8 fixes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update kombu/__init__.py
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Move pytest.importorskip to run earlier
Currently pytest fails to collect tests when `azure` is not present. Moving the pytest.importorskip clause before `from kombu.transport import azureservicebus` fixes it.
```
==================================== ERRORS ====================================
__________ ERROR collecting t/unit/transport/test_azureservicebus.py ___________
ImportError while importing test module '/build/python-kombu/src/kombu-5.1.0/t/unit/transport/test_azureservicebus.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.9/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
t/unit/transport/test_azureservicebus.py:12: in <module>
from kombu.transport import azureservicebus
kombu/transport/azureservicebus.py:66: in <module>
import azure.core.exceptions
E ModuleNotFoundError: No module named 'azure'
```
* Add # noqa to make linter happy
* add accept parameter to SimpleQueue class
* Fixed missing accept for get_nowait() and added unittests
* Remove unused **kwargs from SimpleQueue.__init__
* Use self.consumer.accept instead of new attribute in SimpleQueue
* Add tests for simple interface when accept=[]
Co-authored-by: Matus Valo <matusvalo@gmail.com>
* fixed character replace table according to the comment above - dots replaced by dashes, other punctuations replaced by underscores
* optimised with precalculated punctuation set - according to @thedrow suggestion
* queue name tests
* cleanup
* Use a thread-safe implementation of cached_property.
* Restore setter and deleter.
* Restore tests.
* Fix __get__ signature for backport.
* Cleanup.
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
* fix: non kombu json message decoding in SQS transport
* fix: non kombu json message decoding in SQS transport - add tests
Co-authored-by: Max Nikitenko <max.nikitenko@namecheap.com>
Jonas Miederer