Commit Graph

139 Commits

Author SHA1 Message Date
Omer Katz 2c75db39e0 Remove five usage from tests. 2020-07-23 16:33:40 +03:00
Omer Katz 7a6e7cc45b pyupgrade 2020-07-13 16:58:06 +03:00
Krzysztof Jagiello 016f4accb6 Add an SQS transport option for custom botocore config 2020-07-01 21:31:40 +06:00
Matus Valo 75027490c7
Revert incompatible changes in #1193 and additional improvements (#1211)
* Revert incompatible changes introduced in #1193

* Improved integration tests covering connection

* Fix unittests on python2 + flake8 fixes
2020-06-10 15:14:47 +06:00
Matus Valo 16749626a4
default_channel should reconnect automatically (#1209) 2020-06-05 09:20:56 +02:00
Conor Stevenson ff7c1e3041 reuse connection 2020-06-02 14:11:32 +06:00
Conor Stevenson 255b85229b
set _connection in _ensure_connection (#1205)
Co-authored-by: Conor Stevenson <conor@lendingblock.com>
2020-06-01 23:23:05 +02:00
Matus Valo df31de1c6e Fix broken compatibility with Celery 2020-06-01 12:56:32 +06:00
Matus Valo 25a0266651 Ensure connection when connecting to broker 2020-06-01 12:56:32 +06:00
Ihar Nauros 5cc4a76882
Fix for issue #1198: Celery crashes in cases where there aren’t enough available workers to start acting on “in-flight” messages in the SQS queue (#1199)
* Fix for the issue #1172

* Unit test for the fix relating to the issue #1172

* Fix for issue #1198: Celery crashes in cases where there aren’t enough available workers to start acting on “in-flight” messages in the SQS queue

* Fix for issue #1198: fixed lint issues

* Fix for issue #1198: added unit tests

Co-authored-by: inauros <inauros@copyright.com>
2020-05-27 16:18:31 +06:00
Oleh Kuchuk 246cb9e996
Fix connection imaybe_declare (#1196) 2020-05-26 21:03:22 +02:00
Matus Valo 4644a5e940
Revert "Raise RecoverableConnectionError in maybe_declare with retry on and dropped connection" (#1195)
This reverts commit 90f51bcbbd.
2020-05-25 00:06:38 +02:00
Vinay Karanam 7fefb6fe9a Modified Mutex to use redis LuaLock implementation
Fixes #1190
2020-05-17 11:15:19 +06:00
Andrii Maletskyi 5101db177c fix 100% cpu usage on linux while using sqs
fix https://github.com/celery/celery/issues/5299
2020-05-15 07:11:01 +06:00
Ihar Nauros 4dadaad6e7 Unit test for the fix relating to the issue #1172 2020-04-22 14:48:02 +06:00
Oleh Kuchuk 90f51bcbbd Raise RecoverableConnectionError in maybe_declare with retry on and dropped connection 2020-04-15 21:08:18 +06:00
Christian Clauss cca00b1c9d
Travis CI: Fix travis validation issues (#1159)
* Travis CI: Fix travis validation issues

* Whitespace fix for flake8 issue

* Whitespace fix for flake8 issue
2020-03-03 20:10:02 +06:00
Piyush Agarwal b22cb507a1
#1149 Enabled ssl certificate verification against ca certificates when amqps is used for pyamqp transport (#1151)
* enabled ssl certificate verification when amqps is used for pyamqp transport

* unit tests
2020-02-28 22:00:38 +06:00
Jonathan Ingram 5780f1e70b
Add predefined_queues option to SQS transport (#1156) 2020-02-26 09:09:38 +06:00
Shai Berger 16531285d5
Issue 701 (#1141)
* Make Redis-Mutex test fail as it should (#701)

See https://github.com/celery/kombu/issues/701

* Fix issue #701 - decode lock_id returned from Redis

Thanks @zaro
2020-02-02 21:10:56 +06:00
Dave Peticolas 7cb4a0afe5
Support standard values for ssl_cert_reqs query parameter. (#1139)
* Support standard values for ssl_cert_reqs query parameter.

* Pick lint.
2020-02-02 21:08:38 +06:00
Konstantin Seleznev e1dcfaef4b
Adding retry_on_timeout parameter (#1150)
* added retry_on_timeout connparam

* added retry_on_timeout to tests

* updated Pyro4

* revert Pyro4 in requirements

* Trigger notification

* Trigger rebuild
2020-02-02 21:01:56 +06:00
Kevin Fox cbd327dc3b Add support for health_check_interval option in broker_transport_options. (#1145)
* Add support for health_check_interval option in broker_transport_options. #1114

* Added Keivn Fox to authors.

* Do not use default arg for pop() method.

* Add dedicated test case for configurable health_check

* Add test case for when health_check_interval is not supported by connection_class in redis.

* Fix Flake8

* Added more test cases for redis.

* Updated tests.
2020-01-11 11:20:31 +06:00
Nicolas Mota 0c6444ac88 Add peek lock settings to be changed using transport options (#1119) 2019-12-02 04:15:35 +06:00
Ashish Bansal (mrphantom) ccc9e01f32 Fix redis health checks (#1122)
* Fix redis transport health checks functionality

* Add tests for accepts_argument util function

* Reduce default health check interval to 25s
2019-12-02 04:12:13 +06:00
Smite Chow de8d8cf8c1 reset ready before execute callback (#1126)
* reset ready before execute callback

* add test to cover hub._ready reset to empty
2019-12-02 04:11:15 +06:00
Mario Rodriguez Molins 81101877a0 Add missing parameter queue_args in kombu.connection.SimpleBuffer (#1128)
* Add missing parameter queue_args in kombu.connection.SimpleBuffer

`queue_args` parameter was introduced in `kombu.simple.SimpleQueue`
class and at the same time in `kombu.connection.SimpleQueue` method.
However, `kombu.connection.SimpleBuffer` was not updated in the
same way.

As `kombu.connection.SimpleBuffer` is using positional arguments to
create a `kombu.simple.SimpleBuffer` object, `exchange_opts` value in
that instance would be None and `queue_args` is going to bet set with
the expected `exchange_opts` value.

* Add tests for SimpleBuffer and SimpleQueue

- Add tests for SimpleBuffer/SimpleQueue methods in kombu.simple module
- Add tests for SimpleQueue and SimpleBuffer tasks in kombu.connection
2019-11-23 00:19:03 +06:00
Asif Saif Uddin 5006e49c53
Revert "Revert "Revert "Use SIMEMBERS instead of SMEMBERS to check for queue (redis broker) (#1041)" (#1058)" (#1068)" (#1117)
This reverts commit 052f7601d3.
2019-11-10 23:21:24 +06:00
Asif Saif Uddin 597d675ca1
Revert "Issue #1087 redis fix (#1089)" (#1106)
This reverts commit 2f6f5f6a5d.
2019-11-02 23:43:08 +06:00
Aliaksei Urbanski ef54bf06f7 Fix 'NoneType' object has no attribute 'can_read' (#1107)
The root cause of the issue is that the connection
can be erased during an execution of self._receive_one(c).
So c.connection should be checked before every iteration,
not only before the first one.
2019-10-24 00:27:47 +06:00
Nicolas Mota 977fe3d4f6 Add wait timeout settings to receive queue message (#1110)
* Add wait timeout settings to receive queue message

* add tests

* add more tests and remove servicebus from coveragerc
2019-10-23 17:25:15 +03:00
Kris Ciccarello 2d92827746 Issue #1019 Fix redis transport socket timeout (#1113)
* Issue #1019 Fix redis transport socket timeout

Patch from @jschwartzentruber https://github.com/celery/kombu/issues/1019#issuecomment-535157400

* Fix test_register_with_event_loop redis test

* Fix linter

* Bump redis to 3.3.11
2019-10-23 10:23:26 +06:00
Shekhar Kamble c6b09a3ab5 fixes #1097 (#1099)
* sanitise url when include_password is false  (fixes #1085)

* fix flake8 error

* fixes #1097

* fix flake8 errors
2019-09-10 22:46:13 +06:00
Shekhar Kamble fcecc30aed sanitise url when include_password is false (fixes #1085) (#1095)
* sanitise url when include_password is false  (fixes #1085)

* fix flake8 error
2019-09-03 14:57:27 +03:00
Matt Davis 2f6f5f6a5d Issue #1087 redis fix (#1089)
* issue-1087 Revert _lookup api and correct redis implemetnation.  Update variable naming to be self documenting and more readable by non native speakers.  Major overhaul of redis test cases by adding more full featured fakeredis module.

* Add more test cases to boost coverage of kombu redis transport.

* Refactor the producer consumer test cases to be based on original mocks and be passing

* Fix lingering line length issue in test.
2019-08-31 23:54:01 +06:00
Eric Higdon 5f93e868cf Make sure that max_retries=0 is treated differently than None (#1080)
* Make sure that max_retries=0 is treated differently than None

When max_retries=0, it should not retry at all instead of retrying infinitely.

* retry_always should use max_retries=None instead of 0.

* Added test for zero retries

* Removed Offending spaces from blank line
2019-08-13 23:51:40 +06:00
Marcin Puhacz 0ad7d08c4c Fixing MongoDB backend to work properly with TTL (#1076)
* Fixing MongoDB backend to work properly with TTL
Bumping pymongo version to 3.8.0

* Updating authors

* Adding dependencies to travis config

* Fixing travis test cases

* Changing pymongo version to align with mongo result backend
2019-08-07 15:06:13 +03:00
David Szotten 193054d243 Switch from pkg_resources importlib-metadata (#1071)
See e.g. https://github.com/pypa/setuptools/issues/510#issuecomment-463667124

On a moderatly large project (~200 packages) at $work this saves ~150 ms on
`python -c 'from kombu import Exchange'`
2019-07-06 00:33:35 +06:00
Asif Saif Uddin 052f7601d3
Revert "Revert "Use SIMEMBERS instead of SMEMBERS to check for queue (redis broker) (#1041)" (#1058)" (#1068)
This reverts commit fcb8e05cc9.
2019-06-29 14:24:27 +06:00
Matt Davis ac6c7b9754 Issue 129 fix maybe declare (#1066)
* Factor out the binding of channel to entity into a method that can be shared and does not use assert.

* do not use assertion in production because it can be stripped in optimize and is best left for unit tests.

* dont reassign entity here because it breaks a bunch of tests.

* fix pydoc complaints

* Add TDD style test that fails in master and passes in my branch

* finish refactor of existing maybe_declare tests to use shared fixtures

* py2/py3 print difference means skip printing.

* fix pep8 line length
2019-06-29 11:34:35 +06:00
Eugene Beseda ffabaf31dd don't stop receiving tasks on 503 SQS error. (#1064)
* don't stop receiving tasks on 503 SQS error.

* fix linter errors
2019-06-25 00:18:15 +06:00
ngie-eign ec4313b034 Allow users to switch URLs while omitting the resource identifier (#1032)
Prior to this change, one needed to specify a URL using a URI
identifier, e.g., `pyamqp://foo.bar`. This change makes it so calling
`.switch(..)` again results in switching the host, not switching the
resource identifier.

This simplifies setting up connections with just hostnames specifying
the resource identifier once, separately.

Signed-off-by: Enji Cooper <yaneurabeya@gmail.com>
2019-06-24 11:39:18 +06:00
Eugene Beseda 73bed5f7e5 return an empty list on SQS internal error. (#1059)
* return empty list on SQS internal error.

* add test for on_list_ready error codes
2019-06-13 20:27:27 +06:00
Asif Saif Uddin fcb8e05cc9
Revert "Use SIMEMBERS instead of SMEMBERS to check for queue (redis broker) (#1041)" (#1058)
This reverts commit 73d2219887.
2019-06-11 17:49:25 +06:00
Gabriele Santomaggio 772f1d7518 Add events field on memory channel (#1053)
* Add events field on memory channel

Part of https://github.com/celery/kombu/issues/1050, with this fix
the function does not raise the exception.
The init `defaultdict(set)` is the same init from amqp lib.

* Add unit test

* Add more detail test

* FIx flake8
2019-06-04 19:28:02 +06:00
Stevan Milic 73d2219887 Use SIMEMBERS instead of SMEMBERS to check for queue (redis broker) (#1041)
* Add `_lookup_direct` method to virtual channel. (#994)

Add possibility to optimize lookup for queue in direct
exchange set.

* Add `_lookup_direct` method to redis virtual channel. (#994)

Use `SISMEMBER` instead of `SMEMBERS` command to check if queue
exists in a set. Time complexity is increased from O(N) to O(1)
where N is the set cardinality.
2019-05-29 19:45:09 +06:00
Rohan McGovern 57f464895d qpid: ensure supported exchange types are declared (#1034)
This is a follow-up to ddba8aeaf0. Prior to that commit,
"implements" was missing entirely for qpid. The earlier commit
added it, but only declared that the transport was async-capable.

Turns out it's important also to declare the supported exchange
types. In particular, if the transport doesn't declare support
for the 'fanout' exchange type, pidbox wouldn't be used[1] and so
it would still not be possible to revoke a task with terminate=True
when using the qpid transport.

[1] https://github.com/celery/celery/commit/2f58c35340f648
2019-04-04 10:56:43 +03:00
Omer Katz 892f400355
Reintroduce unqiue broadcast queue name generation as optional. (#1033) 2019-03-31 17:37:15 +03:00
Rohan McGovern d9735dee83 qpid: correctly declare support for async event loop (#1029)
qpid transport supports an async interface, but this wasn't declared
properly. It was missing an appropriate 'implements' class property.

This seems to have been missed because:

- kombu 4.x was branched from 3.0.21
- qpid transport was originally added after that in 3.0.24
- Transport.implements was added in commit 6a1abb7e9 on the 4.x branch
  only, before the qpid transport was in that branch
- qpid transport code was then copied between 3.x and 4.x branches but
  addition of Transport.implements was missed.
2019-03-30 08:49:08 +06:00
Omer Katz 5e08ad55b1
Revert "fix/Broadcast-fail-by-give-queue-kwargs (#906)" (#1031)
This reverts commit 87d6423289.
2019-03-27 15:51:13 +02:00