This adds support for handling large payloads in SQS. The 'sqs_extended_client' is imported and utilized for fetching file from S3 as payload when necessary.
As Kombu asynchronously fetches new messages from the queue, not using the standard boto3 APIs, we have to manually fetch the s3 file, rather than rely on the sqs_extended_client to perform that action
Relates to: celery/kombu#279
- protobuf version in versions larger than 4.25.5 when used by google-cloud-pubsub
raises the following error:
site-packages/google/protobuf/internal/well_known_types.py", line 443, in FromTimedelta
raise AttributeError(
AttributeError: Fail to convert to Duration.
Expected a timedelta like object got str: 'str' object has no attribute 'seconds'
Fix this by restriction of the allowed package versions
- Added unit test to validate pubsub and protobuf compatibility
- Enabled google-cloud-pubsub package versions bump
Co-authored-by: Haim Daniel <haimdaniel@gmail.com>
- google-cloud-pubsub version in versions larger than 2.20.3 raises the following error:
site-packages/google/protobuf/internal/well_known_types.py", line 443, in FromTimedelta
raise AttributeError(
AttributeError: Fail to convert to Duration.
Expected a timedelta like object got str: 'str' object has no attribute 'seconds'
Fix this by restriction of the allowed package versions.
Co-authored-by: Haim Daniel <haimdaniel@gmail.com>
* feature(urllib3): add urllib3 client
* test(urllib3): test urllib3 client
* test(urllib3): update http test for urllib3
* test(urllib3): use urllib3 client instead of curl
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* style(urllib3): remove unused imports
* style(urllib3): fix pre-commit errors
* ci(urllib3): remove pycurl dependency
* docs(urllib3): add docs
* style(urllib3): fix failing gh-workflow py3.8
* style(urllib3): add mention of ProxyManager
* style(urllib3): fix pre-commit issues
* style(pycurl): remove curl-related code
* feat(urllib3): add missing request features (header, auth, ssl, proxy, redirects)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(urllib3): improve styling
* test(urllib3): add new tests
* fix(urllib3): fix request auth
* fix(aws): validate certificate on request
* style(): add missing exports
* feat(aws): add ssl certificate verification from boto
* feat(urllib): try to use certifi.where() if request.ca_certs are not provided
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci(pydocstyle): add missing docstring in public class
* test(urllib3): improve test case
* ci(pydocstyle): fix multi-line docstring summary should start at the first line
* feat(urllib3): remove assert_hostname
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test(boto): add test for get_cert_path returning .pem file path
* test(urllib3): add test for _get_pool_key_parts method
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Asif Saif Uddin <auvipy@gmail.com>
Co-authored-by: Tomer Nosrati <tomer.nosrati@gmail.com>
* Add support for Google Pub/Sub as transport broker
* Add tests
* Add docs
* flake8
* flake8
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix future import
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add missing test requirements
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add monitoring dependency
* Fix test for python3.8
* Mock better google's monitoring api
* Flake8
* Add refdoc
* Add extra url to workaround pypy grpcio
* Add extra index url in tox for grpcio/pypy support
* Revert "Add extra url to workaround pypy grpcio"
This reverts commit dfde4d523c.
* pin grpcio version to match extra_index wheel
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reduce poll calls if qos denies msg rx
---------
Co-authored-by: Haim Daniel <haimdaniel@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Tomer Nosrati <tomer.nosrati@gmail.com>
Amit Shah