proxy.py/proxy/http/plugin.py

116 lines
4.1 KiB
Python
Raw Normal View History

# -*- coding: utf-8 -*-
"""
proxy.py
~~~~~~~~
Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
Network monitoring, controls & Application development, testing, debugging.
:copyright: (c) 2013-present by Abhinav Singh and contributors.
:license: BSD, see LICENSE for more details.
"""
import socket
import argparse
from abc import ABC, abstractmethod
from typing import Tuple, List, Union, Optional, TYPE_CHECKING
from ..common.types import Readables, Writables
from ..core.event import EventQueue
from ..core.connection import TcpClientConnection
from .parser import HttpParser
if TYPE_CHECKING:
from ..core.connection import UpstreamConnectionPool
class HttpProtocolHandlerPlugin(ABC):
"""Base HttpProtocolHandler Plugin class.
NOTE: This is an internal plugin and in most cases only useful for core contributors.
If you are looking for proxy server plugins see `<proxy.HttpProxyBasePlugin>`.
Implements various lifecycle events for an accepted client connection.
Following events are of interest:
1. Client Connection Accepted
A new plugin instance is created per accepted client connection.
Add your logic within __init__ constructor for any per connection setup.
2. Client Request Chunk Received
on_client_data is called for every chunk of data sent by the client.
3. Client Request Complete
on_request_complete is called once client request has completed.
4. Server Response Chunk Received
on_response_chunk is called for every chunk received from the server.
5. Client Connection Closed
2021-11-16 23:34:29 +00:00
Add your logic within `on_client_connection_close` for any per connection tear-down.
"""
def __init__(
self,
v2.4.0rc3 (#882) * Merge pull request #840 from abhinavsingh/release-schedule-notes Add release schedule under FAQ * Green CI (#841) * Cleanup parser & url classes (#843) * Optimize parser logic * Add `is_complete` property * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * pre commit fixes * We need `ValueError` * wuff * useless * flake8 * Simplify url Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * pip prod(deps): bump twine from 3.6.0 to 3.7.0 (#845) Bumps [twine](https://github.com/pypa/twine) from 3.6.0 to 3.7.0. - [Release notes](https://github.com/pypa/twine/releases) - [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst) - [Commits](https://github.com/pypa/twine/compare/3.6.0...3.7.0) --- updated-dependencies: - dependency-name: twine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump jasmine from 3.6.3 to 3.10.0 in /dashboard (#844) Bumps [jasmine](https://github.com/jasmine/jasmine-npm) from 3.6.3 to 3.10.0. - [Release notes](https://github.com/jasmine/jasmine-npm/releases) - [Commits](https://github.com/jasmine/jasmine-npm/compare/v3.6.3...v3.10.0) --- updated-dependencies: - dependency-name: jasmine dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * npm: bump chrome-devtools-frontend in /dashboard (#846) Bumps [chrome-devtools-frontend](https://github.com/ChromeDevTools/devtools-frontend) from 1.0.944903 to 1.0.947377. - [Release notes](https://github.com/ChromeDevTools/devtools-frontend/releases) - [Changelog](https://github.com/ChromeDevTools/devtools-frontend/blob/main/docs/release_management.md) - [Commits](https://github.com/ChromeDevTools/devtools-frontend/commits) --- updated-dependencies: - dependency-name: chrome-devtools-frontend dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * pip prod(deps): bump coverage from 6.1.2 to 6.2 (#847) Bumps [coverage](https://github.com/nedbat/coveragepy) from 6.1.2 to 6.2. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](https://github.com/nedbat/coveragepy/compare/6.1.2...6.2) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Fix GHA check-gate to properly identify failures (#849) * pip prod(deps): bump pylint from 2.12.1 to 2.12.2 (#851) Bumps [pylint](https://github.com/PyCQA/pylint) from 2.12.1 to 2.12.2. - [Release notes](https://github.com/PyCQA/pylint/releases) - [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog) - [Commits](https://github.com/PyCQA/pylint/compare/v2.12.1...v2.12.2) --- updated-dependencies: - dependency-name: pylint dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump @types/js-cookie from 2.2.6 to 3.0.1 in /dashboard (#850) Bumps [@types/js-cookie](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/js-cookie) from 2.2.6 to 3.0.1. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/js-cookie) --- updated-dependencies: - dependency-name: "@types/js-cookie" dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * pip prod(deps): bump sphinx from 4.3.0 to 4.3.1 (#853) Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.3.0 to 4.3.1. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.3.0...v4.3.1) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * pip prod(deps): bump paramiko from 2.8.0 to 2.8.1 (#855) * npm: bump ws from 7.4.6 to 8.3.0 in /dashboard (#854) * pip prod(deps): bump uvicorn from 0.15.0 to 0.16.0 (#857) Bumps [uvicorn](https://github.com/encode/uvicorn) from 0.15.0 to 0.16.0. - [Release notes](https://github.com/encode/uvicorn/releases) - [Changelog](https://github.com/encode/uvicorn/blob/master/CHANGELOG.md) - [Commits](https://github.com/encode/uvicorn/compare/0.15.0...0.16.0) --- updated-dependencies: - dependency-name: uvicorn dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump chrome-devtools-frontend in /dashboard (#856) Bumps [chrome-devtools-frontend](https://github.com/ChromeDevTools/devtools-frontend) from 1.0.947377 to 1.0.949424. - [Release notes](https://github.com/ChromeDevTools/devtools-frontend/releases) - [Changelog](https://github.com/ChromeDevTools/devtools-frontend/blob/main/docs/release_management.md) - [Commits](https://github.com/ChromeDevTools/devtools-frontend/commits) --- updated-dependencies: - dependency-name: chrome-devtools-frontend dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Process `--enable-*` flags before loading plugins (#860) * Process `--enable-*` flags before loading plugins * Fix spelling * Ignore RST299 and RST499 * npm: bump http-server from 0.12.3 to 14.0.0 in /dashboard (#858) Bumps [http-server](https://github.com/http-party/http-server) from 0.12.3 to 14.0.0. - [Release notes](https://github.com/http-party/http-server/releases) - [Commits](https://github.com/http-party/http-server/compare/v0.12.3...v14.0.0) --- updated-dependencies: - dependency-name: http-server dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * pip prod(deps): bump furo from 2021.11.15 to 2021.11.23 (#859) Bumps [furo](https://github.com/pradyunsg/furo) from 2021.11.15 to 2021.11.23. - [Release notes](https://github.com/pradyunsg/furo/releases) - [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md) - [Commits](https://github.com/pradyunsg/furo/compare/2021.11.15...2021.11.23) --- updated-dependencies: - dependency-name: furo dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Update web log context fields to match proxy log context fields (#861) * Update web log context fields to match those with proxy log context for consistency * Fix spell * Fix `DEFAULT_WEB_ACCESS_LOG_FORMAT` * pip prod(deps): bump pytest-xdist from 2.4.0 to 2.5.0 (#864) Bumps [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/pytest-dev/pytest-xdist/releases) - [Changelog](https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest-xdist/compare/v2.4.0...v2.5.0) --- updated-dependencies: - dependency-name: pytest-xdist dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump eslint-plugin-node from 10.0.0 to 11.1.0 in /dashboard (#863) Bumps [eslint-plugin-node](https://github.com/mysticatea/eslint-plugin-node) from 10.0.0 to 11.1.0. - [Release notes](https://github.com/mysticatea/eslint-plugin-node/releases) - [Commits](https://github.com/mysticatea/eslint-plugin-node/compare/v10.0.0...v11.1.0) --- updated-dependencies: - dependency-name: eslint-plugin-node dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Fix broken TLS interception & CacheResponsesPlugin because UID is no longer a UUID (#866) * Fix broken TLS interception because uid is now no longer a UUID * Give enough context to work id for them to be unique within a `proxy.py` instance * Use --port=0 by default within `proxy.TestCase` * Attempt to fix weird buildx issue * Add makefile targets within workflow * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Order? * Write scm file for make * Fetch depth * Quote patch * Try with sudo? * https://github.com/docker/buildx/issues/850 * Remove sudo hack * https://github.com/docker/buildx/issues/850\#issuecomment-973270625 * Add explicit deps * Add `requirements-testing.txt` during linting phase * Pin buildx to v0.7.1 * Pin buildx to v0.7.0 * Revert back unnecessary change to dockerignore * Skip container within make workflow (because GHA lacks support for docker on macOS by default) * Repurpose make into developer workflow Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Integrate showing unreleased changelog draft (#873) * pip prod(deps): bump types-paramiko from 2.8.2 to 2.8.4 (#868) * npm: bump @types/jasmine from 3.6.1 to 3.10.2 in /dashboard (#867) * pip prod(deps): bump py-spy from 0.3.10 to 0.3.11 (#875) Bumps [py-spy](https://github.com/benfred/py-spy) from 0.3.10 to 0.3.11. - [Release notes](https://github.com/benfred/py-spy/releases) - [Changelog](https://github.com/benfred/py-spy/blob/master/CHANGELOG.md) - [Commits](https://github.com/benfred/py-spy/compare/v0.3.10...v0.3.11) --- updated-dependencies: - dependency-name: py-spy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * [GHA] Add container integration test & publish containers to GHCR (#818) * Handle KBI in threadless * Remove line-length disable clause * Move `Set PROXYPY_CONTAINER_VERSION env` within pre-setup * Use step output * Use review suggestion * Break line at less essential places * Possibly fix? * alignment necessary? * :) * Fix integration * Add `v` prefix * Load buildx image within docker for integration tests to work * Run container in detached mode * Enable web server for integration test to work * Push to GHCR after successful tests * Tag & Push to GHCR * Add package permission * Login to GHCR * Login to GHCR * Login to GHCR * Explicitly use ghcr.io registry * Add a ghcr manifest creation step which depends upon docker step * Probably we need all images (tags) locally * arm not amd * Fix syntax * Fix typo * `Unable to find image ghcr.io/abhinavsingh/proxy.py:latest locally` * GHCR doesnt support manifest? * Publish multi-platform containers on GHCR (#877) * Build containers without matrix based strategy. Helps with buildx based manifest generation * Use buildx directly in workflows * Add PROXYPY_PKG_PATH arg * Add missing . * --push from buildx * Also add latest tag for GHCR * Fix typo * Explain differences between latest tag on DockerHub (stable) and GHCR (develop) * Publish multi-platform containers to DockerHub (#878) * Use `--local-executor` flag by default for Docker container (#880) * Benchmark gets packaged within wheel if set as a package * Use `--local-executor` flag by default for Docker containers * Dockerfile update * Fix mypy issues * Remove conflicting dir names * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
2021-12-19 16:00:43 +00:00
uid: str,
flags: argparse.Namespace,
client: TcpClientConnection,
request: HttpParser,
event_queue: Optional[EventQueue],
upstream_conn_pool: Optional['UpstreamConnectionPool'] = None,
):
v2.4.0rc3 (#882) * Merge pull request #840 from abhinavsingh/release-schedule-notes Add release schedule under FAQ * Green CI (#841) * Cleanup parser & url classes (#843) * Optimize parser logic * Add `is_complete` property * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * pre commit fixes * We need `ValueError` * wuff * useless * flake8 * Simplify url Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * pip prod(deps): bump twine from 3.6.0 to 3.7.0 (#845) Bumps [twine](https://github.com/pypa/twine) from 3.6.0 to 3.7.0. - [Release notes](https://github.com/pypa/twine/releases) - [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst) - [Commits](https://github.com/pypa/twine/compare/3.6.0...3.7.0) --- updated-dependencies: - dependency-name: twine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump jasmine from 3.6.3 to 3.10.0 in /dashboard (#844) Bumps [jasmine](https://github.com/jasmine/jasmine-npm) from 3.6.3 to 3.10.0. - [Release notes](https://github.com/jasmine/jasmine-npm/releases) - [Commits](https://github.com/jasmine/jasmine-npm/compare/v3.6.3...v3.10.0) --- updated-dependencies: - dependency-name: jasmine dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * npm: bump chrome-devtools-frontend in /dashboard (#846) Bumps [chrome-devtools-frontend](https://github.com/ChromeDevTools/devtools-frontend) from 1.0.944903 to 1.0.947377. - [Release notes](https://github.com/ChromeDevTools/devtools-frontend/releases) - [Changelog](https://github.com/ChromeDevTools/devtools-frontend/blob/main/docs/release_management.md) - [Commits](https://github.com/ChromeDevTools/devtools-frontend/commits) --- updated-dependencies: - dependency-name: chrome-devtools-frontend dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * pip prod(deps): bump coverage from 6.1.2 to 6.2 (#847) Bumps [coverage](https://github.com/nedbat/coveragepy) from 6.1.2 to 6.2. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](https://github.com/nedbat/coveragepy/compare/6.1.2...6.2) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Fix GHA check-gate to properly identify failures (#849) * pip prod(deps): bump pylint from 2.12.1 to 2.12.2 (#851) Bumps [pylint](https://github.com/PyCQA/pylint) from 2.12.1 to 2.12.2. - [Release notes](https://github.com/PyCQA/pylint/releases) - [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog) - [Commits](https://github.com/PyCQA/pylint/compare/v2.12.1...v2.12.2) --- updated-dependencies: - dependency-name: pylint dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump @types/js-cookie from 2.2.6 to 3.0.1 in /dashboard (#850) Bumps [@types/js-cookie](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/js-cookie) from 2.2.6 to 3.0.1. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/js-cookie) --- updated-dependencies: - dependency-name: "@types/js-cookie" dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * pip prod(deps): bump sphinx from 4.3.0 to 4.3.1 (#853) Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.3.0 to 4.3.1. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.3.0...v4.3.1) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * pip prod(deps): bump paramiko from 2.8.0 to 2.8.1 (#855) * npm: bump ws from 7.4.6 to 8.3.0 in /dashboard (#854) * pip prod(deps): bump uvicorn from 0.15.0 to 0.16.0 (#857) Bumps [uvicorn](https://github.com/encode/uvicorn) from 0.15.0 to 0.16.0. - [Release notes](https://github.com/encode/uvicorn/releases) - [Changelog](https://github.com/encode/uvicorn/blob/master/CHANGELOG.md) - [Commits](https://github.com/encode/uvicorn/compare/0.15.0...0.16.0) --- updated-dependencies: - dependency-name: uvicorn dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump chrome-devtools-frontend in /dashboard (#856) Bumps [chrome-devtools-frontend](https://github.com/ChromeDevTools/devtools-frontend) from 1.0.947377 to 1.0.949424. - [Release notes](https://github.com/ChromeDevTools/devtools-frontend/releases) - [Changelog](https://github.com/ChromeDevTools/devtools-frontend/blob/main/docs/release_management.md) - [Commits](https://github.com/ChromeDevTools/devtools-frontend/commits) --- updated-dependencies: - dependency-name: chrome-devtools-frontend dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Process `--enable-*` flags before loading plugins (#860) * Process `--enable-*` flags before loading plugins * Fix spelling * Ignore RST299 and RST499 * npm: bump http-server from 0.12.3 to 14.0.0 in /dashboard (#858) Bumps [http-server](https://github.com/http-party/http-server) from 0.12.3 to 14.0.0. - [Release notes](https://github.com/http-party/http-server/releases) - [Commits](https://github.com/http-party/http-server/compare/v0.12.3...v14.0.0) --- updated-dependencies: - dependency-name: http-server dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * pip prod(deps): bump furo from 2021.11.15 to 2021.11.23 (#859) Bumps [furo](https://github.com/pradyunsg/furo) from 2021.11.15 to 2021.11.23. - [Release notes](https://github.com/pradyunsg/furo/releases) - [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md) - [Commits](https://github.com/pradyunsg/furo/compare/2021.11.15...2021.11.23) --- updated-dependencies: - dependency-name: furo dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Update web log context fields to match proxy log context fields (#861) * Update web log context fields to match those with proxy log context for consistency * Fix spell * Fix `DEFAULT_WEB_ACCESS_LOG_FORMAT` * pip prod(deps): bump pytest-xdist from 2.4.0 to 2.5.0 (#864) Bumps [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/pytest-dev/pytest-xdist/releases) - [Changelog](https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest-xdist/compare/v2.4.0...v2.5.0) --- updated-dependencies: - dependency-name: pytest-xdist dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump eslint-plugin-node from 10.0.0 to 11.1.0 in /dashboard (#863) Bumps [eslint-plugin-node](https://github.com/mysticatea/eslint-plugin-node) from 10.0.0 to 11.1.0. - [Release notes](https://github.com/mysticatea/eslint-plugin-node/releases) - [Commits](https://github.com/mysticatea/eslint-plugin-node/compare/v10.0.0...v11.1.0) --- updated-dependencies: - dependency-name: eslint-plugin-node dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Fix broken TLS interception & CacheResponsesPlugin because UID is no longer a UUID (#866) * Fix broken TLS interception because uid is now no longer a UUID * Give enough context to work id for them to be unique within a `proxy.py` instance * Use --port=0 by default within `proxy.TestCase` * Attempt to fix weird buildx issue * Add makefile targets within workflow * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Order? * Write scm file for make * Fetch depth * Quote patch * Try with sudo? * https://github.com/docker/buildx/issues/850 * Remove sudo hack * https://github.com/docker/buildx/issues/850\#issuecomment-973270625 * Add explicit deps * Add `requirements-testing.txt` during linting phase * Pin buildx to v0.7.1 * Pin buildx to v0.7.0 * Revert back unnecessary change to dockerignore * Skip container within make workflow (because GHA lacks support for docker on macOS by default) * Repurpose make into developer workflow Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Integrate showing unreleased changelog draft (#873) * pip prod(deps): bump types-paramiko from 2.8.2 to 2.8.4 (#868) * npm: bump @types/jasmine from 3.6.1 to 3.10.2 in /dashboard (#867) * pip prod(deps): bump py-spy from 0.3.10 to 0.3.11 (#875) Bumps [py-spy](https://github.com/benfred/py-spy) from 0.3.10 to 0.3.11. - [Release notes](https://github.com/benfred/py-spy/releases) - [Changelog](https://github.com/benfred/py-spy/blob/master/CHANGELOG.md) - [Commits](https://github.com/benfred/py-spy/compare/v0.3.10...v0.3.11) --- updated-dependencies: - dependency-name: py-spy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * [GHA] Add container integration test & publish containers to GHCR (#818) * Handle KBI in threadless * Remove line-length disable clause * Move `Set PROXYPY_CONTAINER_VERSION env` within pre-setup * Use step output * Use review suggestion * Break line at less essential places * Possibly fix? * alignment necessary? * :) * Fix integration * Add `v` prefix * Load buildx image within docker for integration tests to work * Run container in detached mode * Enable web server for integration test to work * Push to GHCR after successful tests * Tag & Push to GHCR * Add package permission * Login to GHCR * Login to GHCR * Login to GHCR * Explicitly use ghcr.io registry * Add a ghcr manifest creation step which depends upon docker step * Probably we need all images (tags) locally * arm not amd * Fix syntax * Fix typo * `Unable to find image ghcr.io/abhinavsingh/proxy.py:latest locally` * GHCR doesnt support manifest? * Publish multi-platform containers on GHCR (#877) * Build containers without matrix based strategy. Helps with buildx based manifest generation * Use buildx directly in workflows * Add PROXYPY_PKG_PATH arg * Add missing . * --push from buildx * Also add latest tag for GHCR * Fix typo * Explain differences between latest tag on DockerHub (stable) and GHCR (develop) * Publish multi-platform containers to DockerHub (#878) * Use `--local-executor` flag by default for Docker container (#880) * Benchmark gets packaged within wheel if set as a package * Use `--local-executor` flag by default for Docker containers * Dockerfile update * Fix mypy issues * Remove conflicting dir names * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
2021-12-19 16:00:43 +00:00
self.uid: str = uid
self.flags: argparse.Namespace = flags
self.client: TcpClientConnection = client
self.request: HttpParser = request
self.event_queue = event_queue
self.upstream_conn_pool = upstream_conn_pool
super().__init__()
@staticmethod
@abstractmethod
def protocols() -> List[int]:
raise NotImplementedError()
@abstractmethod
Async `get_events`, `handle_event`, `handle_readables`, `handle_writables` (#769) * Asynchronous `handle_event` and `LocalExecutor` thread * Bail out on first task completion * mypy * Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded) * Print open file diff from `benchmark.sh` * Add `--local-executor` flag, disabled by default for now until tests are updated * Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now) * Async `get_events` * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address tests after async changes * mypy and flake8 * spelldoc * `check.py` and trailing comma * Rename to `_assertions.py` * Add missing `pytest-mock` and `pytest-asyncio` deps * Add `pytest-mock` to `pylint` deps * Correct use of `parameterize` and add `PT007` to flake8 ignores * Fix mypy hints broken for `< Python3.9` * Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix for pre-python-3.9 versions * `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions * Install setuptools via `lib-dep` until we recommend editable install * Deprecate support for `Python 3.6` * Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929 * Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906 * Make `Threadless` agnostic of `multiprocessing.Process` * Acceptors must dispatch to local executor in non-blocking fashion * No daemon for executor processes and fix shutdown logic * Only return fds from `_selected_events` not all events data * Refactor logic * Prefix private methods with `_` * `work_queue` and not `client_queue` * Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor` * Make `LocalExecutor` agnostic of `threading.Thread` * `LocalExecutor` now implements `Threadless` * `get_events` and `get_descriptors` now must return int and not sock. `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify` * Fix `main` tests * Apply suggestions from code review Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua> * Apply code review recommendations manually * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert back `Any` and use `addr or None` * Address `flake8` * Update tests to use `fileno` * Fix doc build * Fix doc spell, use tear down and not teardown * Doc updates * Add back support for `Python 3.6` * Acceptors dont need loop initialization * On Python 3.6 `asyncio.new_event_loop()` is necessary * Make doc happy * `--threaded` needs a new event loop for 3.7 too * Always use `asyncio.new_event_loop()` for threaded mode Added e2e integration tests (subprocess & curl) for all modes. * Lint fixes Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
2021-11-23 09:32:00 +00:00
def get_descriptors(self) -> Tuple[List[int], List[int]]:
"""Implementations must return a list of descriptions that they wish to
read from and write into."""
return [], [] # pragma: no cover
@abstractmethod
Async `get_events`, `handle_event`, `handle_readables`, `handle_writables` (#769) * Asynchronous `handle_event` and `LocalExecutor` thread * Bail out on first task completion * mypy * Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded) * Print open file diff from `benchmark.sh` * Add `--local-executor` flag, disabled by default for now until tests are updated * Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now) * Async `get_events` * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address tests after async changes * mypy and flake8 * spelldoc * `check.py` and trailing comma * Rename to `_assertions.py` * Add missing `pytest-mock` and `pytest-asyncio` deps * Add `pytest-mock` to `pylint` deps * Correct use of `parameterize` and add `PT007` to flake8 ignores * Fix mypy hints broken for `< Python3.9` * Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix for pre-python-3.9 versions * `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions * Install setuptools via `lib-dep` until we recommend editable install * Deprecate support for `Python 3.6` * Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929 * Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906 * Make `Threadless` agnostic of `multiprocessing.Process` * Acceptors must dispatch to local executor in non-blocking fashion * No daemon for executor processes and fix shutdown logic * Only return fds from `_selected_events` not all events data * Refactor logic * Prefix private methods with `_` * `work_queue` and not `client_queue` * Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor` * Make `LocalExecutor` agnostic of `threading.Thread` * `LocalExecutor` now implements `Threadless` * `get_events` and `get_descriptors` now must return int and not sock. `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify` * Fix `main` tests * Apply suggestions from code review Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua> * Apply code review recommendations manually * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert back `Any` and use `addr or None` * Address `flake8` * Update tests to use `fileno` * Fix doc build * Fix doc spell, use tear down and not teardown * Doc updates * Add back support for `Python 3.6` * Acceptors dont need loop initialization * On Python 3.6 `asyncio.new_event_loop()` is necessary * Make doc happy * `--threaded` needs a new event loop for 3.7 too * Always use `asyncio.new_event_loop()` for threaded mode Added e2e integration tests (subprocess & curl) for all modes. * Lint fixes Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
2021-11-23 09:32:00 +00:00
async def write_to_descriptors(self, w: Writables) -> bool:
"""Implementations must now write/flush data over the socket.
Note that buffer management is in-build into the connection classes.
2021-11-16 23:34:29 +00:00
Hence implementations MUST call
:meth:`~proxy.core.connection.TcpConnection.flush` here, to send
any buffered data over the socket.
"""
return False # pragma: no cover
@abstractmethod
Async `get_events`, `handle_event`, `handle_readables`, `handle_writables` (#769) * Asynchronous `handle_event` and `LocalExecutor` thread * Bail out on first task completion * mypy * Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded) * Print open file diff from `benchmark.sh` * Add `--local-executor` flag, disabled by default for now until tests are updated * Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now) * Async `get_events` * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address tests after async changes * mypy and flake8 * spelldoc * `check.py` and trailing comma * Rename to `_assertions.py` * Add missing `pytest-mock` and `pytest-asyncio` deps * Add `pytest-mock` to `pylint` deps * Correct use of `parameterize` and add `PT007` to flake8 ignores * Fix mypy hints broken for `< Python3.9` * Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix for pre-python-3.9 versions * `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions * Install setuptools via `lib-dep` until we recommend editable install * Deprecate support for `Python 3.6` * Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929 * Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906 * Make `Threadless` agnostic of `multiprocessing.Process` * Acceptors must dispatch to local executor in non-blocking fashion * No daemon for executor processes and fix shutdown logic * Only return fds from `_selected_events` not all events data * Refactor logic * Prefix private methods with `_` * `work_queue` and not `client_queue` * Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor` * Make `LocalExecutor` agnostic of `threading.Thread` * `LocalExecutor` now implements `Threadless` * `get_events` and `get_descriptors` now must return int and not sock. `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify` * Fix `main` tests * Apply suggestions from code review Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua> * Apply code review recommendations manually * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert back `Any` and use `addr or None` * Address `flake8` * Update tests to use `fileno` * Fix doc build * Fix doc spell, use tear down and not teardown * Doc updates * Add back support for `Python 3.6` * Acceptors dont need loop initialization * On Python 3.6 `asyncio.new_event_loop()` is necessary * Make doc happy * `--threaded` needs a new event loop for 3.7 too * Always use `asyncio.new_event_loop()` for threaded mode Added e2e integration tests (subprocess & curl) for all modes. * Lint fixes Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
2021-11-23 09:32:00 +00:00
async def read_from_descriptors(self, r: Readables) -> bool:
"""Implementations must now read data over the socket."""
return False # pragma: no cover
@abstractmethod
def on_client_data(self, raw: memoryview) -> Optional[memoryview]:
"""Called only after original request has been completely received."""
return raw # pragma: no cover
@abstractmethod
def on_request_complete(self) -> Union[socket.socket, bool]:
"""Called right after client request parser has reached COMPLETE state."""
return False # pragma: no cover
@abstractmethod
def on_response_chunk(self, chunk: List[memoryview]) -> List[memoryview]:
"""Handle data chunks as received from the server.
Return optionally modified chunk to return back to client."""
return chunk # pragma: no cover
@abstractmethod
def on_client_connection_close(self) -> None:
"""Client connection shutdown has been received, flush has been called,
perform any cleanup work here.
"""
pass # pragma: no cover