proxy.py/proxy/http/server/plugin.py

125 lines
4.2 KiB
Python
Raw Normal View History

# -*- coding: utf-8 -*-
"""
proxy.py
~~~~~~~~
Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
Network monitoring, controls & Application development, testing, debugging.
:copyright: (c) 2013-present by Abhinav Singh and contributors.
:license: BSD, see LICENSE for more details.
"""
import argparse
from abc import ABC, abstractmethod
from typing import Any, Dict, List, Optional, Tuple
from ..websocket import WebsocketFrame
from ..parser import HttpParser
from ...common.types import Readables, Writables
from ...core.connection import TcpClientConnection
from ...core.event import EventQueue
class HttpWebServerBasePlugin(ABC):
"""Web Server Plugin for routing of requests."""
def __init__(
self,
v2.4.0rc3 (#882) * Merge pull request #840 from abhinavsingh/release-schedule-notes Add release schedule under FAQ * Green CI (#841) * Cleanup parser & url classes (#843) * Optimize parser logic * Add `is_complete` property * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * pre commit fixes * We need `ValueError` * wuff * useless * flake8 * Simplify url Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * pip prod(deps): bump twine from 3.6.0 to 3.7.0 (#845) Bumps [twine](https://github.com/pypa/twine) from 3.6.0 to 3.7.0. - [Release notes](https://github.com/pypa/twine/releases) - [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst) - [Commits](https://github.com/pypa/twine/compare/3.6.0...3.7.0) --- updated-dependencies: - dependency-name: twine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump jasmine from 3.6.3 to 3.10.0 in /dashboard (#844) Bumps [jasmine](https://github.com/jasmine/jasmine-npm) from 3.6.3 to 3.10.0. - [Release notes](https://github.com/jasmine/jasmine-npm/releases) - [Commits](https://github.com/jasmine/jasmine-npm/compare/v3.6.3...v3.10.0) --- updated-dependencies: - dependency-name: jasmine dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * npm: bump chrome-devtools-frontend in /dashboard (#846) Bumps [chrome-devtools-frontend](https://github.com/ChromeDevTools/devtools-frontend) from 1.0.944903 to 1.0.947377. - [Release notes](https://github.com/ChromeDevTools/devtools-frontend/releases) - [Changelog](https://github.com/ChromeDevTools/devtools-frontend/blob/main/docs/release_management.md) - [Commits](https://github.com/ChromeDevTools/devtools-frontend/commits) --- updated-dependencies: - dependency-name: chrome-devtools-frontend dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * pip prod(deps): bump coverage from 6.1.2 to 6.2 (#847) Bumps [coverage](https://github.com/nedbat/coveragepy) from 6.1.2 to 6.2. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](https://github.com/nedbat/coveragepy/compare/6.1.2...6.2) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Fix GHA check-gate to properly identify failures (#849) * pip prod(deps): bump pylint from 2.12.1 to 2.12.2 (#851) Bumps [pylint](https://github.com/PyCQA/pylint) from 2.12.1 to 2.12.2. - [Release notes](https://github.com/PyCQA/pylint/releases) - [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog) - [Commits](https://github.com/PyCQA/pylint/compare/v2.12.1...v2.12.2) --- updated-dependencies: - dependency-name: pylint dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump @types/js-cookie from 2.2.6 to 3.0.1 in /dashboard (#850) Bumps [@types/js-cookie](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/js-cookie) from 2.2.6 to 3.0.1. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/js-cookie) --- updated-dependencies: - dependency-name: "@types/js-cookie" dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * pip prod(deps): bump sphinx from 4.3.0 to 4.3.1 (#853) Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.3.0 to 4.3.1. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.3.0...v4.3.1) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * pip prod(deps): bump paramiko from 2.8.0 to 2.8.1 (#855) * npm: bump ws from 7.4.6 to 8.3.0 in /dashboard (#854) * pip prod(deps): bump uvicorn from 0.15.0 to 0.16.0 (#857) Bumps [uvicorn](https://github.com/encode/uvicorn) from 0.15.0 to 0.16.0. - [Release notes](https://github.com/encode/uvicorn/releases) - [Changelog](https://github.com/encode/uvicorn/blob/master/CHANGELOG.md) - [Commits](https://github.com/encode/uvicorn/compare/0.15.0...0.16.0) --- updated-dependencies: - dependency-name: uvicorn dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump chrome-devtools-frontend in /dashboard (#856) Bumps [chrome-devtools-frontend](https://github.com/ChromeDevTools/devtools-frontend) from 1.0.947377 to 1.0.949424. - [Release notes](https://github.com/ChromeDevTools/devtools-frontend/releases) - [Changelog](https://github.com/ChromeDevTools/devtools-frontend/blob/main/docs/release_management.md) - [Commits](https://github.com/ChromeDevTools/devtools-frontend/commits) --- updated-dependencies: - dependency-name: chrome-devtools-frontend dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Process `--enable-*` flags before loading plugins (#860) * Process `--enable-*` flags before loading plugins * Fix spelling * Ignore RST299 and RST499 * npm: bump http-server from 0.12.3 to 14.0.0 in /dashboard (#858) Bumps [http-server](https://github.com/http-party/http-server) from 0.12.3 to 14.0.0. - [Release notes](https://github.com/http-party/http-server/releases) - [Commits](https://github.com/http-party/http-server/compare/v0.12.3...v14.0.0) --- updated-dependencies: - dependency-name: http-server dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * pip prod(deps): bump furo from 2021.11.15 to 2021.11.23 (#859) Bumps [furo](https://github.com/pradyunsg/furo) from 2021.11.15 to 2021.11.23. - [Release notes](https://github.com/pradyunsg/furo/releases) - [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md) - [Commits](https://github.com/pradyunsg/furo/compare/2021.11.15...2021.11.23) --- updated-dependencies: - dependency-name: furo dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Update web log context fields to match proxy log context fields (#861) * Update web log context fields to match those with proxy log context for consistency * Fix spell * Fix `DEFAULT_WEB_ACCESS_LOG_FORMAT` * pip prod(deps): bump pytest-xdist from 2.4.0 to 2.5.0 (#864) Bumps [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/pytest-dev/pytest-xdist/releases) - [Changelog](https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest-xdist/compare/v2.4.0...v2.5.0) --- updated-dependencies: - dependency-name: pytest-xdist dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * npm: bump eslint-plugin-node from 10.0.0 to 11.1.0 in /dashboard (#863) Bumps [eslint-plugin-node](https://github.com/mysticatea/eslint-plugin-node) from 10.0.0 to 11.1.0. - [Release notes](https://github.com/mysticatea/eslint-plugin-node/releases) - [Commits](https://github.com/mysticatea/eslint-plugin-node/compare/v10.0.0...v11.1.0) --- updated-dependencies: - dependency-name: eslint-plugin-node dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * Fix broken TLS interception & CacheResponsesPlugin because UID is no longer a UUID (#866) * Fix broken TLS interception because uid is now no longer a UUID * Give enough context to work id for them to be unique within a `proxy.py` instance * Use --port=0 by default within `proxy.TestCase` * Attempt to fix weird buildx issue * Add makefile targets within workflow * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Order? * Write scm file for make * Fetch depth * Quote patch * Try with sudo? * https://github.com/docker/buildx/issues/850 * Remove sudo hack * https://github.com/docker/buildx/issues/850\#issuecomment-973270625 * Add explicit deps * Add `requirements-testing.txt` during linting phase * Pin buildx to v0.7.1 * Pin buildx to v0.7.0 * Revert back unnecessary change to dockerignore * Skip container within make workflow (because GHA lacks support for docker on macOS by default) * Repurpose make into developer workflow Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Integrate showing unreleased changelog draft (#873) * pip prod(deps): bump types-paramiko from 2.8.2 to 2.8.4 (#868) * npm: bump @types/jasmine from 3.6.1 to 3.10.2 in /dashboard (#867) * pip prod(deps): bump py-spy from 0.3.10 to 0.3.11 (#875) Bumps [py-spy](https://github.com/benfred/py-spy) from 0.3.10 to 0.3.11. - [Release notes](https://github.com/benfred/py-spy/releases) - [Changelog](https://github.com/benfred/py-spy/blob/master/CHANGELOG.md) - [Commits](https://github.com/benfred/py-spy/compare/v0.3.10...v0.3.11) --- updated-dependencies: - dependency-name: py-spy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com> * [GHA] Add container integration test & publish containers to GHCR (#818) * Handle KBI in threadless * Remove line-length disable clause * Move `Set PROXYPY_CONTAINER_VERSION env` within pre-setup * Use step output * Use review suggestion * Break line at less essential places * Possibly fix? * alignment necessary? * :) * Fix integration * Add `v` prefix * Load buildx image within docker for integration tests to work * Run container in detached mode * Enable web server for integration test to work * Push to GHCR after successful tests * Tag & Push to GHCR * Add package permission * Login to GHCR * Login to GHCR * Login to GHCR * Explicitly use ghcr.io registry * Add a ghcr manifest creation step which depends upon docker step * Probably we need all images (tags) locally * arm not amd * Fix syntax * Fix typo * `Unable to find image ghcr.io/abhinavsingh/proxy.py:latest locally` * GHCR doesnt support manifest? * Publish multi-platform containers on GHCR (#877) * Build containers without matrix based strategy. Helps with buildx based manifest generation * Use buildx directly in workflows * Add PROXYPY_PKG_PATH arg * Add missing . * --push from buildx * Also add latest tag for GHCR * Fix typo * Explain differences between latest tag on DockerHub (stable) and GHCR (develop) * Publish multi-platform containers to DockerHub (#878) * Use `--local-executor` flag by default for Docker container (#880) * Benchmark gets packaged within wheel if set as a package * Use `--local-executor` flag by default for Docker containers * Dockerfile update * Fix mypy issues * Remove conflicting dir names * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
2021-12-19 16:00:43 +00:00
uid: str,
flags: argparse.Namespace,
client: TcpClientConnection,
event_queue: EventQueue,
):
self.uid = uid
self.flags = flags
self.client = client
self.event_queue = event_queue
def name(self) -> str:
"""A unique name for your plugin.
Defaults to name of the class. This helps plugin developers to directly
access a specific plugin by its name."""
return self.__class__.__name__ # pragma: no cover
# TODO(abhinavsingh): get_descriptors, write_to_descriptors, read_from_descriptors
# can be placed into their own abstract class which can then be shared by
# HttpProxyBasePlugin, HttpWebServerBasePlugin and HttpProtocolHandlerPlugin class.
#
# Currently code has been shamelessly copied. Also these methods are not
# marked as abstract to avoid breaking custom plugins written by users for
# previous versions of proxy.py
#
# Since 3.4.0
#
# @abstractmethod
Async `get_events`, `handle_event`, `handle_readables`, `handle_writables` (#769) * Asynchronous `handle_event` and `LocalExecutor` thread * Bail out on first task completion * mypy * Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded) * Print open file diff from `benchmark.sh` * Add `--local-executor` flag, disabled by default for now until tests are updated * Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now) * Async `get_events` * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address tests after async changes * mypy and flake8 * spelldoc * `check.py` and trailing comma * Rename to `_assertions.py` * Add missing `pytest-mock` and `pytest-asyncio` deps * Add `pytest-mock` to `pylint` deps * Correct use of `parameterize` and add `PT007` to flake8 ignores * Fix mypy hints broken for `< Python3.9` * Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix for pre-python-3.9 versions * `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions * Install setuptools via `lib-dep` until we recommend editable install * Deprecate support for `Python 3.6` * Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929 * Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906 * Make `Threadless` agnostic of `multiprocessing.Process` * Acceptors must dispatch to local executor in non-blocking fashion * No daemon for executor processes and fix shutdown logic * Only return fds from `_selected_events` not all events data * Refactor logic * Prefix private methods with `_` * `work_queue` and not `client_queue` * Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor` * Make `LocalExecutor` agnostic of `threading.Thread` * `LocalExecutor` now implements `Threadless` * `get_events` and `get_descriptors` now must return int and not sock. `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify` * Fix `main` tests * Apply suggestions from code review Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua> * Apply code review recommendations manually * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert back `Any` and use `addr or None` * Address `flake8` * Update tests to use `fileno` * Fix doc build * Fix doc spell, use tear down and not teardown * Doc updates * Add back support for `Python 3.6` * Acceptors dont need loop initialization * On Python 3.6 `asyncio.new_event_loop()` is necessary * Make doc happy * `--threaded` needs a new event loop for 3.7 too * Always use `asyncio.new_event_loop()` for threaded mode Added e2e integration tests (subprocess & curl) for all modes. * Lint fixes Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
2021-11-23 09:32:00 +00:00
def get_descriptors(self) -> Tuple[List[int], List[int]]:
return [], [] # pragma: no cover
# @abstractmethod
def write_to_descriptors(self, w: Writables) -> bool:
"""Implementations must now write/flush data over the socket.
Note that buffer management is in-build into the connection classes.
2021-11-16 23:34:29 +00:00
Hence implementations MUST call
:meth:`~proxy.core.connection.connection.TcpConnection.flush`
here, to send any buffered data over the socket.
"""
return False # pragma: no cover
# @abstractmethod
def read_from_descriptors(self, r: Readables) -> bool:
"""Implementations must now read data over the socket."""
return False # pragma: no cover
@abstractmethod
def routes(self) -> List[Tuple[int, str]]:
"""Return List(protocol, path) that this plugin handles."""
raise NotImplementedError() # pragma: no cover
@abstractmethod
def handle_request(self, request: HttpParser) -> None:
"""Handle the request and serve response."""
raise NotImplementedError() # pragma: no cover
def on_client_connection_close(self) -> None:
"""Client has closed the connection, do any clean up task now."""
pass
# No longer abstract since v2.4.0
#
# @abstractmethod
def on_websocket_open(self) -> None:
"""Called when websocket handshake has finished."""
pass # pragma: no cover
# No longer abstract since v2.4.0
#
# @abstractmethod
def on_websocket_message(self, frame: WebsocketFrame) -> None:
"""Handle websocket frame."""
return None # pragma: no cover
# Deprecated since v2.4.0
#
# Instead use on_client_connection_close.
#
# This callback is no longer invoked. Kindly
# update your plugin before upgrading to v2.4.0.
#
# @abstractmethod
# def on_websocket_close(self) -> None:
# """Called when websocket connection has been closed."""
# raise NotImplementedError() # pragma: no cover
def on_access_log(self, context: Dict[str, Any]) -> Optional[Dict[str, Any]]:
"""Use this method to override default access log format (see
DEFAULT_WEB_ACCESS_LOG_FORMAT) or to add/update/modify passed context
for usage by default access logger.
Return updated log context to use for default logging format, OR
Return None if plugin has logged the request.
"""
return context