Commit Graph

15 Commits

Author SHA1 Message Date
Abhinav Singh 44d72431e3
Async `get_events`, `handle_event`, `handle_readables`, `handle_writables` (#769)
* Asynchronous `handle_event` and `LocalExecutor` thread

* Bail out on first task completion

* mypy

* Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded)

* Print open file diff from `benchmark.sh`

* Add `--local-executor` flag, disabled by default for now until tests are updated

* Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now)

* Async `get_events`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address tests after async changes

* mypy and flake8

* spelldoc

* `check.py` and trailing comma

* Rename to `_assertions.py`

* Add missing `pytest-mock` and `pytest-asyncio` deps

* Add `pytest-mock` to `pylint` deps

* Correct use of `parameterize` and add `PT007` to flake8 ignores

* Fix mypy hints broken for `< Python3.9`

* Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix for pre-python-3.9 versions

* `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions

* Install setuptools via `lib-dep` until we recommend editable install

* Deprecate support for `Python 3.6`

* Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929

* Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906

* Make `Threadless` agnostic of `multiprocessing.Process`

* Acceptors must dispatch to local executor in non-blocking fashion

* No daemon for executor processes and fix shutdown logic

* Only return fds from `_selected_events` not all events data

* Refactor logic

* Prefix private methods with `_`

* `work_queue` and not `client_queue`

* Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor`

* Make `LocalExecutor` agnostic of `threading.Thread`

* `LocalExecutor` now implements `Threadless`

* `get_events` and `get_descriptors` now must return int and not sock.  `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify`

* Fix `main` tests

* Apply suggestions from code review

Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>

* Apply code review recommendations manually

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert back `Any` and use `addr or None`

* Address `flake8`

* Update tests to use `fileno`

* Fix doc build

* Fix doc spell, use tear down and not teardown

* Doc updates

* Add back support for `Python 3.6`

* Acceptors dont need loop initialization

* On Python 3.6 `asyncio.new_event_loop()` is necessary

* Make doc happy

* `--threaded` needs a new event loop for 3.7 too

* Always use `asyncio.new_event_loop()` for threaded mode

Added e2e integration tests (subprocess & curl) for all modes.

* Lint fixes

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
2021-11-23 15:02:00 +05:30
Abhinav Singh 5eb4fa4acd
Add `.vscode` settings (#761) 2021-11-20 01:25:05 +05:30
Abhinav Singh 3cfce5265c
Add `TcpUpstreamConnectionHandler` class (#760)
* Add `TcpUpstreamConnectionHandler` which can be used as standalone or as mixin

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* `TcpUpstreamConnectionHandler` is now an abstract class

* Fix mypy

* `nitpick_ignore` the `proxy.core.base.tcp_upstream.TcpUpstreamConnectionHandler` class

* Add mypy exception for now

* Fix flake

* Fix docstring

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-11-20 00:05:24 +05:30
Sviatoslav Sydorenko fa89944891
Add initial Sphinx docs 2021-11-18 01:36:13 +01:00
Abhinav Singh d72ee2227d
`--enable-proxy-protocol` : HAProxy Protocol v1 (#735)
* Introduce `--haproxy-protocol` flag

* Complete proxy protocol v1 implementation, enable using `--enable-proxy-protocol` flag

* link checks

* Advertise support for haproxy protocol in readme

* Add make target `lib-scm-version`

* `make lib-version` is now `make lib-check`

* Dont enforce -dev part of version within README

* Add provision to update readme flags using check

* Wrap help text within console

* Add closing ticks

* Remove verbose logging and update homebrew formulae (may be fixed?)
2021-11-14 10:43:20 +05:30
Abhinav Singh e38f1a8fee
Test cov (#721)
* nocover for abstract classes

* Add event manager test

* Assert call args

* Sponsored by `Jaxl Innovations Private Limited`
2021-11-10 21:40:51 +05:30
Abhinav Singh 3cb0a62583
Add a `--unix-socket-path` flag (#697)
* Add a `--unix-socket-path` flag.

When available `--hostname` and `--port` flags are ignored.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* `print` statement is allowed only in `flags.py` and `version-check.py`.  All other places must use a `logger` instance

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add guard for `AF_UNIX` on Windows

* Comment out assertion on Windows for now

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-11-08 01:11:46 +05:30
Abhinav Singh f48771fb41
Top-level notion of work not client (#695)
* Top-level notion of work not client

* Update ssl echo server example
2021-11-07 21:43:38 +05:30
Abhinav Singh d3cee32909
Pool (#694)
* Refactor pool

* mypy fixes

* Fix import (relative)

* Add WebScraper example skeleton & ConnectionPool skeleton

* Add ConnectionPool class

* Integrate ConnectionPool with proxy server (experimental)

* Lint fixes

* Remove unused imports. TODO: Put pool behind a flag. Default to false for now

* Make ConnectionPool multiprocess safe.  Later we want to make it safe but without using locks

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove unused imports

* Return created flag from acquire

* Guard connection pool behind --enable-conn-pool flag

* Flag belongs within connection pool class

* spelling

* self.upstream = None only for pool config

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-11-07 21:06:41 +05:30
Abhinav Singh da23c7f70c
Work (#693)
* Refactor work acceptor and executor

* Lint fixes

* Fix expression-not-assigned pylint error
2021-11-07 05:02:52 +05:30
Abhinav Singh 6121fe9004
must_flush_before_shutdown until entire client buffer has been flushed (#686) 2021-11-06 18:26:39 +05:30
Abhinav Singh 628de71ca8
Adopt BaseTcpServerHandler within HttpProtocolHandler (#681)
* Rename .server to .upstream

* Lint fixes

* Mark internal methods with _ prefix

* Fix broken test

* lint changes

* Wah, double client :D

* Avoid selector initialization for threadless mode

* remove unused imports

* Now HttpProtocolHandler implements BaseTcpServerHandler

* Consistent return and guard againt upstream.closed

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* AcceptorPool as context manager

* Group multiprocessing imports together

* Use com.jaxl bundle identifier as proxy.py will eventually move under jaxl org

* revisit devtools integration :)

* Emit all necessary events for devtools integration

* Lint fixes

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-11-06 14:01:13 +05:30
Sviatoslav Sydorenko 05a78567ac
Enable the `add-trailing-comma` pre-commit fixer (#661)
Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com>
2021-11-04 16:58:36 +05:30
Sviatoslav Sydorenko 1bd97678c8
Correct spelling mistakes caught by `codespell` (#656)
Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com>
2021-11-03 17:33:05 +05:30
Abhinav Singh 4520ae31a6
Refactor base server interfaces into core modules (#461)
* Ensure pending buffers are flushed before shutting down in base_server.py

Handle unsupported scheme cases within connect_tunnel.py

* Move base implementations within core module

* Update ssl_echo_server
2020-11-24 21:37:11 +05:30