* Changes for Python 3.11 support
* Updated README.md for versioning info
* Update `httpx==0.27.0` to avoid `cgi` deprecation warning from pytest on Python 3.11
* Make tests work for 3.11
* Declare support for 3.11
* Use 3.11-alpine for Docker images
* Preserve pylint version for `python_version <= 3.10`
* Preserve httpx version for <= 3.10
* `httpx` usage fix in tests for <=3.10
* Adjust pylint and pytest for >= 3.11
* Use 3.11.8, bad-option-value and httpx proxies fix
* tox for 3.11
* Fix for `TOXENV: py`
* -vv for pytest
* Downgrade to `pytest-asyncio==0.21.1`
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove asyncio_mode=strict
* try with `pytest-cov==4.1.0` for 3.11
* bump coverage for 3.11
* Try `3.11` in GitHub workflow which installs >3.11.8 unavailable via pyenv yet
* Revert back to `-v`
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix broken examples
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix mypy
* Pylint ignore
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Refactor into an internal task submodule of work
* As context managers
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add missing license
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Pre-release cleanups
* Add listener pool test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add multi listener test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* isort `proxy.py` main class
* isort init and main
* isort common
* pre-commit fix
* isort dashboard and testing
* isort plugins
* isort core
* Only sort top level http py files
* isort http exception and websocket
* Remove proxy auth plugin from proxy package exports and force discover `PLUGIN_PROXY_AUTH` flags
* isort parser and web server
* no setattr
* isort all
* Enable pre-commit isort hook
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Turn usual suspects to warnings, not error
* Add `HttpClientConnection` skeleton
* Fix doc build
* Update references in http tests
* Make `work` core agnostic to work object construction by adding an abstract static method to `Work` interface called `create`
* Make mypy happy
* Fix tests broken due to change in how work objects are now constructed
* Doc ko bhi happy karo
* Move all TCP server related flags within `tcp_server.py` and also move the encryption functionality within TCP base server
* Templatize `BaseTcpServerHandler` which now expects a client connection object bound to `TcpClientConnection`. This will allow for custom `HttpClientConnection` object in future to be used by `HttpProtocolHandler`
* Pass necessary flags to allow self-signed certificates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix https integration tests
* Affected by https://github.com/abhinavsingh/proxy.py/issues/994
* Fix docs
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* work module
* Fix imports
* String based typing for multiprocessing.synchronize
* Fix `test_accepts_client_from_server_socket`
* Move staticmethod outside of threadless pool class
* Fix doc build
* Fix test mock
* mp grouped together
* pylint happy
* import only for type checking
* doc build
* wrong import order
* Add `Descriptors` type
* Add a `DescriptorsHandlerMixin` class used throughout the http framework
* Remove dependency upon `HasFileno` ie `typing_extension` too
* Define `SelectableEvents` type
* Fix doc
* Blank line
* Remove dep on `typing-extensions`
* Discover base plugin class
* await on now async handlers
* Work can also be TcpServerConnection, not just TcpClientConnection. More over, it can be any generic work type
* Add py_class_role and py_obj
* Port internal integration tests into public repo
* Fix proxy py addr
* Use cross-platform compat shasum
* Add response pkt utility
* Unused import
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix tests as some content is now by default gzipped based upon min compression config
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove unused
* Update necessary tests to use `okResponse` utility
* Add option to explicitly disable compression
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Bail out of lock earlier
* Avoid calling `get_events` for unfinished work tasks. Use `cached_property` for `HttpParser` optimization
* Add `type: ignore[no-any-return]`, odd scenario
* We dont have to rebuild response packet repeatedly within `WebServerPlugin`
* Parse line and header in one invocation
* Minor optimizations and update benchmark to use `oha` instead of `hey`
* Remove `flask` from benchmark, only benchmark `asgi` or `async` based libraries. Use `uvicorn` and 10 workers for `blacksheep`. Use `oha` instead of `hey`
* Add benchmark for `starlette`
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add missing dep
* pre-commit
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add benchmark results to top-level README
* mypy
* Push down WIP example
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Spelling
* Link rtfd within `Internal Documentation` for now
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Asynchronous `handle_event` and `LocalExecutor` thread
* Bail out on first task completion
* mypy
* Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded)
* Print open file diff from `benchmark.sh`
* Add `--local-executor` flag, disabled by default for now until tests are updated
* Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now)
* Async `get_events`
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address tests after async changes
* mypy and flake8
* spelldoc
* `check.py` and trailing comma
* Rename to `_assertions.py`
* Add missing `pytest-mock` and `pytest-asyncio` deps
* Add `pytest-mock` to `pylint` deps
* Correct use of `parameterize` and add `PT007` to flake8 ignores
* Fix mypy hints broken for `< Python3.9`
* Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix for pre-python-3.9 versions
* `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions
* Install setuptools via `lib-dep` until we recommend editable install
* Deprecate support for `Python 3.6`
* Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929
* Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906
* Make `Threadless` agnostic of `multiprocessing.Process`
* Acceptors must dispatch to local executor in non-blocking fashion
* No daemon for executor processes and fix shutdown logic
* Only return fds from `_selected_events` not all events data
* Refactor logic
* Prefix private methods with `_`
* `work_queue` and not `client_queue`
* Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor`
* Make `LocalExecutor` agnostic of `threading.Thread`
* `LocalExecutor` now implements `Threadless`
* `get_events` and `get_descriptors` now must return int and not sock. `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify`
* Fix `main` tests
* Apply suggestions from code review
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
* Apply code review recommendations manually
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert back `Any` and use `addr or None`
* Address `flake8`
* Update tests to use `fileno`
* Fix doc build
* Fix doc spell, use tear down and not teardown
* Doc updates
* Add back support for `Python 3.6`
* Acceptors dont need loop initialization
* On Python 3.6 `asyncio.new_event_loop()` is necessary
* Make doc happy
* `--threaded` needs a new event loop for 3.7 too
* Always use `asyncio.new_event_loop()` for threaded mode
Added e2e integration tests (subprocess & curl) for all modes.
* Lint fixes
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
* Remove abstractmethod for proxy plugin base class, remove unused methods from bundled plugins
* Move httpStatusCodes, httpMethods and Url within top-level proxy.http package
* Custom Url parser for our needs
* lint fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix proxy_pool plugin as scheme can be None if not present in the Url
* Address the ambiguous ipv6:port scenario along with valid cases
* lint checks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* docstring
* Abstract into `http.parser` module
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix#398 HTTP/1.0 related issue
* lint checks
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* deprecate server_file_or_404
* Optionally compress static content. Currently only if content length higher than 300
* trailing comma
* Allow `--plugins` flag to be used multiple times
Following are valid invocation:
1) `--plugins A`
2) `--plugins A,B`
3) `--plugins A --plugins B`
4) `--plugins A,B --plugins C`
* mypy
* Flake8
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* correct type
* Add `HttpParser.is_https_tunnel()` utility method
* mypy
* lint checks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Abstract out a Listener class
* unused
* Use connection instead of manager queue
* For web close connection of client requested via headers
* Remove eventing WIP module
* Sub and Unsub ack
* Fix tests
* mypy and flake8
* comma
* Move callback within EventSubscriber constructor
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip test_unix_path_listener on Windows
* Spelling fix
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Allow overriding work_klass via Proxy context manager kwargs
* Decouple acceptor and executor pools
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add `--num_acceptors` flag and better load balancing
* Remove unused
* Lint errors
* Another arg not kwarg
* Move start work staticmethods within ExecutorPool
* mypy fixes
* Update README with `--num-acceptors` flag
* Rename `Proxy.pool` to `Proxy.acceptors`
* Add SetupShutdownContextManager abstraction
* Match --num-acceptors logic with PR description
* Rename executor utility methods and add docstring
* Remove work_klass from constructors and pass it via flags
* Update docstring for pools as they no longer accept a work_klass argument
* Turn work_klass into a flag. main() no longer accepts input_args (only kwargs opts). Similarly, Proxy doesnt accept any input_args now (only kwargs opts)
* Expose default work klass in README
* Expose `HttpProtocolHandler` and `HttpProtocolHandlerPlugin` within `proxy.http` module
* Start to fix tests
* Fix tests
* mypy and flake8
* Trailing comma
* Remove unused var
* Unused arg
* uff
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Move flags to where they belong
* Move `get_default_plugins` within FlagParser as it depends upon args
TODO: We need plugin dependency system
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add a `--unix-socket-path` flag.
When available `--hostname` and `--port` flags are ignored.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* `print` statement is allowed only in `flags.py` and `version-check.py`. All other places must use a `logger` instance
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add guard for `AF_UNIX` on Windows
* Comment out assertion on Windows for now
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Move Proxy.initialize within FlagParser.initialize. Also move other staticmethods from within proxy class into utils
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove unused imports
* Fix `import-outside-toplevel` error
* add `make lib-flake8` and `make lib-mypy` targets
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Refactor pool
* mypy fixes
* Fix import (relative)
* Add WebScraper example skeleton & ConnectionPool skeleton
* Add ConnectionPool class
* Integrate ConnectionPool with proxy server (experimental)
* Lint fixes
* Remove unused imports. TODO: Put pool behind a flag. Default to false for now
* Make ConnectionPool multiprocess safe. Later we want to make it safe but without using locks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove unused imports
* Return created flag from acquire
* Guard connection pool behind --enable-conn-pool flag
* Flag belongs within connection pool class
* spelling
* self.upstream = None only for pool config
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Rename .server to .upstream
* Lint fixes
* Mark internal methods with _ prefix
* Fix broken test
* lint changes
* Wah, double client :D
* Avoid selector initialization for threadless mode
* remove unused imports
* Now HttpProtocolHandler implements BaseTcpServerHandler
* Consistent return and guard againt upstream.closed
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* AcceptorPool as context manager
* Group multiprocessing imports together
* Use com.jaxl bundle identifier as proxy.py will eventually move under jaxl org
* revisit devtools integration :)
* Emit all necessary events for devtools integration
* Lint fixes
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Async proxy pool
* Async proxy pool
* Late upstream initialization and exception guards
* Close upstream proxy connection on client connection close
* Refactor into EventManager
* Fix tests accounting in the event manager
* Ensure each process initializes logger
* pragma no cover
* Teardown connection when proxy pool upstream proxy closes
* Add ability to customize access log format and add additional context to it
* Maintain total size for response bytes in access logs
* Fix tests broken due to new plugin methods missing mock
* Update pubsub_eventing to use EventManager to avoid entire bootstrapping step
* Ensure pending buffers are flushed before shutting down in base_server.py
Handle unsupported scheme cases within connect_tunnel.py
* Move base implementations within core module
* Update ssl_echo_server
* Go flagless to allow custom user defined flags. Fixes#301
* Add --cache-dir flag for cache plugin (when used with on-disk store)
* Enable discovery of flags from external plugins, example those that reside outside of proxy.py package and loaded on demand. This also allows external flags to surface in --help section
* Define --filtered-client-ips flag for FilterByClientIpPlugin
* Move wrap_socket for SSL server within utils.
Also complete proxy.common.pki gen_csr and sign_csr actions. Used by Makefile sign-https-certificates.
* Add SSL echo server and client example
* Add examples documentation
* Move wrap functionality within respective connection classes. Also decouple websocket client handshake method
* Add a TCP echo client example that works with TCP echo server example
* Better document acceptor module and add a TCP Echo Server example
* autopep8 formating
* Rename ThreadlessWork --> Work class
* Make initialize, is_inactive and shutdown as optional interface methods.
Also introduce Readables & Writables custom types.
* Move websocket code into its own module
* Add websocket client example
* Cleanup websocket client
Abhinav Singh