* Bail out of lock earlier
* Avoid calling `get_events` for unfinished work tasks. Use `cached_property` for `HttpParser` optimization
* Add `type: ignore[no-any-return]`, odd scenario
* We dont have to rebuild response packet repeatedly within `WebServerPlugin`
* Parse line and header in one invocation
* Minor optimizations and update benchmark to use `oha` instead of `hey`
* Remove `flask` from benchmark, only benchmark `asgi` or `async` based libraries. Use `uvicorn` and 10 workers for `blacksheep`. Use `oha` instead of `hey`
* Add benchmark for `starlette`
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add missing dep
* pre-commit
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Optimize `find_http_line` which is in critical path
* Update benchmark results
* Keep the loop hot, TCP no delay, cleanup inactive check periodically
* Check for shutdown signal with tick
* Use non-reentrant `NonBlockingQueue` implementation instead of `queue.Queue`
* Fix listener test
* lint and doc
* Asynchronous `handle_event` and `LocalExecutor` thread
* Bail out on first task completion
* mypy
* Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded)
* Print open file diff from `benchmark.sh`
* Add `--local-executor` flag, disabled by default for now until tests are updated
* Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now)
* Async `get_events`
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address tests after async changes
* mypy and flake8
* spelldoc
* `check.py` and trailing comma
* Rename to `_assertions.py`
* Add missing `pytest-mock` and `pytest-asyncio` deps
* Add `pytest-mock` to `pylint` deps
* Correct use of `parameterize` and add `PT007` to flake8 ignores
* Fix mypy hints broken for `< Python3.9`
* Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix for pre-python-3.9 versions
* `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions
* Install setuptools via `lib-dep` until we recommend editable install
* Deprecate support for `Python 3.6`
* Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929
* Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906
* Make `Threadless` agnostic of `multiprocessing.Process`
* Acceptors must dispatch to local executor in non-blocking fashion
* No daemon for executor processes and fix shutdown logic
* Only return fds from `_selected_events` not all events data
* Refactor logic
* Prefix private methods with `_`
* `work_queue` and not `client_queue`
* Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor`
* Make `LocalExecutor` agnostic of `threading.Thread`
* `LocalExecutor` now implements `Threadless`
* `get_events` and `get_descriptors` now must return int and not sock. `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify`
* Fix `main` tests
* Apply suggestions from code review
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
* Apply code review recommendations manually
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert back `Any` and use `addr or None`
* Address `flake8`
* Update tests to use `fileno`
* Fix doc build
* Fix doc spell, use tear down and not teardown
* Doc updates
* Add back support for `Python 3.6`
* Acceptors dont need loop initialization
* On Python 3.6 `asyncio.new_event_loop()` is necessary
* Make doc happy
* `--threaded` needs a new event loop for 3.7 too
* Always use `asyncio.new_event_loop()` for threaded mode
Added e2e integration tests (subprocess & curl) for all modes.
* Lint fixes
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
* Use threads for delegation. Now `run_once` lock before `accept` not `select`
* Add support to use master proxy within proxy pool plugin. When used, proxy pool plugin will be a no-op for the master node
* Fix acceptor tests now that mask is being used
* Use `cached_property` for web server routes
* Use `select(timeout=1)` otherwise acceptor wont join if total blocking
* mypy, flake, doc spell fixes
* R0205: Class `cached_property` inherits from object, can be safely removed from bases in python3 (useless-object-inheritance)
* Update `make lib-profile`
* Optimize `utils.find_http_line`
* Pass work to executors within their own multiprocessing lock
* Fix tests
* Add `(_py_class_role, Url)` to fix rtfd :D
* Abstract out a Listener class
* unused
* Use connection instead of manager queue
* For web close connection of client requested via headers
* Remove eventing WIP module
* Sub and Unsub ack
* Fix tests
* mypy and flake8
* comma
* Move callback within EventSubscriber constructor
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip test_unix_path_listener on Windows
* Spelling fix
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Allow overriding work_klass via Proxy context manager kwargs
* Decouple acceptor and executor pools
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add `--num_acceptors` flag and better load balancing
* Remove unused
* Lint errors
* Another arg not kwarg
* Move start work staticmethods within ExecutorPool
* mypy fixes
* Update README with `--num-acceptors` flag
* Rename `Proxy.pool` to `Proxy.acceptors`
* Add SetupShutdownContextManager abstraction
* Match --num-acceptors logic with PR description
* Rename executor utility methods and add docstring
* Remove work_klass from constructors and pass it via flags
* Update docstring for pools as they no longer accept a work_klass argument
* Turn work_klass into a flag. main() no longer accepts input_args (only kwargs opts). Similarly, Proxy doesnt accept any input_args now (only kwargs opts)
* Expose default work klass in README
* Expose `HttpProtocolHandler` and `HttpProtocolHandlerPlugin` within `proxy.http` module
* Start to fix tests
* Fix tests
* mypy and flake8
* Trailing comma
* Remove unused var
* Unused arg
* uff
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* print mode via acceptor pool
* `OP_NO_TLSv1_1` by default for upstream connection negotiations. Fixes#639
* Proper fix for flaky static web server test.
Diff in payload was due to a different compression algorithm being used.
* mypy fixes
* Explicit `multiprocessing.Manager.shutdown`
Multiprocessing manager is used within eventing core. From doc,
it appears to start a BaseManager which starts a server????
Seriously???? Anyways, using multiprocessing manager is a PITA
and mistake, as it doesn't even give us performance we expect.
Our proxy server can handle more requests than what multiprocess
manager can exchange between processes.
* `--threadless is now ON by default for `Python 3.8+` on `mac` and `linux` environments
* Clarity around why multiprocessing.Manager must be deprecated
* Add `--threaded` flag which can be used to fallback for environments where `--threadless` is now default
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* never used
* Update README
* Use `threaded=True` in tests which were written for threaded model
* Fix issue where sharing manager between global event queue and subscriber can lead to TypeError
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Expose within __all__
* Enable `--numprocesses=auto` for `pytest.ini`
* make lib-lint
* Also consider `--plugins` flag when bootstrapping plugins
* Add `from .dashboard import ProxyDashboard` in top-level `__init__.py` to make `ProxyDashboard` flags auto discoverable
* Move `--enable-dashboard` to top-level
* Move logging utility within `Logger` class
* Consider comma separated --plugin and --plugins during discover_plugins
* Refactor plugin related utilities in Plugins module
* mypy and lint
* Fix unused import
* Safe to use tempdir on Github actions to avoid race conditions???
* pki (generically disk based file) based tests are flaky on macOS under parallel execution
* Move flags to where they belong
* Move `get_default_plugins` within FlagParser as it depends upon args
TODO: We need plugin dependency system
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add a `--unix-socket-path` flag.
When available `--hostname` and `--port` flags are ignored.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* `print` statement is allowed only in `flags.py` and `version-check.py`. All other places must use a `logger` instance
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add guard for `AF_UNIX` on Windows
* Comment out assertion on Windows for now
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Refactor pool
* mypy fixes
* Fix import (relative)
* Add WebScraper example skeleton & ConnectionPool skeleton
* Add ConnectionPool class
* Integrate ConnectionPool with proxy server (experimental)
* Lint fixes
* Remove unused imports. TODO: Put pool behind a flag. Default to false for now
* Make ConnectionPool multiprocess safe. Later we want to make it safe but without using locks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove unused imports
* Return created flag from acquire
* Guard connection pool behind --enable-conn-pool flag
* Flag belongs within connection pool class
* spelling
* self.upstream = None only for pool config
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add FAQ: OSError when wrapping client for TLS Interception
* Silence exception log for several valid "cert verification failed" by client during tls interception
* Lint checks
* Move exception handling within wrap_server/wrap_client methods
* Lint fixes
* Use certifi/cacert.pem as default --ca-file flag value
* Address tests after DEFAULT_CA_FILE change
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Use suggested fix in #683 to remove pytest ignore
* Handle should not flush now which can block, instead let base tcp handler do its magic
* test speed up, doc string, logging enhancements
* Move macOS to the end of workflows
* Fix mypy warnings
* Rename .server to .upstream
* Lint fixes
* Mark internal methods with _ prefix
* Fix broken test
* lint changes
* Wah, double client :D
* Avoid selector initialization for threadless mode
* remove unused imports
* Now HttpProtocolHandler implements BaseTcpServerHandler
* Consistent return and guard againt upstream.closed
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* AcceptorPool as context manager
* Group multiprocessing imports together
* Use com.jaxl bundle identifier as proxy.py will eventually move under jaxl org
* revisit devtools integration :)
* Emit all necessary events for devtools integration
* Lint fixes
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Make reverse proxy plugin use proxy.py core loop for async io operations
* Address lint errors
* Deprecate on_websocket_close and replace with on_client_connection_close
* Lint fixes
* Retry on SSLWantReadError and SSLWantWriteError
* Rename is_py3 to is_py2 for more logical guard
* Add stubs for missing tests, add few more tests for core modules
* Lint fixes
* Line too long fix
* Remove unnecessary KeyboardInterrupt
* Consistent workflow names
* Update homebrew formulae. Doesnt seems to work now
* test_enable_dashboard and test_enable_events
* test_enable_dashboard and test_enable_events
* Fix problem where empty plugin string was passed as plugin module
* test_enable_devtools and remove redundant guards for None and "" which was there due to a bug
* Async proxy pool
* Async proxy pool
* Late upstream initialization and exception guards
* Close upstream proxy connection on client connection close
* Refactor into EventManager
* Fix tests accounting in the event manager
* Ensure each process initializes logger
* pragma no cover
* Teardown connection when proxy pool upstream proxy closes
* Add ability to customize access log format and add additional context to it
* Maintain total size for response bytes in access logs
* Fix tests broken due to new plugin methods missing mock
* Update pubsub_eventing to use EventManager to avoid entire bootstrapping step
* Build docker from 3.10-alpine
* Bump version to 3.4.0
* Add instructions for how to run dashboard
* Order of menu
* Override dashboard png path until submitted
* Add some doc string for top-level Proxy class. Also some TODOs and warnings regarding PID file overwrite
* Allow HttpProxyBasePlugin implementations to register custom descriptors for read/write events
* Remove hardcoded adblock regex into json config. Update upstream filter to block facebook, not google
* ProxyPoolPlugin and ReverseProxyPlugin must now be updated to use get/read/write descriptor APIs
* Add get/read/write descriptor API for HttpWebServerBasePlugin too
* Surface actual listening port via flags.port
* Go flagless to allow custom user defined flags. Fixes#301
* Add --cache-dir flag for cache plugin (when used with on-disk store)
* Enable discovery of flags from external plugins, example those that reside outside of proxy.py package and loaded on demand. This also allows external flags to surface in --help section
* Define --filtered-client-ips flag for FilterByClientIpPlugin
* Allow plugins to add custom command line flags. Addresses #301
* Reduce dependency over Flags class. This will be deprecated so that adhoc flags can be added without any additional manual configuration
* Fix: Argument 1 to "mock_default_args" of "TestMain" has incompatible type "Namespace"; expected "Mock"
* Reduce Flags class to just the initializer.
* Store list of action dest in FlagParser
* Better document acceptor module and add a TCP Echo Server example
* autopep8 formating
* Rename ThreadlessWork --> Work class
* Make initialize, is_inactive and shutdown as optional interface methods.
Also introduce Readables & Writables custom types.
* Move websocket code into its own module
* Add websocket client example
* Cleanup websocket client
Abhinav Singh