* Bail out of lock earlier
* Avoid calling `get_events` for unfinished work tasks. Use `cached_property` for `HttpParser` optimization
* Add `type: ignore[no-any-return]`, odd scenario
* We dont have to rebuild response packet repeatedly within `WebServerPlugin`
* Parse line and header in one invocation
* Minor optimizations and update benchmark to use `oha` instead of `hey`
* Remove `flask` from benchmark, only benchmark `asgi` or `async` based libraries. Use `uvicorn` and 10 workers for `blacksheep`. Use `oha` instead of `hey`
* Add benchmark for `starlette`
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add missing dep
* pre-commit
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Optimize `find_http_line` which is in critical path
* Update benchmark results
* Keep the loop hot, TCP no delay, cleanup inactive check periodically
* Check for shutdown signal with tick
* Use non-reentrant `NonBlockingQueue` implementation instead of `queue.Queue`
* Fix listener test
* lint and doc
* Asynchronous `handle_event` and `LocalExecutor` thread
* Bail out on first task completion
* mypy
* Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded)
* Print open file diff from `benchmark.sh`
* Add `--local-executor` flag, disabled by default for now until tests are updated
* Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now)
* Async `get_events`
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address tests after async changes
* mypy and flake8
* spelldoc
* `check.py` and trailing comma
* Rename to `_assertions.py`
* Add missing `pytest-mock` and `pytest-asyncio` deps
* Add `pytest-mock` to `pylint` deps
* Correct use of `parameterize` and add `PT007` to flake8 ignores
* Fix mypy hints broken for `< Python3.9`
* Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix for pre-python-3.9 versions
* `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions
* Install setuptools via `lib-dep` until we recommend editable install
* Deprecate support for `Python 3.6`
* Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929
* Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906
* Make `Threadless` agnostic of `multiprocessing.Process`
* Acceptors must dispatch to local executor in non-blocking fashion
* No daemon for executor processes and fix shutdown logic
* Only return fds from `_selected_events` not all events data
* Refactor logic
* Prefix private methods with `_`
* `work_queue` and not `client_queue`
* Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor`
* Make `LocalExecutor` agnostic of `threading.Thread`
* `LocalExecutor` now implements `Threadless`
* `get_events` and `get_descriptors` now must return int and not sock. `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify`
* Fix `main` tests
* Apply suggestions from code review
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
* Apply code review recommendations manually
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert back `Any` and use `addr or None`
* Address `flake8`
* Update tests to use `fileno`
* Fix doc build
* Fix doc spell, use tear down and not teardown
* Doc updates
* Add back support for `Python 3.6`
* Acceptors dont need loop initialization
* On Python 3.6 `asyncio.new_event_loop()` is necessary
* Make doc happy
* `--threaded` needs a new event loop for 3.7 too
* Always use `asyncio.new_event_loop()` for threaded mode
Added e2e integration tests (subprocess & curl) for all modes.
* Lint fixes
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
* Introduce `--haproxy-protocol` flag
* Complete proxy protocol v1 implementation, enable using `--enable-proxy-protocol` flag
* link checks
* Advertise support for haproxy protocol in readme
* Add make target `lib-scm-version`
* `make lib-version` is now `make lib-check`
* Dont enforce -dev part of version within README
* Add provision to update readme flags using check
* Wrap help text within console
* Add closing ticks
* Remove verbose logging and update homebrew formulae (may be fixed?)
* Return DEFAULT_404_RESPONSE by default from static server when file doesnt exist
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix web server with proxy test
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Custom Url parser for our needs
* lint fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix proxy_pool plugin as scheme can be None if not present in the Url
* Address the ambiguous ipv6:port scenario along with valid cases
* lint checks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* docstring
* Abstract into `http.parser` module
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix#398 HTTP/1.0 related issue
* lint checks
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* deprecate server_file_or_404
* Optionally compress static content. Currently only if content length higher than 300
* trailing comma
* Allow `--plugins` flag to be used multiple times
Following are valid invocation:
1) `--plugins A`
2) `--plugins A,B`
3) `--plugins A --plugins B`
4) `--plugins A,B --plugins C`
* mypy
* Flake8
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* correct type
* Add `HttpParser.is_https_tunnel()` utility method
* mypy
* lint checks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Allow overriding work_klass via Proxy context manager kwargs
* Decouple acceptor and executor pools
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add `--num_acceptors` flag and better load balancing
* Remove unused
* Lint errors
* Another arg not kwarg
* Move start work staticmethods within ExecutorPool
* mypy fixes
* Update README with `--num-acceptors` flag
* Rename `Proxy.pool` to `Proxy.acceptors`
* Add SetupShutdownContextManager abstraction
* Match --num-acceptors logic with PR description
* Rename executor utility methods and add docstring
* Remove work_klass from constructors and pass it via flags
* Update docstring for pools as they no longer accept a work_klass argument
* Turn work_klass into a flag. main() no longer accepts input_args (only kwargs opts). Similarly, Proxy doesnt accept any input_args now (only kwargs opts)
* Expose default work klass in README
* Expose `HttpProtocolHandler` and `HttpProtocolHandlerPlugin` within `proxy.http` module
* Start to fix tests
* Fix tests
* mypy and flake8
* Trailing comma
* Remove unused var
* Unused arg
* uff
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Explicit `multiprocessing.Manager.shutdown`
Multiprocessing manager is used within eventing core. From doc,
it appears to start a BaseManager which starts a server????
Seriously???? Anyways, using multiprocessing manager is a PITA
and mistake, as it doesn't even give us performance we expect.
Our proxy server can handle more requests than what multiprocess
manager can exchange between processes.
* `--threadless is now ON by default for `Python 3.8+` on `mac` and `linux` environments
* Clarity around why multiprocessing.Manager must be deprecated
* Add `--threaded` flag which can be used to fallback for environments where `--threadless` is now default
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* never used
* Update README
* Use `threaded=True` in tests which were written for threaded model
* Fix issue where sharing manager between global event queue and subscriber can lead to TypeError
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Expose within __all__
* Enable `--numprocesses=auto` for `pytest.ini`
* make lib-lint
* Also consider `--plugins` flag when bootstrapping plugins
* Add `from .dashboard import ProxyDashboard` in top-level `__init__.py` to make `ProxyDashboard` flags auto discoverable
* Move `--enable-dashboard` to top-level
* Move logging utility within `Logger` class
* Consider comma separated --plugin and --plugins during discover_plugins
* Refactor plugin related utilities in Plugins module
* mypy and lint
* Fix unused import
* Safe to use tempdir on Github actions to avoid race conditions???
* pki (generically disk based file) based tests are flaky on macOS under parallel execution
* Add FAQ: OSError when wrapping client for TLS Interception
* Silence exception log for several valid "cert verification failed" by client during tls interception
* Lint checks
* Move exception handling within wrap_server/wrap_client methods
* Lint fixes
* Use certifi/cacert.pem as default --ca-file flag value
* Address tests after DEFAULT_CA_FILE change
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Use suggested fix in #683 to remove pytest ignore
* Handle should not flush now which can block, instead let base tcp handler do its magic
* test speed up, doc string, logging enhancements
* Move macOS to the end of workflows
* Fix mypy warnings
* Rename .server to .upstream
* Lint fixes
* Mark internal methods with _ prefix
* Fix broken test
* lint changes
* Wah, double client :D
* Avoid selector initialization for threadless mode
* remove unused imports
* Now HttpProtocolHandler implements BaseTcpServerHandler
* Consistent return and guard againt upstream.closed
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* AcceptorPool as context manager
* Group multiprocessing imports together
* Use com.jaxl bundle identifier as proxy.py will eventually move under jaxl org
* revisit devtools integration :)
* Emit all necessary events for devtools integration
* Lint fixes
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Make reverse proxy plugin use proxy.py core loop for async io operations
* Address lint errors
* Deprecate on_websocket_close and replace with on_client_connection_close
* Lint fixes
* Retry on SSLWantReadError and SSLWantWriteError
* Async proxy pool
* Async proxy pool
* Late upstream initialization and exception guards
* Close upstream proxy connection on client connection close
* Refactor into EventManager
* Fix tests accounting in the event manager
* Ensure each process initializes logger
* pragma no cover
* Teardown connection when proxy pool upstream proxy closes
* Add ability to customize access log format and add additional context to it
* Maintain total size for response bytes in access logs
* Fix tests broken due to new plugin methods missing mock
* Update pubsub_eventing to use EventManager to avoid entire bootstrapping step
* Allow plugins to add custom command line flags. Addresses #301
* Reduce dependency over Flags class. This will be deprecated so that adhoc flags can be added without any additional manual configuration
* Fix: Argument 1 to "mock_default_args" of "TestMain" has incompatible type "Namespace"; expected "Mock"
* Reduce Flags class to just the initializer.
* Store list of action dest in FlagParser
* Add DEFAULT_HTTP_PORT constant
* Use DEFAULT_HTTP_PORT in tests
* Refactor into exception module
* Refactor into inspector module
* Refactor into server module
* Refactor into proxy module
* Update doc and banner
* Update banner to match GitHub
* Update older banners too
* Add update_desc to .gitignore
* Update banner for dashboard to match github
* also update html, js, css
* Allow resources to load from http and ws when running w/o https
* Move dashboard backend (dashboard.py) within proxy module. Now shipped with pip install proxy.py
* Update ref to dashboard backend in github workflows
* Add git-pre-commit hook file.
Enable it by symlinking as .git/hooks/pre-commit
* Also enable static server for dashboard serving
* Dynamically load devtools instead of on page load
* Add support for passing flags as kwargs to main / start methods.
* Fix tests for refactored code
* Allow proxy.main, proxy.start, proxy.TestCase.
Also update README.md to reflect the same.
* Use Any for **opts
* Move main as __init__ to avoid name conflicts
* Fix tests
* Update setup.py entry_point
* Explicitly install requirements before setup.py
* Explicitly mention packages of interest
* ipv6 fails on ubuntu, use ipv4
* Make typing-extensions optional
* Instead of putting it all under __init__.py, move main.py to proxy.py
* Simply make setup.py module free
* autopep8
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
Abhinav Singh