* Asynchronous `handle_event` and `LocalExecutor` thread
* Bail out on first task completion
* mypy
* Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded)
* Print open file diff from `benchmark.sh`
* Add `--local-executor` flag, disabled by default for now until tests are updated
* Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now)
* Async `get_events`
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address tests after async changes
* mypy and flake8
* spelldoc
* `check.py` and trailing comma
* Rename to `_assertions.py`
* Add missing `pytest-mock` and `pytest-asyncio` deps
* Add `pytest-mock` to `pylint` deps
* Correct use of `parameterize` and add `PT007` to flake8 ignores
* Fix mypy hints broken for `< Python3.9`
* Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix for pre-python-3.9 versions
* `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions
* Install setuptools via `lib-dep` until we recommend editable install
* Deprecate support for `Python 3.6`
* Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929
* Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906
* Make `Threadless` agnostic of `multiprocessing.Process`
* Acceptors must dispatch to local executor in non-blocking fashion
* No daemon for executor processes and fix shutdown logic
* Only return fds from `_selected_events` not all events data
* Refactor logic
* Prefix private methods with `_`
* `work_queue` and not `client_queue`
* Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor`
* Make `LocalExecutor` agnostic of `threading.Thread`
* `LocalExecutor` now implements `Threadless`
* `get_events` and `get_descriptors` now must return int and not sock. `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify`
* Fix `main` tests
* Apply suggestions from code review
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
* Apply code review recommendations manually
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert back `Any` and use `addr or None`
* Address `flake8`
* Update tests to use `fileno`
* Fix doc build
* Fix doc spell, use tear down and not teardown
* Doc updates
* Add back support for `Python 3.6`
* Acceptors dont need loop initialization
* On Python 3.6 `asyncio.new_event_loop()` is necessary
* Make doc happy
* `--threaded` needs a new event loop for 3.7 too
* Always use `asyncio.new_event_loop()` for threaded mode
Added e2e integration tests (subprocess & curl) for all modes.
* Lint fixes
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
* Update `make lib-profile`
* Optimize `utils.find_http_line`
* Pass work to executors within their own multiprocessing lock
* Fix tests
* Add `(_py_class_role, Url)` to fix rtfd :D
* Allow override of `--auth-plugin`.
This can be used in scenarios where a hardcoded `--basic-auth`
credentials cannot be used across all your users. May-be you
want to connect to a database before authorizing the request.
For all such scenario, disable in-build default auth plugin
by providing `--auth-plugin` flag.
* Allow usage of `--auth-plugin` flag without having to provide `--basic-auth` flag. When `--basic-auth` flag is used, default auth plugin will load. When `--basic-auth` and `--auth-plugin` flag is used, custom auth plugin will load
* Address test_main broken after auth_plugin flag
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update README
* Fix long line
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Abstract out a Listener class
* unused
* Use connection instead of manager queue
* For web close connection of client requested via headers
* Remove eventing WIP module
* Sub and Unsub ack
* Fix tests
* mypy and flake8
* comma
* Move callback within EventSubscriber constructor
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip test_unix_path_listener on Windows
* Spelling fix
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Allow overriding work_klass via Proxy context manager kwargs
* Decouple acceptor and executor pools
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add `--num_acceptors` flag and better load balancing
* Remove unused
* Lint errors
* Another arg not kwarg
* Move start work staticmethods within ExecutorPool
* mypy fixes
* Update README with `--num-acceptors` flag
* Rename `Proxy.pool` to `Proxy.acceptors`
* Add SetupShutdownContextManager abstraction
* Match --num-acceptors logic with PR description
* Rename executor utility methods and add docstring
* Remove work_klass from constructors and pass it via flags
* Update docstring for pools as they no longer accept a work_klass argument
* Turn work_klass into a flag. main() no longer accepts input_args (only kwargs opts). Similarly, Proxy doesnt accept any input_args now (only kwargs opts)
* Expose default work klass in README
* Expose `HttpProtocolHandler` and `HttpProtocolHandlerPlugin` within `proxy.http` module
* Start to fix tests
* Fix tests
* mypy and flake8
* Trailing comma
* Remove unused var
* Unused arg
* uff
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Explicit `multiprocessing.Manager.shutdown`
Multiprocessing manager is used within eventing core. From doc,
it appears to start a BaseManager which starts a server????
Seriously???? Anyways, using multiprocessing manager is a PITA
and mistake, as it doesn't even give us performance we expect.
Our proxy server can handle more requests than what multiprocess
manager can exchange between processes.
* `--threadless is now ON by default for `Python 3.8+` on `mac` and `linux` environments
* Clarity around why multiprocessing.Manager must be deprecated
* Add `--threaded` flag which can be used to fallback for environments where `--threadless` is now default
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* never used
* Update README
* Use `threaded=True` in tests which were written for threaded model
* Fix issue where sharing manager between global event queue and subscriber can lead to TypeError
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Expose within __all__
* Enable `--numprocesses=auto` for `pytest.ini`
* make lib-lint
* Also consider `--plugins` flag when bootstrapping plugins
* Add `from .dashboard import ProxyDashboard` in top-level `__init__.py` to make `ProxyDashboard` flags auto discoverable
* Move `--enable-dashboard` to top-level
* Move logging utility within `Logger` class
* Consider comma separated --plugin and --plugins during discover_plugins
* Refactor plugin related utilities in Plugins module
* mypy and lint
* Fix unused import
* Safe to use tempdir on Github actions to avoid race conditions???
* pki (generically disk based file) based tests are flaky on macOS under parallel execution
* Add a `--unix-socket-path` flag.
When available `--hostname` and `--port` flags are ignored.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* `print` statement is allowed only in `flags.py` and `version-check.py`. All other places must use a `logger` instance
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add guard for `AF_UNIX` on Windows
* Comment out assertion on Windows for now
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Move Proxy.initialize within FlagParser.initialize. Also move other staticmethods from within proxy class into utils
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove unused imports
* Fix `import-outside-toplevel` error
* add `make lib-flake8` and `make lib-mypy` targets
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Rename is_py3 to is_py2 for more logical guard
* Add stubs for missing tests, add few more tests for core modules
* Lint fixes
* Line too long fix
* Remove unnecessary KeyboardInterrupt
* Consistent workflow names
* Update homebrew formulae. Doesnt seems to work now
* test_enable_dashboard and test_enable_events
* test_enable_dashboard and test_enable_events
* Fix problem where empty plugin string was passed as plugin module
* test_enable_devtools and remove redundant guards for None and "" which was there due to a bug
* Async proxy pool
* Async proxy pool
* Late upstream initialization and exception guards
* Close upstream proxy connection on client connection close
* Refactor into EventManager
* Fix tests accounting in the event manager
* Ensure each process initializes logger
* pragma no cover
* Teardown connection when proxy pool upstream proxy closes
* Add ability to customize access log format and add additional context to it
* Maintain total size for response bytes in access logs
* Fix tests broken due to new plugin methods missing mock
* Update pubsub_eventing to use EventManager to avoid entire bootstrapping step
* Go flagless to allow custom user defined flags. Fixes#301
* Add --cache-dir flag for cache plugin (when used with on-disk store)
* Enable discovery of flags from external plugins, example those that reside outside of proxy.py package and loaded on demand. This also allows external flags to surface in --help section
* Define --filtered-client-ips flag for FilterByClientIpPlugin
* Allow plugins to add custom command line flags. Addresses #301
* Reduce dependency over Flags class. This will be deprecated so that adhoc flags can be added without any additional manual configuration
* Fix: Argument 1 to "mock_default_args" of "TestMain" has incompatible type "Namespace"; expected "Mock"
* Reduce Flags class to just the initializer.
* Store list of action dest in FlagParser
* Initialize MacOS Menubar application
* Dashboard plugin at-least needs a shutdown hook to teardown any thread/processes started by dashboard backend plugin
* Add menu bar icon
* Add respective test directories
* Sync test banners
* Move plugin tests under its own package
* Enable daemon for threads, other this wont shutdown cleanly
* Dynamically load devtools instead of on page load
* Add support for passing flags as kwargs to main / start methods.
* Fix tests for refactored code
* Allow proxy.main, proxy.start, proxy.TestCase.
Also update README.md to reflect the same.
* Use Any for **opts
* Move main as __init__ to avoid name conflicts
* Fix tests
* Update setup.py entry_point
* Explicitly install requirements before setup.py
* Explicitly mention packages of interest
* ipv6 fails on ubuntu, use ipv4
* Make typing-extensions optional
* Instead of putting it all under __init__.py, move main.py to proxy.py
* Simply make setup.py module free
* autopep8
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
Abhinav Singh