Commit Graph

5 Commits

Author SHA1 Message Date
Abhinav Singh 0d1fe198d3
`DescriptorsHandlerMixin` and `Descriptors`, `SelectableEvents` types (#938)
* Add `Descriptors` type

* Add a `DescriptorsHandlerMixin` class used throughout the http framework

* Remove dependency upon `HasFileno` ie `typing_extension` too

* Define `SelectableEvents` type

* Fix doc

* Blank line

* Remove dep on `typing-extensions`

* Discover base plugin class

* await on now async handlers
2022-01-05 00:32:16 +05:30
Abhinav Singh 44d72431e3
Async `get_events`, `handle_event`, `handle_readables`, `handle_writables` (#769)
* Asynchronous `handle_event` and `LocalExecutor` thread

* Bail out on first task completion

* mypy

* Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded)

* Print open file diff from `benchmark.sh`

* Add `--local-executor` flag, disabled by default for now until tests are updated

* Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now)

* Async `get_events`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address tests after async changes

* mypy and flake8

* spelldoc

* `check.py` and trailing comma

* Rename to `_assertions.py`

* Add missing `pytest-mock` and `pytest-asyncio` deps

* Add `pytest-mock` to `pylint` deps

* Correct use of `parameterize` and add `PT007` to flake8 ignores

* Fix mypy hints broken for `< Python3.9`

* Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix for pre-python-3.9 versions

* `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions

* Install setuptools via `lib-dep` until we recommend editable install

* Deprecate support for `Python 3.6`

* Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929

* Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906

* Make `Threadless` agnostic of `multiprocessing.Process`

* Acceptors must dispatch to local executor in non-blocking fashion

* No daemon for executor processes and fix shutdown logic

* Only return fds from `_selected_events` not all events data

* Refactor logic

* Prefix private methods with `_`

* `work_queue` and not `client_queue`

* Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor`

* Make `LocalExecutor` agnostic of `threading.Thread`

* `LocalExecutor` now implements `Threadless`

* `get_events` and `get_descriptors` now must return int and not sock.  `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify`

* Fix `main` tests

* Apply suggestions from code review

Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>

* Apply code review recommendations manually

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert back `Any` and use `addr or None`

* Address `flake8`

* Update tests to use `fileno`

* Fix doc build

* Fix doc spell, use tear down and not teardown

* Doc updates

* Add back support for `Python 3.6`

* Acceptors dont need loop initialization

* On Python 3.6 `asyncio.new_event_loop()` is necessary

* Make doc happy

* `--threaded` needs a new event loop for 3.7 too

* Always use `asyncio.new_event_loop()` for threaded mode

Added e2e integration tests (subprocess & curl) for all modes.

* Lint fixes

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
2021-11-23 15:02:00 +05:30
Abhinav Singh 3aa1dc2824
Test refactor + Docker image CI (#154)
* Move tests into individual modules too

* Ensure one test class per file

* Fix docker image after refactoring

* Add github actions workflow for building docker image

* Fix image name

* Setup python required for extracting proxy version

* Version will also require deps
2019-10-29 20:41:39 -07:00
Abhinav Singh 0e2194d683
Update README.md (#152)
* Update flags

* Update debugging instructions and run instructions for develops

* Update references to plugins directory

* For readability add sections for run from command line using pip

* Move internal doc under developer section

* Add option to pass fully-qualified plugin path
2019-10-28 16:14:34 -07:00
Abhinav Singh e14548252c
Proxy.py Dashboard (#141)
* Remove redundant variables

* Initialize frontend dashboard app (written in typescript)

* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection

* Add read_and_build_static_file_response method for reusability in plugins

* teardown websocket connection when opcode CONNECTION_CLOSE is received

* First draft of proxy.py dashboard

* Remove uglify, obfuscator is superb enough

* Correct generic V

* First draft of dashboard

* ProtocolConfig is now Flags

* First big refactor toward no-single-file-module

* Working tests

* Update dashboard for refactored imports

* Remove proxy.py as now we can just call python -m proxy -h

* Fix setup.py for refactored code

* Banner update

* Lint check

* Fix dashboard static serving and no UNDER_TEST constant necessary

* Add support for plugin imports when specified in path/to/module.MyPlugin

* Update README with instructions to run proxy.py after refactor

* Move dashboard under /dashboard path

* Rename to devtools.ts

* remove unused

* Update github workflow for new directory structure

* Update test command too

* Fix coverage generation

* *.py is an invalid syntax on windows

* No * on windows

* Enable execution via github zip downloads

* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory

* Forget zip runs for now

* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load

* Enforce eslint with standard style

* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)

* Remove extra empty line

* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected

* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected

* Fix tests

* Move common code under common sub-module

* Move flags under common module

* Move acceptor under core

* Move connection under core submodule

* Move chunk_parser under http

* Move http_parser as http/parser

* Move http_methods as http/methods

* Move http_proxy as http/proxy

* Move web_server as http/server

* Move status_codes as http/codes

* move websocket as http/websocket

* Move exception under http/exception, also move http/proxy exceptions under http/exceptions

* move protocol_handler as http/handler

* move devtools as http/devtools

* Move version under common/version

* Lifecycle if now core Event

* autopep8

* Add core event queue

* Register / unregister handler

* Enable inspection support for frontend dashboard

* Dont give an illusion of exception for HttpProtocolExceptions

* Update readme for refactored codebase

* DictQueueType everywhere

* Move all websocket API related code under WebsocketApi class

* Inspection enabled on tab switch.

1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.

* Corresponding ends of the work queues can be closed immediately.

Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.

* No need of a manager for shared multiprocess Lock.

This unnecessarily creates additional manager process.

* Move threadless into its own module

* Merge acceptor and acceptor_pool tests

* Defer os.close

* Change content display with tab clicks.

Also ensure relay manager shutdown.

* Remove --cov flags

* Use right type for SyncManager

* Ensure coverage again

* Print help to discover flags, --cov certainly not available on Travis for some reason

* Add pytest-cov to requirements-testing

* Re-add windows on .travis also add changelog to readme

* Use 3.7 and no pip upgrade since it fails on travis windows

* Attempt to fix pip install on windows

* Disable windows on travis, it fails and uses 3.8.  Try reporting coverage from github actions

* Move away from coveralls, use codecov

* Codecov app installation either didnt work or token still needs to be passed

* Remove travis CI

* Use https://github.com/codecov/codecov-action for coverage uploads

* Remove run codecov

* Ha, codecov action only works on linux, what a mess

* Add cookie.js though unable to use it with es5/es6 modules yet

* Enable testing for python 3.8 also Build dashboard during testing

* No python 3.8 on github actions yet

* Autopep8

* Add separate workflows for library (python) and dashboard (node) app

* Type jobs not job

* Add checkout

* Fix parsing node version

* Fix dashboard build on windows

* Show codecov instead of coveralls
2019-10-28 14:57:33 -07:00