Commit Graph

151 Commits

Author SHA1 Message Date
Abhinav Singh 93f6fd63e4
Add timeout to requests (#1321) 2023-03-14 17:24:34 +05:30
Abhinav Singh bb8f8bee2f
Use only a single acceptor in tests to avoid Windows pitfalls (#1199)
* Use only a single acceptor in tests to avoid Windows pitfalls

* Revert `timeout=1`, likely single acceptor on Windows during tests on GHA is the escape route
2022-06-16 02:42:21 +05:30
Abhinav Singh 315cd17fbd
Add `timeout=1` when joining acceptors (#1196)
* Add a context manager test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* `IS_WINDOWS` must also be True for cygwin environments.  Addresses #1169

* Give an acceptor join timeout of 10 seconds instead of None

* Reduce timeout to just 1 sec

* Install necessary types (requests is translative dep)

* Remove conflicting types

* Give precommit the necessary type

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-06-15 23:48:20 +05:30
Abhinav Singh 69f8b548ee
Allow multiport with ephemeral & unix socket support (#1078)
* Allow multiport with ephemeral & unix socket support

* Fix unix tests
2022-01-31 01:56:34 +05:30
Abhinav Singh bc577f348b
Add `--cache-by-content-type` flag (#1070)
Add `--cache-by-content-type` flag
2022-01-27 05:04:54 +05:30
Abhinav Singh 3fd608eb72
[Jupyter] Request creation notebook (#1065)
* Add plugin tests, responses notebook and enhancements to `build_http_packet`

* Add js code snip for ws example

* Fix tests

* ignore all ipynb from codespell

* ignore all ipynb from codespell

* Fix tests and doc spell
2022-01-26 19:50:12 +05:30
Abhinav Singh fd6d1b36ef
[HttpParser] Memory view compliant, Zero copies (#1067)
* Remove usage of `tobytes`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix chunk parser

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Lint fixes

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-26 18:58:37 +05:30
Abhinav Singh f48aac4523
[Optimize] Avoid using `tobytes` for zero-copies (#1066)
* Avoid using `tobytes` where possible

* `send` accepts `Union[memoryview, bytes]` now

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-26 17:42:32 +05:30
Abhinav Singh 600b3e7c21
[Jupyter] Add a response notebook (#1064)
* Add a response generation jupyter notebook

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Make codespell happy

* precommit codespell exclude

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-26 15:28:27 +05:30
Abhinav Singh 32a0759f96
[Fix] Using `okResponse()` without content hangs the connection (#1062)
* It hangs because of no content-length or connection close header

* Fix tests
2022-01-26 14:40:38 +05:30
Abhinav Singh f084342296
`jupyter` notebook based tutorial (#1059)
* `jupyter` notebook based tutorial

* Move within `tutorial` directory

* Fix spell

* Add `as_non_blocking` option during wrap

* `as_non_blocking=False` by defaut
2022-01-25 19:26:18 +05:30
Abhinav Singh d616fc87db
[CacheResponsesPlugin] Add ability to cache request packets (#1056)
* Start of post encryption tests

* Assertion on post encryption callback

* Add `--cache-requests` flag

* Clean up `on_client_data` API as we no longer have a chain within core http protocol handler

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix flake8 warnings

* Fix `inconsistent-return-statements`

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-24 23:38:57 +05:30
Abhinav Singh ac183f5242
[Core] Invoke selector.close on shutdown (#1055)
[Core] Invoke `selector.close` on shutdown
2022-01-24 19:20:17 +05:30
Abhinav Singh d1920460d0
Pre `v2.4.0rc8` cleanups (#1053)
* Pre-release cleanups

* Add listener pool test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add multi listener test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-24 11:36:51 +05:30
Abhinav Singh 2bebbacbd3
[Work] `kwargs` independent `work_klass` creation and work core (#1051)
* `kwargs` independent work klass and core

* Fix tests

* Add a `create` method to base class

* Lint fixes
2022-01-24 03:16:08 +05:30
Abhinav Singh c3b06f121b
[Work] Refactor as FD work (#1049)
* Abstract out FD based work implementation

* No need of local/remote abstractions

* fix type

* Add `BaseLocalExecutor`

* Fix lint and tests
2022-01-24 01:38:36 +05:30
Abhinav Singh ac840aec78
[Work] Refactor for generic work type (#1048)
* Add `TcpOrTlsSocket` type

* isort

* Update to use Fd executor

* Define `HostPort` type

* Fix fileno

* spellfix
2022-01-23 23:20:04 +05:30
Abhinav Singh d7a568e7fd
[CacheByContentTypePlugin] Prepare for content type parsing (#1038)
* Move default cache directory within `proxy.py` instance data directory

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add `CacheByContentTypePlugin` skeleton and remove unnecessary mixin

* Fix startup issues after removal of mixin

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* .

* Fix broken mock

* doc fix

* doc

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-23 18:05:51 +05:30
Abhinav Singh 7845cbb28d
[SOCKS4] Packet parser & builder (#1047)
* Initial commit

* Parse socks4 packet

* Lint check

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix lint issues

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add handler test skeleton

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-23 16:57:58 +05:30
Abhinav Singh 25e6cb0fe5
[ReverseProxy] Add integration tests (#1037)
* Add reverse proxy integration test

* Fix reverse proxy url for https integration

* Enable reverse proxy for docker build and test
2022-01-22 19:41:02 +05:30
Abhinav Singh 67d9066cfc
[Multiport] `Acceptor` can now accept work from multiple descriptors (#1036)
* Update acceptor to accept work from multiple ports

* import order

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix acceptor tests

* Tweak

* fix tests

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-22 18:54:27 +05:30
Abhinav Singh cc126c3476
[Multiport] Prepare for listening on multiple ports (#1031)
* Convert `--port` to list of integers

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Prepare for multiport listener

* Multi listeners, but will fail in theory as no port override is currently performed

* Separate `listener` module

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update flags in readme

* Fix imports in listener test

* Fix flag parsing for `port` and `ports`

* Fix tests

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-22 15:53:36 +05:30
Abhinav Singh 1569146d70
[ReverseProxy] Move within core lib with ability to write its plugin (#1033)
* Move reverse proxy core within core lib

* Fix main test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix doc and lint

* Fix lint

* Top level category for reverse plugin

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-21 22:29:34 +05:30
Abhinav Singh f5bf5b9714
Move `Tls` module within core (#1030) 2022-01-21 15:21:11 +05:30
Abhinav Singh 627b42f923
Handle `SIGINFO`. Try `kill -s INFO <pid>`. (#1024)
* Handle `SIGINFO`.  Try `kill -s INFO <pid>`.

Also remove dirty hack added in flags to incorporate `--basic-auth`
flag.  Add `__pycache__` to ignore list.  Disable http proxy
during acceptor benchmark.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* not on windows

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* no cover

* # pylint: disable=E1101

* Curl retry on error and check_output 2 minute timeout

* check output timeout None

* Update `faulthandler_timeout` to 2 minutes

* Disable `test_circular_imports`, `isort` integration now works

* Fix curl flags

* Revert back to older flags

* SIGINFO attribute might not even exist

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-20 23:54:44 +05:30
Abhinav Singh 85ad44b46e
[Windows] Remove `--threadless` from `TestCase` (#1025)
[Windows] Remove `--threadless` from `TestCase`
2022-01-20 20:58:42 +05:30
Abhinav Singh 54e74a911f
[isort] Lib modules (#1016)
* isort `proxy.py` main class

* isort init and main

* isort common

* pre-commit fix

* isort dashboard and testing

* isort plugins

* isort core

* Only sort top level http py files

* isort http exception and websocket

* Remove proxy auth plugin from proxy package exports and force discover `PLUGIN_PROXY_AUTH` flags

* isort parser and web server

* no setattr

* isort all

* Enable pre-commit isort hook

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-20 15:34:54 +05:30
Abhinav Singh 7beef281c6
[Coverage] For newly added components (#1014)
* Add newly added code cov

* Fix spelling
2022-01-20 11:45:29 +05:30
Abhinav Singh efd1cac3ea
Restrict request handling to `DEFAULT_ALLOWED_URL_SCHEMES` (#1002)
* Raise `HttpProtocolException` if request line scheme do not match `DEFAULT_ALLOWED_URL_SCHEMES`

* ignore WPS329

* Fix tests

* Pin to 4.3.2

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Test coverage for exception handling

* type ignore

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-19 19:20:19 +05:30
Sowmya Sudha Singh d046cea71c
[Windows] `--threaded` mode integration tests works locally but fails on GHA (#1009)
* Enable remote threadless and threaded integration test for windows

* Run only threaded on windows

* Use powershell for execution of integration script on Windows

* Update test_integration.py

* Update test_integration.py

Co-authored-by: sowmyasudhasingh <sowmyasudhasingh@gmail.com>
Co-authored-by: Abhinav Singh <126065+abhinavsingh@users.noreply.github.com>
2022-01-19 18:51:27 +05:30
Abhinav Singh 21499ad1cb
Bail out early for non-HTTP but HTTP looking protocols (#972)
* Add support in `Url` to parse all types of schemes

* .

* Guard handler against http looking protocol but not web or proxy requests

* Fix condition for web server protocol detection

* doc happy

* Update flags and type check imports only
2022-01-18 00:11:48 +05:30
Abhinav Singh 552fb998df
[Middleware] `HttpClientConnection` preparation (#995)
* Turn usual suspects to warnings, not error

* Add `HttpClientConnection` skeleton

* Fix doc build

* Update references in http tests

* Make `work` core agnostic to work object construction by adding an abstract static method to `Work` interface called `create`

* Make mypy happy

* Fix tests broken due to change in how work objects are now constructed

* Doc ko bhi happy karo
2022-01-16 18:34:33 +05:30
Abhinav Singh c6fceb639a
[Middleware] Capability in the core to allow custom client connection classes (#993)
* Move all TCP server related flags within `tcp_server.py` and also move the encryption functionality within TCP base server

* Templatize `BaseTcpServerHandler` which now expects a client connection object bound to `TcpClientConnection`.  This will allow for custom `HttpClientConnection` object in future to be used by `HttpProtocolHandler`

* Pass necessary flags to allow self-signed certificates

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix https integration tests

* Affected by https://github.com/abhinavsingh/proxy.py/issues/994

* Fix docs

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-16 03:22:55 +05:30
Abhinav Singh d17dc9e448
[SshTunnel] WIP (#992)
[SshTunnel] WIP
2022-01-16 00:22:34 +05:30
Abhinav Singh 0ffa7ca9fb
[TlsInterception] Fix broken `ChunkedResponsePlugin` for `Python < 3.10` (#986)
* Add TLS interception integration tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix CI

* unused

* Well 3.9 just worked locally

* Dispatching empty byte to client results in OSError for Python < 3.10

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Uncomment old integration tests

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-14 14:35:28 +05:30
Abhinav Singh 2714d3d22a
Pass separate `--ca-cert-dir` flag for parallel TLS interception tests (#984)
* Pass separate `--ca-cert-dir` flag for parallel TLS interception tests

* Temp disable `test_modify_post_response_integration`

* mock ca cert dir

* Is threaded an issue with TLS interception?

* Disable modify chunk response for python < 3.10

* Disable modify chunk response for python < 3.10
2022-01-14 12:02:17 +05:30
Abhinav Singh 8a28910224
[TlsInterception] GHA integration tests (#981)
* Add TLS interception integration tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixture to gen certificate once for the `test_integration` module

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use https endpoints in tls interception tests

* Fix modify post data integration test

* Only start 3 acceptor & 3 workers during integration run

* disable chunk response

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-14 00:08:12 +05:30
Abhinav Singh dd2476f02a
Refactor into separate `Work` module (#977)
* work module

* Fix imports

* String based typing for multiprocessing.synchronize

* Fix `test_accepts_client_from_server_socket`

* Move staticmethod outside of threadless pool class

* Fix doc build

* Fix test mock

* mp grouped together

* pylint happy

* import only for type checking

* doc build

* wrong import order
2022-01-13 16:07:51 +05:30
Abhinav Singh 05a8ff9fe7
Test submodule and refactor (#971)
* Refactor tests into submodules

* isort tests

* Add malicious request headers test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-13 00:36:37 +05:30
Abhinav Singh 01bfc7dc30
[HttpProtocolHandler] Handle invalid request parsing exceptions (#957)
* Handle invalid request parsing exception when raised, log the bytes for later inspection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-11 02:27:19 +05:30
Abhinav Singh 32acdcb9fe
Decouple transport framework from dashboard plugin (#953)
* Decouple transport framework from dashboard plugin

* Move `InspectTrafficPlugin` within `http.inspector` module

* Avoid exporting plugins within `__init__.py` files

* Use `/transport/` prefix to avoid #945 conflict issue

* Add todo
2022-01-10 19:48:17 +05:30
Abhinav Singh 35e9c57af7
`isort` everything except lib (for now) (#952)
* isort the tests folder

* Carry over changes from #672

* Disable pre-commit

* Revert flake8 config change

* isort examples too
2022-01-10 13:28:22 +05:30
Abhinav Singh 9b2e7bba7d
Expose `UpstreamConnectionPool` to web & proxy plugins (#946)
* Expose conn pool to plugins

* Fix reusable state handling

* Separate `release` and `retain` methods

* Fix conn pool tests

* Fix tests
2022-01-08 23:18:19 +05:30
Abhinav Singh a2e1fc68a0
Add `--port-file` flag (#942)
* Add `--port-file` flag

* Use `--port-file` flag for integration tests using `get_available_port`

* Use temp dir

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix `base_klass` variable related lint issues

* Fix main tests

* Fix integration

* Use timeout when terminating proc

* Skip integration on win instead of xmark

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-06 02:08:23 +05:30
Abhinav Singh 0d1fe198d3
`DescriptorsHandlerMixin` and `Descriptors`, `SelectableEvents` types (#938)
* Add `Descriptors` type

* Add a `DescriptorsHandlerMixin` class used throughout the http framework

* Remove dependency upon `HasFileno` ie `typing_extension` too

* Define `SelectableEvents` type

* Fix doc

* Blank line

* Remove dep on `typing-extensions`

* Discover base plugin class

* await on now async handlers
2022-01-05 00:32:16 +05:30
Abhinav Singh 7ef2785c31
Use `128 KB` as default value for `DEFAULT_BUFFER_SIZE` (#926)
* Add `TlsInterception` acceptance test

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Generate CA cert if not available

* Fix lint and tests

* Fix args

* Remove acceptance tests for now

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-01-02 22:22:53 +05:30
Abhinav Singh db0c923447
[TlsParser] Refactored implementation from #748 (#922)
* Refactored TlsParser based upon work done in #748

* Add missing `tls_server_hello.data`, thanks to @JerryKwan

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Pass `check.py`

* Run check.py locally

* Fix lint errors

* Fix indentation issue

* Ignore linkcheck for cloudflare links, GHA is getting a 403 reply, while the link actually works

* Fix lint

* codespell skip

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-12-31 22:54:33 +05:30
Abhinav Singh d22d551168
Optimize how `HttpProtocolHandler` delegates to the core plugins (#925)
* Add `protocols` abstract static method to `HttpProtocolHandlerBase` which defines which HTTP specification is followed by the core plugin

* lint

* Fix tests

* Lint fixes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-12-31 15:58:36 +05:30
Abhinav Singh 46c942f947
Hook `UpstreamConnectionPool` lifecycle within `Threadless` (#921)
* Hook connection pool lifecycle within threadless

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix test

* Fix spell

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-12-29 17:37:15 +05:30
Abhinav Singh 263c067301
Define work lifecycle events for pool (#918)
* Define work lifecycle events for pool

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use isinstance

* Use mocker fixture to pass CI on 3.6 and 3.7

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-12-29 00:30:32 +05:30