Commit Graph

12 Commits

Author SHA1 Message Date
Abhinav Singh a1bb659488
Threadless execution using coroutines (#134)
* Workers need not register/unregister sock for every loop

* No need of explicit socket.settimeout(0) which is same as socket.setblocking(False)

* Remove settimeout assertion

* Only store sender side of Pipe().  Also ensure both end of the Pipe() are closed on shutdown

* Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes

* Use time.time throughout.  Remove incomplete test_cache_responses_plugin to avoid resource leak in tests

* Remove unused

* Wrap selector register/unregister within a context manager

* Refactor in preparation of threadless request handling

* MyPy generator fix

* Add --threadless flag

* Internally call them acceptors

* Internally use acceptors

* Add Threadless class.

Also no need to pass family over pipe to acceptors.

* Make threadless work for a single client :)

* Threadless is soon be our default

* Close client queue

* Use context manager for register/unregister

* Fix Acceptor tests broken after refactoring

* Use asyncio tasks to invoke ProtocolHandle.handle_events

This gives all client threads a chance to respond without
waiting for other handlers to return.

* Explicitly initialize event loop per Threadless process

* Mypy fixes

* Add ThreadlessWork abstract class implemented by ProtocolHandler

* Add benchmark.py

Avoid TIME_WAIT by properly shutting down the connection.

* Add benchmark.py as part of testing workflow

* When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state

* MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence

* Unwrap if wrapped before shutdown

* Unwrap if wrapped before shutdown

* socket.SHUT_RDWR will cause leaks

* MyPy

* Add instructions for monitor.sh

* Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized

* Add Fast & Scalable section

* Update internal classes section

* Dont print out local dir path in help text :)

* Refactor

* Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang

* Add chrome_with_proxy.sh helper script

* Handle OSError during client.flush which can happen due to invalid protocol type for socket error

* Remove redundant e

* Add classmethods to quickly construct a parser object

* Don't raise from TcpConnection abstract class.

This allows both client/socket side of communication to handle
exceptions as necessary. We might refactor this again later to remove
redundant code :)

* Disable response parsing when TLS interception is enabled. See issue #127

* remove unused imports

* Within webserver parse pipelined requests only if we have a route

* Add ShortLinkPlugin plugin

* Add more shortlinks

* Add ShortLinkPlugin to README.md

* Add path forwarding too instead of leaving as excercise ;)

* Add shortlink to TOC

* Ensure no socket leaks

* Ensure no leaks

* Naming

* Default number of clients 1

* Avoid shortlinking localhost

* Stress more
2019-10-15 23:56:39 -07:00
Abhinav Singh 6455a34973
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128)
* Add tests for is_http_1_1_keep_alive

* Add ModifyPostDataPlugin in README

* Fixes #126

* Refactor HttpProxyBasePlugin API

* before_upstream_connection too can drop request by returning None

* Remove HTTP Server startup during tests, no longer used

* Removed unused imports

* Simplify load_plugins
2019-10-12 17:43:19 -07:00
Abhinav Singh a19da5dcb9
Update issue templates (#123) 2019-10-10 16:18:12 -07:00
Abhinav Singh 3b9587b6f1
Ensure explicit flush before upgrade while TLS intercepting (#121)
* Ensure explicit flush before upgrade + nonblocking server sock.

* tls interception tests

* prepare for v1.1.1

* Add requirements.txt for typing extensions

* Add setup.py under lint workflow
2019-10-10 15:15:27 -07:00
Abhinav Singh a9c8659ffa
Use selectors.DefaultSelector instead of select.select (#106)
* Use selectors.DefaultSelector instead of select.select

* Unregister to avoid endless loop

* Cleanup event register/unregiter

* Cleanup event registration

* Add google-fluentd.conf.  Use if running proxy.py on Google Cloud.

* Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?)

* Fix tests for selectors

* Only include proxy.py and tests.py in coverage report

* Only include proxy.py for coverage

* remove redundant integration test, will rewrite using mocks

* Proper unregister of events

* Change multi core accept model to avoid client TIME_WAIT. Fixes #97

* Catch BlockingIOError

* Remove redundant comments

* Simplify with AcceptorPool

* Pass family to acceptor processes

* Remove plugin.access_log for core plugins

* Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server

* Add support for websocket upgrade

* Websocket frame parser

* Enable websocket based routing

* Add WebsocketClient

* Websocket

* mypy fixes

* Sync GitHub workflow lint and makefile lint commands.

For now comment out tests which are broken :(  New tests coming next.

* Start fixing tests for new code

* Fix formatting

* Fix main tests

* Add worker tests

* GitHub only ran windows tests, may be require unique names

* Use 3.6/3.7 dev versions for GitHub actions

* Add AcceptorPool test

* Add x64 and x86 matrix for actions tests

* Dont use dev versions since they dont exists for x86

* Ha no x86 support itself

* Add backer link

* Remove support badge for 3.5 as it doesnt support typing

* Update read me with changed architecture notes

* Update read me with changed architecture notes

* Add `import proxy` usage instructions.

* Add pydoc reference for developers

* Put pydoc as internal documentation
2019-10-02 00:09:35 -07:00
Abhinav Singh 3a9c8bc8a2
Create FUNDING.yml 2019-09-27 09:56:31 -07:00
Abhinav Singh ccb1dbeb5f
Pluggable WebServer (#98)
* Enable windows and pre/dev python releases

* Move twine to requirements-release.txt

* 3.5 do not support type annotations

* Use inbuild build_request, build_response methods now.

* First cut of web server route plugin architecture

* Add route decorator

* mypy

* disable TestMultiCoreRequestDispatcherIntegration for github

* Anyways it doesnt format as advertised
2019-09-26 02:07:16 -07:00
Abhinav Singh 292ae497fc
Test coverage and mypy compliance (#94)
* Add tests and mypy compliance for tests

* Add pragma: no cover for abstract class methods

* mypy --strict compliance for tests.py

* Refactor and test coverage

* lint fixes
2019-09-25 15:51:12 -07:00
Abhinav Singh 92142a70d9
Add support for TLS/SSL interception and HTTPS server (#86)
* Handle OSError thrown when curl --proxy-cacert flag is invalid

* Add server-cert

* Dockerfile dont need --ipv4 since now its auto-detected and deprecated

* Add make server-cert info

* Add support for HTTPS interception.

1) Start as

    proxy.py --ca-key-file ca-key.pem --ca-cert-file ca-cert.pem --ca-signing-key-file ca-signing-key.pem

2) Test using curl

    curl -v -L -x proxy.py:8899 --cacert ca-cert.pem https://google.com

Now proxy.py is able to decrypt HTTPS traffic between curl and
google.com.  Generated certificates are stored under ~/.proxy.py
directory.

* Add comments for why interception do not work if client is communicating with proxy.py over https

* Fix tests

* Exit with message when trying to https intercept and also serve over https

* lint

* Disable OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_TLSv1, OP_NO_TLSv1_1 when wrapping upstream server socket

* Ensure client.conn.shutdown is called if necessary.

* Add mypy check for plugin_examples.py

* Exception handling when client sends invalid requests

* Verify plugin_examples

* mypy compat

* HttpRequestRejected
2019-09-24 19:02:57 -07:00
Abhinav Singh 397f92e561
Can serve proxy traffic over HTTPS (#85)
* Can serve proxy traffic over HTTPS.

Pass --certfile and --keyfile arguments to enable SSL/TLS layer.
Example:

1) Start proxy.py as

./proxy.py --certfile cert.pem --keyfile key.pem

2) Then verify via curl

curl -v -x https://localhost:8899 --proxy-cacert cert.pem http://httpbin.org/get

* Add --ignore-missing-imports for missing multiprocessing.reduction module

* mock socket.fromfd to avoid sys calls

* Add HTTPS info

* Add TLS encryption

* Add TLS encryption
2019-09-23 17:38:10 -07:00
Abhinav Singh ae1f1d654c
mypy coverage (#78)
* mypy coverage

* First stab at mypy --strict complaince

* mypy --strict compliance.

* Run mypy during Quality Check phase
2019-09-22 21:11:55 -07:00
Abhinav Singh 5eb4d8a97c
Use Pipe instead of Queue when passing sockets (#69)
* Use pipes instead of queues.

Possible solution for #68

* Handle ConnectionRefusedError

* Close corresponding pipes on shutdown

* Abstract classes for plugins.

* Add github workflows

* Use pytest for github workflow

* Add Windows/Mac workflows

* Use os.matrix

* 3.8 and 3.9 probably dont exists yet

* Do not fail-fast, also python 3.5 seems to be throwing syntax error for typing

* autopep8

* Disable windows-latest workflow which seems to hang
2019-09-16 20:18:14 -07:00