2019-10-02 07:09:35 +00:00
[](https://github.com/abhinavsingh/proxy.py)
2013-08-31 09:19:22 +00:00
2019-09-21 22:04:44 +00:00
[](https://opensource.org/licenses/BSD-3-Clause)
2019-09-23 07:30:52 +00:00
[](https://pypi.org/project/proxy.py/)
[](https://hub.docker.com/r/abhinavsingh/proxy.py)
2019-09-17 03:18:14 +00:00
[](https://travis-ci.org/abhinavsingh/proxy.py/)
2019-09-23 07:30:52 +00:00
[](https://github.com/abhinavsingh/proxy.py)
2019-09-17 03:18:14 +00:00
[](https://coveralls.io/github/abhinavsingh/proxy.py?branch=develop)
2019-10-02 07:09:35 +00:00
[](https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/iOS_Simulator_Guide/Introduction/Introduction.html)
[](https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/iOS_Simulator_Guide/Introduction/Introduction.html)
[](https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/iOS_Simulator_Guide/Introduction/Introduction.html)
2019-09-23 07:30:52 +00:00
[](https://www.android.com/)
[](https://developer.android.com/studio/run/emulator-networking.html#proxy)
[](https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/iOS_Simulator_Guide/Introduction/Introduction.html)
[](https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/iOS_Simulator_Guide/Introduction/Introduction.html)
[](https://gitHub.com/abhinavsingh/proxy.py/graphs/commit-activity)
[](https://twitter.com/imoracle)
[](https://github.com/abhinavsingh/proxy.py/issues)
2019-10-02 07:09:35 +00:00
[](https://gitter.im/proxy-py/community)
2019-09-23 07:30:52 +00:00
2019-09-21 22:04:44 +00:00
[](https://www.python.org/downloads/release/python-360/)
[](https://www.python.org/downloads/release/python-370/)
2019-09-23 07:30:52 +00:00
[](http://mypy-lang.org/)
2019-02-09 04:25:09 +00:00
2019-10-02 07:09:35 +00:00
[](https://opencollective.com/proxypy)
2019-09-25 03:44:30 +00:00
Table of Contents
=================
* [Features ](#features )
* [Install ](#install )
* [Stable version ](#stable-version )
* [Development version ](#development-version )
2019-09-25 17:09:40 +00:00
* [Start proxy.py ](#start-proxypy )
* [Command Line ](#command-line )
* [Docker Image ](#docker-image )
2019-09-25 03:44:30 +00:00
* [Plugin Examples ](#plugin-examples )
2019-09-27 20:00:11 +00:00
* [ProposedRestApiPlugin ](#proposedrestapiplugin )
2019-09-25 03:44:30 +00:00
* [RedirectToCustomServerPlugin ](#redirecttocustomserverplugin )
* [FilterByUpstreamHostPlugin ](#filterbyupstreamhostplugin )
* [CacheResponsesPlugin ](#cacheresponsesplugin )
* [ManInTheMiddlePlugin ](#maninthemiddleplugin )
* [Plugin Ordering ](#plugin-ordering )
* [End-to-End Encryption ](#end-to-end-encryption )
2019-09-27 16:18:58 +00:00
* [TLS Interception ](#tls-interception )
2019-10-02 07:09:35 +00:00
* [import proxy.py ](#import-proxypy )
2019-09-25 05:43:29 +00:00
* [Plugin Developer and Contributor Guide ](#plugin-developer-and-contributor-guide )
* [Everything is a plugin ](#everything-is-a-plugin )
2019-10-02 07:09:35 +00:00
* [Internal Architecture ](#internal-architecture )
* [Internal Documentation ](#internal-documentation )
* [Sending a Pull Request ](#sending-a-pull-request )
* [Frequently Asked Questions ](#frequently-asked-questions )
* [Unable to connect with proxy.py from remote host ](#unable-to-connect-with-proxypy-from-remote-host )
* [Basic auth not working with a browser ](#basic-auth-not-working-with-a-browser )
* [Docker image not working on MacOS ](#docker-image-not-working-on-macos )
2019-09-25 05:43:29 +00:00
* [Flags ](#flags )
2019-09-25 03:44:30 +00:00
2013-08-31 09:19:22 +00:00
Features
2019-09-25 03:44:30 +00:00
========
2013-08-31 09:19:22 +00:00
2019-09-21 22:04:44 +00:00
- Lightweight
- Distributed as a single file module `~50KB`
2019-09-22 02:20:27 +00:00
- Uses only `~5-20MB` RAM
2019-09-21 22:04:44 +00:00
- No external dependency other than standard Python library
- Programmable
- Optionally enable builtin Web Server
- Customize proxy and http routing via [plugins ](https://github.com/abhinavsingh/proxy.py/blob/develop/plugin_examples.py )
2019-09-25 02:02:57 +00:00
- Enable plugin using command line option e.g. `--plugins plugin_examples.CacheResponsesPlugin`
2019-09-21 22:04:44 +00:00
- Plugin API is currently in development state, expect breaking changes.
2019-09-24 00:38:10 +00:00
- Secure
2019-09-25 02:02:57 +00:00
- Enable end-to-end encryption between clients and `proxy.py` using TLS
2019-09-25 03:44:30 +00:00
- See [End-to-End Encryption ](#end-to-end-encryption )
2019-09-25 02:02:57 +00:00
- Man-In-The-Middle
- Can decrypt TLS traffic between clients and upstream servers
2019-09-25 03:44:30 +00:00
- See [TLS Encryption ](#tls-interception )
2019-09-24 00:38:10 +00:00
- Supported proxy protocols
- `http`
- `https`
- `http2`
- `websockets`
2019-08-23 00:09:35 +00:00
- Optimized for large file uploads and downloads
2018-12-14 13:14:56 +00:00
- IPv4 and IPv6 support
- Basic authentication support
2019-08-12 05:41:39 +00:00
- Can serve a [PAC (Proxy Auto-configuration) ](https://en.wikipedia.org/wiki/Proxy_auto-config ) file
2019-09-24 00:38:10 +00:00
- See `--pac-file` and `--pac-file-url-path` flags
2013-08-31 09:19:22 +00:00
Install
2019-09-25 03:44:30 +00:00
=======
2013-08-31 09:19:22 +00:00
2019-09-25 17:09:40 +00:00
## Stable version
2013-08-31 09:19:22 +00:00
2019-02-09 04:25:09 +00:00
$ pip install --upgrade proxy.py
2013-08-31 09:19:22 +00:00
2019-09-25 17:09:40 +00:00
## Development version
2019-09-17 03:18:14 +00:00
$ pip install git+https://github.com/abhinavsingh/proxy.py.git@develop
2019-09-25 17:09:40 +00:00
For `Docker` usage see [Docker Image ](#docker-image ).
Start proxy.py
==============
## Command line
Simply type `proxy.py` on command line to start it with default configuration.
```
$ proxy.py
...[redacted]... - Loaded plugin < class ' proxy . HttpProxyPlugin ' >
...[redacted]... - Starting 8 workers
...[redacted]... - Started server on ::1:8899
```
Things to notice from above logs:
- `Loaded plugin` - `proxy.py` will load `HttpProxyPlugin` by default. It adds `http(s)`
proxy server capabilities to `proxy.py`
- `Started N workers` - Use `--num-workers` flag to customize number of `Worker` processes.
By default, `proxy.py` will start as many workers as there are CPU cores on the machine.
- `Started server on ::1:8899` - By default, `proxy.py` listens on IPv6 `::1` , which
is equivalent of IPv4 `127.0.0.1` . If you want to access `proxy.py` externally,
use `--hostname ::` or `--hostname 0.0.0.0` or bind to any other interface available
on your machine.
- `Port 8899` - Use `--port` flag to customize default TCP port.
All the logs above are `INFO` level logs, default `--log-level` for `proxy.py` .
Lets start `proxy.py` with `DEBUG` level logging:
```
$ proxy.py --log-level d
...[redacted]... - Open file descriptor soft limit set to 1024
...[redacted]... - Loaded plugin < class ' proxy . HttpProxyPlugin ' >
...[redacted]... - Started 8 workers
...[redacted]... - Started server on ::1:8899
```
As we can see, before starting up:
- `proxy.py` also tried to set open file limit `ulimit` on the system.
- Default value for `--open-file-limit` used is `1024` .
- `--open-file-limit` flag is a no-op on `Windows` operating systems.
See [flags ](#flags ) for full list of available configuration options.
## Docker image
2019-02-09 05:13:34 +00:00
2019-09-23 07:30:52 +00:00
$ docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:v1.0.0
2019-09-25 17:09:40 +00:00
By default `docker` binary is started with IPv4 networking flags:
2019-09-23 07:30:52 +00:00
2019-09-25 03:44:30 +00:00
--hostname 0.0.0.0 --port 8899
2019-09-23 07:30:52 +00:00
To override input flags, start docker image as follows.
For example, to check `proxy.py --version` :
$ docker run -it \
-p 8899:8899 \
--rm abhinavsingh/proxy.py:v1.0.0 \
--version
[](https://github.com/moby/vpnkit/issues/469)
`docker` image is currently broken on `macOS` due to incompatibility with [vpnkit ](https://github.com/moby/vpnkit/issues/469 ).
2019-02-09 05:13:34 +00:00
2019-09-25 03:44:30 +00:00
Plugin Examples
===============
See [plugin_examples.py ](https://github.com/abhinavsingh/proxy.py/blob/develop/plugin_examples.py ) for full code.
All the examples below also works with `https` traffic but require additional flags and certificate generation.
See [TLS Interception ](#tls-interception ).
2019-09-27 20:00:11 +00:00
## ProposedRestApiPlugin
Mock responses for your server REST API.
Use to test and develop client side applications
without need of an actual upstream REST API server.
Start `proxy.py` as:
```
$ proxy.py \
--plugins plugin_examples.ProposedRestApiPlugin
```
Verify mock API response using `curl -x localhost:8899 http://api.example.com/v1/users/`
```
{"count": 2, "next": null, "previous": null, "results": [{"email": "you@example.com", "groups": [], "url": "api.example.com/v1/users/1/", "username": "admin"}, {"email": "someone@example.com", "groups": [], "url": "api.example.com/v1/users/2/", "username": "admin"}]}
```
Verify the same by inspecting `proxy.py` logs:
```
2019-09-27 12:44:02,212 - INFO - pid:7077 - access_log:1210 - ::1:64792 - GET None:None/v1/users/ - None None - 0 byte
```
Access log shows `None:None` as server `ip:port` . `None` simply means that
the server connection was never made, since response was returned by our plugin.
Now modify `ProposedRestApiPlugin` to returns REST API mock
responses as expected by your clients.
2019-09-25 05:43:29 +00:00
## RedirectToCustomServerPlugin
2019-09-25 03:44:30 +00:00
Redirects all incoming `http` requests to custom web server.
By default, it redirects client requests to inbuilt web server,
also running on `8899` port.
Start `proxy.py` and enable inbuilt web server:
```
$ proxy.py \
--enable-web-server \
--plugins plugin_examples.RedirectToCustomServerPlugin
```
Verify using `curl -v -x localhost:8899 http://google.com`
```
... [redacted] ...
< HTTP / 1 . 1 404 NOT FOUND
< Server: proxy . py v1 . 0 . 0
< Connection: Close
<
* Closing connection 0
```
Above `404` response was returned from `proxy.py` web server.
Verify the same by inspecting the logs for `proxy.py` .
Along with the proxy request log, you must also see a http web server request log.
```
2019-09-24 19:09:33,602 - INFO - pid:49996 - access_log:1241 - ::1:49525 - GET /
2019-09-24 19:09:33,603 - INFO - pid:49995 - access_log:1157 - ::1:49524 - GET localhost:8899/ - 404 NOT FOUND - 70 bytes
```
2019-09-25 05:43:29 +00:00
## FilterByUpstreamHostPlugin
2019-09-25 03:44:30 +00:00
Drops traffic by inspecting upstream host.
By default, plugin drops traffic for `google.com` and `www.google.com` .
Start `proxy.py` as:
```
$ proxy.py \
--plugins plugin_examples.FilterByUpstreamHostPlugin
```
Verify using `curl -v -x localhost:8899 http://google.com` :
```
... [redacted] ...
< HTTP / 1 . 1 418 I ' m a tea pot
< Proxy-agent: proxy . py v1 . 0 . 0
* no chunk, no close, no size. Assume close to signal end
<
* Closing connection 0
```
Above `418 I'm a tea pot` is sent by our plugin.
Verify the same by inspecting logs for `proxy.py` :
```
2019-09-27 16:18:58 +00:00
2019-09-24 19:21:37,893 - ERROR - pid:50074 - handle_readables:1347 - ProtocolException type raised
2019-09-25 03:44:30 +00:00
Traceback (most recent call last):
... [redacted] ...
2019-09-24 19:21:37,897 - INFO - pid:50074 - access_log:1157 - ::1:49911 - GET None:None/ - None None - 0 bytes
```
2019-09-25 05:43:29 +00:00
## CacheResponsesPlugin
2019-09-25 03:44:30 +00:00
Caches Upstream Server Responses.
Start `proxy.py` as:
```
$ proxy.py \
--plugins plugin_examples.CacheResponsesPlugin
```
Verify using `curl -v -x localhost:8899 http://httpbin.org/get` :
```
... [redacted] ...
< HTTP / 1 . 1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Type: application / json
< Date: Wed , 25 Sep 2019 02:24:25 GMT
< Referrer-Policy: no-referrer-when-downgrade
< Server: nginx
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-XSS-Protection: 1 ; mode = block
< Content-Length: 202
< Connection: keep-alive
<
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "httpbin.org",
"User-Agent": "curl/7.54.0"
},
"origin": "1.2.3.4, 5.6.7.8",
"url": "https://httpbin.org/get"
}
* Connection #0 to host localhost left intact
```
Get path to the cache file from `proxy.py` logs:
```
... [redacted] ... - GET httpbin.org:80/get - 200 OK - 556 bytes
... [redacted] ... - Cached response at /var/folders/k9/x93q0_xn1ls9zy76m2mf2k_00000gn/T/httpbin.org-1569378301.407512.txt
```
Verify contents of the cache file `cat /path/to/your/cache/httpbin.org.txt`
```
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 25 Sep 2019 02:24:25 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 202
Connection: keep-alive
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "httpbin.org",
"User-Agent": "curl/7.54.0"
},
"origin": "1.2.3.4, 5.6.7.8",
"url": "https://httpbin.org/get"
}
```
2019-09-25 05:43:29 +00:00
## ManInTheMiddlePlugin
2019-09-25 03:44:30 +00:00
Modifies upstream server responses.
Start `proxy.py` as:
```
$ proxy.py \
--plugins plugin_examples.ManInTheMiddlePlugin
```
Verify using `curl -v -x localhost:8899 http://google.com` :
```
... [redacted] ...
< HTTP / 1 . 1 200 OK
< Content-Length: 28
<
* Connection #0 to host localhost left intact
Hello from man in the middle
```
Response body `Hello from man in the middle` is sent by our plugin.
2019-09-25 05:43:29 +00:00
## Plugin Ordering
2019-09-25 03:44:30 +00:00
When using multiple plugins, depending upon plugin functionality,
it might be worth considering the order in which plugins are passed
on the command line.
Plugins are called in the same order as they are passed. Example,
say we are using both `FilterByUpstreamHostPlugin` and
`RedirectToCustomServerPlugin` . Idea is to drop all incoming `http`
requests for `google.com` and `www.google.com` and redirect other
`http` requests to our inbuilt web server.
Hence, in this scenario it is important to use
`FilterByUpstreamHostPlugin` before `RedirectToCustomServerPlugin` .
If we enable `RedirectToCustomServerPlugin` before `FilterByUpstreamHostPlugin` ,
2019-09-25 05:43:29 +00:00
`google` requests will also get redirected to inbuilt web server,
instead of being dropped.
2019-09-25 03:44:30 +00:00
End-to-End Encryption
=====================
By default, `proxy.py` uses `http` protocol for communication with clients e.g. `curl` , `browser` .
2019-09-25 05:43:29 +00:00
For enabling end-to-end encrypting using `tls` / `https` first generate certificates:
2019-09-25 03:44:30 +00:00
```
make https-certificates
```
Start `proxy.py` as:
```
$ proxy.py \
--cert-file https-cert.pem \
--key-file https-key.pem
```
Verify using `curl -x https://localhost:8899 --proxy-cacert https-cert.pem https://httpbin.org/get` :
```
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "httpbin.org",
"User-Agent": "curl/7.54.0"
},
"origin": "1.2.3.4, 5.6.7.8",
"url": "https://httpbin.org/get"
}
```
TLS Interception
=================
2019-09-25 05:43:29 +00:00
By default, `proxy.py` doesn't decrypt `https` traffic between client and server.
2019-09-25 03:44:30 +00:00
To enable TLS interception first generate CA certificates:
```
make ca-certificates
```
2019-09-26 01:27:26 +00:00
Lets also enable `CacheResponsePlugin` so that we can verify decrypted
response from the server. Start `proxy.py` as:
2019-09-25 03:44:30 +00:00
```
$ proxy.py \
2019-09-26 01:27:26 +00:00
--plugins plugin_examples.CacheResponsesPlugin \
2019-09-25 03:44:30 +00:00
--ca-key-file ca-key.pem \
--ca-cert-file ca-cert.pem \
--ca-signing-key-file ca-signing-key.pem
```
2019-09-26 01:27:26 +00:00
Verify using `curl -v -x localhost:8899 --cacert ca-cert.pem https://httpbin.org/get`
```
* issuer: C=US; ST=CA; L=SanFrancisco; O=proxy.py; OU=CA; CN=Proxy PY CA; emailAddress=proxyca@mailserver.com
* SSL certificate verify ok.
> GET /get HTTP/1.1
... [redacted] ...
< Connection: keep-alive
<
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "httpbin.org",
"User-Agent": "curl/7.54.0"
},
"origin": "1.2.3.4, 5.6.7.8",
"url": "https://httpbin.org/get"
}
```
The `issuer` line confirms that response was intercepted.
Also verify the contents of cached response file. Get path to the cache
file from `proxy.py` logs.
`$ cat /path/to/your/tmp/directory/httpbin.org-1569452863.924174.txt`
2019-09-25 03:44:30 +00:00
```
2019-09-26 01:27:26 +00:00
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 25 Sep 2019 23:07:05 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 202
Connection: keep-alive
2019-09-25 03:44:30 +00:00
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "httpbin.org",
"User-Agent": "curl/7.54.0"
},
"origin": "1.2.3.4, 5.6.7.8",
"url": "https://httpbin.org/get"
}
2019-09-26 01:27:26 +00:00
2019-09-25 03:44:30 +00:00
```
2019-09-26 01:27:26 +00:00
Viola!!! If you remove CA flags, encrypted data will be found in the
cached file instead of plain text.
Now use CA flags other
2019-09-25 05:43:29 +00:00
[plugin examples ](#plugin-examples ) to make them work for `https` traffic.
2019-10-02 07:09:35 +00:00
import proxy.py
===============
You can directly import `proxy.py` into your `Python` code. Example:
```
$ python
>>> import proxy
>>>
>>> # Generate HTTP GET request
>>> proxy.build_http_request(b'GET', b'/')
b'GET / HTTP/1.1\r\n\r\n'
>>>
>>> # Generate HTTP GET request with headers
>>> proxy.build_http_request(b'GET', b'/',
headers={b'Connection': b'close'})
b'GET / HTTP/1.1\r\nConnection: close\r\n\r\n'
>>>
>>> # Generate HTTP POST request with headers and body
>>> import json
>>> proxy.build_http_request(b'POST', b'/form',
headers={b'Content-type': b'application/json'},
body=proxy.bytes_(json.dumps({'email': 'hello@world.com'})))
b'POST /form HTTP/1.1\r\nContent-type: application/json\r\n\r\n{"email": "hello@world.com"}'
```
To start `proxy.py` server from imported `proxy.py` module, simply do:
```
import sys
import proxy
if __name__ == '__main__':
proxy.main(sys.argv[1:])
```
2019-09-25 05:43:29 +00:00
Plugin Developer and Contributor Guide
======================================
## Everything is a plugin
As you might have guessed by now, in `proxy.py` everything is a plugin.
- We enabled proxy server plugins using `--plugins` flag.
All the [plugin examples ](#plugin-examples ) were implementing
`HttpProxyBasePlugin` . See documentation of
[HttpProxyBasePlugin ](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L894-L938 )
for available lifecycle hooks. Use `HttpProxyBasePlugin` to modify
behavior of http(s) proxy protocol between client and upstream server.
Example, [FilterByUpstreamHostPlugin ](#filterbyupstreamhostplugin ).
- We also enabled inbuilt web server using `--enable-web-server` .
2019-09-27 16:18:58 +00:00
Inbuilt web server implements `ProtocolHandlerPlugin` plugin.
See documentation of [ProtocolHandlerPlugin ](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L793-L850 )
for available lifecycle hooks. Use `ProtocolHandlerPlugin` to add
2019-09-25 05:43:29 +00:00
new features for http(s) clients. Example,
[HttpWebServerPlugin ](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L1185-L1260 ).
- There also is a `--disable-http-proxy` flag. It disables inbuilt proxy server.
Use this flag with `--enable-web-server` flag to run `proxy.py` as a programmable
http(s) server. [HttpProxyPlugin ](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L941-L1182 )
2019-09-27 16:18:58 +00:00
also implements `ProtocolHandlerPlugin` .
2019-09-25 05:43:29 +00:00
2019-10-02 07:09:35 +00:00
## Internal Architecture
2019-09-25 05:43:29 +00:00
2019-09-27 16:18:58 +00:00
- [ProtocolHandler ](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L1263-L1440 )
2019-09-25 05:43:29 +00:00
thread is started with the accepted [TcpClientConnection ](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L230-L237 ).
2019-09-27 16:18:58 +00:00
`ProtocolHandler` is responsible for parsing incoming client request and invoking
`ProtocolHandlerPlugin` lifecycle hooks.
2019-09-25 05:43:29 +00:00
2019-09-27 16:18:58 +00:00
- `HttpProxyPlugin` which implements `ProtocolHandlerPlugin` also has its own plugin
2019-09-25 05:43:29 +00:00
mechanism. Its responsibility is to establish connection between client and
upstream [TcpServerConnection ](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L204-L227 )
and invoke `HttpProxyBasePlugin` lifecycle hooks.
2019-09-27 16:18:58 +00:00
- `ProtocolHandler` threads are started by [Worker ](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L424-L472 )
2019-09-25 05:43:29 +00:00
processes.
- `--num-workers` `Worker` processes are started by
2019-10-02 07:09:35 +00:00
[AcceptorPool ](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L368-L421 )
on start-up.
- `AcceptorPool` listens on server socket and pass the handler to `Worker` processes.
Workers are responsible for accepting new client connections and starting
`ProtocolHandler` thread.
2019-09-25 05:43:29 +00:00
2019-10-02 07:09:35 +00:00
## Sending a Pull Request
2019-09-25 03:44:30 +00:00
2019-10-02 07:09:35 +00:00
Install dependencies for local development testing:
`$ pip install -r requirements-testing.txt`
2019-09-25 22:51:12 +00:00
Every pull request goes through set of tests which must pass:
- `mypy` : Run `make lint` locally for compliance check.
Fix all warnings and errors before sending out a PR.
2019-10-02 07:09:35 +00:00
- `coverage` : Run `make coverage` locally for coverage report.
2019-09-25 22:51:12 +00:00
Its ideal to add tests for any critical change. Depending upon
the change, it's ok if test coverage falls by `<0.5%` .
- `formatting` : Run `make autopep8` locally to format the code in-place.
`autopep8` is run with `--aggresive` flag. Sometimes it _may_ result in
2019-10-02 07:09:35 +00:00
weird formatting. But let's stick to one consistent formatting tool.
I am open to flag changes for `autopep8` .
## Internal Documentation
Browse through internal class hierarchy and documentation using `pydoc3` .
Example:
```
$ pydoc3 proxy
Help on module proxy:
NAME
proxy
DESCRIPTION
proxy.py
~~~~~~~~
Lightweight, Programmable, TLS interceptor Proxy for HTTP(S), HTTP2, WebSockets protocols in a single Python file.
:copyright: (c) 2013-present by Abhinav Singh and contributors.
:license: BSD, see LICENSE for more details.
CLASSES
abc.ABC(builtins.object)
HttpProxyBasePlugin
HttpWebServerRoutePlugin
HttpWebServerPacFilePlugin
ProtocolHandlerPlugin
HttpProxyPlugin
HttpWebServerPlugin
TcpConnection
TcpClientConnection
TcpServerConnection
builtins.Exception(builtins.BaseException)
ProtocolException
HttpRequestRejected
ProxyAuthenticationFailed
ProxyConnectionFailed
TcpConnectionUninitializedException
builtins.object
AcceptorPool
ChunkParser
HttpParser
ProtocolConfig
Websocket
WebsocketFrame
builtins.tuple(builtins.object)
ChunkParserStates
HttpParserStates
HttpParserTypes
HttpProtocolTypes
TcpConnectionTypes
multiprocessing.context.Process(multiprocessing.process.BaseProcess)
Worker
threading.Thread(builtins.object)
ProtocolHandler
... [ truncated ] ...
```
Frequently Asked Questions
==========================
## Unable to connect with proxy.py from remote host
Make sure `proxy.py` is listening on correct network interface.
Try following flags:
- For IPv6 `--hostname ::`
- For IPv4 `--hostname 0.0.0.0`
## Basic auth not working with a browser
Most likely it's a browser integration issue with system keychain.
- First verify that basic auth is working using `curl`
`curl -v -x username:password@localhost:8899 https://httpbin.org/get`
- See [this thread ](https://github.com/abhinavsingh/proxy.py/issues/89#issuecomment-534845710 )
for further details.
## Docker image not working on macOS
It's a compatibility issue with `vpnkit` .
See [moby/vpnkit exhausts docker resources ](https://github.com/abhinavsingh/proxy.py/issues/43 )
and [Connection refused: The proxy could not connect ](https://github.com/moby/vpnkit/issues/469 )
for some background.
2019-09-25 22:51:12 +00:00
2019-09-25 05:43:29 +00:00
Flags
2019-09-25 03:44:30 +00:00
=====
2013-08-31 09:19:22 +00:00
```
2019-07-20 19:47:16 +00:00
$ proxy.py -h
usage: proxy.py [-h] [--backlog BACKLOG] [--basic-auth BASIC_AUTH]
2019-09-25 02:02:57 +00:00
[--ca-key-file CA_KEY_FILE] [--ca-cert-dir CA_CERT_DIR]
[--ca-cert-file CA_CERT_FILE]
[--ca-signing-key-file CA_SIGNING_KEY_FILE]
[--cert-file CERT_FILE]
2018-12-13 21:08:08 +00:00
[--client-recvbuf-size CLIENT_RECVBUF_SIZE]
2019-09-02 22:58:37 +00:00
[--disable-headers DISABLE_HEADERS] [--disable-http-proxy]
2019-09-25 02:02:57 +00:00
[--enable-web-server] [--hostname HOSTNAME]
[--key-file KEY_FILE] [--log-level LOG_LEVEL]
[--log-file LOG_FILE] [--log-format LOG_FORMAT]
[--num-workers NUM_WORKERS]
2019-08-23 00:09:35 +00:00
[--open-file-limit OPEN_FILE_LIMIT] [--pac-file PAC_FILE]
2019-08-26 16:52:43 +00:00
[--pac-file-url-path PAC_FILE_URL_PATH] [--pid-file PID_FILE]
[--plugins PLUGINS] [--port PORT]
[--server-recvbuf-size SERVER_RECVBUF_SIZE] [--version]
2013-08-31 09:19:22 +00:00
2019-09-14 04:18:05 +00:00
proxy.py v1.0.0
2013-08-31 09:19:22 +00:00
2019-08-23 00:09:35 +00:00
optional arguments:
2013-08-31 09:19:22 +00:00
-h, --help show this help message and exit
2018-12-13 17:01:56 +00:00
--backlog BACKLOG Default: 100. Maximum number of pending connections to
proxy server
--basic-auth BASIC_AUTH
Default: No authentication. Specify colon separated
user:password to enable basic authentication.
2019-09-25 02:02:57 +00:00
--ca-key-file CA_KEY_FILE
Default: None. CA key to use for signing dynamically
generated HTTPS certificates. If used, must also pass
--ca-cert-file and --ca-signing-key-file
--ca-cert-dir CA_CERT_DIR
Default: ~/.proxy.py. Directory to store dynamically
generated certificates. Also see --ca-key-file, --ca-
cert-file and --ca-signing-key-file
--ca-cert-file CA_CERT_FILE
Default: None. Signing certificate to use for signing
dynamically generated HTTPS certificates. If used,
must also pass --ca-key-file and --ca-signing-key-file
--ca-signing-key-file CA_SIGNING_KEY_FILE
Default: None. CA signing key to use for dynamic
generation of HTTPS certificates. If used, must also
pass --ca-key-file and --ca-cert-file
--cert-file CERT_FILE
Default: None. Server certificate to enable end-to-end
TLS encryption with clients. If used, must also pass
--key-file.
2018-12-13 21:08:08 +00:00
--client-recvbuf-size CLIENT_RECVBUF_SIZE
2019-08-26 16:37:25 +00:00
Default: 1 MB. Maximum amount of data received from
2018-12-13 21:08:08 +00:00
the client in a single recv() operation. Bump this
value for faster uploads at the expense of increased
RAM.
2019-09-02 22:58:37 +00:00
--disable-headers DISABLE_HEADERS
Default: None. Comma separated list of headers to
2019-09-14 04:18:05 +00:00
remove before dispatching client request to upstream
2019-09-02 22:58:37 +00:00
server.
--disable-http-proxy Default: False. Whether to disable
proxy.HttpProxyPlugin.
2019-08-24 17:23:30 +00:00
--enable-web-server Default: False. Whether to enable
2019-08-23 00:09:35 +00:00
proxy.HttpWebServerPlugin.
2019-09-25 02:02:57 +00:00
--hostname HOSTNAME Default: ::1. Server IP address.
--key-file KEY_FILE Default: None. Server key file to enable end-to-end
TLS encryption with clients. If used, must also pass
--cert-file.
2013-08-31 09:19:22 +00:00
--log-level LOG_LEVEL
2019-07-20 19:17:01 +00:00
Valid options: DEBUG, INFO (default), WARNING, ERROR,
2019-09-14 04:18:05 +00:00
CRITICAL. Both upper and lowercase values are allowed.
You may also simply use the leading character e.g.
--log-level d
2019-08-26 16:37:25 +00:00
--log-file LOG_FILE Default: sys.stdout. Log file destination.
2019-08-20 05:59:11 +00:00
--log-format LOG_FORMAT
Log format for Python logger.
2019-08-24 17:23:30 +00:00
--num-workers NUM_WORKERS
Defaults to number of CPU cores.
2019-07-20 19:47:16 +00:00
--open-file-limit OPEN_FILE_LIMIT
Default: 1024. Maximum number of files (TCP
connections) that proxy.py can open concurrently.
2019-07-20 19:17:01 +00:00
--pac-file PAC_FILE A file (Proxy Auto Configuration) or string to serve
2019-09-14 04:18:05 +00:00
when the server receives a direct file request. Using
this option enables proxy.HttpWebServerPlugin.
2019-08-23 00:09:35 +00:00
--pac-file-url-path PAC_FILE_URL_PATH
2019-09-14 04:18:05 +00:00
Default: /. Web server path to serve the PAC file.
2019-08-26 16:52:43 +00:00
--pid-file PID_FILE Default: None. Save parent process ID to a file.
2019-08-23 00:09:35 +00:00
--plugins PLUGINS Comma separated plugins
--port PORT Default: 8899. Server port.
2019-07-20 19:47:16 +00:00
--server-recvbuf-size SERVER_RECVBUF_SIZE
2019-08-26 16:37:25 +00:00
Default: 1 MB. Maximum amount of data received from
2019-07-20 19:47:16 +00:00
the server in a single recv() operation. Bump this
value for faster downloads at the expense of increased
RAM.
2019-09-14 04:18:05 +00:00
--version, -v Prints proxy.py version.
2013-08-31 09:19:22 +00:00
2019-07-20 19:17:01 +00:00
Proxy.py not working? Report at:
2013-08-31 09:19:22 +00:00
https://github.com/abhinavsingh/proxy.py/issues/new
```
