1133 lines
38 KiB
Markdown
1133 lines
38 KiB
Markdown
[](https://github.com/abhinavsingh/proxy.py)
|
||
|
||
[](https://opensource.org/licenses/BSD-3-Clause)
|
||
[](https://pypi.org/project/proxy.py/)
|
||
[](https://hub.docker.com/r/abhinavsingh/proxy.py)
|
||
[](https://travis-ci.org/abhinavsingh/proxy.py/)
|
||
[](https://github.com/abhinavsingh/proxy.py)
|
||
[](https://codecov.io/gh/abhinavsingh/proxy.py)
|
||
|
||
[](https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/iOS_Simulator_Guide/Introduction/Introduction.html)
|
||
[](https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/iOS_Simulator_Guide/Introduction/Introduction.html)
|
||
[](https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/iOS_Simulator_Guide/Introduction/Introduction.html)
|
||
[](https://www.android.com/)
|
||
[](https://developer.android.com/studio/run/emulator-networking.html#proxy)
|
||
[](https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/iOS_Simulator_Guide/Introduction/Introduction.html)
|
||
[](https://developer.apple.com/library/archive/documentation/IDEs/Conceptual/iOS_Simulator_Guide/Introduction/Introduction.html)
|
||
|
||
[](https://gitHub.com/abhinavsingh/proxy.py/graphs/commit-activity)
|
||
[](https://twitter.com/imoracle)
|
||
[](https://github.com/abhinavsingh/proxy.py/issues)
|
||
[](https://gitter.im/proxy-py/community)
|
||
|
||
[](https://www.python.org/downloads/release/python-360/)
|
||
[](https://www.python.org/downloads/release/python-370/)
|
||
[](http://mypy-lang.org/)
|
||
|
||
[](https://opencollective.com/proxypy)
|
||
|
||
Table of Contents
|
||
=================
|
||
|
||
* [Features](#features)
|
||
* [Install](#install)
|
||
* [Stable version](#stable-version)
|
||
* [Development version](#development-version)
|
||
* [Start proxy.py](#start-proxypy)
|
||
* [From command line when installed using PIP](#from-command-line-when-installed-using-pip)
|
||
* [Run it](#run-it)
|
||
* [Reading logs](#reading-logs)
|
||
* [Enable DEBUG logging](#enable-debug-logging)
|
||
* [From command line using repo source](#from-command-line-using-repo-source)
|
||
* [Docker Image](#docker-image)
|
||
* [Stable version](#stable-version-from-docker-hub)
|
||
* [Development version](#build-development-version-locally)
|
||
* [Customize Startup Flags](#customize-startup-flags)
|
||
* [Plugin Examples](#plugin-examples)
|
||
* [ShortLinkPlugin](#shortlinkplugin)
|
||
* [ModifyPostDataPlugin](#modifypostdataplugin)
|
||
* [MockRestApiPlugin](#mockrestapiplugin)
|
||
* [RedirectToCustomServerPlugin](#redirecttocustomserverplugin)
|
||
* [FilterByUpstreamHostPlugin](#filterbyupstreamhostplugin)
|
||
* [CacheResponsesPlugin](#cacheresponsesplugin)
|
||
* [ManInTheMiddlePlugin](#maninthemiddleplugin)
|
||
* [Plugin Ordering](#plugin-ordering)
|
||
* [End-to-End Encryption](#end-to-end-encryption)
|
||
* [TLS Interception](#tls-interception)
|
||
* [Embed proxy.py](#embed-proxypy)
|
||
* [Plugin Developer and Contributor Guide](#plugin-developer-and-contributor-guide)
|
||
* [Everything is a plugin](#everything-is-a-plugin)
|
||
* [Internal Architecture](#internal-architecture)
|
||
* [Internal Documentation](#internal-documentation)
|
||
* [Sending a Pull Request](#sending-a-pull-request)
|
||
* [Utilities](#utilities)
|
||
* [TCP](#tcp-sockets)
|
||
* [new_socket_connection](#new_socket_connection)
|
||
* [socket_connection](#socket_connection)
|
||
* [Http](#http-client)
|
||
* [build_http_request](#build_http_request)
|
||
* [build_http_response](#build_http_response)
|
||
* [Websocket](#websocket)
|
||
* [WebsocketFrame](#websocketframe)
|
||
* [WebsocketClient](#websocketclient)
|
||
* [Frequently Asked Questions](#frequently-asked-questions)
|
||
* [SyntaxError: invalid syntax](#syntaxerror-invalid-syntax)
|
||
* [Unable to connect with proxy.py from remote host](#unable-to-connect-with-proxypy-from-remote-host)
|
||
* [Basic auth not working with a browser](#basic-auth-not-working-with-a-browser)
|
||
* [Docker image not working on MacOS](#docker-image-not-working-on-macos)
|
||
* [Unable to load custom plugins](#unable-to-load-custom-plugins)
|
||
* [ValueError: filedescriptor out of range in select](#valueerror-filedescriptor-out-of-range-in-select)
|
||
* [Flags](#flags)
|
||
* [Changelog](#changelog)
|
||
|
||
Features
|
||
========
|
||
|
||
- Fast & Scalable
|
||
- Scales by using all available cores on the system
|
||
- Threadless executions using coroutine
|
||
- Made to handle `tens-of-thousands` connections / sec
|
||
```
|
||
# On Macbook Pro 2015 / 2.8 GHz Intel Core i7
|
||
$ hey -n 10000 -c 100 http://localhost:8899/
|
||
|
||
Summary:
|
||
Total: 0.6157 secs
|
||
Slowest: 0.1049 secs
|
||
Fastest: 0.0007 secs
|
||
Average: 0.0055 secs
|
||
Requests/sec: 16240.5444
|
||
|
||
Total data: 800000 bytes
|
||
Size/request: 80 bytes
|
||
|
||
Response time histogram:
|
||
0.001 [1] |
|
||
0.011 [9565] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
|
||
0.022 [332] |■
|
||
```
|
||
- Lightweight
|
||
- Uses only `~5-20MB` RAM
|
||
- No external dependency other than standard Python library
|
||
- Programmable
|
||
- Optionally enable builtin Web Server
|
||
- Customize proxy and http routing via [plugins](https://github.com/abhinavsingh/proxy.py/blob/develop/plugin_examples)
|
||
- Enable plugin using command line option e.g. `--plugins plugin_examples/cache_responses.CacheResponsesPlugin`
|
||
- Plugin API is currently in development phase, expect breaking changes.
|
||
- Realtime Dashboard
|
||
- Optionally enable bundled dashboard.
|
||
- Available at `http://localhost:8899/dashboard`.
|
||
- Inspect, Monitor, Control and Configure `proxy.py` at runtime.
|
||
- Extend dashboard using plugins.
|
||
- Dashboard is currently in development phase, expect breaking changes.
|
||
- Secure
|
||
- Enable end-to-end encryption between clients and `proxy.py` using TLS
|
||
- See [End-to-End Encryption](#end-to-end-encryption)
|
||
- Man-In-The-Middle
|
||
- Can decrypt TLS traffic between clients and upstream servers
|
||
- See [TLS Encryption](#tls-interception)
|
||
- Supported proxy protocols
|
||
- `http`
|
||
- `https`
|
||
- `http2`
|
||
- `websockets`
|
||
- Optimized for large file uploads and downloads
|
||
- IPv4 and IPv6 support
|
||
- Basic authentication support
|
||
- Can serve a [PAC (Proxy Auto-configuration)](https://en.wikipedia.org/wiki/Proxy_auto-config) file
|
||
- See `--pac-file` and `--pac-file-url-path` flags
|
||
|
||
Install
|
||
=======
|
||
|
||
## Stable version
|
||
|
||
Install from `PyPi`
|
||
|
||
$ pip install --upgrade proxy.py
|
||
|
||
or from GitHub `master` branch
|
||
|
||
$ pip install git+https://github.com/abhinavsingh/proxy.py.git@master
|
||
|
||
## Development version
|
||
|
||
$ pip install git+https://github.com/abhinavsingh/proxy.py.git@develop
|
||
|
||
For `Docker` installation see [Docker Image](#docker-image).
|
||
|
||
Start proxy.py
|
||
==============
|
||
|
||
## From command line when installed using PIP
|
||
|
||
When `proxy.py` is installed using `pip`,
|
||
an executable named `proxy` is added under your `$PATH`.
|
||
|
||
#### Run it
|
||
|
||
Simply type `proxy` on command line to start it with default configuration.
|
||
|
||
```
|
||
$ proxy
|
||
...[redacted]... - Loaded plugin proxy.http_proxy.HttpProxyPlugin
|
||
...[redacted]... - Starting 8 workers
|
||
...[redacted]... - Started server on ::1:8899
|
||
```
|
||
|
||
#### Reading logs
|
||
|
||
Things to notice from above logs:
|
||
|
||
- `Loaded plugin` - `proxy.py` will load `proxy.http.proxy.HttpProxyPlugin` by default.
|
||
As name suggests, this core plugin adds `http(s)` proxy server capabilities to `proxy.py`
|
||
|
||
- `Started N workers` - Use `--num-workers` flag to customize number of worker processes.
|
||
By default, `proxy.py` will start as many workers as there are CPU cores on the machine.
|
||
|
||
- `Started server on ::1:8899` - By default, `proxy.py` listens on IPv6 `::1`, which
|
||
is equivalent of IPv4 `127.0.0.1`. If you want to access `proxy.py` externally,
|
||
use `--hostname ::` or `--hostname 0.0.0.0` or bind to any other interface available
|
||
on your machine.
|
||
|
||
- `Port 8899` - Use `--port` flag to customize default TCP port.
|
||
|
||
#### Enable DEBUG logging
|
||
|
||
All the logs above are `INFO` level logs, default `--log-level` for `proxy.py`.
|
||
|
||
Lets start `proxy.py` with `DEBUG` level logging:
|
||
|
||
```
|
||
$ proxy --log-level d
|
||
...[redacted]... - Open file descriptor soft limit set to 1024
|
||
...[redacted]... - Loaded plugin proxy.http_proxy.HttpProxyPlugin
|
||
...[redacted]... - Started 8 workers
|
||
...[redacted]... - Started server on ::1:8899
|
||
```
|
||
|
||
As we can see, before starting up:
|
||
|
||
- `proxy.py` also tried to set open file limit `ulimit` on the system.
|
||
- Default value for `--open-file-limit` used is `1024`.
|
||
- `--open-file-limit` flag is a no-op on `Windows` operating systems.
|
||
|
||
See [flags](#flags) for full list of available configuration options.
|
||
|
||
## From command line using repo source
|
||
|
||
If you are trying to run `proxy.py` from source code,
|
||
there is no binary file named `proxy` in the source code.
|
||
|
||
To start `proxy.py` from source code follow these instructions:
|
||
|
||
- Clone repo
|
||
|
||
```
|
||
$ git clone https://github.com/abhinavsingh/proxy.py.git
|
||
$ cd proxy.py
|
||
```
|
||
|
||
- Create a Python 3 virtual env
|
||
|
||
```
|
||
$ python3 -m venv venv
|
||
$ source venv/bin/activate
|
||
```
|
||
|
||
- Install deps
|
||
|
||
```
|
||
$ pip install -r requirements.txt
|
||
$ pip install -r requirements-testing.txt
|
||
```
|
||
|
||
- Run tests
|
||
|
||
```
|
||
$ make
|
||
```
|
||
|
||
- Run proxy.py
|
||
|
||
```
|
||
$ python -m proxy
|
||
```
|
||
|
||
Also see [Plugin Developer and Contributor Guide](#plugin-developer-and-contributor-guide)
|
||
if you plan to work with `proxy.py` source code.
|
||
|
||
## Docker image
|
||
|
||
#### Stable Version from Docker Hub
|
||
|
||
$ docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:latest
|
||
|
||
#### Build Development Version Locally
|
||
|
||
$ git clone https://github.com/abhinavsingh/proxy.py.git
|
||
$ cd proxy.py
|
||
$ make container
|
||
$ docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:latest
|
||
|
||
#### Customize startup flags
|
||
|
||
By default `docker` binary is started with IPv4 networking flags:
|
||
|
||
--hostname 0.0.0.0 --port 8899
|
||
|
||
To override input flags, start docker image as follows.
|
||
For example, to check `proxy.py` version within Docker image:
|
||
|
||
$ docker run -it \
|
||
-p 8899:8899 \
|
||
--rm abhinavsingh/proxy.py:latest \
|
||
-v
|
||
|
||
[](https://github.com/moby/vpnkit/issues/469)
|
||
`docker` image is currently broken on `macOS` due to incompatibility with [vpnkit](https://github.com/moby/vpnkit/issues/469).
|
||
|
||
Plugin Examples
|
||
===============
|
||
|
||
See [plugin_examples](https://github.com/abhinavsingh/proxy.py/tree/develop/plugin_examples) for full code.
|
||
|
||
All the examples below also works with `https` traffic but require additional flags and certificate generation.
|
||
See [TLS Interception](#tls-interception).
|
||
|
||
## ShortLinkPlugin
|
||
|
||
Add support for short links in your favorite browsers / applications.
|
||
|
||
Start `proxy.py` as:
|
||
|
||
```
|
||
$ proxy \
|
||
--plugins plugin_examples/shortlink.ShortLinkPlugin
|
||
```
|
||
|
||
Now you can speed up your daily browsing experience by visiting your
|
||
favorite website using single character domain names :). This works
|
||
across all browsers.
|
||
|
||
Following short links are enabled by default:
|
||
|
||
Short Link | Destination URL
|
||
:--------: | :---------------:
|
||
a/ | amazon.com
|
||
i/ | instagram.com
|
||
l/ | linkedin.com
|
||
f/ | facebook.com
|
||
g/ | google.com
|
||
t/ | twitter.com
|
||
w/ | web.whatsapp.com
|
||
y/ | youtube.com
|
||
proxy/ | localhost:8899
|
||
|
||
## ModifyPostDataPlugin
|
||
|
||
Modifies POST request body before sending request to upstream server.
|
||
|
||
Start `proxy.py` as:
|
||
|
||
```
|
||
$ proxy \
|
||
--plugins plugin_examples/modify_post_data.ModifyPostDataPlugin
|
||
```
|
||
|
||
By default plugin replaces POST body content with hardcoded `b'{"key": "modified"}'`
|
||
and enforced `Content-Type: application/json`.
|
||
|
||
Verify the same using `curl -x localhost:8899 -d '{"key": "value"}' http://httpbin.org/post`
|
||
|
||
```
|
||
{
|
||
"args": {},
|
||
"data": "{\"key\": \"modified\"}",
|
||
"files": {},
|
||
"form": {},
|
||
"headers": {
|
||
"Accept": "*/*",
|
||
"Content-Length": "19",
|
||
"Content-Type": "application/json",
|
||
"Host": "httpbin.org",
|
||
"User-Agent": "curl/7.54.0"
|
||
},
|
||
"json": {
|
||
"key": "modified"
|
||
},
|
||
"origin": "1.2.3.4, 5.6.7.8",
|
||
"url": "https://httpbin.org/post"
|
||
}
|
||
```
|
||
|
||
Note following from the response above:
|
||
|
||
1. POST data was modified `"data": "{\"key\": \"modified\"}"`.
|
||
Original `curl` command data was `{"key": "value"}`.
|
||
2. Our `curl` command did not add any `Content-Type` header,
|
||
but our plugin did add one `"Content-Type": "application/json"`.
|
||
Same can also be verified by looking at `json` field in the output above:
|
||
```
|
||
"json": {
|
||
"key": "modified"
|
||
},
|
||
```
|
||
3. Our plugin also added a `Content-Length` header to match length
|
||
of modified body.
|
||
|
||
## MockRestApiPlugin
|
||
|
||
Mock responses for your server REST API.
|
||
Use to test and develop client side applications
|
||
without need of an actual upstream REST API server.
|
||
|
||
Start `proxy.py` as:
|
||
|
||
```
|
||
$ proxy \
|
||
--plugins plugin_examples/mock_rest_api.ProposedRestApiPlugin
|
||
```
|
||
|
||
Verify mock API response using `curl -x localhost:8899 http://api.example.com/v1/users/`
|
||
|
||
```
|
||
{"count": 2, "next": null, "previous": null, "results": [{"email": "you@example.com", "groups": [], "url": "api.example.com/v1/users/1/", "username": "admin"}, {"email": "someone@example.com", "groups": [], "url": "api.example.com/v1/users/2/", "username": "admin"}]}
|
||
```
|
||
|
||
Verify the same by inspecting `proxy.py` logs:
|
||
|
||
```
|
||
2019-09-27 12:44:02,212 - INFO - pid:7077 - access_log:1210 - ::1:64792 - GET None:None/v1/users/ - None None - 0 byte
|
||
```
|
||
|
||
Access log shows `None:None` as server `ip:port`. `None` simply means that
|
||
the server connection was never made, since response was returned by our plugin.
|
||
|
||
Now modify `ProposedRestApiPlugin` to returns REST API mock
|
||
responses as expected by your clients.
|
||
|
||
## RedirectToCustomServerPlugin
|
||
|
||
Redirects all incoming `http` requests to custom web server.
|
||
By default, it redirects client requests to inbuilt web server,
|
||
also running on `8899` port.
|
||
|
||
Start `proxy.py` and enable inbuilt web server:
|
||
|
||
```
|
||
$ proxy \
|
||
--enable-web-server \
|
||
--plugins plugin_examples/redirect_to_custom_server.RedirectToCustomServerPlugin
|
||
```
|
||
|
||
Verify using `curl -v -x localhost:8899 http://google.com`
|
||
|
||
```
|
||
... [redacted] ...
|
||
< HTTP/1.1 404 NOT FOUND
|
||
< Server: proxy.py v1.0.0
|
||
< Connection: Close
|
||
<
|
||
* Closing connection 0
|
||
```
|
||
|
||
Above `404` response was returned from `proxy.py` web server.
|
||
|
||
Verify the same by inspecting the logs for `proxy.py`.
|
||
Along with the proxy request log, you must also see a http web server request log.
|
||
|
||
```
|
||
2019-09-24 19:09:33,602 - INFO - pid:49996 - access_log:1241 - ::1:49525 - GET /
|
||
2019-09-24 19:09:33,603 - INFO - pid:49995 - access_log:1157 - ::1:49524 - GET localhost:8899/ - 404 NOT FOUND - 70 bytes
|
||
```
|
||
|
||
## FilterByUpstreamHostPlugin
|
||
|
||
Drops traffic by inspecting upstream host.
|
||
By default, plugin drops traffic for `google.com` and `www.google.com`.
|
||
|
||
Start `proxy.py` as:
|
||
|
||
```
|
||
$ proxy \
|
||
--plugins plugin_examples/filter_by_upstream.FilterByUpstreamHostPlugin
|
||
```
|
||
|
||
Verify using `curl -v -x localhost:8899 http://google.com`:
|
||
|
||
```
|
||
... [redacted] ...
|
||
< HTTP/1.1 418 I'm a tea pot
|
||
< Proxy-agent: proxy.py v1.0.0
|
||
* no chunk, no close, no size. Assume close to signal end
|
||
<
|
||
* Closing connection 0
|
||
```
|
||
|
||
Above `418 I'm a tea pot` is sent by our plugin.
|
||
|
||
Verify the same by inspecting logs for `proxy.py`:
|
||
|
||
```
|
||
2019-09-24 19:21:37,893 - ERROR - pid:50074 - handle_readables:1347 - HttpProtocolException type raised
|
||
Traceback (most recent call last):
|
||
... [redacted] ...
|
||
2019-09-24 19:21:37,897 - INFO - pid:50074 - access_log:1157 - ::1:49911 - GET None:None/ - None None - 0 bytes
|
||
```
|
||
|
||
## CacheResponsesPlugin
|
||
|
||
Caches Upstream Server Responses.
|
||
|
||
Start `proxy.py` as:
|
||
|
||
```
|
||
$ proxy \
|
||
--plugins plugin_examples/cache_responses.CacheResponsesPlugin
|
||
```
|
||
|
||
Verify using `curl -v -x localhost:8899 http://httpbin.org/get`:
|
||
|
||
```
|
||
... [redacted] ...
|
||
< HTTP/1.1 200 OK
|
||
< Access-Control-Allow-Credentials: true
|
||
< Access-Control-Allow-Origin: *
|
||
< Content-Type: application/json
|
||
< Date: Wed, 25 Sep 2019 02:24:25 GMT
|
||
< Referrer-Policy: no-referrer-when-downgrade
|
||
< Server: nginx
|
||
< X-Content-Type-Options: nosniff
|
||
< X-Frame-Options: DENY
|
||
< X-XSS-Protection: 1; mode=block
|
||
< Content-Length: 202
|
||
< Connection: keep-alive
|
||
<
|
||
{
|
||
"args": {},
|
||
"headers": {
|
||
"Accept": "*/*",
|
||
"Host": "httpbin.org",
|
||
"User-Agent": "curl/7.54.0"
|
||
},
|
||
"origin": "1.2.3.4, 5.6.7.8",
|
||
"url": "https://httpbin.org/get"
|
||
}
|
||
* Connection #0 to host localhost left intact
|
||
```
|
||
|
||
Get path to the cache file from `proxy.py` logs:
|
||
|
||
```
|
||
... [redacted] ... - GET httpbin.org:80/get - 200 OK - 556 bytes
|
||
... [redacted] ... - Cached response at /var/folders/k9/x93q0_xn1ls9zy76m2mf2k_00000gn/T/httpbin.org-1569378301.407512.txt
|
||
```
|
||
|
||
Verify contents of the cache file `cat /path/to/your/cache/httpbin.org.txt`
|
||
|
||
```
|
||
HTTP/1.1 200 OK
|
||
Access-Control-Allow-Credentials: true
|
||
Access-Control-Allow-Origin: *
|
||
Content-Type: application/json
|
||
Date: Wed, 25 Sep 2019 02:24:25 GMT
|
||
Referrer-Policy: no-referrer-when-downgrade
|
||
Server: nginx
|
||
X-Content-Type-Options: nosniff
|
||
X-Frame-Options: DENY
|
||
X-XSS-Protection: 1; mode=block
|
||
Content-Length: 202
|
||
Connection: keep-alive
|
||
|
||
{
|
||
"args": {},
|
||
"headers": {
|
||
"Accept": "*/*",
|
||
"Host": "httpbin.org",
|
||
"User-Agent": "curl/7.54.0"
|
||
},
|
||
"origin": "1.2.3.4, 5.6.7.8",
|
||
"url": "https://httpbin.org/get"
|
||
}
|
||
```
|
||
|
||
## ManInTheMiddlePlugin
|
||
|
||
Modifies upstream server responses.
|
||
|
||
Start `proxy.py` as:
|
||
|
||
```
|
||
$ proxy \
|
||
--plugins plugin_examples/man_in_the_middle.ManInTheMiddlePlugin
|
||
```
|
||
|
||
Verify using `curl -v -x localhost:8899 http://google.com`:
|
||
|
||
```
|
||
... [redacted] ...
|
||
< HTTP/1.1 200 OK
|
||
< Content-Length: 28
|
||
<
|
||
* Connection #0 to host localhost left intact
|
||
Hello from man in the middle
|
||
```
|
||
|
||
Response body `Hello from man in the middle` is sent by our plugin.
|
||
|
||
## Plugin Ordering
|
||
|
||
When using multiple plugins, depending upon plugin functionality,
|
||
it might be worth considering the order in which plugins are passed
|
||
on the command line.
|
||
|
||
Plugins are called in the same order as they are passed. Example,
|
||
say we are using both `FilterByUpstreamHostPlugin` and
|
||
`RedirectToCustomServerPlugin`. Idea is to drop all incoming `http`
|
||
requests for `google.com` and `www.google.com` and redirect other
|
||
`http` requests to our inbuilt web server.
|
||
|
||
Hence, in this scenario it is important to use
|
||
`FilterByUpstreamHostPlugin` before `RedirectToCustomServerPlugin`.
|
||
If we enable `RedirectToCustomServerPlugin` before `FilterByUpstreamHostPlugin`,
|
||
`google` requests will also get redirected to inbuilt web server,
|
||
instead of being dropped.
|
||
|
||
End-to-End Encryption
|
||
=====================
|
||
|
||
By default, `proxy.py` uses `http` protocol for communication with clients e.g. `curl`, `browser`.
|
||
For enabling end-to-end encrypting using `tls` / `https` first generate certificates:
|
||
|
||
```
|
||
make https-certificates
|
||
```
|
||
|
||
Start `proxy.py` as:
|
||
|
||
```
|
||
$ proxy \
|
||
--cert-file https-cert.pem \
|
||
--key-file https-key.pem
|
||
```
|
||
|
||
Verify using `curl -x https://localhost:8899 --proxy-cacert https-cert.pem https://httpbin.org/get`:
|
||
|
||
```
|
||
{
|
||
"args": {},
|
||
"headers": {
|
||
"Accept": "*/*",
|
||
"Host": "httpbin.org",
|
||
"User-Agent": "curl/7.54.0"
|
||
},
|
||
"origin": "1.2.3.4, 5.6.7.8",
|
||
"url": "https://httpbin.org/get"
|
||
}
|
||
```
|
||
|
||
TLS Interception
|
||
=================
|
||
|
||
By default, `proxy.py` will not decrypt `https` traffic between client and server.
|
||
To enable TLS interception first generate CA certificates:
|
||
|
||
```
|
||
make ca-certificates
|
||
```
|
||
|
||
Lets also enable `CacheResponsePlugin` so that we can verify decrypted
|
||
response from the server. Start `proxy.py` as:
|
||
|
||
```
|
||
$ proxy \
|
||
--plugins plugin_examples/cache_responses.CacheResponsesPlugin \
|
||
--ca-key-file ca-key.pem \
|
||
--ca-cert-file ca-cert.pem \
|
||
--ca-signing-key-file ca-signing-key.pem
|
||
```
|
||
|
||
Verify using `curl -v -x localhost:8899 --cacert ca-cert.pem https://httpbin.org/get`
|
||
|
||
```
|
||
* issuer: C=US; ST=CA; L=SanFrancisco; O=proxy.py; OU=CA; CN=Proxy PY CA; emailAddress=proxyca@mailserver.com
|
||
* SSL certificate verify ok.
|
||
> GET /get HTTP/1.1
|
||
... [redacted] ...
|
||
< Connection: keep-alive
|
||
<
|
||
{
|
||
"args": {},
|
||
"headers": {
|
||
"Accept": "*/*",
|
||
"Host": "httpbin.org",
|
||
"User-Agent": "curl/7.54.0"
|
||
},
|
||
"origin": "1.2.3.4, 5.6.7.8",
|
||
"url": "https://httpbin.org/get"
|
||
}
|
||
```
|
||
|
||
The `issuer` line confirms that response was intercepted.
|
||
|
||
Also verify the contents of cached response file. Get path to the cache
|
||
file from `proxy.py` logs.
|
||
|
||
`$ cat /path/to/your/tmp/directory/httpbin.org-1569452863.924174.txt`
|
||
|
||
```
|
||
HTTP/1.1 200 OK
|
||
Access-Control-Allow-Credentials: true
|
||
Access-Control-Allow-Origin: *
|
||
Content-Type: application/json
|
||
Date: Wed, 25 Sep 2019 23:07:05 GMT
|
||
Referrer-Policy: no-referrer-when-downgrade
|
||
Server: nginx
|
||
X-Content-Type-Options: nosniff
|
||
X-Frame-Options: DENY
|
||
X-XSS-Protection: 1; mode=block
|
||
Content-Length: 202
|
||
Connection: keep-alive
|
||
|
||
{
|
||
"args": {},
|
||
"headers": {
|
||
"Accept": "*/*",
|
||
"Host": "httpbin.org",
|
||
"User-Agent": "curl/7.54.0"
|
||
},
|
||
"origin": "1.2.3.4, 5.6.7.8",
|
||
"url": "https://httpbin.org/get"
|
||
}
|
||
```
|
||
|
||
Viola!!! If you remove CA flags, encrypted data will be found in the
|
||
cached file instead of plain text.
|
||
|
||
Now use CA flags with other
|
||
[plugin examples](#plugin-examples) to see them work with `https` traffic.
|
||
|
||
Embed proxy.py
|
||
==============
|
||
|
||
To start `proxy.py` in embedded mode:
|
||
|
||
```
|
||
from proxy.main import main
|
||
|
||
if __name__ == '__main__':
|
||
main([])
|
||
```
|
||
|
||
or, to start with arguments:
|
||
|
||
```
|
||
from proxy.main import main
|
||
|
||
if __name__ == '__main__':
|
||
main([
|
||
'--hostname', '::1',
|
||
'--port', '8899'
|
||
])
|
||
```
|
||
|
||
Plugin Developer and Contributor Guide
|
||
======================================
|
||
|
||
Contributors must start `proxy.py` from source to verify and develop new features / fixes.
|
||
|
||
See [Run proxy.py from command line using repo source](#from-command-line-using-repo-source) for details.
|
||
|
||
## Everything is a plugin
|
||
|
||
As you might have guessed by now, in `proxy.py` everything is a plugin.
|
||
|
||
- We enabled proxy server plugins using `--plugins` flag.
|
||
All the [plugin examples](#plugin-examples) were implementing
|
||
`HttpProxyBasePlugin`. See documentation of
|
||
[HttpProxyBasePlugin](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L894-L938)
|
||
for available lifecycle hooks. Use `HttpProxyBasePlugin` to modify
|
||
behavior of http(s) proxy protocol between client and upstream server.
|
||
Example, [FilterByUpstreamHostPlugin](#filterbyupstreamhostplugin).
|
||
|
||
- We also enabled inbuilt web server using `--enable-web-server`.
|
||
Inbuilt web server implements `HttpProtocolHandlerPlugin` plugin.
|
||
See documentation of [HttpProtocolHandlerPlugin](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L793-L850)
|
||
for available lifecycle hooks. Use `HttpProtocolHandlerPlugin` to add
|
||
new features for http(s) clients. Example,
|
||
[HttpWebServerPlugin](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L1185-L1260).
|
||
|
||
- There also is a `--disable-http-proxy` flag. It disables inbuilt proxy server.
|
||
Use this flag with `--enable-web-server` flag to run `proxy.py` as a programmable
|
||
http(s) server. [HttpProxyPlugin](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L941-L1182)
|
||
also implements `HttpProtocolHandlerPlugin`.
|
||
|
||
## Internal Architecture
|
||
|
||
- [HttpProtocolHandler](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L1263-L1440)
|
||
thread is started with the accepted [TcpClientConnection](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L230-L237).
|
||
`HttpProtocolHandler` is responsible for parsing incoming client request and invoking
|
||
`HttpProtocolHandlerPlugin` lifecycle hooks.
|
||
|
||
- `HttpProxyPlugin` which implements `HttpProtocolHandlerPlugin` also has its own plugin
|
||
mechanism. Its responsibility is to establish connection between client and
|
||
upstream [TcpServerConnection](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L204-L227)
|
||
and invoke `HttpProxyBasePlugin` lifecycle hooks.
|
||
|
||
- `HttpProtocolHandler` threads are started by [Acceptor](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L424-L472)
|
||
processes.
|
||
|
||
- `--num-workers` `Acceptor` processes are started by
|
||
[AcceptorPool](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L368-L421)
|
||
on start-up.
|
||
|
||
- `AcceptorPool` listens on server socket and pass the handler to `Acceptor` processes.
|
||
Workers are responsible for accepting new client connections and starting
|
||
`HttpProtocolHandler` thread.
|
||
|
||
## Sending a Pull Request
|
||
|
||
Install dependencies for local development testing:
|
||
|
||
`$ pip install -r requirements-testing.txt`
|
||
|
||
Every pull request goes through set of tests which must pass:
|
||
|
||
- `mypy`: Run `make lint` locally for compliance check.
|
||
Fix all warnings and errors before sending out a PR.
|
||
|
||
- `coverage`: Run `make coverage` locally for coverage report.
|
||
Its ideal to add tests for any critical change. Depending upon
|
||
the change, it's ok if test coverage falls by `<0.5%`.
|
||
|
||
- `formatting`: Run `make autopep8` locally to format the code in-place.
|
||
`autopep8` is run with `--aggresive` flag. Sometimes it _may_ result in
|
||
weird formatting. But let's stick to one consistent formatting tool.
|
||
I am open to flag changes for `autopep8`.
|
||
|
||
## Utilities
|
||
|
||
## TCP Sockets
|
||
|
||
#### new_socket_connection
|
||
|
||
Attempts to create an IPv4 connection, then IPv6 and
|
||
finally a dual stack connection to provided address.
|
||
|
||
```
|
||
>>> conn = new_socket_connection(('httpbin.org', 80))
|
||
>>> ...[ use connection ]...
|
||
>>> conn.close()
|
||
```
|
||
|
||
#### socket_connection
|
||
|
||
`socket_connection` is a convenient decorator + context manager
|
||
around `new_socket_connection` which ensures `conn.close` is implicit.
|
||
|
||
As a context manager:
|
||
|
||
```
|
||
>>> with socket_connection(('httpbin.org', 80)) as conn:
|
||
>>> ... [ use connection ] ...
|
||
```
|
||
|
||
As a decorator:
|
||
|
||
```
|
||
>>> @socket_connection(('httpbin.org', 80))
|
||
>>> def my_api_call(conn, *args, **kwargs):
|
||
>>> ... [ use connection ] ...
|
||
```
|
||
|
||
## Http Client
|
||
|
||
#### build_http_request
|
||
|
||
##### Generate HTTP GET request
|
||
|
||
```
|
||
>>> build_http_request(b'GET', b'/')
|
||
b'GET / HTTP/1.1\r\n\r\n'
|
||
>>>
|
||
```
|
||
|
||
##### Generate HTTP GET request with headers
|
||
|
||
```
|
||
>>> build_http_request(b'GET', b'/',
|
||
headers={b'Connection': b'close'})
|
||
b'GET / HTTP/1.1\r\nConnection: close\r\n\r\n'
|
||
>>>
|
||
```
|
||
|
||
##### Generate HTTP POST request with headers and body
|
||
|
||
```
|
||
>>> import json
|
||
>>> build_http_request(b'POST', b'/form',
|
||
headers={b'Content-type': b'application/json'},
|
||
body=proxy.bytes_(json.dumps({'email': 'hello@world.com'})))
|
||
b'POST /form HTTP/1.1\r\nContent-type: application/json\r\n\r\n{"email": "hello@world.com"}'
|
||
```
|
||
|
||
#### build_http_response
|
||
|
||
TODO
|
||
|
||
## Websocket
|
||
|
||
#### WebsocketFrame
|
||
|
||
TODO
|
||
|
||
#### WebsocketClient
|
||
|
||
TODO
|
||
|
||
## Internal Documentation
|
||
|
||
Browse through internal class hierarchy and documentation using `pydoc3`.
|
||
Example:
|
||
|
||
```
|
||
$ pydoc3 proxy
|
||
|
||
PACKAGE CONTENTS
|
||
__main__
|
||
common (package)
|
||
core (package)
|
||
http (package)
|
||
main
|
||
|
||
FILE
|
||
/Users/abhinav/Dev/proxy.py/proxy/__init__.py
|
||
```
|
||
|
||
Frequently Asked Questions
|
||
==========================
|
||
|
||
## SyntaxError: invalid syntax
|
||
|
||
Make sure you are using `Python 3`. Verify the version before running `proxy.py`:
|
||
|
||
`$ python --version`
|
||
|
||
## Unable to connect with proxy.py from remote host
|
||
|
||
Make sure `proxy.py` is listening on correct network interface.
|
||
Try following flags:
|
||
|
||
- For IPv6 `--hostname ::`
|
||
- For IPv4 `--hostname 0.0.0.0`
|
||
|
||
## Basic auth not working with a browser
|
||
|
||
Most likely it's a browser integration issue with system keychain.
|
||
|
||
- First verify that basic auth is working using `curl`
|
||
|
||
`curl -v -x username:password@localhost:8899 https://httpbin.org/get`
|
||
|
||
- See [this thread](https://github.com/abhinavsingh/proxy.py/issues/89#issuecomment-534845710)
|
||
for further details.
|
||
|
||
## Docker image not working on macOS
|
||
|
||
It's a compatibility issue with `vpnkit`.
|
||
|
||
See [moby/vpnkit exhausts docker resources](https://github.com/abhinavsingh/proxy.py/issues/43)
|
||
and [Connection refused: The proxy could not connect](https://github.com/moby/vpnkit/issues/469)
|
||
for some background.
|
||
|
||
## Unable to load custom plugins
|
||
|
||
Make sure your plugin modules are discoverable by adding them to `PYTHONPATH`. Example:
|
||
|
||
`PYTHONPATH=/path/to/my/app proxy --plugins my_app.proxyPlugin`
|
||
|
||
```
|
||
...[redacted]... - Loaded plugin proxy.HttpProxyPlugin
|
||
...[redacted]... - Loaded plugin my_app.proxyPlugin
|
||
```
|
||
|
||
or, make sure to pass fully-qualified path as parameter, e.g.
|
||
|
||
`proxy --plugins /path/to/my/app/my_app.proxyPlugin`
|
||
|
||
## GCE log viewer integration for proxy.py
|
||
|
||
A starter [fluentd.conf](https://github.com/abhinavsingh/proxy.py/blob/develop/fluentd.conf)
|
||
template is available.
|
||
|
||
1. Copy this configuration file as `proxy.py.conf` under
|
||
`/etc/google-fluentd/config.d/`
|
||
|
||
2. Update `path` field to log file path as used with `--log-file` flag.
|
||
By default `/tmp/proxy.log` path is tailed.
|
||
|
||
3. Reload `google-fluentd`:
|
||
|
||
`sudo service google-fluentd restart`
|
||
|
||
Now `proxy.py` logs can be browsed using
|
||
[GCE log viewer](https://console.cloud.google.com/logs/viewer).
|
||
|
||
## ValueError: filedescriptor out of range in select
|
||
|
||
`proxy.py` is made to handle thousands of connections per second
|
||
without any socket leaks.
|
||
|
||
1. Make use of `--open-file-limit` flag to customize `ulimit -n`.
|
||
- To set a value upper than the hard limit, run as root.
|
||
2. Make sure to adjust `--backlog` flag for higher concurrency.
|
||
|
||
If nothing helps, [open an issue](https://github.com/abhinavsingh/proxy.py/issues/new)
|
||
with `requests per second` sent and output of following debug script:
|
||
|
||
```
|
||
$ ./helper/monitor_open_files.sh <proxy-py-pid>
|
||
```
|
||
|
||
Flags
|
||
=====
|
||
|
||
```
|
||
❯ proxy -h
|
||
usage: proxy [-h] [--backlog BACKLOG] [--basic-auth BASIC_AUTH]
|
||
[--ca-key-file CA_KEY_FILE] [--ca-cert-dir CA_CERT_DIR]
|
||
[--ca-cert-file CA_CERT_FILE]
|
||
[--ca-signing-key-file CA_SIGNING_KEY_FILE]
|
||
[--cert-file CERT_FILE]
|
||
[--client-recvbuf-size CLIENT_RECVBUF_SIZE]
|
||
[--devtools-ws-path DEVTOOLS_WS_PATH]
|
||
[--disable-headers DISABLE_HEADERS] [--disable-http-proxy]
|
||
[--enable-devtools] [--enable-events] [--enable-static-server]
|
||
[--enable-web-server] [--hostname HOSTNAME] [--key-file KEY_FILE]
|
||
[--log-level LOG_LEVEL] [--log-file LOG_FILE]
|
||
[--log-format LOG_FORMAT] [--num-workers NUM_WORKERS]
|
||
[--open-file-limit OPEN_FILE_LIMIT] [--pac-file PAC_FILE]
|
||
[--pac-file-url-path PAC_FILE_URL_PATH] [--pid-file PID_FILE]
|
||
[--plugins PLUGINS] [--port PORT]
|
||
[--server-recvbuf-size SERVER_RECVBUF_SIZE]
|
||
[--static-server-dir STATIC_SERVER_DIR] [--threadless]
|
||
[--timeout TIMEOUT] [--version]
|
||
|
||
proxy.py v2.0.0
|
||
|
||
optional arguments:
|
||
-h, --help show this help message and exit
|
||
--backlog BACKLOG Default: 100. Maximum number of pending connections to
|
||
proxy server
|
||
--basic-auth BASIC_AUTH
|
||
Default: No authentication. Specify colon separated
|
||
user:password to enable basic authentication.
|
||
--ca-key-file CA_KEY_FILE
|
||
Default: None. CA key to use for signing dynamically
|
||
generated HTTPS certificates. If used, must also pass
|
||
--ca-cert-file and --ca-signing-key-file
|
||
--ca-cert-dir CA_CERT_DIR
|
||
Default: ~/.proxy.py. Directory to store dynamically
|
||
generated certificates. Also see --ca-key-file, --ca-
|
||
cert-file and --ca-signing-key-file
|
||
--ca-cert-file CA_CERT_FILE
|
||
Default: None. Signing certificate to use for signing
|
||
dynamically generated HTTPS certificates. If used,
|
||
must also pass --ca-key-file and --ca-signing-key-file
|
||
--ca-signing-key-file CA_SIGNING_KEY_FILE
|
||
Default: None. CA signing key to use for dynamic
|
||
generation of HTTPS certificates. If used, must also
|
||
pass --ca-key-file and --ca-cert-file
|
||
--cert-file CERT_FILE
|
||
Default: None. Server certificate to enable end-to-end
|
||
TLS encryption with clients. If used, must also pass
|
||
--key-file.
|
||
--client-recvbuf-size CLIENT_RECVBUF_SIZE
|
||
Default: 1 MB. Maximum amount of data received from
|
||
the client in a single recv() operation. Bump this
|
||
value for faster uploads at the expense of increased
|
||
RAM.
|
||
--devtools-ws-path DEVTOOLS_WS_PATH
|
||
Default: /devtools. Only applicable if --enable-
|
||
devtools is used.
|
||
--disable-headers DISABLE_HEADERS
|
||
Default: None. Comma separated list of headers to
|
||
remove before dispatching client request to upstream
|
||
server.
|
||
--disable-http-proxy Default: False. Whether to disable
|
||
proxy.HttpProxyPlugin.
|
||
--enable-devtools Default: False. Enables integration with Chrome
|
||
Devtool Frontend.
|
||
--enable-events Default: False. Enables core to dispatch lifecycle
|
||
events. Plugins can be used to subscribe for core
|
||
events.
|
||
--enable-static-server
|
||
Default: False. Enable inbuilt static file server.
|
||
Optionally, also use --static-server-dir to serve
|
||
static content from custom directory. By default,
|
||
static file server serves from public folder.
|
||
--enable-web-server Default: False. Whether to enable
|
||
proxy.HttpWebServerPlugin.
|
||
--hostname HOSTNAME Default: ::1. Server IP address.
|
||
--key-file KEY_FILE Default: None. Server key file to enable end-to-end
|
||
TLS encryption with clients. If used, must also pass
|
||
--cert-file.
|
||
--log-level LOG_LEVEL
|
||
Valid options: DEBUG, INFO (default), WARNING, ERROR,
|
||
CRITICAL. Both upper and lowercase values are allowed.
|
||
You may also simply use the leading character e.g.
|
||
--log-level d
|
||
--log-file LOG_FILE Default: sys.stdout. Log file destination.
|
||
--log-format LOG_FORMAT
|
||
Log format for Python logger.
|
||
--num-workers NUM_WORKERS
|
||
Defaults to number of CPU cores.
|
||
--open-file-limit OPEN_FILE_LIMIT
|
||
Default: 1024. Maximum number of files (TCP
|
||
connections) that proxy.py can open concurrently.
|
||
--pac-file PAC_FILE A file (Proxy Auto Configuration) or string to serve
|
||
when the server receives a direct file request. Using
|
||
this option enables proxy.HttpWebServerPlugin.
|
||
--pac-file-url-path PAC_FILE_URL_PATH
|
||
Default: /. Web server path to serve the PAC file.
|
||
--pid-file PID_FILE Default: None. Save parent process ID to a file.
|
||
--plugins PLUGINS Comma separated plugins
|
||
--port PORT Default: 8899. Server port.
|
||
--server-recvbuf-size SERVER_RECVBUF_SIZE
|
||
Default: 1 MB. Maximum amount of data received from
|
||
the server in a single recv() operation. Bump this
|
||
value for faster downloads at the expense of increased
|
||
RAM.
|
||
--static-server-dir STATIC_SERVER_DIR
|
||
Default: "public" folder in directory where proxy.py
|
||
is placed. This option is only applicable when static
|
||
server is also enabled. See --enable-static-server.
|
||
--threadless Default: False. When disabled a new thread is spawned
|
||
to handle each client connection.
|
||
--timeout TIMEOUT Default: 10. Number of seconds after which an inactive
|
||
connection must be dropped. Inactivity is defined by
|
||
no data sent or received by the client.
|
||
--version, -v Prints proxy.py version.
|
||
|
||
Proxy.py not working? Report at:
|
||
https://github.com/abhinavsingh/proxy.py/issues/new
|
||
```
|
||
|
||
Changelog
|
||
=========
|
||
|
||
- `v2.x`
|
||
- No longer ~~a single file module~~.
|
||
- Added dashboard app.
|
||
- `v1.x`
|
||
- `Python3` only.
|
||
- Deprecated support for ~~Python 2.x~~.
|
||
- Added support for multi accept.
|
||
- Added plugin support.
|
||
- `v0.x`
|
||
- Single file.
|
||
- Single threaded server.
|
||
|
||
For detailed changelog refer
|