Rooba
/
proxy.py
mirror of https://github.com/abhinavsingh/proxy.py.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add more context in intro (#713) 

* Add more context in intro

* Take another pass

* Add threadless in log section
This commit is contained in:
Abhinav Singh 2021-11-09 03:53:17 +05:30 committed by GitHub
parent fa18129f9b
commit fec9a14303
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 153 additions and 123 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

276
README.md
View File

@ -27,6 +27,7 @@
- [Features](#features)
- [Install](#install)
- [Stable vs Develop](#stable-vs-develop)
- [Using PIP](#using-pip)
- [Stable version](#stable-version-with-pip)
- [Development version](#development-version-with-pip)
@ -119,7 +120,7 @@
- Fast & Scalable
- Scales by using all available cores on the system
- Threadless executions using coroutine
- Threadless executions using asyncio
- Made to handle `tens-of-thousands` connections / sec
```console
@ -146,36 +147,56 @@
- Uses only `~5-20MB` RAM
- No external dependency other than standard Python library
- Programmable
- Optionally enable builtin Web Server
- Customize proxy and http routing via [plugins](https://github.com/abhinavsingh/proxy.py/tree/develop/proxy/plugin)
- Enable plugin using command line option e.g. `--plugins proxy.plugin.CacheResponsesPlugin`
- Plugin API is currently in development phase, expect breaking changes.
- Customize proxy behavior using [Proxy Server Plugins](#http-proxy-plugins). Example:
- `--plugins proxy.plugin.ProxyPoolPlugin`
- Optionally, enable builtin [Web Server Plugins](#http-web-server-plugins). Example:
- `--plugins proxy.plugin.ReverseProxyPlugin`
- Plugin API is currently in development phase, expect breaking changes
- Realtime Dashboard
- Optionally enable bundled dashboard.
- Optionally, enable [proxy.py dashboard](#run-dashboard).
- Available at `http://localhost:8899/dashboard`.
- Inspect, Monitor, Control and Configure `proxy.py` at runtime.
- Extend dashboard using plugins.
- Dashboard is currently in development phase, expect breaking changes.
- [Inspect, Monitor, Control and Configure](#inspect-traffic) `proxy.py` at runtime
- [Chrome DevTools Protocol](#chrome-devtools-protocol) support
- Extend dashboard using plugins
- Dashboard is currently in development phase, expect breaking changes
- Secure
- Enable end-to-end encryption between clients and `proxy.py` using TLS
- Enable end-to-end encryption between clients and `proxy.py`
- See [End-to-End Encryption](#end-to-end-encryption)
- Private
- Everyone deserves privacy. Browse with malware and adult content protection
- See [DNS-over-HTTPS](#cloudflarednsresolverplugin)
- Man-In-The-Middle
- Can decrypt TLS traffic between clients and upstream servers
- See [TLS Interception](#tls-interception)
- Supported proxy protocols
- `http(s)`
- `http1`
- `http1.1` pipeline
- `http1.1` with pipeline
- `http2`
- `websockets`
- Static file server support
- See `--enable-static-server` and `--static-server-dir` flags
- Optimized for large file uploads and downloads
- IPv4 and IPv6 support
- See `--client-recvbuf-size` and `--server-recvbuf-size` flag
- `IPv4` and `IPv6` support
- See `--hostname` flag
- Unix domain socket support
- See `--unix-socket-path` flag
- Basic authentication support
- Can serve a [PAC (Proxy Auto-configuration)](https://en.wikipedia.org/wiki/Proxy_auto-config) file
- See `--basic-auth` flag
- PAC (Proxy Auto-configuration) support
- See `--pac-file` and `--pac-file-url-path` flags
# Install
## Stable vs Develop
`master` branch contains latest stable code and is available via `PyPi` repository
`develop` branch contains cutting edge changes
Development branch is kept stable *(most of the times)*. But if you want 100% reliability and serving users in production environment, always use stable version from `PyPi` or `Docker` container from `hub.docker.com`.
## Using PIP
### Stable Version with PIP
@ -200,18 +221,17 @@ or from GitHub `master` branch
## Using Docker
#### Stable Version from Docker Hub
### Stable Version from Docker Hub
```console
❯ docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:latest
```
#### Build Development Version Locally
### Build Development Version Locally
```console
❯ git clone https://github.com/abhinavsingh/proxy.py.git
❯ cd proxy.py
❯ make container
❯ cd proxy.py && make container
❯ docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:latest
```
@ -239,38 +259,41 @@ or from GitHub `master` branch
When `proxy.py` is installed using `pip`,
an executable named `proxy` is placed under your `$PATH`.
#### Run it
### Run it
Simply type `proxy` on command line to start it with default configuration.
Simply type `proxy` on command line to start with default configuration.
```console
❯ proxy
...[redacted]... - Loaded plugin proxy.http_proxy.HttpProxyPlugin
...[redacted]... - Starting 8 workers
...[redacted]... - Starting 8 threadless workers
...[redacted]... - Started server on ::1:8899
```
#### Understanding logs
### Understanding logs
Things to notice from above logs:
- `Loaded plugin` - `proxy.py` will load `proxy.http.proxy.HttpProxyPlugin` by default.
As name suggests, this core plugin adds `http(s)` proxy server capabilities to `proxy.py`
- `Loaded plugin`
- `proxy.py` will load `proxy.http.proxy.HttpProxyPlugin` by default
- As name suggests, this core plugin adds `http(s)` proxy server capabilities to `proxy.py` instance
- `Started N workers` - Use `--num-workers` flag to customize number of worker processes.
By default, `proxy.py` will start as many workers as there are CPU cores on the machine.
- `Started N threadless workers`
- By default, `proxy.py` will start as many workers as there are CPU cores on the machine
- Use `--num-workers` flag to customize number of worker processes
- See [Threads vs Threadless](#threads-vs-threadless) to understand how to control execution mode
- `Started server on ::1:8899` - By default, `proxy.py` listens on IPv6 `::1`,
which is equivalent of IPv4 `127.0.0.1`. If you want to access `proxy.py` externally,
use `--hostname ::` or `--hostname 0.0.0.0` or bind to any other interface available
on your machine. See [CustomNetworkInterface](#customnetworkinterface) for how to customize
your system public IP as seen by the upstream servers.
- `Started server on ::1:8899`
- By default, `proxy.py` listens on IPv6 `::1`, which is equivalent of IPv4 `127.0.0.1`
- If you want to access `proxy.py` from external host, use `--hostname ::` or `--hostname 0.0.0.0` or bind to any other interface available on your machine.
- See [CustomNetworkInterface](#customnetworkinterface) for how to customize `proxy.py` *public IP seen by upstream servers*.
- `Port 8899` - Use `--port` flag to customize default TCP port.
- `Port 8899`
- Use `--port` flag to customize default TCP port.
#### Enable DEBUG logging
### Enable DEBUG logging
All the logs above are `INFO` level logs, default `--log-level` for `proxy.py`.
All the logs above are `INFO` level logs, default `--log-level` for `proxy.py`
Lets start `proxy.py` with `DEBUG` level logging:
@ -282,11 +305,18 @@ Lets start `proxy.py` with `DEBUG` level logging:
...[redacted]... - Started server on ::1:8899
```
As we can see, before starting up:
You can use single letter to customize log level. Example:
- `d = DEBUG`
- `i = INFO`
- `w = WARNING`
- `e = ERROR`
- `c = CRITICAL`
- `proxy.py` also tried to set open file limit `ulimit` on the system.
- Default value for `--open-file-limit` used is `1024`.
- `--open-file-limit` flag is a no-op on `Windows` operating systems.
As we can see from the above logs, before starting up:
- `proxy.py` tried to set open file limit `ulimit` on the system
- Default value for `--open-file-limit` used is `1024`
- `--open-file-limit` flag is a no-op on `Windows` operating systems
See [flags](#flags) for full list of available configuration options.
@ -323,25 +353,24 @@ To start `proxy.py` from source code follow these instructions:
❯ make
```
- Run proxy.py
- Run `proxy.py`
```console
❯ python -m proxy
```
Also see [Plugin Developer and Contributor Guide](#plugin-developer-and-contributor-guide)
See [Plugin Developer and Contributor Guide](#plugin-developer-and-contributor-guide)
if you plan to work with `proxy.py` source code.
## Docker image
#### Customize startup flags
### Customize startup flags
By default `docker` binary is started with IPv4 networking flags:
--hostname 0.0.0.0 --port 8899
To override input flags, start docker image as follows.
For example, to check `proxy.py` version within Docker image:
You can override flag from command line when starting the docker container. For example, to check `proxy.py` version within the docker container, run:
❯ docker run -it \
-p 8899:8899 \
@ -1380,32 +1409,30 @@ As a decorator:
### build_http_request
#### Generate HTTP GET request
- Generate HTTP GET request
```python
>>> build_http_request(b'GET', b'/')
b'GET / HTTP/1.1\r\n\r\n'
>>>
```
```python
>>> build_http_request(b'GET', b'/')
b'GET / HTTP/1.1\r\n\r\n'
```
#### Generate HTTP GET request with headers
- Generate HTTP GET request with headers
```python
>>> build_http_request(b'GET', b'/',
headers={b'Connection': b'close'})
b'GET / HTTP/1.1\r\nConnection: close\r\n\r\n'
>>>
```
```python
>>> build_http_request(b'GET', b'/',
headers={b'Connection': b'close'})
b'GET / HTTP/1.1\r\nConnection: close\r\n\r\n'
```
#### Generate HTTP POST request with headers and body
- Generate HTTP POST request with headers and body
```python
>>> import json
>>> build_http_request(b'POST', b'/form',
headers={b'Content-type': b'application/json'},
body=proxy.bytes_(json.dumps({'email': 'hello@world.com'})))
b'POST /form HTTP/1.1\r\nContent-type: application/json\r\n\r\n{"email": "hello@world.com"}'
```
```python
>>> import json
>>> build_http_request(b'POST', b'/form',
headers={b'Content-type': b'application/json'},
body=proxy.bytes_(json.dumps({'email': 'hello@world.com'})))
b'POST /form HTTP/1.1\r\nContent-type: application/json\r\n\r\n{"email": "hello@world.com"}'
```
### build_http_response
@ -1422,66 +1449,66 @@ build_http_response(
### API Usage
#### gen_private_key
- gen_private_key
```python
gen_private_key(
key_path: str,
password: str,
bits: int = 2048,
timeout: int = 10) -> bool
```
```python
gen_private_key(
key_path: str,
password: str,
bits: int = 2048,
timeout: int = 10) -> bool
```
#### gen_public_key
- gen_public_key
```python
gen_public_key(
public_key_path: str,
private_key_path: str,
private_key_password: str,
subject: str,
alt_subj_names: Optional[List[str]] = None,
extended_key_usage: Optional[str] = None,
validity_in_days: int = 365,
timeout: int = 10) -> bool
```
```python
gen_public_key(
public_key_path: str,
private_key_path: str,
private_key_password: str,
subject: str,
alt_subj_names: Optional[List[str]] = None,
extended_key_usage: Optional[str] = None,
validity_in_days: int = 365,
timeout: int = 10) -> bool
```
#### remove_passphrase
- remove_passphrase
```python
remove_passphrase(
key_in_path: str,
password: str,
key_out_path: str,
timeout: int = 10) -> bool
```
```python
remove_passphrase(
key_in_path: str,
password: str,
key_out_path: str,
timeout: int = 10) -> bool
```
#### gen_csr
- gen_csr
```python
gen_csr(
csr_path: str,
key_path: str,
password: str,
crt_path: str,
timeout: int = 10) -> bool
```
```python
gen_csr(
csr_path: str,
key_path: str,
password: str,
crt_path: str,
timeout: int = 10) -> bool
```
#### sign_csr
- sign_csr
```python
sign_csr(
csr_path: str,
crt_path: str,
ca_key_path: str,
ca_key_password: str,
ca_crt_path: str,
serial: str,
alt_subj_names: Optional[List[str]] = None,
extended_key_usage: Optional[str] = None,
validity_in_days: int = 365,
timeout: int = 10) -> bool
```
```python
sign_csr(
csr_path: str,
crt_path: str,
ca_key_path: str,
ca_key_password: str,
ca_crt_path: str,
serial: str,
alt_subj_names: Optional[List[str]] = None,
extended_key_usage: Optional[str] = None,
validity_in_days: int = 365,
timeout: int = 10) -> bool
```
See [pki.py](https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/common/pki.py) and
[test_pki.py](https://github.com/abhinavsingh/proxy.py/blob/develop/tests/common/test_pki.py)
@ -1520,8 +1547,7 @@ optional arguments:
## Internal Documentation
Browse through internal class hierarchy and documentation using `pydoc3`.
Example:
Code is well documented. Browse through internal class hierarchy and documentation using `pydoc3`
```console
❯ pydoc3 proxy
@ -1539,8 +1565,6 @@ FILE
# Run Dashboard
**This is a WIP and may not work as documented**
Dashboard is currently under development and not yet bundled with `pip` packages.
To run dashboard, you must checkout the source.
@ -1579,6 +1603,8 @@ Visit dashboard:
## Inspect Traffic
***This is a WIP and may not work as documented***
Wait for embedded `Chrome Dev Console` to load. Currently, detail about all traffic flowing
through `proxy.py` is pushed to the `Inspect Traffic` tab. However, received payloads are not
yet integrated with the embedded dev console.
@ -1603,19 +1629,23 @@ Now point your CDT instance to `ws://localhost:8899/devtools`.
## Threads vs Threadless
### Pre v2.x
### `v1.x`
`proxy.py` used to spawn new threads for handling client requests.
### Starting v2.0
### `v2.0+`
`proxy.py` added support for threadless execution of client requests using `asyncio`.
### Starting v2.4.0
### `v2.4.0+`
Threadless execution was turned ON by default for `Python 3.8+` on `mac` and `linux` environments. `proxy.py` threadless execution has been reported safe on these environments by our users. If you are running into trouble, fallback to threaded mode using `--threaded` flag.
Threadless execution was turned ON by default for `Python 3.8+` on `mac` and `linux` environments.
For `windows` and `Python < 3.8`, you can still try out threadless mode by starting `proxy.py` with `--threadless` flag. If threadless works for you, consider sending a PR by editing `_env_threadless_compliant` method in the `proxy/common/constants.py` file.
`proxy.py` threadless execution has been reported safe on these environments by our users. If you are running into trouble, fallback to threaded mode using `--threaded` flag.
For `windows` and `Python < 3.8`, you can still try out threadless mode by starting `proxy.py` with `--threadless` flag.
If threadless works for you, consider sending a PR by editing `_env_threadless_compliant` method in the `proxy/common/constants.py` file.
## SyntaxError: invalid syntax