proxy.py/examples
Abhinav Singh 44d72431e3
Async `get_events`, `handle_event`, `handle_readables`, `handle_writables` (#769)
* Asynchronous `handle_event` and `LocalExecutor` thread

* Bail out on first task completion

* mypy

* Add `helper/benchmark.sh` and fix threaded which must now use asyncio (reduced performance of threaded)

* Print open file diff from `benchmark.sh`

* Add `--local-executor` flag, disabled by default for now until tests are updated

* Async `handle_readables` and `handle_writables` for `HttpProtocolHandlerPlugin` interface (doesnt impact proxy/web plugins for now)

* Async `get_events`

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Address tests after async changes

* mypy and flake8

* spelldoc

* `check.py` and trailing comma

* Rename to `_assertions.py`

* Add missing `pytest-mock` and `pytest-asyncio` deps

* Add `pytest-mock` to `pylint` deps

* Correct use of `parameterize` and add `PT007` to flake8 ignores

* Fix mypy hints broken for `< Python3.9`

* Remove usage of `asynccontextmanager` which is not available for all Python versions that `proxy.py` supports

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix for pre-python-3.9 versions

* `AsyncTask` apis `set_name` and `get_name` are not available on all supported versions

* Install setuptools via `lib-dep` until we recommend editable install

* Deprecate support for `Python 3.6`

* Use recommendation suggested here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753840929

* Address recommendation here https://github.com/abhinavsingh/proxy.py/pull/769\#discussion_r753841906

* Make `Threadless` agnostic of `multiprocessing.Process`

* Acceptors must dispatch to local executor in non-blocking fashion

* No daemon for executor processes and fix shutdown logic

* Only return fds from `_selected_events` not all events data

* Refactor logic

* Prefix private methods with `_`

* `work_queue` and not `client_queue`

* Turn `Threadless` into an abstract executor. Introduce `RemoteExecutor`

* Make `LocalExecutor` agnostic of `threading.Thread`

* `LocalExecutor` now implements `Threadless`

* `get_events` and `get_descriptors` now must return int and not sock.  `Threadless` now avoids repeated register/unregister and instead make use of `selectors.modify`

* Fix `main` tests

* Apply suggestions from code review

Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>

* Apply code review recommendations manually

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert back `Any` and use `addr or None`

* Address `flake8`

* Update tests to use `fileno`

* Fix doc build

* Fix doc spell, use tear down and not teardown

* Doc updates

* Add back support for `Python 3.6`

* Acceptors dont need loop initialization

* On Python 3.6 `asyncio.new_event_loop()` is necessary

* Make doc happy

* `--threaded` needs a new event loop for 3.7 too

* Always use `asyncio.new_event_loop()` for threaded mode

Added e2e integration tests (subprocess & curl) for all modes.

* Lint fixes

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sviatoslav Sydorenko <wk@sydorenko.org.ua>
2021-11-23 15:02:00 +05:30
..
README.md Pool (#694) 2021-11-07 21:06:41 +05:30
https_connect_tunnel.py No abstract method for proxy plugin (#738) 2021-11-15 03:17:12 +05:30
pubsub_eventing.py Add `Listener`, Web server close on header, use `Pipe` instead of `Manager` in eventing core (#720) 2021-11-10 20:47:28 +05:30
ssl_echo_client.py Add a `--unix-socket-path` flag (#697) 2021-11-08 01:11:46 +05:30
ssl_echo_server.py Add `--num-acceptors` flag + Allow `work_klass` via `Proxy` context manager kwargs (#714) 2021-11-10 04:27:40 +05:30
tcp_echo_client.py Add a `--unix-socket-path` flag (#697) 2021-11-08 01:11:46 +05:30
tcp_echo_server.py Add `--num-acceptors` flag + Allow `work_klass` via `Proxy` context manager kwargs (#714) 2021-11-10 04:27:40 +05:30
web_scraper.py Async `get_events`, `handle_event`, `handle_readables`, `handle_writables` (#769) 2021-11-23 15:02:00 +05:30
websocket_client.py Add a `--unix-socket-path` flag (#697) 2021-11-08 01:11:46 +05:30

README.md

Proxy Library Examples

This directory contains examples that demonstrate proxy.py core library capabilities.

Looking for proxy.py plugin examples? Check proxy/plugin directory.

Table of Contents

Generic Work Acceptor and Executor

  1. Makes use of proxy.core.AcceptorPool and proxy.core.Work
  2. Demonstrates how to perform generic work using proxy.py core.

Start web_scraper.py as:

 PYTHONPATH=. python examples/web_scraper.py

WebSocket Client

  1. Makes use of proxy.http.websocket.WebsocketClient which is built on-top of asyncio
  2. websocket_client.py by default opens a WebSocket connection to ws://echo.websocket.org.
  3. Client will exchange num_echos = 10 packets with the server and then shutdown.

Start websocket_client.py as:

 PYTHONPATH=. python examples/websocket_client.py
Received b'hello' after 306 millisec
Received b'hello' after 308 millisec
Received b'hello' after 277 millisec
Received b'hello' after 334 millisec
Received b'hello' after 296 millisec
Received b'hello' after 317 millisec
Received b'hello' after 307 millisec
Received b'hello' after 307 millisec
Received b'hello' after 306 millisec
Received b'hello' after 307 millisec
Received b'hello' after 309 millisec

TCP Echo Server

  1. Makes use of proxy.core.acceptor.AcceptorPool, same multicore acceptor used internally by proxy.py server.
  2. Implements proxy.core.acceptor.Work interface to handle incoming client connections.

Start tcp_echo_server.py as:

 PYTHONPATH=. python examples/tcp_echo_server.py
Connection accepted from ('::1', 53285, 0, 0)
Connection closed by client ('::1', 53285, 0, 0)

TCP Echo Client

  1. Makes use of proxy.common.utils.socket_connection to establish a TCP socket connection with our TCP echo server.
  2. Exchanges packet with server in an infinite loop. Press CTRL+C to stop.

Start tcp_echo_client.py as:

 PYTHONPATH=. python examples/tcp_echo_client.py
b'hello'
b'hello'
b'hello'
b'hello'
b'hello'
...
...
...
^CTraceback (most recent call last):
  File "examples/tcp_echo_client.py", line 18, in <module>
    data = client.recv(DEFAULT_BUFFER_SIZE)
KeyboardInterrupt

SSL Echo Server

  1. Same as tcp_echo_server.py.
  2. Internally uses proxy.common.utils.wrap_socket to enable SSL encryption.
  3. Uses https-key.pem and https-signed-cert.pem for SSL encryption. See End-to-End Encryption for instructions on how to generate SSL certificates.

Start ssl_echo_server.py as:

 PYTHONPATH=. python examples/ssl_echo_server.py

SSL Echo Client

  1. Makes use of proxy.core.connection.TcpServerConnection to establish a SSL connection with our ssl_echo_server.py.
  2. Uses generated ca-cert.pem for SSL certificate verification.

Start ssl_echo_client.py as:

 PYTHONPATH=. python examples/ssl_echo_client.py

PubSub Eventing

  1. Makes use of proxy.py core eventing module.
  2. A proxy.core.event.EventDispatcher thread is started.
  3. A proxy.core.event.EventSubscriber thread is started.
  4. A multiprocessing.Process publisher is started.
  5. Main thread also publishes into EventDispatcher queue.
  6. Events from both the main thread and another process are received by the subscriber.

Start pubsub_eventing.py as:

 PYTHONPATH=. python examples/pubsub_eventing.py
DEBUG:proxy.core.event.subscriber:Subscribed relay sub id 5eb22010764f4d44900f41e2fb408ca6 from core events
publisher starting
^Cpublisher shutdown
bye!!!
DEBUG:proxy.core.event.subscriber:Un-subscribed relay sub id 5eb22010764f4d44900f41e2fb408ca6 from core events
Received 52724 events from main thread, 60172 events from another process, in 21.50117802619934 seconds

HTTPS Connect Tunnel

A simple HTTP proxy server supporting only CONNECT (https) requests.

  1. Uses HttpParser for request parsing.
  2. Uses TcpServerConnection to establish upstream connection.
  3. Overrides BaseServer methods to also register read/write events for upstream connection.

Start https_connect_tunnel.py as:

 PYTHONPATH=. python examples/https_connect_tunnel.py

Send https requests via tunnel as:

 curl -x localhost:12345 https://httpbin.org/get