Move plugin_examples/ as proxy.plugin and update readme (#179)

* Update dev guide

* Move plugin_examples/ as proxy.plugin

* Update proxy.plugin ref path in readme

* Remove unnecessary port flag

* Remove plugin_examples from github workflows

* dashboard folder is a npm package not python package anymore

* Plugins can now be tried using Docker image
This commit is contained in:
Abhinav Singh 2019-11-15 14:47:50 -08:00 committed by GitHub
parent 148c260472
commit 131e9366ac
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
21 changed files with 125 additions and 205 deletions

View File

@ -25,8 +25,8 @@ jobs:
pip install -r requirements-testing.txt
- name: Quality Check
run: |
flake8 --ignore=W504 --max-line-length=127 proxy/ tests/ benchmark/ plugin_examples/ setup.py
mypy --strict --ignore-missing-imports proxy/ tests/ benchmark/ plugin_examples/ setup.py
flake8 --ignore=W504 --max-line-length=127 proxy/ tests/ benchmark/ setup.py
mypy --strict --ignore-missing-imports proxy/ tests/ benchmark/ setup.py
- name: Run Tests
run: pytest --cov=proxy tests/
- name: Upload coverage to Codecov

View File

@ -21,5 +21,4 @@ COPY --from=builder /deps /usr/local
EXPOSE 8899/tcp
ENTRYPOINT [ "proxy" ]
CMD [ "--hostname=0.0.0.0", \
"--port=8899" ]
CMD [ "--hostname=0.0.0.0" ]

View File

@ -27,7 +27,6 @@ autopep8:
autopep8 --recursive --in-place --aggressive proxy/*.py
autopep8 --recursive --in-place --aggressive proxy/*/*.py
autopep8 --recursive --in-place --aggressive tests/*.py
autopep8 --recursive --in-place --aggressive plugin_examples/*.py
autopep8 --recursive --in-place --aggressive benchmark/*.py
autopep8 --recursive --in-place --aggressive setup.py
@ -59,8 +58,8 @@ lib-clean:
rm -rf .hypothesis
lib-lint:
flake8 --ignore=W504 --max-line-length=127 proxy/ tests/ benchmark/ plugin_examples/ setup.py
mypy --strict --ignore-missing-imports proxy/ tests/ benchmark/ plugin_examples/ setup.py
flake8 --ignore=W504 --max-line-length=127 proxy/ tests/ benchmark/ setup.py
mypy --strict --ignore-missing-imports proxy/ tests/ benchmark/ setup.py
lib-test: lib-lint
python -m unittest discover

View File

@ -65,7 +65,10 @@ Table of Contents
* [Everything is a plugin](#everything-is-a-plugin)
* [Internal Architecture](#internal-architecture)
* [Internal Documentation](#internal-documentation)
* [Sending a Pull Request](#sending-a-pull-request)
* [Development Guide](#development-guide)
* [Setup Local Environment](#setup-local-environment)
* [Setup pre-commit hook](#setup-pre-commit-hook)
* [Sending a Pull Request](#sending-a-pull-request)
* [Utilities](#utilities)
* [TCP](#tcp-sockets)
* [new_socket_connection](#new_socket_connection)
@ -120,8 +123,8 @@ Features
- No external dependency other than standard Python library
- Programmable
- Optionally enable builtin Web Server
- Customize proxy and http routing via [plugins](https://github.com/abhinavsingh/proxy.py/blob/develop/plugin_examples)
- Enable plugin using command line option e.g. `--plugins plugin_examples/cache_responses.CacheResponsesPlugin`
- Customize proxy and http routing via [plugins](https://github.com/abhinavsingh/proxy.py/tree/develop/proxy/plugin)
- Enable plugin using command line option e.g. `--plugins proxy.plugin.CacheResponsesPlugin`
- Plugin API is currently in development phase, expect breaking changes.
- Realtime Dashboard
- Optionally enable bundled dashboard.
@ -300,10 +303,12 @@ For example, to check `proxy.py` version within Docker image:
Plugin Examples
===============
See [plugin_examples](https://github.com/abhinavsingh/proxy.py/tree/develop/plugin_examples) for full code.
All the examples below also works with `https` traffic but require additional flags and certificate generation.
See [TLS Interception](#tls-interception).
- See [plugin](https://github.com/abhinavsingh/proxy.py/tree/develop/proxy/plugin) module for full code.
- All the bundled plugin examples also works with `https` traffic
- Require additional flags and certificate generation
- See [TLS Interception](#tls-interception).
- Plugin examples are also bundled with Docker image.
- See [Customize startup flags](#customize-startup-flags) to try plugins with Docker image.
## ShortLinkPlugin
@ -313,7 +318,7 @@ Start `proxy.py` as:
```
$ proxy \
--plugins plugin_examples/shortlink.ShortLinkPlugin
--plugins proxy.plugin.ShortLinkPlugin
```
Now you can speed up your daily browsing experience by visiting your
@ -342,7 +347,7 @@ Start `proxy.py` as:
```
$ proxy \
--plugins plugin_examples/modify_post_data.ModifyPostDataPlugin
--plugins proxy.plugin.ModifyPostDataPlugin
```
By default plugin replaces POST body content with hardcoded `b'{"key": "modified"}'`
@ -396,7 +401,7 @@ Start `proxy.py` as:
```
$ proxy \
--plugins plugin_examples/mock_rest_api.ProposedRestApiPlugin
--plugins proxy.plugin.ProposedRestApiPlugin
```
Verify mock API response using `curl -x localhost:8899 http://api.example.com/v1/users/`
@ -428,7 +433,7 @@ Start `proxy.py` and enable inbuilt web server:
```
$ proxy \
--enable-web-server \
--plugins plugin_examples/redirect_to_custom_server.RedirectToCustomServerPlugin
--plugins proxy.plugin.RedirectToCustomServerPlugin
```
Verify using `curl -v -x localhost:8899 http://google.com`
@ -461,7 +466,7 @@ Start `proxy.py` as:
```
$ proxy \
--plugins plugin_examples/filter_by_upstream.FilterByUpstreamHostPlugin
--plugins proxy.plugin.FilterByUpstreamHostPlugin
```
Verify using `curl -v -x localhost:8899 http://google.com`:
@ -494,7 +499,7 @@ Start `proxy.py` as:
```
$ proxy \
--plugins plugin_examples/cache_responses.CacheResponsesPlugin
--plugins proxy.plugin.CacheResponsesPlugin
```
Verify using `curl -v -x localhost:8899 http://httpbin.org/get`:
@ -570,7 +575,7 @@ Start `proxy.py` as:
```
$ proxy \
--plugins plugin_examples/man_in_the_middle.ManInTheMiddlePlugin
--plugins proxy.plugin.ManInTheMiddlePlugin
```
Verify using `curl -v -x localhost:8899 http://google.com`:
@ -652,7 +657,7 @@ response from the server. Start `proxy.py` as:
```
$ proxy \
--plugins plugin_examples/cache_responses.CacheResponsesPlugin \
--plugins proxy.plugin.CacheResponsesPlugin \
--ca-key-file ca-key.pem \
--ca-cert-file ca-cert.pem \
--ca-signing-key-file ca-signing-key.pem
@ -861,10 +866,6 @@ class TestProxyPyEmbedded(unittest.TestCase):
Plugin Developer and Contributor Guide
======================================
Contributors must start `proxy.py` from source to verify and develop new features / fixes.
See [Run proxy.py from command line using repo source](#from-command-line-using-repo-source) for details.
## Everything is a plugin
As you might have guessed by now, in `proxy.py` everything is a plugin.
@ -912,25 +913,25 @@ and invoke `HttpProxyBasePlugin` lifecycle hooks.
Workers are responsible for accepting new client connections and starting
`HttpProtocolHandler` thread.
## Sending a Pull Request
## Development Guide
Install dependencies for local development testing:
#### Setup Local Environment
`$ pip install -r requirements-testing.txt`
Contributors must start `proxy.py` from source to verify and develop new features / fixes.
See [Run proxy.py from command line using repo source](#from-command-line-using-repo-source) for details.
Every pull request goes through set of tests which must pass:
#### Setup pre-commit hook
- `mypy`: Run `make lint` locally for compliance check.
Fix all warnings and errors before sending out a PR.
1. `cd /path/to/proxy.py`
2. `ln -s $(PWD)/git-pre-commit .git/hooks/pre-commit`
- `coverage`: Run `make coverage` locally for coverage report.
Its ideal to add tests for any critical change. Depending upon
the change, it's ok if test coverage falls by `<0.5%`.
Pre-commit hook ensures lint checking and library tests passes.
- `formatting`: Run `make autopep8` locally to format the code in-place.
`autopep8` is run with `--aggresive` flag. Sometimes it _may_ result in
weird formatting. But let's stick to one consistent formatting tool.
I am open to flag changes for `autopep8`.
#### Sending a Pull Request
Every pull request is tested using GitHub actions.
See [GitHub workflow](https://github.com/abhinavsingh/proxy.py/tree/develop/.github/workflows)
for list of tests.
## Utilities
@ -1051,12 +1052,10 @@ Make sure plugin modules are discoverable by adding them to `PYTHONPATH`. Examp
...[redacted]... - Loaded plugin my_app.proxyPlugin
```
or, make sure to pass fully-qualified path as parameter, e.g.
OR, simply pass fully-qualified path as parameter, e.g.
`proxy --plugins /path/to/my/app/my_app.proxyPlugin`
Note that `pip install proxy.py` don't ship [plugin_examples](https://github.com/abhinavsingh/proxy.py/blob/develop/plugin_examples).
## Unable to connect with proxy.py from remote host
Make sure `proxy.py` is listening on correct network interface.
@ -1108,7 +1107,6 @@ Now `proxy.py` logs can be browsed using
without any socket leaks.
1. Make use of `--open-file-limit` flag to customize `ulimit -n`.
- To set a value upper than the hard limit, run as root.
2. Make sure to adjust `--backlog` flag for higher concurrency.
If nothing helps, [open an issue](https://github.com/abhinavsingh/proxy.py/issues/new)
@ -1131,7 +1129,7 @@ usage: proxy [-h] [--backlog BACKLOG] [--basic-auth BASIC_AUTH]
[--client-recvbuf-size CLIENT_RECVBUF_SIZE]
[--devtools-ws-path DEVTOOLS_WS_PATH]
[--disable-headers DISABLE_HEADERS] [--disable-http-proxy]
[--enable-devtools] [--enable-events]
[--enable-dashboard] [--enable-devtools] [--enable-events]
[--enable-static-server] [--enable-web-server]
[--hostname HOSTNAME] [--key-file KEY_FILE]
[--log-level LOG_LEVEL] [--log-file LOG_FILE]
@ -1186,6 +1184,7 @@ optional arguments:
server.
--disable-http-proxy Default: False. Whether to disable
proxy.HttpProxyPlugin.
--enable-dashboard Default: False. Enables proxy.py dashboard.
--enable-devtools Default: False. Enables integration with Chrome
Devtool Frontend. Also see --devtools-ws-path.
--enable-events Default: False. Enables core to dispatch lifecycle

View File

@ -1,10 +0,0 @@
# -*- coding: utf-8 -*-
"""
proxy.py
~~~~~~~~
Fast, Lightweight, Programmable, TLS interception capable
proxy server for Application debugging, testing and development.
:copyright: (c) 2013-present by Abhinav Singh and contributors.
:license: BSD, see LICENSE for more details.
"""

View File

@ -1 +0,0 @@
# Proxy.py Plugins

View File

@ -1,10 +0,0 @@
# -*- coding: utf-8 -*-
"""
proxy.py
~~~~~~~~
Fast, Lightweight, Programmable, TLS interception capable
proxy server for Application debugging, testing and development.
:copyright: (c) 2013-present by Abhinav Singh and contributors.
:license: BSD, see LICENSE for more details.
"""

View File

@ -1,87 +0,0 @@
# -*- coding: utf-8 -*-
"""
proxy.py
~~~~~~~~
Fast, Lightweight, Programmable, TLS interception capable
proxy server for Application debugging, testing and development.
:copyright: (c) 2013-present by Abhinav Singh and contributors.
:license: BSD, see LICENSE for more details.
"""
from setuptools import setup, find_packages
VERSION = (0, 1, 0)
__version__ = '.'.join(map(str, VERSION[0:3]))
__description__ = '⚡⚡⚡ Fast, Lightweight, Programmable Proxy Server in a single Python file.'
__author__ = 'Abhinav Singh'
__author_email__ = 'mailsforabhinav@gmail.com'
__homepage__ = 'https://github.com/abhinavsingh/proxy.py'
__download_url__ = '%s/archive/master.zip' % __homepage__
__license__ = 'BSD'
setup(
name='proxy.py-plugins',
version=__version__,
author=__author__,
author_email=__author_email__,
url=__homepage__,
description=__description__,
long_description=open('README.md').read().strip(),
long_description_content_type='text/markdown',
download_url=__download_url__,
license=__license__,
packages=find_packages(),
install_requires=['proxy.py'],
classifiers=[
'Development Status :: 5 - Production/Stable',
'Environment :: Console',
'Environment :: No Input/Output (Daemon)',
'Environment :: Web Environment',
'Environment :: MacOS X',
'Environment :: Plugins',
'Environment :: Win32 (MS Windows)',
'Framework :: Robot Framework',
'Framework :: Robot Framework :: Library',
'Intended Audience :: Developers',
'Intended Audience :: Education',
'Intended Audience :: End Users/Desktop',
'Intended Audience :: System Administrators',
'Intended Audience :: Science/Research',
'License :: OSI Approved :: BSD License',
'Natural Language :: English',
'Operating System :: MacOS',
'Operating System :: MacOS :: MacOS 9',
'Operating System :: MacOS :: MacOS X',
'Operating System :: POSIX',
'Operating System :: POSIX :: Linux',
'Operating System :: Unix',
'Operating System :: Microsoft',
'Operating System :: Microsoft :: Windows',
'Operating System :: Microsoft :: Windows :: Windows 10',
'Operating System :: Android',
'Operating System :: OS Independent',
'Programming Language :: Python :: Implementation',
'Programming Language :: Python :: 3 :: Only',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Topic :: Internet',
'Topic :: Internet :: Proxy Servers',
'Topic :: Internet :: WWW/HTTP',
'Topic :: Internet :: WWW/HTTP :: Browsers',
'Topic :: Internet :: WWW/HTTP :: Dynamic Content',
'Topic :: Internet :: WWW/HTTP :: Dynamic Content :: CGI Tools/Libraries',
'Topic :: Internet :: WWW/HTTP :: HTTP Servers',
'Topic :: Scientific/Engineering :: Information Analysis',
'Topic :: Software Development :: Debuggers',
'Topic :: Software Development :: Libraries :: Python Modules',
'Topic :: System :: Monitoring',
'Topic :: System :: Networking',
'Topic :: System :: Networking :: Firewalls',
'Topic :: System :: Networking :: Monitoring',
'Topic :: Utilities',
'Typing :: Typed',
],
)

View File

@ -251,7 +251,7 @@ class Flags:
args.enable_events)),
plugins=Flags.load_plugins(
bytes_(
'%s%s' %
'%s,%s' %
(text_(COMMA).join(collections.OrderedDict(default_plugins).keys()),
opts.get('plugins', args.plugins)))),
pid_file=cast(Optional[str], opts.get('pid_file', args.pid_file)))

29
proxy/plugin/__init__.py Normal file
View File

@ -0,0 +1,29 @@
# -*- coding: utf-8 -*-
"""
proxy.py
~~~~~~~~
Fast, Lightweight, Programmable, TLS interception capable
proxy server for Application debugging, testing and development.
:copyright: (c) 2013-present by Abhinav Singh and contributors.
:license: BSD, see LICENSE for more details.
"""
from .cache_responses import CacheResponsesPlugin
from .filter_by_upstream import FilterByUpstreamHostPlugin
from .man_in_the_middle import ManInTheMiddlePlugin
from .mock_rest_api import ProposedRestApiPlugin
from .modify_post_data import ModifyPostDataPlugin
from .redirect_to_custom_server import RedirectToCustomServerPlugin
from .shortlink import ShortLinkPlugin
from .web_server_route import WebServerPlugin
__all__ = [
'CacheResponsesPlugin',
'FilterByUpstreamHostPlugin',
'ManInTheMiddlePlugin',
'ProposedRestApiPlugin',
'ModifyPostDataPlugin',
'RedirectToCustomServerPlugin',
'ShortLinkPlugin',
'WebServerPlugin',
]

View File

@ -14,9 +14,9 @@ import time
import logging
from typing import Optional, BinaryIO, Any
from proxy.http.parser import HttpParser
from proxy.http.proxy import HttpProxyBasePlugin
from proxy.common.utils import text_
from ..common.utils import text_
from ..http.parser import HttpParser
from ..http.proxy import HttpProxyBasePlugin
logger = logging.getLogger(__name__)

View File

@ -10,10 +10,10 @@
"""
from typing import Optional
from proxy.http.proxy import HttpProxyBasePlugin
from proxy.http.exception import HttpRequestRejected
from proxy.http.parser import HttpParser
from proxy.http.codes import httpStatusCodes
from ..http.exception import HttpRequestRejected
from ..http.parser import HttpParser
from ..http.codes import httpStatusCodes
from ..http.proxy import HttpProxyBasePlugin
class FilterByUpstreamHostPlugin(HttpProxyBasePlugin):

View File

@ -10,10 +10,10 @@
"""
from typing import Optional
from proxy.http.proxy import HttpProxyBasePlugin
from proxy.http.parser import HttpParser
from proxy.http.codes import httpStatusCodes
from proxy.common.utils import build_http_response
from ..common.utils import build_http_response
from ..http.parser import HttpParser
from ..http.codes import httpStatusCodes
from ..http.proxy import HttpProxyBasePlugin
class ManInTheMiddlePlugin(HttpProxyBasePlugin):

View File

@ -11,10 +11,10 @@
import json
from typing import Optional
from proxy.http.parser import HttpParser
from proxy.http.proxy import HttpProxyBasePlugin
from proxy.http.codes import httpStatusCodes
from proxy.common.utils import bytes_, build_http_response, text_
from ..common.utils import bytes_, build_http_response, text_
from ..http.parser import HttpParser
from ..http.proxy import HttpProxyBasePlugin
from ..http.codes import httpStatusCodes
class ProposedRestApiPlugin(HttpProxyBasePlugin):

View File

@ -10,10 +10,10 @@
"""
from typing import Optional
from proxy.http.parser import HttpParser
from proxy.http.proxy import HttpProxyBasePlugin
from proxy.http.methods import httpMethods
from proxy.common.utils import bytes_
from ..common.utils import bytes_
from ..http.parser import HttpParser
from ..http.proxy import HttpProxyBasePlugin
from ..http.methods import httpMethods
class ModifyPostDataPlugin(HttpProxyBasePlugin):

View File

@ -11,9 +11,9 @@
from urllib import parse as urlparse
from typing import Optional
from proxy.http.proxy import HttpProxyBasePlugin
from proxy.http.parser import HttpParser
from proxy.http.methods import httpMethods
from ..http.proxy import HttpProxyBasePlugin
from ..http.parser import HttpParser
from ..http.methods import httpMethods
class RedirectToCustomServerPlugin(HttpProxyBasePlugin):

View File

@ -10,11 +10,11 @@
"""
from typing import Optional
from proxy.http.proxy import HttpProxyBasePlugin
from proxy.http.parser import HttpParser
from proxy.http.codes import httpStatusCodes
from proxy.common.constants import DOT, SLASH
from proxy.common.utils import build_http_response
from ..common.constants import DOT, SLASH
from ..common.utils import build_http_response
from ..http.parser import HttpParser
from ..http.codes import httpStatusCodes
from ..http.proxy import HttpProxyBasePlugin
class ShortLinkPlugin(HttpProxyBasePlugin):

View File

@ -11,11 +11,11 @@
import logging
from typing import List, Tuple
from proxy.http.server import HttpWebServerBasePlugin, httpProtocolTypes
from proxy.http.websocket import WebsocketFrame
from proxy.http.parser import HttpParser
from proxy.http.codes import httpStatusCodes
from proxy.common.utils import build_http_response
from ..common.utils import build_http_response
from ..http.parser import HttpParser
from ..http.codes import httpStatusCodes
from ..http.websocket import WebsocketFrame
from ..http.server import HttpWebServerBasePlugin, httpProtocolTypes
logger = logging.getLogger(__name__)

View File

@ -30,7 +30,14 @@ setup(
long_description_content_type='text/markdown',
download_url=__download_url__,
license=__license__,
packages=['proxy', 'proxy.common', 'proxy.core', 'proxy.http'],
packages=[
'proxy',
'proxy.common',
'proxy.core',
'proxy.dashboard',
'proxy.http',
'proxy.plugin'
],
install_requires=open('requirements.txt', 'r').read().strip().split(),
entry_points={
'console_scripts': [

View File

@ -22,8 +22,7 @@ from proxy.common.utils import build_http_request, bytes_, build_http_response
from proxy.common.constants import PROXY_AGENT_HEADER_VALUE
from proxy.http.codes import httpStatusCodes
from plugin_examples import mock_rest_api
from plugin_examples import redirect_to_custom_server
from proxy.plugin import ProposedRestApiPlugin, RedirectToCustomServerPlugin
from .utils import get_plugin_by_test_name
@ -97,9 +96,9 @@ class TestHttpProxyPluginExamples(unittest.TestCase):
path = b'/v1/users/'
self._conn.recv.return_value = build_http_request(
b'GET', b'http://%s%s' % (
mock_rest_api.ProposedRestApiPlugin.API_SERVER, path),
ProposedRestApiPlugin.API_SERVER, path),
headers={
b'Host': mock_rest_api.ProposedRestApiPlugin.API_SERVER,
b'Host': ProposedRestApiPlugin.API_SERVER,
}
)
self.mock_selector.return_value.select.side_effect = [
@ -118,7 +117,7 @@ class TestHttpProxyPluginExamples(unittest.TestCase):
headers={b'Content-Type': b'application/json'},
body=bytes_(
json.dumps(
mock_rest_api.ProposedRestApiPlugin.REST_API_SPEC[path]))
ProposedRestApiPlugin.REST_API_SPEC[path]))
))
@mock.patch('proxy.http.proxy.TcpServerConnection')
@ -140,7 +139,7 @@ class TestHttpProxyPluginExamples(unittest.TestCase):
self.protocol_handler.run_once()
upstream = urlparse.urlsplit(
redirect_to_custom_server.RedirectToCustomServerPlugin.UPSTREAM_SERVER)
RedirectToCustomServerPlugin.UPSTREAM_SERVER)
mock_server_conn.assert_called_with('localhost', 8899)
mock_server_conn.return_value.queue.assert_called_with(
build_http_request(

View File

@ -1,26 +1,22 @@
from typing import Type
from proxy.http.proxy import HttpProxyBasePlugin
from plugin_examples import modify_post_data
from plugin_examples import mock_rest_api
from plugin_examples import redirect_to_custom_server
from plugin_examples import filter_by_upstream
from plugin_examples import cache_responses
from plugin_examples import man_in_the_middle
from proxy.plugin import ModifyPostDataPlugin, ProposedRestApiPlugin, RedirectToCustomServerPlugin, \
FilterByUpstreamHostPlugin, CacheResponsesPlugin, ManInTheMiddlePlugin
def get_plugin_by_test_name(test_name: str) -> Type[HttpProxyBasePlugin]:
plugin: Type[HttpProxyBasePlugin] = modify_post_data.ModifyPostDataPlugin
plugin: Type[HttpProxyBasePlugin] = ModifyPostDataPlugin
if test_name == 'test_modify_post_data_plugin':
plugin = modify_post_data.ModifyPostDataPlugin
plugin = ModifyPostDataPlugin
elif test_name == 'test_proposed_rest_api_plugin':
plugin = mock_rest_api.ProposedRestApiPlugin
plugin = ProposedRestApiPlugin
elif test_name == 'test_redirect_to_custom_server_plugin':
plugin = redirect_to_custom_server.RedirectToCustomServerPlugin
plugin = RedirectToCustomServerPlugin
elif test_name == 'test_filter_by_upstream_host_plugin':
plugin = filter_by_upstream.FilterByUpstreamHostPlugin
plugin = FilterByUpstreamHostPlugin
elif test_name == 'test_cache_responses_plugin':
plugin = cache_responses.CacheResponsesPlugin
plugin = CacheResponsesPlugin
elif test_name == 'test_man_in_the_middle_plugin':
plugin = man_in_the_middle.ManInTheMiddlePlugin
plugin = ManInTheMiddlePlugin
return plugin