2019-10-02 07:09:35 +00:00
[](https://github.com/abhinavsingh/proxy.py)
2013-08-31 09:19:22 +00:00
2019-09-21 22:04:44 +00:00
[](https://opensource.org/licenses/BSD-3-Clause)
2019-12-02 08:08:15 +00:00
[](https://pypi.org/project/proxy.py/)
[](https://hub.docker.com/r/abhinavsingh/proxy.py)
GitHub workflow badge (#244)
* v2.x (#173)
* Always update latest tag for docker releases
* Update issue templates (#123)
* Invoke HttpWebServerBasePlugin.handle_request for each request in HTTP/1.1 pipeline (#125)
* Add tests to verify certificate generation
* Separate out tests for ProtocolHandler and WebServerPlugin
* Keep-alive connections for web server.
TODO: Only keep-alivei if HTTP/1.1
* Add request.path to avoid build_url repeatedly whose name is also slightly misleading
* Fix example usage of request.path
* Pipeline only for HTTP/1.1
* Lint fix
* Teardown HTTP/1.1 keep-alive request when Connection: close header is sent
* Add instructions on how to build docker image locally
* Move access_log to separate function for pretty logging
* Reduce docker image size
* Ensure teardown is always accompanied with Connection: close header
Fix tests
* Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128)
* Add tests for is_http_1_1_keep_alive
* Add ModifyPostDataPlugin in README
* Fixes #126
* Refactor HttpProxyBasePlugin API
* before_upstream_connection too can drop request by returning None
* Remove HTTP Server startup during tests, no longer used
* Removed unused imports
* Simplify load_plugins
* Add --timeout flag with default value of 10 second. (#129)
* Add --timeout flag with default value of 5. This value was previously hardcoded to 30
* --timeout=10 by default
* Dispatch 408 timeout when connection is dropped due to inactivity
* Add httpStatusCodes named tuple
* Update plugin client connection reference after TLS connection upgrade
* Test plugin examples (#130)
* Add tests for plugin_examples.* to ensure we never break functionality
* Add tests for plugin_examples.*
* Test man in the middle
* Lint fixes
* Checkin
* Add tests for plugin examples with TLS encryption enabled
* Threadless execution using coroutines (#134)
* Workers need not register/unregister sock for every loop
* No need of explicit socket.settimeout(0) which is same as socket.setblocking(False)
* Remove settimeout assertion
* Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown
* Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes
* Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests
* Remove unused
* Wrap selector register/unregister within a context manager
* Refactor in preparation of threadless request handling
* MyPy generator fix
* Add --threadless flag
* Internally call them acceptors
* Internally use acceptors
* Add Threadless class.
Also no need to pass family over pipe to acceptors.
* Make threadless work for a single client :)
* Threadless is soon be our default
* Close client queue
* Use context manager for register/unregister
* Fix Acceptor tests broken after refactoring
* Use asyncio tasks to invoke ProtocolHandle.handle_events
This gives all client threads a chance to respond without
waiting for other handlers to return.
* Explicitly initialize event loop per Threadless process
* Mypy fixes
* Add ThreadlessWork abstract class implemented by ProtocolHandler
* Add benchmark.py
Avoid TIME_WAIT by properly shutting down the connection.
* Add benchmark.py as part of testing workflow
* When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state
* MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence
* Unwrap if wrapped before shutdown
* Unwrap if wrapped before shutdown
* socket.SHUT_RDWR will cause leaks
* MyPy
* Add instructions for monitor.sh
* Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized
* Add Fast & Scalable section
* Update internal classes section
* Dont print out local dir path in help text :)
* Refactor
* Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang
* Add chrome_with_proxy.sh helper script
* Handle OSError during client.flush which can happen due to invalid protocol type for socket error
* Remove redundant e
* Add classmethods to quickly construct a parser object
* Don't raise from TcpConnection abstract class.
This allows both client/socket side of communication to handle
exceptions as necessary. We might refactor this again later to remove
redundant code :)
* Disable response parsing when TLS interception is enabled. See issue #127
* remove unused imports
* Within webserver parse pipelined requests only if we have a route
* Add ShortLinkPlugin plugin
* Add more shortlinks
* Add ShortLinkPlugin to README.md
* Add path forwarding too instead of leaving as excercise ;)
* Add shortlink to TOC
* Ensure no socket leaks
* Ensure no leaks
* Naming
* Default number of clients 1
* Avoid shortlinking localhost
* Stress more
* Remove pip upgrade for windows which seems to be failing on travis (#136)
* Remove pip upgrade for windows which seems to be failing on travis
* Remove windows testing on Travis, pip install is failing
* Add pipeline response parsing tests (#137)
* Add pipeline response parsing tests
* build_http_response now only adds content-length if transfer-encoding is
not provided.
Also return pending raw chunks from ChunkParser so that we can parse
pipelined chunk responses.
* os.close only for threadless (#138)
* os.close only for Threadless to avoid fd leaks
* Remove os.close mock which is only called for threadless
* Update pytest from 5.2.1 to 5.2.2 (#142)
* Update setuptools from 41.4.0 to 41.5.0 (#145)
* Update typing-extensions from 3.7.4 to 3.7.4.1 (#147)
* Update flake8 from 3.7.8 to 3.7.9 (#148)
* Update setuptools from 41.5.0 to 41.5.1 (#149)
* Update py-spy from 0.2.2 to 0.3.0 (#144)
* Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
* Update mypy==0.740 (#151)
* Update README.md (#152)
* Update flags
* Update debugging instructions and run instructions for develops
* Update references to plugins directory
* For readability add sections for run from command line using pip
* Move internal doc under developer section
* Add option to pass fully-qualified plugin path
* Update setuptools from 41.5.1 to 41.6.0 (#153)
* Test refactor + Docker image CI (#154)
* Move tests into individual modules too
* Ensure one test class per file
* Fix docker image after refactoring
* Add github actions workflow for building docker image
* Fix image name
* Setup python required for extracting proxy version
* Version will also require deps
* Separate packages for Dashboard (#157)
* Refactor Makefile and add dashboard setup.py
* Package dashboard as proxy.py-dashboard pip package
* Give dashboard releases its own version
* Fix lib-package reference
* Add non-blocking embedded mode feature (#159)
* Fixes #158
* mypy fixes
* Instructions for non-blocking embed mode
* Toggle running flag before shutdown
* Add private / public key generation utils which comply with new requirements on Mac OS 10.15 (#160)
* Add utilities to generate private key and public keys with alternate cnames
* Add separate package proxy.py-plugins, fixes #156
* Generate certificates to comply with Mac requirements.
* Add utility for CSR generation and signing
* Fixes #161
* Add initial pki tests
* Give structure to dashboard app (#163)
* Separate out files for different responsibilities.
1. Add src/plugins directory. This directory holds one typescript file
per plugin. Each plugin is optionally can be displayed as a tab on
the UI.
2. Move WebsocketApi to ws.ts. This file contains all websocket APIs
provided by dashboard.py backend.
* Make dashboard pluggable
* Move devtools under core too
* Register tabs dynamically
* Typescript fixes for abstract interfaces
* Initialize plugin app body skeleton
* Call activated / deactivated on tab change
* Move plugin name within plugin classes and initialize plugin within proxy dashboard constructor
* templatize api development plugin
* eslint fixes
* use globs
* Remove useless constructors
* Move traffic_control outside of core plugin, it maps to several plugin examples like redirectToUpstreamHost, filterByUpstreamHost plugins (#165)
* Introduce sendMessage websocket api which allows for callbacks (#166)
* Introduce sendMessage websocket api which allows for callbacks, deprecate lastPingId in favor of callbacks
* Let InspectTrafficPlugin handle all pushed inspection events
* Add proxy.main.TestCase for unit testing Python application with proxy.py (#167)
* Add demonstration of how to use proxy.py within Python application unittests
* mypy fixes
* test_with_proxy example
* Add docs for proxy.main.TestCase.
Also wait for proxy.py server to come up before running the tests.
* Consistent dashboard look and feel across plugins (#169)
* Explicitly link version changelog in TOC
* Separate out app header body builder
* Ensure unsubscribe when disabling inspection. Fixes #164
* Avoid creation of new manager per dashboard instance.
* Add UI header for all plugins (tabs)
* Ensure app body for all plugin skeleton
* Move app-header and app-body within core for consistent dashboard look and feel
* Consistent UI header body for plugins
* autopep8
* Dashboard Inspect traffic tab + devtools (#170)
* Explicitly link version changelog in TOC
* Separate out app header body builder
* Ensure unsubscribe when disabling inspection. Fixes #164
* Avoid creation of new manager per dashboard instance.
* Add UI header for all plugins (tabs)
* Ensure app body for all plugin skeleton
* Move app-header and app-body within core for consistent dashboard look and feel
* Consistent UI header body for plugins
* autopep8
* make devtools
* convert to es6
* Add inspect_traffic plugin devtools app
* trigger re-build, github UI is stuck
* Dynamically load devtools within inspect traffic view
* Just copy devtools into public/dashboard folder
* Works but not how we wanted, devtools takes over entire body and doesnt contain itself within a div
* Load devtools within iframe
* Load devtools within iframe (#171)
* Allow to pass flags as kwargs too in embed mode (#172)
* Dynamically load devtools instead of on page load
* Add support for passing flags as kwargs to main / start methods.
* Fix tests for refactored code
* Allow proxy.main, proxy.start, proxy.TestCase.
Also update README.md to reflect the same.
* Use Any for **opts
* Move main as __init__ to avoid name conflicts
* Fix tests
* Update setup.py entry_point
* Explicitly install requirements before setup.py
* Explicitly mention packages of interest
* ipv6 fails on ubuntu, use ipv4
* Make typing-extensions optional
* Instead of putting it all under __init__.py, move main.py to proxy.py
* Simply make setup.py module free
* autopep8
* Devtools Protocol (#174)
* Refine docs
* Decouple relay from dashboard.
Will be re-used by devtools protocol plugin.
* Just have a single manager for all eventing
* Ofcourse managers cant be shared across processes
* Remove unused
* Add DevtoolsProtocolPlugin
* Emit REQUEST_COMPLETE core event
* Emit only if --enable-events used
* Add event emitter for response cycle
* Fill up core events to devtools protocol expectations
* Serve static content with Cache-Control header and gzip compression
* Add PWA manifest.json and icons from sample PWA apps (replace later)
* Catch any exception and be ssl agnostic
* Add CSP headers and avoid inline scripts
* Re-enable iframe and deobfuscation
* Embed plugins within <section/> block
* Make tab switching agnostic of block name
* Add support for browser history on tab change
* Default hash to #home
* Switch to tab if hash is already set
* Expand canvas to fill screen even without content
* Remove inline css for embedded devtools
* Make dashboard backend websocket API pluggable
* doc
* Move dashboard backend within proxy module, now ships via same pip package (#177)
* Allow resources to load from http and ws when running w/o https
* Move dashboard backend (dashboard.py) within proxy module. Now shipped with pip install proxy.py
* Update ref to dashboard backend in github workflows
* Add git-pre-commit hook file.
Enable it by symlinking as .git/hooks/pre-commit
* Also enable static server for dashboard serving
* Move plugin_examples/ as proxy.plugin and update readme (#179)
* Update dev guide
* Move plugin_examples/ as proxy.plugin
* Update proxy.plugin ref path in readme
* Remove unnecessary port flag
* Remove plugin_examples from github workflows
* dashboard folder is a npm package not python package anymore
* Plugins can now be tried using Docker image
* Move benchmark module within proxy (#181)
* Move benchmark within proxy module
* chmod 0644 for benchmark.py which was executable till now
* Turn utilities into its own section
* Update pytest from 5.2.3 to 5.2.4 (#180)
* Doc & Banner update to match GitHub (#182)
* Update doc and banner
* Update banner to match GitHub
* Update older banners too
* Add update_desc to .gitignore
* Update banner for dashboard to match github
* also update html, js, css
* Update twine from 2.0.0 to 3.0.0 (#183)
* Update pytest from 5.2.4 to 5.3.0 (#186)
* Testing support improvements (#185)
* Introduce proxy.Proxy context manager.
This is similar to already existing context manager `start` but
`proxy.Proxy` is a class with __enter__ and __exit__ methods. This
allows usage of `proxy.Proxy` both as context manager and for manually
setup and teardown of `proxy.py` during test setUpClass and
teardownClass methods.
* Gracefully shutdown threadless processes
* Update tests and add a VCR method. See #184
* Refactor routes
* Add Proxy to __all__
* Move TestCase under proxy.testing and test_embed.py under tests.embed module to avoid conflict with http module due to a http directory under proxy folder
* Add a base cache plugin class which can be customized for custom cache behaviors
* See #184. Add VCRPlugin which can be enabled within tests using a context manager, e.g. with self.vcr(): ...
* Make cache plugin pluggable + make cache storage pluggable
* Make dashboard npm module agnostic of top level directory
* Symlink dashboard public folder
* Dump devtools within dashboard public folder
* Remove unused 3rd party js
* Initialize Menubar (#188)
* Initialize MacOS Menubar application
* Dashboard plugin at-least needs a shutdown hook to teardown any thread/processes started by dashboard backend plugin
* Add menu bar icon
* Add respective test directories
* Sync test banners
* Move plugin tests under its own package
* Enable daemon for threads, other this wont shutdown cleanly
* Update twine from 3.0.0 to 3.1.0 (#190)
* Update setuptools from 41.6.0 to 42.0.0 (#191)
* Memory optimizations (#189)
* Avoid persisting raw content in memory within parser, simply parse and throw-away. Addresses #187
* Clarity in test comments
* Update setuptools from 42.0.0 to 42.0.1 (#193)
* Make connection queue / recv work with memoryview to avoid copies (#192)
* connection.recv now returns a memoryview
* Make connection.queue also memoryview compliant
* autopep8
* wrap in memoryview as necessary
* Add default timeout for socket_connection and test_embed urllib
* Fix tests
* Skip TestProxyPyEmbedded for now, verifying GitHub actions
* Add timeout for wait_for_server and skip only if GITHUB_ACTIONS env variable is set
* Verify if GitHub Action fails due to wait_for_server spinning forever
* Add test for wait_for_server timeout error exception
* GitHub action hangs irrespective of wait_for_server timeout, disable TestEmbed for GitHub actions
* Cleanup (#194)
* Add basic README description for dashboard
* Use spaces for all except makefile
* enable tests for py 3.5
* Python 3.5 support label
* Avoid clash of names
* Add py3.8 support and bump node to 12.x (#195)
* Add py3.8 support and bump node to 12.x
* Add 10.x, 11.x, 12.x matrix for dashboard testing
* Add Python 3.8 support label
* Single tested with label
* autopep8 (#196)
* autopep8
* Update TestCase section
* Update pytest from 5.3.0 to 5.3.1 (#197)
* Update twine from 3.1.0 to 3.1.1 (#200)
* Add reverse proxy example (#201)
* Add reverse proxy example
* Add separate sections for http proxy and web server plugins
* Add doc
* Add proxy over ssh tunnel functionality (#198)
* update mypy to 0.750 (#204)
* Test Core Eventing (#205)
* Add core event tests
* Update .gitignore with coverage
* Add shortlink gif
* Add event dispatcher test
* Test event subscriber
* Test Dashboard backend (#206)
* Update shortlink gif name
* Conditionally run workflows as necessary
* Use pytest
* It works but github workflow is not reporting any status :(
* Separate out badges
* Add python_requires to setup.py
* Update setuptools from 42.0.1 to 42.0.2 (#207)
* Add tox.ini (#208)
* Homebrew formula (#209)
* Add homebrew formula
* Build PyPi package and Homebrew installation verification
* Check develop
* bdist_wheel reported as error: invalid command "bdist_wheel"
* Move under stable/develop folders to keep Proxy class name same
* uff
* develop installs proxy not proxy.py binary
* Prepend site-packages
* Install typing-extensions explicitly with brew
* Use find_packages
* Most likely failing due to lack of find_packages in current develop branch
* Fix windows setup.py build
* test_static_web_server_serves seems flaky on Ubuntu python 3.8
* Add instructions to install using homebrew
* Disable test_static_web_server_serves on GitHub actions, seems flaky
* Packaging (#210)
* Move docker installation steps above
* Try brewing with virtualenv
* depends on python
* Update homebrew formula for stable release
* Just test brewing on latest python
* Add support for regex based routing. Fixes #203 (#211)
* Remove public folder references (#212)
* Refactor (#213)
* Add DEFAULT_HTTP_PORT constant
* Use DEFAULT_HTTP_PORT in tests
* Refactor into exception module
* Refactor into inspector module
* Refactor into server module
* Refactor into proxy module
* Build docker of Python 3.8 (#214)
* Move homebrew under helper (#215)
* Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET on no internet (#216)
* Catch TimeoutError and OSError (host unreachable)
* Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET
* Enable mccabe (#217)
* No need of per day or week stats (#218)
* Make HTTP handler constructor free of socket file number (#219)
* Refactor into acceptor module
* Add tunnel doc
* Make fileno free
* Autopep8
* Response parser now reaches COMPLETE even when no body is expected (#220)
* Stash current changes
* Refactor into connection module
* Response parser state complete when no body expect
* Raise NotImplementedError if invalid state reached within parser
* Update tox from 3.14.1 to 3.14.2 (#221)
* Update paramiko from 2.6.0 to 2.7.0 (#225)
* Update paramiko from 2.7.0 to 2.7.1 (#227)
* Proxy Pool Plugin (#228)
* Add proxy pool example. See #226
* Add ProxyPoolPlugin to doc
* Update pytest from 5.3.1 to 5.3.2 (#229)
* Update coverage from 4.5.4 to 5.0 (#230)
* Update mypy from 0.750 to 0.760 (#232)
* Update mypy from 0.760 to 0.761 (#235)
* Move manager initialization outside of top level scope. Fixes #233 (#236)
* Share lock to acceptors via pool (#238)
* Move manager initialization outside of top level scope. Fixes #233
* Share lock to acceptor via pool
* Optionally initialize manager in main thread and use the same for EventQueue initialization (#239)
* Highlight language syntax (#240)
* Highlight lang syntax
* zsh prompt
* Update coverage from 5.0 to 5.0.1 (#241)
* Integration testing (#243)
* Add tests for public/private/csr generation
* Add integration testing skeleton for mac and ubuntu
* Merge integration within lib test to avoid too many workflows
* Disable integration testing on windows for now
* Use sudo to start integration test script as lsof fails on MacOS.
lsof: WARNING: can't stat() vmhgfs file system
* Add basic integration testing for now to assert proxy works as expected when started out of develop branch
* Add a call to inbuilt http server to verify it works
* wait for server to accept requests
Co-authored-by: pyup.io bot <github-bot@pyup.io>
* Add github workflow badges
Co-authored-by: pyup.io bot <github-bot@pyup.io>
2019-12-26 01:05:39 +00:00
[](https://github.com/abhinavsingh/proxy.py)
[](https://github.com/abhinavsingh/proxy.py/actions)
[](https://github.com/abhinavsingh/proxy.py/actions)
[](https://github.com/abhinavsingh/proxy.py/actions)
[](https://github.com/abhinavsingh/proxy.py/actions)
2019-12-02 23:55:08 +00:00
[](https://codecov.io/gh/abhinavsingh/proxy.py)
2019-12-02 08:08:15 +00:00
2019-12-01 06:54:20 +00:00
[](https://abhinavsingh.com/proxy-py-a-lightweight-single-file-http-proxy-server-in-python/)
[](https://abhinavsingh.com/proxy-py-a-lightweight-single-file-http-proxy-server-in-python/)
[](https://abhinavsingh.com/proxy-py-a-lightweight-single-file-http-proxy-server-in-python/)
2019-09-23 07:30:52 +00:00
[](https://gitHub.com/abhinavsingh/proxy.py/graphs/commit-activity)
[](https://twitter.com/imoracle)
[](https://github.com/abhinavsingh/proxy.py/issues)
2019-10-02 07:09:35 +00:00
[](https://gitter.im/proxy-py/community)
2019-09-23 07:30:52 +00:00
2021-10-29 17:45:23 +00:00
[](https://www.python.org/)
2019-11-27 05:44:20 +00:00
[](http://mypy-lang.org/)
2019-02-09 04:25:09 +00:00
2021-01-10 16:42:56 +00:00
# Table of Contents
- [Features ](#features )
- [Install ](#install )
2021-11-08 22:23:17 +00:00
- [Stable vs Develop ](#stable-vs-develop )
2021-01-10 16:42:56 +00:00
- [Using PIP ](#using-pip )
- [Stable version ](#stable-version-with-pip )
- [Development version ](#development-version-with-pip )
- [Using Docker ](#using-docker )
- [Stable version ](#stable-version-from-docker-hub )
- [Development version ](#build-development-version-locally )
- [Using HomeBrew ](#using-homebrew )
- [Stable version ](#stable-version-with-homebrew )
- [Development version ](#development-version-with-homebrew )
- [Start proxy.py ](#start-proxypy )
- [From command line when installed using PIP ](#from-command-line-when-installed-using-pip )
- [Run it ](#run-it )
- [Understanding logs ](#understanding-logs )
- [Enable DEBUG logging ](#enable-debug-logging )
- [From command line using repo source ](#from-command-line-using-repo-source )
- [Docker Image ](#docker-image )
- [Customize Startup Flags ](#customize-startup-flags )
- [Plugin Examples ](#plugin-examples )
- [HTTP Proxy Plugins ](#http-proxy-plugins )
- [ShortLink Plugin ](#shortlinkplugin )
- [Modify Post Data Plugin ](#modifypostdataplugin )
- [Mock Api Plugin ](#mockrestapiplugin )
- [Redirect To Custom Server Plugin ](#redirecttocustomserverplugin )
- [Filter By Upstream Host Plugin ](#filterbyupstreamhostplugin )
- [Cache Responses Plugin ](#cacheresponsesplugin )
- [Man-In-The-Middle Plugin ](#maninthemiddleplugin )
- [Proxy Pool Plugin ](#proxypoolplugin )
- [FilterByClientIpPlugin ](#filterbyclientipplugin )
- [ModifyChunkResponsePlugin ](#modifychunkresponseplugin )
2021-11-05 02:06:02 +00:00
- [CloudflareDnsResolverPlugin ](#cloudflarednsresolverplugin )
- [CustomDnsResolverPlugin ](#customdnsresolverplugin )
2021-11-06 15:19:55 +00:00
- [CustomNetworkInterface ](#customnetworkinterface )
2021-01-10 16:42:56 +00:00
- [HTTP Web Server Plugins ](#http-web-server-plugins )
- [Reverse Proxy ](#reverse-proxy )
- [Web Server Route ](#web-server-route )
- [Plugin Ordering ](#plugin-ordering )
- [End-to-End Encryption ](#end-to-end-encryption )
- [TLS Interception ](#tls-interception )
- [TLS Interception With Docker ](#tls-interception-with-docker )
- [Proxy Over SSH Tunnel ](#proxy-over-ssh-tunnel )
- [Proxy Remote Requests Locally ](#proxy-remote-requests-locally )
- [Proxy Local Requests Remotely ](#proxy-local-requests-remotely )
- [Embed proxy.py ](#embed-proxypy )
- [Blocking Mode ](#blocking-mode )
- [Non-blocking Mode ](#non-blocking-mode )
2021-11-04 21:41:22 +00:00
- [Ephemeral Port ](#ephemeral-port )
2021-01-10 16:42:56 +00:00
- [Loading Plugins ](#loading-plugins )
- [Unit testing with proxy.py ](#unit-testing-with-proxypy )
- [proxy.TestCase ](#proxytestcase )
- [Override Startup Flags ](#override-startup-flags )
- [With unittest.TestCase ](#with-unittesttestcase )
- [Utilities ](#utilities )
- [TCP ](#tcp-sockets )
- [new_socket_connection ](#new_socket_connection )
- [socket_connection ](#socket_connection )
- [Http ](#http-client )
- [build_http_request ](#build_http_request )
- [build_http_response ](#build_http_response )
- [Public Key Infrastructure ](#pki )
- [API Usage ](#api-usage )
- [CLI Usage ](#cli-usage )
2021-10-29 23:02:05 +00:00
- [Run Dashboard ](#run-dashboard )
- [Inspect Traffic ](#inspect-traffic )
2021-11-06 15:19:55 +00:00
- [Chrome DevTools Protocol ](#chrome-devtools-protocol )
2021-01-10 16:42:56 +00:00
- [Frequently Asked Questions ](#frequently-asked-questions )
- [Threads vs Threadless ](#threads-vs-threadless )
- [SyntaxError: invalid syntax ](#syntaxerror-invalid-syntax )
- [Unable to load plugins ](#unable-to-load-plugins )
- [Unable to connect with proxy.py from remote host ](#unable-to-connect-with-proxypy-from-remote-host )
- [Basic auth not working with a browser ](#basic-auth-not-working-with-a-browser )
- [Docker image not working on MacOS ](#docker-image-not-working-on-macos )
- [ValueError: filedescriptor out of range in select ](#valueerror-filedescriptor-out-of-range-in-select )
- [None:None in access logs ](#nonenone-in-access-logs )
2021-11-06 21:20:11 +00:00
- [OSError when wrapping client for TLS Interception ](#oserror-when-wrapping-client-for-tls-interception )
2021-11-06 15:19:55 +00:00
- [Plugin Developer and Contributor Guide ](#plugin-developer-and-contributor-guide )
- [High level architecture ](#high-level-architecture )
- [Everything is a plugin ](#everything-is-a-plugin )
- [Internal Documentation ](#internal-documentation )
- [Development Guide ](#development-guide )
- [Setup Local Environment ](#setup-local-environment )
- [Setup Git Hooks ](#setup-git-hooks )
- [Sending a Pull Request ](#sending-a-pull-request )
2021-01-10 16:42:56 +00:00
- [Flags ](#flags )
- [Changelog ](#changelog )
- [v2.x ](#v2x )
- [v1.x ](#v1x )
- [v0.x ](#v0x )
# Features
2019-10-16 06:56:39 +00:00
- Fast & Scalable
2021-01-10 16:42:56 +00:00
- Scales by using all available cores on the system
2021-11-08 22:23:17 +00:00
- Threadless executions using asyncio
2021-01-10 16:42:56 +00:00
- Made to handle `tens-of-thousands` connections / sec
2021-11-06 15:19:55 +00:00
```console
2021-01-10 16:42:56 +00:00
# On Macbook Pro 2015 / 2.8 GHz Intel Core i7
❯
Summary:
Total: 0.6157 secs
Slowest: 0.1049 secs
Fastest: 0.0007 secs
Average: 0.0055 secs
Requests/sec: 16240.5444
Total data: 800000 bytes
Size/request: 80 bytes
Response time histogram:
0.001 [1] |
0.011 [9565] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.022 [332] |■
```
2019-09-21 22:04:44 +00:00
- Lightweight
2021-01-10 16:42:56 +00:00
- Uses only `~5-20MB` RAM
- No external dependency other than standard Python library
2019-09-21 22:04:44 +00:00
- Programmable
2021-11-08 22:23:17 +00:00
- Customize proxy behavior using [Proxy Server Plugins ](#http-proxy-plugins ). Example:
- `--plugins proxy.plugin.ProxyPoolPlugin`
- Optionally, enable builtin [Web Server Plugins ](#http-web-server-plugins ). Example:
- `--plugins proxy.plugin.ReverseProxyPlugin`
- Plugin API is currently in development phase, expect breaking changes
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
- Realtime Dashboard
2021-11-08 22:23:17 +00:00
- Optionally, enable [proxy.py dashboard ](#run-dashboard ).
2021-01-10 16:42:56 +00:00
- Available at `http://localhost:8899/dashboard` .
2021-11-08 22:23:17 +00:00
- [Inspect, Monitor, Control and Configure ](#inspect-traffic ) `proxy.py` at runtime
- [Chrome DevTools Protocol ](#chrome-devtools-protocol ) support
- Extend dashboard using plugins
- Dashboard is currently in development phase, expect breaking changes
2019-09-24 00:38:10 +00:00
- Secure
2021-11-08 22:23:17 +00:00
- Enable end-to-end encryption between clients and `proxy.py`
2021-01-10 16:42:56 +00:00
- See [End-to-End Encryption ](#end-to-end-encryption )
2021-11-08 22:23:17 +00:00
- Private
- Everyone deserves privacy. Browse with malware and adult content protection
- See [DNS-over-HTTPS ](#cloudflarednsresolverplugin )
2019-09-25 02:02:57 +00:00
- Man-In-The-Middle
2021-01-10 16:42:56 +00:00
- Can decrypt TLS traffic between clients and upstream servers
- See [TLS Interception ](#tls-interception )
2019-09-24 00:38:10 +00:00
- Supported proxy protocols
2021-01-10 16:42:56 +00:00
- `http(s)`
- `http1`
2021-11-08 22:23:17 +00:00
- `http1.1` with pipeline
2021-01-10 16:42:56 +00:00
- `http2`
- `websockets`
2021-11-08 22:23:17 +00:00
- Static file server support
- See `--enable-static-server` and `--static-server-dir` flags
2019-08-23 00:09:35 +00:00
- Optimized for large file uploads and downloads
2021-11-08 22:23:17 +00:00
- See `--client-recvbuf-size` and `--server-recvbuf-size` flag
- `IPv4` and `IPv6` support
- See `--hostname` flag
- Unix domain socket support
- See `--unix-socket-path` flag
2018-12-14 13:14:56 +00:00
- Basic authentication support
2021-11-08 22:23:17 +00:00
- See `--basic-auth` flag
- PAC (Proxy Auto-configuration) support
2021-01-10 16:42:56 +00:00
- See `--pac-file` and `--pac-file-url-path` flags
2013-08-31 09:19:22 +00:00
2021-01-10 16:42:56 +00:00
# Install
2013-08-31 09:19:22 +00:00
2021-11-08 22:23:17 +00:00
## Stable vs Develop
`master` branch contains latest stable code and is available via `PyPi` repository
`develop` branch contains cutting edge changes
Development branch is kept stable *(most of the times)* . But if you want 100% reliability and serving users in production environment, always use stable version from `PyPi` or `Docker` container from `hub.docker.com` .
2019-12-02 03:01:28 +00:00
## Using PIP
### Stable Version with PIP
2013-08-31 09:19:22 +00:00
2019-10-10 05:36:47 +00:00
Install from `PyPi`
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
```
2013-08-31 09:19:22 +00:00
2019-10-10 05:36:47 +00:00
or from GitHub `master` branch
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
```
2019-10-10 05:36:47 +00:00
2019-12-02 03:01:28 +00:00
### Development Version with PIP
2019-09-17 03:18:14 +00:00
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
```
2019-09-17 03:18:14 +00:00
2019-12-02 04:08:50 +00:00
## Using Docker
2021-11-08 22:23:17 +00:00
### Stable Version from Docker Hub
2019-12-02 04:08:50 +00:00
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
```
2019-12-02 04:08:50 +00:00
2021-11-08 22:23:17 +00:00
### Build Development Version Locally
2019-12-02 04:08:50 +00:00
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2021-11-08 22:23:17 +00:00
❯ & & make container
2019-12-22 04:57:29 +00:00
❯
```
2019-12-02 04:08:50 +00:00
[](https://github.com/moby/vpnkit/issues/469)
`docker` image is currently broken on `macOS` due to incompatibility with [vpnkit ](https://github.com/moby/vpnkit/issues/469 ).
2019-12-02 03:01:28 +00:00
## Using HomeBrew
### Stable Version with HomeBrew
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
```
2019-12-02 03:01:28 +00:00
### Development Version with HomeBrew
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
```
2019-12-02 03:01:28 +00:00
2021-01-10 16:42:56 +00:00
# Start proxy.py
2019-09-25 17:09:40 +00:00
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
## From command line when installed using PIP
2019-09-25 17:09:40 +00:00
2019-10-28 23:14:34 +00:00
When `proxy.py` is installed using `pip` ,
2019-11-08 04:53:08 +00:00
an executable named `proxy` is placed under your `$PATH` .
2019-10-28 23:14:34 +00:00
2021-11-08 22:23:17 +00:00
### Run it
2019-10-28 23:14:34 +00:00
2021-11-08 22:23:17 +00:00
Simply type `proxy` on command line to start with default configuration.
2019-09-25 17:09:40 +00:00
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
...[redacted]... - Loaded plugin proxy.http_proxy.HttpProxyPlugin
2021-11-08 22:23:17 +00:00
...[redacted]... - Starting 8 threadless workers
2019-09-25 17:09:40 +00:00
...[redacted]... - Started server on ::1:8899
```
2021-11-08 22:23:17 +00:00
### Understanding logs
2019-10-28 23:14:34 +00:00
2019-09-25 17:09:40 +00:00
Things to notice from above logs:
2021-11-08 22:23:17 +00:00
- `Loaded plugin`
- `proxy.py` will load `proxy.http.proxy.HttpProxyPlugin` by default
- As name suggests, this core plugin adds `http(s)` proxy server capabilities to `proxy.py` instance
2019-09-25 17:09:40 +00:00
2021-11-08 22:23:17 +00:00
- `Started N threadless workers`
- By default, `proxy.py` will start as many workers as there are CPU cores on the machine
- Use `--num-workers` flag to customize number of worker processes
- See [Threads vs Threadless ](#threads-vs-threadless ) to understand how to control execution mode
2019-09-25 17:09:40 +00:00
2021-11-08 22:23:17 +00:00
- `Started server on ::1:8899`
- By default, `proxy.py` listens on IPv6 `::1` , which is equivalent of IPv4 `127.0.0.1`
- If you want to access `proxy.py` from external host, use `--hostname ::` or `--hostname 0.0.0.0` or bind to any other interface available on your machine.
- See [CustomNetworkInterface ](#customnetworkinterface ) for how to customize `proxy.py` *public IP seen by upstream servers* .
2019-09-25 17:09:40 +00:00
2021-11-08 22:23:17 +00:00
- `Port 8899`
- Use `--port` flag to customize default TCP port.
2019-09-25 17:09:40 +00:00
2021-11-08 22:23:17 +00:00
### Enable DEBUG logging
2019-10-28 23:14:34 +00:00
2021-11-08 22:23:17 +00:00
All the logs above are `INFO` level logs, default `--log-level` for `proxy.py`
2019-09-25 17:09:40 +00:00
Lets start `proxy.py` with `DEBUG` level logging:
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-09-25 17:09:40 +00:00
...[redacted]... - Open file descriptor soft limit set to 1024
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
...[redacted]... - Loaded plugin proxy.http_proxy.HttpProxyPlugin
2019-09-25 17:09:40 +00:00
...[redacted]... - Started 8 workers
...[redacted]... - Started server on ::1:8899
```
2021-11-08 22:23:17 +00:00
You can use single letter to customize log level. Example:
- `d = DEBUG`
- `i = INFO`
- `w = WARNING`
- `e = ERROR`
- `c = CRITICAL`
As we can see from the above logs, before starting up:
2019-09-25 17:09:40 +00:00
2021-11-08 22:23:17 +00:00
- `proxy.py` tried to set open file limit `ulimit` on the system
- Default value for `--open-file-limit` used is `1024`
- `--open-file-limit` flag is a no-op on `Windows` operating systems
2019-09-25 17:09:40 +00:00
See [flags ](#flags ) for full list of available configuration options.
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
## From command line using repo source
If you are trying to run `proxy.py` from source code,
there is no binary file named `proxy` in the source code.
2019-10-28 23:14:34 +00:00
To start `proxy.py` from source code follow these instructions:
- Clone repo
2021-11-06 15:19:55 +00:00
```console
2021-01-10 16:42:56 +00:00
❯
❯
```
2019-10-28 23:14:34 +00:00
- Create a Python 3 virtual env
2021-11-06 15:19:55 +00:00
```console
2021-01-10 16:42:56 +00:00
❯
❯
```
2019-10-28 23:14:34 +00:00
- Install deps
2021-11-06 15:19:55 +00:00
```console
2021-11-07 20:40:53 +00:00
❯
2021-01-10 16:42:56 +00:00
```
2019-10-28 23:14:34 +00:00
2021-11-07 20:40:53 +00:00
- Optionally, run tests
2019-10-28 23:14:34 +00:00
2021-11-06 15:19:55 +00:00
```console
2021-01-10 16:42:56 +00:00
❯
```
2019-10-28 23:14:34 +00:00
2021-11-08 22:23:17 +00:00
- Run `proxy.py`
2019-10-28 23:14:34 +00:00
2021-11-06 15:19:55 +00:00
```console
2021-01-10 16:42:56 +00:00
❯
```
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
2021-11-08 22:23:17 +00:00
See [Plugin Developer and Contributor Guide ](#plugin-developer-and-contributor-guide )
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
if you plan to work with `proxy.py` source code.
2019-09-25 17:09:40 +00:00
## Docker image
2019-02-09 05:13:34 +00:00
2021-11-08 22:23:17 +00:00
### Customize startup flags
2019-10-11 05:15:07 +00:00
2019-09-25 17:09:40 +00:00
By default `docker` binary is started with IPv4 networking flags:
2019-09-23 07:30:52 +00:00
2019-09-25 03:44:30 +00:00
--hostname 0.0.0.0 --port 8899
2019-09-23 07:30:52 +00:00
2021-11-08 22:23:17 +00:00
You can override flag from command line when starting the docker container. For example, to check `proxy.py` version within the docker container, run:
2019-09-23 07:30:52 +00:00
2019-12-22 04:57:29 +00:00
❯
2019-09-23 07:30:52 +00:00
-p 8899:8899 \
2019-10-10 22:15:27 +00:00
--rm abhinavsingh/proxy.py:latest \
2019-10-11 05:15:07 +00:00
-v
2019-09-23 07:30:52 +00:00
2021-01-10 16:42:56 +00:00
# Plugin Examples
2019-09-25 03:44:30 +00:00
2019-11-15 22:47:50 +00:00
- See [plugin ](https://github.com/abhinavsingh/proxy.py/tree/develop/proxy/plugin ) module for full code.
- All the bundled plugin examples also works with `https` traffic
2021-01-10 16:42:56 +00:00
- Require additional flags and certificate generation
- See [TLS Interception ](#tls-interception ).
2019-11-15 22:47:50 +00:00
- Plugin examples are also bundled with Docker image.
2021-01-10 16:42:56 +00:00
- See [Customize startup flags ](#customize-startup-flags ) to try plugins with Docker image.
2019-09-25 03:44:30 +00:00
2019-11-30 05:28:31 +00:00
## HTTP Proxy Plugins
### ShortLinkPlugin
2019-10-16 06:56:39 +00:00
Add support for short links in your favorite browsers / applications.
2019-12-01 06:54:20 +00:00
[](https://github.com/abhinavsingh/proxy.py#shortlinkplugin)
2019-12-01 06:04:43 +00:00
2019-10-16 06:56:39 +00:00
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.ShortLinkPlugin
2019-10-16 06:56:39 +00:00
```
Now you can speed up your daily browsing experience by visiting your
2021-01-10 16:42:56 +00:00
favorite website using single character domain names :). This works
2019-10-16 06:56:39 +00:00
across all browsers.
Following short links are enabled by default:
2021-01-10 16:42:56 +00:00
| Short Link | Destination URL |
| :--------: | :--------------: |
| a/ | amazon.com |
| i/ | instagram.com |
| l/ | linkedin.com |
| f/ | facebook.com |
| g/ | google.com |
| t/ | twitter.com |
| w/ | web.whatsapp.com |
| y/ | youtube.com |
| proxy/ | localhost:8899 |
2019-10-16 06:56:39 +00:00
2019-11-30 05:28:31 +00:00
### ModifyPostDataPlugin
2019-10-13 00:43:19 +00:00
Modifies POST request body before sending request to upstream server.
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.ModifyPostDataPlugin
2019-10-13 00:43:19 +00:00
```
By default plugin replaces POST body content with hardcoded `b'{"key": "modified"}'`
and enforced `Content-Type: application/json` .
Verify the same using `curl -x localhost:8899 -d '{"key": "value"}' http://httpbin.org/post`
2021-11-06 15:19:55 +00:00
```console
2019-10-13 00:43:19 +00:00
{
"args": {},
"data": "{\"key\": \"modified\"}",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Content-Length": "19",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "curl/7.54.0"
},
"json": {
"key": "modified"
},
"origin": "1.2.3.4, 5.6.7.8",
"url": "https://httpbin.org/post"
}
```
Note following from the response above:
1. POST data was modified `"data": "{\"key\": \"modified\"}"` .
Original `curl` command data was `{"key": "value"}` .
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
2. Our `curl` command did not add any `Content-Type` header,
2019-10-13 00:43:19 +00:00
but our plugin did add one `"Content-Type": "application/json"` .
Same can also be verified by looking at `json` field in the output above:
```
"json": {
"key": "modified"
},
```
3. Our plugin also added a `Content-Length` header to match length
of modified body.
2019-11-30 05:28:31 +00:00
### MockRestApiPlugin
2019-09-27 20:00:11 +00:00
Mock responses for your server REST API.
Use to test and develop client side applications
without need of an actual upstream REST API server.
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.ProposedRestApiPlugin
2019-09-27 20:00:11 +00:00
```
Verify mock API response using `curl -x localhost:8899 http://api.example.com/v1/users/`
2021-11-06 15:19:55 +00:00
```console
2019-09-27 20:00:11 +00:00
{"count": 2, "next": null, "previous": null, "results": [{"email": "you@example.com", "groups": [], "url": "api.example.com/v1/users/1/", "username": "admin"}, {"email": "someone@example.com", "groups": [], "url": "api.example.com/v1/users/2/", "username": "admin"}]}
```
Verify the same by inspecting `proxy.py` logs:
2021-11-06 15:19:55 +00:00
```console
2019-09-27 20:00:11 +00:00
2019-09-27 12:44:02,212 - INFO - pid:7077 - access_log:1210 - ::1:64792 - GET None:None/v1/users/ - None None - 0 byte
```
2021-01-10 16:42:56 +00:00
Access log shows `None:None` as server `ip:port` . `None` simply means that
2019-09-27 20:00:11 +00:00
the server connection was never made, since response was returned by our plugin.
2019-12-22 04:57:29 +00:00
Now modify `ProposedRestApiPlugin` to returns REST API mock
2019-09-27 20:00:11 +00:00
responses as expected by your clients.
2019-11-30 05:28:31 +00:00
### RedirectToCustomServerPlugin
2019-09-25 03:44:30 +00:00
2019-12-22 04:57:29 +00:00
Redirects all incoming `http` requests to custom web server.
By default, it redirects client requests to inbuilt web server,
2019-09-25 03:44:30 +00:00
also running on `8899` port.
Start `proxy.py` and enable inbuilt web server:
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-09-25 03:44:30 +00:00
--enable-web-server \
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.RedirectToCustomServerPlugin
2019-09-25 03:44:30 +00:00
```
Verify using `curl -v -x localhost:8899 http://google.com`
```
... [redacted] ...
< HTTP / 1 . 1 404 NOT FOUND
< Server: proxy . py v1 . 0 . 0
< Connection: Close
2019-12-22 04:57:29 +00:00
<
2019-09-25 03:44:30 +00:00
* Closing connection 0
```
2019-12-22 04:57:29 +00:00
Above `404` response was returned from `proxy.py` web server.
2019-09-25 03:44:30 +00:00
2019-12-22 04:57:29 +00:00
Verify the same by inspecting the logs for `proxy.py` .
2019-09-25 03:44:30 +00:00
Along with the proxy request log, you must also see a http web server request log.
```
2019-09-24 19:09:33,602 - INFO - pid:49996 - access_log:1241 - ::1:49525 - GET /
2019-09-24 19:09:33,603 - INFO - pid:49995 - access_log:1157 - ::1:49524 - GET localhost:8899/ - 404 NOT FOUND - 70 bytes
```
2019-11-30 05:28:31 +00:00
### FilterByUpstreamHostPlugin
2019-09-25 03:44:30 +00:00
2019-12-22 04:57:29 +00:00
Drops traffic by inspecting upstream host.
2021-11-06 15:19:55 +00:00
By default, plugin drops traffic for `facebook.com` and `www.facebok.com` .
2019-09-25 03:44:30 +00:00
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.FilterByUpstreamHostPlugin
2019-09-25 03:44:30 +00:00
```
2021-11-06 15:19:55 +00:00
Verify using `curl -v -x localhost:8899 http://facebook.com` :
2019-09-25 03:44:30 +00:00
2021-11-06 15:19:55 +00:00
```console
2019-09-25 03:44:30 +00:00
... [redacted] ...
< HTTP / 1 . 1 418 I ' m a tea pot
< Proxy-agent: proxy . py v1 . 0 . 0
* no chunk, no close, no size. Assume close to signal end
2019-12-22 04:57:29 +00:00
<
2019-09-25 03:44:30 +00:00
* Closing connection 0
```
Above `418 I'm a tea pot` is sent by our plugin.
Verify the same by inspecting logs for `proxy.py` :
2021-11-06 15:19:55 +00:00
```console
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
2019-09-24 19:21:37,893 - ERROR - pid:50074 - handle_readables:1347 - HttpProtocolException type raised
2019-09-25 03:44:30 +00:00
Traceback (most recent call last):
... [redacted] ...
2019-09-24 19:21:37,897 - INFO - pid:50074 - access_log:1157 - ::1:49911 - GET None:None/ - None None - 0 bytes
```
2019-11-30 05:28:31 +00:00
### CacheResponsesPlugin
2019-09-25 03:44:30 +00:00
Caches Upstream Server Responses.
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.CacheResponsesPlugin
2019-09-25 03:44:30 +00:00
```
Verify using `curl -v -x localhost:8899 http://httpbin.org/get` :
2021-11-06 15:19:55 +00:00
```console
2019-09-25 03:44:30 +00:00
... [redacted] ...
< HTTP / 1 . 1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Type: application / json
< Date: Wed , 25 Sep 2019 02:24:25 GMT
< Referrer-Policy: no-referrer-when-downgrade
< Server: nginx
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-XSS-Protection: 1 ; mode = block
< Content-Length: 202
< Connection: keep-alive
2019-12-22 04:57:29 +00:00
<
2019-09-25 03:44:30 +00:00
{
2019-10-13 00:43:19 +00:00
"args": {},
2019-09-25 03:44:30 +00:00
"headers": {
2019-10-13 00:43:19 +00:00
"Accept": "*/*",
"Host": "httpbin.org",
2019-09-25 03:44:30 +00:00
"User-Agent": "curl/7.54.0"
2019-10-13 00:43:19 +00:00
},
"origin": "1.2.3.4, 5.6.7.8",
2019-09-25 03:44:30 +00:00
"url": "https://httpbin.org/get"
}
* Connection #0 to host localhost left intact
```
Get path to the cache file from `proxy.py` logs:
2021-11-06 15:19:55 +00:00
```console
2019-09-25 03:44:30 +00:00
... [redacted] ... - GET httpbin.org:80/get - 200 OK - 556 bytes
... [redacted] ... - Cached response at /var/folders/k9/x93q0_xn1ls9zy76m2mf2k_00000gn/T/httpbin.org-1569378301.407512.txt
```
Verify contents of the cache file `cat /path/to/your/cache/httpbin.org.txt`
2021-11-06 15:19:55 +00:00
```console
2019-09-25 03:44:30 +00:00
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 25 Sep 2019 02:24:25 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 202
Connection: keep-alive
{
2019-10-13 00:43:19 +00:00
"args": {},
2019-09-25 03:44:30 +00:00
"headers": {
2019-10-13 00:43:19 +00:00
"Accept": "*/*",
"Host": "httpbin.org",
2019-09-25 03:44:30 +00:00
"User-Agent": "curl/7.54.0"
2019-10-13 00:43:19 +00:00
},
"origin": "1.2.3.4, 5.6.7.8",
2019-09-25 03:44:30 +00:00
"url": "https://httpbin.org/get"
}
```
2019-11-30 05:28:31 +00:00
### ManInTheMiddlePlugin
2019-09-25 03:44:30 +00:00
Modifies upstream server responses.
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.ManInTheMiddlePlugin
2019-09-25 03:44:30 +00:00
```
Verify using `curl -v -x localhost:8899 http://google.com` :
2021-11-06 15:19:55 +00:00
```console
2019-09-25 03:44:30 +00:00
... [redacted] ...
< HTTP / 1 . 1 200 OK
< Content-Length: 28
2019-12-22 04:57:29 +00:00
<
2019-09-25 03:44:30 +00:00
* Connection #0 to host localhost left intact
Hello from man in the middle
```
Response body `Hello from man in the middle` is sent by our plugin.
2019-12-10 03:38:49 +00:00
### ProxyPoolPlugin
Forward incoming proxy requests to a set of upstream proxy servers.
By default, `ProxyPoolPlugin` is hard-coded to use
`localhost:9000` and `localhost:9001` as upstream proxy server.
Let's start upstream proxies first.
Start `proxy.py` on port `9000` and `9001`
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-12-10 03:38:49 +00:00
```
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-12-10 03:38:49 +00:00
```
Now, start `proxy.py` with `ProxyPoolPlugin` (on default `8899` port):
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-12-10 03:38:49 +00:00
--plugins proxy.plugin.ProxyPoolPlugin
```
Make a curl request via `8899` proxy:
`curl -v -x localhost:8899 http://httpbin.org/get`
Verify that `8899` proxy forwards requests to upstream proxies
by checking respective logs.
2020-07-01 22:00:29 +00:00
### FilterByClientIpPlugin
2021-01-10 16:42:56 +00:00
Reject traffic from specific IP addresses. By default this
2020-07-01 22:00:29 +00:00
plugin blocks traffic from `127.0.0.1` and `::1` .
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2020-07-01 22:00:29 +00:00
❯
--plugins proxy.plugin.FilterByClientIpPlugin
```
Send a request using `curl -v -x localhost:8899 http://google.com` :
2021-11-06 15:19:55 +00:00
```console
2020-07-01 22:00:29 +00:00
... [redacted] ...
> Proxy-Connection: Keep-Alive
>
< HTTP / 1 . 1 418 I ' m a tea pot
< Connection: close
<
* Closing connection 0
```
Modify plugin to your taste e.g. Allow specific IP addresses only.
### ModifyChunkResponsePlugin
2021-01-10 16:42:56 +00:00
This plugin demonstrate how to modify chunked encoded responses. In able to do so, this plugin uses `proxy.py` core to parse the chunked encoded response. Then we reconstruct the response using custom hardcoded chunks, ignoring original chunks received from upstream server.
2020-07-01 22:00:29 +00:00
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2020-07-01 22:00:29 +00:00
❯
--plugins proxy.plugin.ModifyChunkResponsePlugin
```
Verify using `curl -v -x localhost:8899 http://httpbin.org/stream/5` :
2021-11-06 15:19:55 +00:00
```console
2020-07-01 22:00:29 +00:00
... [redacted] ...
modify
chunk
response
plugin
* Connection #0 to host localhost left intact
* Closing connection 0
```
Modify `ModifyChunkResponsePlugin` to your taste. Example, instead of sending hardcoded chunks, parse and modify the original `JSON` chunks received from the upstream server.
2021-11-05 02:06:02 +00:00
### CloudflareDnsResolverPlugin
This plugin uses `Cloudflare` hosted `DNS-over-HTTPS` [API ](https://developers.cloudflare.com/1.1.1.1/encrypted-dns/dns-over-https/make-api-requests/dns-json ) (json).
2021-11-06 15:19:55 +00:00
`DoH` mandates a HTTP2 compliant client. Unfortunately `proxy.py`
doesn't provide that yet, so we use a dependency. Install it:
```console
❯
```
Now start `proxy.py` as:
2021-11-05 02:06:02 +00:00
2021-11-06 15:19:55 +00:00
```console
2021-11-05 02:06:02 +00:00
❯
--plugins proxy.plugin.CloudflareDnsResolverPlugin
```
2021-11-06 15:19:55 +00:00
By default, `CloudflareDnsResolverPlugin` runs in `security` mode and provides malware protection.
Use `--cloudflare-dns-mode family` to also enable adult content protection too.
2021-11-05 02:06:02 +00:00
### CustomDnsResolverPlugin
This plugin demonstrate how to use a custom DNS resolution implementation with `proxy.py` .
This example plugin currently uses Python's in-built resolution mechanism. Customize code
to your taste. Example, query your custom DNS server, implement DoH or other mechanisms.
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2021-11-05 02:06:02 +00:00
❯
--plugins proxy.plugin.CustomDnsResolverPlugin
```
2021-11-06 15:19:55 +00:00
### CustomNetworkInterface
`HttpProxyBasePlugin.resolve_dns` callback can also be used to configure `network interface` which must be used as the `source_address` for connection to the upstream server.
See [this thread ](https://github.com/abhinavsingh/proxy.py/issues/535#issuecomment-961510862 )
for more details.
PS: There is no plugin named, but [CustomDnsResolverPlugin ](#customdnsresolverplugin )
can be easily customized according to your needs.
2021-11-05 02:06:02 +00:00
2019-11-30 05:28:31 +00:00
## HTTP Web Server Plugins
### Reverse Proxy
Extend in-built Web Server to add Reverse Proxy capabilities.
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2020-02-02 20:42:20 +00:00
❯
2019-11-30 05:28:31 +00:00
--plugins proxy.plugin.ReverseProxyPlugin
```
With default configuration, `ReverseProxyPlugin` plugin is equivalent to
following `Nginx` config:
2021-11-06 15:19:55 +00:00
```console
2019-11-30 05:28:31 +00:00
location /get {
proxy_pass http://httpbin.org/get
}
```
Verify using `curl -v localhost:8899/get` :
2021-11-06 15:19:55 +00:00
```console
2019-11-30 05:28:31 +00:00
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "localhost",
"User-Agent": "curl/7.64.1"
},
"origin": "1.2.3.4, 5.6.7.8",
"url": "https://localhost/get"
}
```
### Web Server Route
Demonstrates inbuilt web server routing using plugin.
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2020-02-02 20:42:20 +00:00
❯
2019-11-30 05:28:31 +00:00
--plugins proxy.plugin.WebServerPlugin
```
Verify using `curl -v localhost:8899/http-route-example` , should return:
2021-11-06 15:19:55 +00:00
```console
2019-11-30 05:28:31 +00:00
HTTP route response
```
2019-09-25 05:43:29 +00:00
## Plugin Ordering
2019-09-25 03:44:30 +00:00
2019-12-22 04:57:29 +00:00
When using multiple plugins, depending upon plugin functionality,
it might be worth considering the order in which plugins are passed
2019-09-25 03:44:30 +00:00
on the command line.
2019-12-22 04:57:29 +00:00
Plugins are called in the same order as they are passed. Example,
say we are using both `FilterByUpstreamHostPlugin` and
`RedirectToCustomServerPlugin` . Idea is to drop all incoming `http`
2021-11-06 15:19:55 +00:00
requests for `facebook.com` and `www.facebook.com` and redirect other
2019-09-25 03:44:30 +00:00
`http` requests to our inbuilt web server.
2019-12-22 04:57:29 +00:00
Hence, in this scenario it is important to use
`FilterByUpstreamHostPlugin` before `RedirectToCustomServerPlugin` .
2019-09-25 03:44:30 +00:00
If we enable `RedirectToCustomServerPlugin` before `FilterByUpstreamHostPlugin` ,
2021-11-06 15:19:55 +00:00
`facebook` requests will also get redirected to inbuilt web server,
2019-09-25 05:43:29 +00:00
instead of being dropped.
2019-09-25 03:44:30 +00:00
2021-01-10 16:42:56 +00:00
# End-to-End Encryption
2019-09-25 03:44:30 +00:00
2021-11-06 15:19:55 +00:00
By default, `proxy.py` uses `http` protocol for communication with clients e.g. `curl` , `browser` . For enabling end-to-end encrypting using `tls` / `https` first generate certificates. **Checkout** the repository and run:
2019-09-25 03:44:30 +00:00
2021-11-06 15:19:55 +00:00
```console
2019-09-25 03:44:30 +00:00
make https-certificates
```
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-09-25 03:44:30 +00:00
--cert-file https-cert.pem \
--key-file https-key.pem
```
Verify using `curl -x https://localhost:8899 --proxy-cacert https-cert.pem https://httpbin.org/get` :
2021-11-06 15:19:55 +00:00
```console
2019-09-25 03:44:30 +00:00
{
2019-10-13 00:43:19 +00:00
"args": {},
2019-09-25 03:44:30 +00:00
"headers": {
2019-10-13 00:43:19 +00:00
"Accept": "*/*",
"Host": "httpbin.org",
2019-09-25 03:44:30 +00:00
"User-Agent": "curl/7.54.0"
2019-10-13 00:43:19 +00:00
},
"origin": "1.2.3.4, 5.6.7.8",
2019-09-25 03:44:30 +00:00
"url": "https://httpbin.org/get"
}
```
2021-01-10 16:42:56 +00:00
If you want to avoid passing `--proxy-cacert` flag, also consider signing generated SSL certificates. Example:
2020-07-08 11:08:58 +00:00
First, generate CA certificates:
2021-11-06 15:19:55 +00:00
```console
2020-07-08 11:08:58 +00:00
make ca-certificates
```
Then, sign SSL certificate:
2021-11-06 15:19:55 +00:00
```console
2020-07-08 11:08:58 +00:00
make sign-https-certificates
```
2021-01-10 16:42:56 +00:00
Now restart the server with `--cert-file https-signed-cert.pem` flag. Note that you must also trust generated `ca-cert.pem` in your system keychain.
2020-07-08 11:08:58 +00:00
2021-01-10 16:42:56 +00:00
# TLS Interception
2019-09-25 03:44:30 +00:00
2019-12-22 04:57:29 +00:00
By default, `proxy.py` will not decrypt `https` traffic between client and server.
2020-03-25 08:00:19 +00:00
To enable TLS interception first generate root CA certificates:
2019-09-25 03:44:30 +00:00
2021-11-06 15:19:55 +00:00
```console
2020-03-25 08:00:19 +00:00
❯
2019-09-25 03:44:30 +00:00
```
2019-09-26 01:27:26 +00:00
Lets also enable `CacheResponsePlugin` so that we can verify decrypted
2021-01-10 16:42:56 +00:00
response from the server. Start `proxy.py` as:
2019-09-25 03:44:30 +00:00
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.CacheResponsesPlugin \
2019-09-25 03:44:30 +00:00
--ca-key-file ca-key.pem \
--ca-cert-file ca-cert.pem \
--ca-signing-key-file ca-signing-key.pem
```
2020-06-09 06:37:00 +00:00
[](https://github.com/abhinavsingh/proxy.py#flags) Also provide explicit CA bundle path needed for validation of peer certificates. See `--ca-file` flag.
2020-03-25 08:00:19 +00:00
Verify TLS interception using `curl`
2021-11-06 15:19:55 +00:00
```console
2020-03-25 08:00:19 +00:00
❯
```
2019-09-26 01:27:26 +00:00
2021-11-06 15:19:55 +00:00
```console
2019-09-26 01:27:26 +00:00
* issuer: C=US; ST=CA; L=SanFrancisco; O=proxy.py; OU=CA; CN=Proxy PY CA; emailAddress=proxyca@mailserver.com
* SSL certificate verify ok.
> GET /get HTTP/1.1
... [redacted] ...
< Connection: keep-alive
2019-12-22 04:57:29 +00:00
<
2019-09-26 01:27:26 +00:00
{
2019-10-13 00:43:19 +00:00
"args": {},
2019-09-26 01:27:26 +00:00
"headers": {
2019-10-13 00:43:19 +00:00
"Accept": "*/*",
"Host": "httpbin.org",
2019-09-26 01:27:26 +00:00
"User-Agent": "curl/7.54.0"
2019-10-13 00:43:19 +00:00
},
"origin": "1.2.3.4, 5.6.7.8",
2019-09-26 01:27:26 +00:00
"url": "https://httpbin.org/get"
}
```
The `issuer` line confirms that response was intercepted.
2021-01-10 16:42:56 +00:00
Also verify the contents of cached response file. Get path to the cache
2019-09-26 01:27:26 +00:00
file from `proxy.py` logs.
2019-12-22 04:57:29 +00:00
`❯
2019-09-25 03:44:30 +00:00
2021-11-06 15:19:55 +00:00
```console
2019-09-26 01:27:26 +00:00
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 25 Sep 2019 23:07:05 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 202
Connection: keep-alive
2019-09-25 03:44:30 +00:00
{
2019-10-13 00:43:19 +00:00
"args": {},
2019-09-25 03:44:30 +00:00
"headers": {
2019-10-13 00:43:19 +00:00
"Accept": "*/*",
"Host": "httpbin.org",
2019-09-25 03:44:30 +00:00
"User-Agent": "curl/7.54.0"
2019-10-13 00:43:19 +00:00
},
"origin": "1.2.3.4, 5.6.7.8",
2019-09-25 03:44:30 +00:00
"url": "https://httpbin.org/get"
}
```
2021-01-10 16:42:56 +00:00
Viola!!! If you remove CA flags, encrypted data will be found in the
2019-09-26 01:27:26 +00:00
cached file instead of plain text.
2019-12-22 04:57:29 +00:00
Now use CA flags with other
2019-10-10 05:36:47 +00:00
[plugin examples ](#plugin-examples ) to see them work with `https` traffic.
2019-09-25 05:43:29 +00:00
2020-06-17 12:24:43 +00:00
## TLS Interception With Docker
Important notes about TLS Interception with Docker container:
2021-01-10 16:42:56 +00:00
- Since `v2.2.0` , `proxy.py` docker container also ships with `openssl` . This allows `proxy.py`
to generate certificates on the fly for TLS Interception.
2020-06-17 12:24:43 +00:00
- For security reasons, `proxy.py` docker container doesn't ship with CA certificates.
Here is how to start a `proxy.py` docker container
with TLS Interception:
1. Generate CA certificates on host computer
2021-11-06 15:19:55 +00:00
```console
2021-01-10 16:42:56 +00:00
❯
```
2020-06-17 12:24:43 +00:00
2021-01-10 16:42:56 +00:00
2. Copy all generated certificates into a separate directory. We'll later mount this directory into our docker container
2020-06-17 12:24:43 +00:00
2021-11-06 15:19:55 +00:00
```console
2021-01-10 16:42:56 +00:00
❯
❯
```
2020-06-17 12:24:43 +00:00
3. Start docker container
2021-11-06 15:19:55 +00:00
```console
2021-01-10 16:42:56 +00:00
❯
-v /tmp/ca-certificates:/tmp/ca-certificates \
-p 8899:8899 \
abhinavsingh/proxy.py:latest \
--hostname 0.0.0.0 \
--plugins proxy.plugin.CacheResponsesPlugin \
--ca-key-file /tmp/ca-certificates/ca-key.pem \
--ca-cert-file /tmp/ca-certificates/ca-cert.pem \
--ca-signing-key /tmp/ca-certificates/ca-signing-key.pem
```
2020-06-17 12:24:43 +00:00
2021-01-10 16:42:56 +00:00
- `-v /tmp/ca-certificates:/tmp/ca-certificates` flag mounts our CA certificate directory in container environment
- `--plugins proxy.plugin.CacheResponsesPlugin` enables `CacheResponsesPlugin` so that we can inspect intercepted traffic
- `--ca-*` flags enable TLS Interception.
2020-06-17 12:24:43 +00:00
4. From another terminal, try TLS Interception using `curl` . You can omit `--cacert` flag if CA certificate is already trusted by the system.
2021-11-06 15:19:55 +00:00
```console
2021-01-10 16:42:56 +00:00
❯
--cacert ca-cert.pem \
-x 127.0.0.1:8899 \
https://httpbin.org/get
```
2020-06-17 12:24:43 +00:00
5. Verify `issuer` field from response headers.
2021-11-06 15:19:55 +00:00
```console
2021-01-10 16:42:56 +00:00
* Server certificate:
* subject: CN=httpbin.org; C=NA; ST=Unavailable; L=Unavailable; O=Unavailable; OU=Unavailable
* start date: Jun 17 09:26:57 2020 GMT
* expire date: Jun 17 09:26:57 2022 GMT
* subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
* issuer: CN=example.com
* SSL certificate verify ok.
```
2020-06-17 12:24:43 +00:00
6. Back on docker terminal, copy response dump path logs.
2021-11-06 15:19:55 +00:00
```console
2021-01-10 16:42:56 +00:00
...[redacted]... [I] access_log:338 - 172.17.0.1:56498 - CONNECT httpbin.org:443 - 1031 bytes - 1216.70 ms
...[redacted]... [I] close:49 - Cached response at /tmp/httpbin.org-ae1a927d064e4ab386ea319eb38fe251.txt
```
2020-06-17 12:24:43 +00:00
7. In another terminal, `cat` the response dump:
2021-11-06 15:19:55 +00:00
```console
2021-01-10 16:42:56 +00:00
❯
HTTP/1.1 200 OK
...[redacted]...
{
...[redacted]...,
"url": "http://httpbin.org/get"
}
```
2020-06-17 12:24:43 +00:00
2021-01-10 16:42:56 +00:00
# Proxy Over SSH Tunnel
2019-12-02 23:55:08 +00:00
2020-06-17 12:24:43 +00:00
**This is a WIP and may not work as documented**
2021-11-05 02:06:02 +00:00
Requires `paramiko` to work.
See [requirements-tunnel.txt ](https://github.com/abhinavsingh/proxy.py/blob/develop/requirements-tunnel.txt )
2019-12-02 23:55:08 +00:00
## Proxy Remote Requests Locally
|
+------------+ | +----------+
| LOCAL | | | REMOTE |
| HOST | < == SSH ==== :8900 == | SERVER |
+------------+ | +----------+
:8899 proxy.py |
|
FIREWALL
(allow tcp/22)
## What
Proxy HTTP(s) requests made on a `remote` server through `proxy.py` server
running on `localhost` .
### How
2021-01-10 16:42:56 +00:00
- Requested `remote` port is forwarded over the SSH connection.
- `proxy.py` running on the `localhost` handles and responds to
2019-12-02 23:55:08 +00:00
`remote` proxy requests.
### Requirements
1. `localhost` MUST have SSH access to the `remote` server
2. `remote` server MUST be configured to proxy HTTP(s) requests
through the forwarded port number e.g. `:8900` .
- `remote` and `localhost` ports CAN be same e.g. `:8899` .
- `:8900` is chosen in ascii art for differentiation purposes.
### Try it
Start `proxy.py` as:
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
❯
2019-12-02 23:55:08 +00:00
--tunnel-username username \
--tunnel-hostname ip.address.or.domain.name \
--tunnel-port 22 \
--tunnel-remote-host 127.0.0.1
--tunnel-remote-port 8899
```
2019-12-22 04:57:29 +00:00
Make a HTTP proxy request on `remote` server and
2019-12-02 23:55:08 +00:00
verify that response contains public IP address of `localhost` as origin:
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
❯
2019-12-02 23:55:08 +00:00
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "httpbin.org",
"User-Agent": "curl/7.54.0"
},
"origin": "x.x.x.x, y.y.y.y",
"url": "https://httpbin.org/get"
}
```
Also, verify that `proxy.py` logs on `localhost` contains `remote` IP as client IP.
2021-11-06 15:19:55 +00:00
```console
2019-12-02 23:55:08 +00:00
access_log:328 - remote:52067 - GET httpbin.org:80
```
## Proxy Local Requests Remotely
|
+------------+ | +----------+
| LOCAL | | | REMOTE |
| HOST | === SSH =====> | SERVER |
+------------+ | +----------+
| :8899 proxy.py
|
FIREWALL
(allow tcp/22)
2021-01-10 16:42:56 +00:00
# Embed proxy.py
2019-10-10 05:36:47 +00:00
2019-11-08 04:53:08 +00:00
## Blocking Mode
2019-11-14 00:38:59 +00:00
Start `proxy.py` in embedded mode with default configuration
2019-11-15 03:00:07 +00:00
by using `proxy.main` method. Example:
2019-10-10 05:36:47 +00:00
2019-12-22 04:57:29 +00:00
```python
2019-11-14 00:38:59 +00:00
import proxy
2019-10-10 05:36:47 +00:00
2019-10-28 23:14:34 +00:00
if __name__ == '__main__':
2019-11-14 00:38:59 +00:00
proxy.main()
2019-10-10 05:36:47 +00:00
```
2019-10-16 06:56:39 +00:00
2019-11-14 00:38:59 +00:00
Customize startup flags by passing list of input arguments:
2019-10-02 07:09:35 +00:00
2019-12-22 04:57:29 +00:00
```python
2019-11-14 00:38:59 +00:00
import proxy
2019-10-02 07:09:35 +00:00
if __name__ == '__main__':
2019-11-14 00:38:59 +00:00
proxy.main([
2019-10-28 23:14:34 +00:00
'--hostname', '::1',
'--port', '8899'
])
2019-10-02 07:09:35 +00:00
```
2019-11-14 00:38:59 +00:00
or, customize startup flags by passing them as kwargs:
2019-12-22 04:57:29 +00:00
```python
2019-11-14 00:38:59 +00:00
import ipaddress
import proxy
if __name__ == '__main__':
proxy.main(
hostname=ipaddress.IPv6Address('::1'),
port=8899
)
```
2019-11-08 04:53:08 +00:00
Note that:
1. Calling `main` is simply equivalent to starting `proxy.py` from command line.
2. `main` will block until `proxy.py` shuts down.
## Non-blocking Mode
2019-11-14 00:38:59 +00:00
Start `proxy.py` in non-blocking embedded mode with default configuration
2021-11-06 15:19:55 +00:00
by using `Proxy` context manager: Example:
2019-11-08 04:53:08 +00:00
2019-12-22 04:57:29 +00:00
```python
2019-11-14 00:38:59 +00:00
import proxy
2019-11-08 04:53:08 +00:00
if __name__ == '__main__':
2021-11-04 21:41:22 +00:00
with proxy.Proxy([]) as p:
2019-11-08 04:53:08 +00:00
# ... your logic here ...
```
Note that:
2021-11-04 21:41:22 +00:00
1. `Proxy` is similar to `main` , except `Proxy` does not block.
2021-11-06 15:19:55 +00:00
2. Internally `Proxy` is a context manager.
3. It will start `proxy.py` when called and will shut it down
2021-11-04 21:41:22 +00:00
once the scope ends.
2021-11-06 15:19:55 +00:00
4. Just like `main` , startup flags with `Proxy`
2019-11-14 00:38:59 +00:00
can be customized by either passing flags as list of
2021-11-04 21:41:22 +00:00
input arguments e.g. `Proxy(['--port', '8899'])` or
by using passing flags as kwargs e.g. `Proxy(port=8899)` .
## Ephemeral Port
Use `--port=0` to bind `proxy.py` on a random port allocated by the kernel.
In embedded mode, you can access this port. Example:
```python
import proxy
if __name__ == '__main__':
with proxy.Proxy([]) as p:
print(p.pool.flags.port)
```
`pool.flags.port` will give you access to the random port allocated by the kernel.
2019-11-08 04:53:08 +00:00
2020-08-14 11:50:19 +00:00
## Loading Plugins
2021-11-06 15:19:55 +00:00
Users can use `--plugins` flag multiple times to load multiple plugins.
See [Unable to load plugins ](#unable-to-load-plugins ) if you are running into issues.
When using in embedded mode, you have a few more options. Example:
1. Provide a fully-qualified name of the plugin class as `bytes` to the `proxy.main` method or `proxy.Proxy` context manager.
2. Provide `type` instance of the plugin class. This is especially useful if you plan to define plugins at runtime.
Example, load a single plugin using `--plugins` flag:
2020-08-14 11:50:19 +00:00
```python
import proxy
if __name__ == '__main__':
proxy.main([
'--plugins', 'proxy.plugin.CacheResponsesPlugin',
])
```
2021-11-06 15:19:55 +00:00
For simplicity, you can also pass the list of plugins as a keyword argument to `proxy.main` or the `Proxy` constructor.
Example:
2020-08-14 11:50:19 +00:00
```python
import proxy
from proxy.plugin import FilterByUpstreamHostPlugin
if __name__ == '__main__':
proxy.main([], plugins=[
b'proxy.plugin.CacheResponsesPlugin',
FilterByUpstreamHostPlugin,
])
```
2021-01-10 16:42:56 +00:00
# Unit testing with proxy.py
2019-11-12 20:25:47 +00:00
2019-11-14 00:38:59 +00:00
## proxy.TestCase
2019-11-12 20:25:47 +00:00
2021-11-04 21:41:22 +00:00
To setup and teardown `proxy.py` for your Python `unittest` classes,
2019-11-14 00:38:59 +00:00
simply use `proxy.TestCase` instead of `unittest.TestCase` .
2019-11-12 20:25:47 +00:00
Example:
2019-12-22 04:57:29 +00:00
```python
2019-11-14 00:38:59 +00:00
import proxy
2019-11-12 20:25:47 +00:00
2019-11-14 00:38:59 +00:00
class TestProxyPyEmbedded(proxy.TestCase):
2019-11-12 20:25:47 +00:00
def test_my_application_with_proxy(self) -> None:
self.assertTrue(True)
```
Note that:
2019-11-14 00:38:59 +00:00
1. `proxy.TestCase` overrides `unittest.TestCase.run()` method to setup and teardown `proxy.py` .
2019-11-12 20:25:47 +00:00
2. `proxy.py` server will listen on a random available port on the system.
2021-11-04 21:41:22 +00:00
This random port is available as `self.PROXY.pool.flags.port` within your test cases.
2019-11-12 20:25:47 +00:00
3. Only a single worker is started by default (`--num-workers 1`) for faster setup and teardown.
2019-11-14 00:38:59 +00:00
4. Most importantly, `proxy.TestCase` also ensures `proxy.py` server
2019-11-27 06:43:05 +00:00
is up and running before proceeding with execution of tests. By default,
2019-12-22 04:57:29 +00:00
`proxy.TestCase` will wait for `10 seconds` for `proxy.py` server to start,
2019-11-27 06:43:05 +00:00
upon failure a `TimeoutError` exception will be raised.
2019-11-12 20:25:47 +00:00
## Override startup flags
To override default startup flags, define a `PROXY_PY_STARTUP_FLAGS` variable in your test class.
Example:
2019-12-22 04:57:29 +00:00
```python
2019-11-12 20:25:47 +00:00
class TestProxyPyEmbedded(TestCase):
PROXY_PY_STARTUP_FLAGS = [
'--num-workers', '1',
'--enable-web-server',
]
def test_my_application_with_proxy(self) -> None:
self.assertTrue(True)
```
See [test_embed.py ](https://github.com/abhinavsingh/proxy.py/blob/develop/tests/test_embed.py )
for full working example.
## With unittest.TestCase
2019-11-14 00:38:59 +00:00
If for some reasons you are unable to directly use `proxy.TestCase` ,
2019-11-12 20:25:47 +00:00
then simply override `unittest.TestCase.run` yourself to setup and teardown `proxy.py` .
Example:
2019-12-22 04:57:29 +00:00
```python
2019-11-12 20:25:47 +00:00
import unittest
2019-11-14 00:38:59 +00:00
import proxy
2019-11-12 20:25:47 +00:00
class TestProxyPyEmbedded(unittest.TestCase):
def test_my_application_with_proxy(self) -> None:
self.assertTrue(True)
def run(self, result: Optional[unittest.TestResult] = None) -> Any:
2019-11-14 00:38:59 +00:00
with proxy.start([
2019-11-12 20:25:47 +00:00
'--num-workers', '1',
'--port', '... random port ...']):
super().run(result)
```
2019-12-22 04:57:29 +00:00
or simply setup / teardown `proxy.py` within
2019-11-27 06:43:05 +00:00
`setUpClass` and `teardownClass` class methods.
2021-01-10 16:42:56 +00:00
# Utilities
2019-10-28 23:14:34 +00:00
## TCP Sockets
2020-01-07 03:51:18 +00:00
### new_socket_connection
2019-10-28 23:14:34 +00:00
Attempts to create an IPv4 connection, then IPv6 and
finally a dual stack connection to provided address.
2019-12-22 04:57:29 +00:00
```python
2019-10-28 23:14:34 +00:00
>>> conn = new_socket_connection(('httpbin.org', 80))
>>> ...[ use connection ]...
>>> conn.close()
```
2020-01-07 03:51:18 +00:00
### socket_connection
2019-10-28 23:14:34 +00:00
`socket_connection` is a convenient decorator + context manager
around `new_socket_connection` which ensures `conn.close` is implicit.
As a context manager:
2019-12-22 04:57:29 +00:00
```python
2019-10-28 23:14:34 +00:00
>>> with socket_connection(('httpbin.org', 80)) as conn:
>>> ... [ use connection ] ...
```
As a decorator:
2019-12-22 04:57:29 +00:00
```python
2019-10-28 23:14:34 +00:00
>>> @socket_connection (('httpbin.org', 80))
>>> def my_api_call(conn, *args, * *kwargs):
>>> ... [ use connection ] ...
```
## Http Client
2020-01-07 03:51:18 +00:00
### build_http_request
2019-10-28 23:14:34 +00:00
2021-11-08 22:23:17 +00:00
- Generate HTTP GET request
2019-10-28 23:14:34 +00:00
2021-11-08 22:23:17 +00:00
```python
>>> build_http_request(b'GET', b'/')
b'GET / HTTP/1.1\r\n\r\n'
```
2019-10-28 23:14:34 +00:00
2021-11-08 22:23:17 +00:00
- Generate HTTP GET request with headers
2019-10-28 23:14:34 +00:00
2021-11-08 22:23:17 +00:00
```python
>>> build_http_request(b'GET', b'/',
headers={b'Connection': b'close'})
b'GET / HTTP/1.1\r\nConnection: close\r\n\r\n'
```
2019-10-28 23:14:34 +00:00
2021-11-08 22:23:17 +00:00
- Generate HTTP POST request with headers and body
2019-10-28 23:14:34 +00:00
2021-11-08 22:23:17 +00:00
```python
>>> import json
>>> build_http_request(b'POST', b'/form',
headers={b'Content-type': b'application/json'},
body=proxy.bytes_(json.dumps({'email': 'hello@world.com'})))
b'POST /form HTTP/1.1\r\nContent-type: application/json\r\n\r\n{"email": "hello@world.com"}'
```
2019-10-28 23:14:34 +00:00
2020-01-07 03:51:18 +00:00
### build_http_response
2019-10-28 23:14:34 +00:00
2020-02-03 04:49:01 +00:00
```python
build_http_response(
status_code: int,
protocol_version: bytes = HTTP_1_1,
reason: Optional[bytes] = None,
headers: Optional[Dict[bytes, bytes]] = None,
body: Optional[bytes] = None) -> bytes
```
2019-10-28 23:14:34 +00:00
2020-01-07 03:51:18 +00:00
## PKI
2019-10-28 23:14:34 +00:00
2020-01-07 03:51:18 +00:00
### API Usage
2019-10-28 23:14:34 +00:00
2021-11-08 22:23:17 +00:00
- gen_private_key
2020-02-03 04:49:01 +00:00
2021-11-08 22:23:17 +00:00
```python
gen_private_key(
key_path: str,
password: str,
bits: int = 2048,
timeout: int = 10) -> bool
```
2020-02-03 04:49:01 +00:00
2021-11-08 22:23:17 +00:00
- gen_public_key
```python
gen_public_key(
public_key_path: str,
private_key_path: str,
private_key_password: str,
subject: str,
alt_subj_names: Optional[List[str]] = None,
extended_key_usage: Optional[str] = None,
validity_in_days: int = 365,
timeout: int = 10) -> bool
```
2020-02-03 04:49:01 +00:00
2021-11-08 22:23:17 +00:00
- remove_passphrase
2020-02-03 04:49:01 +00:00
2021-11-08 22:23:17 +00:00
```python
remove_passphrase(
key_in_path: str,
password: str,
key_out_path: str,
timeout: int = 10) -> bool
```
2020-02-03 04:49:01 +00:00
2021-11-08 22:23:17 +00:00
- gen_csr
2020-02-03 04:49:01 +00:00
2021-11-08 22:23:17 +00:00
```python
gen_csr(
csr_path: str,
key_path: str,
password: str,
crt_path: str,
timeout: int = 10) -> bool
```
2019-10-28 23:14:34 +00:00
2021-11-08 22:23:17 +00:00
- sign_csr
```python
sign_csr(
csr_path: str,
crt_path: str,
ca_key_path: str,
ca_key_password: str,
ca_crt_path: str,
serial: str,
alt_subj_names: Optional[List[str]] = None,
extended_key_usage: Optional[str] = None,
validity_in_days: int = 365,
timeout: int = 10) -> bool
```
2020-02-03 04:49:01 +00:00
See [pki.py ](https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/common/pki.py ) and
[test_pki.py ](https://github.com/abhinavsingh/proxy.py/blob/develop/tests/common/test_pki.py )
2020-01-07 19:38:25 +00:00
for usage examples.
2019-10-28 23:14:34 +00:00
2020-01-07 03:51:18 +00:00
### CLI Usage
Use `proxy.common.pki` module for:
2021-01-10 16:42:56 +00:00
1. Generation of public and private keys
2. Generating CSR requests
3. Signing CSR requests using custom CA.
2020-01-07 03:51:18 +00:00
2021-11-06 15:19:55 +00:00
```console
2020-01-07 03:51:18 +00:00
python -m proxy.common.pki -h
usage: pki.py [-h] [--password PASSWORD] [--private-key-path PRIVATE_KEY_PATH]
[--public-key-path PUBLIC_KEY_PATH] [--subject SUBJECT]
action
2020-06-09 06:37:00 +00:00
proxy.py v2.2.0 : PKI Utility
2020-01-07 03:51:18 +00:00
positional arguments:
action Valid actions: remove_passphrase, gen_private_key,
gen_public_key, gen_csr, sign_csr
optional arguments:
-h, --help show this help message and exit
--password PASSWORD Password to use for encryption. Default: proxy.py
--private-key-path PRIVATE_KEY_PATH
Private key path
--public-key-path PUBLIC_KEY_PATH
Public key path
--subject SUBJECT Subject to use for public key generation. Default:
/CN=example.com
```
2019-10-28 23:14:34 +00:00
2019-10-02 07:09:35 +00:00
## Internal Documentation
2021-11-08 22:23:17 +00:00
Code is well documented. Browse through internal class hierarchy and documentation using `pydoc3`
2019-10-02 07:09:35 +00:00
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯
2019-10-02 07:09:35 +00:00
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
PACKAGE CONTENTS
__main__
common (package)
core (package)
http (package)
main
FILE
/Users/abhinav/Dev/proxy.py/proxy/__init__.py
2019-10-02 07:09:35 +00:00
```
2021-10-29 23:02:05 +00:00
# Run Dashboard
2021-11-05 02:06:02 +00:00
Dashboard is currently under development and not yet bundled with `pip` packages.
To run dashboard, you must checkout the source.
2021-10-29 23:02:05 +00:00
Dashboard is written in Typescript and SCSS, so let's build it first using:
2021-11-06 15:19:55 +00:00
```console
❯
2021-10-29 23:02:05 +00:00
```
2021-11-07 17:32:34 +00:00
Also build the embedded `Chrome DevTools` if you plan on using it:
```console
❯
```
2021-10-29 23:02:05 +00:00
Now start `proxy.py` with dashboard plugin and by overriding root directory for static server:
2021-11-06 15:19:55 +00:00
```console
❯
2021-10-29 23:02:05 +00:00
...[redacted]... - Loaded plugin proxy.http.server.HttpWebServerPlugin
...[redacted]... - Loaded plugin proxy.dashboard.dashboard.ProxyDashboard
...[redacted]... - Loaded plugin proxy.dashboard.inspect_traffic.InspectTrafficPlugin
...[redacted]... - Loaded plugin proxy.http.inspector.DevtoolsProtocolPlugin
...[redacted]... - Loaded plugin proxy.http.proxy.HttpProxyPlugin
...[redacted]... - Listening on ::1:8899
...[redacted]... - Core Event enabled
```
Currently, enabling dashboard will also enable all the dashboard plugins.
Visit dashboard:
2021-11-06 15:19:55 +00:00
```console
❯
2021-10-29 23:02:05 +00:00
```
## Inspect Traffic
2021-11-08 22:23:17 +00:00
***This is a WIP and may not work as documented***
2021-11-05 02:06:02 +00:00
Wait for embedded `Chrome Dev Console` to load. Currently, detail about all traffic flowing
through `proxy.py` is pushed to the `Inspect Traffic` tab. However, received payloads are not
yet integrated with the embedded dev console.
2021-10-29 23:02:05 +00:00
2021-11-05 02:06:02 +00:00
Current functionality can be verified by opening the `Dev Console` of dashboard and inspecting
the websocket connection that dashboard established with the `proxy.py` server.
2021-10-29 23:02:05 +00:00
2021-10-31 03:35:35 +00:00
[](https://github.com/abhinavsingh/proxy.py)
2021-10-29 23:02:05 +00:00
2021-11-06 15:19:55 +00:00
# Chrome DevTools Protocol
For scenarios where you want direct access to `Chrome DevTools` protocol websocket endpoint,
start `proxy.py` as:
```console
❯
```
Now point your CDT instance to `ws://localhost:8899/devtools` .
2021-01-10 16:42:56 +00:00
# Frequently Asked Questions
2019-10-02 07:09:35 +00:00
2020-01-07 05:23:38 +00:00
## Threads vs Threadless
2021-11-08 22:23:17 +00:00
### `v1.x`
2020-01-07 05:23:38 +00:00
2021-11-08 20:04:17 +00:00
`proxy.py` used to spawn new threads for handling client requests.
2020-01-07 05:23:38 +00:00
2021-11-08 22:23:17 +00:00
### `v2.0+`
2020-01-07 19:38:25 +00:00
2021-11-08 20:04:17 +00:00
`proxy.py` added support for threadless execution of client requests using `asyncio` .
2021-11-08 22:23:17 +00:00
### `v2.4.0+`
Threadless execution was turned ON by default for `Python 3.8+` on `mac` and `linux` environments.
`proxy.py` threadless execution has been reported safe on these environments by our users. If you are running into trouble, fallback to threaded mode using `--threaded` flag.
2021-11-08 20:04:17 +00:00
2021-11-08 22:23:17 +00:00
For `windows` and `Python < 3.8` , you can still try out threadless mode by starting `proxy.py` with `--threadless` flag.
2021-11-08 20:04:17 +00:00
2021-11-08 22:23:17 +00:00
If threadless works for you, consider sending a PR by editing `_env_threadless_compliant` method in the `proxy/common/constants.py` file.
2020-01-07 05:23:38 +00:00
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
## SyntaxError: invalid syntax
2020-02-03 04:49:01 +00:00
`proxy.py` is strictly typed and uses Python `typing` annotations. Example:
```python
>>> my_strings : List[str] = []
>>> #############^^^^^^^^^#####
```
Hence a Python version that understands typing annotations is required.
2020-03-24 15:39:32 +00:00
Make sure you are using `Python 3.6+` .
2020-02-03 04:49:01 +00:00
Verify the version before running `proxy.py` :
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
2019-12-22 04:57:29 +00:00
`❯
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
2020-02-03 04:49:01 +00:00
All `typing` annotations can be replaced with `comment-only` annotations. Example:
```python
>>> my_strings = [] # List[str]
>>> ################^^^^^^^^^^^
```
2020-03-24 15:39:32 +00:00
It will enable `proxy.py` to run on Python `pre-3.6` , even on `2.7` .
2020-02-03 04:49:01 +00:00
However, as all future versions of Python will support `typing` annotations,
this has not been considered.
2019-11-12 23:29:37 +00:00
## Unable to load plugins
2021-01-10 16:42:56 +00:00
Make sure plugin modules are discoverable by adding them to `PYTHONPATH` . Example:
2019-11-12 23:29:37 +00:00
`PYTHONPATH=/path/to/my/app proxy --plugins my_app.proxyPlugin`
2021-11-06 15:19:55 +00:00
```console
2019-11-12 23:29:37 +00:00
...[redacted]... - Loaded plugin proxy.HttpProxyPlugin
...[redacted]... - Loaded plugin my_app.proxyPlugin
```
2019-11-15 22:47:50 +00:00
OR, simply pass fully-qualified path as parameter, e.g.
2019-11-12 23:29:37 +00:00
`proxy --plugins /path/to/my/app/my_app.proxyPlugin`
2019-10-02 07:09:35 +00:00
## Unable to connect with proxy.py from remote host
Make sure `proxy.py` is listening on correct network interface.
Try following flags:
- For IPv6 `--hostname ::`
- For IPv4 `--hostname 0.0.0.0`
## Basic auth not working with a browser
Most likely it's a browser integration issue with system keychain.
- First verify that basic auth is working using `curl`
2021-01-10 16:42:56 +00:00
`curl -v -x username:password@localhost:8899 https://httpbin.org/get`
2019-10-02 07:09:35 +00:00
- See [this thread ](https://github.com/abhinavsingh/proxy.py/issues/89#issuecomment-534845710 )
for further details.
## Docker image not working on macOS
It's a compatibility issue with `vpnkit` .
See [moby/vpnkit exhausts docker resources ](https://github.com/abhinavsingh/proxy.py/issues/43 )
and [Connection refused: The proxy could not connect ](https://github.com/moby/vpnkit/issues/469 )
for some background.
2019-10-10 05:36:47 +00:00
## GCE log viewer integration for proxy.py
A starter [fluentd.conf ](https://github.com/abhinavsingh/proxy.py/blob/develop/fluentd.conf )
template is available.
1. Copy this configuration file as `proxy.py.conf` under
`/etc/google-fluentd/config.d/`
2. Update `path` field to log file path as used with `--log-file` flag.
By default `/tmp/proxy.log` path is tailed.
3. Reload `google-fluentd` :
`sudo service google-fluentd restart`
Now `proxy.py` logs can be browsed using
[GCE log viewer ](https://console.cloud.google.com/logs/viewer ).
## ValueError: filedescriptor out of range in select
2019-10-28 23:14:34 +00:00
`proxy.py` is made to handle thousands of connections per second
without any socket leaks.
2019-10-10 05:36:47 +00:00
1. Make use of `--open-file-limit` flag to customize `ulimit -n` .
2. Make sure to adjust `--backlog` flag for higher concurrency.
If nothing helps, [open an issue ](https://github.com/abhinavsingh/proxy.py/issues/new )
with `requests per second` sent and output of following debug script:
2021-11-06 15:19:55 +00:00
```console
2019-12-22 04:57:29 +00:00
❯ < proxy-py-pid >
2019-10-10 05:36:47 +00:00
```
2019-09-25 22:51:12 +00:00
2019-11-19 04:45:51 +00:00
## None:None in access logs
2021-01-10 16:42:56 +00:00
Sometimes you may see `None:None` in access logs. It simply means
2019-11-19 04:45:51 +00:00
that an upstream server connection was never established i.e.
`upstream_host=None` , `upstream_port=None` .
There can be several reasons for no upstream connection,
few obvious ones include:
2019-12-22 04:57:29 +00:00
1. Client established a connection but never completed the request.
2019-11-19 04:45:51 +00:00
2. A plugin returned a response prematurely, avoiding connection to upstream server.
2021-11-06 21:20:11 +00:00
## OSError when wrapping client for TLS Interception
With `TLS Interception` on, you might occasionally see following exceptions:
```console
2021-11-06 23:33:34,540 - pid:91032 [E] server.intercept:678 - OSError when wrapping client
Traceback (most recent call last):
...[redacted]...
...[redacted]...
...[redacted]...
ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:997)
...[redacted]... - CONNECT oauth2.googleapis.com:443 - 0 bytes - 272.08 ms
```
Some clients can throw `TLSV1_ALERT_UNKNOWN_CA` if they cannot verify the certificate of the server
because it is signed by an unknown issuer CA. Which is the case when we are doing TLS interception.
This can be for a variety of reasons e.g. certificate pinning etc.
Another exception you might see is `CERTIFICATE_VERIFY_FAILED` :
```console
2021-11-06 23:36:02,002 - pid:91033 [E] handler.handle_readables:293 - Exception while receiving from client connection < socket.socket fd = 28, family = AddressFamily.AF_INET, type = SocketKind.SOCK_STREAM, proto = 0, laddr = ('127.0.0.1', 8899 ) , raddr = ('127.0.0.1', 51961 ) > with reason SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')
Traceback (most recent call last):
...[redacted]...
...[redacted]...
...[redacted]...
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)
...[redacted]... - CONNECT init.push.apple.com:443 - 0 bytes - 892.99 ms
```
In future, we might support serving original HTTPS content for such clients while still
performing TLS interception in the background. This will keep the clients happy without
impacting our ability to TLS intercept. Unfortunately, this feature is currently not available.
Another example with `SSLEOFError` exception:
```console
2021-11-06 23:46:40,446 - pid:91034 [E] server.intercept:678 - OSError when wrapping client
Traceback (most recent call last):
...[redacted]...
...[redacted]...
...[redacted]...
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:997)
...[redacted]... - CONNECT stock.adobe.io:443 - 0 bytes - 685.32 ms
```
2021-11-06 15:19:55 +00:00
# Plugin Developer and Contributor Guide
## High level architecture
```console
+-------------+
2021-11-08 16:52:23 +00:00
| |
2021-11-06 15:19:55 +00:00
| Proxy([]) |
2021-11-08 16:52:23 +00:00
| |
2021-11-06 15:19:55 +00:00
+------+------+
|
|
+-----------v--------------+
2021-11-08 16:52:23 +00:00
| |
2021-11-06 15:19:55 +00:00
| AcceptorPool(...) |
2021-11-08 16:52:23 +00:00
| |
2021-11-06 15:19:55 +00:00
+------------+-------------+
|
+-----------------+ | +-----------------+
2021-11-08 16:52:23 +00:00
| | | | |
2021-11-06 15:19:55 +00:00
| Acceptor(..) < ------------- + ----------- > Acceptor(..) |
2021-11-08 16:52:23 +00:00
| | | |
+---+-------------+ +---------+-------+
| |
| |
| +------++------++------++------++------+ |
| | || || || || | |
+----> || || || || < ----- +
| || || || || |
+------++------++------++------++------+
Threadless Worker Processes
2021-11-06 15:19:55 +00:00
```
`proxy.py` is made with performance in mind. By default, `proxy.py`
will try to utilize all available CPU cores to it for accepting new
client connections. This is achieved by starting `AcceptorPool` which
listens on configured server port. Then, `AcceptorPool` starts `Acceptor`
processes (`--num-workers`) to accept incoming client connections.
Each `Acceptor` process delegates the accepted client connection
to a `Work` class. Currently, `HttpProtocolHandler` is the default
work klass hardcoded into the code.
`HttpProtocolHandler` simply assumes that incoming clients will follow
HTTP specification. Specific HTTP proxy and HTTP server implementations
are written as plugins of `HttpProtocolHandler` .
See documentation of `HttpProtocolHandlerPlugin` for available lifecycle hooks.
Use `HttpProtocolHandlerPlugin` to add new features for http(s) clients. Example,
See `HttpWebServerPlugin` .
## Everything is a plugin
Within `proxy.py` everything is a plugin.
- We enabled `proxy server` plugins using `--plugins` flag.
Proxy server `HttpProxyPlugin` is a plugin of `HttpProtocolHandler` .
Further, Proxy server allows plugin through `HttpProxyBasePlugin` specification.
- All the proxy server [plugin examples ](#plugin-examples ) were implementing
`HttpProxyBasePlugin` . See documentation of `HttpProxyBasePlugin` for available
lifecycle hooks. Use `HttpProxyBasePlugin` to modify behavior of http(s) proxy protocol
between client and upstream server. Example,
[FilterByUpstreamHostPlugin ](#filterbyupstreamhostplugin ).
- We also enabled inbuilt `web server` using `--enable-web-server` .
Web server `HttpWebServerPlugin` is a plugin of `HttpProtocolHandler`
and implements `HttpProtocolHandlerPlugin` specification.
- There also is a `--disable-http-proxy` flag. It disables inbuilt proxy server.
Use this flag with `--enable-web-server` flag to run `proxy.py` as a programmable
http(s) server.
## Development Guide
### Setup Local Environment
Contributors must start `proxy.py` from source to verify and develop new features / fixes.
See [Run proxy.py from command line using repo source ](#from-command-line-using-repo-source ) for details.
[](https://github.com/abhinavsingh/proxy.py/issues/642#issuecomment-960819271) On `macOS`
you must install `Python` using `pyenv` , as `Python` installed via `homebrew` tends
to be problematic. See linked thread for more details.
### Setup Git Hooks
Pre-commit hook ensures tests are passing.
1. `cd /path/to/proxy.py`
2. `ln -s $(PWD)/git-pre-commit .git/hooks/pre-commit`
Pre-push hook ensures lint and tests are passing.
1. `cd /path/to/proxy.py`
2. `ln -s $(PWD)/git-pre-push .git/hooks/pre-push`
### Sending a Pull Request
Every pull request is tested using GitHub actions.
See [GitHub workflow ](https://github.com/abhinavsingh/proxy.py/tree/develop/.github/workflows )
for list of tests.
2021-01-10 16:42:56 +00:00
# Flags
2013-08-31 09:19:22 +00:00
2021-11-06 15:19:55 +00:00
```console
2019-10-28 23:14:34 +00:00
❯
2021-11-08 20:04:17 +00:00
usage: proxy [-h] [--enable-events] [--enable-conn-pool] [--threadless] [--threaded]
[--pid-file PID_FILE] [--backlog BACKLOG] [--hostname HOSTNAME]
[--port PORT] [--num-workers NUM_WORKERS]
[--unix-socket-path UNIX_SOCKET_PATH]
[--client-recvbuf-size CLIENT_RECVBUF_SIZE] [--key-file KEY_FILE]
[--timeout TIMEOUT] [--version] [--log-level LOG_LEVEL]
[--log-file LOG_FILE] [--log-format LOG_FORMAT]
[--open-file-limit OPEN_FILE_LIMIT] [--plugins PLUGINS] [--enable-dashboard]
[--disable-http-proxy] [--ca-key-file CA_KEY_FILE]
[--ca-cert-dir CA_CERT_DIR] [--ca-cert-file CA_CERT_FILE]
[--ca-file CA_FILE] [--ca-signing-key-file CA_SIGNING_KEY_FILE]
[--cert-file CERT_FILE] [--disable-headers DISABLE_HEADERS]
[--server-recvbuf-size SERVER_RECVBUF_SIZE] [--basic-auth BASIC_AUTH]
[--cache-dir CACHE_DIR] [--filtered-upstream-hosts FILTERED_UPSTREAM_HOSTS]
[--enable-web-server] [--enable-static-server]
[--static-server-dir STATIC_SERVER_DIR] [--pac-file PAC_FILE]
[--pac-file-url-path PAC_FILE_URL_PATH]
2021-11-07 21:51:42 +00:00
[--filtered-client-ips FILTERED_CLIENT_IPS]
[--filtered-url-regex-config FILTERED_URL_REGEX_CONFIG]
[--cloudflare-dns-mode CLOUDFLARE_DNS_MODE]
2021-01-10 16:42:56 +00:00
2021-10-29 23:02:05 +00:00
proxy.py v2.4.0
2013-08-31 09:19:22 +00:00
2021-10-29 17:45:23 +00:00
options:
2013-08-31 09:19:22 +00:00
-h, --help show this help message and exit
2021-11-08 20:04:17 +00:00
--enable-events Default: False. Enables core to dispatch lifecycle events.
Plugins can be used to subscribe for core events.
2021-11-07 21:51:42 +00:00
--enable-conn-pool Default: False. (WIP) Enable upstream connection pooling.
2021-11-08 20:04:17 +00:00
--threadless Default: True. Enabled by default on Python 3.8+ (mac, linux).
When disabled a new thread is spawned to handle each client
connection.
--threaded Default: False. Disabled by default on Python < 3.8 and
windows. When enabled a new thread is spawned to handle each
2021-11-07 21:51:42 +00:00
client connection.
--pid-file PID_FILE Default: None. Save parent process ID to a file.
2021-11-08 20:04:17 +00:00
--backlog BACKLOG Default: 100. Maximum number of pending connections to proxy
server
2021-01-10 16:42:56 +00:00
--hostname HOSTNAME Default: ::1. Server IP address.
--port PORT Default: 8899. Server port.
--num-workers NUM_WORKERS
Defaults to number of CPU cores.
2021-11-07 21:51:42 +00:00
--unix-socket-path UNIX_SOCKET_PATH
2021-11-08 20:04:17 +00:00
Default: None. Unix socket path to use. When provided --host
and --port flags are ignored
2018-12-13 21:08:08 +00:00
--client-recvbuf-size CLIENT_RECVBUF_SIZE
2021-11-08 20:04:17 +00:00
Default: 1 MB. Maximum amount of data received from the client
in a single recv() operation. Bump this value for faster
uploads at the expense of increased RAM.
--key-file KEY_FILE Default: None. Server key file to enable end-to-end TLS
encryption with clients. If used, must also pass --cert-file.
--timeout TIMEOUT Default: 10.0. Number of seconds after which an inactive
connection must be dropped. Inactivity is defined by no data
sent or received by the client.
2021-01-10 16:42:56 +00:00
--version, -v Prints proxy.py version.
2013-08-31 09:19:22 +00:00
--log-level LOG_LEVEL
2021-11-08 20:04:17 +00:00
Valid options: DEBUG, INFO (default), WARNING, ERROR,
CRITICAL. Both upper and lowercase values are allowed. You may
also simply use the leading character e.g. --log-level d
2019-08-26 16:37:25 +00:00
--log-file LOG_FILE Default: sys.stdout. Log file destination.
2019-08-20 05:59:11 +00:00
--log-format LOG_FORMAT
Log format for Python logger.
2019-07-20 19:47:16 +00:00
--open-file-limit OPEN_FILE_LIMIT
2021-11-07 21:51:42 +00:00
Default: 1024. Maximum number of files (TCP connections) that
proxy.py can open concurrently.
2019-08-23 00:09:35 +00:00
--plugins PLUGINS Comma separated plugins
2021-11-08 20:04:17 +00:00
--enable-dashboard Default: False. Enables proxy.py dashboard.
2021-11-07 21:51:42 +00:00
--disable-http-proxy Default: False. Whether to disable proxy.HttpProxyPlugin.
2021-01-10 16:42:56 +00:00
--ca-key-file CA_KEY_FILE
2021-11-08 20:04:17 +00:00
Default: None. CA key to use for signing dynamically generated
HTTPS certificates. If used, must also pass --ca-cert-file and
--ca-signing-key-file
2021-01-10 16:42:56 +00:00
--ca-cert-dir CA_CERT_DIR
2021-11-07 21:51:42 +00:00
Default: ~/.proxy.py. Directory to store dynamically generated
certificates. Also see --ca-key-file, --ca-cert-file and --ca-
signing-key-file
2021-01-10 16:42:56 +00:00
--ca-cert-file CA_CERT_FILE
2021-11-08 20:04:17 +00:00
Default: None. Signing certificate to use for signing
dynamically generated HTTPS certificates. If used, must also
pass --ca-key-file and --ca-signing-key-file
2021-11-07 21:51:42 +00:00
--ca-file CA_FILE Default:
/Users/abhinavsingh/Dev/proxy.py/venv310/lib/python3.10/site-
2021-11-08 20:04:17 +00:00
packages/certifi/cacert.pem. Provide path to custom CA bundle
for peer certificate verification
2021-01-10 16:42:56 +00:00
--ca-signing-key-file CA_SIGNING_KEY_FILE
2021-11-08 20:04:17 +00:00
Default: None. CA signing key to use for dynamic generation of
HTTPS certificates. If used, must also pass --ca-key-file and
--ca-cert-file
2021-01-10 16:42:56 +00:00
--cert-file CERT_FILE
2021-11-08 20:04:17 +00:00
Default: None. Server certificate to enable end-to-end TLS
encryption with clients. If used, must also pass --key-file.
2021-01-10 16:42:56 +00:00
--disable-headers DISABLE_HEADERS
2021-11-08 20:04:17 +00:00
Default: None. Comma separated list of headers to remove
before dispatching client request to upstream server.
2019-07-20 19:47:16 +00:00
--server-recvbuf-size SERVER_RECVBUF_SIZE
2021-11-08 20:04:17 +00:00
Default: 1 MB. Maximum amount of data received from the server
in a single recv() operation. Bump this value for faster
downloads at the expense of increased RAM.
2021-01-10 16:42:56 +00:00
--basic-auth BASIC_AUTH
2021-11-08 20:04:17 +00:00
Default: No authentication. Specify colon separated
user:password to enable basic authentication.
2021-01-10 16:42:56 +00:00
--cache-dir CACHE_DIR
2021-11-08 20:04:17 +00:00
Default: A temporary directory. Flag only applicable when
cache plugin is used with on-disk storage.
2021-11-07 21:51:42 +00:00
--filtered-upstream-hosts FILTERED_UPSTREAM_HOSTS
2021-11-08 20:04:17 +00:00
Default: Blocks Facebook. Comma separated list of IPv4 and
IPv6 addresses.
2021-11-07 21:51:42 +00:00
--enable-web-server Default: False. Whether to enable proxy.HttpWebServerPlugin.
--enable-static-server
2021-11-08 20:04:17 +00:00
Default: False. Enable inbuilt static file server. Optionally,
also use --static-server-dir to serve static content from
custom directory. By default, static file server serves out of
installed proxy.py python module folder.
2019-10-10 05:36:47 +00:00
--static-server-dir STATIC_SERVER_DIR
2021-11-08 20:04:17 +00:00
Default: "public" folder in directory where proxy.py is
placed. This option is only applicable when static server is
also enabled. See --enable-static-server.
--pac-file PAC_FILE A file (Proxy Auto Configuration) or string to serve when the
server receives a direct file request. Using this option
enables proxy.HttpWebServerPlugin.
2021-01-10 16:42:56 +00:00
--pac-file-url-path PAC_FILE_URL_PATH
Default: /. Web server path to serve the PAC file.
--filtered-client-ips FILTERED_CLIENT_IPS
2021-11-07 21:51:42 +00:00
Default: 127.0.0.1,::1. Comma separated list of IPv4 and IPv6
addresses.
--filtered-url-regex-config FILTERED_URL_REGEX_CONFIG
2021-11-08 20:04:17 +00:00
Default: No config. Comma separated list of IPv4 and IPv6
addresses.
2021-11-07 21:51:42 +00:00
--cloudflare-dns-mode CLOUDFLARE_DNS_MODE
2021-11-08 20:04:17 +00:00
Default: security. Either "security" (for malware protection)
or "family" (for malware and adult content protection)
2013-08-31 09:19:22 +00:00
2021-10-29 17:45:23 +00:00
Proxy.py not working? Report at: https://github.com/abhinavsingh/proxy.py/issues/new
2013-08-31 09:19:22 +00:00
```
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
2021-01-10 16:42:56 +00:00
# Changelog
Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
2019-11-12 23:29:37 +00:00
## v2.x
- No longer ~~a single file module~~ .
- Added support for threadless execution.
- Added dashboard app.
2019-11-22 05:16:01 +00:00
- Added support for unit testing.
2019-11-12 23:29:37 +00:00
## v1.x
- `Python3` only.
2021-01-10 16:42:56 +00:00
- Deprecated support for ~~Python 2.x~~ .
2019-11-12 23:29:37 +00:00
- Added support multi core accept.
- Added plugin support.
## v0.x
- Single file.
- Single threaded server.
For detailed changelog refer to release PRs or commit history.
