Rooba
/
proxy.py
mirror of https://github.com/abhinavsingh/proxy.py.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
proxy.py/README.md

1833 lines
56 KiB
Markdown
Raw Normal View History

Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
[![Proxy.Py](https://raw.githubusercontent.com/abhinavsingh/proxy.py/develop/ProxyPy.png)](https://github.com/abhinavsingh/proxy.py)
updated README.md
2013-08-31 09:19:22 +00:00
Add plugin usage instruction
2019-09-21 22:04:44 +00:00
[![License](https://img.shields.io/github/license/abhinavsingh/proxy.py.svg)](https://opensource.org/licenses/BSD-3-Clause)
Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET on no internet (#216) * Catch TimeoutError and OSError (host unreachable) * Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET
2019-12-02 08:08:15 +00:00
[![PyPi Monthly](https://img.shields.io/pypi/dm/proxy.py.svg?color=green)](https://pypi.org/project/proxy.py/)
[![Docker Pulls](https://img.shields.io/docker/pulls/abhinavsingh/proxy.py?color=green)](https://hub.docker.com/r/abhinavsingh/proxy.py)
GitHub workflow badge (#244) * v2.x (#173) * Always update latest tag for docker releases * Update issue templates (#123) * Invoke HttpWebServerBasePlugin.handle_request for each request in HTTP/1.1 pipeline (#125) * Add tests to verify certificate generation * Separate out tests for ProtocolHandler and WebServerPlugin * Keep-alive connections for web server. TODO: Only keep-alivei if HTTP/1.1 * Add request.path to avoid build_url repeatedly whose name is also slightly misleading * Fix example usage of request.path * Pipeline only for HTTP/1.1 * Lint fix * Teardown HTTP/1.1 keep-alive request when Connection: close header is sent * Add instructions on how to build docker image locally * Move access_log to separate function for pretty logging * Reduce docker image size * Ensure teardown is always accompanied with Connection: close header Fix tests * Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins * Add --timeout flag with default value of 10 second. (#129) * Add --timeout flag with default value of 5. This value was previously hardcoded to 30 * --timeout=10 by default * Dispatch 408 timeout when connection is dropped due to inactivity * Add httpStatusCodes named tuple * Update plugin client connection reference after TLS connection upgrade * Test plugin examples (#130) * Add tests for plugin_examples.* to ensure we never break functionality * Add tests for plugin_examples.* * Test man in the middle * Lint fixes * Checkin * Add tests for plugin examples with TLS encryption enabled * Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more * Remove pip upgrade for windows which seems to be failing on travis (#136) * Remove pip upgrade for windows which seems to be failing on travis * Remove windows testing on Travis, pip install is failing * Add pipeline response parsing tests (#137) * Add pipeline response parsing tests * build_http_response now only adds content-length if transfer-encoding is not provided. Also return pending raw chunks from ChunkParser so that we can parse pipelined chunk responses. * os.close only for threadless (#138) * os.close only for Threadless to avoid fd leaks * Remove os.close mock which is only called for threadless * Update pytest from 5.2.1 to 5.2.2 (#142) * Update setuptools from 41.4.0 to 41.5.0 (#145) * Update typing-extensions from 3.7.4 to 3.7.4.1 (#147) * Update flake8 from 3.7.8 to 3.7.9 (#148) * Update setuptools from 41.5.0 to 41.5.1 (#149) * Update py-spy from 0.2.2 to 0.3.0 (#144) * Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls * Update mypy==0.740 (#151) * Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path * Update setuptools from 41.5.1 to 41.6.0 (#153) * Test refactor + Docker image CI (#154) * Move tests into individual modules too * Ensure one test class per file * Fix docker image after refactoring * Add github actions workflow for building docker image * Fix image name * Setup python required for extracting proxy version * Version will also require deps * Separate packages for Dashboard (#157) * Refactor Makefile and add dashboard setup.py * Package dashboard as proxy.py-dashboard pip package * Give dashboard releases its own version * Fix lib-package reference * Add non-blocking embedded mode feature (#159) * Fixes #158 * mypy fixes * Instructions for non-blocking embed mode * Toggle running flag before shutdown * Add private / public key generation utils which comply with new requirements on Mac OS 10.15 (#160) * Add utilities to generate private key and public keys with alternate cnames * Add separate package proxy.py-plugins, fixes #156 * Generate certificates to comply with Mac requirements. * Add utility for CSR generation and signing * Fixes #161 * Add initial pki tests * Give structure to dashboard app (#163) * Separate out files for different responsibilities. 1. Add src/plugins directory. This directory holds one typescript file per plugin. Each plugin is optionally can be displayed as a tab on the UI. 2. Move WebsocketApi to ws.ts. This file contains all websocket APIs provided by dashboard.py backend. * Make dashboard pluggable * Move devtools under core too * Register tabs dynamically * Typescript fixes for abstract interfaces * Initialize plugin app body skeleton * Call activated / deactivated on tab change * Move plugin name within plugin classes and initialize plugin within proxy dashboard constructor * templatize api development plugin * eslint fixes * use globs * Remove useless constructors * Move traffic_control outside of core plugin, it maps to several plugin examples like redirectToUpstreamHost, filterByUpstreamHost plugins (#165) * Introduce sendMessage websocket api which allows for callbacks (#166) * Introduce sendMessage websocket api which allows for callbacks, deprecate lastPingId in favor of callbacks * Let InspectTrafficPlugin handle all pushed inspection events * Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests. * Consistent dashboard look and feel across plugins (#169) * Explicitly link version changelog in TOC * Separate out app header body builder * Ensure unsubscribe when disabling inspection. Fixes #164 * Avoid creation of new manager per dashboard instance. * Add UI header for all plugins (tabs) * Ensure app body for all plugin skeleton * Move app-header and app-body within core for consistent dashboard look and feel * Consistent UI header body for plugins * autopep8 * Dashboard Inspect traffic tab + devtools (#170) * Explicitly link version changelog in TOC * Separate out app header body builder * Ensure unsubscribe when disabling inspection. Fixes #164 * Avoid creation of new manager per dashboard instance. * Add UI header for all plugins (tabs) * Ensure app body for all plugin skeleton * Move app-header and app-body within core for consistent dashboard look and feel * Consistent UI header body for plugins * autopep8 * make devtools * convert to es6 * Add inspect_traffic plugin devtools app * trigger re-build, github UI is stuck * Dynamically load devtools within inspect traffic view * Just copy devtools into public/dashboard folder * Works but not how we wanted, devtools takes over entire body and doesnt contain itself within a div * Load devtools within iframe * Load devtools within iframe (#171) * Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8 * Devtools Protocol (#174) * Refine docs * Decouple relay from dashboard. Will be re-used by devtools protocol plugin. * Just have a single manager for all eventing * Ofcourse managers cant be shared across processes * Remove unused * Add DevtoolsProtocolPlugin * Emit REQUEST_COMPLETE core event * Emit only if --enable-events used * Add event emitter for response cycle * Fill up core events to devtools protocol expectations * Serve static content with Cache-Control header and gzip compression * Add PWA manifest.json and icons from sample PWA apps (replace later) * Catch any exception and be ssl agnostic * Add CSP headers and avoid inline scripts * Re-enable iframe and deobfuscation * Embed plugins within <section/> block * Make tab switching agnostic of block name * Add support for browser history on tab change * Default hash to #home * Switch to tab if hash is already set * Expand canvas to fill screen even without content * Remove inline css for embedded devtools * Make dashboard backend websocket API pluggable * doc * Move dashboard backend within proxy module, now ships via same pip package (#177) * Allow resources to load from http and ws when running w/o https * Move dashboard backend (dashboard.py) within proxy module. Now shipped with pip install proxy.py * Update ref to dashboard backend in github workflows * Add git-pre-commit hook file. Enable it by symlinking as .git/hooks/pre-commit * Also enable static server for dashboard serving * Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image * Move benchmark module within proxy (#181) * Move benchmark within proxy module * chmod 0644 for benchmark.py which was executable till now * Turn utilities into its own section * Update pytest from 5.2.3 to 5.2.4 (#180) * Doc & Banner update to match GitHub (#182) * Update doc and banner * Update banner to match GitHub * Update older banners too * Add update_desc to .gitignore * Update banner for dashboard to match github * also update html, js, css * Update twine from 2.0.0 to 3.0.0 (#183) * Update pytest from 5.2.4 to 5.3.0 (#186) * Testing support improvements (#185) * Introduce proxy.Proxy context manager. This is similar to already existing context manager `start` but `proxy.Proxy` is a class with __enter__ and __exit__ methods. This allows usage of `proxy.Proxy` both as context manager and for manually setup and teardown of `proxy.py` during test setUpClass and teardownClass methods. * Gracefully shutdown threadless processes * Update tests and add a VCR method. See #184 * Refactor routes * Add Proxy to __all__ * Move TestCase under proxy.testing and test_embed.py under tests.embed module to avoid conflict with http module due to a http directory under proxy folder * Add a base cache plugin class which can be customized for custom cache behaviors * See #184. Add VCRPlugin which can be enabled within tests using a context manager, e.g. with self.vcr(): ... * Make cache plugin pluggable + make cache storage pluggable * Make dashboard npm module agnostic of top level directory * Symlink dashboard public folder * Dump devtools within dashboard public folder * Remove unused 3rd party js * Initialize Menubar (#188) * Initialize MacOS Menubar application * Dashboard plugin at-least needs a shutdown hook to teardown any thread/processes started by dashboard backend plugin * Add menu bar icon * Add respective test directories * Sync test banners * Move plugin tests under its own package * Enable daemon for threads, other this wont shutdown cleanly * Update twine from 3.0.0 to 3.1.0 (#190) * Update setuptools from 41.6.0 to 42.0.0 (#191) * Memory optimizations (#189) * Avoid persisting raw content in memory within parser, simply parse and throw-away. Addresses #187 * Clarity in test comments * Update setuptools from 42.0.0 to 42.0.1 (#193) * Make connection queue / recv work with memoryview to avoid copies (#192) * connection.recv now returns a memoryview * Make connection.queue also memoryview compliant * autopep8 * wrap in memoryview as necessary * Add default timeout for socket_connection and test_embed urllib * Fix tests * Skip TestProxyPyEmbedded for now, verifying GitHub actions * Add timeout for wait_for_server and skip only if GITHUB_ACTIONS env variable is set * Verify if GitHub Action fails due to wait_for_server spinning forever * Add test for wait_for_server timeout error exception * GitHub action hangs irrespective of wait_for_server timeout, disable TestEmbed for GitHub actions * Cleanup (#194) * Add basic README description for dashboard * Use spaces for all except makefile * enable tests for py 3.5 * Python 3.5 support label * Avoid clash of names * Add py3.8 support and bump node to 12.x (#195) * Add py3.8 support and bump node to 12.x * Add 10.x, 11.x, 12.x matrix for dashboard testing * Add Python 3.8 support label * Single tested with label * autopep8 (#196) * autopep8 * Update TestCase section * Update pytest from 5.3.0 to 5.3.1 (#197) * Update twine from 3.1.0 to 3.1.1 (#200) * Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc * Add proxy over ssh tunnel functionality (#198) * update mypy to 0.750 (#204) * Test Core Eventing (#205) * Add core event tests * Update .gitignore with coverage * Add shortlink gif * Add event dispatcher test * Test event subscriber * Test Dashboard backend (#206) * Update shortlink gif name * Conditionally run workflows as necessary * Use pytest * It works but github workflow is not reporting any status :( * Separate out badges * Add python_requires to setup.py * Update setuptools from 42.0.1 to 42.0.2 (#207) * Add tox.ini (#208) * Homebrew formula (#209) * Add homebrew formula * Build PyPi package and Homebrew installation verification * Check develop * bdist_wheel reported as error: invalid command "bdist_wheel" * Move under stable/develop folders to keep Proxy class name same * uff * develop installs proxy not proxy.py binary * Prepend site-packages * Install typing-extensions explicitly with brew * Use find_packages * Most likely failing due to lack of find_packages in current develop branch * Fix windows setup.py build * test_static_web_server_serves seems flaky on Ubuntu python 3.8 * Add instructions to install using homebrew * Disable test_static_web_server_serves on GitHub actions, seems flaky * Packaging (#210) * Move docker installation steps above * Try brewing with virtualenv * depends on python * Update homebrew formula for stable release * Just test brewing on latest python * Add support for regex based routing. Fixes #203 (#211) * Remove public folder references (#212) * Refactor (#213) * Add DEFAULT_HTTP_PORT constant * Use DEFAULT_HTTP_PORT in tests * Refactor into exception module * Refactor into inspector module * Refactor into server module * Refactor into proxy module * Build docker of Python 3.8 (#214) * Move homebrew under helper (#215) * Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET on no internet (#216) * Catch TimeoutError and OSError (host unreachable) * Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET * Enable mccabe (#217) * No need of per day or week stats (#218) * Make HTTP handler constructor free of socket file number (#219) * Refactor into acceptor module * Add tunnel doc * Make fileno free * Autopep8 * Response parser now reaches COMPLETE even when no body is expected (#220) * Stash current changes * Refactor into connection module * Response parser state complete when no body expect * Raise NotImplementedError if invalid state reached within parser * Update tox from 3.14.1 to 3.14.2 (#221) * Update paramiko from 2.6.0 to 2.7.0 (#225) * Update paramiko from 2.7.0 to 2.7.1 (#227) * Proxy Pool Plugin (#228) * Add proxy pool example. See #226 * Add ProxyPoolPlugin to doc * Update pytest from 5.3.1 to 5.3.2 (#229) * Update coverage from 4.5.4 to 5.0 (#230) * Update mypy from 0.750 to 0.760 (#232) * Update mypy from 0.760 to 0.761 (#235) * Move manager initialization outside of top level scope. Fixes #233 (#236) * Share lock to acceptors via pool (#238) * Move manager initialization outside of top level scope. Fixes #233 * Share lock to acceptor via pool * Optionally initialize manager in main thread and use the same for EventQueue initialization (#239) * Highlight language syntax (#240) * Highlight lang syntax * zsh prompt * Update coverage from 5.0 to 5.0.1 (#241) * Integration testing (#243) * Add tests for public/private/csr generation * Add integration testing skeleton for mac and ubuntu * Merge integration within lib test to avoid too many workflows * Disable integration testing on windows for now * Use sudo to start integration test script as lsof fails on MacOS. lsof: WARNING: can't stat() vmhgfs file system * Add basic integration testing for now to assert proxy works as expected when started out of develop branch * Add a call to inbuilt http server to verify it works * wait for server to accept requests Co-authored-by: pyup.io bot <github-bot@pyup.io> * Add github workflow badges Co-authored-by: pyup.io bot <github-bot@pyup.io>
2019-12-26 01:05:39 +00:00
[![No Dependencies](https://img.shields.io/static/v1?label=dependencies&message=none&color=green)](https://github.com/abhinavsingh/proxy.py)
[![Proxy.py Library Build Status](https://github.com/abhinavsingh/proxy.py/workflows/Proxy.py%20Library/badge.svg)](https://github.com/abhinavsingh/proxy.py/actions)
[![Proxy.py Docker Build Status](https://github.com/abhinavsingh/proxy.py/workflows/Proxy.py%20Docker/badge.svg)](https://github.com/abhinavsingh/proxy.py/actions)
[![Proxy.py Docker Build Status](https://github.com/abhinavsingh/proxy.py/workflows/Proxy.py%20Dashboard/badge.svg)](https://github.com/abhinavsingh/proxy.py/actions)
[![Proxy.py Docker Build Status](https://github.com/abhinavsingh/proxy.py/workflows/Proxy.py%20Brew/badge.svg)](https://github.com/abhinavsingh/proxy.py/actions)
Make HTTP handler constructor free of socket file number (#219) * Refactor into acceptor module * Add tunnel doc * Make fileno free * Autopep8
2019-12-02 23:55:08 +00:00
[![Coverage](https://codecov.io/gh/abhinavsingh/proxy.py/branch/develop/graph/badge.svg)](https://codecov.io/gh/abhinavsingh/proxy.py)
Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET on no internet (#216) * Catch TimeoutError and OSError (host unreachable) * Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET
2019-12-02 08:08:15 +00:00
Test Dashboard backend (#206) * Update shortlink gif name * Conditionally run workflows as necessary * Use pytest * It works but github workflow is not reporting any status :( * Separate out badges * Add python_requires to setup.py
2019-12-01 06:54:20 +00:00
[![Tested With MacOS, Ubuntu, Windows, Android, Android Emulator, iOS, iOS Simulator](https://img.shields.io/static/v1?label=tested%20with&message=mac%20OS%20%F0%9F%92%BB%20%7C%20Ubuntu%20%F0%9F%96%A5%20%7C%20Windows%20%F0%9F%92%BB&color=brightgreen)](https://abhinavsingh.com/proxy-py-a-lightweight-single-file-http-proxy-server-in-python/)
[![Android, Android Emulator](https://img.shields.io/static/v1?label=tested%20with&message=Android%20%F0%9F%93%B1%20%7C%20Android%20Emulator%20%F0%9F%93%B1&color=brightgreen)](https://abhinavsingh.com/proxy-py-a-lightweight-single-file-http-proxy-server-in-python/)
[![iOS, iOS Simulator](https://img.shields.io/static/v1?label=tested%20with&message=iOS%20%F0%9F%93%B1%20%7C%20iOS%20Simulator%20%F0%9F%93%B1&color=brightgreen)](https://abhinavsingh.com/proxy-py-a-lightweight-single-file-http-proxy-server-in-python/)
Make docker v1.0.0 release (#82) * Make docker v1.0.0 release * 👍 * 👍 📱 💻 💡 * 👍 📱 💻 💡
2019-09-23 07:30:52 +00:00
[![Maintenance](https://img.shields.io/static/v1?label=maintained%3F&message=yes&color=green)](https://gitHub.com/abhinavsingh/proxy.py/graphs/commit-activity)
[![Ask Me Anything](https://img.shields.io/static/v1?label=need%20help%3F&message=ask&color=green)](https://twitter.com/imoracle)
[![Contributions Welcome](https://img.shields.io/static/v1?label=contributions&message=welcome%20%F0%9F%91%8D&color=green)](https://github.com/abhinavsingh/proxy.py/issues)
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
[![Gitter](https://badges.gitter.im/proxy-py/community.svg)](https://gitter.im/proxy-py/community)
Make docker v1.0.0 release (#82) * Make docker v1.0.0 release * 👍 * 👍 📱 💻 💡 * 👍 📱 💻 💡
2019-09-23 07:30:52 +00:00
[README] Mention 3.9 support tested locally on Mac using pyenv install 3.9-dev (#288)
2020-02-03 04:49:01 +00:00
[![Python 3.x](https://img.shields.io/static/v1?label=Python&message=3.6%20%7C%203.7%20%7C%203.8%20%7C%203.9&color=blue)](https://www.python.org/)
Add py3.8 support and bump node to 12.x (#195) * Add py3.8 support and bump node to 12.x * Add 10.x, 11.x, 12.x matrix for dashboard testing * Add Python 3.8 support label * Single tested with label
2019-11-27 05:44:20 +00:00
[![Checked with mypy](https://img.shields.io/static/v1?label=MyPy&message=checked&color=blue)](http://mypy-lang.org/)
Add Travis build status
2019-02-09 04:25:09 +00:00
Consistent dashboard look and feel across plugins (#169) * Explicitly link version changelog in TOC * Separate out app header body builder * Ensure unsubscribe when disabling inspection. Fixes #164 * Avoid creation of new manager per dashboard instance. * Add UI header for all plugins (tabs) * Ensure app body for all plugin skeleton * Move app-header and app-body within core for consistent dashboard look and feel * Consistent UI header body for plugins * autopep8
2019-11-12 23:29:37 +00:00
[![Become a Backer](https://opencollective.com/proxypy/tiers/backer.svg?avatarHeight=72)](https://opencollective.com/proxypy)
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Table of Contents
=================
* [Features](#features)
* [Install](#install)
Homebrew formula (#209) * Add homebrew formula * Build PyPi package and Homebrew installation verification * Check develop * bdist_wheel reported as error: invalid command "bdist_wheel" * Move under stable/develop folders to keep Proxy class name same * uff * develop installs proxy not proxy.py binary * Prepend site-packages * Install typing-extensions explicitly with brew * Use find_packages * Most likely failing due to lack of find_packages in current develop branch * Fix windows setup.py build * test_static_web_server_serves seems flaky on Ubuntu python 3.8 * Add instructions to install using homebrew * Disable test_static_web_server_serves on GitHub actions, seems flaky
2019-12-02 03:01:28 +00:00
* [Using PIP](#using-pip)
* [Stable version](#stable-version-with-pip)
* [Development version](#development-version-with-pip)
Packaging (#210) * Move docker installation steps above * Try brewing with virtualenv * depends on python * Update homebrew formula for stable release * Just test brewing on latest python
2019-12-02 04:08:50 +00:00
* [Using Docker](#using-docker)
* [Stable version](#stable-version-from-docker-hub)
* [Development version](#build-development-version-locally)
Homebrew formula (#209) * Add homebrew formula * Build PyPi package and Homebrew installation verification * Check develop * bdist_wheel reported as error: invalid command "bdist_wheel" * Move under stable/develop folders to keep Proxy class name same * uff * develop installs proxy not proxy.py binary * Prepend site-packages * Install typing-extensions explicitly with brew * Use find_packages * Most likely failing due to lack of find_packages in current develop branch * Fix windows setup.py build * test_static_web_server_serves seems flaky on Ubuntu python 3.8 * Add instructions to install using homebrew * Disable test_static_web_server_serves on GitHub actions, seems flaky
2019-12-02 03:01:28 +00:00
* [Using HomeBrew](#using-homebrew)
* [Stable version](#stable-version-with-homebrew)
* [Development version](#development-version-with-homebrew)
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
* [Start proxy.py](#start-proxypy)
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
* [From command line when installed using PIP](#from-command-line-when-installed-using-pip)
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
* [Run it](#run-it)
Doc & Banner update to match GitHub (#182) * Update doc and banner * Update banner to match GitHub * Update older banners too * Add update_desc to .gitignore * Update banner for dashboard to match github * also update html, js, css
2019-11-19 04:45:51 +00:00
* [Understanding logs](#understanding-logs)
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
* [Enable DEBUG logging](#enable-debug-logging)
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
* [From command line using repo source](#from-command-line-using-repo-source)
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
* [Docker Image](#docker-image)
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
* [Customize Startup Flags](#customize-startup-flags)
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
* [Plugin Examples](#plugin-examples)
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
* [HTTP Proxy Plugins](#http-proxy-plugins)
Proxy Pool Plugin (#228) * Add proxy pool example. See #226 * Add ProxyPoolPlugin to doc
2019-12-10 03:38:49 +00:00
* [ShortLink Plugin](#shortlinkplugin)
* [Modify Post Data Plugin](#modifypostdataplugin)
* [Mock Api Plugin](#mockrestapiplugin)
* [Redirect To Custom Server Plugin](#redirecttocustomserverplugin)
* [Filter By Upstream Host Plugin](#filterbyupstreamhostplugin)
* [Cache Responses Plugin](#cacheresponsesplugin)
* [Man-In-The-Middle Plugin](#maninthemiddleplugin)
* [Proxy Pool Plugin](#proxypoolplugin)
Document FilterByClientIpPlugin & ModifyChunkResponsePlugin (#387)
2020-07-01 22:00:29 +00:00
* [FilterByClientIpPlugin](#filterbyclientipplugin)
* [ModifyChunkResponsePlugin](#modifychunkresponseplugin)
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
* [HTTP Web Server Plugins](#http-web-server-plugins)
* [Reverse Proxy](#reverse-proxy)
* [Web Server Route](#web-server-route)
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
* [Plugin Ordering](#plugin-ordering)
* [End-to-End Encryption](#end-to-end-encryption)
Defer SSL Wrap (#100) * Always deduce address family as we have a structure * Convert TcpConnection into an ABC. * Fix tests * Rename MultiCoreRequestDispatcher to WorkerPool * Consistent naming refactor * Make WorkerPool independent of protocol config object * Fix tests * Proper fix for test_on_client_connection_called_on_teardown * TLS interception * better logging
2019-09-27 16:18:58 +00:00
* [TLS Interception](#tls-interception)
Enable docker containers for TLS Interception (#375) * Move static values to constants.py * Install openssl in docker containers for TLS interception * Complete TLS Interception with Docker doc
2020-06-17 12:24:43 +00:00
* [TLS Interception With Docker](#tls-interception-with-docker)
Make HTTP handler constructor free of socket file number (#219) * Refactor into acceptor module * Add tunnel doc * Make fileno free * Autopep8
2019-12-02 23:55:08 +00:00
* [Proxy Over SSH Tunnel](#proxy-over-ssh-tunnel)
* [Proxy Remote Requests Locally](#proxy-remote-requests-locally)
* [Proxy Local Requests Remotely](#proxy-local-requests-remotely)
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
* [Embed proxy.py](#embed-proxypy)
Add non-blocking embedded mode feature (#159) * Fixes #158 * mypy fixes * Instructions for non-blocking embed mode * Toggle running flag before shutdown
2019-11-08 04:53:08 +00:00
* [Blocking Mode](#blocking-mode)
* [Non-blocking Mode](#non-blocking-mode)
Documentation for plugin loading in embedded mode (#422)
2020-08-14 11:50:19 +00:00
* [Loading Plugins](#loading-plugins)
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
* [Unit testing with proxy.py](#unit-testing-with-proxypy)
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
* [proxy.TestCase](#proxytestcase)
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
* [Override Startup Flags](#override-startup-flags)
* [With unittest.TestCase](#with-unittesttestcase)
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
* [Plugin Developer and Contributor Guide](#plugin-developer-and-contributor-guide)
* [Everything is a plugin](#everything-is-a-plugin)
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
* [Internal Architecture](#internal-architecture)
* [Internal Documentation](#internal-documentation)
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
* [Development Guide](#development-guide)
* [Setup Local Environment](#setup-local-environment)
* [Setup pre-commit hook](#setup-pre-commit-hook)
* [Sending a Pull Request](#sending-a-pull-request)
Move benchmark module within proxy (#181) * Move benchmark within proxy module * chmod 0644 for benchmark.py which was executable till now * Turn utilities into its own section
2019-11-16 07:49:51 +00:00
* [Utilities](#utilities)
* [TCP](#tcp-sockets)
* [new_socket_connection](#new_socket_connection)
* [socket_connection](#socket_connection)
* [Http](#http-client)
* [build_http_request](#build_http_request)
* [build_http_response](#build_http_response)
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
* [Public Key Infrastructure](#pki)
* [API Usage](#api-usage)
* [CLI Usage](#cli-usage)
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
* [Frequently Asked Questions](#frequently-asked-questions)
Add py.typed marker, add version checker, remove deprecated methods (#255) * Add py.typed marker * Add version-check.py * Remove deprecated assertDictContainsSubset usage
2020-01-07 05:23:38 +00:00
* [Threads vs Threadless](#threads-vs-threadless)
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
* [SyntaxError: invalid syntax](#syntaxerror-invalid-syntax)
Consistent dashboard look and feel across plugins (#169) * Explicitly link version changelog in TOC * Separate out app header body builder * Ensure unsubscribe when disabling inspection. Fixes #164 * Avoid creation of new manager per dashboard instance. * Add UI header for all plugins (tabs) * Ensure app body for all plugin skeleton * Move app-header and app-body within core for consistent dashboard look and feel * Consistent UI header body for plugins * autopep8
2019-11-12 23:29:37 +00:00
* [Unable to load plugins](#unable-to-load-plugins)
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
* [Unable to connect with proxy.py from remote host](#unable-to-connect-with-proxypy-from-remote-host)
* [Basic auth not working with a browser](#basic-auth-not-working-with-a-browser)
* [Docker image not working on MacOS](#docker-image-not-working-on-macos)
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
* [ValueError: filedescriptor out of range in select](#valueerror-filedescriptor-out-of-range-in-select)
Doc & Banner update to match GitHub (#182) * Update doc and banner * Update banner to match GitHub * Update older banners too * Add update_desc to .gitignore * Update banner for dashboard to match github * also update html, js, css
2019-11-19 04:45:51 +00:00
* [None:None in access logs](#nonenone-in-access-logs)
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
* [Flags](#flags)
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
* [Changelog](#changelog)
Consistent dashboard look and feel across plugins (#169) * Explicitly link version changelog in TOC * Separate out app header body builder * Ensure unsubscribe when disabling inspection. Fixes #164 * Avoid creation of new manager per dashboard instance. * Add UI header for all plugins (tabs) * Ensure app body for all plugin skeleton * Move app-header and app-body within core for consistent dashboard look and feel * Consistent UI header body for plugins * autopep8
2019-11-12 23:29:37 +00:00
* [v2.x](#v2x)
* [v1.x](#v1x)
* [v0.x](#v0x)
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
updated README.md
2013-08-31 09:19:22 +00:00
Features
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
========
updated README.md
2013-08-31 09:19:22 +00:00
Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more
2019-10-16 06:56:39 +00:00
- Fast & Scalable
- Scales by using all available cores on the system
- Threadless executions using coroutine
- Made to handle `tens-of-thousands` connections / sec
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more
2019-10-16 06:56:39 +00:00
# On Macbook Pro 2015 / 2.8 GHz Intel Core i7
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
❯ hey -n 10000 -c 100 http://localhost:8899/
Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more
2019-10-16 06:56:39 +00:00
Summary:
Total: 0.6157 secs
Slowest: 0.1049 secs
Fastest: 0.0007 secs
Average: 0.0055 secs
Requests/sec: 16240.5444
Total data: 800000 bytes
Size/request: 80 bytes
Response time histogram:
0.001 [1] |
0.011 [9565] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.022 [332] |■
```
Add plugin usage instruction
2019-09-21 22:04:44 +00:00
- Lightweight
Use ipaddress type and avoid explicit Connection: close header to upstream (#76) * Fixes #71 and address some of #75 * Fix tests as IPV6 is default now * Add comments and raise from e
2019-09-22 02:20:27 +00:00
- Uses only `~5-20MB` RAM
Add plugin usage instruction
2019-09-21 22:04:44 +00:00
- No external dependency other than standard Python library
- Programmable
- Optionally enable builtin Web Server
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
- Customize proxy and http routing via [plugins](https://github.com/abhinavsingh/proxy.py/tree/develop/proxy/plugin)
- Enable plugin using command line option e.g. `--plugins proxy.plugin.CacheResponsesPlugin`
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
- Plugin API is currently in development phase, expect breaking changes.
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
- Realtime Dashboard
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
- Optionally enable bundled dashboard.
- Available at `http://localhost:8899/dashboard`.
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
- Inspect, Monitor, Control and Configure `proxy.py` at runtime.
- Extend dashboard using plugins.
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
- Dashboard is currently in development phase, expect breaking changes.
Can serve proxy traffic over HTTPS (#85) * Can serve proxy traffic over HTTPS. Pass --certfile and --keyfile arguments to enable SSL/TLS layer. Example: 1) Start proxy.py as ./proxy.py --certfile cert.pem --keyfile key.pem 2) Then verify via curl curl -v -x https://localhost:8899 --proxy-cacert cert.pem http://httpbin.org/get * Add --ignore-missing-imports for missing multiprocessing.reduction module * mock socket.fromfd to avoid sys calls * Add HTTPS info * Add TLS encryption * Add TLS encryption
2019-09-24 00:38:10 +00:00
- Secure
Add support for TLS/SSL interception and HTTPS server (#86) * Handle OSError thrown when curl --proxy-cacert flag is invalid * Add server-cert * Dockerfile dont need --ipv4 since now its auto-detected and deprecated * Add make server-cert info * Add support for HTTPS interception. 1) Start as proxy.py --ca-key-file ca-key.pem --ca-cert-file ca-cert.pem --ca-signing-key-file ca-signing-key.pem 2) Test using curl curl -v -L -x proxy.py:8899 --cacert ca-cert.pem https://google.com Now proxy.py is able to decrypt HTTPS traffic between curl and google.com. Generated certificates are stored under ~/.proxy.py directory. * Add comments for why interception do not work if client is communicating with proxy.py over https * Fix tests * Exit with message when trying to https intercept and also serve over https * lint * Disable OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_TLSv1, OP_NO_TLSv1_1 when wrapping upstream server socket * Ensure client.conn.shutdown is called if necessary. * Add mypy check for plugin_examples.py * Exception handling when client sends invalid requests * Verify plugin_examples * mypy compat * HttpRequestRejected
2019-09-25 02:02:57 +00:00
- Enable end-to-end encryption between clients and `proxy.py` using TLS
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
- See [End-to-End Encryption](#end-to-end-encryption)
Add support for TLS/SSL interception and HTTPS server (#86) * Handle OSError thrown when curl --proxy-cacert flag is invalid * Add server-cert * Dockerfile dont need --ipv4 since now its auto-detected and deprecated * Add make server-cert info * Add support for HTTPS interception. 1) Start as proxy.py --ca-key-file ca-key.pem --ca-cert-file ca-cert.pem --ca-signing-key-file ca-signing-key.pem 2) Test using curl curl -v -L -x proxy.py:8899 --cacert ca-cert.pem https://google.com Now proxy.py is able to decrypt HTTPS traffic between curl and google.com. Generated certificates are stored under ~/.proxy.py directory. * Add comments for why interception do not work if client is communicating with proxy.py over https * Fix tests * Exit with message when trying to https intercept and also serve over https * lint * Disable OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_TLSv1, OP_NO_TLSv1_1 when wrapping upstream server socket * Ensure client.conn.shutdown is called if necessary. * Add mypy check for plugin_examples.py * Exception handling when client sends invalid requests * Verify plugin_examples * mypy compat * HttpRequestRejected
2019-09-25 02:02:57 +00:00
- Man-In-The-Middle
- Can decrypt TLS traffic between clients and upstream servers
Add non-blocking embedded mode feature (#159) * Fixes #158 * mypy fixes * Instructions for non-blocking embed mode * Toggle running flag before shutdown
2019-11-08 04:53:08 +00:00
- See [TLS Interception](#tls-interception)
Can serve proxy traffic over HTTPS (#85) * Can serve proxy traffic over HTTPS. Pass --certfile and --keyfile arguments to enable SSL/TLS layer. Example: 1) Start proxy.py as ./proxy.py --certfile cert.pem --keyfile key.pem 2) Then verify via curl curl -v -x https://localhost:8899 --proxy-cacert cert.pem http://httpbin.org/get * Add --ignore-missing-imports for missing multiprocessing.reduction module * mock socket.fromfd to avoid sys calls * Add HTTPS info * Add TLS encryption * Add TLS encryption
2019-09-24 00:38:10 +00:00
- Supported proxy protocols
Add private / public key generation utils which comply with new requirements on Mac OS 10.15 (#160) * Add utilities to generate private key and public keys with alternate cnames * Add separate package proxy.py-plugins, fixes #156 * Generate certificates to comply with Mac requirements. * Add utility for CSR generation and signing * Fixes #161 * Add initial pki tests
2019-11-12 00:17:13 +00:00
- `http(s)`
- `http1`
- `http1.1` pipeline
Can serve proxy traffic over HTTPS (#85) * Can serve proxy traffic over HTTPS. Pass --certfile and --keyfile arguments to enable SSL/TLS layer. Example: 1) Start proxy.py as ./proxy.py --certfile cert.pem --keyfile key.pem 2) Then verify via curl curl -v -x https://localhost:8899 --proxy-cacert cert.pem http://httpbin.org/get * Add --ignore-missing-imports for missing multiprocessing.reduction module * mock socket.fromfd to avoid sys calls * Add HTTPS info * Add TLS encryption * Add TLS encryption
2019-09-24 00:38:10 +00:00
- `http2`
- `websockets`
Per worker queue instead of shared one. 1. MultiCoreDispatcher now dispatches request in round robin fashion. 2. Added Python3 static type hinting add Google Cloud Run Button Preserve proxy.py name as otherwise it breaks module system. Just provide a custom service name to fix Google Cloud Run issue Custom service name for Google Cloud Run. Specify port in app.json as required by Google Cloud Run Go cannot unmarshall int :) Port desc Looks like PORT cannot be overridden, for now use port 8080 in Dockerfile Remove cloudrun integration. While proxy.py runs successfully within the container, gvisor limitations doesn't allow proxy.py to accept client connections. Specifically membarrier syscall isn't allowed. Update README.md
2019-08-23 00:09:35 +00:00
- Optimized for large file uploads and downloads
Avoid setting open file limits on Windows OS. Python resource module is unavailable on Windows OS.
2018-12-14 13:14:56 +00:00
- IPv4 and IPv6 support
- Basic authentication support
Link Travis Board for build status image
2019-08-12 05:41:39 +00:00
- Can serve a [PAC (Proxy Auto-configuration)](https://en.wikipedia.org/wiki/Proxy_auto-config) file
Can serve proxy traffic over HTTPS (#85) * Can serve proxy traffic over HTTPS. Pass --certfile and --keyfile arguments to enable SSL/TLS layer. Example: 1) Start proxy.py as ./proxy.py --certfile cert.pem --keyfile key.pem 2) Then verify via curl curl -v -x https://localhost:8899 --proxy-cacert cert.pem http://httpbin.org/get * Add --ignore-missing-imports for missing multiprocessing.reduction module * mock socket.fromfd to avoid sys calls * Add HTTPS info * Add TLS encryption * Add TLS encryption
2019-09-24 00:38:10 +00:00
- See `--pac-file` and `--pac-file-url-path` flags
updated README.md
2013-08-31 09:19:22 +00:00
Install
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
=======
updated README.md
2013-08-31 09:19:22 +00:00
Homebrew formula (#209) * Add homebrew formula * Build PyPi package and Homebrew installation verification * Check develop * bdist_wheel reported as error: invalid command "bdist_wheel" * Move under stable/develop folders to keep Proxy class name same * uff * develop installs proxy not proxy.py binary * Prepend site-packages * Install typing-extensions explicitly with brew * Use find_packages * Most likely failing due to lack of find_packages in current develop branch * Fix windows setup.py build * test_static_web_server_serves seems flaky on Ubuntu python 3.8 * Add instructions to install using homebrew * Disable test_static_web_server_serves on GitHub actions, seems flaky
2019-12-02 03:01:28 +00:00
## Using PIP
### Stable Version with PIP
updated README.md
2013-08-31 09:19:22 +00:00
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
Install from `PyPi`
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ pip install --upgrade proxy.py
```
updated README.md
2013-08-31 09:19:22 +00:00
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
or from GitHub `master` branch
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ pip install git+https://github.com/abhinavsingh/proxy.py.git@master
```
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
Homebrew formula (#209) * Add homebrew formula * Build PyPi package and Homebrew installation verification * Check develop * bdist_wheel reported as error: invalid command "bdist_wheel" * Move under stable/develop folders to keep Proxy class name same * uff * develop installs proxy not proxy.py binary * Prepend site-packages * Install typing-extensions explicitly with brew * Use find_packages * Most likely failing due to lack of find_packages in current develop branch * Fix windows setup.py build * test_static_web_server_serves seems flaky on Ubuntu python 3.8 * Add instructions to install using homebrew * Disable test_static_web_server_serves on GitHub actions, seems flaky
2019-12-02 03:01:28 +00:00
### Development Version with PIP
Use Pipe instead of Queue when passing sockets (#69) * Use pipes instead of queues. Possible solution for #68 * Handle ConnectionRefusedError * Close corresponding pipes on shutdown * Abstract classes for plugins. * Add github workflows * Use pytest for github workflow * Add Windows/Mac workflows * Use os.matrix * 3.8 and 3.9 probably dont exists yet * Do not fail-fast, also python 3.5 seems to be throwing syntax error for typing * autopep8 * Disable windows-latest workflow which seems to hang
2019-09-17 03:18:14 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ pip install git+https://github.com/abhinavsingh/proxy.py.git@develop
```
Use Pipe instead of Queue when passing sockets (#69) * Use pipes instead of queues. Possible solution for #68 * Handle ConnectionRefusedError * Close corresponding pipes on shutdown * Abstract classes for plugins. * Add github workflows * Use pytest for github workflow * Add Windows/Mac workflows * Use os.matrix * 3.8 and 3.9 probably dont exists yet * Do not fail-fast, also python 3.5 seems to be throwing syntax error for typing * autopep8 * Disable windows-latest workflow which seems to hang
2019-09-17 03:18:14 +00:00
Packaging (#210) * Move docker installation steps above * Try brewing with virtualenv * depends on python * Update homebrew formula for stable release * Just test brewing on latest python
2019-12-02 04:08:50 +00:00
## Using Docker
#### Stable Version from Docker Hub
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:latest
```
Packaging (#210) * Move docker installation steps above * Try brewing with virtualenv * depends on python * Update homebrew formula for stable release * Just test brewing on latest python
2019-12-02 04:08:50 +00:00
#### Build Development Version Locally
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ git clone https://github.com/abhinavsingh/proxy.py.git
❯ cd proxy.py
❯ make container
❯ docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:latest
```
Packaging (#210) * Move docker installation steps above * Try brewing with virtualenv * depends on python * Update homebrew formula for stable release * Just test brewing on latest python
2019-12-02 04:08:50 +00:00
[![WARNING](https://img.shields.io/static/v1?label=MacOS&message=warning&color=red)](https://github.com/moby/vpnkit/issues/469)
`docker` image is currently broken on `macOS` due to incompatibility with [vpnkit](https://github.com/moby/vpnkit/issues/469).
Homebrew formula (#209) * Add homebrew formula * Build PyPi package and Homebrew installation verification * Check develop * bdist_wheel reported as error: invalid command "bdist_wheel" * Move under stable/develop folders to keep Proxy class name same * uff * develop installs proxy not proxy.py binary * Prepend site-packages * Install typing-extensions explicitly with brew * Use find_packages * Most likely failing due to lack of find_packages in current develop branch * Fix windows setup.py build * test_static_web_server_serves seems flaky on Ubuntu python 3.8 * Add instructions to install using homebrew * Disable test_static_web_server_serves on GitHub actions, seems flaky
2019-12-02 03:01:28 +00:00
## Using HomeBrew
### Stable Version with HomeBrew
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ brew install https://raw.githubusercontent.com/abhinavsingh/proxy.py/develop/helper/homebrew/stable/proxy.rb
```
Homebrew formula (#209) * Add homebrew formula * Build PyPi package and Homebrew installation verification * Check develop * bdist_wheel reported as error: invalid command "bdist_wheel" * Move under stable/develop folders to keep Proxy class name same * uff * develop installs proxy not proxy.py binary * Prepend site-packages * Install typing-extensions explicitly with brew * Use find_packages * Most likely failing due to lack of find_packages in current develop branch * Fix windows setup.py build * test_static_web_server_serves seems flaky on Ubuntu python 3.8 * Add instructions to install using homebrew * Disable test_static_web_server_serves on GitHub actions, seems flaky
2019-12-02 03:01:28 +00:00
### Development Version with HomeBrew
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ brew install https://raw.githubusercontent.com/abhinavsingh/proxy.py/develop/helper/homebrew/develop/proxy.rb
```
Homebrew formula (#209) * Add homebrew formula * Build PyPi package and Homebrew installation verification * Check develop * bdist_wheel reported as error: invalid command "bdist_wheel" * Move under stable/develop folders to keep Proxy class name same * uff * develop installs proxy not proxy.py binary * Prepend site-packages * Install typing-extensions explicitly with brew * Use find_packages * Most likely failing due to lack of find_packages in current develop branch * Fix windows setup.py build * test_static_web_server_serves seems flaky on Ubuntu python 3.8 * Add instructions to install using homebrew * Disable test_static_web_server_serves on GitHub actions, seems flaky
2019-12-02 03:01:28 +00:00
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
Start proxy.py
==============
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
## From command line when installed using PIP
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
When `proxy.py` is installed using `pip`,
Add non-blocking embedded mode feature (#159) * Fixes #158 * mypy fixes * Instructions for non-blocking embed mode * Toggle running flag before shutdown
2019-11-08 04:53:08 +00:00
an executable named `proxy` is placed under your `$PATH`.
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
#### Run it
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
Simply type `proxy` on command line to start it with default configuration.
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
...[redacted]... - Loaded plugin proxy.http_proxy.HttpProxyPlugin
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
...[redacted]... - Starting 8 workers
...[redacted]... - Started server on ::1:8899
```
Doc & Banner update to match GitHub (#182) * Update doc and banner * Update banner to match GitHub * Update older banners too * Add update_desc to .gitignore * Update banner for dashboard to match github * also update html, js, css
2019-11-19 04:45:51 +00:00
#### Understanding logs
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
Things to notice from above logs:
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
- `Loaded plugin` - `proxy.py` will load `proxy.http.proxy.HttpProxyPlugin` by default.
As name suggests, this core plugin adds `http(s)` proxy server capabilities to `proxy.py`
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
- `Started N workers` - Use `--num-workers` flag to customize number of worker processes.
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
By default, `proxy.py` will start as many workers as there are CPU cores on the machine.
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
- `Started server on ::1:8899` - By default, `proxy.py` listens on IPv6 `::1`, which
is equivalent of IPv4 `127.0.0.1`. If you want to access `proxy.py` externally,
use `--hostname ::` or `--hostname 0.0.0.0` or bind to any other interface available
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
on your machine.
- `Port 8899` - Use `--port` flag to customize default TCP port.
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
#### Enable DEBUG logging
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
All the logs above are `INFO` level logs, default `--log-level` for `proxy.py`.
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
Lets start `proxy.py` with `DEBUG` level logging:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy --log-level d
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
...[redacted]... - Open file descriptor soft limit set to 1024
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
...[redacted]... - Loaded plugin proxy.http_proxy.HttpProxyPlugin
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
...[redacted]... - Started 8 workers
...[redacted]... - Started server on ::1:8899
```
As we can see, before starting up:
- `proxy.py` also tried to set open file limit `ulimit` on the system.
- Default value for `--open-file-limit` used is `1024`.
- `--open-file-limit` flag is a no-op on `Windows` operating systems.
See [flags](#flags) for full list of available configuration options.
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
## From command line using repo source
If you are trying to run `proxy.py` from source code,
there is no binary file named `proxy` in the source code.
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
To start `proxy.py` from source code follow these instructions:
- Clone repo
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ git clone https://github.com/abhinavsingh/proxy.py.git
❯ cd proxy.py
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
```
- Create a Python 3 virtual env
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ python3 -m venv venv
❯ source venv/bin/activate
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
```
- Install deps
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ pip install -r requirements.txt
❯ pip install -r requirements-testing.txt
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
```
- Run tests
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ make
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
```
- Run proxy.py
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ python -m proxy
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
```
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
Also see [Plugin Developer and Contributor Guide](#plugin-developer-and-contributor-guide)
if you plan to work with `proxy.py` source code.
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
## Docker image
Release docker container and prep for v0.4
2019-02-09 05:13:34 +00:00
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
#### Customize startup flags
Invoke HttpWebServerBasePlugin.handle_request for each request in HTTP/1.1 pipeline (#125) * Add tests to verify certificate generation * Separate out tests for ProtocolHandler and WebServerPlugin * Keep-alive connections for web server. TODO: Only keep-alivei if HTTP/1.1 * Add request.path to avoid build_url repeatedly whose name is also slightly misleading * Fix example usage of request.path * Pipeline only for HTTP/1.1 * Lint fix * Teardown HTTP/1.1 keep-alive request when Connection: close header is sent * Add instructions on how to build docker image locally * Move access_log to separate function for pretty logging
2019-10-11 05:15:07 +00:00
Add starting proxy.py instructions
2019-09-25 17:09:40 +00:00
By default `docker` binary is started with IPv4 networking flags:
Make docker v1.0.0 release (#82) * Make docker v1.0.0 release * 👍 * 👍 📱 💻 💡 * 👍 📱 💻 💡
2019-09-23 07:30:52 +00:00
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
--hostname 0.0.0.0 --port 8899
Make docker v1.0.0 release (#82) * Make docker v1.0.0 release * 👍 * 👍 📱 💻 💡 * 👍 📱 💻 💡
2019-09-23 07:30:52 +00:00
To override input flags, start docker image as follows.
Invoke HttpWebServerBasePlugin.handle_request for each request in HTTP/1.1 pipeline (#125) * Add tests to verify certificate generation * Separate out tests for ProtocolHandler and WebServerPlugin * Keep-alive connections for web server. TODO: Only keep-alivei if HTTP/1.1 * Add request.path to avoid build_url repeatedly whose name is also slightly misleading * Fix example usage of request.path * Pipeline only for HTTP/1.1 * Lint fix * Teardown HTTP/1.1 keep-alive request when Connection: close header is sent * Add instructions on how to build docker image locally * Move access_log to separate function for pretty logging
2019-10-11 05:15:07 +00:00
For example, to check `proxy.py` version within Docker image:
Make docker v1.0.0 release (#82) * Make docker v1.0.0 release * 👍 * 👍 📱 💻 💡 * 👍 📱 💻 💡
2019-09-23 07:30:52 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
❯ docker run -it \
Make docker v1.0.0 release (#82) * Make docker v1.0.0 release * 👍 * 👍 📱 💻 💡 * 👍 📱 💻 💡
2019-09-23 07:30:52 +00:00
-p 8899:8899 \
Ensure explicit flush before upgrade while TLS intercepting (#121) * Ensure explicit flush before upgrade + nonblocking server sock. * tls interception tests * prepare for v1.1.1 * Add requirements.txt for typing extensions * Add setup.py under lint workflow
2019-10-10 22:15:27 +00:00
--rm abhinavsingh/proxy.py:latest \
Invoke HttpWebServerBasePlugin.handle_request for each request in HTTP/1.1 pipeline (#125) * Add tests to verify certificate generation * Separate out tests for ProtocolHandler and WebServerPlugin * Keep-alive connections for web server. TODO: Only keep-alivei if HTTP/1.1 * Add request.path to avoid build_url repeatedly whose name is also slightly misleading * Fix example usage of request.path * Pipeline only for HTTP/1.1 * Lint fix * Teardown HTTP/1.1 keep-alive request when Connection: close header is sent * Add instructions on how to build docker image locally * Move access_log to separate function for pretty logging
2019-10-11 05:15:07 +00:00
-v
Make docker v1.0.0 release (#82) * Make docker v1.0.0 release * 👍 * 👍 📱 💻 💡 * 👍 📱 💻 💡
2019-09-23 07:30:52 +00:00
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Plugin Examples
===============
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
- See [plugin](https://github.com/abhinavsingh/proxy.py/tree/develop/proxy/plugin) module for full code.
- All the bundled plugin examples also works with `https` traffic
- Require additional flags and certificate generation
- See [TLS Interception](#tls-interception).
- Plugin examples are also bundled with Docker image.
- See [Customize startup flags](#customize-startup-flags) to try plugins with Docker image.
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
## HTTP Proxy Plugins
### ShortLinkPlugin
Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more
2019-10-16 06:56:39 +00:00
Add support for short links in your favorite browsers / applications.
Test Dashboard backend (#206) * Update shortlink gif name * Conditionally run workflows as necessary * Use pytest * It works but github workflow is not reporting any status :( * Separate out badges * Add python_requires to setup.py
2019-12-01 06:54:20 +00:00
[![Shortlink Plugin](https://raw.githubusercontent.com/abhinavsingh/proxy.py/develop/shortlink.gif)](https://github.com/abhinavsingh/proxy.py#shortlinkplugin)
Test Core Eventing (#205) * Add core event tests * Update .gitignore with coverage * Add shortlink gif * Add event dispatcher test * Test event subscriber
2019-12-01 06:04:43 +00:00
Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more
2019-10-16 06:56:39 +00:00
Start `proxy.py` as:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy \
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.ShortLinkPlugin
Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more
2019-10-16 06:56:39 +00:00
```
Now you can speed up your daily browsing experience by visiting your
favorite website using single character domain names :). This works
across all browsers.
Following short links are enabled by default:
Short Link | Destination URL
:--------: | :---------------:
a/ | amazon.com
i/ | instagram.com
l/ | linkedin.com
f/ | facebook.com
g/ | google.com
t/ | twitter.com
w/ | web.whatsapp.com
y/ | youtube.com
proxy/ | localhost:8899
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
### ModifyPostDataPlugin
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
Modifies POST request body before sending request to upstream server.
Start `proxy.py` as:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy \
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.ModifyPostDataPlugin
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
```
By default plugin replaces POST body content with hardcoded `b'{"key": "modified"}'`
and enforced `Content-Type: application/json`.
Verify the same using `curl -x localhost:8899 -d '{"key": "value"}' http://httpbin.org/post`
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
{
"args": {},
"data": "{\"key\": \"modified\"}",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Content-Length": "19",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "curl/7.54.0"
},
"json": {
"key": "modified"
},
"origin": "1.2.3.4, 5.6.7.8",
"url": "https://httpbin.org/post"
}
```
Note following from the response above:
1. POST data was modified `"data": "{\"key\": \"modified\"}"`.
Original `curl` command data was `{"key": "value"}`.
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
2. Our `curl` command did not add any `Content-Type` header,
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
but our plugin did add one `"Content-Type": "application/json"`.
Same can also be verified by looking at `json` field in the output above:
```
"json": {
"key": "modified"
},
```
3. Our plugin also added a `Content-Length` header to match length
of modified body.
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
### MockRestApiPlugin
Add ProposedRestApiPlugin example. (#101) * Add ProposedRestApiPlugin example. This plugin is an example to demonstrate how `proxy.py` can help you test and debug client applications without actually having a need of backend server. Simply modify ProposedRestApiPlugin as per need to return json responses for your API specs. * Add README instructions * Correct username
2019-09-27 20:00:11 +00:00
Mock responses for your server REST API.
Use to test and develop client side applications
without need of an actual upstream REST API server.
Start `proxy.py` as:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy \
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.ProposedRestApiPlugin
Add ProposedRestApiPlugin example. (#101) * Add ProposedRestApiPlugin example. This plugin is an example to demonstrate how `proxy.py` can help you test and debug client applications without actually having a need of backend server. Simply modify ProposedRestApiPlugin as per need to return json responses for your API specs. * Add README instructions * Correct username
2019-09-27 20:00:11 +00:00
```
Verify mock API response using `curl -x localhost:8899 http://api.example.com/v1/users/`
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Add ProposedRestApiPlugin example. (#101) * Add ProposedRestApiPlugin example. This plugin is an example to demonstrate how `proxy.py` can help you test and debug client applications without actually having a need of backend server. Simply modify ProposedRestApiPlugin as per need to return json responses for your API specs. * Add README instructions * Correct username
2019-09-27 20:00:11 +00:00
{"count": 2, "next": null, "previous": null, "results": [{"email": "you@example.com", "groups": [], "url": "api.example.com/v1/users/1/", "username": "admin"}, {"email": "someone@example.com", "groups": [], "url": "api.example.com/v1/users/2/", "username": "admin"}]}
```
Verify the same by inspecting `proxy.py` logs:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Add ProposedRestApiPlugin example. (#101) * Add ProposedRestApiPlugin example. This plugin is an example to demonstrate how `proxy.py` can help you test and debug client applications without actually having a need of backend server. Simply modify ProposedRestApiPlugin as per need to return json responses for your API specs. * Add README instructions * Correct username
2019-09-27 20:00:11 +00:00
2019-09-27 12:44:02,212 - INFO - pid:7077 - access_log:1210 - ::1:64792 - GET None:None/v1/users/ - None None - 0 byte
```
Access log shows `None:None` as server `ip:port`. `None` simply means that
the server connection was never made, since response was returned by our plugin.
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
Now modify `ProposedRestApiPlugin` to returns REST API mock
Add ProposedRestApiPlugin example. (#101) * Add ProposedRestApiPlugin example. This plugin is an example to demonstrate how `proxy.py` can help you test and debug client applications without actually having a need of backend server. Simply modify ProposedRestApiPlugin as per need to return json responses for your API specs. * Add README instructions * Correct username
2019-09-27 20:00:11 +00:00
responses as expected by your clients.
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
### RedirectToCustomServerPlugin
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
Redirects all incoming `http` requests to custom web server.
By default, it redirects client requests to inbuilt web server,
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
also running on `8899` port.
Start `proxy.py` and enable inbuilt web server:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy \
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
--enable-web-server \
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.RedirectToCustomServerPlugin
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
```
Verify using `curl -v -x localhost:8899 http://google.com`
```
... [redacted] ...
< HTTP/1.1 404 NOT FOUND
< Server: proxy.py v1.0.0
< Connection: Close
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
<
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
* Closing connection 0
```
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
Above `404` response was returned from `proxy.py` web server.
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
Verify the same by inspecting the logs for `proxy.py`.
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Along with the proxy request log, you must also see a http web server request log.
```
2019-09-24 19:09:33,602 - INFO - pid:49996 - access_log:1241 - ::1:49525 - GET /
2019-09-24 19:09:33,603 - INFO - pid:49995 - access_log:1157 - ::1:49524 - GET localhost:8899/ - 404 NOT FOUND - 70 bytes
```
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
### FilterByUpstreamHostPlugin
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
Drops traffic by inspecting upstream host.
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
By default, plugin drops traffic for `google.com` and `www.google.com`.
Start `proxy.py` as:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy \
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.FilterByUpstreamHostPlugin
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
```
Verify using `curl -v -x localhost:8899 http://google.com`:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
... [redacted] ...
< HTTP/1.1 418 I'm a tea pot
< Proxy-agent: proxy.py v1.0.0
* no chunk, no close, no size. Assume close to signal end
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
<
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
* Closing connection 0
```
Above `418 I'm a tea pot` is sent by our plugin.
Verify the same by inspecting logs for `proxy.py`:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
2019-09-24 19:21:37,893 - ERROR - pid:50074 - handle_readables:1347 - HttpProtocolException type raised
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Traceback (most recent call last):
... [redacted] ...
2019-09-24 19:21:37,897 - INFO - pid:50074 - access_log:1157 - ::1:49911 - GET None:None/ - None None - 0 bytes
```
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
### CacheResponsesPlugin
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Caches Upstream Server Responses.
Start `proxy.py` as:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy \
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.CacheResponsesPlugin
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
```
Verify using `curl -v -x localhost:8899 http://httpbin.org/get`:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
... [redacted] ...
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Type: application/json
< Date: Wed, 25 Sep 2019 02:24:25 GMT
< Referrer-Policy: no-referrer-when-downgrade
< Server: nginx
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-XSS-Protection: 1; mode=block
< Content-Length: 202
< Connection: keep-alive
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
<
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
{
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
"args": {},
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
"headers": {
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
"Accept": "*/*",
"Host": "httpbin.org",
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
"User-Agent": "curl/7.54.0"
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
},
"origin": "1.2.3.4, 5.6.7.8",
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
"url": "https://httpbin.org/get"
}
* Connection #0 to host localhost left intact
```
Get path to the cache file from `proxy.py` logs:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
... [redacted] ... - GET httpbin.org:80/get - 200 OK - 556 bytes
... [redacted] ... - Cached response at /var/folders/k9/x93q0_xn1ls9zy76m2mf2k_00000gn/T/httpbin.org-1569378301.407512.txt
```
Verify contents of the cache file `cat /path/to/your/cache/httpbin.org.txt`
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 25 Sep 2019 02:24:25 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 202
Connection: keep-alive
{
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
"args": {},
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
"headers": {
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
"Accept": "*/*",
"Host": "httpbin.org",
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
"User-Agent": "curl/7.54.0"
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
},
"origin": "1.2.3.4, 5.6.7.8",
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
"url": "https://httpbin.org/get"
}
```
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
### ManInTheMiddlePlugin
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Modifies upstream server responses.
Start `proxy.py` as:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy \
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.ManInTheMiddlePlugin
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
```
Verify using `curl -v -x localhost:8899 http://google.com`:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
... [redacted] ...
< HTTP/1.1 200 OK
< Content-Length: 28
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
<
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
* Connection #0 to host localhost left intact
Hello from man in the middle
```
Response body `Hello from man in the middle` is sent by our plugin.
Proxy Pool Plugin (#228) * Add proxy pool example. See #226 * Add ProxyPoolPlugin to doc
2019-12-10 03:38:49 +00:00
### ProxyPoolPlugin
Forward incoming proxy requests to a set of upstream proxy servers.
By default, `ProxyPoolPlugin` is hard-coded to use
`localhost:9000` and `localhost:9001` as upstream proxy server.
Let's start upstream proxies first.
Start `proxy.py` on port `9000` and `9001`
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy --port 9000
Proxy Pool Plugin (#228) * Add proxy pool example. See #226 * Add ProxyPoolPlugin to doc
2019-12-10 03:38:49 +00:00
```
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy --port 9001
Proxy Pool Plugin (#228) * Add proxy pool example. See #226 * Add ProxyPoolPlugin to doc
2019-12-10 03:38:49 +00:00
```
Now, start `proxy.py` with `ProxyPoolPlugin` (on default `8899` port):
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy \
Proxy Pool Plugin (#228) * Add proxy pool example. See #226 * Add ProxyPoolPlugin to doc
2019-12-10 03:38:49 +00:00
--plugins proxy.plugin.ProxyPoolPlugin
```
Make a curl request via `8899` proxy:
`curl -v -x localhost:8899 http://httpbin.org/get`
Verify that `8899` proxy forwards requests to upstream proxies
by checking respective logs.
Document FilterByClientIpPlugin & ModifyChunkResponsePlugin (#387)
2020-07-01 22:00:29 +00:00
### FilterByClientIpPlugin
Reject traffic from specific IP addresses. By default this
plugin blocks traffic from `127.0.0.1` and `::1`.
Start `proxy.py` as:
```bash
❯ proxy \
--plugins proxy.plugin.FilterByClientIpPlugin
```
Send a request using `curl -v -x localhost:8899 http://google.com`:
```bash
... [redacted] ...
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 418 I'm a tea pot
< Connection: close
<
* Closing connection 0
```
Modify plugin to your taste e.g. Allow specific IP addresses only.
### ModifyChunkResponsePlugin
This plugin demonstrate how to modify chunked encoded responses. In able to do so, this plugin uses `proxy.py` core to parse the chunked encoded response. Then we reconstruct the response using custom hardcoded chunks, ignoring original chunks received from upstream server.
Start `proxy.py` as:
```bash
❯ proxy \
--plugins proxy.plugin.ModifyChunkResponsePlugin
```
Verify using `curl -v -x localhost:8899 http://httpbin.org/stream/5`:
```bash
... [redacted] ...
modify
chunk
response
plugin
* Connection #0 to host localhost left intact
* Closing connection 0
```
Modify `ModifyChunkResponsePlugin` to your taste. Example, instead of sending hardcoded chunks, parse and modify the original `JSON` chunks received from the upstream server.
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
## HTTP Web Server Plugins
### Reverse Proxy
Extend in-built Web Server to add Reverse Proxy capabilities.
Start `proxy.py` as:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Fixes #285 (#286)
2020-02-02 20:42:20 +00:00
❯ proxy --enable-web-server \
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
--plugins proxy.plugin.ReverseProxyPlugin
```
With default configuration, `ReverseProxyPlugin` plugin is equivalent to
following `Nginx` config:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
location /get {
proxy_pass http://httpbin.org/get
}
```
Verify using `curl -v localhost:8899/get`:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "localhost",
"User-Agent": "curl/7.64.1"
},
"origin": "1.2.3.4, 5.6.7.8",
"url": "https://localhost/get"
}
```
### Web Server Route
Demonstrates inbuilt web server routing using plugin.
Start `proxy.py` as:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Fixes #285 (#286)
2020-02-02 20:42:20 +00:00
❯ proxy --enable-web-server \
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
--plugins proxy.plugin.WebServerPlugin
```
Verify using `curl -v localhost:8899/http-route-example`, should return:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc
2019-11-30 05:28:31 +00:00
HTTP route response
```
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
## Plugin Ordering
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
When using multiple plugins, depending upon plugin functionality,
it might be worth considering the order in which plugins are passed
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
on the command line.
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
Plugins are called in the same order as they are passed. Example,
say we are using both `FilterByUpstreamHostPlugin` and
`RedirectToCustomServerPlugin`. Idea is to drop all incoming `http`
requests for `google.com` and `www.google.com` and redirect other
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
`http` requests to our inbuilt web server.
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
Hence, in this scenario it is important to use
`FilterByUpstreamHostPlugin` before `RedirectToCustomServerPlugin`.
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
If we enable `RedirectToCustomServerPlugin` before `FilterByUpstreamHostPlugin`,
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
`google` requests will also get redirected to inbuilt web server,
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
instead of being dropped.
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
End-to-End Encryption
=====================
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
By default, `proxy.py` uses `http` protocol for communication with clients e.g. `curl`, `browser`.
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
For enabling end-to-end encrypting using `tls` / `https` first generate certificates:
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
make https-certificates
```
Start `proxy.py` as:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy \
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
--cert-file https-cert.pem \
--key-file https-key.pem
```
Verify using `curl -x https://localhost:8899 --proxy-cacert https-cert.pem https://httpbin.org/get`:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
{
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
"args": {},
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
"headers": {
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
"Accept": "*/*",
"Host": "httpbin.org",
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
"User-Agent": "curl/7.54.0"
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
},
"origin": "1.2.3.4, 5.6.7.8",
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
"url": "https://httpbin.org/get"
}
```
Add SSL echo server & client example (#395) * Move wrap_socket for SSL server within utils. Also complete proxy.common.pki gen_csr and sign_csr actions. Used by Makefile sign-https-certificates. * Add SSL echo server and client example * Add examples documentation
2020-07-08 11:08:58 +00:00
If you want to avoid passing `--proxy-cacert` flag, also consider signing generated SSL certificates. Example:
First, generate CA certificates:
```bash
make ca-certificates
```
Then, sign SSL certificate:
```bash
make sign-https-certificates
```
Now restart the server with `--cert-file https-signed-cert.pem` flag. Note that you must also trust generated `ca-cert.pem` in your system keychain.
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
TLS Interception
=================
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
By default, `proxy.py` will not decrypt `https` traffic between client and server.
Add flag to specify custom system CA Path (#321) * Fixes #320 * Update README and add codecov.yml * Update codecov.yml
2020-03-25 08:00:19 +00:00
To enable TLS interception first generate root CA certificates:
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Add flag to specify custom system CA Path (#321) * Fixes #320 * Update README and add codecov.yml * Update codecov.yml
2020-03-25 08:00:19 +00:00
```bash
❯ make ca-certificates
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
```
Add HttpParser.build_response utility (#96)
2019-09-26 01:27:26 +00:00
Lets also enable `CacheResponsePlugin` so that we can verify decrypted
Add flag to specify custom system CA Path (#321) * Fixes #320 * Update README and add codecov.yml * Update codecov.yml
2020-03-25 08:00:19 +00:00
response from the server. Start `proxy.py` as:
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ proxy \
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
--plugins proxy.plugin.CacheResponsesPlugin \
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
--ca-key-file ca-key.pem \
--ca-cert-file ca-cert.pem \
--ca-signing-key-file ca-signing-key.pem
```
Add flag to specify custom system CA Path (#321) * Fixes #320 * Update README and add codecov.yml * Update codecov.yml
2020-03-25 08:00:19 +00:00
TLS Interception Cert Generation (#362) * Use common.pki for interception certificate generation * Fix tests * Dont use certificate fields that we dont need, it leads to certificate generation error on Ubuntu * Prepare for v2.2.0 * npm audit fix
2020-06-09 06:37:00 +00:00
[![NOTE](https://img.shields.io/static/v1?label=MacOS&message=note&color=yellow)](https://github.com/abhinavsingh/proxy.py#flags) Also provide explicit CA bundle path needed for validation of peer certificates. See `--ca-file` flag.
Add flag to specify custom system CA Path (#321) * Fixes #320 * Update README and add codecov.yml * Update codecov.yml
2020-03-25 08:00:19 +00:00
Verify TLS interception using `curl`
```bash
❯ curl -v -x localhost:8899 --cacert ca-cert.pem https://httpbin.org/get
```
Add HttpParser.build_response utility (#96)
2019-09-26 01:27:26 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Add HttpParser.build_response utility (#96)
2019-09-26 01:27:26 +00:00
* issuer: C=US; ST=CA; L=SanFrancisco; O=proxy.py; OU=CA; CN=Proxy PY CA; emailAddress=proxyca@mailserver.com
* SSL certificate verify ok.
> GET /get HTTP/1.1
... [redacted] ...
< Connection: keep-alive
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
<
Add HttpParser.build_response utility (#96)
2019-09-26 01:27:26 +00:00
{
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
"args": {},
Add HttpParser.build_response utility (#96)
2019-09-26 01:27:26 +00:00
"headers": {
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
"Accept": "*/*",
"Host": "httpbin.org",
Add HttpParser.build_response utility (#96)
2019-09-26 01:27:26 +00:00
"User-Agent": "curl/7.54.0"
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
},
"origin": "1.2.3.4, 5.6.7.8",
Add HttpParser.build_response utility (#96)
2019-09-26 01:27:26 +00:00
"url": "https://httpbin.org/get"
}
```
The `issuer` line confirms that response was intercepted.
Also verify the contents of cached response file. Get path to the cache
file from `proxy.py` logs.
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
`❯ cat /path/to/your/tmp/directory/httpbin.org-1569452863.924174.txt`
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Add HttpParser.build_response utility (#96)
2019-09-26 01:27:26 +00:00
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 25 Sep 2019 23:07:05 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 202
Connection: keep-alive
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
{
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
"args": {},
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
"headers": {
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
"Accept": "*/*",
"Host": "httpbin.org",
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
"User-Agent": "curl/7.54.0"
Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins
2019-10-13 00:43:19 +00:00
},
"origin": "1.2.3.4, 5.6.7.8",
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
"url": "https://httpbin.org/get"
}
```
Add HttpParser.build_response utility (#96)
2019-09-26 01:27:26 +00:00
Viola!!! If you remove CA flags, encrypted data will be found in the
cached file instead of plain text.
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
Now use CA flags with other
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
[plugin examples](#plugin-examples) to see them work with `https` traffic.
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
Enable docker containers for TLS Interception (#375) * Move static values to constants.py * Install openssl in docker containers for TLS interception * Complete TLS Interception with Docker doc
2020-06-17 12:24:43 +00:00
## TLS Interception With Docker
Important notes about TLS Interception with Docker container:
- Since `v2.2.0`, `proxy.py` docker container also ships with `openssl`. This allows `proxy.py`
to generate certificates on the fly for TLS Interception.
- For security reasons, `proxy.py` docker container doesn't ship with CA certificates.
Here is how to start a `proxy.py` docker container
with TLS Interception:
1. Generate CA certificates on host computer
```bash
❯ make ca-certificates
```
2. Copy all generated certificates into a separate directory. We'll later mount this directory into our docker container
```bash
❯ mkdir /tmp/ca-certificates
❯ cp ca-cert.pem ca-key.pem ca-signing-key.pem /tmp/ca-certificates
```
3. Start docker container
```bash
❯ docker run -it --rm \
-v /tmp/ca-certificates:/tmp/ca-certificates \
-p 8899:8899 \
abhinavsingh/proxy.py:latest \
--hostname 0.0.0.0 \
--plugins proxy.plugin.CacheResponsesPlugin \
--ca-key-file /tmp/ca-certificates/ca-key.pem \
--ca-cert-file /tmp/ca-certificates/ca-cert.pem \
--ca-signing-key /tmp/ca-certificates/ca-signing-key.pem
```
- `-v /tmp/ca-certificates:/tmp/ca-certificates` flag mounts our CA certificate directory in container environment
- `--plugins proxy.plugin.CacheResponsesPlugin` enables `CacheResponsesPlugin` so that we can inspect intercepted traffic
- `--ca-*` flags enable TLS Interception.
4. From another terminal, try TLS Interception using `curl`. You can omit `--cacert` flag if CA certificate is already trusted by the system.
```bash
❯ curl -v \
--cacert ca-cert.pem \
-x 127.0.0.1:8899 \
https://httpbin.org/get
```
5. Verify `issuer` field from response headers.
```bash
* Server certificate:
* subject: CN=httpbin.org; C=NA; ST=Unavailable; L=Unavailable; O=Unavailable; OU=Unavailable
* start date: Jun 17 09:26:57 2020 GMT
* expire date: Jun 17 09:26:57 2022 GMT
* subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
* issuer: CN=example.com
* SSL certificate verify ok.
```
6. Back on docker terminal, copy response dump path logs.
```bash
...[redacted]... [I] access_log:338 - 172.17.0.1:56498 - CONNECT httpbin.org:443 - 1031 bytes - 1216.70 ms
...[redacted]... [I] close:49 - Cached response at /tmp/httpbin.org-ae1a927d064e4ab386ea319eb38fe251.txt
```
7. In another terminal, `cat` the response dump:
```bash
❯ docker exec -it $(docker ps | grep proxy.py | awk '{ print $1 }') cat /tmp/httpbin.org-ae1a927d064e4ab386ea319eb38fe251.txt
HTTP/1.1 200 OK
...[redacted]...
{
...[redacted]...,
"url": "http://httpbin.org/get"
}
```
Make HTTP handler constructor free of socket file number (#219) * Refactor into acceptor module * Add tunnel doc * Make fileno free * Autopep8
2019-12-02 23:55:08 +00:00
Proxy Over SSH Tunnel
=====================
Enable docker containers for TLS Interception (#375) * Move static values to constants.py * Install openssl in docker containers for TLS interception * Complete TLS Interception with Docker doc
2020-06-17 12:24:43 +00:00
**This is a WIP and may not work as documented**
Make HTTP handler constructor free of socket file number (#219) * Refactor into acceptor module * Add tunnel doc * Make fileno free * Autopep8
2019-12-02 23:55:08 +00:00
Requires `paramiko` to work. See [requirements-tunnel.txt](https://github.com/abhinavsingh/proxy.py/blob/develop/requirements-tunnel.txt)
## Proxy Remote Requests Locally
|
+------------+ | +----------+
| LOCAL | | | REMOTE |
| HOST | <== SSH ==== :8900 == | SERVER |
+------------+ | +----------+
:8899 proxy.py |
|
FIREWALL
(allow tcp/22)
## What
Proxy HTTP(s) requests made on a `remote` server through `proxy.py` server
running on `localhost`.
### How
* Requested `remote` port is forwarded over the SSH connection.
* `proxy.py` running on the `localhost` handles and responds to
`remote` proxy requests.
### Requirements
1. `localhost` MUST have SSH access to the `remote` server
2. `remote` server MUST be configured to proxy HTTP(s) requests
through the forwarded port number e.g. `:8900`.
- `remote` and `localhost` ports CAN be same e.g. `:8899`.
- `:8900` is chosen in ascii art for differentiation purposes.
### Try it
Start `proxy.py` as:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ # On localhost
❯ proxy --enable-tunnel \
Make HTTP handler constructor free of socket file number (#219) * Refactor into acceptor module * Add tunnel doc * Make fileno free * Autopep8
2019-12-02 23:55:08 +00:00
--tunnel-username username \
--tunnel-hostname ip.address.or.domain.name \
--tunnel-port 22 \
--tunnel-remote-host 127.0.0.1
--tunnel-remote-port 8899
```
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
Make a HTTP proxy request on `remote` server and
Make HTTP handler constructor free of socket file number (#219) * Refactor into acceptor module * Add tunnel doc * Make fileno free * Autopep8
2019-12-02 23:55:08 +00:00
verify that response contains public IP address of `localhost` as origin:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ # On remote
❯ curl -x 127.0.0.1:8899 http://httpbin.org/get
Make HTTP handler constructor free of socket file number (#219) * Refactor into acceptor module * Add tunnel doc * Make fileno free * Autopep8
2019-12-02 23:55:08 +00:00
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "httpbin.org",
"User-Agent": "curl/7.54.0"
},
"origin": "x.x.x.x, y.y.y.y",
"url": "https://httpbin.org/get"
}
```
Also, verify that `proxy.py` logs on `localhost` contains `remote` IP as client IP.
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Make HTTP handler constructor free of socket file number (#219) * Refactor into acceptor module * Add tunnel doc * Make fileno free * Autopep8
2019-12-02 23:55:08 +00:00
access_log:328 - remote:52067 - GET httpbin.org:80
```
## Proxy Local Requests Remotely
|
+------------+ | +----------+
| LOCAL | | | REMOTE |
| HOST | === SSH =====> | SERVER |
+------------+ | +----------+
| :8899 proxy.py
|
FIREWALL
(allow tcp/22)
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
Embed proxy.py
==============
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
Add non-blocking embedded mode feature (#159) * Fixes #158 * mypy fixes * Instructions for non-blocking embed mode * Toggle running flag before shutdown
2019-11-08 04:53:08 +00:00
## Blocking Mode
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
Start `proxy.py` in embedded mode with default configuration
Devtools Protocol (#174) * Refine docs * Decouple relay from dashboard. Will be re-used by devtools protocol plugin. * Just have a single manager for all eventing * Ofcourse managers cant be shared across processes * Remove unused * Add DevtoolsProtocolPlugin * Emit REQUEST_COMPLETE core event * Emit only if --enable-events used * Add event emitter for response cycle * Fill up core events to devtools protocol expectations * Serve static content with Cache-Control header and gzip compression * Add PWA manifest.json and icons from sample PWA apps (replace later) * Catch any exception and be ssl agnostic * Add CSP headers and avoid inline scripts * Re-enable iframe and deobfuscation * Embed plugins within <section/> block * Make tab switching agnostic of block name * Add support for browser history on tab change * Default hash to #home * Switch to tab if hash is already set * Expand canvas to fill screen even without content * Remove inline css for embedded devtools * Make dashboard backend websocket API pluggable * doc
2019-11-15 03:00:07 +00:00
by using `proxy.main` method. Example:
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
import proxy
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
if __name__ == '__main__':
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
proxy.main()
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
```
Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more
2019-10-16 06:56:39 +00:00
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
Customize startup flags by passing list of input arguments:
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
import proxy
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
if __name__ == '__main__':
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
proxy.main([
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
'--hostname', '::1',
'--port', '8899'
])
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
```
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
or, customize startup flags by passing them as kwargs:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
import ipaddress
import proxy
if __name__ == '__main__':
proxy.main(
hostname=ipaddress.IPv6Address('::1'),
port=8899
)
```
Add non-blocking embedded mode feature (#159) * Fixes #158 * mypy fixes * Instructions for non-blocking embed mode * Toggle running flag before shutdown
2019-11-08 04:53:08 +00:00
Note that:
1. Calling `main` is simply equivalent to starting `proxy.py` from command line.
2. `main` will block until `proxy.py` shuts down.
## Non-blocking Mode
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
Start `proxy.py` in non-blocking embedded mode with default configuration
by using `start` method: Example:
Add non-blocking embedded mode feature (#159) * Fixes #158 * mypy fixes * Instructions for non-blocking embed mode * Toggle running flag before shutdown
2019-11-08 04:53:08 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
import proxy
Add non-blocking embedded mode feature (#159) * Fixes #158 * mypy fixes * Instructions for non-blocking embed mode * Toggle running flag before shutdown
2019-11-08 04:53:08 +00:00
if __name__ == '__main__':
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
with proxy.start([]):
Add non-blocking embedded mode feature (#159) * Fixes #158 * mypy fixes * Instructions for non-blocking embed mode * Toggle running flag before shutdown
2019-11-08 04:53:08 +00:00
# ... your logic here ...
```
Note that:
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
1. `start` is similar to `main`, except `start` won't block.
1. `start` is a context manager.
It will start `proxy.py` when called and will shut it down
once scope ends.
Devtools Protocol (#174) * Refine docs * Decouple relay from dashboard. Will be re-used by devtools protocol plugin. * Just have a single manager for all eventing * Ofcourse managers cant be shared across processes * Remove unused * Add DevtoolsProtocolPlugin * Emit REQUEST_COMPLETE core event * Emit only if --enable-events used * Add event emitter for response cycle * Fill up core events to devtools protocol expectations * Serve static content with Cache-Control header and gzip compression * Add PWA manifest.json and icons from sample PWA apps (replace later) * Catch any exception and be ssl agnostic * Add CSP headers and avoid inline scripts * Re-enable iframe and deobfuscation * Embed plugins within <section/> block * Make tab switching agnostic of block name * Add support for browser history on tab change * Default hash to #home * Switch to tab if hash is already set * Expand canvas to fill screen even without content * Remove inline css for embedded devtools * Make dashboard backend websocket API pluggable * doc
2019-11-15 03:00:07 +00:00
3. Just like `main`, startup flags with `start` method
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
can be customized by either passing flags as list of
Devtools Protocol (#174) * Refine docs * Decouple relay from dashboard. Will be re-used by devtools protocol plugin. * Just have a single manager for all eventing * Ofcourse managers cant be shared across processes * Remove unused * Add DevtoolsProtocolPlugin * Emit REQUEST_COMPLETE core event * Emit only if --enable-events used * Add event emitter for response cycle * Fill up core events to devtools protocol expectations * Serve static content with Cache-Control header and gzip compression * Add PWA manifest.json and icons from sample PWA apps (replace later) * Catch any exception and be ssl agnostic * Add CSP headers and avoid inline scripts * Re-enable iframe and deobfuscation * Embed plugins within <section/> block * Make tab switching agnostic of block name * Add support for browser history on tab change * Default hash to #home * Switch to tab if hash is already set * Expand canvas to fill screen even without content * Remove inline css for embedded devtools * Make dashboard backend websocket API pluggable * doc
2019-11-15 03:00:07 +00:00
input arguments e.g. `start(['--port', '8899'])` or
by using passing flags as kwargs e.g. `start(port=8899)`.
Add non-blocking embedded mode feature (#159) * Fixes #158 * mypy fixes * Instructions for non-blocking embed mode * Toggle running flag before shutdown
2019-11-08 04:53:08 +00:00
Documentation for plugin loading in embedded mode (#422)
2020-08-14 11:50:19 +00:00
## Loading Plugins
You can, of course, list plugins to load in the input arguments list of `proxy.main`, `proxy.start` or the `Proxy` constructor. Use the `--plugins` flag as when starting from command line:
```python
import proxy
if __name__ == '__main__':
proxy.main([
'--plugins', 'proxy.plugin.CacheResponsesPlugin',
])
```
However, for simplicity you can pass the list of plugins to load as a keyword argument to `proxy.main`, `proxy.start` or the `Proxy` constructor:
```python
import proxy
from proxy.plugin import FilterByUpstreamHostPlugin
if __name__ == '__main__':
proxy.main([], plugins=[
b'proxy.plugin.CacheResponsesPlugin',
FilterByUpstreamHostPlugin,
])
```
Note that it supports:
1. The fully-qualified name of a class as `bytes`
2. Any `type` instance for a Proxy.py plugin class. This is espacially useful for custom plugins defined locally.
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
Unit testing with proxy.py
==========================
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
## proxy.TestCase
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
To setup and teardown `proxy.py` for your Python unittest classes,
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
simply use `proxy.TestCase` instead of `unittest.TestCase`.
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
Example:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
import proxy
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
class TestProxyPyEmbedded(proxy.TestCase):
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
def test_my_application_with_proxy(self) -> None:
self.assertTrue(True)
```
Note that:
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
1. `proxy.TestCase` overrides `unittest.TestCase.run()` method to setup and teardown `proxy.py`.
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
2. `proxy.py` server will listen on a random available port on the system.
autopep8 (#196) * autopep8 * Update TestCase section
2019-11-27 06:43:05 +00:00
This random port is available as `self.PROXY_PORT` within your test cases.
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
3. Only a single worker is started by default (`--num-workers 1`) for faster setup and teardown.
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
4. Most importantly, `proxy.TestCase` also ensures `proxy.py` server
autopep8 (#196) * autopep8 * Update TestCase section
2019-11-27 06:43:05 +00:00
is up and running before proceeding with execution of tests. By default,
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
`proxy.TestCase` will wait for `10 seconds` for `proxy.py` server to start,
autopep8 (#196) * autopep8 * Update TestCase section
2019-11-27 06:43:05 +00:00
upon failure a `TimeoutError` exception will be raised.
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
## Override startup flags
To override default startup flags, define a `PROXY_PY_STARTUP_FLAGS` variable in your test class.
Example:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
class TestProxyPyEmbedded(TestCase):
PROXY_PY_STARTUP_FLAGS = [
'--num-workers', '1',
'--enable-web-server',
]
def test_my_application_with_proxy(self) -> None:
self.assertTrue(True)
```
See [test_embed.py](https://github.com/abhinavsingh/proxy.py/blob/develop/tests/test_embed.py)
for full working example.
## With unittest.TestCase
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
If for some reasons you are unable to directly use `proxy.TestCase`,
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
then simply override `unittest.TestCase.run` yourself to setup and teardown `proxy.py`.
Example:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
import unittest
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
import proxy
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
class TestProxyPyEmbedded(unittest.TestCase):
def test_my_application_with_proxy(self) -> None:
self.assertTrue(True)
def run(self, result: Optional[unittest.TestResult] = None) -> Any:
Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8
2019-11-14 00:38:59 +00:00
with proxy.start([
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests.
2019-11-12 20:25:47 +00:00
'--num-workers', '1',
'--port', '... random port ...']):
super().run(result)
```
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
or simply setup / teardown `proxy.py` within
autopep8 (#196) * autopep8 * Update TestCase section
2019-11-27 06:43:05 +00:00
`setUpClass` and `teardownClass` class methods.
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
Plugin Developer and Contributor Guide
======================================
## Everything is a plugin
As you might have guessed by now, in `proxy.py` everything is a plugin.
- We enabled proxy server plugins using `--plugins` flag.
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
All the [plugin examples](#plugin-examples) were implementing
`HttpProxyBasePlugin`. See documentation of
[HttpProxyBasePlugin](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L894-L938)
for available lifecycle hooks. Use `HttpProxyBasePlugin` to modify
behavior of http(s) proxy protocol between client and upstream server.
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
Example, [FilterByUpstreamHostPlugin](#filterbyupstreamhostplugin).
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
- We also enabled inbuilt web server using `--enable-web-server`.
Inbuilt web server implements `HttpProtocolHandlerPlugin` plugin.
See documentation of [HttpProtocolHandlerPlugin](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L793-L850)
for available lifecycle hooks. Use `HttpProtocolHandlerPlugin` to add
new features for http(s) clients. Example,
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
[HttpWebServerPlugin](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L1185-L1260).
- There also is a `--disable-http-proxy` flag. It disables inbuilt proxy server.
Use this flag with `--enable-web-server` flag to run `proxy.py` as a programmable
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
http(s) server. [HttpProxyPlugin](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L941-L1182)
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
also implements `HttpProtocolHandlerPlugin`.
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
## Internal Architecture
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
- [HttpProtocolHandler](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L1263-L1440)
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
thread is started with the accepted [TcpClientConnection](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L230-L237).
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
`HttpProtocolHandler` is responsible for parsing incoming client request and invoking
`HttpProtocolHandlerPlugin` lifecycle hooks.
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
- `HttpProxyPlugin` which implements `HttpProtocolHandlerPlugin` also has its own plugin
mechanism. Its responsibility is to establish connection between client and
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
upstream [TcpServerConnection](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L204-L227)
and invoke `HttpProxyBasePlugin` lifecycle hooks.
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
- `HttpProtocolHandler` threads are started by [Acceptor](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L424-L472)
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
processes.
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
- `--num-workers` `Acceptor` processes are started by
[AcceptorPool](https://github.com/abhinavsingh/proxy.py/blob/b03629fa0df1595eb4995427bc601063be7fdca9/proxy.py#L368-L421)
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
on start-up.
Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more
2019-10-16 06:56:39 +00:00
- `AcceptorPool` listens on server socket and pass the handler to `Acceptor` processes.
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
Workers are responsible for accepting new client connections and starting
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
`HttpProtocolHandler` thread.
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
## Development Guide
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
Move benchmark module within proxy (#181) * Move benchmark within proxy module * chmod 0644 for benchmark.py which was executable till now * Turn utilities into its own section
2019-11-16 07:49:51 +00:00
### Setup Local Environment
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
Contributors must start `proxy.py` from source to verify and develop new features / fixes.
Move benchmark module within proxy (#181) * Move benchmark within proxy module * chmod 0644 for benchmark.py which was executable till now * Turn utilities into its own section
2019-11-16 07:49:51 +00:00
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
See [Run proxy.py from command line using repo source](#from-command-line-using-repo-source) for details.
Test coverage and mypy compliance (#94) * Add tests and mypy compliance for tests * Add pragma: no cover for abstract class methods * mypy --strict compliance for tests.py * Refactor and test coverage * lint fixes
2019-09-25 22:51:12 +00:00
Move benchmark module within proxy (#181) * Move benchmark within proxy module * chmod 0644 for benchmark.py which was executable till now * Turn utilities into its own section
2019-11-16 07:49:51 +00:00
### Setup pre-commit hook
Pre-commit hook ensures lint checking and tests execution.
Test coverage and mypy compliance (#94) * Add tests and mypy compliance for tests * Add pragma: no cover for abstract class methods * mypy --strict compliance for tests.py * Refactor and test coverage * lint fixes
2019-09-25 22:51:12 +00:00
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
1. `cd /path/to/proxy.py`
2. `ln -s $(PWD)/git-pre-commit .git/hooks/pre-commit`
Test coverage and mypy compliance (#94) * Add tests and mypy compliance for tests * Add pragma: no cover for abstract class methods * mypy --strict compliance for tests.py * Refactor and test coverage * lint fixes
2019-09-25 22:51:12 +00:00
Move benchmark module within proxy (#181) * Move benchmark within proxy module * chmod 0644 for benchmark.py which was executable till now * Turn utilities into its own section
2019-11-16 07:49:51 +00:00
### Sending a Pull Request
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
Every pull request is tested using GitHub actions.
Move benchmark module within proxy (#181) * Move benchmark within proxy module * chmod 0644 for benchmark.py which was executable till now * Turn utilities into its own section
2019-11-16 07:49:51 +00:00
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
See [GitHub workflow](https://github.com/abhinavsingh/proxy.py/tree/develop/.github/workflows)
for list of tests.
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
Move benchmark module within proxy (#181) * Move benchmark within proxy module * chmod 0644 for benchmark.py which was executable till now * Turn utilities into its own section
2019-11-16 07:49:51 +00:00
Utilities
=========
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
## TCP Sockets
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
### new_socket_connection
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
Attempts to create an IPv4 connection, then IPv6 and
finally a dual stack connection to provided address.
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
>>> conn = new_socket_connection(('httpbin.org', 80))
>>> ...[ use connection ]...
>>> conn.close()
```
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
### socket_connection
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
`socket_connection` is a convenient decorator + context manager
around `new_socket_connection` which ensures `conn.close` is implicit.
As a context manager:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
>>> with socket_connection(('httpbin.org', 80)) as conn:
>>> ... [ use connection ] ...
```
As a decorator:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
>>> @socket_connection(('httpbin.org', 80))
>>> def my_api_call(conn, *args, **kwargs):
>>> ... [ use connection ] ...
```
## Http Client
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
### build_http_request
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
#### Generate HTTP GET request
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
>>> build_http_request(b'GET', b'/')
b'GET / HTTP/1.1\r\n\r\n'
>>>
```
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
#### Generate HTTP GET request with headers
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
>>> build_http_request(b'GET', b'/',
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
headers={b'Connection': b'close'})
b'GET / HTTP/1.1\r\nConnection: close\r\n\r\n'
>>>
```
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
#### Generate HTTP POST request with headers and body
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```python
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
>>> import json
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
>>> build_http_request(b'POST', b'/form',
headers={b'Content-type': b'application/json'},
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
body=proxy.bytes_(json.dumps({'email': 'hello@world.com'})))
b'POST /form HTTP/1.1\r\nContent-type: application/json\r\n\r\n{"email": "hello@world.com"}'
```
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
### build_http_response
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
[README] Mention 3.9 support tested locally on Mac using pyenv install 3.9-dev (#288)
2020-02-03 04:49:01 +00:00
```python
build_http_response(
status_code: int,
protocol_version: bytes = HTTP_1_1,
reason: Optional[bytes] = None,
headers: Optional[Dict[bytes, bytes]] = None,
body: Optional[bytes] = None) -> bytes
```
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
## PKI
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
### API Usage
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
#### gen_private_key
[README] Mention 3.9 support tested locally on Mac using pyenv install 3.9-dev (#288)
2020-02-03 04:49:01 +00:00
```python
gen_private_key(
key_path: str,
password: str,
bits: int = 2048,
timeout: int = 10) -> bool
```
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
#### gen_public_key
[README] Mention 3.9 support tested locally on Mac using pyenv install 3.9-dev (#288)
2020-02-03 04:49:01 +00:00
```python
gen_public_key(
public_key_path: str,
private_key_path: str,
private_key_password: str,
subject: str,
alt_subj_names: Optional[List[str]] = None,
extended_key_usage: Optional[str] = None,
validity_in_days: int = 365,
timeout: int = 10) -> bool
```
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
#### remove_passphrase
[README] Mention 3.9 support tested locally on Mac using pyenv install 3.9-dev (#288)
2020-02-03 04:49:01 +00:00
```python
remove_passphrase(
key_in_path: str,
password: str,
key_out_path: str,
timeout: int = 10) -> bool
```
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
#### gen_csr
[README] Mention 3.9 support tested locally on Mac using pyenv install 3.9-dev (#288)
2020-02-03 04:49:01 +00:00
```python
gen_csr(
csr_path: str,
key_path: str,
password: str,
crt_path: str,
timeout: int = 10) -> bool
```
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
#### sign_csr
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
[README] Mention 3.9 support tested locally on Mac using pyenv install 3.9-dev (#288)
2020-02-03 04:49:01 +00:00
```python
sign_csr(
csr_path: str,
crt_path: str,
ca_key_path: str,
ca_key_password: str,
ca_crt_path: str,
serial: str,
alt_subj_names: Optional[List[str]] = None,
extended_key_usage: Optional[str] = None,
validity_in_days: int = 365,
timeout: int = 10) -> bool
```
See [pki.py](https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/common/pki.py) and
[test_pki.py](https://github.com/abhinavsingh/proxy.py/blob/develop/tests/common/test_pki.py)
Refactor into event submodule (#257)
2020-01-07 19:38:25 +00:00
for usage examples.
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
### CLI Usage
Use `proxy.common.pki` module for:
1) Generation of public and private keys
2) Generating CSR requests
3) Signing CSR requests using custom CA.
```bash
python -m proxy.common.pki -h
usage: pki.py [-h] [--password PASSWORD] [--private-key-path PRIVATE_KEY_PATH]
[--public-key-path PUBLIC_KEY_PATH] [--subject SUBJECT]
action
TLS Interception Cert Generation (#362) * Use common.pki for interception certificate generation * Fix tests * Dont use certificate fields that we dont need, it leads to certificate generation error on Ubuntu * Prepare for v2.2.0 * npm audit fix
2020-06-09 06:37:00 +00:00
proxy.py v2.2.0 : PKI Utility
Add CLI usage for pki.py and update Makefile (#254) * Add CLI usage for pki.py * Bump to 2.1.0 * Replace direct openssl invocation with pki utility * Bolder * Ordering and version in README * Refine help
2020-01-07 03:51:18 +00:00
positional arguments:
action Valid actions: remove_passphrase, gen_private_key,
gen_public_key, gen_csr, sign_csr
optional arguments:
-h, --help show this help message and exit
--password PASSWORD Password to use for encryption. Default: proxy.py
--private-key-path PRIVATE_KEY_PATH
Private key path
--public-key-path PUBLIC_KEY_PATH
Public key path
--subject SUBJECT Subject to use for public key generation. Default:
/CN=example.com
```
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
## Internal Documentation
Browse through internal class hierarchy and documentation using `pydoc3`.
Example:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ pydoc3 proxy
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
PACKAGE CONTENTS
__main__
common (package)
core (package)
http (package)
main
FILE
/Users/abhinav/Dev/proxy.py/proxy/__init__.py
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
```
Frequently Asked Questions
==========================
Add py.typed marker, add version checker, remove deprecated methods (#255) * Add py.typed marker * Add version-check.py * Remove deprecated assertDictContainsSubset usage
2020-01-07 05:23:38 +00:00
## Threads vs Threadless
Pre v2.x, `proxy.py` used to spawn new threads for handling
client requests.
Starting v2.x, `proxy.py` added support for threadless execution of
client requests using `asyncio`.
In future, threadless execution will be the default mode.
Refactor into event submodule (#257)
2020-01-07 19:38:25 +00:00
Add py.typed marker, add version checker, remove deprecated methods (#255) * Add py.typed marker * Add version-check.py * Remove deprecated assertDictContainsSubset usage
2020-01-07 05:23:38 +00:00
Till then if you are interested in trying it out,
start `proxy.py` with `--threadless` flag.
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
## SyntaxError: invalid syntax
[README] Mention 3.9 support tested locally on Mac using pyenv install 3.9-dev (#288)
2020-02-03 04:49:01 +00:00
`proxy.py` is strictly typed and uses Python `typing` annotations. Example:
```python
>>> my_strings : List[str] = []
>>> #############^^^^^^^^^#####
```
Hence a Python version that understands typing annotations is required.
Update dependencies (#318) Also fixes #317
2020-03-24 15:39:32 +00:00
Make sure you are using `Python 3.6+`.
[README] Mention 3.9 support tested locally on Mac using pyenv install 3.9-dev (#288)
2020-02-03 04:49:01 +00:00
Verify the version before running `proxy.py`:
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
`❯ python --version`
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
[README] Mention 3.9 support tested locally on Mac using pyenv install 3.9-dev (#288)
2020-02-03 04:49:01 +00:00
All `typing` annotations can be replaced with `comment-only` annotations. Example:
```python
>>> my_strings = [] # List[str]
>>> ################^^^^^^^^^^^
```
Update dependencies (#318) Also fixes #317
2020-03-24 15:39:32 +00:00
It will enable `proxy.py` to run on Python `pre-3.6`, even on `2.7`.
[README] Mention 3.9 support tested locally on Mac using pyenv install 3.9-dev (#288)
2020-02-03 04:49:01 +00:00
However, as all future versions of Python will support `typing` annotations,
this has not been considered.
Consistent dashboard look and feel across plugins (#169) * Explicitly link version changelog in TOC * Separate out app header body builder * Ensure unsubscribe when disabling inspection. Fixes #164 * Avoid creation of new manager per dashboard instance. * Add UI header for all plugins (tabs) * Ensure app body for all plugin skeleton * Move app-header and app-body within core for consistent dashboard look and feel * Consistent UI header body for plugins * autopep8
2019-11-12 23:29:37 +00:00
## Unable to load plugins
Make sure plugin modules are discoverable by adding them to `PYTHONPATH`. Example:
`PYTHONPATH=/path/to/my/app proxy --plugins my_app.proxyPlugin`
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Consistent dashboard look and feel across plugins (#169) * Explicitly link version changelog in TOC * Separate out app header body builder * Ensure unsubscribe when disabling inspection. Fixes #164 * Avoid creation of new manager per dashboard instance. * Add UI header for all plugins (tabs) * Ensure app body for all plugin skeleton * Move app-header and app-body within core for consistent dashboard look and feel * Consistent UI header body for plugins * autopep8
2019-11-12 23:29:37 +00:00
...[redacted]... - Loaded plugin proxy.HttpProxyPlugin
...[redacted]... - Loaded plugin my_app.proxyPlugin
```
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
OR, simply pass fully-qualified path as parameter, e.g.
Consistent dashboard look and feel across plugins (#169) * Explicitly link version changelog in TOC * Separate out app header body builder * Ensure unsubscribe when disabling inspection. Fixes #164 * Avoid creation of new manager per dashboard instance. * Add UI header for all plugins (tabs) * Ensure app body for all plugin skeleton * Move app-header and app-body within core for consistent dashboard look and feel * Consistent UI header body for plugins * autopep8
2019-11-12 23:29:37 +00:00
`proxy --plugins /path/to/my/app/my_app.proxyPlugin`
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
## Unable to connect with proxy.py from remote host
Make sure `proxy.py` is listening on correct network interface.
Try following flags:
- For IPv6 `--hostname ::`
- For IPv4 `--hostname 0.0.0.0`
## Basic auth not working with a browser
Most likely it's a browser integration issue with system keychain.
- First verify that basic auth is working using `curl`
`curl -v -x username:password@localhost:8899 https://httpbin.org/get`
- See [this thread](https://github.com/abhinavsingh/proxy.py/issues/89#issuecomment-534845710)
for further details.
## Docker image not working on macOS
It's a compatibility issue with `vpnkit`.
See [moby/vpnkit exhausts docker resources](https://github.com/abhinavsingh/proxy.py/issues/43)
and [Connection refused: The proxy could not connect](https://github.com/moby/vpnkit/issues/469)
for some background.
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
## GCE log viewer integration for proxy.py
A starter [fluentd.conf](https://github.com/abhinavsingh/proxy.py/blob/develop/fluentd.conf)
template is available.
1. Copy this configuration file as `proxy.py.conf` under
`/etc/google-fluentd/config.d/`
2. Update `path` field to log file path as used with `--log-file` flag.
By default `/tmp/proxy.log` path is tailed.
3. Reload `google-fluentd`:
`sudo service google-fluentd restart`
Now `proxy.py` logs can be browsed using
[GCE log viewer](https://console.cloud.google.com/logs/viewer).
## ValueError: filedescriptor out of range in select
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
`proxy.py` is made to handle thousands of connections per second
without any socket leaks.
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
1. Make use of `--open-file-limit` flag to customize `ulimit -n`.
2. Make sure to adjust `--backlog` flag for higher concurrency.
If nothing helps, [open an issue](https://github.com/abhinavsingh/proxy.py/issues/new)
with `requests per second` sent and output of following debug script:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
❯ ./helper/monitor_open_files.sh <proxy-py-pid>
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
```
Test coverage and mypy compliance (#94) * Add tests and mypy compliance for tests * Add pragma: no cover for abstract class methods * mypy --strict compliance for tests.py * Refactor and test coverage * lint fixes
2019-09-25 22:51:12 +00:00
Doc & Banner update to match GitHub (#182) * Update doc and banner * Update banner to match GitHub * Update older banners too * Add update_desc to .gitignore * Update banner for dashboard to match github * also update html, js, css
2019-11-19 04:45:51 +00:00
## None:None in access logs
Sometimes you may see `None:None` in access logs. It simply means
that an upstream server connection was never established i.e.
`upstream_host=None`, `upstream_port=None`.
There can be several reasons for no upstream connection,
few obvious ones include:
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
1. Client established a connection but never completed the request.
Doc & Banner update to match GitHub (#182) * Update doc and banner * Update banner to match GitHub * Update older banners too * Add update_desc to .gitignore * Update banner for dashboard to match github * also update html, js, css
2019-11-19 04:45:51 +00:00
2. A plugin returned a response prematurely, avoiding connection to upstream server.
Lint & Doc (#91) * Remove redundant __init__ from plugins * autopep8 * Add plugin developer and contributor guide * Wordings
2019-09-25 05:43:29 +00:00
Flags
Doc update (#90) * Add documentation * Add TOC * Add links to encryption and tls * fix cert-file and key-file * Add example for verification of https and tls interception * Bigger fonts * Redact
2019-09-25 03:44:30 +00:00
=====
updated README.md
2013-08-31 09:19:22 +00:00
Highlight language syntax (#240) * Highlight lang syntax * zsh prompt
2019-12-22 04:57:29 +00:00
```bash
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
❯ proxy -h
usage: proxy [-h] [--backlog BACKLOG] [--basic-auth BASIC_AUTH]
[--ca-key-file CA_KEY_FILE] [--ca-cert-dir CA_CERT_DIR]
[--ca-cert-file CA_CERT_FILE]
[--ca-signing-key-file CA_SIGNING_KEY_FILE]
[--cert-file CERT_FILE]
[--client-recvbuf-size CLIENT_RECVBUF_SIZE]
[--devtools-ws-path DEVTOOLS_WS_PATH]
[--disable-headers DISABLE_HEADERS] [--disable-http-proxy]
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
[--enable-dashboard] [--enable-devtools] [--enable-events]
Devtools Protocol (#174) * Refine docs * Decouple relay from dashboard. Will be re-used by devtools protocol plugin. * Just have a single manager for all eventing * Ofcourse managers cant be shared across processes * Remove unused * Add DevtoolsProtocolPlugin * Emit REQUEST_COMPLETE core event * Emit only if --enable-events used * Add event emitter for response cycle * Fill up core events to devtools protocol expectations * Serve static content with Cache-Control header and gzip compression * Add PWA manifest.json and icons from sample PWA apps (replace later) * Catch any exception and be ssl agnostic * Add CSP headers and avoid inline scripts * Re-enable iframe and deobfuscation * Embed plugins within <section/> block * Make tab switching agnostic of block name * Add support for browser history on tab change * Default hash to #home * Switch to tab if hash is already set * Expand canvas to fill screen even without content * Remove inline css for embedded devtools * Make dashboard backend websocket API pluggable * doc
2019-11-15 03:00:07 +00:00
[--enable-static-server] [--enable-web-server]
[--hostname HOSTNAME] [--key-file KEY_FILE]
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
[--log-level LOG_LEVEL] [--log-file LOG_FILE]
[--log-format LOG_FORMAT] [--num-workers NUM_WORKERS]
[--open-file-limit OPEN_FILE_LIMIT] [--pac-file PAC_FILE]
Devtools Protocol (#174) * Refine docs * Decouple relay from dashboard. Will be re-used by devtools protocol plugin. * Just have a single manager for all eventing * Ofcourse managers cant be shared across processes * Remove unused * Add DevtoolsProtocolPlugin * Emit REQUEST_COMPLETE core event * Emit only if --enable-events used * Add event emitter for response cycle * Fill up core events to devtools protocol expectations * Serve static content with Cache-Control header and gzip compression * Add PWA manifest.json and icons from sample PWA apps (replace later) * Catch any exception and be ssl agnostic * Add CSP headers and avoid inline scripts * Re-enable iframe and deobfuscation * Embed plugins within <section/> block * Make tab switching agnostic of block name * Add support for browser history on tab change * Default hash to #home * Switch to tab if hash is already set * Expand canvas to fill screen even without content * Remove inline css for embedded devtools * Make dashboard backend websocket API pluggable * doc
2019-11-15 03:00:07 +00:00
[--pac-file-url-path PAC_FILE_URL_PATH]
[--pid-file PID_FILE] [--plugins PLUGINS] [--port PORT]
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
[--server-recvbuf-size SERVER_RECVBUF_SIZE]
[--static-server-dir STATIC_SERVER_DIR] [--threadless]
[--timeout TIMEOUT] [--version]
TLS Interception Cert Generation (#362) * Use common.pki for interception certificate generation * Fix tests * Dont use certificate fields that we dont need, it leads to certificate generation error on Ubuntu * Prepare for v2.2.0 * npm audit fix
2020-06-09 06:37:00 +00:00
proxy.py v2.2.0
updated README.md
2013-08-31 09:19:22 +00:00
Per worker queue instead of shared one. 1. MultiCoreDispatcher now dispatches request in round robin fashion. 2. Added Python3 static type hinting add Google Cloud Run Button Preserve proxy.py name as otherwise it breaks module system. Just provide a custom service name to fix Google Cloud Run issue Custom service name for Google Cloud Run. Specify port in app.json as required by Google Cloud Run Go cannot unmarshall int :) Port desc Looks like PORT cannot be overridden, for now use port 8080 in Dockerfile Remove cloudrun integration. While proxy.py runs successfully within the container, gvisor limitations doesn't allow proxy.py to accept client connections. Specifically membarrier syscall isn't allowed. Update README.md
2019-08-23 00:09:35 +00:00
optional arguments:
updated README.md
2013-08-31 09:19:22 +00:00
-h, --help show this help message and exit
Update README to reflect basic auth support. Also refactor proxy exception handling code.
2018-12-13 17:01:56 +00:00
--backlog BACKLOG Default: 100. Maximum number of pending connections to
proxy server
--basic-auth BASIC_AUTH
Default: No authentication. Specify colon separated
user:password to enable basic authentication.
Add support for TLS/SSL interception and HTTPS server (#86) * Handle OSError thrown when curl --proxy-cacert flag is invalid * Add server-cert * Dockerfile dont need --ipv4 since now its auto-detected and deprecated * Add make server-cert info * Add support for HTTPS interception. 1) Start as proxy.py --ca-key-file ca-key.pem --ca-cert-file ca-cert.pem --ca-signing-key-file ca-signing-key.pem 2) Test using curl curl -v -L -x proxy.py:8899 --cacert ca-cert.pem https://google.com Now proxy.py is able to decrypt HTTPS traffic between curl and google.com. Generated certificates are stored under ~/.proxy.py directory. * Add comments for why interception do not work if client is communicating with proxy.py over https * Fix tests * Exit with message when trying to https intercept and also serve over https * lint * Disable OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_TLSv1, OP_NO_TLSv1_1 when wrapping upstream server socket * Ensure client.conn.shutdown is called if necessary. * Add mypy check for plugin_examples.py * Exception handling when client sends invalid requests * Verify plugin_examples * mypy compat * HttpRequestRejected
2019-09-25 02:02:57 +00:00
--ca-key-file CA_KEY_FILE
Default: None. CA key to use for signing dynamically
generated HTTPS certificates. If used, must also pass
--ca-cert-file and --ca-signing-key-file
--ca-cert-dir CA_CERT_DIR
Default: ~/.proxy.py. Directory to store dynamically
generated certificates. Also see --ca-key-file, --ca-
cert-file and --ca-signing-key-file
--ca-cert-file CA_CERT_FILE
Default: None. Signing certificate to use for signing
dynamically generated HTTPS certificates. If used,
must also pass --ca-key-file and --ca-signing-key-file
Add flag to specify custom system CA Path (#321) * Fixes #320 * Update README and add codecov.yml * Update codecov.yml
2020-03-25 08:00:19 +00:00
--ca-file CA_FILE Default: None. Provide path to custom CA file for peer
certificate validation. Specially useful on MacOS.
Add support for TLS/SSL interception and HTTPS server (#86) * Handle OSError thrown when curl --proxy-cacert flag is invalid * Add server-cert * Dockerfile dont need --ipv4 since now its auto-detected and deprecated * Add make server-cert info * Add support for HTTPS interception. 1) Start as proxy.py --ca-key-file ca-key.pem --ca-cert-file ca-cert.pem --ca-signing-key-file ca-signing-key.pem 2) Test using curl curl -v -L -x proxy.py:8899 --cacert ca-cert.pem https://google.com Now proxy.py is able to decrypt HTTPS traffic between curl and google.com. Generated certificates are stored under ~/.proxy.py directory. * Add comments for why interception do not work if client is communicating with proxy.py over https * Fix tests * Exit with message when trying to https intercept and also serve over https * lint * Disable OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_TLSv1, OP_NO_TLSv1_1 when wrapping upstream server socket * Ensure client.conn.shutdown is called if necessary. * Add mypy check for plugin_examples.py * Exception handling when client sends invalid requests * Verify plugin_examples * mypy compat * HttpRequestRejected
2019-09-25 02:02:57 +00:00
--ca-signing-key-file CA_SIGNING_KEY_FILE
Default: None. CA signing key to use for dynamic
generation of HTTPS certificates. If used, must also
pass --ca-key-file and --ca-cert-file
--cert-file CERT_FILE
Default: None. Server certificate to enable end-to-end
TLS encryption with clients. If used, must also pass
--key-file.
Provision to specify custom buffer sizes. Now --server-recvbuf-size and --client-recvbuf-size flags can be used for customizing maximum recv buffer sizes. Bumping these values will result in higher RAM consumption during download / upload operations but will provide better download / upload speeds / throughputs and lower CPU utilization. Fixes #22
2018-12-13 21:08:08 +00:00
--client-recvbuf-size CLIENT_RECVBUF_SIZE
Add option (--log-file) to redirect logs to a file
2019-08-26 16:37:25 +00:00
Default: 1 MB. Maximum amount of data received from
Provision to specify custom buffer sizes. Now --server-recvbuf-size and --client-recvbuf-size flags can be used for customizing maximum recv buffer sizes. Bumping these values will result in higher RAM consumption during download / upload operations but will provide better download / upload speeds / throughputs and lower CPU utilization. Fixes #22
2018-12-13 21:08:08 +00:00
the client in a single recv() operation. Bump this
value for faster uploads at the expense of increased
RAM.
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
--devtools-ws-path DEVTOOLS_WS_PATH
Default: /devtools. Only applicable if --enable-
devtools is used.
Add --disable-headers option. Fixes #31
2019-09-02 22:58:37 +00:00
--disable-headers DISABLE_HEADERS
Default: None. Comma separated list of headers to
Address mypy errors. Unfortunately collections.namedtuple isn't supported by mypy and most likely never will be. Idea will be to use Typing.NamedTuple but this require some code refactoring.
2019-09-14 04:18:05 +00:00
remove before dispatching client request to upstream
Add --disable-headers option. Fixes #31
2019-09-02 22:58:37 +00:00
server.
--disable-http-proxy Default: False. Whether to disable
proxy.HttpProxyPlugin.
Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image
2019-11-15 22:47:50 +00:00
--enable-dashboard Default: False. Enables proxy.py dashboard.
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
--enable-devtools Default: False. Enables integration with Chrome
Devtools Protocol (#174) * Refine docs * Decouple relay from dashboard. Will be re-used by devtools protocol plugin. * Just have a single manager for all eventing * Ofcourse managers cant be shared across processes * Remove unused * Add DevtoolsProtocolPlugin * Emit REQUEST_COMPLETE core event * Emit only if --enable-events used * Add event emitter for response cycle * Fill up core events to devtools protocol expectations * Serve static content with Cache-Control header and gzip compression * Add PWA manifest.json and icons from sample PWA apps (replace later) * Catch any exception and be ssl agnostic * Add CSP headers and avoid inline scripts * Re-enable iframe and deobfuscation * Embed plugins within <section/> block * Make tab switching agnostic of block name * Add support for browser history on tab change * Default hash to #home * Switch to tab if hash is already set * Expand canvas to fill screen even without content * Remove inline css for embedded devtools * Make dashboard backend websocket API pluggable * doc
2019-11-15 03:00:07 +00:00
Devtool Frontend. Also see --devtools-ws-path.
Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path
2019-10-28 23:14:34 +00:00
--enable-events Default: False. Enables core to dispatch lifecycle
events. Plugins can be used to subscribe for core
events.
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
--enable-static-server
Default: False. Enable inbuilt static file server.
Optionally, also use --static-server-dir to serve
static content from custom directory. By default,
Remove public folder references (#212)
2019-12-02 05:30:29 +00:00
static file server serves out of installed proxy.py
python module folder.
Add --version flag also used by Makefile for releases.
2019-08-24 17:23:30 +00:00
--enable-web-server Default: False. Whether to enable
Per worker queue instead of shared one. 1. MultiCoreDispatcher now dispatches request in round robin fashion. 2. Added Python3 static type hinting add Google Cloud Run Button Preserve proxy.py name as otherwise it breaks module system. Just provide a custom service name to fix Google Cloud Run issue Custom service name for Google Cloud Run. Specify port in app.json as required by Google Cloud Run Go cannot unmarshall int :) Port desc Looks like PORT cannot be overridden, for now use port 8080 in Dockerfile Remove cloudrun integration. While proxy.py runs successfully within the container, gvisor limitations doesn't allow proxy.py to accept client connections. Specifically membarrier syscall isn't allowed. Update README.md
2019-08-23 00:09:35 +00:00
proxy.HttpWebServerPlugin.
Add support for TLS/SSL interception and HTTPS server (#86) * Handle OSError thrown when curl --proxy-cacert flag is invalid * Add server-cert * Dockerfile dont need --ipv4 since now its auto-detected and deprecated * Add make server-cert info * Add support for HTTPS interception. 1) Start as proxy.py --ca-key-file ca-key.pem --ca-cert-file ca-cert.pem --ca-signing-key-file ca-signing-key.pem 2) Test using curl curl -v -L -x proxy.py:8899 --cacert ca-cert.pem https://google.com Now proxy.py is able to decrypt HTTPS traffic between curl and google.com. Generated certificates are stored under ~/.proxy.py directory. * Add comments for why interception do not work if client is communicating with proxy.py over https * Fix tests * Exit with message when trying to https intercept and also serve over https * lint * Disable OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_TLSv1, OP_NO_TLSv1_1 when wrapping upstream server socket * Ensure client.conn.shutdown is called if necessary. * Add mypy check for plugin_examples.py * Exception handling when client sends invalid requests * Verify plugin_examples * mypy compat * HttpRequestRejected
2019-09-25 02:02:57 +00:00
--hostname HOSTNAME Default: ::1. Server IP address.
--key-file KEY_FILE Default: None. Server key file to enable end-to-end
TLS encryption with clients. If used, must also pass
--cert-file.
updated README.md
2013-08-31 09:19:22 +00:00
--log-level LOG_LEVEL
Update README.md
2019-07-20 19:17:01 +00:00
Valid options: DEBUG, INFO (default), WARNING, ERROR,
Address mypy errors. Unfortunately collections.namedtuple isn't supported by mypy and most likely never will be. Idea will be to use Typing.NamedTuple but this require some code refactoring.
2019-09-14 04:18:05 +00:00
CRITICAL. Both upper and lowercase values are allowed.
You may also simply use the leading character e.g.
--log-level d
Add option (--log-file) to redirect logs to a file
2019-08-26 16:37:25 +00:00
--log-file LOG_FILE Default: sys.stdout. Log file destination.
Core changes: 1. Rename HttpServer to MultiCoreRequestDispatcher, there wasn't anything Http about that class, all Http implementation happens in HttpProxy class 2. PY 2.7 DEPRECATION ERRORS 3. Add --log-format cli option 4. Remove _ from HttpProxy private methods, refactor it's core for clarify (remove redundant close() and other details)
2019-08-20 05:59:11 +00:00
--log-format LOG_FORMAT
Log format for Python logger.
Add --version flag also used by Makefile for releases.
2019-08-24 17:23:30 +00:00
--num-workers NUM_WORKERS
Defaults to number of CPU cores.
Pre-spawn worker processes to utilize all cores.
2019-07-20 19:47:16 +00:00
--open-file-limit OPEN_FILE_LIMIT
Default: 1024. Maximum number of files (TCP
connections) that proxy.py can open concurrently.
Update README.md
2019-07-20 19:17:01 +00:00
--pac-file PAC_FILE A file (Proxy Auto Configuration) or string to serve
Address mypy errors. Unfortunately collections.namedtuple isn't supported by mypy and most likely never will be. Idea will be to use Typing.NamedTuple but this require some code refactoring.
2019-09-14 04:18:05 +00:00
when the server receives a direct file request. Using
this option enables proxy.HttpWebServerPlugin.
Per worker queue instead of shared one. 1. MultiCoreDispatcher now dispatches request in round robin fashion. 2. Added Python3 static type hinting add Google Cloud Run Button Preserve proxy.py name as otherwise it breaks module system. Just provide a custom service name to fix Google Cloud Run issue Custom service name for Google Cloud Run. Specify port in app.json as required by Google Cloud Run Go cannot unmarshall int :) Port desc Looks like PORT cannot be overridden, for now use port 8080 in Dockerfile Remove cloudrun integration. While proxy.py runs successfully within the container, gvisor limitations doesn't allow proxy.py to accept client connections. Specifically membarrier syscall isn't allowed. Update README.md
2019-08-23 00:09:35 +00:00
--pac-file-url-path PAC_FILE_URL_PATH
Address mypy errors. Unfortunately collections.namedtuple isn't supported by mypy and most likely never will be. Idea will be to use Typing.NamedTuple but this require some code refactoring.
2019-09-14 04:18:05 +00:00
Default: /. Web server path to serve the PAC file.
Add --pid-file option to dump parent process ID to a file. Rename to --disable-http-proxy. Is more appropriate since proxy is always enabled. unittest.skipIf right guards for travis
2019-08-26 16:52:43 +00:00
--pid-file PID_FILE Default: None. Save parent process ID to a file.
Per worker queue instead of shared one. 1. MultiCoreDispatcher now dispatches request in round robin fashion. 2. Added Python3 static type hinting add Google Cloud Run Button Preserve proxy.py name as otherwise it breaks module system. Just provide a custom service name to fix Google Cloud Run issue Custom service name for Google Cloud Run. Specify port in app.json as required by Google Cloud Run Go cannot unmarshall int :) Port desc Looks like PORT cannot be overridden, for now use port 8080 in Dockerfile Remove cloudrun integration. While proxy.py runs successfully within the container, gvisor limitations doesn't allow proxy.py to accept client connections. Specifically membarrier syscall isn't allowed. Update README.md
2019-08-23 00:09:35 +00:00
--plugins PLUGINS Comma separated plugins
--port PORT Default: 8899. Server port.
Pre-spawn worker processes to utilize all cores.
2019-07-20 19:47:16 +00:00
--server-recvbuf-size SERVER_RECVBUF_SIZE
Add option (--log-file) to redirect logs to a file
2019-08-26 16:37:25 +00:00
Default: 1 MB. Maximum amount of data received from
Pre-spawn worker processes to utilize all cores.
2019-07-20 19:47:16 +00:00
the server in a single recv() operation. Bump this
value for faster downloads at the expense of increased
RAM.
Chrome Devtool Integration first steps (#109) * Initialize skeleton electron app * Attempt to open devtools * Electron free * Initialize public/devtools * Add basic support for static file serving and chrome devtools. 1. No cache header management for static file serving yet. 2. No chunked encoded responses for static files yet. 3. Chrome Devtool initialization. * Fix static serving with query params * profile using py-spy * Complete websocket client loop * lint check * Add support for building websocket frames * Remove redundant CDT params * Lint check * Refactor web server base plugin name * Devtools integrated, need more polish * Add START_TIME global var * lint fix * Remove outdated chrome rdp * Add FAQs * Add FAQs * socket_connection decorator + context manager * Defer SSL handshake and plugin initialize until protocol handler thread has started. This is a follow up to this PR https://github.com/abhinavsingh/proxy.py/pull/111 * Add tests for new_socket_connection and its friend socket_connection * Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92 * Lint fixes * uff ye str and bytes * Remove explicit flushes outside of write ready descriptor handlers * add links to import proxy * Only try websocket upgrade if a route is registered * Add plugin_examples.WebServerPlugin and use precision logging for levelname * Remove redundant comments * Add --devtools-ws-path flag * Add on_websocket_open and on_websocket_close callbacks * Add empty stubs for incomplete CDT responses * Ensure client is ready before final flush * Shutdown on write side of socket, may be client is still reading * Since client.closed can be set, explicitly call client.connection.closed * Add ModifyPostDataPlugin example. Was first asked and referenced here https://github.com/abhinavsingh/proxy.py/issues/115 * Start adding TestHttpProxyPlugin * Fixes #116
2019-10-10 05:36:47 +00:00
--static-server-dir STATIC_SERVER_DIR
Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more
2019-10-16 06:56:39 +00:00
Default: "public" folder in directory where proxy.py
is placed. This option is only applicable when static
server is also enabled. See --enable-static-server.
--threadless Default: False. When disabled a new thread is spawned
to handle each client connection.
Add --timeout flag with default value of 10 second. (#129) * Add --timeout flag with default value of 5. This value was previously hardcoded to 30 * --timeout=10 by default * Dispatch 408 timeout when connection is dropped due to inactivity * Add httpStatusCodes named tuple * Update plugin client connection reference after TLS connection upgrade
2019-10-13 04:02:17 +00:00
--timeout TIMEOUT Default: 10. Number of seconds after which an inactive
connection must be dropped. Inactivity is defined by
no data sent or received by the client.
Address mypy errors. Unfortunately collections.namedtuple isn't supported by mypy and most likely never will be. Idea will be to use Typing.NamedTuple but this require some code refactoring.
2019-09-14 04:18:05 +00:00
--version, -v Prints proxy.py version.
updated README.md
2013-08-31 09:19:22 +00:00
Update README.md
2019-07-20 19:17:01 +00:00
Proxy.py not working? Report at:
updated README.md
2013-08-31 09:19:22 +00:00
https://github.com/abhinavsingh/proxy.py/issues/new
```
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
Changelog
=========
Consistent dashboard look and feel across plugins (#169) * Explicitly link version changelog in TOC * Separate out app header body builder * Ensure unsubscribe when disabling inspection. Fixes #164 * Avoid creation of new manager per dashboard instance. * Add UI header for all plugins (tabs) * Ensure app body for all plugin skeleton * Move app-header and app-body within core for consistent dashboard look and feel * Consistent UI header body for plugins * autopep8
2019-11-12 23:29:37 +00:00
## v2.x
- No longer ~~a single file module~~.
- Added support for threadless execution.
- Added dashboard app.
Testing support improvements (#185) * Introduce proxy.Proxy context manager. This is similar to already existing context manager `start` but `proxy.Proxy` is a class with __enter__ and __exit__ methods. This allows usage of `proxy.Proxy` both as context manager and for manually setup and teardown of `proxy.py` during test setUpClass and teardownClass methods. * Gracefully shutdown threadless processes * Update tests and add a VCR method. See #184 * Refactor routes * Add Proxy to __all__ * Move TestCase under proxy.testing and test_embed.py under tests.embed module to avoid conflict with http module due to a http directory under proxy folder * Add a base cache plugin class which can be customized for custom cache behaviors * See #184. Add VCRPlugin which can be enabled within tests using a context manager, e.g. with self.vcr(): ... * Make cache plugin pluggable + make cache storage pluggable * Make dashboard npm module agnostic of top level directory * Symlink dashboard public folder * Dump devtools within dashboard public folder * Remove unused 3rd party js
2019-11-22 05:16:01 +00:00
- Added support for unit testing.
Consistent dashboard look and feel across plugins (#169) * Explicitly link version changelog in TOC * Separate out app header body builder * Ensure unsubscribe when disabling inspection. Fixes #164 * Avoid creation of new manager per dashboard instance. * Add UI header for all plugins (tabs) * Ensure app body for all plugin skeleton * Move app-header and app-body within core for consistent dashboard look and feel * Consistent UI header body for plugins * autopep8
2019-11-12 23:29:37 +00:00
## v1.x
- `Python3` only.
- Deprecated support for ~~Python 2.x~~.
- Added support multi core accept.
- Added plugin support.
## v0.x
- Single file.
- Single threaded server.
For detailed changelog refer to release PRs or commit history.