Rooba
/
proxy.py
mirror of https://github.com/abhinavsingh/proxy.py.git
1
1
Fork 0
Code Issues Packages Projects Releases Wiki Activity
proxy.py/benchmark/__init__.py

11 lines
349 B
Python
Raw Normal View History

v2.x (#173) * Always update latest tag for docker releases * Update issue templates (#123) * Invoke HttpWebServerBasePlugin.handle_request for each request in HTTP/1.1 pipeline (#125) * Add tests to verify certificate generation * Separate out tests for ProtocolHandler and WebServerPlugin * Keep-alive connections for web server. TODO: Only keep-alivei if HTTP/1.1 * Add request.path to avoid build_url repeatedly whose name is also slightly misleading * Fix example usage of request.path * Pipeline only for HTTP/1.1 * Lint fix * Teardown HTTP/1.1 keep-alive request when Connection: close header is sent * Add instructions on how to build docker image locally * Move access_log to separate function for pretty logging * Reduce docker image size * Ensure teardown is always accompanied with Connection: close header Fix tests * Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128) * Add tests for is_http_1_1_keep_alive * Add ModifyPostDataPlugin in README * Fixes #126 * Refactor HttpProxyBasePlugin API * before_upstream_connection too can drop request by returning None * Remove HTTP Server startup during tests, no longer used * Removed unused imports * Simplify load_plugins * Add --timeout flag with default value of 10 second. (#129) * Add --timeout flag with default value of 5. This value was previously hardcoded to 30 * --timeout=10 by default * Dispatch 408 timeout when connection is dropped due to inactivity * Add httpStatusCodes named tuple * Update plugin client connection reference after TLS connection upgrade * Test plugin examples (#130) * Add tests for plugin_examples.* to ensure we never break functionality * Add tests for plugin_examples.* * Test man in the middle * Lint fixes * Checkin * Add tests for plugin examples with TLS encryption enabled * Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more * Remove pip upgrade for windows which seems to be failing on travis (#136) * Remove pip upgrade for windows which seems to be failing on travis * Remove windows testing on Travis, pip install is failing * Add pipeline response parsing tests (#137) * Add pipeline response parsing tests * build_http_response now only adds content-length if transfer-encoding is not provided. Also return pending raw chunks from ChunkParser so that we can parse pipelined chunk responses. * os.close only for threadless (#138) * os.close only for Threadless to avoid fd leaks * Remove os.close mock which is only called for threadless * Update pytest from 5.2.1 to 5.2.2 (#142) * Update setuptools from 41.4.0 to 41.5.0 (#145) * Update typing-extensions from 3.7.4 to 3.7.4.1 (#147) * Update flake8 from 3.7.8 to 3.7.9 (#148) * Update setuptools from 41.5.0 to 41.5.1 (#149) * Update py-spy from 0.2.2 to 0.3.0 (#144) * Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls * Update mypy==0.740 (#151) * Update README.md (#152) * Update flags * Update debugging instructions and run instructions for develops * Update references to plugins directory * For readability add sections for run from command line using pip * Move internal doc under developer section * Add option to pass fully-qualified plugin path * Update setuptools from 41.5.1 to 41.6.0 (#153) * Test refactor + Docker image CI (#154) * Move tests into individual modules too * Ensure one test class per file * Fix docker image after refactoring * Add github actions workflow for building docker image * Fix image name * Setup python required for extracting proxy version * Version will also require deps * Separate packages for Dashboard (#157) * Refactor Makefile and add dashboard setup.py * Package dashboard as proxy.py-dashboard pip package * Give dashboard releases its own version * Fix lib-package reference * Add non-blocking embedded mode feature (#159) * Fixes #158 * mypy fixes * Instructions for non-blocking embed mode * Toggle running flag before shutdown * Add private / public key generation utils which comply with new requirements on Mac OS 10.15 (#160) * Add utilities to generate private key and public keys with alternate cnames * Add separate package proxy.py-plugins, fixes #156 * Generate certificates to comply with Mac requirements. * Add utility for CSR generation and signing * Fixes #161 * Add initial pki tests * Give structure to dashboard app (#163) * Separate out files for different responsibilities. 1. Add src/plugins directory. This directory holds one typescript file per plugin. Each plugin is optionally can be displayed as a tab on the UI. 2. Move WebsocketApi to ws.ts. This file contains all websocket APIs provided by dashboard.py backend. * Make dashboard pluggable * Move devtools under core too * Register tabs dynamically * Typescript fixes for abstract interfaces * Initialize plugin app body skeleton * Call activated / deactivated on tab change * Move plugin name within plugin classes and initialize plugin within proxy dashboard constructor * templatize api development plugin * eslint fixes * use globs * Remove useless constructors * Move traffic_control outside of core plugin, it maps to several plugin examples like redirectToUpstreamHost, filterByUpstreamHost plugins (#165) * Introduce sendMessage websocket api which allows for callbacks (#166) * Introduce sendMessage websocket api which allows for callbacks, deprecate lastPingId in favor of callbacks * Let InspectTrafficPlugin handle all pushed inspection events * Add proxy.main.TestCase for unit testing Python application with proxy.py (#167) * Add demonstration of how to use proxy.py within Python application unittests * mypy fixes * test_with_proxy example * Add docs for proxy.main.TestCase. Also wait for proxy.py server to come up before running the tests. * Consistent dashboard look and feel across plugins (#169) * Explicitly link version changelog in TOC * Separate out app header body builder * Ensure unsubscribe when disabling inspection. Fixes #164 * Avoid creation of new manager per dashboard instance. * Add UI header for all plugins (tabs) * Ensure app body for all plugin skeleton * Move app-header and app-body within core for consistent dashboard look and feel * Consistent UI header body for plugins * autopep8 * Dashboard Inspect traffic tab + devtools (#170) * Explicitly link version changelog in TOC * Separate out app header body builder * Ensure unsubscribe when disabling inspection. Fixes #164 * Avoid creation of new manager per dashboard instance. * Add UI header for all plugins (tabs) * Ensure app body for all plugin skeleton * Move app-header and app-body within core for consistent dashboard look and feel * Consistent UI header body for plugins * autopep8 * make devtools * convert to es6 * Add inspect_traffic plugin devtools app * trigger re-build, github UI is stuck * Dynamically load devtools within inspect traffic view * Just copy devtools into public/dashboard folder * Works but not how we wanted, devtools takes over entire body and doesnt contain itself within a div * Load devtools within iframe * Load devtools within iframe (#171) * Allow to pass flags as kwargs too in embed mode (#172) * Dynamically load devtools instead of on page load * Add support for passing flags as kwargs to main / start methods. * Fix tests for refactored code * Allow proxy.main, proxy.start, proxy.TestCase. Also update README.md to reflect the same. * Use Any for **opts * Move main as __init__ to avoid name conflicts * Fix tests * Update setup.py entry_point * Explicitly install requirements before setup.py * Explicitly mention packages of interest * ipv6 fails on ubuntu, use ipv4 * Make typing-extensions optional * Instead of putting it all under __init__.py, move main.py to proxy.py * Simply make setup.py module free * autopep8 * Devtools Protocol (#174) * Refine docs * Decouple relay from dashboard. Will be re-used by devtools protocol plugin. * Just have a single manager for all eventing * Ofcourse managers cant be shared across processes * Remove unused * Add DevtoolsProtocolPlugin * Emit REQUEST_COMPLETE core event * Emit only if --enable-events used * Add event emitter for response cycle * Fill up core events to devtools protocol expectations * Serve static content with Cache-Control header and gzip compression * Add PWA manifest.json and icons from sample PWA apps (replace later) * Catch any exception and be ssl agnostic * Add CSP headers and avoid inline scripts * Re-enable iframe and deobfuscation * Embed plugins within <section/> block * Make tab switching agnostic of block name * Add support for browser history on tab change * Default hash to #home * Switch to tab if hash is already set * Expand canvas to fill screen even without content * Remove inline css for embedded devtools * Make dashboard backend websocket API pluggable * doc * Move dashboard backend within proxy module, now ships via same pip package (#177) * Allow resources to load from http and ws when running w/o https * Move dashboard backend (dashboard.py) within proxy module. Now shipped with pip install proxy.py * Update ref to dashboard backend in github workflows * Add git-pre-commit hook file. Enable it by symlinking as .git/hooks/pre-commit * Also enable static server for dashboard serving * Move plugin_examples/ as proxy.plugin and update readme (#179) * Update dev guide * Move plugin_examples/ as proxy.plugin * Update proxy.plugin ref path in readme * Remove unnecessary port flag * Remove plugin_examples from github workflows * dashboard folder is a npm package not python package anymore * Plugins can now be tried using Docker image * Move benchmark module within proxy (#181) * Move benchmark within proxy module * chmod 0644 for benchmark.py which was executable till now * Turn utilities into its own section * Update pytest from 5.2.3 to 5.2.4 (#180) * Doc & Banner update to match GitHub (#182) * Update doc and banner * Update banner to match GitHub * Update older banners too * Add update_desc to .gitignore * Update banner for dashboard to match github * also update html, js, css * Update twine from 2.0.0 to 3.0.0 (#183) * Update pytest from 5.2.4 to 5.3.0 (#186) * Testing support improvements (#185) * Introduce proxy.Proxy context manager. This is similar to already existing context manager `start` but `proxy.Proxy` is a class with __enter__ and __exit__ methods. This allows usage of `proxy.Proxy` both as context manager and for manually setup and teardown of `proxy.py` during test setUpClass and teardownClass methods. * Gracefully shutdown threadless processes * Update tests and add a VCR method. See #184 * Refactor routes * Add Proxy to __all__ * Move TestCase under proxy.testing and test_embed.py under tests.embed module to avoid conflict with http module due to a http directory under proxy folder * Add a base cache plugin class which can be customized for custom cache behaviors * See #184. Add VCRPlugin which can be enabled within tests using a context manager, e.g. with self.vcr(): ... * Make cache plugin pluggable + make cache storage pluggable * Make dashboard npm module agnostic of top level directory * Symlink dashboard public folder * Dump devtools within dashboard public folder * Remove unused 3rd party js * Initialize Menubar (#188) * Initialize MacOS Menubar application * Dashboard plugin at-least needs a shutdown hook to teardown any thread/processes started by dashboard backend plugin * Add menu bar icon * Add respective test directories * Sync test banners * Move plugin tests under its own package * Enable daemon for threads, other this wont shutdown cleanly * Update twine from 3.0.0 to 3.1.0 (#190) * Update setuptools from 41.6.0 to 42.0.0 (#191) * Memory optimizations (#189) * Avoid persisting raw content in memory within parser, simply parse and throw-away. Addresses #187 * Clarity in test comments * Update setuptools from 42.0.0 to 42.0.1 (#193) * Make connection queue / recv work with memoryview to avoid copies (#192) * connection.recv now returns a memoryview * Make connection.queue also memoryview compliant * autopep8 * wrap in memoryview as necessary * Add default timeout for socket_connection and test_embed urllib * Fix tests * Skip TestProxyPyEmbedded for now, verifying GitHub actions * Add timeout for wait_for_server and skip only if GITHUB_ACTIONS env variable is set * Verify if GitHub Action fails due to wait_for_server spinning forever * Add test for wait_for_server timeout error exception * GitHub action hangs irrespective of wait_for_server timeout, disable TestEmbed for GitHub actions * Cleanup (#194) * Add basic README description for dashboard * Use spaces for all except makefile * enable tests for py 3.5 * Python 3.5 support label * Avoid clash of names * Add py3.8 support and bump node to 12.x (#195) * Add py3.8 support and bump node to 12.x * Add 10.x, 11.x, 12.x matrix for dashboard testing * Add Python 3.8 support label * Single tested with label * autopep8 (#196) * autopep8 * Update TestCase section * Update pytest from 5.3.0 to 5.3.1 (#197) * Update twine from 3.1.0 to 3.1.1 (#200) * Add reverse proxy example (#201) * Add reverse proxy example * Add separate sections for http proxy and web server plugins * Add doc * Add proxy over ssh tunnel functionality (#198) * update mypy to 0.750 (#204) * Test Core Eventing (#205) * Add core event tests * Update .gitignore with coverage * Add shortlink gif * Add event dispatcher test * Test event subscriber * Test Dashboard backend (#206) * Update shortlink gif name * Conditionally run workflows as necessary * Use pytest * It works but github workflow is not reporting any status :( * Separate out badges * Add python_requires to setup.py * Update setuptools from 42.0.1 to 42.0.2 (#207) * Add tox.ini (#208) * Homebrew formula (#209) * Add homebrew formula * Build PyPi package and Homebrew installation verification * Check develop * bdist_wheel reported as error: invalid command "bdist_wheel" * Move under stable/develop folders to keep Proxy class name same * uff * develop installs proxy not proxy.py binary * Prepend site-packages * Install typing-extensions explicitly with brew * Use find_packages * Most likely failing due to lack of find_packages in current develop branch * Fix windows setup.py build * test_static_web_server_serves seems flaky on Ubuntu python 3.8 * Add instructions to install using homebrew * Disable test_static_web_server_serves on GitHub actions, seems flaky * Packaging (#210) * Move docker installation steps above * Try brewing with virtualenv * depends on python * Update homebrew formula for stable release * Just test brewing on latest python * Add support for regex based routing. Fixes #203 (#211) * Remove public folder references (#212) * Refactor (#213) * Add DEFAULT_HTTP_PORT constant * Use DEFAULT_HTTP_PORT in tests * Refactor into exception module * Refactor into inspector module * Refactor into server module * Refactor into proxy module * Build docker of Python 3.8 (#214) * Move homebrew under helper (#215) * Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET on no internet (#216) * Catch TimeoutError and OSError (host unreachable) * Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET * Enable mccabe (#217) * No need of per day or week stats (#218) * Make HTTP handler constructor free of socket file number (#219) * Refactor into acceptor module * Add tunnel doc * Make fileno free * Autopep8 * Response parser now reaches COMPLETE even when no body is expected (#220) * Stash current changes * Refactor into connection module * Response parser state complete when no body expect * Raise NotImplementedError if invalid state reached within parser * Update tox from 3.14.1 to 3.14.2 (#221) * Update paramiko from 2.6.0 to 2.7.0 (#225) * Update paramiko from 2.7.0 to 2.7.1 (#227) * Proxy Pool Plugin (#228) * Add proxy pool example. See #226 * Add ProxyPoolPlugin to doc * Update pytest from 5.3.1 to 5.3.2 (#229) * Update coverage from 4.5.4 to 5.0 (#230) * Update mypy from 0.750 to 0.760 (#232) * Update mypy from 0.760 to 0.761 (#235) * Move manager initialization outside of top level scope. Fixes #233 (#236) * Share lock to acceptors via pool (#238) * Move manager initialization outside of top level scope. Fixes #233 * Share lock to acceptor via pool * Optionally initialize manager in main thread and use the same for EventQueue initialization (#239) * Highlight language syntax (#240) * Highlight lang syntax * zsh prompt * Update coverage from 5.0 to 5.0.1 (#241) * Integration testing (#243) * Add tests for public/private/csr generation * Add integration testing skeleton for mac and ubuntu * Merge integration within lib test to avoid too many workflows * Disable integration testing on windows for now * Use sudo to start integration test script as lsof fails on MacOS. lsof: WARNING: can't stat() vmhgfs file system * Add basic integration testing for now to assert proxy works as expected when started out of develop branch * Add a call to inbuilt http server to verify it works * wait for server to accept requests Co-authored-by: pyup.io bot <github-bot@pyup.io>
2019-12-26 00:47:15 +00:00
# -*- coding: utf-8 -*-
"""
proxy.py
~~~~~~~~
⚡⚡⚡ Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
Network monitoring, controls & Application development, testing, debugging.
:copyright: (c) 2013-present by Abhinav Singh and contributors.
:license: BSD, see LICENSE for more details.
"""