* Add py3.8 support and bump node to 12.x
* Add 10.x, 11.x, 12.x matrix for dashboard testing
* Add Python 3.8 support label
* Single tested with label
* Add basic README description for dashboard
* Use spaces for all except makefile
* enable tests for py 3.5
* Python 3.5 support label
* Avoid clash of names
* Introduce proxy.Proxy context manager.
This is similar to already existing context manager `start` but
`proxy.Proxy` is a class with __enter__ and __exit__ methods. This
allows usage of `proxy.Proxy` both as context manager and for manually
setup and teardown of `proxy.py` during test setUpClass and
teardownClass methods.
* Gracefully shutdown threadless processes
* Update tests and add a VCR method. See #184
* Refactor routes
* Add Proxy to __all__
* Move TestCase under proxy.testing and test_embed.py under tests.embed module to avoid conflict with http module due to a http directory under proxy folder
* Add a base cache plugin class which can be customized for custom cache behaviors
* See #184. Add VCRPlugin which can be enabled within tests using a context manager, e.g. with self.vcr(): ...
* Make cache plugin pluggable + make cache storage pluggable
* Make dashboard npm module agnostic of top level directory
* Symlink dashboard public folder
* Dump devtools within dashboard public folder
* Remove unused 3rd party js
* Update doc and banner
* Update banner to match GitHub
* Update older banners too
* Add update_desc to .gitignore
* Update banner for dashboard to match github
* also update html, js, css
* Update dev guide
* Move plugin_examples/ as proxy.plugin
* Update proxy.plugin ref path in readme
* Remove unnecessary port flag
* Remove plugin_examples from github workflows
* dashboard folder is a npm package not python package anymore
* Plugins can now be tried using Docker image
* Refine docs
* Decouple relay from dashboard.
Will be re-used by devtools protocol plugin.
* Just have a single manager for all eventing
* Ofcourse managers cant be shared across processes
* Remove unused
* Add DevtoolsProtocolPlugin
* Emit REQUEST_COMPLETE core event
* Emit only if --enable-events used
* Add event emitter for response cycle
* Fill up core events to devtools protocol expectations
* Serve static content with Cache-Control header and gzip compression
* Add PWA manifest.json and icons from sample PWA apps (replace later)
* Catch any exception and be ssl agnostic
* Add CSP headers and avoid inline scripts
* Re-enable iframe and deobfuscation
* Embed plugins within <section/> block
* Make tab switching agnostic of block name
* Add support for browser history on tab change
* Default hash to #home
* Switch to tab if hash is already set
* Expand canvas to fill screen even without content
* Remove inline css for embedded devtools
* Make dashboard backend websocket API pluggable
* doc
* Dynamically load devtools instead of on page load
* Add support for passing flags as kwargs to main / start methods.
* Fix tests for refactored code
* Allow proxy.main, proxy.start, proxy.TestCase.
Also update README.md to reflect the same.
* Use Any for **opts
* Move main as __init__ to avoid name conflicts
* Fix tests
* Update setup.py entry_point
* Explicitly install requirements before setup.py
* Explicitly mention packages of interest
* ipv6 fails on ubuntu, use ipv4
* Make typing-extensions optional
* Instead of putting it all under __init__.py, move main.py to proxy.py
* Simply make setup.py module free
* autopep8
* Explicitly link version changelog in TOC
* Separate out app header body builder
* Ensure unsubscribe when disabling inspection. Fixes#164
* Avoid creation of new manager per dashboard instance.
* Add UI header for all plugins (tabs)
* Ensure app body for all plugin skeleton
* Move app-header and app-body within core for consistent dashboard look and feel
* Consistent UI header body for plugins
* autopep8
* Add demonstration of how to use proxy.py within Python application unittests
* mypy fixes
* test_with_proxy example
* Add docs for proxy.main.TestCase.
Also wait for proxy.py server to come up before running the tests.
* Add utilities to generate private key and public keys with alternate cnames
* Add separate package proxy.py-plugins, fixes#156
* Generate certificates to comply with Mac requirements.
* Add utility for CSR generation and signing
* Fixes#161
* Add initial pki tests
* Move tests into individual modules too
* Ensure one test class per file
* Fix docker image after refactoring
* Add github actions workflow for building docker image
* Fix image name
* Setup python required for extracting proxy version
* Version will also require deps
* Update flags
* Update debugging instructions and run instructions for develops
* Update references to plugins directory
* For readability add sections for run from command line using pip
* Move internal doc under developer section
* Add option to pass fully-qualified plugin path
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
* Workers need not register/unregister sock for every loop
* No need of explicit socket.settimeout(0) which is same as socket.setblocking(False)
* Remove settimeout assertion
* Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown
* Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes
* Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests
* Remove unused
* Wrap selector register/unregister within a context manager
* Refactor in preparation of threadless request handling
* MyPy generator fix
* Add --threadless flag
* Internally call them acceptors
* Internally use acceptors
* Add Threadless class.
Also no need to pass family over pipe to acceptors.
* Make threadless work for a single client :)
* Threadless is soon be our default
* Close client queue
* Use context manager for register/unregister
* Fix Acceptor tests broken after refactoring
* Use asyncio tasks to invoke ProtocolHandle.handle_events
This gives all client threads a chance to respond without
waiting for other handlers to return.
* Explicitly initialize event loop per Threadless process
* Mypy fixes
* Add ThreadlessWork abstract class implemented by ProtocolHandler
* Add benchmark.py
Avoid TIME_WAIT by properly shutting down the connection.
* Add benchmark.py as part of testing workflow
* When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state
* MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence
* Unwrap if wrapped before shutdown
* Unwrap if wrapped before shutdown
* socket.SHUT_RDWR will cause leaks
* MyPy
* Add instructions for monitor.sh
* Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized
* Add Fast & Scalable section
* Update internal classes section
* Dont print out local dir path in help text :)
* Refactor
* Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang
* Add chrome_with_proxy.sh helper script
* Handle OSError during client.flush which can happen due to invalid protocol type for socket error
* Remove redundant e
* Add classmethods to quickly construct a parser object
* Don't raise from TcpConnection abstract class.
This allows both client/socket side of communication to handle
exceptions as necessary. We might refactor this again later to remove
redundant code :)
* Disable response parsing when TLS interception is enabled. See issue #127
* remove unused imports
* Within webserver parse pipelined requests only if we have a route
* Add ShortLinkPlugin plugin
* Add more shortlinks
* Add ShortLinkPlugin to README.md
* Add path forwarding too instead of leaving as excercise ;)
* Add shortlink to TOC
* Ensure no socket leaks
* Ensure no leaks
* Naming
* Default number of clients 1
* Avoid shortlinking localhost
* Stress more
* Add --timeout flag with default value of 5. This value was previously hardcoded to 30
* --timeout=10 by default
* Dispatch 408 timeout when connection is dropped due to inactivity
* Add httpStatusCodes named tuple
* Update plugin client connection reference after TLS connection upgrade
* Add tests for is_http_1_1_keep_alive
* Add ModifyPostDataPlugin in README
* Fixes#126
* Refactor HttpProxyBasePlugin API
* before_upstream_connection too can drop request by returning None
* Remove HTTP Server startup during tests, no longer used
* Removed unused imports
* Simplify load_plugins
* Add tests to verify certificate generation
* Separate out tests for ProtocolHandler and WebServerPlugin
* Keep-alive connections for web server.
TODO: Only keep-alivei if HTTP/1.1
* Add request.path to avoid build_url repeatedly whose name is also slightly misleading
* Fix example usage of request.path
* Pipeline only for HTTP/1.1
* Lint fix
* Teardown HTTP/1.1 keep-alive request when Connection: close header is sent
* Add instructions on how to build docker image locally
* Move access_log to separate function for pretty logging
* Initialize skeleton electron app
* Attempt to open devtools
* Electron free
* Initialize public/devtools
* Add basic support for static file serving and chrome devtools.
1. No cache header management for static file serving yet.
2. No chunked encoded responses for static files yet.
3. Chrome Devtool initialization.
* Fix static serving with query params
* profile using py-spy
* Complete websocket client loop
* lint check
* Add support for building websocket frames
* Remove redundant CDT params
* Lint check
* Refactor web server base plugin name
* Devtools integrated, need more polish
* Add START_TIME global var
* lint fix
* Remove outdated chrome rdp
* Add FAQs
* Add FAQs
* socket_connection decorator + context manager
* Defer SSL handshake and plugin initialize until protocol handler thread
has started.
This is a follow up to this PR
https://github.com/abhinavsingh/proxy.py/pull/111
* Add tests for new_socket_connection and its friend socket_connection
* Address an issue which came back after being fixed in https://github.com/abhinavsingh/proxy.py/pull/92
* Lint fixes
* uff ye str and bytes
* Remove explicit flushes outside of write ready descriptor handlers
* add links to import proxy
* Only try websocket upgrade if a route is registered
* Add plugin_examples.WebServerPlugin and use precision logging for levelname
* Remove redundant comments
* Add --devtools-ws-path flag
* Add on_websocket_open and on_websocket_close callbacks
* Add empty stubs for incomplete CDT responses
* Ensure client is ready before final flush
* Shutdown on write side of socket, may be client is still reading
* Since client.closed can be set, explicitly call client.connection.closed
* Add ModifyPostDataPlugin example.
Was first asked and referenced here
https://github.com/abhinavsingh/proxy.py/issues/115
* Start adding TestHttpProxyPlugin
* Fixes#116
* Use selectors.DefaultSelector instead of select.select
* Unregister to avoid endless loop
* Cleanup event register/unregiter
* Cleanup event registration
* Add google-fluentd.conf. Use if running proxy.py on Google Cloud.
* Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?)
* Fix tests for selectors
* Only include proxy.py and tests.py in coverage report
* Only include proxy.py for coverage
* remove redundant integration test, will rewrite using mocks
* Proper unregister of events
* Change multi core accept model to avoid client TIME_WAIT. Fixes#97
* Catch BlockingIOError
* Remove redundant comments
* Simplify with AcceptorPool
* Pass family to acceptor processes
* Remove plugin.access_log for core plugins
* Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server
* Add support for websocket upgrade
* Websocket frame parser
* Enable websocket based routing
* Add WebsocketClient
* Websocket
* mypy fixes
* Sync GitHub workflow lint and makefile lint commands.
For now comment out tests which are broken :( New tests coming next.
* Start fixing tests for new code
* Fix formatting
* Fix main tests
* Add worker tests
* GitHub only ran windows tests, may be require unique names
* Use 3.6/3.7 dev versions for GitHub actions
* Add AcceptorPool test
* Add x64 and x86 matrix for actions tests
* Dont use dev versions since they dont exists for x86
* Ha no x86 support itself
* Add backer link
* Remove support badge for 3.5 as it doesnt support typing
* Update read me with changed architecture notes
* Update read me with changed architecture notes
* Add `import proxy` usage instructions.
* Add pydoc reference for developers
* Put pydoc as internal documentation
* Add ProposedRestApiPlugin example.
This plugin is an example to demonstrate how `proxy.py` can help you
test and debug client applications without actually having a need of
backend server.
Simply modify ProposedRestApiPlugin as per need to return json
responses for your API specs.
* Add README instructions
* Correct username
* Always deduce address family as we have a structure
* Convert TcpConnection into an ABC.
* Fix tests
* Rename MultiCoreRequestDispatcher to WorkerPool
* Consistent naming refactor
* Make WorkerPool independent of protocol config object
* Fix tests
* Proper fix for test_on_client_connection_called_on_teardown
* TLS interception
* better logging
* Add tests and mypy compliance for tests
* Add pragma: no cover for abstract class methods
* mypy --strict compliance for tests.py
* Refactor and test coverage
* lint fixes
* Add documentation
* Add TOC
* Add links to encryption and tls
* fix cert-file and key-file
* Add example for verification of https and tls interception
* Bigger fonts
* Redact
* Handle OSError thrown when curl --proxy-cacert flag is invalid
* Add server-cert
* Dockerfile dont need --ipv4 since now its auto-detected and deprecated
* Add make server-cert info
* Add support for HTTPS interception.
1) Start as
proxy.py --ca-key-file ca-key.pem --ca-cert-file ca-cert.pem --ca-signing-key-file ca-signing-key.pem
2) Test using curl
curl -v -L -x proxy.py:8899 --cacert ca-cert.pem https://google.com
Now proxy.py is able to decrypt HTTPS traffic between curl and
google.com. Generated certificates are stored under ~/.proxy.py
directory.
* Add comments for why interception do not work if client is communicating with proxy.py over https
* Fix tests
* Exit with message when trying to https intercept and also serve over https
* lint
* Disable OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_TLSv1, OP_NO_TLSv1_1 when wrapping upstream server socket
* Ensure client.conn.shutdown is called if necessary.
* Add mypy check for plugin_examples.py
* Exception handling when client sends invalid requests
* Verify plugin_examples
* mypy compat
* HttpRequestRejected
* Use pipes instead of queues.
Possible solution for #68
* Handle ConnectionRefusedError
* Close corresponding pipes on shutdown
* Abstract classes for plugins.
* Add github workflows
* Use pytest for github workflow
* Add Windows/Mac workflows
* Use os.matrix
* 3.8 and 3.9 probably dont exists yet
* Do not fail-fast, also python 3.5 seems to be throwing syntax error for typing
* autopep8
* Disable windows-latest workflow which seems to hang
Unfortunately collections.namedtuple isn't supported by mypy and most
likely never will be. Idea will be to use Typing.NamedTuple but this
require some code refactoring.
No use mirrors
Enforce coverage3
Add coverage badge and use python -m to invoke coverage
Enforce pip3, remove SyntaxError reported line
Use develop branch for coverage status
1. MultiCoreDispatcher now dispatches request in round robin fashion.
2. Added Python3 static type hinting
add Google Cloud Run Button
Preserve proxy.py name as otherwise it breaks module system. Just provide a custom service name to fix Google Cloud Run issue
Custom service name for Google Cloud Run.
Specify port in app.json as required by Google Cloud Run
Go cannot unmarshall int :)
Port desc
Looks like PORT cannot be overridden, for now use port 8080 in Dockerfile
Remove cloudrun integration.
While proxy.py runs successfully within the container, gvisor
limitations doesn't allow proxy.py to accept client connections.
Specifically membarrier syscall isn't allowed.
Update README.md
1. Rename HttpServer to MultiCoreRequestDispatcher, there wasn't
anything Http about that class, all Http implementation happens in
HttpProxy class
2. PY 2.7 DEPRECATION ERRORS
3. Add --log-format cli option
4. Remove _ from HttpProxy private methods, refactor it's core for
clarify (remove redundant close() and other details)
Abhinav Singh