* Expose within __all__
* Enable `--numprocesses=auto` for `pytest.ini`
* make lib-lint
* Also consider `--plugins` flag when bootstrapping plugins
* Add `from .dashboard import ProxyDashboard` in top-level `__init__.py` to make `ProxyDashboard` flags auto discoverable
* Move `--enable-dashboard` to top-level
* Move logging utility within `Logger` class
* Consider comma separated --plugin and --plugins during discover_plugins
* Refactor plugin related utilities in Plugins module
* mypy and lint
* Fix unused import
* Safe to use tempdir on Github actions to avoid race conditions???
* pki (generically disk based file) based tests are flaky on macOS under parallel execution
* Move flags to where they belong
* Move `get_default_plugins` within FlagParser as it depends upon args
TODO: We need plugin dependency system
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add a `--unix-socket-path` flag.
When available `--hostname` and `--port` flags are ignored.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* `print` statement is allowed only in `flags.py` and `version-check.py`. All other places must use a `logger` instance
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add guard for `AF_UNIX` on Windows
* Comment out assertion on Windows for now
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Move Proxy.initialize within FlagParser.initialize. Also move other staticmethods from within proxy class into utils
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* remove unused imports
* Fix `import-outside-toplevel` error
* add `make lib-flake8` and `make lib-mypy` targets
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add FAQ: OSError when wrapping client for TLS Interception
* Silence exception log for several valid "cert verification failed" by client during tls interception
* Lint checks
* Move exception handling within wrap_server/wrap_client methods
* Lint fixes
* Use certifi/cacert.pem as default --ca-file flag value
* Address tests after DEFAULT_CA_FILE change
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Use suggested fix in #683 to remove pytest ignore
* Handle should not flush now which can block, instead let base tcp handler do its magic
* test speed up, doc string, logging enhancements
* Move macOS to the end of workflows
* Fix mypy warnings
* Rename .server to .upstream
* Lint fixes
* Mark internal methods with _ prefix
* Fix broken test
* lint changes
* Wah, double client :D
* Avoid selector initialization for threadless mode
* remove unused imports
* Now HttpProtocolHandler implements BaseTcpServerHandler
* Consistent return and guard againt upstream.closed
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* AcceptorPool as context manager
* Group multiprocessing imports together
* Use com.jaxl bundle identifier as proxy.py will eventually move under jaxl org
* revisit devtools integration :)
* Emit all necessary events for devtools integration
* Lint fixes
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Make reverse proxy plugin use proxy.py core loop for async io operations
* Address lint errors
* Deprecate on_websocket_close and replace with on_client_connection_close
* Lint fixes
* Retry on SSLWantReadError and SSLWantWriteError
* Async proxy pool
* Async proxy pool
* Late upstream initialization and exception guards
* Close upstream proxy connection on client connection close
* Refactor into EventManager
* Fix tests accounting in the event manager
* Ensure each process initializes logger
* pragma no cover
* Teardown connection when proxy pool upstream proxy closes
* Add ability to customize access log format and add additional context to it
* Maintain total size for response bytes in access logs
* Fix tests broken due to new plugin methods missing mock
* Update pubsub_eventing to use EventManager to avoid entire bootstrapping step
* Build docker from 3.10-alpine
* Bump version to 3.4.0
* Add instructions for how to run dashboard
* Order of menu
* Override dashboard png path until submitted
* Add some doc string for top-level Proxy class. Also some TODOs and warnings regarding PID file overwrite
* Allow HttpProxyBasePlugin implementations to register custom descriptors for read/write events
* Remove hardcoded adblock regex into json config. Update upstream filter to block facebook, not google
* ProxyPoolPlugin and ReverseProxyPlugin must now be updated to use get/read/write descriptor APIs
* Add get/read/write descriptor API for HttpWebServerBasePlugin too
* Surface actual listening port via flags.port
* Go flagless to allow custom user defined flags. Fixes#301
* Add --cache-dir flag for cache plugin (when used with on-disk store)
* Enable discovery of flags from external plugins, example those that reside outside of proxy.py package and loaded on demand. This also allows external flags to surface in --help section
* Define --filtered-client-ips flag for FilterByClientIpPlugin
* Allow plugins to add custom command line flags. Addresses #301
* Reduce dependency over Flags class. This will be deprecated so that adhoc flags can be added without any additional manual configuration
* Fix: Argument 1 to "mock_default_args" of "TestMain" has incompatible type "Namespace"; expected "Mock"
* Reduce Flags class to just the initializer.
* Store list of action dest in FlagParser
* Move wrap_socket for SSL server within utils.
Also complete proxy.common.pki gen_csr and sign_csr actions. Used by Makefile sign-https-certificates.
* Add SSL echo server and client example
* Add examples documentation
* Better document acceptor module and add a TCP Echo Server example
* autopep8 formating
* Rename ThreadlessWork --> Work class
* Make initialize, is_inactive and shutdown as optional interface methods.
Also introduce Readables & Writables custom types.
* Move websocket code into its own module
* Add websocket client example
* Cleanup websocket client
* Update to latest code signing recommendations
* Move HttpProtocolHandlerPlugin into separate file
* Dont add subject attributes if not provided by upstream. Also handle subprocess.TimeoutExpired raised during certificate generation. Instead of retries, we simply close the connection on timeout
* Remove plugin specific flag initialization methods for now
* Use common.pki for interception certificate generation
* Fix tests
* Dont use certificate fields that we dont need, it leads to certificate generation error on Ubuntu
* Prepare for v2.2.0
* npm audit fix
* Add fix required to run on Python 3.6. Python 3.5.x is no longer supported as it reports syntax error and no longer recognize typing syntax
* Prepare for v2.1.2
* Add CLI usage for pki.py
* Bump to 2.1.0
* Replace direct openssl invocation with pki utility
* Bolder
* Ordering and version in README
* Refine help
* Add DEFAULT_HTTP_PORT constant
* Use DEFAULT_HTTP_PORT in tests
* Refactor into exception module
* Refactor into inspector module
* Refactor into server module
* Refactor into proxy module
* connection.recv now returns a memoryview
* Make connection.queue also memoryview compliant
* autopep8
* wrap in memoryview as necessary
* Add default timeout for socket_connection and test_embed urllib
* Fix tests
* Skip TestProxyPyEmbedded for now, verifying GitHub actions
* Add timeout for wait_for_server and skip only if GITHUB_ACTIONS env variable is set
* Verify if GitHub Action fails due to wait_for_server spinning forever
* Add test for wait_for_server timeout error exception
* GitHub action hangs irrespective of wait_for_server timeout, disable TestEmbed for GitHub actions
* Introduce proxy.Proxy context manager.
This is similar to already existing context manager `start` but
`proxy.Proxy` is a class with __enter__ and __exit__ methods. This
allows usage of `proxy.Proxy` both as context manager and for manually
setup and teardown of `proxy.py` during test setUpClass and
teardownClass methods.
* Gracefully shutdown threadless processes
* Update tests and add a VCR method. See #184
* Refactor routes
* Add Proxy to __all__
* Move TestCase under proxy.testing and test_embed.py under tests.embed module to avoid conflict with http module due to a http directory under proxy folder
* Add a base cache plugin class which can be customized for custom cache behaviors
* See #184. Add VCRPlugin which can be enabled within tests using a context manager, e.g. with self.vcr(): ...
* Make cache plugin pluggable + make cache storage pluggable
* Make dashboard npm module agnostic of top level directory
* Symlink dashboard public folder
* Dump devtools within dashboard public folder
* Remove unused 3rd party js
* Update doc and banner
* Update banner to match GitHub
* Update older banners too
* Add update_desc to .gitignore
* Update banner for dashboard to match github
* also update html, js, css
* Update dev guide
* Move plugin_examples/ as proxy.plugin
* Update proxy.plugin ref path in readme
* Remove unnecessary port flag
* Remove plugin_examples from github workflows
* dashboard folder is a npm package not python package anymore
* Plugins can now be tried using Docker image
* Allow resources to load from http and ws when running w/o https
* Move dashboard backend (dashboard.py) within proxy module. Now shipped with pip install proxy.py
* Update ref to dashboard backend in github workflows
* Add git-pre-commit hook file.
Enable it by symlinking as .git/hooks/pre-commit
* Also enable static server for dashboard serving
* Refine docs
* Decouple relay from dashboard.
Will be re-used by devtools protocol plugin.
* Just have a single manager for all eventing
* Ofcourse managers cant be shared across processes
* Remove unused
* Add DevtoolsProtocolPlugin
* Emit REQUEST_COMPLETE core event
* Emit only if --enable-events used
* Add event emitter for response cycle
* Fill up core events to devtools protocol expectations
* Serve static content with Cache-Control header and gzip compression
* Add PWA manifest.json and icons from sample PWA apps (replace later)
* Catch any exception and be ssl agnostic
* Add CSP headers and avoid inline scripts
* Re-enable iframe and deobfuscation
* Embed plugins within <section/> block
* Make tab switching agnostic of block name
* Add support for browser history on tab change
* Default hash to #home
* Switch to tab if hash is already set
* Expand canvas to fill screen even without content
* Remove inline css for embedded devtools
* Make dashboard backend websocket API pluggable
* doc
* Dynamically load devtools instead of on page load
* Add support for passing flags as kwargs to main / start methods.
* Fix tests for refactored code
* Allow proxy.main, proxy.start, proxy.TestCase.
Also update README.md to reflect the same.
* Use Any for **opts
* Move main as __init__ to avoid name conflicts
* Fix tests
* Update setup.py entry_point
* Explicitly install requirements before setup.py
* Explicitly mention packages of interest
* ipv6 fails on ubuntu, use ipv4
* Make typing-extensions optional
* Instead of putting it all under __init__.py, move main.py to proxy.py
* Simply make setup.py module free
* autopep8
* Add demonstration of how to use proxy.py within Python application unittests
* mypy fixes
* test_with_proxy example
* Add docs for proxy.main.TestCase.
Also wait for proxy.py server to come up before running the tests.
* Add utilities to generate private key and public keys with alternate cnames
* Add separate package proxy.py-plugins, fixes#156
* Generate certificates to comply with Mac requirements.
* Add utility for CSR generation and signing
* Fixes#161
* Add initial pki tests
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
Abhinav Singh