* Always update latest tag for docker releases
* Update issue templates (#123)
* Invoke HttpWebServerBasePlugin.handle_request for each request in HTTP/1.1 pipeline (#125)
* Add tests to verify certificate generation
* Separate out tests for ProtocolHandler and WebServerPlugin
* Keep-alive connections for web server.
TODO: Only keep-alivei if HTTP/1.1
* Add request.path to avoid build_url repeatedly whose name is also slightly misleading
* Fix example usage of request.path
* Pipeline only for HTTP/1.1
* Lint fix
* Teardown HTTP/1.1 keep-alive request when Connection: close header is sent
* Add instructions on how to build docker image locally
* Move access_log to separate function for pretty logging
* Reduce docker image size
* Ensure teardown is always accompanied with Connection: close header
Fix tests
* Invoke proxy plugin handle_request for each request in HTTP/1.1 pipeline or when TLS interception is enabled (#128)
* Add tests for is_http_1_1_keep_alive
* Add ModifyPostDataPlugin in README
* Fixes#126
* Refactor HttpProxyBasePlugin API
* before_upstream_connection too can drop request by returning None
* Remove HTTP Server startup during tests, no longer used
* Removed unused imports
* Simplify load_plugins
* Add --timeout flag with default value of 10 second. (#129)
* Add --timeout flag with default value of 5. This value was previously hardcoded to 30
* --timeout=10 by default
* Dispatch 408 timeout when connection is dropped due to inactivity
* Add httpStatusCodes named tuple
* Update plugin client connection reference after TLS connection upgrade
* Test plugin examples (#130)
* Add tests for plugin_examples.* to ensure we never break functionality
* Add tests for plugin_examples.*
* Test man in the middle
* Lint fixes
* Checkin
* Add tests for plugin examples with TLS encryption enabled
* Threadless execution using coroutines (#134)
* Workers need not register/unregister sock for every loop
* No need of explicit socket.settimeout(0) which is same as socket.setblocking(False)
* Remove settimeout assertion
* Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown
* Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes
* Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests
* Remove unused
* Wrap selector register/unregister within a context manager
* Refactor in preparation of threadless request handling
* MyPy generator fix
* Add --threadless flag
* Internally call them acceptors
* Internally use acceptors
* Add Threadless class.
Also no need to pass family over pipe to acceptors.
* Make threadless work for a single client :)
* Threadless is soon be our default
* Close client queue
* Use context manager for register/unregister
* Fix Acceptor tests broken after refactoring
* Use asyncio tasks to invoke ProtocolHandle.handle_events
This gives all client threads a chance to respond without
waiting for other handlers to return.
* Explicitly initialize event loop per Threadless process
* Mypy fixes
* Add ThreadlessWork abstract class implemented by ProtocolHandler
* Add benchmark.py
Avoid TIME_WAIT by properly shutting down the connection.
* Add benchmark.py as part of testing workflow
* When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state
* MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence
* Unwrap if wrapped before shutdown
* Unwrap if wrapped before shutdown
* socket.SHUT_RDWR will cause leaks
* MyPy
* Add instructions for monitor.sh
* Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized
* Add Fast & Scalable section
* Update internal classes section
* Dont print out local dir path in help text :)
* Refactor
* Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang
* Add chrome_with_proxy.sh helper script
* Handle OSError during client.flush which can happen due to invalid protocol type for socket error
* Remove redundant e
* Add classmethods to quickly construct a parser object
* Don't raise from TcpConnection abstract class.
This allows both client/socket side of communication to handle
exceptions as necessary. We might refactor this again later to remove
redundant code :)
* Disable response parsing when TLS interception is enabled. See issue #127
* remove unused imports
* Within webserver parse pipelined requests only if we have a route
* Add ShortLinkPlugin plugin
* Add more shortlinks
* Add ShortLinkPlugin to README.md
* Add path forwarding too instead of leaving as excercise ;)
* Add shortlink to TOC
* Ensure no socket leaks
* Ensure no leaks
* Naming
* Default number of clients 1
* Avoid shortlinking localhost
* Stress more
* Remove pip upgrade for windows which seems to be failing on travis (#136)
* Remove pip upgrade for windows which seems to be failing on travis
* Remove windows testing on Travis, pip install is failing
* Add pipeline response parsing tests (#137)
* Add pipeline response parsing tests
* build_http_response now only adds content-length if transfer-encoding is
not provided.
Also return pending raw chunks from ChunkParser so that we can parse
pipelined chunk responses.
* os.close only for threadless (#138)
* os.close only for Threadless to avoid fd leaks
* Remove os.close mock which is only called for threadless
* Update pytest from 5.2.1 to 5.2.2 (#142)
* Update setuptools from 41.4.0 to 41.5.0 (#145)
* Update typing-extensions from 3.7.4 to 3.7.4.1 (#147)
* Update flake8 from 3.7.8 to 3.7.9 (#148)
* Update setuptools from 41.5.0 to 41.5.1 (#149)
* Update py-spy from 0.2.2 to 0.3.0 (#144)
* Proxy.py Dashboard (#141)
* Remove redundant variables
* Initialize frontend dashboard app (written in typescript)
* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection
* Add read_and_build_static_file_response method for reusability in plugins
* teardown websocket connection when opcode CONNECTION_CLOSE is received
* First draft of proxy.py dashboard
* Remove uglify, obfuscator is superb enough
* Correct generic V
* First draft of dashboard
* ProtocolConfig is now Flags
* First big refactor toward no-single-file-module
* Working tests
* Update dashboard for refactored imports
* Remove proxy.py as now we can just call python -m proxy -h
* Fix setup.py for refactored code
* Banner update
* Lint check
* Fix dashboard static serving and no UNDER_TEST constant necessary
* Add support for plugin imports when specified in path/to/module.MyPlugin
* Update README with instructions to run proxy.py after refactor
* Move dashboard under /dashboard path
* Rename to devtools.ts
* remove unused
* Update github workflow for new directory structure
* Update test command too
* Fix coverage generation
* *.py is an invalid syntax on windows
* No * on windows
* Enable execution via github zip downloads
* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory
* Forget zip runs for now
* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load
* Enforce eslint with standard style
* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)
* Remove extra empty line
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected
* Fix tests
* Move common code under common sub-module
* Move flags under common module
* Move acceptor under core
* Move connection under core submodule
* Move chunk_parser under http
* Move http_parser as http/parser
* Move http_methods as http/methods
* Move http_proxy as http/proxy
* Move web_server as http/server
* Move status_codes as http/codes
* move websocket as http/websocket
* Move exception under http/exception, also move http/proxy exceptions under http/exceptions
* move protocol_handler as http/handler
* move devtools as http/devtools
* Move version under common/version
* Lifecycle if now core Event
* autopep8
* Add core event queue
* Register / unregister handler
* Enable inspection support for frontend dashboard
* Dont give an illusion of exception for HttpProtocolExceptions
* Update readme for refactored codebase
* DictQueueType everywhere
* Move all websocket API related code under WebsocketApi class
* Inspection enabled on tab switch.
1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.
* Corresponding ends of the work queues can be closed immediately.
Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.
* No need of a manager for shared multiprocess Lock.
This unnecessarily creates additional manager process.
* Move threadless into its own module
* Merge acceptor and acceptor_pool tests
* Defer os.close
* Change content display with tab clicks.
Also ensure relay manager shutdown.
* Remove --cov flags
* Use right type for SyncManager
* Ensure coverage again
* Print help to discover flags, --cov certainly not available on Travis for some reason
* Add pytest-cov to requirements-testing
* Re-add windows on .travis also add changelog to readme
* Use 3.7 and no pip upgrade since it fails on travis windows
* Attempt to fix pip install on windows
* Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions
* Move away from coveralls, use codecov
* Codecov app installation either didnt work or token still needs to be passed
* Remove travis CI
* Use https://github.com/codecov/codecov-action for coverage uploads
* Remove run codecov
* Ha, codecov action only works on linux, what a mess
* Add cookie.js though unable to use it with es5/es6 modules yet
* Enable testing for python 3.8 also Build dashboard during testing
* No python 3.8 on github actions yet
* Autopep8
* Add separate workflows for library (python) and dashboard (node) app
* Type jobs not job
* Add checkout
* Fix parsing node version
* Fix dashboard build on windows
* Show codecov instead of coveralls
* Update mypy==0.740 (#151)
* Update README.md (#152)
* Update flags
* Update debugging instructions and run instructions for develops
* Update references to plugins directory
* For readability add sections for run from command line using pip
* Move internal doc under developer section
* Add option to pass fully-qualified plugin path
* Update setuptools from 41.5.1 to 41.6.0 (#153)
* Test refactor + Docker image CI (#154)
* Move tests into individual modules too
* Ensure one test class per file
* Fix docker image after refactoring
* Add github actions workflow for building docker image
* Fix image name
* Setup python required for extracting proxy version
* Version will also require deps
* Separate packages for Dashboard (#157)
* Refactor Makefile and add dashboard setup.py
* Package dashboard as proxy.py-dashboard pip package
* Give dashboard releases its own version
* Fix lib-package reference
* Add non-blocking embedded mode feature (#159)
* Fixes#158
* mypy fixes
* Instructions for non-blocking embed mode
* Toggle running flag before shutdown
* Add private / public key generation utils which comply with new requirements on Mac OS 10.15 (#160)
* Add utilities to generate private key and public keys with alternate cnames
* Add separate package proxy.py-plugins, fixes#156
* Generate certificates to comply with Mac requirements.
* Add utility for CSR generation and signing
* Fixes#161
* Add initial pki tests
* Give structure to dashboard app (#163)
* Separate out files for different responsibilities.
1. Add src/plugins directory. This directory holds one typescript file
per plugin. Each plugin is optionally can be displayed as a tab on
the UI.
2. Move WebsocketApi to ws.ts. This file contains all websocket APIs
provided by dashboard.py backend.
* Make dashboard pluggable
* Move devtools under core too
* Register tabs dynamically
* Typescript fixes for abstract interfaces
* Initialize plugin app body skeleton
* Call activated / deactivated on tab change
* Move plugin name within plugin classes and initialize plugin within proxy dashboard constructor
* templatize api development plugin
* eslint fixes
* use globs
* Remove useless constructors
* Move traffic_control outside of core plugin, it maps to several plugin examples like redirectToUpstreamHost, filterByUpstreamHost plugins (#165)
* Introduce sendMessage websocket api which allows for callbacks (#166)
* Introduce sendMessage websocket api which allows for callbacks, deprecate lastPingId in favor of callbacks
* Let InspectTrafficPlugin handle all pushed inspection events
* Add proxy.main.TestCase for unit testing Python application with proxy.py (#167)
* Add demonstration of how to use proxy.py within Python application unittests
* mypy fixes
* test_with_proxy example
* Add docs for proxy.main.TestCase.
Also wait for proxy.py server to come up before running the tests.
* Consistent dashboard look and feel across plugins (#169)
* Explicitly link version changelog in TOC
* Separate out app header body builder
* Ensure unsubscribe when disabling inspection. Fixes#164
* Avoid creation of new manager per dashboard instance.
* Add UI header for all plugins (tabs)
* Ensure app body for all plugin skeleton
* Move app-header and app-body within core for consistent dashboard look and feel
* Consistent UI header body for plugins
* autopep8
* Dashboard Inspect traffic tab + devtools (#170)
* Explicitly link version changelog in TOC
* Separate out app header body builder
* Ensure unsubscribe when disabling inspection. Fixes#164
* Avoid creation of new manager per dashboard instance.
* Add UI header for all plugins (tabs)
* Ensure app body for all plugin skeleton
* Move app-header and app-body within core for consistent dashboard look and feel
* Consistent UI header body for plugins
* autopep8
* make devtools
* convert to es6
* Add inspect_traffic plugin devtools app
* trigger re-build, github UI is stuck
* Dynamically load devtools within inspect traffic view
* Just copy devtools into public/dashboard folder
* Works but not how we wanted, devtools takes over entire body and doesnt contain itself within a div
* Load devtools within iframe
* Load devtools within iframe (#171)
* Allow to pass flags as kwargs too in embed mode (#172)
* Dynamically load devtools instead of on page load
* Add support for passing flags as kwargs to main / start methods.
* Fix tests for refactored code
* Allow proxy.main, proxy.start, proxy.TestCase.
Also update README.md to reflect the same.
* Use Any for **opts
* Move main as __init__ to avoid name conflicts
* Fix tests
* Update setup.py entry_point
* Explicitly install requirements before setup.py
* Explicitly mention packages of interest
* ipv6 fails on ubuntu, use ipv4
* Make typing-extensions optional
* Instead of putting it all under __init__.py, move main.py to proxy.py
* Simply make setup.py module free
* autopep8
* Devtools Protocol (#174)
* Refine docs
* Decouple relay from dashboard.
Will be re-used by devtools protocol plugin.
* Just have a single manager for all eventing
* Ofcourse managers cant be shared across processes
* Remove unused
* Add DevtoolsProtocolPlugin
* Emit REQUEST_COMPLETE core event
* Emit only if --enable-events used
* Add event emitter for response cycle
* Fill up core events to devtools protocol expectations
* Serve static content with Cache-Control header and gzip compression
* Add PWA manifest.json and icons from sample PWA apps (replace later)
* Catch any exception and be ssl agnostic
* Add CSP headers and avoid inline scripts
* Re-enable iframe and deobfuscation
* Embed plugins within <section/> block
* Make tab switching agnostic of block name
* Add support for browser history on tab change
* Default hash to #home
* Switch to tab if hash is already set
* Expand canvas to fill screen even without content
* Remove inline css for embedded devtools
* Make dashboard backend websocket API pluggable
* doc
* Move dashboard backend within proxy module, now ships via same pip package (#177)
* Allow resources to load from http and ws when running w/o https
* Move dashboard backend (dashboard.py) within proxy module. Now shipped with pip install proxy.py
* Update ref to dashboard backend in github workflows
* Add git-pre-commit hook file.
Enable it by symlinking as .git/hooks/pre-commit
* Also enable static server for dashboard serving
* Move plugin_examples/ as proxy.plugin and update readme (#179)
* Update dev guide
* Move plugin_examples/ as proxy.plugin
* Update proxy.plugin ref path in readme
* Remove unnecessary port flag
* Remove plugin_examples from github workflows
* dashboard folder is a npm package not python package anymore
* Plugins can now be tried using Docker image
* Move benchmark module within proxy (#181)
* Move benchmark within proxy module
* chmod 0644 for benchmark.py which was executable till now
* Turn utilities into its own section
* Update pytest from 5.2.3 to 5.2.4 (#180)
* Doc & Banner update to match GitHub (#182)
* Update doc and banner
* Update banner to match GitHub
* Update older banners too
* Add update_desc to .gitignore
* Update banner for dashboard to match github
* also update html, js, css
* Update twine from 2.0.0 to 3.0.0 (#183)
* Update pytest from 5.2.4 to 5.3.0 (#186)
* Testing support improvements (#185)
* Introduce proxy.Proxy context manager.
This is similar to already existing context manager `start` but
`proxy.Proxy` is a class with __enter__ and __exit__ methods. This
allows usage of `proxy.Proxy` both as context manager and for manually
setup and teardown of `proxy.py` during test setUpClass and
teardownClass methods.
* Gracefully shutdown threadless processes
* Update tests and add a VCR method. See #184
* Refactor routes
* Add Proxy to __all__
* Move TestCase under proxy.testing and test_embed.py under tests.embed module to avoid conflict with http module due to a http directory under proxy folder
* Add a base cache plugin class which can be customized for custom cache behaviors
* See #184. Add VCRPlugin which can be enabled within tests using a context manager, e.g. with self.vcr(): ...
* Make cache plugin pluggable + make cache storage pluggable
* Make dashboard npm module agnostic of top level directory
* Symlink dashboard public folder
* Dump devtools within dashboard public folder
* Remove unused 3rd party js
* Initialize Menubar (#188)
* Initialize MacOS Menubar application
* Dashboard plugin at-least needs a shutdown hook to teardown any thread/processes started by dashboard backend plugin
* Add menu bar icon
* Add respective test directories
* Sync test banners
* Move plugin tests under its own package
* Enable daemon for threads, other this wont shutdown cleanly
* Update twine from 3.0.0 to 3.1.0 (#190)
* Update setuptools from 41.6.0 to 42.0.0 (#191)
* Memory optimizations (#189)
* Avoid persisting raw content in memory within parser, simply parse and throw-away. Addresses #187
* Clarity in test comments
* Update setuptools from 42.0.0 to 42.0.1 (#193)
* Make connection queue / recv work with memoryview to avoid copies (#192)
* connection.recv now returns a memoryview
* Make connection.queue also memoryview compliant
* autopep8
* wrap in memoryview as necessary
* Add default timeout for socket_connection and test_embed urllib
* Fix tests
* Skip TestProxyPyEmbedded for now, verifying GitHub actions
* Add timeout for wait_for_server and skip only if GITHUB_ACTIONS env variable is set
* Verify if GitHub Action fails due to wait_for_server spinning forever
* Add test for wait_for_server timeout error exception
* GitHub action hangs irrespective of wait_for_server timeout, disable TestEmbed for GitHub actions
* Cleanup (#194)
* Add basic README description for dashboard
* Use spaces for all except makefile
* enable tests for py 3.5
* Python 3.5 support label
* Avoid clash of names
* Add py3.8 support and bump node to 12.x (#195)
* Add py3.8 support and bump node to 12.x
* Add 10.x, 11.x, 12.x matrix for dashboard testing
* Add Python 3.8 support label
* Single tested with label
* autopep8 (#196)
* autopep8
* Update TestCase section
* Update pytest from 5.3.0 to 5.3.1 (#197)
* Update twine from 3.1.0 to 3.1.1 (#200)
* Add reverse proxy example (#201)
* Add reverse proxy example
* Add separate sections for http proxy and web server plugins
* Add doc
* Add proxy over ssh tunnel functionality (#198)
* update mypy to 0.750 (#204)
* Test Core Eventing (#205)
* Add core event tests
* Update .gitignore with coverage
* Add shortlink gif
* Add event dispatcher test
* Test event subscriber
* Test Dashboard backend (#206)
* Update shortlink gif name
* Conditionally run workflows as necessary
* Use pytest
* It works but github workflow is not reporting any status :(
* Separate out badges
* Add python_requires to setup.py
* Update setuptools from 42.0.1 to 42.0.2 (#207)
* Add tox.ini (#208)
* Homebrew formula (#209)
* Add homebrew formula
* Build PyPi package and Homebrew installation verification
* Check develop
* bdist_wheel reported as error: invalid command "bdist_wheel"
* Move under stable/develop folders to keep Proxy class name same
* uff
* develop installs proxy not proxy.py binary
* Prepend site-packages
* Install typing-extensions explicitly with brew
* Use find_packages
* Most likely failing due to lack of find_packages in current develop branch
* Fix windows setup.py build
* test_static_web_server_serves seems flaky on Ubuntu python 3.8
* Add instructions to install using homebrew
* Disable test_static_web_server_serves on GitHub actions, seems flaky
* Packaging (#210)
* Move docker installation steps above
* Try brewing with virtualenv
* depends on python
* Update homebrew formula for stable release
* Just test brewing on latest python
* Add support for regex based routing. Fixes#203 (#211)
* Remove public folder references (#212)
* Refactor (#213)
* Add DEFAULT_HTTP_PORT constant
* Use DEFAULT_HTTP_PORT in tests
* Refactor into exception module
* Refactor into inspector module
* Refactor into server module
* Refactor into proxy module
* Build docker of Python 3.8 (#214)
* Move homebrew under helper (#215)
* Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET on no internet (#216)
* Catch TimeoutError and OSError (host unreachable)
* Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET
* Enable mccabe (#217)
* No need of per day or week stats (#218)
* Make HTTP handler constructor free of socket file number (#219)
* Refactor into acceptor module
* Add tunnel doc
* Make fileno free
* Autopep8
* Response parser now reaches COMPLETE even when no body is expected (#220)
* Stash current changes
* Refactor into connection module
* Response parser state complete when no body expect
* Raise NotImplementedError if invalid state reached within parser
* Update tox from 3.14.1 to 3.14.2 (#221)
* Update paramiko from 2.6.0 to 2.7.0 (#225)
* Update paramiko from 2.7.0 to 2.7.1 (#227)
* Proxy Pool Plugin (#228)
* Add proxy pool example. See #226
* Add ProxyPoolPlugin to doc
* Update pytest from 5.3.1 to 5.3.2 (#229)
* Update coverage from 4.5.4 to 5.0 (#230)
* Update mypy from 0.750 to 0.760 (#232)
* Update mypy from 0.760 to 0.761 (#235)
* Move manager initialization outside of top level scope. Fixes#233 (#236)
* Share lock to acceptors via pool (#238)
* Move manager initialization outside of top level scope. Fixes#233
* Share lock to acceptor via pool
* Optionally initialize manager in main thread and use the same for EventQueue initialization (#239)
* Highlight language syntax (#240)
* Highlight lang syntax
* zsh prompt
* Update coverage from 5.0 to 5.0.1 (#241)
* Integration testing (#243)
* Add tests for public/private/csr generation
* Add integration testing skeleton for mac and ubuntu
* Merge integration within lib test to avoid too many workflows
* Disable integration testing on windows for now
* Use sudo to start integration test script as lsof fails on MacOS.
lsof: WARNING: can't stat() vmhgfs file system
* Add basic integration testing for now to assert proxy works as expected when started out of develop branch
* Add a call to inbuilt http server to verify it works
* wait for server to accept requests
Co-authored-by: pyup.io bot <github-bot@pyup.io>
Abhinav Singh