Commit Graph

34 Commits

Author SHA1 Message Date
Abhinav Singh bff171ec26
Fix basic auth condition (#482)
* Fix basic auth condition

* Prepare for v2.3.1
2021-01-10 22:12:56 +05:30
Abhinav Singh 137ce457bb
Separate basic auth plugin outside of core server (#453)
* Separate basic auth plugin outside of core

* Put basic auth plugin at top
2020-10-14 20:00:29 +05:30
Abhinav Singh 0744cd8e7f
Go flagless to allow custom user defined flags. (#452)
* Go flagless to allow custom user defined flags. Fixes #301

* Add --cache-dir flag for cache plugin (when used with on-disk store)

* Enable discovery of flags from external plugins, example those that reside outside of proxy.py package and loaded on demand.  This also allows external flags to surface in --help section

* Define --filtered-client-ips flag for FilterByClientIpPlugin
2020-10-14 10:51:56 +05:30
Abhinav Singh 4804c534f8
Update mypy (#449) 2020-10-10 16:23:29 +05:30
Abhinav Singh 8cc349be48
Allow plugins to add custom command line flags (#438)
* Allow plugins to add custom command line flags.  Addresses #301

* Reduce dependency over Flags class.  This will be deprecated so that adhoc flags can be added without any additional manual configuration

* Fix: Argument 1 to "mock_default_args" of "TestMain" has incompatible type "Namespace"; expected "Mock"

* Reduce Flags class to just the initializer.

* Store list of action dest in FlagParser
2020-10-03 16:25:43 +05:30
Pascal COMBES e3a1b7f963 Ensure plugins are loaded only once.
Also changed module name for plugins passed by type.
2020-08-11 22:14:51 +02:00
Pascal COMBES 1b8d9a8e67 Pass a list plugin class objects or bytes to proxy when used in embeded mode.
No automated tests for the feature yet.
2020-08-11 16:28:32 +02:00
Pascal COMBES 929800200e Remove test for 'HttpWebServerRouteHandler'
This does not exist (as fas as I can see) and it bother the linter (Mypy),
when I tell it `klass` is a `type` instance.
2020-08-11 16:12:28 +02:00
Abhinav Singh b7c4c5f175
Add SSL echo server & client example (#395)
* Move wrap_socket for SSL server within utils.

Also complete proxy.common.pki gen_csr and sign_csr actions. Used by Makefile sign-https-certificates.

* Add SSL echo server and client example

* Add examples documentation
2020-07-08 16:38:58 +05:30
Abhinav Singh c884338f42
Core acceptor pool doc, cleanup and standalone example (#393)
* Better document acceptor module and add a TCP Echo Server example

* autopep8 formating

* Rename ThreadlessWork --> Work class

* Make initialize, is_inactive and shutdown as optional interface methods.

Also introduce Readables & Writables custom types.

* Move websocket code into its own module

* Add websocket client example

* Cleanup websocket client
2020-07-07 18:01:49 +05:30
Abhinav Singh 1b0ed923d7
Refactor plugin base classes for plugin specific flags (#388)
* Update to latest code signing recommendations

* Move HttpProtocolHandlerPlugin into separate file

* Dont add subject attributes if not provided by upstream. Also handle subprocess.TimeoutExpired raised during certificate generation.  Instead of retries, we simply close the connection on timeout

* Remove plugin specific flag initialization methods for now
2020-07-04 18:17:11 +05:30
Abhinav Singh 4117c5f0d5
Add FilterByClientIpPlugin example (#381) 2020-06-21 21:57:03 +05:30
Abhinav Singh 62012ed181
Enable docker containers for TLS Interception (#375)
* Move static values to constants.py

* Install openssl in docker containers for TLS interception

* Complete TLS Interception with Docker doc
2020-06-17 17:54:43 +05:30
Abhinav Singh 1b2966140c
Copy certificate subject from peercertificate, use ssl.PROTOCOL_TLSv1_2 for client wrap and allow TLSv1_1 for server wrap (#370) 2020-06-13 21:42:12 +05:30
Abhinav Singh d6e60774ae
Add DEFAULT_MAX_SEND_SIZE and handle SSLWantWriteError errors when dispatching to upstream servers (#368) 2020-06-13 16:05:01 +05:30
Abhinav Singh e7aa8a28f7
TLS Interception Cert Generation (#362)
* Use common.pki for interception certificate generation

* Fix tests

* Dont use certificate fields that we dont need, it leads to certificate generation error on Ubuntu

* Prepare for v2.2.0

* npm audit fix
2020-06-09 12:07:00 +05:30
Abhinav Singh ab1198268c
Add flag to specify custom system CA Path (#321)
* Fixes #320

* Update README and add codecov.yml

* Update codecov.yml
2020-03-25 13:30:19 +05:30
Abhinav Singh 434e2502ec
Add urlparse fix for Python 3.6.x . Deprecate support for Python 3.5.x (#278)
* Add fix required to run on Python 3.6.  Python 3.5.x is no longer supported as it reports syntax error and no longer recognize typing syntax

* Prepare for v2.1.2
2020-01-30 22:35:41 -08:00
Abhinav Singh 87a54a0781
Fixes #267 (#277)
* Fixes #267

* Prepare for v2.1.1
2020-01-30 21:54:02 -08:00
Abhinav Singh ac29e34137
Add CLI usage for pki.py and update Makefile (#254)
* Add CLI usage for pki.py

* Bump to 2.1.0

* Replace direct openssl invocation with pki utility

* Bolder

* Ordering and version in README

* Refine help
2020-01-06 19:51:18 -08:00
Abhinav Singh 63e6d22566
Optionally initialize manager in main thread and use the same for EventQueue initialization (#239) 2019-12-20 17:10:35 -08:00
Abhinav Singh 6137fd6f82
Refactor (#213)
* Add DEFAULT_HTTP_PORT constant

* Use DEFAULT_HTTP_PORT in tests

* Refactor into exception module

* Refactor into inspector module

* Refactor into server module

* Refactor into proxy module
2019-12-01 22:46:00 -08:00
Abhinav Singh 64192250ee
Remove public folder references (#212) 2019-12-01 21:30:29 -08:00
Abhinav Singh 093e852df5
Make connection queue / recv work with memoryview to avoid copies (#192)
* connection.recv now returns a memoryview

* Make connection.queue also memoryview compliant

* autopep8

* wrap in memoryview as necessary

* Add default timeout for socket_connection and test_embed urllib

* Fix tests

* Skip TestProxyPyEmbedded for now, verifying GitHub actions

* Add timeout for wait_for_server and skip only if GITHUB_ACTIONS env variable is set

* Verify if GitHub Action fails due to wait_for_server spinning forever

* Add test for wait_for_server timeout error exception

* GitHub action hangs irrespective of wait_for_server timeout, disable TestEmbed for GitHub actions
2019-11-26 17:59:26 -08:00
Abhinav Singh 0c4e1cbda9
Testing support improvements (#185)
* Introduce proxy.Proxy context manager.

This is similar to already existing context manager `start` but
`proxy.Proxy` is a class with __enter__ and __exit__ methods. This
allows usage of `proxy.Proxy` both as context manager and for manually
setup and teardown of `proxy.py` during test setUpClass and
teardownClass methods.

* Gracefully shutdown threadless processes

* Update tests and add a VCR method. See #184

* Refactor routes

* Add Proxy to __all__

* Move TestCase under proxy.testing and test_embed.py under tests.embed module to avoid conflict with http module due to a http directory under proxy folder

* Add a base cache plugin class which can be customized for custom cache behaviors

* See #184. Add VCRPlugin which can be enabled within tests using a context manager, e.g. with self.vcr(): ...

* Make cache plugin pluggable + make cache storage pluggable

* Make dashboard npm module agnostic of top level directory

* Symlink dashboard public folder

* Dump devtools within dashboard public folder

* Remove unused 3rd party js
2019-11-21 21:16:01 -08:00
Abhinav Singh ad42e0d74d
Doc & Banner update to match GitHub (#182)
* Update doc and banner

* Update banner to match GitHub

* Update older banners too

* Add update_desc to .gitignore

* Update banner for dashboard to match github

* also update html, js, css
2019-11-18 20:45:51 -08:00
Abhinav Singh 131e9366ac
Move plugin_examples/ as proxy.plugin and update readme (#179)
* Update dev guide

* Move plugin_examples/ as proxy.plugin

* Update proxy.plugin ref path in readme

* Remove unnecessary port flag

* Remove plugin_examples from github workflows

* dashboard folder is a npm package not python package anymore

* Plugins can now be tried using Docker image
2019-11-15 14:47:50 -08:00
Abhinav Singh 148c260472
Move dashboard backend within proxy module, now ships via same pip package (#177)
* Allow resources to load from http and ws when running w/o https

* Move dashboard backend (dashboard.py) within proxy module. Now shipped with pip install proxy.py

* Update ref to dashboard backend in github workflows

* Add git-pre-commit hook file.

Enable it by symlinking as .git/hooks/pre-commit

* Also enable static server for dashboard serving
2019-11-15 13:29:48 -08:00
Abhinav Singh 439d58fdc2
Devtools Protocol (#174)
* Refine docs

* Decouple relay from dashboard.

Will be re-used by devtools protocol plugin.

* Just have a single manager for all eventing

* Ofcourse managers cant be shared across processes

* Remove unused

* Add DevtoolsProtocolPlugin

* Emit REQUEST_COMPLETE core event

* Emit only if --enable-events used

* Add event emitter for response cycle

* Fill up core events to devtools protocol expectations

* Serve static content with Cache-Control header and gzip compression

* Add PWA manifest.json and icons from sample PWA apps (replace later)

* Catch any exception and be ssl agnostic

* Add CSP headers and avoid inline scripts

* Re-enable iframe and deobfuscation

* Embed plugins within <section/> block

* Make tab switching agnostic of block name

* Add support for browser history on tab change

* Default hash to #home

* Switch to tab if hash is already set

* Expand canvas to fill screen even without content

* Remove inline css for embedded devtools

* Make dashboard backend websocket API pluggable

* doc
2019-11-14 19:00:07 -08:00
Abhinav Singh c943dd7e27
Allow to pass flags as kwargs too in embed mode (#172)
* Dynamically load devtools instead of on page load

* Add support for passing flags as kwargs to main / start methods.

* Fix tests for refactored code

* Allow proxy.main, proxy.start, proxy.TestCase.

Also update README.md to reflect the same.

* Use Any for **opts

* Move main as __init__ to avoid name conflicts

* Fix tests

* Update setup.py entry_point

* Explicitly install requirements before setup.py

* Explicitly mention packages of interest

* ipv6 fails on ubuntu, use ipv4

* Make typing-extensions optional

* Instead of putting it all under __init__.py, move main.py to proxy.py

* Simply make setup.py module free

* autopep8
2019-11-13 16:38:59 -08:00
Abhinav Singh 3d65366c7b
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167)
* Add demonstration of how to use proxy.py within Python application unittests

* mypy fixes

* test_with_proxy example

* Add docs for proxy.main.TestCase.

Also wait for proxy.py server to come up before running the tests.
2019-11-12 12:25:47 -08:00
Abhinav Singh 0cc4e5e625
Add private / public key generation utils which comply with new requirements on Mac OS 10.15 (#160)
* Add utilities to generate private key and public keys with alternate cnames

* Add separate package proxy.py-plugins, fixes #156

* Generate certificates to comply with Mac requirements.

* Add utility for CSR generation and signing

* Fixes #161

* Add initial pki tests
2019-11-11 16:17:13 -08:00
Abhinav Singh 1ce16b95de
Update mypy==0.740 (#151) 2019-10-28 15:29:33 -07:00
Abhinav Singh e14548252c
Proxy.py Dashboard (#141)
* Remove redundant variables

* Initialize frontend dashboard app (written in typescript)

* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection

* Add read_and_build_static_file_response method for reusability in plugins

* teardown websocket connection when opcode CONNECTION_CLOSE is received

* First draft of proxy.py dashboard

* Remove uglify, obfuscator is superb enough

* Correct generic V

* First draft of dashboard

* ProtocolConfig is now Flags

* First big refactor toward no-single-file-module

* Working tests

* Update dashboard for refactored imports

* Remove proxy.py as now we can just call python -m proxy -h

* Fix setup.py for refactored code

* Banner update

* Lint check

* Fix dashboard static serving and no UNDER_TEST constant necessary

* Add support for plugin imports when specified in path/to/module.MyPlugin

* Update README with instructions to run proxy.py after refactor

* Move dashboard under /dashboard path

* Rename to devtools.ts

* remove unused

* Update github workflow for new directory structure

* Update test command too

* Fix coverage generation

* *.py is an invalid syntax on windows

* No * on windows

* Enable execution via github zip downloads

* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory

* Forget zip runs for now

* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load

* Enforce eslint with standard style

* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)

* Remove extra empty line

* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected

* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected

* Fix tests

* Move common code under common sub-module

* Move flags under common module

* Move acceptor under core

* Move connection under core submodule

* Move chunk_parser under http

* Move http_parser as http/parser

* Move http_methods as http/methods

* Move http_proxy as http/proxy

* Move web_server as http/server

* Move status_codes as http/codes

* move websocket as http/websocket

* Move exception under http/exception, also move http/proxy exceptions under http/exceptions

* move protocol_handler as http/handler

* move devtools as http/devtools

* Move version under common/version

* Lifecycle if now core Event

* autopep8

* Add core event queue

* Register / unregister handler

* Enable inspection support for frontend dashboard

* Dont give an illusion of exception for HttpProtocolExceptions

* Update readme for refactored codebase

* DictQueueType everywhere

* Move all websocket API related code under WebsocketApi class

* Inspection enabled on tab switch.

1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.

* Corresponding ends of the work queues can be closed immediately.

Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.

* No need of a manager for shared multiprocess Lock.

This unnecessarily creates additional manager process.

* Move threadless into its own module

* Merge acceptor and acceptor_pool tests

* Defer os.close

* Change content display with tab clicks.

Also ensure relay manager shutdown.

* Remove --cov flags

* Use right type for SyncManager

* Ensure coverage again

* Print help to discover flags, --cov certainly not available on Travis for some reason

* Add pytest-cov to requirements-testing

* Re-add windows on .travis also add changelog to readme

* Use 3.7 and no pip upgrade since it fails on travis windows

* Attempt to fix pip install on windows

* Disable windows on travis, it fails and uses 3.8.  Try reporting coverage from github actions

* Move away from coveralls, use codecov

* Codecov app installation either didnt work or token still needs to be passed

* Remove travis CI

* Use https://github.com/codecov/codecov-action for coverage uploads

* Remove run codecov

* Ha, codecov action only works on linux, what a mess

* Add cookie.js though unable to use it with es5/es6 modules yet

* Enable testing for python 3.8 also Build dashboard during testing

* No python 3.8 on github actions yet

* Autopep8

* Add separate workflows for library (python) and dashboard (node) app

* Type jobs not job

* Add checkout

* Fix parsing node version

* Fix dashboard build on windows

* Show codecov instead of coveralls
2019-10-28 14:57:33 -07:00