Commit Graph

55 Commits

Author SHA1 Message Date
Mike aedf5933f2
Add plugin "FilterByURLRegexPlugin" (#397)
* Initial draft of filter_by_url_regex.py

* Add FilterByURLRegexPlugin

* Fix dictionary key & add logging

* Add proper logging

* Add better logging

* Add logging

* move code to handle_client_request

* development logging

* development

* development

* development

* dev

* dev

* dev

* dev

* dev

* dev

* dev

* dev

* dev

* dev

* dev

* Fix blocked log

* Add to FILTER_LIST, some tidy up

* Update FILTER_LIST

* dev

* remove scheme from url

* Add to FILTER_LIST

* Add to FILTER_LIST

* Update FILTER_LIST

* commenting

* Update FILTER_LIST

* After autopep8

* Fix Anomalous backslash in string (pep8)

* Address code quality checks - flake8 F401 & W605

* Address flake8 errors

* Attempt to fix flake8 errors

* Fix linting issues

* Address flake8 W292

* Attempt to create tests

* Add FilterByURLRegexPlugin

* Rename test

* Work on tests

* Work on tests

* Work on tests

Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com>
2020-07-13 10:40:34 +05:30
Abhinav Singh 1867d58338
Add cross ref for how to generate SSL certs. (#401) 2020-07-12 21:24:20 +05:30
Abhinav Singh 73bcf1d057
Add core pubsub eventing example and add menubar item skeleton (#396)
* Initialize menu bar items with click handler and open a popover for preferences

* Add Core PubSub eventing example

* Remove hardcoded request ids
2020-07-09 21:31:29 +05:30
Abhinav Singh b7c4c5f175
Add SSL echo server & client example (#395)
* Move wrap_socket for SSL server within utils.

Also complete proxy.common.pki gen_csr and sign_csr actions. Used by Makefile sign-https-certificates.

* Add SSL echo server and client example

* Add examples documentation
2020-07-08 16:38:58 +05:30
Abhinav Singh 682114e9e0
Decouple SSL wrap logic into connection classes (#394)
* Move wrap functionality within respective connection classes. Also decouple websocket client handshake method

* Add a TCP echo client example that works with TCP echo server example
2020-07-08 13:11:12 +05:30
Abhinav Singh c884338f42
Core acceptor pool doc, cleanup and standalone example (#393)
* Better document acceptor module and add a TCP Echo Server example

* autopep8 formating

* Rename ThreadlessWork --> Work class

* Make initialize, is_inactive and shutdown as optional interface methods.

Also introduce Readables & Writables custom types.

* Move websocket code into its own module

* Add websocket client example

* Cleanup websocket client
2020-07-07 18:01:49 +05:30
Abhinav Singh 1b0ed923d7
Refactor plugin base classes for plugin specific flags (#388)
* Update to latest code signing recommendations

* Move HttpProtocolHandlerPlugin into separate file

* Dont add subject attributes if not provided by upstream. Also handle subprocess.TimeoutExpired raised during certificate generation.  Instead of retries, we simply close the connection on timeout

* Remove plugin specific flag initialization methods for now
2020-07-04 18:17:11 +05:30
Abhinav Singh ea227b1cdf
Document FilterByClientIpPlugin & ModifyChunkResponsePlugin (#387) 2020-07-02 03:30:29 +05:30
Abhinav Singh 4117c5f0d5
Add FilterByClientIpPlugin example (#381) 2020-06-21 21:57:03 +05:30
Abhinav Singh 62012ed181
Enable docker containers for TLS Interception (#375)
* Move static values to constants.py

* Install openssl in docker containers for TLS interception

* Complete TLS Interception with Docker doc
2020-06-17 17:54:43 +05:30
Abhinav Singh 1b2966140c
Copy certificate subject from peercertificate, use ssl.PROTOCOL_TLSv1_2 for client wrap and allow TLSv1_1 for server wrap (#370) 2020-06-13 21:42:12 +05:30
Abhinav Singh d6e60774ae
Add DEFAULT_MAX_SEND_SIZE and handle SSLWantWriteError errors when dispatching to upstream servers (#368) 2020-06-13 16:05:01 +05:30
Abhinav Singh 14fda0c95a
Fixes #356 WebSockets not working with TLS interception (#365) 2020-06-11 22:45:09 +05:30
Abhinav Singh e7aa8a28f7
TLS Interception Cert Generation (#362)
* Use common.pki for interception certificate generation

* Fix tests

* Dont use certificate fields that we dont need, it leads to certificate generation error on Ubuntu

* Prepare for v2.2.0

* npm audit fix
2020-06-09 12:07:00 +05:30
Abhinav Singh 7a67d0ffe4
urlsplit compliance for https CONNECT (#360)
* Add fix to be complaint with urlparse.urlsplit semantics which expects a fully qualified URL

* Log exception stacktrace on OSError
2020-06-06 19:10:41 +05:30
Abhinav Singh 08824869c2
Fix & Re-enable VCR testing on Linux (#323)
* Fix/Re-enable VCR test on Linux

* Disable tests for Windows due to weird permission issues
2020-03-26 12:31:00 +05:30
Abhinav Singh ab1198268c
Add flag to specify custom system CA Path (#321)
* Fixes #320

* Update README and add codecov.yml

* Update codecov.yml
2020-03-25 13:30:19 +05:30
Abhinav Singh bb736aa4a0
Remove -dev from python-version (#287)
* Remove -dev from python-version. See https://twitter.com/imoracle/status/1224070108064043008 for background

* Remove benchmark module which was broken on 3.6.9 (mypy broken). Also remove 3.5 support from README
2020-02-02 13:05:06 -08:00
pva 27e3cadf03
Fix ProxyPoolPlugin.rebuild_original_path 2020-02-01 10:16:19 -08:00
Bénouare 12de12d2a7
(#258) Fix generation of certif -> chg Time.time() to self.uid.int (#258)
Co-authored-by: Abhinav Singh <mailsforabhinav@gmail.com>
2020-02-01 08:51:25 -08:00
Abhinav Singh 434e2502ec
Add urlparse fix for Python 3.6.x . Deprecate support for Python 3.5.x (#278)
* Add fix required to run on Python 3.6.  Python 3.5.x is no longer supported as it reports syntax error and no longer recognize typing syntax

* Prepare for v2.1.2
2020-01-30 22:35:41 -08:00
Abhinav Singh 87a54a0781
Fixes #267 (#277)
* Fixes #267

* Prepare for v2.1.1
2020-01-30 21:54:02 -08:00
Abhinav Singh 6740b2857a
Refactor into event submodule (#257) 2020-01-07 11:38:25 -08:00
Abhinav Singh ee4e4ce41e
Add py.typed marker, add version checker, remove deprecated methods (#255)
* Add py.typed marker

* Add version-check.py

* Remove deprecated assertDictContainsSubset usage
2020-01-06 21:23:38 -08:00
Abhinav Singh ac29e34137
Add CLI usage for pki.py and update Makefile (#254)
* Add CLI usage for pki.py

* Bump to 2.1.0

* Replace direct openssl invocation with pki utility

* Bolder

* Ordering and version in README

* Refine help
2020-01-06 19:51:18 -08:00
Abhinav Singh 63e6d22566
Optionally initialize manager in main thread and use the same for EventQueue initialization (#239) 2019-12-20 17:10:35 -08:00
Abhinav Singh c6c09395f9
Share lock to acceptors via pool (#238)
* Move manager initialization outside of top level scope.  Fixes #233

* Share lock to acceptor via pool
2019-12-20 15:13:46 -08:00
Abhinav Singh 625dc4d3e5
Move manager initialization outside of top level scope. Fixes #233 (#236) 2019-12-20 14:01:26 -08:00
Abhinav Singh ba5c84dfbd
Proxy Pool Plugin (#228)
* Add proxy pool example. See #226

* Add ProxyPoolPlugin to doc
2019-12-09 19:38:49 -08:00
Abhinav Singh ccaf868921
Response parser now reaches COMPLETE even when no body is expected (#220)
* Stash current changes

* Refactor into connection module

* Response parser state complete when no body expect

* Raise NotImplementedError if invalid state reached within parser
2019-12-02 19:18:43 -08:00
Abhinav Singh 269484df2e
Make HTTP handler constructor free of socket file number (#219)
* Refactor into acceptor module

* Add tunnel doc

* Make fileno free

* Autopep8
2019-12-02 15:55:08 -08:00
Abhinav Singh 5e714374af
Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET on no internet (#216)
* Catch TimeoutError and OSError (host unreachable)

* Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET
2019-12-02 00:08:15 -08:00
Abhinav Singh 6137fd6f82
Refactor (#213)
* Add DEFAULT_HTTP_PORT constant

* Use DEFAULT_HTTP_PORT in tests

* Refactor into exception module

* Refactor into inspector module

* Refactor into server module

* Refactor into proxy module
2019-12-01 22:46:00 -08:00
Abhinav Singh 64192250ee
Remove public folder references (#212) 2019-12-01 21:30:29 -08:00
Abhinav Singh fff914e22b
Add support for regex based routing. Fixes #203 (#211) 2019-12-01 20:57:05 -08:00
Abhinav Singh 654696d2f9
Test Core Eventing (#205)
* Add core event tests

* Update .gitignore with coverage

* Add shortlink gif

* Add event dispatcher test

* Test event subscriber
2019-11-30 22:04:43 -08:00
Abhinav Singh 6f4fe34d65
Add proxy over ssh tunnel functionality (#198) 2019-11-30 09:03:23 -08:00
Abhinav Singh b88d480cdd
Add reverse proxy example (#201)
* Add reverse proxy example

* Add separate sections for http proxy and web server plugins

* Add doc
2019-11-29 21:28:31 -08:00
Abhinav Singh ae92adc432
autopep8 (#196)
* autopep8

* Update TestCase section
2019-11-26 22:43:05 -08:00
Abhinav Singh 093e852df5
Make connection queue / recv work with memoryview to avoid copies (#192)
* connection.recv now returns a memoryview

* Make connection.queue also memoryview compliant

* autopep8

* wrap in memoryview as necessary

* Add default timeout for socket_connection and test_embed urllib

* Fix tests

* Skip TestProxyPyEmbedded for now, verifying GitHub actions

* Add timeout for wait_for_server and skip only if GITHUB_ACTIONS env variable is set

* Verify if GitHub Action fails due to wait_for_server spinning forever

* Add test for wait_for_server timeout error exception

* GitHub action hangs irrespective of wait_for_server timeout, disable TestEmbed for GitHub actions
2019-11-26 17:59:26 -08:00
Abhinav Singh 9767010425
Memory optimizations (#189)
* Avoid persisting raw content in memory within parser, simply parse and throw-away. Addresses #187

* Clarity in test comments
2019-11-24 11:59:41 -08:00
Abhinav Singh e03cae30d2
Initialize Menubar (#188)
* Initialize MacOS Menubar application

* Dashboard plugin at-least needs a shutdown hook to teardown any thread/processes started by dashboard backend plugin

* Add menu bar icon

* Add respective test directories

* Sync test banners

* Move plugin tests under its own package

* Enable daemon for threads, other this wont shutdown cleanly
2019-11-22 15:18:01 -08:00
Abhinav Singh 0c4e1cbda9
Testing support improvements (#185)
* Introduce proxy.Proxy context manager.

This is similar to already existing context manager `start` but
`proxy.Proxy` is a class with __enter__ and __exit__ methods. This
allows usage of `proxy.Proxy` both as context manager and for manually
setup and teardown of `proxy.py` during test setUpClass and
teardownClass methods.

* Gracefully shutdown threadless processes

* Update tests and add a VCR method. See #184

* Refactor routes

* Add Proxy to __all__

* Move TestCase under proxy.testing and test_embed.py under tests.embed module to avoid conflict with http module due to a http directory under proxy folder

* Add a base cache plugin class which can be customized for custom cache behaviors

* See #184. Add VCRPlugin which can be enabled within tests using a context manager, e.g. with self.vcr(): ...

* Make cache plugin pluggable + make cache storage pluggable

* Make dashboard npm module agnostic of top level directory

* Symlink dashboard public folder

* Dump devtools within dashboard public folder

* Remove unused 3rd party js
2019-11-21 21:16:01 -08:00
Abhinav Singh ad42e0d74d
Doc & Banner update to match GitHub (#182)
* Update doc and banner

* Update banner to match GitHub

* Update older banners too

* Add update_desc to .gitignore

* Update banner for dashboard to match github

* also update html, js, css
2019-11-18 20:45:51 -08:00
Abhinav Singh bb7006b67a
Move benchmark module within proxy (#181)
* Move benchmark within proxy module

* chmod 0644 for benchmark.py which was executable till now

* Turn utilities into its own section
2019-11-15 23:49:51 -08:00
Abhinav Singh 131e9366ac
Move plugin_examples/ as proxy.plugin and update readme (#179)
* Update dev guide

* Move plugin_examples/ as proxy.plugin

* Update proxy.plugin ref path in readme

* Remove unnecessary port flag

* Remove plugin_examples from github workflows

* dashboard folder is a npm package not python package anymore

* Plugins can now be tried using Docker image
2019-11-15 14:47:50 -08:00
Abhinav Singh 148c260472
Move dashboard backend within proxy module, now ships via same pip package (#177)
* Allow resources to load from http and ws when running w/o https

* Move dashboard backend (dashboard.py) within proxy module. Now shipped with pip install proxy.py

* Update ref to dashboard backend in github workflows

* Add git-pre-commit hook file.

Enable it by symlinking as .git/hooks/pre-commit

* Also enable static server for dashboard serving
2019-11-15 13:29:48 -08:00
Abhinav Singh 439d58fdc2
Devtools Protocol (#174)
* Refine docs

* Decouple relay from dashboard.

Will be re-used by devtools protocol plugin.

* Just have a single manager for all eventing

* Ofcourse managers cant be shared across processes

* Remove unused

* Add DevtoolsProtocolPlugin

* Emit REQUEST_COMPLETE core event

* Emit only if --enable-events used

* Add event emitter for response cycle

* Fill up core events to devtools protocol expectations

* Serve static content with Cache-Control header and gzip compression

* Add PWA manifest.json and icons from sample PWA apps (replace later)

* Catch any exception and be ssl agnostic

* Add CSP headers and avoid inline scripts

* Re-enable iframe and deobfuscation

* Embed plugins within <section/> block

* Make tab switching agnostic of block name

* Add support for browser history on tab change

* Default hash to #home

* Switch to tab if hash is already set

* Expand canvas to fill screen even without content

* Remove inline css for embedded devtools

* Make dashboard backend websocket API pluggable

* doc
2019-11-14 19:00:07 -08:00
Abhinav Singh c943dd7e27
Allow to pass flags as kwargs too in embed mode (#172)
* Dynamically load devtools instead of on page load

* Add support for passing flags as kwargs to main / start methods.

* Fix tests for refactored code

* Allow proxy.main, proxy.start, proxy.TestCase.

Also update README.md to reflect the same.

* Use Any for **opts

* Move main as __init__ to avoid name conflicts

* Fix tests

* Update setup.py entry_point

* Explicitly install requirements before setup.py

* Explicitly mention packages of interest

* ipv6 fails on ubuntu, use ipv4

* Make typing-extensions optional

* Instead of putting it all under __init__.py, move main.py to proxy.py

* Simply make setup.py module free

* autopep8
2019-11-13 16:38:59 -08:00
Abhinav Singh e42cfefcf6
Consistent dashboard look and feel across plugins (#169)
* Explicitly link version changelog in TOC

* Separate out app header body builder

* Ensure unsubscribe when disabling inspection.  Fixes #164

* Avoid creation of new manager per dashboard instance.

* Add UI header for all plugins (tabs)

* Ensure app body for all plugin skeleton

* Move app-header and app-body within core for consistent dashboard look and feel

* Consistent UI header body for plugins

* autopep8
2019-11-12 15:29:37 -08:00