Commit Graph

30 Commits

Author SHA1 Message Date
Abhinav Singh 63e6d22566
Optionally initialize manager in main thread and use the same for EventQueue initialization (#239) 2019-12-20 17:10:35 -08:00
Abhinav Singh c6c09395f9
Share lock to acceptors via pool (#238)
* Move manager initialization outside of top level scope.  Fixes #233

* Share lock to acceptor via pool
2019-12-20 15:13:46 -08:00
Abhinav Singh 625dc4d3e5
Move manager initialization outside of top level scope. Fixes #233 (#236) 2019-12-20 14:01:26 -08:00
Abhinav Singh ba5c84dfbd
Proxy Pool Plugin (#228)
* Add proxy pool example. See #226

* Add ProxyPoolPlugin to doc
2019-12-09 19:38:49 -08:00
Abhinav Singh ccaf868921
Response parser now reaches COMPLETE even when no body is expected (#220)
* Stash current changes

* Refactor into connection module

* Response parser state complete when no body expect

* Raise NotImplementedError if invalid state reached within parser
2019-12-02 19:18:43 -08:00
Abhinav Singh 269484df2e
Make HTTP handler constructor free of socket file number (#219)
* Refactor into acceptor module

* Add tunnel doc

* Make fileno free

* Autopep8
2019-12-02 15:55:08 -08:00
Abhinav Singh 5e714374af
Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET on no internet (#216)
* Catch TimeoutError and OSError (host unreachable)

* Handle ETIMEDOUT, EHOSTUNREACH, ECONNRESET
2019-12-02 00:08:15 -08:00
Abhinav Singh 6137fd6f82
Refactor (#213)
* Add DEFAULT_HTTP_PORT constant

* Use DEFAULT_HTTP_PORT in tests

* Refactor into exception module

* Refactor into inspector module

* Refactor into server module

* Refactor into proxy module
2019-12-01 22:46:00 -08:00
Abhinav Singh 64192250ee
Remove public folder references (#212) 2019-12-01 21:30:29 -08:00
Abhinav Singh fff914e22b
Add support for regex based routing. Fixes #203 (#211) 2019-12-01 20:57:05 -08:00
Abhinav Singh 654696d2f9
Test Core Eventing (#205)
* Add core event tests

* Update .gitignore with coverage

* Add shortlink gif

* Add event dispatcher test

* Test event subscriber
2019-11-30 22:04:43 -08:00
Abhinav Singh 6f4fe34d65
Add proxy over ssh tunnel functionality (#198) 2019-11-30 09:03:23 -08:00
Abhinav Singh b88d480cdd
Add reverse proxy example (#201)
* Add reverse proxy example

* Add separate sections for http proxy and web server plugins

* Add doc
2019-11-29 21:28:31 -08:00
Abhinav Singh ae92adc432
autopep8 (#196)
* autopep8

* Update TestCase section
2019-11-26 22:43:05 -08:00
Abhinav Singh 093e852df5
Make connection queue / recv work with memoryview to avoid copies (#192)
* connection.recv now returns a memoryview

* Make connection.queue also memoryview compliant

* autopep8

* wrap in memoryview as necessary

* Add default timeout for socket_connection and test_embed urllib

* Fix tests

* Skip TestProxyPyEmbedded for now, verifying GitHub actions

* Add timeout for wait_for_server and skip only if GITHUB_ACTIONS env variable is set

* Verify if GitHub Action fails due to wait_for_server spinning forever

* Add test for wait_for_server timeout error exception

* GitHub action hangs irrespective of wait_for_server timeout, disable TestEmbed for GitHub actions
2019-11-26 17:59:26 -08:00
Abhinav Singh 9767010425
Memory optimizations (#189)
* Avoid persisting raw content in memory within parser, simply parse and throw-away. Addresses #187

* Clarity in test comments
2019-11-24 11:59:41 -08:00
Abhinav Singh e03cae30d2
Initialize Menubar (#188)
* Initialize MacOS Menubar application

* Dashboard plugin at-least needs a shutdown hook to teardown any thread/processes started by dashboard backend plugin

* Add menu bar icon

* Add respective test directories

* Sync test banners

* Move plugin tests under its own package

* Enable daemon for threads, other this wont shutdown cleanly
2019-11-22 15:18:01 -08:00
Abhinav Singh 0c4e1cbda9
Testing support improvements (#185)
* Introduce proxy.Proxy context manager.

This is similar to already existing context manager `start` but
`proxy.Proxy` is a class with __enter__ and __exit__ methods. This
allows usage of `proxy.Proxy` both as context manager and for manually
setup and teardown of `proxy.py` during test setUpClass and
teardownClass methods.

* Gracefully shutdown threadless processes

* Update tests and add a VCR method. See #184

* Refactor routes

* Add Proxy to __all__

* Move TestCase under proxy.testing and test_embed.py under tests.embed module to avoid conflict with http module due to a http directory under proxy folder

* Add a base cache plugin class which can be customized for custom cache behaviors

* See #184. Add VCRPlugin which can be enabled within tests using a context manager, e.g. with self.vcr(): ...

* Make cache plugin pluggable + make cache storage pluggable

* Make dashboard npm module agnostic of top level directory

* Symlink dashboard public folder

* Dump devtools within dashboard public folder

* Remove unused 3rd party js
2019-11-21 21:16:01 -08:00
Abhinav Singh ad42e0d74d
Doc & Banner update to match GitHub (#182)
* Update doc and banner

* Update banner to match GitHub

* Update older banners too

* Add update_desc to .gitignore

* Update banner for dashboard to match github

* also update html, js, css
2019-11-18 20:45:51 -08:00
Abhinav Singh bb7006b67a
Move benchmark module within proxy (#181)
* Move benchmark within proxy module

* chmod 0644 for benchmark.py which was executable till now

* Turn utilities into its own section
2019-11-15 23:49:51 -08:00
Abhinav Singh 131e9366ac
Move plugin_examples/ as proxy.plugin and update readme (#179)
* Update dev guide

* Move plugin_examples/ as proxy.plugin

* Update proxy.plugin ref path in readme

* Remove unnecessary port flag

* Remove plugin_examples from github workflows

* dashboard folder is a npm package not python package anymore

* Plugins can now be tried using Docker image
2019-11-15 14:47:50 -08:00
Abhinav Singh 148c260472
Move dashboard backend within proxy module, now ships via same pip package (#177)
* Allow resources to load from http and ws when running w/o https

* Move dashboard backend (dashboard.py) within proxy module. Now shipped with pip install proxy.py

* Update ref to dashboard backend in github workflows

* Add git-pre-commit hook file.

Enable it by symlinking as .git/hooks/pre-commit

* Also enable static server for dashboard serving
2019-11-15 13:29:48 -08:00
Abhinav Singh 439d58fdc2
Devtools Protocol (#174)
* Refine docs

* Decouple relay from dashboard.

Will be re-used by devtools protocol plugin.

* Just have a single manager for all eventing

* Ofcourse managers cant be shared across processes

* Remove unused

* Add DevtoolsProtocolPlugin

* Emit REQUEST_COMPLETE core event

* Emit only if --enable-events used

* Add event emitter for response cycle

* Fill up core events to devtools protocol expectations

* Serve static content with Cache-Control header and gzip compression

* Add PWA manifest.json and icons from sample PWA apps (replace later)

* Catch any exception and be ssl agnostic

* Add CSP headers and avoid inline scripts

* Re-enable iframe and deobfuscation

* Embed plugins within <section/> block

* Make tab switching agnostic of block name

* Add support for browser history on tab change

* Default hash to #home

* Switch to tab if hash is already set

* Expand canvas to fill screen even without content

* Remove inline css for embedded devtools

* Make dashboard backend websocket API pluggable

* doc
2019-11-14 19:00:07 -08:00
Abhinav Singh c943dd7e27
Allow to pass flags as kwargs too in embed mode (#172)
* Dynamically load devtools instead of on page load

* Add support for passing flags as kwargs to main / start methods.

* Fix tests for refactored code

* Allow proxy.main, proxy.start, proxy.TestCase.

Also update README.md to reflect the same.

* Use Any for **opts

* Move main as __init__ to avoid name conflicts

* Fix tests

* Update setup.py entry_point

* Explicitly install requirements before setup.py

* Explicitly mention packages of interest

* ipv6 fails on ubuntu, use ipv4

* Make typing-extensions optional

* Instead of putting it all under __init__.py, move main.py to proxy.py

* Simply make setup.py module free

* autopep8
2019-11-13 16:38:59 -08:00
Abhinav Singh e42cfefcf6
Consistent dashboard look and feel across plugins (#169)
* Explicitly link version changelog in TOC

* Separate out app header body builder

* Ensure unsubscribe when disabling inspection.  Fixes #164

* Avoid creation of new manager per dashboard instance.

* Add UI header for all plugins (tabs)

* Ensure app body for all plugin skeleton

* Move app-header and app-body within core for consistent dashboard look and feel

* Consistent UI header body for plugins

* autopep8
2019-11-12 15:29:37 -08:00
Abhinav Singh 3d65366c7b
Add proxy.main.TestCase for unit testing Python application with proxy.py (#167)
* Add demonstration of how to use proxy.py within Python application unittests

* mypy fixes

* test_with_proxy example

* Add docs for proxy.main.TestCase.

Also wait for proxy.py server to come up before running the tests.
2019-11-12 12:25:47 -08:00
Abhinav Singh 0cc4e5e625
Add private / public key generation utils which comply with new requirements on Mac OS 10.15 (#160)
* Add utilities to generate private key and public keys with alternate cnames

* Add separate package proxy.py-plugins, fixes #156

* Generate certificates to comply with Mac requirements.

* Add utility for CSR generation and signing

* Fixes #161

* Add initial pki tests
2019-11-11 16:17:13 -08:00
Abhinav Singh 93d8a55c0e
Add non-blocking embedded mode feature (#159)
* Fixes #158

* mypy fixes

* Instructions for non-blocking embed mode

* Toggle running flag before shutdown
2019-11-07 20:53:08 -08:00
Abhinav Singh 1ce16b95de
Update mypy==0.740 (#151) 2019-10-28 15:29:33 -07:00
Abhinav Singh e14548252c
Proxy.py Dashboard (#141)
* Remove redundant variables

* Initialize frontend dashboard app (written in typescript)

* Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection

* Add read_and_build_static_file_response method for reusability in plugins

* teardown websocket connection when opcode CONNECTION_CLOSE is received

* First draft of proxy.py dashboard

* Remove uglify, obfuscator is superb enough

* Correct generic V

* First draft of dashboard

* ProtocolConfig is now Flags

* First big refactor toward no-single-file-module

* Working tests

* Update dashboard for refactored imports

* Remove proxy.py as now we can just call python -m proxy -h

* Fix setup.py for refactored code

* Banner update

* Lint check

* Fix dashboard static serving and no UNDER_TEST constant necessary

* Add support for plugin imports when specified in path/to/module.MyPlugin

* Update README with instructions to run proxy.py after refactor

* Move dashboard under /dashboard path

* Rename to devtools.ts

* remove unused

* Update github workflow for new directory structure

* Update test command too

* Fix coverage generation

* *.py is an invalid syntax on windows

* No * on windows

* Enable execution via github zip downloads

* Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory

* Forget zip runs for now

* Initialize ProxyDashboard on page load rather than within typescript i.e. on script load

* Enforce eslint with standard style

* Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python)

* Remove extra empty line

* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected

* Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected

* Fix tests

* Move common code under common sub-module

* Move flags under common module

* Move acceptor under core

* Move connection under core submodule

* Move chunk_parser under http

* Move http_parser as http/parser

* Move http_methods as http/methods

* Move http_proxy as http/proxy

* Move web_server as http/server

* Move status_codes as http/codes

* move websocket as http/websocket

* Move exception under http/exception, also move http/proxy exceptions under http/exceptions

* move protocol_handler as http/handler

* move devtools as http/devtools

* Move version under common/version

* Lifecycle if now core Event

* autopep8

* Add core event queue

* Register / unregister handler

* Enable inspection support for frontend dashboard

* Dont give an illusion of exception for HttpProtocolExceptions

* Update readme for refactored codebase

* DictQueueType everywhere

* Move all websocket API related code under WebsocketApi class

* Inspection enabled on tab switch.

1. Additionally now acceptors are assigned an int id.
2. Fix tests to match change in constructor.

* Corresponding ends of the work queues can be closed immediately.

Since work queues between AcceptorPool and Acceptor process is used only
once, close corresponding ends asap instead of at shutdown.

* No need of a manager for shared multiprocess Lock.

This unnecessarily creates additional manager process.

* Move threadless into its own module

* Merge acceptor and acceptor_pool tests

* Defer os.close

* Change content display with tab clicks.

Also ensure relay manager shutdown.

* Remove --cov flags

* Use right type for SyncManager

* Ensure coverage again

* Print help to discover flags, --cov certainly not available on Travis for some reason

* Add pytest-cov to requirements-testing

* Re-add windows on .travis also add changelog to readme

* Use 3.7 and no pip upgrade since it fails on travis windows

* Attempt to fix pip install on windows

* Disable windows on travis, it fails and uses 3.8.  Try reporting coverage from github actions

* Move away from coveralls, use codecov

* Codecov app installation either didnt work or token still needs to be passed

* Remove travis CI

* Use https://github.com/codecov/codecov-action for coverage uploads

* Remove run codecov

* Ha, codecov action only works on linux, what a mess

* Add cookie.js though unable to use it with es5/es6 modules yet

* Enable testing for python 3.8 also Build dashboard during testing

* No python 3.8 on github actions yet

* Autopep8

* Add separate workflows for library (python) and dashboard (node) app

* Type jobs not job

* Add checkout

* Fix parsing node version

* Fix dashboard build on windows

* Show codecov instead of coveralls
2019-10-28 14:57:33 -07:00