proxy.py/.github/workflows/testing.yml

35 lines
1.1 KiB
YAML
Raw Normal View History

name: Proxy.py
on: [push]
jobs:
build:
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
runs-on: ${{ matrix.os }}-latest
name: Python ${{ matrix.python }} on ${{ matrix.os }}
strategy:
matrix:
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
os: [macOS, ubuntu, windows]
python: [3.6, 3.7]
max-parallel: 4
fail-fast: false
steps:
- uses: actions/checkout@v1
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
- name: Setup Python
uses: actions/setup-python@v1
with:
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
python-version: ${{ matrix.python }}-dev
architecture: x64
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements-testing.txt
- name: Quality Check
run: |
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
# The GitHub editor is 127 chars wide
# W504 screams for line break after binary operators
Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more
2019-10-16 06:56:39 +00:00
flake8 --ignore=W504 --max-line-length=127 proxy.py plugin_examples.py tests.py setup.py benchmark.py
# mypy compliance check
Threadless execution using coroutines (#134) * Workers need not register/unregister sock for every loop * No need of explicit socket.settimeout(0) which is same as socket.setblocking(False) * Remove settimeout assertion * Only store sender side of Pipe(). Also ensure both end of the Pipe() are closed on shutdown * Make now global. Also we seem to be using datetime.utcnow and time.time for similar purposes * Use time.time throughout. Remove incomplete test_cache_responses_plugin to avoid resource leak in tests * Remove unused * Wrap selector register/unregister within a context manager * Refactor in preparation of threadless request handling * MyPy generator fix * Add --threadless flag * Internally call them acceptors * Internally use acceptors * Add Threadless class. Also no need to pass family over pipe to acceptors. * Make threadless work for a single client :) * Threadless is soon be our default * Close client queue * Use context manager for register/unregister * Fix Acceptor tests broken after refactoring * Use asyncio tasks to invoke ProtocolHandle.handle_events This gives all client threads a chance to respond without waiting for other handlers to return. * Explicitly initialize event loop per Threadless process * Mypy fixes * Add ThreadlessWork abstract class implemented by ProtocolHandler * Add benchmark.py Avoid TIME_WAIT by properly shutting down the connection. * Add benchmark.py as part of testing workflow * When e2e encryption is enabled, unwrap socket before shutdown to ensure CLOSED state * MyPy fixes, Union should have worked, but likely unwrap is not part of socket.socket hence * Unwrap if wrapped before shutdown * Unwrap if wrapped before shutdown * socket.SHUT_RDWR will cause leaks * MyPy * Add instructions for monitor.sh * Avoid recursive exception in new_socket_connection and only invoke plugins/shutdown if server connection was initialized * Add Fast & Scalable section * Update internal classes section * Dont print out local dir path in help text :) * Refactor * Fix a bug where response parser for HTTP only requests was reused for pipelined requests resulting in a hang * Add chrome_with_proxy.sh helper script * Handle OSError during client.flush which can happen due to invalid protocol type for socket error * Remove redundant e * Add classmethods to quickly construct a parser object * Don't raise from TcpConnection abstract class. This allows both client/socket side of communication to handle exceptions as necessary. We might refactor this again later to remove redundant code :) * Disable response parsing when TLS interception is enabled. See issue #127 * remove unused imports * Within webserver parse pipelined requests only if we have a route * Add ShortLinkPlugin plugin * Add more shortlinks * Add ShortLinkPlugin to README.md * Add path forwarding too instead of leaving as excercise ;) * Add shortlink to TOC * Ensure no socket leaks * Ensure no leaks * Naming * Default number of clients 1 * Avoid shortlinking localhost * Stress more
2019-10-16 06:56:39 +00:00
mypy --strict --ignore-missing-imports proxy.py plugin_examples.py tests.py setup.py benchmark.py
- name: Run Tests
Use selectors.DefaultSelector instead of select.select (#106) * Use selectors.DefaultSelector instead of select.select * Unregister to avoid endless loop * Cleanup event register/unregiter * Cleanup event registration * Add google-fluentd.conf. Use if running proxy.py on Google Cloud. * Send server error from proxy if for whatever reason we fail to process the request (or should it be BadRequest based upon situation?) * Fix tests for selectors * Only include proxy.py and tests.py in coverage report * Only include proxy.py for coverage * remove redundant integration test, will rewrite using mocks * Proper unregister of events * Change multi core accept model to avoid client TIME_WAIT. Fixes #97 * Catch BlockingIOError * Remove redundant comments * Simplify with AcceptorPool * Pass family to acceptor processes * Remove plugin.access_log for core plugins * Return 501 not implemented for web socket upgrade requests to inbuilt HTTP server * Add support for websocket upgrade * Websocket frame parser * Enable websocket based routing * Add WebsocketClient * Websocket * mypy fixes * Sync GitHub workflow lint and makefile lint commands. For now comment out tests which are broken :( New tests coming next. * Start fixing tests for new code * Fix formatting * Fix main tests * Add worker tests * GitHub only ran windows tests, may be require unique names * Use 3.6/3.7 dev versions for GitHub actions * Add AcceptorPool test * Add x64 and x86 matrix for actions tests * Dont use dev versions since they dont exists for x86 * Ha no x86 support itself * Add backer link * Remove support badge for 3.5 as it doesnt support typing * Update read me with changed architecture notes * Update read me with changed architecture notes * Add `import proxy` usage instructions. * Add pydoc reference for developers * Put pydoc as internal documentation
2019-10-02 07:09:35 +00:00
run: pytest tests.py