proxy.py/proxy/http/websocket/frame.py

173 lines
5.0 KiB
Python
Raw Normal View History

Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
# -*- coding: utf-8 -*-
"""
proxy.py
~~~~~~~~
Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
Network monitoring, controls & Application development, testing, debugging.
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
:copyright: (c) 2013-present by Abhinav Singh and contributors.
:license: BSD, see LICENSE for more details.
"""
import hashlib
import base64
import struct
import secrets
import logging
import io
from typing import TypeVar, Type, Optional, NamedTuple
Proxy.py Dashboard (#141) * Remove redundant variables * Initialize frontend dashboard app (written in typescript) * Add a WebsocketFrame.text method to quickly build a text frame raw packet, also close connection for static file serving, atleast Google Chrome seems to hang up instead of closing the connection * Add read_and_build_static_file_response method for reusability in plugins * teardown websocket connection when opcode CONNECTION_CLOSE is received * First draft of proxy.py dashboard * Remove uglify, obfuscator is superb enough * Correct generic V * First draft of dashboard * ProtocolConfig is now Flags * First big refactor toward no-single-file-module * Working tests * Update dashboard for refactored imports * Remove proxy.py as now we can just call python -m proxy -h * Fix setup.py for refactored code * Banner update * Lint check * Fix dashboard static serving and no UNDER_TEST constant necessary * Add support for plugin imports when specified in path/to/module.MyPlugin * Update README with instructions to run proxy.py after refactor * Move dashboard under /dashboard path * Rename to devtools.ts * remove unused * Update github workflow for new directory structure * Update test command too * Fix coverage generation * *.py is an invalid syntax on windows * No * on windows * Enable execution via github zip downloads * Github Zip downloads cannot be executed as Github puts project under a folder named after Github project, this breaks python interpreter expectation of finding a __main__.py in the root directory * Forget zip runs for now * Initialize ProxyDashboard on page load rather than within typescript i.e. on script load * Enforce eslint with standard style * Add .editorconfig to make editor compatible with various style requirements (Makefile, Typescript, Python) * Remove extra empty line * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Add ability to pass headers with HttpRequestRejected exception, also remove proxy agent header for HttpRequestRejected * Fix tests * Move common code under common sub-module * Move flags under common module * Move acceptor under core * Move connection under core submodule * Move chunk_parser under http * Move http_parser as http/parser * Move http_methods as http/methods * Move http_proxy as http/proxy * Move web_server as http/server * Move status_codes as http/codes * move websocket as http/websocket * Move exception under http/exception, also move http/proxy exceptions under http/exceptions * move protocol_handler as http/handler * move devtools as http/devtools * Move version under common/version * Lifecycle if now core Event * autopep8 * Add core event queue * Register / unregister handler * Enable inspection support for frontend dashboard * Dont give an illusion of exception for HttpProtocolExceptions * Update readme for refactored codebase * DictQueueType everywhere * Move all websocket API related code under WebsocketApi class * Inspection enabled on tab switch. 1. Additionally now acceptors are assigned an int id. 2. Fix tests to match change in constructor. * Corresponding ends of the work queues can be closed immediately. Since work queues between AcceptorPool and Acceptor process is used only once, close corresponding ends asap instead of at shutdown. * No need of a manager for shared multiprocess Lock. This unnecessarily creates additional manager process. * Move threadless into its own module * Merge acceptor and acceptor_pool tests * Defer os.close * Change content display with tab clicks. Also ensure relay manager shutdown. * Remove --cov flags * Use right type for SyncManager * Ensure coverage again * Print help to discover flags, --cov certainly not available on Travis for some reason * Add pytest-cov to requirements-testing * Re-add windows on .travis also add changelog to readme * Use 3.7 and no pip upgrade since it fails on travis windows * Attempt to fix pip install on windows * Disable windows on travis, it fails and uses 3.8. Try reporting coverage from github actions * Move away from coveralls, use codecov * Codecov app installation either didnt work or token still needs to be passed * Remove travis CI * Use https://github.com/codecov/codecov-action for coverage uploads * Remove run codecov * Ha, codecov action only works on linux, what a mess * Add cookie.js though unable to use it with es5/es6 modules yet * Enable testing for python 3.8 also Build dashboard during testing * No python 3.8 on github actions yet * Autopep8 * Add separate workflows for library (python) and dashboard (node) app * Type jobs not job * Add checkout * Fix parsing node version * Fix dashboard build on windows * Show codecov instead of coveralls
2019-10-28 21:57:33 +00:00
WebsocketOpcodes = NamedTuple('WebsocketOpcodes', [
('CONTINUATION_FRAME', int),
('TEXT_FRAME', int),
('BINARY_FRAME', int),
('CONNECTION_CLOSE', int),
('PING', int),
('PONG', int),
])
websocketOpcodes = WebsocketOpcodes(0x0, 0x1, 0x2, 0x8, 0x9, 0xA)
V = TypeVar('V', bound='WebsocketFrame')
logger = logging.getLogger(__name__)
class WebsocketFrame:
"""Websocket frames parser and constructor."""
GUID = b'258EAFA5-E914-47DA-95CA-C5AB0DC85B11'
def __init__(self) -> None:
self.fin: bool = False
self.rsv1: bool = False
self.rsv2: bool = False
self.rsv3: bool = False
self.opcode: int = 0
self.masked: bool = False
self.payload_length: Optional[int] = None
self.mask: Optional[bytes] = None
self.data: Optional[bytes] = None
@classmethod
def text(cls: Type[V], data: bytes) -> bytes:
frame = cls()
frame.fin = True
frame.opcode = websocketOpcodes.TEXT_FRAME
frame.data = data
return frame.build()
def reset(self) -> None:
self.fin = False
self.rsv1 = False
self.rsv2 = False
self.rsv3 = False
self.opcode = 0
self.masked = False
self.payload_length = None
self.mask = None
self.data = None
def parse_fin_and_rsv(self, byte: int) -> None:
self.fin = bool(byte & 1 << 7)
self.rsv1 = bool(byte & 1 << 6)
self.rsv2 = bool(byte & 1 << 5)
self.rsv3 = bool(byte & 1 << 4)
self.opcode = byte & 0b00001111
def parse_mask_and_payload(self, byte: int) -> None:
self.masked = bool(byte & 0b10000000)
self.payload_length = byte & 0b01111111
def build(self) -> bytes:
if self.payload_length is None and self.data:
self.payload_length = len(self.data)
raw = io.BytesIO()
raw.write(
struct.pack(
'!B',
(1 << 7 if self.fin else 0) |
(1 << 6 if self.rsv1 else 0) |
(1 << 5 if self.rsv2 else 0) |
(1 << 4 if self.rsv3 else 0) |
self.opcode
))
assert self.payload_length is not None
if self.payload_length < 126:
raw.write(
struct.pack(
'!B',
(1 << 7 if self.masked else 0) | self.payload_length
)
)
elif self.payload_length < 1 << 16:
raw.write(
struct.pack(
'!BH',
(1 << 7 if self.masked else 0) | 126,
self.payload_length
)
)
elif self.payload_length < 1 << 64:
raw.write(
struct.pack(
'!BHQ',
(1 << 7 if self.masked else 0) | 127,
self.payload_length
)
)
else:
raise ValueError(f'Invalid payload_length { self.payload_length },'
f'maximum allowed { 1 << 64 }')
if self.masked and self.data:
mask = secrets.token_bytes(4) if self.mask is None else self.mask
raw.write(mask)
raw.write(self.apply_mask(self.data, mask))
elif self.data:
raw.write(self.data)
return raw.getvalue()
def parse(self, raw: bytes) -> bytes:
cur = 0
self.parse_fin_and_rsv(raw[cur])
cur += 1
self.parse_mask_and_payload(raw[cur])
cur += 1
if self.payload_length == 126:
data = raw[cur: cur + 2]
self.payload_length, = struct.unpack('!H', data)
cur += 2
elif self.payload_length == 127:
data = raw[cur: cur + 8]
self.payload_length, = struct.unpack('!Q', data)
cur += 8
if self.masked:
self.mask = raw[cur: cur + 4]
cur += 4
assert self.payload_length
self.data = raw[cur: cur + self.payload_length]
cur += self.payload_length
if self.masked:
assert self.mask is not None
self.data = self.apply_mask(self.data, self.mask)
return raw[cur:]
@staticmethod
def apply_mask(data: bytes, mask: bytes) -> bytes:
raw = bytearray(data)
for i in range(len(raw)):
raw[i] = raw[i] ^ mask[i % 4]
return bytes(raw)
@staticmethod
def key_to_accept(key: bytes) -> bytes:
sha1 = hashlib.sha1()
sha1.update(key + WebsocketFrame.GUID)
return base64.b64encode(sha1.digest())