CustomDnsResolver plugin, CloudflareDnsResolver plugin, Allow plugins to configure network interface (#671)

* Add CustomDnsResolver plugin.  Addresses #535 and #664

* Add cloudflare DNS resolver plugin

* Lint fixes

LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2013-2022 by Abhinav Singh and contributors.
+Copyright (c) 2013-present by Abhinav Singh and contributors.
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without modification,

README.md

@@ -56,6 +56,8 @@
     - [Proxy Pool Plugin](#proxypoolplugin)
     - [FilterByClientIpPlugin](#filterbyclientipplugin)
     - [ModifyChunkResponsePlugin](#modifychunkresponseplugin)
+    - [CloudflareDnsResolverPlugin](#cloudflarednsresolverplugin)
+    - [CustomDnsResolverPlugin](#customdnsresolverplugin)
   - [HTTP Web Server Plugins](#http-web-server-plugins)
     - [Reverse Proxy](#reverse-proxy)
     - [Web Server Route](#web-server-route)
@@ -720,6 +722,36 @@ plugin
 
 Modify `ModifyChunkResponsePlugin` to your taste. Example, instead of sending hardcoded chunks, parse and modify the original `JSON` chunks received from the upstream server.
 
+### CloudflareDnsResolverPlugin
+
+This plugin uses `Cloudflare` hosted `DNS-over-HTTPS` [API](https://developers.cloudflare.com/1.1.1.1/encrypted-dns/dns-over-https/make-api-requests/dns-json) (json).
+
+Start `proxy.py` as:
+
+```bash
+❯ proxy \
+    --plugins proxy.plugin.CloudflareDnsResolverPlugin
+```
+
+By default, `CloudflareDnsResolverPlugin` runs in `security` mode (provides malware protection).  Use `--cloudflare-dns-mode family` to also enable
+adult content protection.
+
+### CustomDnsResolverPlugin
+
+This plugin demonstrate how to use a custom DNS resolution implementation with `proxy.py`.
+This example plugin currently uses Python's in-built resolution mechanism.  Customize code
+to your taste.  Example, query your custom DNS server, implement DoH or other mechanisms.
+
+Start `proxy.py` as:
+
+```bash
+❯ proxy \
+    --plugins proxy.plugin.CustomDnsResolverPlugin
+```
+
+`HttpProxyBasePlugin.resolve_dns` can also be used to configure `network interface` which
+must be used as the `source_address` for connection to the upstream server.
+
 ## HTTP Web Server Plugins
 
 ### Reverse Proxy
@@ -1013,7 +1045,9 @@ with TLS Interception:
 
 **This is a WIP and may not work as documented**
 
-Requires `paramiko` to work. See [requirements-tunnel.txt](https://github.com/abhinavsingh/proxy.py/blob/develop/requirements-tunnel.txt)
+Requires `paramiko` to work.
+
+See [requirements-tunnel.txt](https://github.com/abhinavsingh/proxy.py/blob/develop/requirements-tunnel.txt)
 
 ## Proxy Remote Requests Locally
 
@@ -1182,7 +1216,8 @@ if __name__ == '__main__':
 
 ## Loading Plugins
 
-You can, of course, list plugins to load in the input arguments list of `proxy.main` or `Proxy` constructor. Use the `--plugins` flag when starting from command line:
+You can, of course, list plugins to load in the input arguments list of `proxy.main` or
+`Proxy` constructor. Use the `--plugins` flag when starting from command line:
 
 ```python
 import proxy
@@ -1193,7 +1228,8 @@ if __name__ == '__main__':
   ])
 ```
 
-For simplicity you can pass the list of plugins to load as a keyword argument to `proxy.main` or the `Proxy` constructor:
+For simplicity you can pass the list of plugins to load as a keyword argument to `proxy.main` or
+the `Proxy` constructor:
 
 ```python
 import proxy
@@ -1353,8 +1389,8 @@ Contributors must start `proxy.py` from source to verify and develop new feature
 
 See [Run proxy.py from command line using repo source](#from-command-line-using-repo-source) for details.
 
-[![WARNING](https://img.shields.io/static/v1?label=MacOS&message=warning&color=red)]
-(https://github.com/abhinavsingh/proxy.py/issues/642#issuecomment-960819271) On `macOS`
+
+[![WARNING](https://img.shields.io/static/v1?label=MacOS&message=warning&color=red)](https://github.com/abhinavsingh/proxy.py/issues/642#issuecomment-960819271) On `macOS`
 you must install `Python` using `pyenv`, as `Python` installed via `homebrew` tends
 to be problematic.  See linked thread for more details.
 
@@ -1575,7 +1611,8 @@ FILE
 
 # Run Dashboard
 
-Dashboard is currently under development and not yet bundled with `pip` packages.  To run dashboard, you must checkout the source.
+Dashboard is currently under development and not yet bundled with `pip` packages.
+To run dashboard, you must checkout the source.
 
 Dashboard is written in Typescript and SCSS, so let's build it first using:
 
@@ -1606,9 +1643,12 @@ $ open http://localhost:8899/dashboard/
 
 ## Inspect Traffic
 
-Wait for embedded `Chrome Dev Console` to load.  Currently, detail about all traffic flowing through `proxy.py` is pushed to the `Inspect Traffic` tab.  However, received payloads are not yet integrated with the embedded dev console.
+Wait for embedded `Chrome Dev Console` to load.  Currently, detail about all traffic flowing
+through `proxy.py` is pushed to the `Inspect Traffic` tab.  However, received payloads are not
+yet integrated with the embedded dev console.
 
-Current functionality can be verified by opening the `Dev Console` of dashboard and inspecting the websocket connection that dashboard established with the `proxy.py` server.
+Current functionality can be verified by opening the `Dev Console` of dashboard and inspecting
+the websocket connection that dashboard established with the `proxy.py` server.
 
 [![Proxy.Py Dashboard Inspect Traffic](https://raw.githubusercontent.com/abhinavsingh/proxy.py/develop/Dashboard.png)](https://github.com/abhinavsingh/proxy.py)
 

proxy/common/utils.py

@@ -181,7 +181,7 @@ def wrap_socket(
 
 
 def new_socket_connection(
-        addr: Tuple[str, int], timeout: int = DEFAULT_TIMEOUT,
+        addr: Tuple[str, int], timeout: int = DEFAULT_TIMEOUT, source_address: Optional[Tuple[str, int]] = None,
 ) -> socket.socket:
     conn = None
     try:
@@ -205,7 +205,7 @@ def new_socket_connection(
         return conn
 
     # try to establish dual stack IPv4/IPv6 connection.
-    return socket.create_connection(addr, timeout=timeout)
+    return socket.create_connection(addr, timeout=timeout, source_address=source_address)
 
 
 class socket_connection(contextlib.ContextDecorator):

proxy/core/connection/server.py

@@ -33,9 +33,11 @@ class TcpServerConnection(TcpConnection):
             raise TcpConnectionUninitializedException()
         return self._conn
 
-    def connect(self) -> None:
+    def connect(self, addr: Optional[Tuple[str, int]] = None, source_address: Optional[Tuple[str, int]] = None) -> None:
         if self._conn is None:
-            self._conn = new_socket_connection(self.addr)
+            self._conn = new_socket_connection(
+                addr or self.addr, source_address=source_address,
+            )
         self.closed = False
 
     def wrap(self, hostname: str, ca_file: Optional[str]) -> None:

proxy/http/proxy/plugin.py

@@ -77,6 +77,22 @@ class HttpProxyBasePlugin(ABC):
         """Implementations must now read data over the socket."""
         return False  # pragma: no cover
 
+    def resolve_dns(self, host: str, port: int) -> Tuple[Optional[str], Optional[Tuple[str, int]]]:
+        """Resolve upstream server host to an IP address.
+
+        Optionally also override the source address to use for
+        connection with upstream server.
+
+        For upstream IP:
+        Return None to use default resolver available to the system.
+        Return ip address as string to use your custom resolver.
+
+        For source address:
+        Return None to use default source address
+        Return 2-tuple representing (host, port) to use as source address
+        """
+        return None, None
+
     @abstractmethod
     def before_upstream_connection(
             self, request: HttpParser,

proxy/http/proxy/server.py

@@ -509,13 +509,30 @@ class HttpProxyPlugin(HttpProtocolHandlerPlugin):
                     'Connecting to upstream %s:%s' %
                     (text_(host), port),
                 )
-                self.server.connect()
+                # Invoke plugin.resolve_dns
+                upstream_ip, source_addr = None, None
+                for plugin in self.plugins.values():
+                    upstream_ip, source_addr = plugin.resolve_dns(
+                        text_(host), port,
+                    )
+                    if upstream_ip or source_addr:
+                        break
+                # Connect with overridden upstream IP and source address
+                # if any of the plugin returned a non-null value.
+                self.server.connect(
+                    addr=None if not upstream_ip else (
+                        upstream_ip, port,
+                    ), source_address=source_addr,
+                )
                 self.server.connection.setblocking(False)
                 logger.debug(
                     'Connected to upstream %s:%s' %
                     (text_(host), port),
                 )
             except Exception as e:  # TimeoutError, socket.gaierror
+                logger.exception(
+                    'Unable to connect with upstream server', exc_info=e,
+                )
                 self.server.closed = True
                 raise ProxyConnectionFailed(text_(host), port, repr(e)) from e
         else:

proxy/plugin/__init__.py

@@ -21,6 +21,8 @@ from .proxy_pool import ProxyPoolPlugin
 from .filter_by_client_ip import FilterByClientIpPlugin
 from .filter_by_url_regex import FilterByURLRegexPlugin
 from .modify_chunk_response import ModifyChunkResponsePlugin
+from .custom_dns_resolver import CustomDnsResolverPlugin
+from .cloudflare_dns import CloudflareDnsResolverPlugin
 
 __all__ = [
     'CacheResponsesPlugin',
@@ -37,4 +39,6 @@ __all__ = [
     'FilterByClientIpPlugin',
     'ModifyChunkResponsePlugin',
     'FilterByURLRegexPlugin',
+    'CustomDnsResolverPlugin',
+    'CloudflareDnsResolverPlugin',
 ]

proxy/plugin/cloudflare_dns.py

@@ -0,0 +1,94 @@
+# -*- coding: utf-8 -*-
+"""
+    proxy.py
+    ~~~~~~~~
+    ⚡⚡⚡ Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
+    Network monitoring, controls & Application development, testing, debugging.
+
+    :copyright: (c) 2013-present by Abhinav Singh and contributors.
+    :license: BSD, see LICENSE for more details.
+"""
+import logging
+
+try:
+    import httpx
+except ImportError:
+    pass
+
+from typing import Optional, Tuple
+
+from ..common.flag import flags
+from ..http.parser import HttpParser
+from ..http.proxy import HttpProxyBasePlugin
+
+logger = logging.getLogger(__name__)
+
+
+flags.add_argument(
+    '--cloudflare-dns-mode',
+    type=str,
+    default='security',
+    help='Default: security.  Either "security" (for malware protection) ' +
+    'or "family" (for malware and adult content protection)',
+)
+
+
+class CloudflareDnsResolverPlugin(HttpProxyBasePlugin):
+    """This plugin uses Cloudflare DNS resolver to provide protection
+    against malwares and adult content.  Implementation uses DoH specification.
+
+    See https://developers.cloudflare.com/1.1.1.1/1.1.1.1-for-families
+    See https://developers.cloudflare.com/1.1.1.1/encrypted-dns/dns-over-https/make-api-requests/dns-json
+
+    NOTE: For this plugin to work, make sure to bypass proxy for 1.1.1.1
+
+    NOTE: This plugin requires additional dependency because DoH mandates
+    a HTTP2 complaint client.  Install `httpx` dependency as:
+
+    pip install "httpx[http2]"
+    """
+
+    def resolve_dns(self, host: str, port: int) -> Tuple[Optional[str], Optional[Tuple[str, int]]]:
+        try:
+            context = httpx.create_ssl_context(http2=True)
+            # TODO: Support resolution via Authority (SOA) to add support for
+            # AAAA (IPv6) query
+            r = httpx.get(
+                'https://{0}.cloudflare-dns.com/dns-query?name={1}&type=A'.format(
+                    self.flags.cloudflare_dns_mode, host,
+                ),
+                headers={'accept': 'application/dns-json'},
+                verify=context,
+                timeout=httpx.Timeout(timeout=5.0),
+                proxies={
+                    'all://': None,
+                },
+            )
+            if r.status_code != 200:
+                return None, None
+            response = r.json()
+            answers = response.get('Answer', [])
+            if len(answers) == 0:
+                return None, None
+            # TODO: Utilize TTL to cache response locally
+            # instead of making a DNS query repeatedly for the same host.
+            return answers[0]['data'], None
+        except Exception as e:
+            logger.exception('Unable to resolve DNS-over-HTTPS', exc_info=e)
+            return None, None
+
+    def before_upstream_connection(
+            self, request: HttpParser,
+    ) -> Optional[HttpParser]:
+        return request
+
+    def handle_client_request(
+            self, request: HttpParser,
+    ) -> Optional[HttpParser]:
+        return request
+
+    def handle_upstream_chunk(self, chunk: memoryview) -> memoryview:
+        return chunk
+
+    def on_upstream_connection_close(self) -> None:
+        pass

proxy/plugin/custom_dns_resolver.py

@@ -0,0 +1,52 @@
+# -*- coding: utf-8 -*-
+"""
+    proxy.py
+    ~~~~~~~~
+    ⚡⚡⚡ Fast, Lightweight, Pluggable, TLS interception capable proxy server focused on
+    Network monitoring, controls & Application development, testing, debugging.
+
+    :copyright: (c) 2013-present by Abhinav Singh and contributors.
+    :license: BSD, see LICENSE for more details.
+"""
+import socket
+
+from typing import Optional, Tuple
+
+from ..http.parser import HttpParser
+from ..http.proxy import HttpProxyBasePlugin
+
+
+class CustomDnsResolverPlugin(HttpProxyBasePlugin):
+    """This plugin demonstrate how to use your own custom DNS resolver."""
+
+    def resolve_dns(self, host: str, port: int) -> Tuple[Optional[str], Optional[Tuple[str, int]]]:
+        """Here we are using in-built python resolver for demonstration.
+
+        Ideally you would like to query your custom DNS server or even use DoH to make
+        real sense out of this plugin.
+
+        2nd parameter returned is None.  Return a 2-tuple to configure underlying interface
+        to use for connection to the upstream server.
+        """
+        try:
+            return socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP)[0][4][0], None
+        except socket.gaierror:
+            # Ideally we can also thrown HttpRequestRejected or HttpProtocolException here
+            # Returning None simply fallback to core generated exceptions.
+            return None, None
+
+    def before_upstream_connection(
+            self, request: HttpParser,
+    ) -> Optional[HttpParser]:
+        return request
+
+    def handle_client_request(
+            self, request: HttpParser,
+    ) -> Optional[HttpParser]:
+        return request
+
+    def handle_upstream_chunk(self, chunk: memoryview) -> memoryview:
+        return chunk
+
+    def on_upstream_connection_close(self) -> None:
+        pass

tests/common/test_utils.py

@@ -43,7 +43,9 @@ class TestSocketConnectionUtils(unittest.TestCase):
     @mock.patch('socket.create_connection')
     def test_new_socket_connection_dual(self, mock_socket: mock.Mock) -> None:
         conn = new_socket_connection(self.addr_dual)
-        mock_socket.assert_called_with(self.addr_dual, timeout=DEFAULT_TIMEOUT)
+        mock_socket.assert_called_with(
+            self.addr_dual, timeout=DEFAULT_TIMEOUT, source_address=None,
+        )
         self.assertEqual(conn, mock_socket.return_value)
 
     @mock.patch('proxy.common.utils.new_socket_connection')

tests/http/test_http_proxy.py

@@ -60,6 +60,7 @@ class TestHttpProxyPlugin(unittest.TestCase):
         self.plugin.return_value.read_from_descriptors.return_value = False
         self.plugin.return_value.before_upstream_connection.side_effect = lambda r: r
         self.plugin.return_value.handle_client_request.side_effect = lambda r: r
+        self.plugin.return_value.resolve_dns.return_value = None, None
 
         self._conn.recv.return_value = build_http_request(
             b'GET', b'http://upstream.host/not-found.html',

tests/http/test_http_proxy_tls_interception.py

@@ -133,6 +133,7 @@ class TestHttpProxyTlsInterception(unittest.TestCase):
         self.proxy_plugin.return_value.read_from_descriptors.return_value = False
         self.proxy_plugin.return_value.before_upstream_connection.side_effect = lambda r: r
         self.proxy_plugin.return_value.handle_client_request.side_effect = lambda r: r
+        self.proxy_plugin.return_value.resolve_dns.return_value = None, None
 
         self.mock_selector.return_value.select.side_effect = [
             [(