-
-
-
- $!nav if this.title!="docs" else ""!$
-
- mitmproxy 0.9 docs
-
-
-
-
-
- $!title if this.title!="docs" else ""!$
- $!body!$
-
-
-
+
+@!copyright!@
+
+
+ + + +
diff --git a/doc-src/_transparent.graffle/data.plist b/doc-src/_transparent.graffle/data.plist
new file mode 100644
index 000000000..722b4a44a
--- /dev/null
+++ b/doc-src/_transparent.graffle/data.plist
@@ -0,0 +1,771 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ $!body!$
+ @!this.title!@
++ + + +
-
-You will be prompted for super-user access, which you must allow. Next, enter
-the ProxyDroid settings, and change the proxy settings to point to your
-mitmproxy instance. When you're done, it should look something like this:
-
-
-
-In this case, our mitmproxy instance is at the host __maru.otago.ac.nz__,
-running on port __8080__.
-
-When you start mitmproxy, make sure that the upstream certificate option is set
-(use the _--upstream-cert_ command-line option, or enable it interactively
-using the _o_ shortcut):
-
--mitmproxy --upstream-cert -- diff --git a/doc-src/explicit.png b/doc-src/explicit.png new file mode 100644 index 000000000..c9ba26a73 Binary files /dev/null and b/doc-src/explicit.png differ diff --git a/doc-src/explicit_https.png b/doc-src/explicit_https.png new file mode 100644 index 000000000..1f1ca0231 Binary files /dev/null and b/doc-src/explicit_https.png differ diff --git a/doc-src/features/anticache.html b/doc-src/features/anticache.html new file mode 100644 index 000000000..f42903e8a --- /dev/null +++ b/doc-src/features/anticache.html @@ -0,0 +1,18 @@ + +When the __anticache__ option is passed to mitmproxy, it removes headers +(__if-none-match__ and __if-modified-since__) that might elicit a +304-not-modified response from the server. This is useful when you want to make +sure you capture an HTTP exchange in its totality. It's also often used during +[client replay](@!urlTo("clientreplay.html")!@), when you want to make sure the +server responds with complete data. + +
| command-line | --anticache | +
|---|---|
| mitmproxy shortcut | o then a | +
| command-line | -c path | +
|---|---|
| mitmproxy shortcut | c | +
| @!i[0]!@ | @!i[1]!@ |
| command-line | +
+
|
+
|---|
| command-line | +
+
|
+
|---|---|
| mitmproxy shortcut | R | +
| command-line | -P http[s]://hostname[:port] | +
|---|---|
| mitmproxy shortcut | P | +
| command-line | ++ --setheader PATTERN + | +
|---|---|
| mitmproxy shortcut | H | +
| command-line | +
+
|
+
|---|---|
| mitmproxy shortcut | t | +
| command-line | +
+
|
+
|---|---|
| mitmproxy shortcut | u | +
| command-line | --no-upstream-cert | +
|---|
+
+
+Configuring the client to use mitmproxy as an explicit proxy is the simplest
+and most reliable way to intercept traffic. The proxy protocol is codified in
+the [HTTP RFC](http://www.ietf.org/rfc/rfc2068.txt), so the behaviour of both
+the client and the server is well defined, and usually reliable. In the
+simplest possible interaction with mitmproxy, a client connects directly to the
+proxy, and makes a request that looks like this:
+
+Explicit HTTP
+GET http://example.com/index.html HTTP/1.1+ +This is a proxy GET request - an extended form of the vanilla HTTP GET request +that includes a schema and host specification, and it includes all the +information mitmproxy needs to proceed. + +
+
+| 1 | + +The client connects to the proxy and makes a request. | + +
| 2 | + +Mitmproxy connects to the upstream server and simply forwards + the request on. | + +
+
+
+The process for an explicitly proxied HTTPS connection is quite different. The
+client connects to the proxy and makes a request that looks like this:
+
+Explicit HTTPS
+CONNECT example.com:443 HTTP/1.1+ +A conventional proxy can neither view nor manipulate an SSL-encrypted data +stream, so a CONNECT request simply asks the proxy to open a pipe between the +client and server. The proxy here is just a facilitator - it blindly forwards +data in both directions without knowing anything about the contents. The +negotiation of the SSL connection happens over this pipe, and the subsequent +flow of requests and responses are completely opaque to the proxy. + +## The MITM in mitmproxy + +This is where mitmproxy's fundamental trick comes in to play. The MITM in its +name stands for Man-In-The-Middle - a reference to the process we use to +intercept and interfere with these theoretially opaque data streams. The basic +idea is to pretend to be the server to the client, and pretend to be the client +to the server. The tricky part is that the Certificate Authority system is +designed to prevent exactly this attack, by allowing a trusted third-party to +cryptographically sign a server's SSL certificates to verify that the certs are +legit. If this signature is from a non-trusted party, a secure client will +simply drop the connection and refuse to proceed. Despite the many shortcomings +of the CA system as it exists today, this is usually fatal to attempts to MITM +an SSL connection for analysis. + +Our answer to this conundrum is to become a trusted Certificate Authority +ourselves. Mitmproxy includes a full CA implementation that generates +interception certificates on the fly. To get the client to trust these +certificates, we register mitmproxy as a CA with the device manually. + +## Complication 1: What's the remote hostname? + +To proceed with this plan, we need to know the domain name to use in the +interception certificate - the client will verify that the certificate is for +the domain it's connecting to, and abort if this is not the case. At first +blush, it seems that the CONNECT request above gives us all we need - in this +example, both of these values are "example.com". But what if the client had +initiated the connection as follows: + +
CONNECT 10.1.1.1:443 HTTP/1.1+ +Using the IP address is perfectly legitimate because it gives us enough +information to initiate the pipe, even though it doesn't reveal the remote +hostname. + +Mitmproxy has a cunning mechanism that smooths this over - upstream certificate +sniffing. As soon as we see the CONNECT request, we pause the client part of +the conversation, and initiate a simultaneous connection to the server. We +complete the SSL handshake with the server, and inspect the certificates it +used. Now, we use the Common Name in the upstream SSL certificates to generate +the dummy certificate for the client. Voila, we have the correct hostname to +present to the client, even if it was never specified. + + +## Complication 2: Subject Alternate Name + +Enter the next complication. Sometimes, the certificate Common Name is not, in +fact, the hostname that the client is connecting to. This is because of the +optional Subject Alternate Name field in the SSL certificate that allows an +arbitrary number of alternate domains to be specified. If the expected domain +matches any of these, the client wil proceed, even though the domain doesn't +match the certificate Common Name. The answer here is simple: when extract the +CN from the upstream cert, we also extract the SANs, and add them to the +generated dummy certificate. + + +## Complication 3: Server Name Indication + +One of the big limitations of conventional SSL is that each certificate +requires its own IP address. This means that you couldn't do virtual hosting +where multiple domains with independent certificates share the same IP address. +In a world with a rapidly shrinking IPv4 address pool this is a problem, and we +have a solution in the form of the Server Name Indication extension to the SSL +and TLS protocols. This lets the client specify the remote server name at the +start of the SSL handshake, which then lets the server select the right +certificate to complete the process. + +SNI breaks our upstream certificate sniffing process, because when we connect +without using SNI, we get served a default certificate that may have nothing to +do with the certificate expected by the client. The solution is another tricky +complication to the client connection process. After the client connects, we +allow the SSL handshake to continue until just _after_ the SNI value has been +passed to us. Now we can pause the conversation, and initiate an upstream +connection using the correct SNI value, which then serves us the correct +upstream certificate, from which we can extract the expected CN and SANs. + + +## Putting it all together + +Lets put all of this together into the complete explicitly proxied HTTPS flow. + +
+
+| 1 | +The client makes a connection to mitmproxy, and issues an HTTP + CONNECT request. | +
| 2 | + +Mitmproxy responds with a 200 Connection Established, as if it + has set up the CONNECT pipe. | +
| 3 | + +The client believes it's talking to the remote server, and + initiates the SSL connection. It uses SNI to indicate the hostname + it is connecting to. | +
| 4 | + +Mitmproxy connects to the server, and establishes an SSL + connection using the SNI hostname indicated by the client. | + +
| 5 | + +The server responds with the matching SSL certificate, which + contains the CN and SAN values needed to generate the interception + certificate. | +
| 6 | + +Mitmproxy generates the interception cert, and continues the + client SSL handshake paused in step 3. | +
| 7 | + +The client sends the request over the established SSL + connection. | +
| 7 | + +Mitmproxy passes the request on to the server over the SSL + connection initiated in step 4. | +
+
+
+When a transparent proxy is used, the HTTP/S connection is redirected into a
+proxy at the network layer, without any client configuration being required.
+This makes transparent proxying ideal for those situations where you can't
+change client behaviour - proxy-oblivious Android applications being a common
+example.
+
+To achieve this, we need to introduce two extra components. The first new
+component is a router that transparently redirects the TCP connection to the
+proxy. Once the client has initiated the connection, it makes a vanilla HTTP
+request, which might look something like this:
+
+Transparent HTTP
+GET /index.html HTTP/1.1+ +Note that this request differs from the explicit proxy variation, in that it +omits the scheme and hostname. How, then, do we know which upstream host to +forward the request to? The routing mechanism that has performed the +redirection keeps track of the original destination. Each different routing +mechanism has its own ideosyncratic way of exposing this data, so this +introduces the second component required for working transparent proxying: a +host module that knows how to retrieve the original destination address from +the router. Once we have this information, the process is fairly +straight-forward. + +
+
+
+| 1 | +The client makes a connection to the server. | +
| 2 | + +The router redirects the connection to mitmproxy, which is + typically listening on a local port of the same host. Mitmproxy + then consults the routing mechanism to establish what the original + destination was. | +
| 3 | + +Now, we simply read the client's request... | +
| 4 | + +... and forward it upstream. | + +
+
+
+The process for transparently proxying an HTTPS request is a merger of the
+methods we've outlined for transparently proxying HTTP, and explicitly proxying
+HTTPS. We use the routing mechanism to establish the upstream server address,
+and then proceed as for explit HTTPS connections to establish the CN and SANs,
+and cope with SNI.
+
+Transparent HTTPS
+
+
+
+| 1 | +The client makes a connection to the server. | +
| 2 | + +The router redirects the connection to mitmproxy, which is + typically listening on a local port of the same host. Mitmproxy + then consults the routing mechanism to establish what the original + destination was. | +
| 3 | + +The client believes it's talking to the remote server, and + initiates the SSL connection. It uses SNI to indicate the hostname + it is connecting to. | +
| 4 | + +Mitmproxy connects to the server, and establishes an SSL + connection using the SNI hostname indicated by the client. | + +
| 5 | + +The server responds with the matching SSL certificate, which + contains the CN and SAN values needed to generate the interception + certificate. | +
| 6 | + +Mitmproxy generates the interception cert, and continues the + client SSL handshake paused in step 3. | +
| 7 | + +The client sends the request over the established SSL + connection. | +
| 7 | + +Mitmproxy passes the request on to the server over the SSL + connection initiated in step 4. | +
-
-
- Introduction -
- Installation -
- mitmproxy -
- mitmdump -
- Features - - -
- Scripting mitmproxy
-
-
-
- Inline Scripts -
- libmproxy -
- - Setting up SSL interception
-
-
-
- Firefox -
- OSX -
- Windows 7 -
- iPhone/iPad -
- Android -
- - Setting up Transparent Proxying - - -
- Tutorials - - -
- FAQ -
- Administrivia -
@!this.title!@
") - this.layout = countershape.Layout("_websitelayout.html") -else: - ns.title = countershape.template.Template(None, "@!this.title!@
") - this.layout = countershape.Layout("_layout.html") - +this.layout = countershape.Layout("_layout.html") +this.titlePrefix = "mitmproxy 0.9 - " this.markup = markup.Markdown() + ns.docMaintainer = "Aldo Cortesi" ns.docMaintainerEmail = "aldo@corte.si" -ns.copyright = u"\u00a9 mitmproxy project, 2012" - -ns.index = countershape.widgets.SiblingPageIndex('/index.html', divclass="pageindex") +ns.copyright = u"\u00a9 mitmproxy project, 2013" def mpath(p): p = os.path.join(MITMPROXY_SRC, p) return os.path.expanduser(p) -ns.license = file(mpath("LICENSE")).read() ns.index_contents = file(mpath("README.mkd")).read() - - -top = os.path.abspath(os.getcwd()) def example(s): d = file(mpath(s)).read() extemp = """%s
"""
return extemp%(countershape.template.Syntax("py")(d), s)
-
-
ns.example = example
filt_help = []
@@ -64,25 +51,27 @@ filt_help.extend(
ns.filt_help = filt_help
+def nav(page, current, state):
+ if current.match(page, False):
+ pre = '(%s)
-sudo pip install mitmproxy +## Installing the latest release + +A single command will download and install the latest release of mitmproxy, +along with all its dependencies: + ++pip install mitmproxy-## From Source +## Installing from source -- When installing from source, you will need to install the -[dependencies](@!urlTo("intro.html")!@) by hand. -- Then run the following command from the base of the source distribution: +When installing from source, the easiest method is still to use pip. In this +case run: --sudo python setup.py install ++pip install /path/to/source+Note that if you're installing current git master, you will also have to +install the current git master of [netlib](http://github.com/cortesi/netlib) by +hand. ## OSX +- If you're running a Python interpreter installed with homebrew (or similar), +you may have to install some dependencies by hand. - Make sure that XCode is installed from the App Store, and that the command-line tools have been downloaded (XCode/Preferences/Downloads). -- Install __pip__ using the following command: - --sudo easy_install pip -- -- Now use __pip__ to set up the dependencies and do the install: - --sudo pip install mitmproxy -- -This procedure may vary if, for instance, you've installed Python from an -external source like [homebrew](http://mxcl.github.com/homebrew/). In that -case, the easiest way to proceed is to first install __easy_install__, then -continue as above. +- Now use __pip__ to do the installation, as above. There are a few bits of customization you might want to do to make mitmproxy -comfortable to use. At the moment, mitmproxy's color scheme is optimized for a -dark background terminal, so you probably want to change the default. You can -use the OSX open program to create a simple and effective -~/.mailcap file to view HTTP bodies: +comfortable to use on OSX. The default color scheme is optimized for a dark +background terminal, but you can select a palette for a light terminal +background with the --palette option. You can use the OSX open program +to create a simple and effective ~/.mailcap file to view request and +response bodies: -+application/*; /usr/bin/open -Wn %s audio/*; /usr/bin/open -Wn %s image/*; /usr/bin/open -Wn %s diff --git a/doc-src/intro.html b/doc-src/intro.html deleted file mode 100644 index 79687ec61..000000000 --- a/doc-src/intro.html +++ /dev/null @@ -1,4 +0,0 @@ - -@!index_contents!@ - - diff --git a/doc-src/scripting/inlinescripts.html b/doc-src/scripting/inlinescripts.html index 860ad9b64..7abf743c6 100644 --- a/doc-src/scripting/inlinescripts.html +++ b/doc-src/scripting/inlinescripts.html @@ -1,4 +1,3 @@ - __mitmproxy__ has a powerful scripting API that allows you to modify flows on-the-fly or rewrite previously saved flows locally. @@ -22,13 +21,13 @@ response itself. Called once on startup, before any other events. -###clientconnect(ScriptContext, ClientConnect) +### clientconnect(ScriptContext, ClientConnect) Called when a client initiates a connection to the proxy. Note that a connection can correspond to multiple HTTP requests. -###request(ScriptContext, Flow) +### request(ScriptContext, Flow) Called when a client request has been received. The __Flow__ object is guaranteed to have a non-None __request__ attribute. diff --git a/doc-src/transparent.png b/doc-src/transparent.png new file mode 100644 index 000000000..3994d6815 Binary files /dev/null and b/doc-src/transparent.png differ diff --git a/doc-src/transparent/index.py b/doc-src/transparent/index.py index fa2a67da9..d277d7080 100644 --- a/doc-src/transparent/index.py +++ b/doc-src/transparent/index.py @@ -3,5 +3,4 @@ from countershape import Page pages = [ Page("linux.html", "Linux"), Page("osx.html", "OSX"), - Page("openbsd.html", "OpenBSD"), ] diff --git a/doc-src/transparent/openbsd.html b/doc-src/transparent/openbsd.html deleted file mode 100644 index e69de29bb..000000000 diff --git a/doc-src/transparent_https.png b/doc-src/transparent_https.png new file mode 100644 index 000000000..559cddd21 Binary files /dev/null and b/doc-src/transparent_https.png differ diff --git a/doc-src/upstreamcerts.html b/doc-src/upstreamcerts.html deleted file mode 100644 index 32a13a9a8..000000000 --- a/doc-src/upstreamcerts.html +++ /dev/null @@ -1,15 +0,0 @@ -- command-line: _--upstream-cert_ -- mitmproxy shortcut: _o_, then _u_ - -Normally, mitmproxy uses the target domain specified in a client's proxy -request to generate an interception certificate. When __upstream-cert__ mode is -activated a different procedure is followed: a connection is made to the -specified remote server to retrieve its __Common Name__ and __Subject -Alternative Names__. This feature is especially useful when the client -specifies an IP address rather than a host name in the proxy request. If this -is the case, we can only generate a certificate if we can establish the __CN__ -and __SANs__ from the upstream server. - -Note that __upstream-cert__ mode does not work when the remote server relies on -[Server Name Indication](http://en.wikipedia.org/wiki/Server_Name_Indication). -Luckily, SNI is still not very widely used. diff --git a/examples/iframe_injector b/examples/iframe_injector new file mode 100755 index 000000000..6dd28674c --- /dev/null +++ b/examples/iframe_injector @@ -0,0 +1,50 @@ +#!/usr/bin/env python +""" + Zap encoding in requests and inject iframe after body tag in html responses. + Usage: + iframe_injector http://someurl/somefile.html +""" +from libmproxy import controller, proxy +import os +import sys + + +class InjectingMaster(controller.Master): + def __init__(self, server, iframe_url): + controller.Master.__init__(self, server) + self._iframe_url = iframe_url + + def run(self): + try: + return controller.Master.run(self) + except KeyboardInterrupt: + self.shutdown() + + def handle_request(self, msg): + if 'Accept-Encoding' in msg.headers: + msg.headers["Accept-Encoding"] = 'none' + msg._ack() + + def handle_response(self, msg): + if msg.content: + c = msg.replace('', '' % self._iframe_url) + if c > 0: + print 'Iframe injected!' + msg._ack() + + +def main(argv): + if len(argv) != 2: + print "Usage: %s IFRAME_URL" % argv[0] + sys.exit(1) + iframe_url = argv[1] + config = proxy.ProxyConfig( + cacert = os.path.expanduser("~/.mitmproxy/mitmproxy-ca.pem") + ) + server = proxy.ProxyServer(config, 8080) + print 'Starting proxy...' + m = InjectingMaster(server, iframe_url) + m.run() + +if __name__ == '__main__': + main(sys.argv) diff --git a/libmproxy/authentication.py b/libmproxy/authentication.py index e5383f5ac..500ead6b8 100644 --- a/libmproxy/authentication.py +++ b/libmproxy/authentication.py @@ -2,35 +2,49 @@ import binascii import contrib.md5crypt as md5crypt class NullProxyAuth(): - """ No proxy auth at all (returns empty challange headers) """ - def __init__(self, password_manager=None): + """ + No proxy auth at all (returns empty challange headers) + """ + def __init__(self, password_manager): self.password_manager = password_manager self.username = "" - def authenticate(self, auth_value): - """ Tests that the specified user is allowed to use the proxy (stub) """ + def clean(self, headers): + """ + Clean up authentication headers, so they're not passed upstream. + """ + pass + + def authenticate(self, headers): + """ + Tests that the user is allowed to use the proxy + """ return True def auth_challenge_headers(self): - """ Returns a dictionary containing the headers require to challenge the user """ + """ + Returns a dictionary containing the headers require to challenge the user + """ return {} - def get_username(self): - return self.username - class BasicProxyAuth(NullProxyAuth): - - def __init__(self, password_manager, realm="mitmproxy"): + CHALLENGE_HEADER = 'Proxy-Authenticate' + AUTH_HEADER = 'Proxy-Authorization' + def __init__(self, password_manager, realm): NullProxyAuth.__init__(self, password_manager) - self.realm = "mitmproxy" + self.realm = realm - def authenticate(self, auth_value): - if (not auth_value) or (not auth_value[0]): - return False; + def clean(self, headers): + del headers[self.AUTH_HEADER] + + def authenticate(self, headers): + auth_value = headers.get(self.AUTH_HEADER, []) + if not auth_value: + return False try: - scheme, username, password = self.parse_authorization_header(auth_value[0]) - except: + scheme, username, password = self.parse_auth_value(auth_value[0]) + except ValueError: return False if scheme.lower()!='basic': return False @@ -40,14 +54,26 @@ class BasicProxyAuth(NullProxyAuth): return True def auth_challenge_headers(self): - return {'Proxy-Authenticate':'Basic realm="%s"'%self.realm} + return {self.CHALLENGE_HEADER:'Basic realm="%s"'%self.realm} - def parse_authorization_header(self, auth_value): + def unparse_auth_value(self, scheme, username, password): + v = binascii.b2a_base64(username + ":" + password) + return scheme + " " + v + + def parse_auth_value(self, auth_value): words = auth_value.split() + if len(words) != 2: + raise ValueError("Invalid basic auth credential.") scheme = words[0] - user = binascii.a2b_base64(words[1]) - username, password = user.split(':') - return scheme, username, password + try: + user = binascii.a2b_base64(words[1]) + except binascii.Error: + raise ValueError("Invalid basic auth credential: user:password pair not valid base64: %s"%words[1]) + parts = user.split(':') + if len(parts) != 2: + raise ValueError("Invalid basic auth credential: decoded user:password pair not valid: %s"%user) + return scheme, parts[0], parts[1] + class PasswordManager(): def __init__(self): @@ -56,8 +82,8 @@ class PasswordManager(): def test(self, username, password_token): return False -class PermissivePasswordManager(PasswordManager): +class PermissivePasswordManager(PasswordManager): def __init__(self): PasswordManager.__init__(self) @@ -66,16 +92,17 @@ class PermissivePasswordManager(PasswordManager): return True return False -class HtpasswdPasswordManager(PasswordManager): - """ Read usernames and passwords from a file created by Apache htpasswd""" +class HtpasswdPasswordManager(PasswordManager): + """ + Read usernames and passwords from a file created by Apache htpasswd + """ def __init__(self, filehandle): PasswordManager.__init__(self) entries = (line.strip().split(':') for line in filehandle) valid_entries = (entry for entry in entries if len(entry)==2) self.usernames = {username:token for username,token in valid_entries} - def test(self, username, password_token): if username not in self.usernames: return False @@ -84,8 +111,8 @@ class HtpasswdPasswordManager(PasswordManager): expected = md5crypt.md5crypt(password_token, salt, '$'+magic+'$') return expected==full_token -class SingleUserPasswordManager(PasswordManager): +class SingleUserPasswordManager(PasswordManager): def __init__(self, username, password): PasswordManager.__init__(self) self.username = username diff --git a/libmproxy/cmdline.py b/libmproxy/cmdline.py index db1ebf0d3..de70bea80 100644 --- a/libmproxy/cmdline.py +++ b/libmproxy/cmdline.py @@ -248,11 +248,6 @@ def common_options(parser): help="Byte size limit of HTTP request and response bodies."\ " Understands k/m/g suffixes, i.e. 3m for 3 megabytes." ) - parser.add_argument( - "--cert-wait-time", type=float, - action="store", dest="cert_wait_time", default=0, - help="Wait for specified number of seconds after a new cert is generated. This can smooth over small discrepancies between the client and server times." - ) parser.add_argument( "--no-upstream-cert", default=False, @@ -338,46 +333,29 @@ def common_options(parser): group = parser.add_argument_group( "Proxy Authentication", """ - Specification of which users are allowed to access the proxy and the method used for authenticating them. - If authscheme is specified, one must specify a list of authorized users and their passwords. - In case that authscheme is not specified, or set to None, any list of authorized users will be ignored. - """.strip() + Specify which users are allowed to access the proxy and the method + used for authenticating them. These options are ignored if the + proxy is in transparent or reverse proxy mode. + """ ) - - group.add_argument( - "--authscheme", type=str, - action="store", dest="authscheme", default=None, choices=["none", "basic"], - help=""" - Specify the scheme used by the proxy to identify users. - If not none, requires the specification of a list of authorized users. - This option is ignored if the proxy is in transparent or reverse mode. - """.strip() - - ) - user_specification_group = group.add_mutually_exclusive_group() - - user_specification_group.add_argument( "--nonanonymous", action="store_true", dest="auth_nonanonymous", - help="Allow access to any user as long as a username is specified. Ignores the provided password." + help="Allow access to any user long as a credentials are specified." ) user_specification_group.add_argument( "--singleuser", action="store", dest="auth_singleuser", type=str, - help="Allows access to a single user as specified by the option value. Specify a username and password in the form username:password." + metavar="USER", + help="Allows access to a a single user, specified in the form username:password." ) - user_specification_group.add_argument( "--htpasswd", action="store", dest="auth_htpasswd", type=argparse.FileType('r'), + metavar="PATH", help="Allow access to users specified in an Apache htpasswd file." ) - - - - proxy.certificate_option_group(parser) diff --git a/libmproxy/flow.py b/libmproxy/flow.py index 9a6b5527d..2c4c55133 100755 --- a/libmproxy/flow.py +++ b/libmproxy/flow.py @@ -1387,6 +1387,8 @@ class FlowMaster(controller.Master): self.kill_nonreplay = kill def stop_server_playback(self): + if self.server_playback.exit: + self.shutdown() self.server_playback = None def do_server_playback(self, flow): @@ -1420,10 +1422,6 @@ class FlowMaster(controller.Master): self.shutdown() self.client_playback.tick(self) - if self.server_playback: - if self.server_playback.exit and self.server_playback.count() == 0: - self.shutdown() - return controller.Master.tick(self, q) def duplicate_flow(self, f): diff --git a/libmproxy/platform/osx.py b/libmproxy/platform/osx.py index a66c03ed8..dda5d9afc 100644 --- a/libmproxy/platform/osx.py +++ b/libmproxy/platform/osx.py @@ -1,103 +1,23 @@ -import socket, ctypes - -# Python socket module does not have this constant -DIOCNATLOOK = 23 -PFDEV = "/dev/pf" - - -class PF_STATE_XPORT(ctypes.Union): - """ - union pf_state_xport { - u_int16_t port; - u_int16_t call_id; - u_int32_t spi; - }; - """ - _fields_ = [ - ("port", ctypes.c_uint), - ("call_id", ctypes.c_uint), - ("spi", ctypes.c_ulong), - ] - - -class PF_ADDR(ctypes.Union): - """ - struct pf_addr { - union { - struct in_addr v4; - struct in6_addr v6; - u_int8_t addr8[16]; - u_int16_t addr16[8]; - u_int32_t addr32[4]; - } pfa; - } - """ - _fields_ = [ - ("addr8", ctypes.c_byte * 2), - ("addr16", ctypes.c_byte * 4), - ("addr32", ctypes.c_byte * 8), - ] - - -class PFIOC_NATLOOK(ctypes.Structure): - """ - struct pfioc_natlook { - struct pf_addr saddr; - struct pf_addr daddr; - struct pf_addr rsaddr; - struct pf_addr rdaddr; - #ifndef NO_APPLE_EXTENSIONS - union pf_state_xport sxport; - union pf_state_xport dxport; - union pf_state_xport rsxport; - union pf_state_xport rdxport; - sa_family_t af; - u_int8_t proto; - u_int8_t proto_variant; - u_int8_t direction; - #else - u_int16_t sport; - u_int16_t dport; - u_int16_t rsport; - u_int16_t rdport; - sa_family_t af; - u_int8_t proto; - u_int8_t direction; - #endif - }; - """ - _fields_ = [ - ("saddr", PF_ADDR), - ("daddr", PF_ADDR), - ("rsaddr", PF_ADDR), - ("rdaddr", PF_ADDR), - - ("sxport", PF_STATE_XPORT), - ("dxport", PF_STATE_XPORT), - ("rsxport", PF_STATE_XPORT), - ("rdxport", PF_STATE_XPORT), - ("af", ctypes.c_uint), - ("proto", ctypes.c_ushort), - ("proto_variant", ctypes.c_ushort), - ("direction", ctypes.c_ushort), - ] +import subprocess +import pf +""" + Doing this the "right" way by using DIOCNATLOOK on the pf device turns out + to be a pain. Apple has made a number of modifications to the data + structures returned, and compiling userspace tools to test and work with + this turns out to be a pain in the ass. Parsing pfctl output is short, + simple, and works. +""" class Resolver: + STATECMD = ("sudo", "-n", "/sbin/pfctl", "-s", "state") def __init__(self): - self.pfdev = open(PFDEV, "r") + pass def original_addr(self, csock): - """ - The following sttruct defintions are plucked from the current XNU source, found here: - - http://www.opensource.apple.com/source/xnu/xnu-1699.26.8/bsd/net/pfvar.h - - - union pf_state_xport { - u_int16_t port; - u_int16_t call_id; - u_int32_t spi; - }; - """ - pass + peer = csock.getpeername() + try: + stxt = subprocess.check_output(self.STATECMD, stderr=subprocess.STDOUT) + except subprocess.CalledProcessError: + return None + return pf.lookup(peer[0], peer[1], stxt) diff --git a/libmproxy/platform/pf.py b/libmproxy/platform/pf.py new file mode 100644 index 000000000..062d33113 --- /dev/null +++ b/libmproxy/platform/pf.py @@ -0,0 +1,16 @@ + +def lookup(address, port, s): + """ + Parse the pfctl state output s, to look up the destination host + matching the client (address, port). + + Returns an (address, port) tuple, or None. + """ + spec = "%s:%s"%(address, port) + for i in s.split("\n"): + if "ESTABLISHED:ESTABLISHED" in i and spec in i: + s = i.split() + if len(s) > 4: + s = s[4].split(":") + if len(s) == 2: + return s[0], int(s[1]) diff --git a/libmproxy/proxy.py b/libmproxy/proxy.py index 4c57aeb0c..cf006f608 100755 --- a/libmproxy/proxy.py +++ b/libmproxy/proxy.py @@ -38,19 +38,19 @@ class Log(controller.Msg): class ProxyConfig: - def __init__(self, certfile = None, cacert = None, clientcerts = None, cert_wait_time=0, no_upstream_cert=False, body_size_limit = None, reverse_proxy=None, transparent_proxy=None, certdir = None, authenticator=None): + def __init__(self, certfile = None, cacert = None, clientcerts = None, no_upstream_cert=False, body_size_limit = None, reverse_proxy=None, transparent_proxy=None, certdir = None, authenticator=None): assert not (reverse_proxy and transparent_proxy) self.certfile = certfile self.cacert = cacert self.clientcerts = clientcerts - self.certdir = certdir - self.cert_wait_time = cert_wait_time self.no_upstream_cert = no_upstream_cert self.body_size_limit = body_size_limit self.reverse_proxy = reverse_proxy self.transparent_proxy = transparent_proxy self.authenticator = authenticator + self.certstore = certutils.CertStore(certdir) + class RequestReplayThread(threading.Thread): def __init__(self, config, flow, masterq): self.config, self.flow, self.masterq = config, flow, masterq @@ -247,8 +247,7 @@ class ProxyHandler(tcp.BaseHandler): raise ProxyError(502, "Unable to get remote cert: %s"%str(v)) sans = cert.altnames host = cert.cn.decode("utf8").encode("idna") - ret = certutils.dummy_cert(self.config.certdir, self.config.cacert, host, sans) - time.sleep(self.config.cert_wait_time) + ret = self.config.certstore.get_cert(host, sans, self.config.cacert) if not ret: raise ProxyError(502, "mitmproxy: Unable to generate dummy cert.") return ret @@ -270,7 +269,10 @@ class ProxyHandler(tcp.BaseHandler): def read_request(self, client_conn): self.rfile.reset_timestamps() if self.config.transparent_proxy: - host, port = self.config.transparent_proxy["resolver"].original_addr(self.connection) + orig = self.config.transparent_proxy["resolver"].original_addr(self.connection) + if not orig: + raise ProxyError(502, "Transparent mode failure: could not resolve original destination.") + host, port = orig if not self.ssl_established and (port in self.config.transparent_proxy["sslports"]): scheme = "https" certfile = self.find_cert(host, port, None) @@ -311,7 +313,7 @@ class ProxyHandler(tcp.BaseHandler): line = self.get_line(self.rfile) if line == "": return None - if line.startswith("CONNECT"): + if http.parse_init_connect(line): r = http.parse_init_connect(line) if not r: raise ProxyError(400, "Bad HTTP request line: %s"%repr(line)) @@ -332,14 +334,15 @@ class ProxyHandler(tcp.BaseHandler): raise ProxyError(400, str(v)) self.proxy_connect_state = (host, port, httpversion) line = self.rfile.readline(line) + if self.proxy_connect_state: - host, port, httpversion = self.proxy_connect_state r = http.parse_init_http(line) if not r: raise ProxyError(400, "Bad HTTP request line: %s"%repr(line)) method, path, httpversion = r headers = self.read_headers(authenticate=False) + host, port, _ = self.proxy_connect_state content = http.read_http_body_request( self.rfile, self.wfile, headers, httpversion, self.config.body_size_limit ) @@ -348,7 +351,7 @@ class ProxyHandler(tcp.BaseHandler): r = http.parse_init_proxy(line) if not r: raise ProxyError(400, "Bad HTTP request line: %s"%repr(line)) - method, scheme, host, port, path, httpversion = http.parse_init_proxy(line) + method, scheme, host, port, path, httpversion = r headers = self.read_headers(authenticate=True) content = http.read_http_body_request( self.rfile, self.wfile, headers, httpversion, self.config.body_size_limit @@ -359,8 +362,15 @@ class ProxyHandler(tcp.BaseHandler): headers = http.read_headers(self.rfile) if headers is None: raise ProxyError(400, "Invalid headers") - if authenticate and self.config.authenticator and not self.config.authenticator.authenticate(headers.get('Proxy-Authorization', [])): - raise ProxyError(407, "Proxy Authentication Required", self.config.authenticator.auth_challenge_headers()) + if authenticate and self.config.authenticator: + if self.config.authenticator.authenticate(headers): + self.config.authenticator.clean(headers) + else: + raise ProxyError( + 407, + "Proxy Authentication Required", + self.config.authenticator.auth_challenge_headers() + ) return headers def send_response(self, response): @@ -405,13 +415,6 @@ class ProxyServer(tcp.TCPServer): except socket.error, v: raise ProxyServerError('Error starting proxy server: ' + v.strerror) self.masterq = None - if config.certdir: - self.certdir = config.certdir - self.remove_certdir = False - else: - self.certdir = tempfile.mkdtemp(prefix="mitmproxy") - config.certdir = self.certdir - self.remove_certdir = True self.apps = AppRegistry() def start_slave(self, klass, masterq): @@ -430,11 +433,7 @@ class ProxyServer(tcp.TCPServer): pass def handle_shutdown(self): - try: - if self.remove_certdir: - shutil.rmtree(self.certdir) - except OSError: - pass + self.config.certstore.cleanup() class AppRegistry: @@ -537,30 +536,24 @@ def process_proxy_options(parser, options): if not os.path.exists(options.certdir) or not os.path.isdir(options.certdir): parser.error("Dummy cert directory does not exist or is not a directory: %s"%options.certdir) - if options.authscheme and (options.authscheme!='none'): - if not (options.auth_nonanonymous or options.auth_singleuser or options.auth_htpasswd): - parser.error("Proxy authentication scheme is specified, but no allowed user list is given.") - if options.auth_singleuser and len(options.auth_singleuser.split(':'))!=2: - parser.error("Authorized user is not given in correct format username:password") - if options.auth_nonanonymous: - password_manager = authentication.PermissivePasswordManager() - elif options.auth_singleuser: + if (options.auth_nonanonymous or options.auth_singleuser or options.auth_htpasswd): + if options.auth_singleuser: + if len(options.auth_singleuser.split(':')) != 2: + parser.error("Please specify user in the format username:password") username, password = options.auth_singleuser.split(':') password_manager = authentication.SingleUserPasswordManager(username, password) + elif options.auth_nonanonymous: + password_manager = authentication.PermissivePasswordManager() elif options.auth_htpasswd: password_manager = authentication.HtpasswdPasswordManager(options.auth_htpasswd) - # in the meanwhile, basic auth is the only true authentication scheme we support - # so just use it - authenticator = authentication.BasicProxyAuth(password_manager) + authenticator = authentication.BasicProxyAuth(password_manager, "mitmproxy") else: authenticator = authentication.NullProxyAuth(None) - return ProxyConfig( certfile = options.cert, cacert = cacert, clientcerts = options.clientcerts, - cert_wait_time = options.cert_wait_time, body_size_limit = body_size_limit, no_upstream_cert = options.no_upstream_cert, reverse_proxy = rp, diff --git a/setup.py b/setup.py index 6fb25c017..9a5398be9 100644 --- a/setup.py +++ b/setup.py @@ -93,7 +93,4 @@ setup( "Topic :: Software Development :: Testing" ], install_requires=["netlib", "urwid>=1.1", "pyasn1>0.1.2", "pyopenssl>=0.12", "PIL", "lxml"], - dependency_links=[ - 'https://github.com/cortesi/netlib/tarball/master#egg=netlib-0.1' - ], ) diff --git a/test/data/pf01 b/test/data/pf01 new file mode 100644 index 000000000..3139a2895 --- /dev/null +++ b/test/data/pf01 @@ -0,0 +1,4 @@ +No ALTQ support in kernel +ALTQ related functions disabled +ALL tcp 127.0.0.1:8080 <- 5.5.5.6:80 <- 192.168.1.111:40001 FIN_WAIT_2:FIN_WAIT_2 +ALL tcp 127.0.0.1:8080 <- 5.5.5.5:80 <- 192.168.1.111:40000 ESTABLISHED:ESTABLISHED diff --git a/test/test_authentication.py b/test/test_authentication.py new file mode 100644 index 000000000..f7a5ecd34 --- /dev/null +++ b/test/test_authentication.py @@ -0,0 +1,58 @@ +import binascii +from libmproxy import authentication +from netlib import odict +import tutils + + +class TestNullProxyAuth: + def test_simple(self): + na = authentication.NullProxyAuth(authentication.PermissivePasswordManager()) + assert not na.auth_challenge_headers() + assert na.authenticate("foo") + na.clean({}) + + +class TestBasicProxyAuth: + def test_simple(self): + ba = authentication.BasicProxyAuth(authentication.PermissivePasswordManager(), "test") + h = odict.ODictCaseless() + assert ba.auth_challenge_headers() + assert not ba.authenticate(h) + + def test_parse_auth_value(self): + ba = authentication.BasicProxyAuth(authentication.PermissivePasswordManager(), "test") + vals = ("basic", "foo", "bar") + assert ba.parse_auth_value(ba.unparse_auth_value(*vals)) == vals + tutils.raises(ValueError, ba.parse_auth_value, "") + tutils.raises(ValueError, ba.parse_auth_value, "foo bar") + + v = "basic " + binascii.b2a_base64("foo") + tutils.raises(ValueError, ba.parse_auth_value, v) + + def test_authenticate_clean(self): + ba = authentication.BasicProxyAuth(authentication.PermissivePasswordManager(), "test") + + hdrs = odict.ODictCaseless() + vals = ("basic", "foo", "bar") + hdrs[ba.AUTH_HEADER] = [ba.unparse_auth_value(*vals)] + assert ba.authenticate(hdrs) + + ba.clean(hdrs) + assert not ba.AUTH_HEADER in hdrs + + + hdrs[ba.AUTH_HEADER] = [""] + assert not ba.authenticate(hdrs) + + hdrs[ba.AUTH_HEADER] = ["foo"] + assert not ba.authenticate(hdrs) + + vals = ("foo", "foo", "bar") + hdrs[ba.AUTH_HEADER] = [ba.unparse_auth_value(*vals)] + assert not ba.authenticate(hdrs) + + ba = authentication.BasicProxyAuth(authentication.PasswordManager(), "test") + vals = ("basic", "foo", "bar") + hdrs[ba.AUTH_HEADER] = [ba.unparse_auth_value(*vals)] + assert not ba.authenticate(hdrs) + diff --git a/test/test_dump.py b/test/test_dump.py index 0337bb33f..e1241e296 100644 --- a/test/test_dump.py +++ b/test/test_dump.py @@ -48,6 +48,15 @@ class TestDumpMaster: fw.add(t) f.close() + def test_error(self): + cs = StringIO() + o = dump.Options(verbosity=1) + m = dump.DumpMaster(None, o, None, outfile=cs) + f = tutils.tflow_err() + m.handle_request(f.request) + assert m.handle_error(f.error) + assert "error" in cs.getvalue() + def test_replay(self): cs = StringIO() diff --git a/test/test_flow.py b/test/test_flow.py index 277d2407a..2af702ce4 100644 --- a/test/test_flow.py +++ b/test/test_flow.py @@ -624,6 +624,7 @@ class TestFlowMaster: fm.handle_error(flow.Error(f.request, "error")) def test_server_playback(self): + controller.should_exit = False s = flow.State() f = tutils.tflow() @@ -641,8 +642,9 @@ class TestFlowMaster: r = tutils.tflow() r.request.content = "gibble" assert not fm.do_server_playback(r) - assert fm.do_server_playback(tutils.tflow()) + + fm.start_server_playback(pb, False, [], True, False) q = Queue.Queue() fm.tick(q) assert controller.should_exit @@ -650,6 +652,20 @@ class TestFlowMaster: fm.stop_server_playback() assert not fm.server_playback + def test_server_playback_kill(self): + s = flow.State() + f = tutils.tflow() + f.response = tutils.tresp(f.request) + pb = [f] + fm = flow.FlowMaster(None, s) + fm.refresh_server_playback = True + fm.start_server_playback(pb, True, [], False, False) + + f = tutils.tflow() + f.request.host = "nonexistent" + fm.process_new_request(f) + assert "killed" in f.error.msg + def test_stickycookie(self): s = flow.State() fm = flow.FlowMaster(None, s) diff --git a/test/test_platform_pf.py b/test/test_platform_pf.py new file mode 100644 index 000000000..a2e7c3c1d --- /dev/null +++ b/test/test_platform_pf.py @@ -0,0 +1,13 @@ +import tutils +from libmproxy.platform import pf + + +class TestLookup: + def test_simple(self): + p = tutils.test_data.path("data/pf01") + d = open(p).read() + assert pf.lookup("192.168.1.111", 40000, d) == ("5.5.5.5", 80) + assert not pf.lookup("192.168.1.112", 40000, d) + assert not pf.lookup("192.168.1.111", 40001, d) + + diff --git a/test/tools/getcert b/test/tools/getcert index 6447ecc71..8fabefb75 100755 --- a/test/tools/getcert +++ b/test/tools/getcert @@ -5,5 +5,3 @@ import socket, tempfile, ssl, subprocess addr = socket.gethostbyname(sys.argv[1]) print ssl.get_server_certificate((addr, 443)) - - diff --git a/test/tools/getcn b/test/tools/getcn deleted file mode 100755 index 212977c33..000000000 --- a/test/tools/getcn +++ /dev/null @@ -1,17 +0,0 @@ -#!/usr/bin/env python -import sys -sys.path.insert(0, "../../") -from libmproxy import certutils - -if len(sys.argv) > 2: - port = int(sys.argv[2]) -else: - pport = 443 - -cn, san = certutils.get_remote_cn(sys.argv[1], port) -print cn -if san: - for i in san: - print "\t", i - - diff --git a/test/tutils.py b/test/tutils.py index 0e661a4bc..5ce56cb45 100644 --- a/test/tutils.py +++ b/test/tutils.py @@ -4,7 +4,7 @@ from contextlib import contextmanager from libmproxy import proxy, flow, controller, utils from netlib import certutils import human_curl as hurl -import libpathod.test +import libpathod.test, libpathod.pathoc def treq(conn=None): if not conn: @@ -194,7 +194,6 @@ class ReverseProxTest(ProxTestBase): return r - @contextmanager def tmpdir(*args, **kwargs): orig_workdir = os.getcwd() @@ -245,4 +244,5 @@ def raises(exc, obj, *args, **kwargs): ) raise AssertionError("No exception raised.") + test_data = utils.Data(__name__)