Commit Graph

1159 Commits

Author SHA1 Message Date
Aldo Cortesi 8a9352b3f7 First draft conversion of server to PyOpenSSL. 2012-06-13 18:16:47 +12:00
Aldo Cortesi d032504b17 Fix an exception when replaying a flow with no response. 2012-06-10 17:10:43 +12:00
Aldo Cortesi d60fa9918b Localise client connection object manipulation.
This simplifies the call signature for a bunch of functions.
2012-06-10 16:49:59 +12:00
Aldo Cortesi 6ba5f0f35b Add HTTP version to response objects.
Another change in the serialization format.
2012-06-10 13:27:43 +12:00
Aldo Cortesi 52779d9db9 Refactoring of proxy.py
- Correctly pass HTTP request version on to upstream servers
- Adjust tests not to hang due to a pathod response with no content-length
2012-06-10 13:17:18 +12:00
Aldo Cortesi 55ddf853cd Add HTTP version to flow.Request
This is a serialization format change, that makes us incompatible with previous
versions.
2012-06-10 10:46:22 +12:00
Aldo Cortesi a3b47e0cb5 Consolidate HTTP major and minor versions into a single variable. 2012-06-10 10:31:04 +12:00
Aldo Cortesi 8254187bf3 Add proxy.should_connection_close, and strip out unused code. 2012-06-10 10:10:46 +12:00
Aldo Cortesi 18c1b44475 Reverse proxy testing. 2012-06-09 20:41:28 +12:00
Aldo Cortesi 05492baf8d Move from requests to human_curl.
It turns out that _none_ of the Python stdlib or anything that relies on it
supports CONNECT through a proxy. Beggars belief, but there you go.
2012-06-09 16:17:51 +12:00
Aldo Cortesi 22192d1a46 Nose mopup: docs, no cover pragmas, a few missing path specs. 2012-06-09 13:55:55 +12:00
Aldo Cortesi b7b357528c Port mitmproxy test suite entirely to nose. 2012-06-09 13:42:43 +12:00
Aldo Cortesi a63240a848 Move pathod service testing truss to nose. 2012-06-09 12:13:01 +12:00
Aldo Cortesi e78b48ab20 Start conversion to nose.
RIP pry.
2012-06-09 10:57:00 +12:00
Aldo Cortesi 7a312546f3 Shift mitmproxy test suite over to pathod.
This opens a whole brave new world of testing for mitmproxy.
2012-06-08 10:00:16 +12:00
Aldo Cortesi e9109812e1 Split parsing of intial line into separate protocols. 2012-06-03 06:04:57 -07:00
Aldo Cortesi 491f9bdcee Add unit tests for console/help.py 2012-06-03 01:11:07 -07:00
Aldo Cortesi ee2950cd19 Fix a crashing bug when replacing text in a flow with unicode bodies. 2012-05-25 18:10:31 -07:00
Aldo Cortesi 0a90a3eaba Refuse to replay a request with missing content. 2012-05-16 18:24:32 +12:00
Aldo Cortesi 0c2d894cea Add the ability to flag content as missing in a request or a response.
We'll use this in a number of situations. First, we'll soon have response
streaming that directly pipes responses to clients. These will be content-less
from mitmproxy's perspective. Second, we'll be growing new events that fire
after headers are received, but before content is read.
2012-05-16 15:42:58 +12:00
Aldo Cortesi 116fcfcf7a Internal error page for WSGI.
Also, 100% test coverage.
2012-04-27 15:56:42 +12:00
Aldo Cortesi c8d2b2594b Add a WSGI adapter that lets us serve a WSGI app out of mitmproxy.
This commit adds:
    - A WSGI App adapter for mitmproxy
    - An app registry in the proxy instance that lets us link WSGI apps with
    (hostname, port) combinations.
    - Fixes for a number of bugs discovered while creating this feature.
2012-04-24 14:52:29 +12:00
Aldo Cortesi 8c96264304 Serialized data version check. 2012-04-11 10:10:53 +12:00
Aldo Cortesi 4e2d19714c Add an "f" shortcut key to load full body contents. 2012-04-08 19:44:01 +12:00
Aldo Cortesi a4f7728fad XML/HTML pretty view tweaks. 2012-04-07 22:15:31 +12:00
Aldo Cortesi f1dc3f2ab2 Integrate lxml for pretty-printing HTML and XML.
Tackling the pretty-printing performance problem head-on, at the cost of a
major dependency.
2012-04-07 13:47:03 +12:00
Aldo Cortesi ab0e10e60f Serialize requestcount for ClientConnect objects. 2012-04-03 22:37:24 +12:00
Aldo Cortesi 61fab03b24 Add a details page, available from a flow view with the 'X' shortcut
At the moment, this shows the upstream SSL certificate details. More
fine-grained detail that doesn't fit in the flow view itself will be added.
2012-04-03 11:10:25 +12:00
Aldo Cortesi f526e5fa12 Minor unit test bump. 2012-04-03 09:52:26 +12:00
Aldo Cortesi 4979a22d3e Add accessor method for SSLCert object on Response. 2012-04-02 17:02:23 +12:00
Aldo Cortesi ab1d8fa350 Expand SSL cert support
- Capture the remote SSL certificate
- Expose the remote cert as an attribute on Response
- Expand the certutils.SSLCert interface to expose more cert info
2012-04-02 16:21:23 +12:00
Aldo Cortesi c02fdb2463 Refactor proxy.Server to fix a crash when replaying with -n 2012-04-02 13:24:51 +12:00
Aldo Cortesi 15cc09f1b8 Start rationalizing content views.
We now no longer have distinction between "pretty" view and hex/raw. Instead,
we simply a default AUTO view with a global override (M) and a local override
(m).
2012-04-02 10:30:35 +12:00
Aldo Cortesi 7fef0ecdf5 Make "T" pretty view over-ride persistent when switching between flows.
We do this by adding a flow settings mechanism to ConsoleState. This is pretty
rough at the moment and should become more sophisticated as needed.
2012-04-02 09:30:38 +12:00
Aldo Cortesi e9ac4bef20 Add a variant of cleanBin that escapes newlines and tabs.
Use this to fix the hex display option.
2012-03-27 11:25:50 +13:00
Aldo Cortesi a050eeef05 Add a pretty-viewer for images.
This shows basic image information like dimensions, plus extracted EXIF tags
and other metadata.
2012-03-26 11:26:02 +13:00
Aldo Cortesi 2240d2a6a5 Pretty view now indents Javascript.
Thanks to the JSBeautifier project, which is now included in the contrib directory.
2012-03-25 10:56:45 +13:00
Aldo Cortesi 74c51df580 Re-enable simple multipart form parsing and preview. 2012-03-25 10:10:48 +13:00
Aldo Cortesi 62e51018d0 Refactor pretty view mechanism.
Also start adding unit tests for this subsystem.
2012-03-24 14:20:24 +13:00
Aldo Cortesi ed74ed24a0 Add error indications to GridEditor. 2012-03-23 13:28:33 +13:00
Aldo Cortesi 2739cb4861 Add a simple parser for content type specifications. 2012-03-20 10:31:07 +13:00
Aldo Cortesi 5690e7c399 Generalize GridEditor to N columns.
Start adding a replacement rule editor.
2012-03-18 14:39:21 +13:00
Aldo Cortesi 76175672ad Add specification of replacement patterns on the command line. 2012-03-17 17:20:34 +13:00
Aldo Cortesi c8ae1e85b3 Hooks -> ReplaceHooks
It makes more sense to specialize this, which will let me build a nicer
interface for replacement hooks in mitmproxy.
2012-03-17 11:31:05 +13:00
Aldo Cortesi 08f410cacc Add a hooks mechanism, based on filter expressions. 2012-03-16 17:13:11 +13:00
Aldo Cortesi d138af7217 replace() methods now decode and re-encode contents before substitution. 2012-03-16 11:24:18 +13:00
Aldo Cortesi d51b8cab0c Add a decoded context manager.
This simplifies a common chore when modifying traffic - decoding the object,
modifying it, then re-encoding it with the same encoding afterwards. You can
now simply say:

with flow.decoded(request):
    request.content = "bar"
2012-03-16 11:12:56 +13:00
Aldo Cortesi fa6305ee98 Cleanliness fixes.
- Remove unused code during previous commit.
- Code coverage fixes.
2012-03-12 11:25:50 +13:00
Aldo Cortesi d3aad7a185 Merge remote-tracking branch 'taiste/server-replay-pop' 2012-03-10 13:36:50 +13:00
Valtteri Virtanen 041eafba73 Added tests for ServerPlaybackState with nopop 2012-03-05 13:57:57 +02:00
Valtteri Virtanen 5b5b79f5c4 Fixed old tests 2012-03-05 13:40:18 +02:00
Aldo Cortesi e1356dd2b6 Create an SSL certificate class. 2012-03-05 10:22:47 +13:00
András Veres-Szentkirályi 15ad7704d2 Removed imports left unused after Py{OpenSSL,ASN1}
Commits 533f61f67a and
8b841bc9e3 left some imports unused while
swithing to PyOpenSSL and PyASN1 -- this commit removes these imports.
2012-03-01 16:20:34 +01:00
Aldo Cortesi 533f61f67a Use PyOpenSSL and PyASN1 for certificate parsing.
Yes, these are two more major dependencies for mitmproxy, but if we're going to
do all the cool things I want to do with SSL certs, there is no other way.
2012-03-01 21:08:44 +13:00
Aldo Cortesi 8b841bc9e3 Factor out cert operations in to certutils.py. 2012-02-29 13:20:53 +13:00
Aldo Cortesi 688faa9baa Repair unit tests. 2012-02-27 20:34:47 +13:00
Aldo Cortesi 764724748b Fix cert generation harder. 2012-02-27 15:59:29 +13:00
Aldo Cortesi 2ba8296843 Better certificate parsing. 2012-02-27 15:21:05 +13:00
Aldo Cortesi 00942c1431 Add upstream certificate lookup.
This initiates a connection to the server to obtain certificate information to
generate interception certificates. At the moment, the information used is the
Common Name, and the list of Subject Alternative Names.
2012-02-27 15:05:45 +13:00
Aldo Cortesi 986a41d180 Unit test++. 2012-02-25 12:19:54 +13:00
Aldo Cortesi 25fa596cd6 Fix detection of URL-encoded forms.
Thanks to Paul Capestany <capestany@gmail.com> for reporting this.
2012-02-24 13:03:24 +13:00
Aldo Cortesi ddc9155c24 Make "~q" filter work more intuitively.
It now matches any flow that has no response.
2012-02-23 17:06:09 +13:00
Aldo Cortesi 2df9c52c09 Refactor filter matching. 2012-02-23 17:03:58 +13:00
Aldo Cortesi dbd75e02f7 Create ODictCaseless for headers, use vanilla ODict for everything else. 2012-02-20 11:29:36 +13:00
Aldo Cortesi 18029df99c Use ODict for request.get_form_urlencoded and set_form_urlencoded 2012-02-20 11:13:35 +13:00
Aldo Cortesi b0f77dfefd Unit test import cleanups. 2012-02-20 11:04:07 +13:00
Aldo Cortesi fa11b7c9be Use ODict for Request.get_query and Request.set_query 2012-02-20 10:44:47 +13:00
Aldo Cortesi 2616f490fe Rename Headers class to ODict
ODict is an ordered dictionary class that will be useful in many other parts of
our API.
2012-02-20 10:39:00 +13:00
Aldo Cortesi d32d6bc5e3 Add "p" key binding to connection list view to copy a flow. 2012-02-19 00:17:47 +13:00
Aldo Cortesi 8ddc3b4ef2 Add API for duplicating flows. 2012-02-18 23:56:40 +13:00
Aldo Cortesi 7aa79b89e8 Firm up what we consider to be a valid proxy spec. 2012-02-18 16:29:02 +13:00
Aldo Cortesi 6ad8b1a15d Firm up reverse proxy specification.
- Extract proxy spec parsing and unparsing functions.
- Add a status indicator in mitmproxy.
- Add the "R" keybinding for changing the reverse proxy from within mitmproxy.
2012-02-18 16:27:09 +13:00
Aldo Cortesi a7df6e1503 Refactor reverse proxying
- Retain the specification from the Host header as a Request's description.
- Expand upstream proxy specifications to include the scheme. We now say https://hostname:port
- Move the "R" revert keybinding to "v" to make room for a reverse proxy
binding that matches the command-line flag.
2012-02-18 14:45:22 +13:00
Aldo Cortesi acdc2d00b4 Repair unit tests. 2012-02-18 12:27:59 +13:00
Aldo Cortesi da1ccfddeb 100% test coverage for flow.py 2012-02-10 15:55:58 +13:00
Aldo Cortesi 1ad7e91527 Make filter matching act more sensibly. 2012-02-10 15:31:45 +13:00
Aldo Cortesi 5f785e26b9 Add filter for detecting flows with errors.
Also, remove dependency on weird _is_response method.
2012-02-10 15:22:26 +13:00
Aldo Cortesi b14c29b25c Expand test coverage. 2012-02-10 15:04:20 +13:00
Aldo Cortesi 9c985f2d20 Methods for getting and setting form urlencoded data on Request. 2012-02-10 14:27:39 +13:00
Aldo Cortesi 2709441d5b Add get_query and set_query methods to Request. 2012-02-09 16:40:31 +13:00
Aldo Cortesi cdd5a53767 Refactor console.
Split the console implementation out into logical components.
2012-02-07 16:39:37 +13:00
Aldo Cortesi 35a914a549 Fix unit tests broken during previous commit. 2012-01-21 14:39:36 +13:00
Aldo Cortesi d5e3722c97 Fix an issue caused by some editors when editing a request/response body.
Many editors make it hard save a file without a terminating newline on the last
line. When editing message bodies, this can cause problems. For now, I just
strip the newlines off the end of the body when we return from an editor.
2012-01-21 12:43:00 +13:00
Mark E. Haase 05111f093d Add support for filtering by HTTP method (get, post, etc.) using ~m operator. 2011-12-28 17:32:29 -05:00
Aldo Cortesi 67f2610032 Add HTTP body size limit specification to command-line tools. 2011-09-09 15:27:31 +12:00
Aldo Cortesi 28daa93268 Basic infrastructure for request and response body size limits. 2011-09-09 14:49:34 +12:00
Aldo Cortesi e5bded7dee Improve robustness against invalid data. 2011-09-05 07:47:47 +12:00
Aldo Cortesi 45f4768a5c Add attribution and license for tnetstring.py 2011-08-19 21:53:52 +12:00
Aldo Cortesi a566684e32 Move to typed netstrings for serialization.
This change is backwards incompatible with the old serialization format!
2011-08-19 21:30:24 +12:00
Aldo Cortesi b51aac8a86 Code cleanliness - appease pychecker. 2011-08-04 10:34:34 +12:00
Aldo Cortesi 730c78ac53 Move script.Context to flow.ScriptContext 2011-08-04 10:14:44 +12:00
Aldo Cortesi 7a3b871b33 Request class now has a clean pydoc profile. 2011-08-04 09:26:26 +12:00
Aldo Cortesi 0760607a7d Further interface cleaning. 2011-08-03 23:02:33 +12:00
Aldo Cortesi 9042d3f3b9 Clean up interfaces by making some methods pseudo-private. 2011-08-03 22:48:57 +12:00
Aldo Cortesi 57c653be5f Move all HTTP objects to flow.py
That's Request, Response, ClientConnect, ClientDisconnect, Error, and Headers.
2011-08-03 22:41:38 +12:00
Aldo Cortesi e337682d8e Enable "|" command to run a oneshot script on a single flow. 2011-08-03 17:35:18 +12:00
Aldo Cortesi 179cf75862 Add script hooks, enable new engine for mitmdump. 2011-08-03 16:36:20 +12:00
Aldo Cortesi 12d2b1f926 Rip out old script interface, start replacing with new stubs.
Scripts are broken for now.
2011-08-03 13:20:36 +12:00
Aldo Cortesi 62088a6661 Start stubbing out a much more powerful script architecture. 2011-08-03 11:06:29 +12:00
Aldo Cortesi a817db5bd6 Refresh current connection when toggling autodecode.
Also fix the unit tests I forgot to commit...
2011-08-02 20:47:53 +12:00
Aldo Cortesi 1ff6a767d0 Unit test++ 2011-08-02 16:52:47 +12:00
Aldo Cortesi 357502fe03 General cleanup.
Cut out unused variables and code, generally shut up pychecker as much as is
reasonable.
2011-08-02 16:14:33 +12:00
Aldo Cortesi f3742f29da We no longer need to track clientconnections. 2011-08-02 14:56:09 +12:00
Aldo Cortesi 675b3133b4 Improve performance of loading flows from a file hugely.
Fell into the "expensive __eq__ method" trap. Oh, Python, you little scamp.
2011-08-01 11:26:09 +12:00
Aldo Cortesi 43f1c72511 Refactor the way we calculate views of the flow list.
The naive approach we used before recalculated the view on every access, and
consequently had serious performance problems.
2011-08-01 11:17:01 +12:00
Aldo Cortesi ddb5748a76 Add decoding/encoding for requests. 2011-08-01 10:43:01 +12:00
Aldo Cortesi c89c4361c3 Merge remote-tracking branch 'alts/encoding' 2011-07-28 11:19:07 +12:00
Stephen Altamirano 78049abac1 Changes replace logic to function in both Python 2.6.x and 2.7.x
Tests now only assume Python 2.6.x rather than requiring 2.7.x. This does not preclude the use of flags as a kwarg in replace
2011-07-26 22:47:08 -07:00
Stephen Altamirano acd511f676 Adds test method for Response encoding and decoding 2011-07-26 22:38:42 -07:00
Aldo Cortesi 4043829cf2 Add an eventlog option to mitmdump
This shows client connections, disconnections and requests (before a complete
flow is assembled). We need to add an analogous display to mitmproxy.
2011-07-23 12:57:54 +12:00
Aldo Cortesi 47e1695512 Also replace strings path for requests. 2011-07-22 20:52:13 +12:00
Aldo Cortesi 1b961fc4ad Add utility functions to search and replace strings in flows
This is a common task in pentesting scenarios. This commit adds the following
functions:

utils.Headers.replace
proxy.Request.replace
proxy.Response.replace
flow.Flow.replace
2011-07-22 17:48:42 +12:00
Stephen Altamirano ecd4645988 Adds encode counterparts to decode functions 2011-07-17 20:16:47 -07:00
Aldo Cortesi ce41046786 Refine encoding support
- Push decoding down into the LRU cache
- Cope gracefully with corrupted data and incorrect encoding headers
2011-07-17 10:25:25 +12:00
alts 6dc0f105cc Adds support for content encoding, namely gip and deflate 2011-07-16 02:47:06 -07:00
Aldo Cortesi 94ae720a22 Add a pretty-printing mode for urlencoded form data. 2011-07-15 16:46:54 +12:00
Aldo Cortesi 76b4c6ba82 Introduce an anti-compression command-line argument.
This is on by default, which means we avoid compressed content unless the -z
flag is specified.
2011-07-15 15:24:56 +12:00
Aldo Cortesi 7e21ac0eb8 Refine path completion somewhat.
Make it match the behaviour of vim and mutt more closely
2011-07-14 16:54:04 +12:00
Aldo Cortesi 1c9e7b982a Rewrite Headers object to preserve order and case. 2011-07-14 16:01:54 +12:00
Aldo Cortesi 8e176c2086 Cast some data read from dump files to str, to prevent unicode promotion.
This fixes a bug that caused a traceback when de-serialized requests were
replayed. Also adds unit tests for the problem.
2011-07-01 14:20:42 +12:00
Aldo Cortesi 18d4c3a9e9 JSON pretty-printing.
Also rename the display modes ("pretty" instead of "indent"), and expand the
built-in documentation.
2011-06-30 13:27:27 +12:00
Aldo Cortesi f004326855 Try not to hang when user views large request & response bodies
Two different strategies here:

    - Use a simple heuristic to detect if we're looking at XML data when indent
    mode is used. On non-XML data we can hang even on small documents.

    - Only view partial data for large bodies. At the moment the cutoff is
    100k. I might finetune this later.
2011-06-27 15:59:17 +12:00
Aldo Cortesi 7d7803a4d9 Add a hideous kludge to fix not-yet-valid certificates.
- The OpenSSL x509 has no way to explicitly set the notBefore value on
certificates.

- If two systems have the same configured time, it's possible to return a
certificate before the validity start time has arrived.

- We "solve" this by waiting for one second when a certificate is first
generated before returning the cert. The alternative is to rewrite pretty much
all of our certificate generation, a thought too horrible to contemplate.
2011-06-11 15:16:16 +12:00
Aldo Cortesi 613e9a298e Add a new flow loading mechanism.
We now simulate the normal connection flow when we load flows. That means
that we can run scripts, hooks, sticky cookies, etc.
2011-05-15 11:22:35 +12:00
Aldo Cortesi c0bd1a39e4 unit test coverage ++ 2011-03-20 18:52:16 +13:00
Aldo Cortesi c726519e73 Add a stickyauth option.
This allows us to replay an HTTP Authorization header, in the same way as we
replay cookies using stickycookies. This lets us conveniently get at HTTP Basic
Auth protected resources through the proxy, but is not enough to do the same
for HTTP Digest auth. We'll put that on the todo list.
2011-03-20 17:31:54 +13:00
Aldo Cortesi 4f877cde6a Reverse order of flows in mitmproxy.
It matches user expectations much better to have new flows appended to the bottom.
2011-03-20 09:31:39 +13:00
Aldo Cortesi e22fd74d06 Revamp key generation.
We now create three different files in the .mitmproxy directory when a dummy CA
is made:

mitmproxy-ca.pem - the CA, including private key

mitmproxy-ca-cert.p12 - A pkcs12 version of the certificate, for distribution to Windows.

mitmproxy-ca-cert.pem - A PEM version of the certificate, for distribution to everyone else.
2011-03-18 16:45:31 +13:00
Aldo Cortesi 394bd1d6b0 Interrupt interception when deleting an intercepting flow.
Prompting the user for this is annoying.
2011-03-15 17:53:29 +13:00
Aldo Cortesi 6d5c32ad4b Fix hang on shutdown. 2011-03-15 17:21:35 +13:00
Aldo Cortesi fb28e71f0b Do pre-processing of requests before replay.
This enables scripts, anticache, server playback and sticky cookies for
request replays.
2011-03-15 15:11:03 +13:00
Aldo Cortesi fe1e2f16ff Improve responsiveness of request and response viewing.
- Computing the view of a large body is expensive, so we introduce an LRU cache
to hold the latest 20 results.

- Use ListView more correctly, passing it individual urwid.Text snippets,
rather than a single large one. This hugely improves render time.
2011-03-15 13:05:33 +13:00
Aldo Cortesi 2e70a0e44b Fix unit tests on OSX when a system proxy is configured. 2011-03-14 14:05:14 +13:00
Aldo Cortesi e35fd7126b Fix flow formatting unit tests. 2011-03-14 13:49:30 +13:00
Aldo Cortesi 1e47167ac6 Fix a traceback sometimes triggered viewing a connection with no response. 2011-03-13 20:03:13 +13:00
Aldo Cortesi 324d7c3955 Add client plaback to mitmproxy. 2011-03-13 16:50:11 +13:00
Aldo Cortesi 3ef9d2cffc flow.py 100% test coverage 2011-03-13 14:51:25 +13:00
Aldo Cortesi c901cbbada mitmproxy prompted input now display previously set value.
E.g. if you set a limit, then re-enter the limit prompt, you start with the
currently set value.
2011-03-13 11:24:49 +13:00
Aldo Cortesi 9a9a2d9125 Unit tests for proxy.read_chunked 2011-03-12 16:00:01 +13:00
Aldo Cortesi 4ce393cc41 Unit tests for cmdline.py 2011-03-12 15:14:25 +13:00
Aldo Cortesi 96b10ca486 console.py 100% test coverage 2011-03-12 15:08:06 +13:00
Aldo Cortesi fbbc43a5b2 netstring.py 100% unit test coverage. 2011-03-12 14:39:41 +13:00
Aldo Cortesi 9f16a84a9e Make mitmdump handle invalid serialized data gracefully. 2011-03-11 15:16:31 +13:00
Aldo Cortesi 7d85db0da3 Display Errors and killed connections in mitmdump. 2011-03-11 13:06:51 +13:00
Aldo Cortesi daa9653ebe Add --norefresh to stop refreshing server playback to mitmdump.
Also, make cookie parsing for refreshing more error-tolerant.
2011-03-11 11:56:10 +13:00
Aldo Cortesi e99b1d1949 Stub out refresh for server-side replay. 2011-03-09 20:05:30 +13:00
Aldo Cortesi 03f1345385 Add an --anticache option to mitmdump.
This removes all headers that might cause a server to return 304-not-modified.

For now, all the new features are going into mitmdump - everything will be
ported over to mitmproxy once I have the feature set locked down.
2011-03-09 13:15:31 +13:00
Aldo Cortesi 765871bd11 Store timestamps on flow components as a UTC time tuple.
Format is:

(tm_year,tm_mon,tm_mday,tm_hour,tm_min, tm_sec,tm_wday,tm_yday,tm_isdst)
2011-03-07 13:46:02 +13:00
Aldo Cortesi 7330f5fccf Make mitmdump server playback also exit by default.
Like client playback, the --keepserving option makes mitmdump keep serving.
2011-03-06 17:08:56 +13:00
Aldo Cortesi ec00b5a66e Make mitmdump exit after client replay is complete by default.
Add an option --keepserving to make it keep serving after replay.
2011-03-06 16:54:49 +13:00
Aldo Cortesi e794cbc0d8 Add flow_count and active_flow_count methods to flow.State.
Use these in unit tests where sensible.
2011-03-06 16:11:45 +13:00
Aldo Cortesi 42ffded626 Bump unit tests, rearrange mitmdump command-line options slightly. 2011-03-06 16:02:28 +13:00
Aldo Cortesi bad77f3470 Add client replay options to mitmdump. 2011-03-06 11:21:31 +13:00
Aldo Cortesi d9cb083809 Reorganize test suite to remove confusion between test utils and libmproxy utils. 2011-03-05 15:58:48 +13:00
Aldo Cortesi 5da4099ddf Massage namespace to make room for client replay.
Mostly replay -> server_replay
2011-03-05 13:03:26 +13:00
Aldo Cortesi 96471fde1d Basic client playback state structure. 2011-03-04 13:08:43 +13:00
Aldo Cortesi 6921b9ff2a Add an indicator that sticky cookies have been applied in mitmdump. 2011-02-25 21:23:44 +13:00
Aldo Cortesi 8cade9fbbf Move stringification funcs from proxy to dump. 2011-02-25 21:11:44 +13:00
Aldo Cortesi 57f96c5fe0 Minor housekeeping - unused code, nocover directives.
95% test coverage.
2011-02-24 15:44:08 +13:00
Aldo Cortesi 79039eb5d2 More mature sticky cookie primitive. Use it in console.py. 2011-02-24 15:26:34 +13:00
Aldo Cortesi 57947b328e Start abstracting out sticky cookie state. 2011-02-24 10:33:39 +13:00
Aldo Cortesi 39207ffdd2 Add a way for users to specify header significance in server replay.
Also add the --rheader command-line option to mitmdump to let the user specify
an arbitrary number of significant headers. The default is to treat no headers
as significant.
2011-02-23 10:54:51 +13:00
Aldo Cortesi 7bc913c40d Move script hooks into the flow primitives.
This lets handle scripts in corner cases like playback situations more easily.
2011-02-21 11:40:49 +13:00
Aldo Cortesi fe99871df8 Add --kill option to mitmdump
If this option is passed all requests that are not part of a replayed
conversation are killed. If the option is not passed, such requests are passed
through to the server as usual.
2011-02-21 11:08:35 +13:00
Aldo Cortesi fd4dd8cb6b First pass of playback function for mitmdump. 2011-02-21 09:54:39 +13:00
Aldo Cortesi deb79a9c5a Add a simple server playback state object.
We use a loose hash to match incoming requests with recorded flows. At the
moment, this hash is over the host, port, scheme, method, path and content of
the request. Note that headers are not included here - if we do want to include
headers, we would have to do some work to normalize them to remove variations
between user agents, header order, etc. etc.
2011-02-21 08:47:19 +13:00
Aldo Cortesi aa16194518 Clean up and strip down netstrings module. 2011-02-20 14:03:32 +13:00
Aldo Cortesi 7ddba22f51 Certificates are now generated in a temporary per-session directory.
This means that certificates don't accumulate in the conf directory, users
don't have to clear certificates if the CA is regenerated, and the user can
specify a custom CA without invalid certificates being loaded inadvertently.
2011-02-20 13:29:41 +13:00
Aldo Cortesi c2ae8285f4 Revamp SSL configuration.
- Move option parsing utiliities to proxy.py

- Don't have a global config object. Pass it as an argument to ProxyServer.

- Simplify certificate generation logic.
2011-02-20 12:53:42 +13:00
Aldo Cortesi 4fc807cedd Clean up certificate generation.
- Use templates for config files. We can re-introduce customization of the
certificate attributes when we need them.

- Split CA and cert generation into separate functions.

- Generation methods provide an error return when generation fails.

- When the user explicitly specifies a certificate, we don't generate it, but
fail if it doesn't exist.
2011-02-20 12:17:10 +13:00
Aldo Cortesi d7ace1ce9e Bump unit tests for flow.py and dump.py 2011-02-20 09:55:42 +13:00
Aldo Cortesi 9c5c3c2b1a Implement state loading that doesn't change object identity.
We need this to let us load state from copied Flows returned from scripts.
2011-02-20 09:36:13 +13:00
Aldo Cortesi 1549ec8079 Bump test coverage to 100% for flow.py 2011-02-19 17:21:08 +13:00
Aldo Cortesi d9c6be3d13 Fix serialization when a Request has no associated client connection. 2011-02-19 17:03:44 +13:00
Aldo Cortesi 5da27a9905 Refactor Flow primitives to remove HTTP1.0 assumption.
This is a big patch removing the assumption that there's one connection per
Request/Response pair. It touches pretty much every part of mitmproxy, so
expect glitches until everything is ironed out.
2011-02-19 17:00:24 +13:00
Aldo Cortesi cd4eea3934 First pass of script hooks for mitmdump.
Also stub out docs, improve mitmdump error handling.
2011-02-18 12:40:45 +13:00
Aldo Cortesi f028dc7e1b Add filtering to mitmdump. 2011-02-17 10:44:08 +13:00
Aldo Cortesi f0f1fb4b55 Add file writing to mitmdump. 2011-02-17 10:18:38 +13:00
Aldo Cortesi f5511350eb Tweaks and cleanups for mitmdump. 2011-02-16 23:03:46 +13:00
Aldo Cortesi 0dd1aa7cc7 Initial port of mitmdump to Flows. 2011-02-16 22:10:24 +13:00
Aldo Cortesi 66349c9783 FlowMaster bugfixes and unit tests. 2011-02-16 16:43:35 +13:00
Aldo Cortesi 33dac700fa Switch over to new serialization format.
Remove BSON from contrib.
2011-02-16 15:10:00 +13:00
Aldo Cortesi 0c6f846861 First draft of the new serialization mechanism. 2011-02-16 14:33:04 +13:00
Aldo Cortesi 5692c7359c Import Will McGugan's netstring module.
Module is in the Public Domain.

I expect to modify and extend this module, so I've imported into main library
rather than contrib. Code has been reformatted to suite our code standard,
tests have been extrated into /tests directory.
2011-02-16 11:21:06 +13:00
Aldo Cortesi 6339f521fc Repair unit test to match removal of arg from Response class. 2011-02-16 11:20:00 +13:00
Henrik Nordstrom 89627a702a drop unused protocol parameter from Response 2011-02-15 22:24:38 +01:00
Aldo Cortesi 759496e24e Repair unit test suite. 2011-02-11 11:06:30 +13:00
Aldo Cortesi a4eaafab5a Further fine-tuning for pretty_xmlish. 2011-02-06 16:56:13 +13:00
Aldo Cortesi 7156d1a73a Rip out BeautifulSoup, and use a custom XML-ish prettyprinter. 2011-02-06 14:17:30 +13:00
Aldo Cortesi d0c2d9480c Also serialize ClientConnection and flow backups. 2011-02-06 10:28:43 +13:00
Aldo Cortesi 2ad4c5adf3 Get rid of ReplayConnection - we now have only one ClientConnection class. 2011-02-04 10:58:08 +13:00
Aldo Cortesi 2cb7429d38 Change "connection" to the less confusing "client_conn" throughout. 2011-02-03 14:51:32 +13:00
Aldo Cortesi 7924f07971 Fine-tune completion a bit. 2011-02-03 13:50:57 +13:00
Aldo Cortesi 6c89749f0a Add timestamps to flows.
For now, these are only displayed on the connection view screen, with second
granularity.
2011-02-03 13:30:47 +13:00
Aldo Cortesi d8cb826361 Test suite rejiggering and cleanup. 2011-02-03 12:16:03 +13:00
Aldo Cortesi db99da6af5 Improve script handling.
- Display output in external viewer when script exits with error.

- Add a "changed" indicator to show if a request can be reverted.
2011-02-02 11:44:28 +13:00
Aldo Cortesi 405e60215c View script debug output (stderr) in pager. 2011-02-02 10:09:16 +13:00
Aldo Cortesi b886f808be Add an external script API.
External scripts can read a flow, modify it, and then return it to mitmproxy
using a simple API.

The "|" keyboard shortcut within mitmproxy prompts the user for a script.
2011-01-31 13:26:56 +13:00
Aldo Cortesi edb8228dd2 Factor out flow unit tests into speparate file. 2011-01-31 11:48:53 +13:00
Aldo Cortesi 152b97fa0b Restructure to make subclassing Flow unnecessary. 2011-01-31 11:44:52 +13:00
Aldo Cortesi 2a39387d27 UI tweaks
- Make flow view state persistent (request/response, body view mode).
- Don't exit flow view mode when viewing help.
2011-01-30 14:52:51 +13:00
Aldo Cortesi 0742bc4fd0 Nicer statusbar messages, with timed expiry. 2011-01-28 12:18:01 +13:00
Aldo Cortesi dfefe3cdda Make pretty-printing more robust.
Also, since BeautifulSoup is so damn slow, print a statusbar message saying
that we're calculating a pretty version of the response. Maybe I should add
hangman or something, becuase on a 200k document this can take ages.
2011-01-28 12:08:25 +13:00
Aldo Cortesi 93ef691bad Basix XML/HTML pretty-printing in flow viewer. 2011-01-27 17:26:01 +13:00
Aldo Cortesi 460107589c Add tab completion for save and load path specs. 2011-01-27 10:29:37 +13:00
Aldo Cortesi 635f7a971d Add saving and loading of complete flows for later replay and analysis. 2011-01-26 16:50:17 +13:00
Aldo Cortesi 29d8007678 Add serialization hooks to flows and flow component objects. 2011-01-26 14:52:03 +13:00
Aldo Cortesi 7983dbb26a Abstract flow management out of the interactive code. 2011-01-25 15:02:48 +13:00
Aldo Cortesi cb0e328709 Initial checkin. 2010-02-16 17:09:07 +13:00