2ba8296843
Better certificate parsing.
2012-02-27 15:21:05 +13:00
00942c1431
Add upstream certificate lookup.
...
This initiates a connection to the server to obtain certificate information to
generate interception certificates. At the moment, the information used is the
Common Name, and the list of Subject Alternative Names.
2012-02-27 15:05:45 +13:00
986a41d180
Unit test++.
2012-02-25 12:19:54 +13:00
25fa596cd6
Fix detection of URL-encoded forms.
...
Thanks to Paul Capestany <capestany@gmail.com> for reporting this.
2012-02-24 13:03:24 +13:00
ddc9155c24
Make "~q" filter work more intuitively.
...
It now matches any flow that has no response.
2012-02-23 17:06:09 +13:00
2df9c52c09
Refactor filter matching.
2012-02-23 17:03:58 +13:00
dbd75e02f7
Create ODictCaseless for headers, use vanilla ODict for everything else.
2012-02-20 11:29:36 +13:00
18029df99c
Use ODict for request.get_form_urlencoded and set_form_urlencoded
2012-02-20 11:13:35 +13:00
b0f77dfefd
Unit test import cleanups.
2012-02-20 11:04:07 +13:00
fa11b7c9be
Use ODict for Request.get_query and Request.set_query
2012-02-20 10:44:47 +13:00
2616f490fe
Rename Headers class to ODict
...
ODict is an ordered dictionary class that will be useful in many other parts of
our API.
2012-02-20 10:39:00 +13:00
d32d6bc5e3
Add "p" key binding to connection list view to copy a flow.
2012-02-19 00:17:47 +13:00
8ddc3b4ef2
Add API for duplicating flows.
2012-02-18 23:56:40 +13:00
7aa79b89e8
Firm up what we consider to be a valid proxy spec.
2012-02-18 16:29:02 +13:00
6ad8b1a15d
Firm up reverse proxy specification.
...
- Extract proxy spec parsing and unparsing functions.
- Add a status indicator in mitmproxy.
- Add the "R" keybinding for changing the reverse proxy from within mitmproxy.
2012-02-18 16:27:09 +13:00
a7df6e1503
Refactor reverse proxying
...
- Retain the specification from the Host header as a Request's description.
- Expand upstream proxy specifications to include the scheme. We now say https://hostname:port
- Move the "R" revert keybinding to "v" to make room for a reverse proxy
binding that matches the command-line flag.
2012-02-18 14:45:22 +13:00
acdc2d00b4
Repair unit tests.
2012-02-18 12:27:59 +13:00
da1ccfddeb
100% test coverage for flow.py
2012-02-10 15:55:58 +13:00
1ad7e91527
Make filter matching act more sensibly.
2012-02-10 15:31:45 +13:00
5f785e26b9
Add filter for detecting flows with errors.
...
Also, remove dependency on weird _is_response method.
2012-02-10 15:22:26 +13:00
b14c29b25c
Expand test coverage.
2012-02-10 15:04:20 +13:00
9c985f2d20
Methods for getting and setting form urlencoded data on Request.
2012-02-10 14:27:39 +13:00
2709441d5b
Add get_query and set_query methods to Request.
2012-02-09 16:40:31 +13:00
cdd5a53767
Refactor console.
...
Split the console implementation out into logical components.
2012-02-07 16:39:37 +13:00
35a914a549
Fix unit tests broken during previous commit.
2012-01-21 14:39:36 +13:00
d5e3722c97
Fix an issue caused by some editors when editing a request/response body.
...
Many editors make it hard save a file without a terminating newline on the last
line. When editing message bodies, this can cause problems. For now, I just
strip the newlines off the end of the body when we return from an editor.
2012-01-21 12:43:00 +13:00
05111f093d
Add support for filtering by HTTP method (get, post, etc.) using ~m operator.
2011-12-28 17:32:29 -05:00
67f2610032
Add HTTP body size limit specification to command-line tools.
2011-09-09 15:27:31 +12:00
28daa93268
Basic infrastructure for request and response body size limits.
2011-09-09 14:49:34 +12:00
e5bded7dee
Improve robustness against invalid data.
2011-09-05 07:47:47 +12:00
45f4768a5c
Add attribution and license for tnetstring.py
2011-08-19 21:53:52 +12:00
a566684e32
Move to typed netstrings for serialization.
...
This change is backwards incompatible with the old serialization format!
2011-08-19 21:30:24 +12:00
b51aac8a86
Code cleanliness - appease pychecker.
2011-08-04 10:34:34 +12:00
730c78ac53
Move script.Context to flow.ScriptContext
2011-08-04 10:14:44 +12:00
7a3b871b33
Request class now has a clean pydoc profile.
2011-08-04 09:26:26 +12:00
0760607a7d
Further interface cleaning.
2011-08-03 23:02:33 +12:00
9042d3f3b9
Clean up interfaces by making some methods pseudo-private.
2011-08-03 22:48:57 +12:00
57c653be5f
Move all HTTP objects to flow.py
...
That's Request, Response, ClientConnect, ClientDisconnect, Error, and Headers.
2011-08-03 22:41:38 +12:00
e337682d8e
Enable "|" command to run a oneshot script on a single flow.
2011-08-03 17:35:18 +12:00
179cf75862
Add script hooks, enable new engine for mitmdump.
2011-08-03 16:36:20 +12:00
12d2b1f926
Rip out old script interface, start replacing with new stubs.
...
Scripts are broken for now.
2011-08-03 13:20:36 +12:00
62088a6661
Start stubbing out a much more powerful script architecture.
2011-08-03 11:06:29 +12:00
a817db5bd6
Refresh current connection when toggling autodecode.
...
Also fix the unit tests I forgot to commit...
2011-08-02 20:47:53 +12:00
1ff6a767d0
Unit test++
2011-08-02 16:52:47 +12:00
357502fe03
General cleanup.
...
Cut out unused variables and code, generally shut up pychecker as much as is
reasonable.
2011-08-02 16:14:33 +12:00
f3742f29da
We no longer need to track clientconnections.
2011-08-02 14:56:09 +12:00
675b3133b4
Improve performance of loading flows from a file hugely.
...
Fell into the "expensive __eq__ method" trap. Oh, Python, you little scamp.
2011-08-01 11:26:09 +12:00
43f1c72511
Refactor the way we calculate views of the flow list.
...
The naive approach we used before recalculated the view on every access, and
consequently had serious performance problems.
2011-08-01 11:17:01 +12:00
ddb5748a76
Add decoding/encoding for requests.
2011-08-01 10:43:01 +12:00
c89c4361c3
Merge remote-tracking branch 'alts/encoding'
2011-07-28 11:19:07 +12:00
78049abac1
Changes replace logic to function in both Python 2.6.x and 2.7.x
...
Tests now only assume Python 2.6.x rather than requiring 2.7.x. This does not preclude the use of flags as a kwarg in replace
2011-07-26 22:47:08 -07:00
acd511f676
Adds test method for Response encoding and decoding
2011-07-26 22:38:42 -07:00
4043829cf2
Add an eventlog option to mitmdump
...
This shows client connections, disconnections and requests (before a complete
flow is assembled). We need to add an analogous display to mitmproxy.
2011-07-23 12:57:54 +12:00
47e1695512
Also replace strings path for requests.
2011-07-22 20:52:13 +12:00
1b961fc4ad
Add utility functions to search and replace strings in flows
...
This is a common task in pentesting scenarios. This commit adds the following
functions:
utils.Headers.replace
proxy.Request.replace
proxy.Response.replace
flow.Flow.replace
2011-07-22 17:48:42 +12:00
ecd4645988
Adds encode counterparts to decode functions
2011-07-17 20:16:47 -07:00
ce41046786
Refine encoding support
...
- Push decoding down into the LRU cache
- Cope gracefully with corrupted data and incorrect encoding headers
2011-07-17 10:25:25 +12:00
6dc0f105cc
Adds support for content encoding, namely gip and deflate
2011-07-16 02:47:06 -07:00
94ae720a22
Add a pretty-printing mode for urlencoded form data.
2011-07-15 16:46:54 +12:00
76b4c6ba82
Introduce an anti-compression command-line argument.
...
This is on by default, which means we avoid compressed content unless the -z
flag is specified.
2011-07-15 15:24:56 +12:00
7e21ac0eb8
Refine path completion somewhat.
...
Make it match the behaviour of vim and mutt more closely
2011-07-14 16:54:04 +12:00
1c9e7b982a
Rewrite Headers object to preserve order and case.
2011-07-14 16:01:54 +12:00
8e176c2086
Cast some data read from dump files to str, to prevent unicode promotion.
...
This fixes a bug that caused a traceback when de-serialized requests were
replayed. Also adds unit tests for the problem.
2011-07-01 14:20:42 +12:00
18d4c3a9e9
JSON pretty-printing.
...
Also rename the display modes ("pretty" instead of "indent"), and expand the
built-in documentation.
2011-06-30 13:27:27 +12:00
f004326855
Try not to hang when user views large request & response bodies
...
Two different strategies here:
- Use a simple heuristic to detect if we're looking at XML data when indent
mode is used. On non-XML data we can hang even on small documents.
- Only view partial data for large bodies. At the moment the cutoff is
100k. I might finetune this later.
2011-06-27 15:59:17 +12:00
7d7803a4d9
Add a hideous kludge to fix not-yet-valid certificates.
...
- The OpenSSL x509 has no way to explicitly set the notBefore value on
certificates.
- If two systems have the same configured time, it's possible to return a
certificate before the validity start time has arrived.
- We "solve" this by waiting for one second when a certificate is first
generated before returning the cert. The alternative is to rewrite pretty much
all of our certificate generation, a thought too horrible to contemplate.
2011-06-11 15:16:16 +12:00
613e9a298e
Add a new flow loading mechanism.
...
We now simulate the normal connection flow when we load flows. That means
that we can run scripts, hooks, sticky cookies, etc.
2011-05-15 11:22:35 +12:00
c0bd1a39e4
unit test coverage ++
2011-03-20 18:52:16 +13:00
c726519e73
Add a stickyauth option.
...
This allows us to replay an HTTP Authorization header, in the same way as we
replay cookies using stickycookies. This lets us conveniently get at HTTP Basic
Auth protected resources through the proxy, but is not enough to do the same
for HTTP Digest auth. We'll put that on the todo list.
2011-03-20 17:31:54 +13:00
4f877cde6a
Reverse order of flows in mitmproxy.
...
It matches user expectations much better to have new flows appended to the bottom.
2011-03-20 09:31:39 +13:00
e22fd74d06
Revamp key generation.
...
We now create three different files in the .mitmproxy directory when a dummy CA
is made:
mitmproxy-ca.pem - the CA, including private key
mitmproxy-ca-cert.p12 - A pkcs12 version of the certificate, for distribution to Windows.
mitmproxy-ca-cert.pem - A PEM version of the certificate, for distribution to everyone else.
2011-03-18 16:45:31 +13:00
394bd1d6b0
Interrupt interception when deleting an intercepting flow.
...
Prompting the user for this is annoying.
2011-03-15 17:53:29 +13:00
6d5c32ad4b
Fix hang on shutdown.
2011-03-15 17:21:35 +13:00
fb28e71f0b
Do pre-processing of requests before replay.
...
This enables scripts, anticache, server playback and sticky cookies for
request replays.
2011-03-15 15:11:03 +13:00
fe1e2f16ff
Improve responsiveness of request and response viewing.
...
- Computing the view of a large body is expensive, so we introduce an LRU cache
to hold the latest 20 results.
- Use ListView more correctly, passing it individual urwid.Text snippets,
rather than a single large one. This hugely improves render time.
2011-03-15 13:05:33 +13:00
2e70a0e44b
Fix unit tests on OSX when a system proxy is configured.
2011-03-14 14:05:14 +13:00
e35fd7126b
Fix flow formatting unit tests.
2011-03-14 13:49:30 +13:00
1e47167ac6
Fix a traceback sometimes triggered viewing a connection with no response.
2011-03-13 20:03:13 +13:00
324d7c3955
Add client plaback to mitmproxy.
2011-03-13 16:50:11 +13:00
3ef9d2cffc
flow.py 100% test coverage
2011-03-13 14:51:25 +13:00
c901cbbada
mitmproxy prompted input now display previously set value.
...
E.g. if you set a limit, then re-enter the limit prompt, you start with the
currently set value.
2011-03-13 11:24:49 +13:00
9a9a2d9125
Unit tests for proxy.read_chunked
2011-03-12 16:00:01 +13:00
4ce393cc41
Unit tests for cmdline.py
2011-03-12 15:14:25 +13:00
96b10ca486
console.py 100% test coverage
2011-03-12 15:08:06 +13:00
fbbc43a5b2
netstring.py 100% unit test coverage.
2011-03-12 14:39:41 +13:00
9f16a84a9e
Make mitmdump handle invalid serialized data gracefully.
2011-03-11 15:16:31 +13:00
7d85db0da3
Display Errors and killed connections in mitmdump.
2011-03-11 13:06:51 +13:00
daa9653ebe
Add --norefresh to stop refreshing server playback to mitmdump.
...
Also, make cookie parsing for refreshing more error-tolerant.
2011-03-11 11:56:10 +13:00
e99b1d1949
Stub out refresh for server-side replay.
2011-03-09 20:05:30 +13:00
03f1345385
Add an --anticache option to mitmdump.
...
This removes all headers that might cause a server to return 304-not-modified.
For now, all the new features are going into mitmdump - everything will be
ported over to mitmproxy once I have the feature set locked down.
2011-03-09 13:15:31 +13:00
765871bd11
Store timestamps on flow components as a UTC time tuple.
...
Format is:
(tm_year,tm_mon,tm_mday,tm_hour,tm_min, tm_sec,tm_wday,tm_yday,tm_isdst)
2011-03-07 13:46:02 +13:00
7330f5fccf
Make mitmdump server playback also exit by default.
...
Like client playback, the --keepserving option makes mitmdump keep serving.
2011-03-06 17:08:56 +13:00
ec00b5a66e
Make mitmdump exit after client replay is complete by default.
...
Add an option --keepserving to make it keep serving after replay.
2011-03-06 16:54:49 +13:00
e794cbc0d8
Add flow_count and active_flow_count methods to flow.State.
...
Use these in unit tests where sensible.
2011-03-06 16:11:45 +13:00
42ffded626
Bump unit tests, rearrange mitmdump command-line options slightly.
2011-03-06 16:02:28 +13:00
bad77f3470
Add client replay options to mitmdump.
2011-03-06 11:21:31 +13:00
d9cb083809
Reorganize test suite to remove confusion between test utils and libmproxy utils.
2011-03-05 15:58:48 +13:00
5da4099ddf
Massage namespace to make room for client replay.
...
Mostly replay -> server_replay
2011-03-05 13:03:26 +13:00
96471fde1d
Basic client playback state structure.
2011-03-04 13:08:43 +13:00
6921b9ff2a
Add an indicator that sticky cookies have been applied in mitmdump.
2011-02-25 21:23:44 +13:00
8cade9fbbf
Move stringification funcs from proxy to dump.
2011-02-25 21:11:44 +13:00
57f96c5fe0
Minor housekeeping - unused code, nocover directives.
...
95% test coverage.
2011-02-24 15:44:08 +13:00
79039eb5d2
More mature sticky cookie primitive. Use it in console.py.
2011-02-24 15:26:34 +13:00
57947b328e
Start abstracting out sticky cookie state.
2011-02-24 10:33:39 +13:00
39207ffdd2
Add a way for users to specify header significance in server replay.
...
Also add the --rheader command-line option to mitmdump to let the user specify
an arbitrary number of significant headers. The default is to treat no headers
as significant.
2011-02-23 10:54:51 +13:00
7bc913c40d
Move script hooks into the flow primitives.
...
This lets handle scripts in corner cases like playback situations more easily.
2011-02-21 11:40:49 +13:00
fe99871df8
Add --kill option to mitmdump
...
If this option is passed all requests that are not part of a replayed
conversation are killed. If the option is not passed, such requests are passed
through to the server as usual.
2011-02-21 11:08:35 +13:00
fd4dd8cb6b
First pass of playback function for mitmdump.
2011-02-21 09:54:39 +13:00
deb79a9c5a
Add a simple server playback state object.
...
We use a loose hash to match incoming requests with recorded flows. At the
moment, this hash is over the host, port, scheme, method, path and content of
the request. Note that headers are not included here - if we do want to include
headers, we would have to do some work to normalize them to remove variations
between user agents, header order, etc. etc.
2011-02-21 08:47:19 +13:00
aa16194518
Clean up and strip down netstrings module.
2011-02-20 14:03:32 +13:00
7ddba22f51
Certificates are now generated in a temporary per-session directory.
...
This means that certificates don't accumulate in the conf directory, users
don't have to clear certificates if the CA is regenerated, and the user can
specify a custom CA without invalid certificates being loaded inadvertently.
2011-02-20 13:29:41 +13:00
c2ae8285f4
Revamp SSL configuration.
...
- Move option parsing utiliities to proxy.py
- Don't have a global config object. Pass it as an argument to ProxyServer.
- Simplify certificate generation logic.
2011-02-20 12:53:42 +13:00
4fc807cedd
Clean up certificate generation.
...
- Use templates for config files. We can re-introduce customization of the
certificate attributes when we need them.
- Split CA and cert generation into separate functions.
- Generation methods provide an error return when generation fails.
- When the user explicitly specifies a certificate, we don't generate it, but
fail if it doesn't exist.
2011-02-20 12:17:10 +13:00
d7ace1ce9e
Bump unit tests for flow.py and dump.py
2011-02-20 09:55:42 +13:00
9c5c3c2b1a
Implement state loading that doesn't change object identity.
...
We need this to let us load state from copied Flows returned from scripts.
2011-02-20 09:36:13 +13:00
1549ec8079
Bump test coverage to 100% for flow.py
2011-02-19 17:21:08 +13:00
d9c6be3d13
Fix serialization when a Request has no associated client connection.
2011-02-19 17:03:44 +13:00
5da27a9905
Refactor Flow primitives to remove HTTP1.0 assumption.
...
This is a big patch removing the assumption that there's one connection per
Request/Response pair. It touches pretty much every part of mitmproxy, so
expect glitches until everything is ironed out.
2011-02-19 17:00:24 +13:00
cd4eea3934
First pass of script hooks for mitmdump.
...
Also stub out docs, improve mitmdump error handling.
2011-02-18 12:40:45 +13:00
f028dc7e1b
Add filtering to mitmdump.
2011-02-17 10:44:08 +13:00
f0f1fb4b55
Add file writing to mitmdump.
2011-02-17 10:18:38 +13:00
f5511350eb
Tweaks and cleanups for mitmdump.
2011-02-16 23:03:46 +13:00
0dd1aa7cc7
Initial port of mitmdump to Flows.
2011-02-16 22:10:24 +13:00
66349c9783
FlowMaster bugfixes and unit tests.
2011-02-16 16:43:35 +13:00
33dac700fa
Switch over to new serialization format.
...
Remove BSON from contrib.
2011-02-16 15:10:00 +13:00
0c6f846861
First draft of the new serialization mechanism.
2011-02-16 14:33:04 +13:00
5692c7359c
Import Will McGugan's netstring module.
...
Module is in the Public Domain.
I expect to modify and extend this module, so I've imported into main library
rather than contrib. Code has been reformatted to suite our code standard,
tests have been extrated into /tests directory.
2011-02-16 11:21:06 +13:00
6339f521fc
Repair unit test to match removal of arg from Response class.
2011-02-16 11:20:00 +13:00
89627a702a
drop unused protocol parameter from Response
2011-02-15 22:24:38 +01:00
759496e24e
Repair unit test suite.
2011-02-11 11:06:30 +13:00
a4eaafab5a
Further fine-tuning for pretty_xmlish.
2011-02-06 16:56:13 +13:00
7156d1a73a
Rip out BeautifulSoup, and use a custom XML-ish prettyprinter.
2011-02-06 14:17:30 +13:00
d0c2d9480c
Also serialize ClientConnection and flow backups.
2011-02-06 10:28:43 +13:00
2ad4c5adf3
Get rid of ReplayConnection - we now have only one ClientConnection class.
2011-02-04 10:58:08 +13:00
2cb7429d38
Change "connection" to the less confusing "client_conn" throughout.
2011-02-03 14:51:32 +13:00
7924f07971
Fine-tune completion a bit.
2011-02-03 13:50:57 +13:00
6c89749f0a
Add timestamps to flows.
...
For now, these are only displayed on the connection view screen, with second
granularity.
2011-02-03 13:30:47 +13:00
d8cb826361
Test suite rejiggering and cleanup.
2011-02-03 12:16:03 +13:00
db99da6af5
Improve script handling.
...
- Display output in external viewer when script exits with error.
- Add a "changed" indicator to show if a request can be reverted.
2011-02-02 11:44:28 +13:00
405e60215c
View script debug output (stderr) in pager.
2011-02-02 10:09:16 +13:00
b886f808be
Add an external script API.
...
External scripts can read a flow, modify it, and then return it to mitmproxy
using a simple API.
The "|" keyboard shortcut within mitmproxy prompts the user for a script.
2011-01-31 13:26:56 +13:00
edb8228dd2
Factor out flow unit tests into speparate file.
2011-01-31 11:48:53 +13:00
152b97fa0b
Restructure to make subclassing Flow unnecessary.
2011-01-31 11:44:52 +13:00
2a39387d27
UI tweaks
...
- Make flow view state persistent (request/response, body view mode).
- Don't exit flow view mode when viewing help.
2011-01-30 14:52:51 +13:00
0742bc4fd0
Nicer statusbar messages, with timed expiry.
2011-01-28 12:18:01 +13:00
dfefe3cdda
Make pretty-printing more robust.
...
Also, since BeautifulSoup is so damn slow, print a statusbar message saying
that we're calculating a pretty version of the response. Maybe I should add
hangman or something, becuase on a 200k document this can take ages.
2011-01-28 12:08:25 +13:00
93ef691bad
Basix XML/HTML pretty-printing in flow viewer.
2011-01-27 17:26:01 +13:00
460107589c
Add tab completion for save and load path specs.
2011-01-27 10:29:37 +13:00
635f7a971d
Add saving and loading of complete flows for later replay and analysis.
2011-01-26 16:50:17 +13:00
29d8007678
Add serialization hooks to flows and flow component objects.
2011-01-26 14:52:03 +13:00
7983dbb26a
Abstract flow management out of the interactive code.
2011-01-25 15:02:48 +13:00
cb0e328709
Initial checkin.
2010-02-16 17:09:07 +13:00
Aldo Cortesi