Commit Graph

252 Commits

Author SHA1 Message Date
Matteo Luppi a128672ec7
Fix Bug view options menu (#6697)
#### Description

This PR fixes bug described here : #4448 

I set a max-height property and a scroll in case of overflow on the
y-axis.

#### Checklist

 - [x] I have updated tests where applicable.
 - [x] I have added an entry to the CHANGELOG.

---------

Co-authored-by: Maximilian Hils <git@maximilianhils.com>
2024-03-06 21:55:26 +01:00
Denis Stanishevsky 766b5451b7
Allow --allow-hosts and --ignore-hosts to work together (#6711)
#### Description

The mutual exclusivity of the allow-hosts and ignore-hosts parameters
looks like an unnecessary obstacle and does not make much sense.

It is very convenient to use a proxy only for the domain of your
service, but at the same time ignore some subdomains, especially when
they serve some kind of CDNs with a large amount of data.

Although this filtering could be implemented using regexp with negative
lookahead, but it complicates configuration and is not as clear as
conjuction of allow and deny filters.

#### Checklist

 - [x] I have updated tests where applicable.
 - [x] I have added an entry to the CHANGELOG.

---------

Co-authored-by: Denis Stanishevskiy <>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-03-06 21:29:08 +01:00
Gaurav Jain b63a600f84
Allow runtime modifications of HTTP flow filters for server replays (#6695)
#### Description
Fixes #4506
`mitmproxy` during server-replay mode, calculates the hashes of flows
from input files based on user defined filters and uses them to compare
against hashes of incoming requests to serve the corresponding stored
response by matching the hash. However, during runtime, if the user
changes any of the filters, `mitmproxy` fails to recalculate the hashes
of input flows and hence doesn't return the intended response. This PR
fixes this issue by recomputing the hashes for every flow whenever a
filter(option) used for computing hashes is changed.

#### Checklist

 - [x] I have updated tests where applicable.
 - [x] I have added an entry to the CHANGELOG.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Maximilian Hils <git@maximilianhils.com>
2024-02-28 18:57:06 +00:00
Gaurav Jain 92c556afbe
Add `content_view_lines_cutoff` option to mitmdump (#6692)
#### Description

Fixes #4476 
This PR adds the already existing `content_view_lines_cutoff` option to
`mitmdump` to filter the number of content lines shown when
`flow_detail` option is set to `3`. By default this option's value is
set to `512`

Updated tests that check for `cutoff` in responses that cross `512`
lines(with the default value set)
Added tests to ensure that custom values for `content_view_lines_cutoff`
works as intended

#### Verification
1. run `mitmdump --set flow_detail=3 --set content_view_lines_cutoff=3`
2. run `curl --proxy http:/127.0.0.1:8080 "https://google.com/"`

#### Before
<details>
<summary>Output</summary>

```
[15:00:41.441] HTTP(S) proxy listening at *:8080.
[15:00:53.236][127.0.0.1:38826] client connect
[15:00:53.520][127.0.0.1:38826] server connect google.com:443 (142.250.183.238:443)
127.0.0.1:38826: GET https://google.com/ HTTP/2.0
    user-agent: curl/8.5.0
    accept: */*

 << HTTP/2.0 301 Moved Permanently 220b
    location: https://www.google.com/
    content-type: text/html; charset=UTF-8
    content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-7rNdfc7B14cr_4JZskk3AQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
    date: Tue, 27 Feb 2024 09:31:02 GMT
    expires: Thu, 28 Mar 2024 09:31:02 GMT
    cache-control: public, max-age=2592000
    server: gws
    content-length: 220
    x-xss-protection: 0
    x-frame-options: SAMEORIGIN
    alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

    <HTML>
    <HEAD>
      <meta http-equiv="content-type" content="text/html;charset=utf-8">
      <TITLE>301 Moved</TITLE>
    </HEAD>
    <BODY>
      <H1>301 Moved</H1>
      The document has moved
      <A HREF="https://www.google.com/">here</A>
      .
    </BODY>
    </HTML>

[15:00:53.897][127.0.0.1:38826] client disconnect
[15:00:53.897][127.0.0.1:38826] server disconnect google.com:443 (142.250.183.238:443)
```
</details>

Content is not truncated within `3` lines

#### After

<details>
<summary>Output</summary>

```
[14:55:26.042] HTTP(S) proxy listening at *:8080.
[14:55:33.630][127.0.0.1:40554] client connect
[14:55:33.775][127.0.0.1:40554] server connect google.com:443 (142.250.182.142:443)
127.0.0.1:40554: GET https://google.com/ HTTP/2.0
    user-agent: curl/8.5.0
    accept: */*

 << HTTP/2.0 301 Moved Permanently 220b
    location: https://www.google.com/
    content-type: text/html; charset=UTF-8
    content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-LCd_ThPYwSImYoighASUFQ' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
    date: Tue, 27 Feb 2024 09:25:42 GMT
    expires: Thu, 28 Mar 2024 09:25:42 GMT
    cache-control: public, max-age=2592000
    server: gws
    content-length: 220
    x-xss-protection: 0
    x-frame-options: SAMEORIGIN
    alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

    <HTML>
    <HEAD>
      <meta http-equiv="content-type" content="text/html;charset=utf-8">
    (cut off)

[14:55:34.021][127.0.0.1:40554] client disconnect
[14:55:34.022][127.0.0.1:40554] server disconnect google.com:443 (142.250.182.142:443)
```
</details>

Content is truncated within `3` lines

#### Checklist

 - [x] I have updated tests where applicable.
 - [x] I have added an entry to the CHANGELOG.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-02-27 12:50:25 +01:00
Mike Anselmi 3ba4b65ecd
Fix ca_file containing multiple CAs (#6666)
#### Description

Unfortunately 0b5e310881 broke mitmproxy's
ability to issue leaf certificates if `ca_file` contains multiple CAs.
This PR restores that capability.

The issue lies in `mitmproxy/certs.py` - specifically, in the
`from_files` method of the `CertStore` class. Before
0b5e310881, the issuing CA was identified
like this:

``` python
raw = ca_file.read_bytes()
key = load_pem_private_key(raw, passphrase)
…
certs = re.split(rb"(?=-----BEGIN CERTIFICATE-----)", raw)
ca = Cert.from_pem(certs[1])
```

This worked even when `ca_file` contained multiple CAs. For example,
consider this example:

```
-----BEGIN PRIVATE KEY-----
REDACTED
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIICmTCCAfqgAwIBAgIRAIqaBJ6aU5wXT+5JAn8Uj1IwCgYIKoZIzj0EAwQwMDEu
MCwGA1UEAxMlbWFuc2VsbWkgSW50ZXJtZWRpYXRlIENBICgyMDIzLTA0LTExKTAe
Fw0yNDAyMTQxNzEzMDBaFw0yNDAyMTUxNzE0MDBaMB8xHTAbBgNVBAMTFG1pdG1w
cm94eSBJc3N1aW5nIENBMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEeKTLpy6KeiWN
89S9UJoTM9w6HyrEiVByf/ie2wUX3vR/3nHVMTi5XZQD6/8h3A9yUWwtRtVDj55D
STbPKjN6G8t/Qh1dGz2vanNjmWNlceM3DOIlT30vmR5+GrUOwj+7o4HoMIHlMA4G
A1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwEgYD
VR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUkf9kgx5DbP0B66Ir6oLUDV/CIU4w
HwYDVR0jBBgwFoAUqBYQLb8rkIidwq5dYuiDFdKhY48wYAYMKwYBBAGCpGTGKEAB
BFAwTgIBAQQcbWlrZUBtYW5zZWxtaS5jb206aXNzdWluZy1jYQQrYmZhODBJaGMz
VXNUcU0yQlp6ZG1rYW1OLW1XNGtpWTJrM1g5X19kd3Q0bzAKBggqhkjOPQQDBAOB
jAAwgYgCQgF/QLZ0cThJKeoeIk4dY4+Z2MdIey7kvvy1jwlWrsOqOY+UXUWHXRlj
oXZN92imAKqgLU5T859MMqWrjA1LjN7w6QJCAY+2xN7ovry3WSMqwnu86NuPL8vh
fjftZeIW5GWuLzOsEFt7SfWMhkmFc/jcFpZ/qlolPhvr6E170zNhHUSMKvjb
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIICcjCCAdSgAwIBAgIQeK+VyMjx85py1kqGdjn5nTAKBggqhkjOPQQDBDAoMSYw
JAYDVQQDEx1tYW5zZWxtaSBSb290IENBICgyMDIzLTA0LTExKTAeFw0yMzA0MTEw
MjIzMzdaFw0yODA0MDkwMjIzMzdaMDAxLjAsBgNVBAMTJW1hbnNlbG1pIEludGVy
bWVkaWF0ZSBDQSAoMjAyMy0wNC0xMSkwgZswEAYHKoZIzj0CAQYFK4EEACMDgYYA
BADhbjtnME8IDpBab8MogM0HchH0V9B8Qx7nRXK+tbGyyHlgLjIBYF/erJGBP0md
RMAxU5x/nPlHPODRBkJhB0wbcwGC9zjQQyxfhG75806QzkaZSsC0kFZ9d3JG4fjn
uenY1S2VZ5qE0gDc8Qmfyjcx7KqxQKwS9RwtG1ymAmWgvmGIJ6OBlDCBkTAOBgNV
HQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB/wIBATAdBgNVHQ4EFgQUqBYQLb8r
kIidwq5dYuiDFdKhY48wHwYDVR0jBBgwFoAU6+JptrrPiOuWZyKvwg0EJz5TkW8w
KwYDVR0fBCQwIjAgoB6gHIYaaHR0cDovL2NhLm1hbnNlbG1pLmNvbS9jcmwwCgYI
KoZIzj0EAwQDgYsAMIGHAkEw/A2kiL4Kt9/8PND89vlXEehqeymFgats62a3UMU2
g3zATES/WI2E1IgqIsmB6ILRgWMslzpH+NGP+JJkFCz9cwJCAJaRlnOc3xafHZta
0Xwr5YUVAAfmVOj9a2XAsNlNuMy2t8cmeeGmQjrZwCuivlzLOp19Ge3Mi0dnzeV+
5L2rBOCL
-----END CERTIFICATE-----
```

`certs` would have three elements: the private key, the issuing CA and
the intermediate CA. As a result, `ca = Cert.from_pem(certs[1])` would
select the first CA (the issuing CA).

From 0b5e310881 onward, we instead have

``` python
raw = ca_file.read_bytes()
key = load_pem_private_key(raw, passphrase)
…
certs = x509.load_pem_x509_certificates(raw)
ca = Cert(certs[-1])
```

Now, `certs` would have only two elements: the issuing CA and the
intermediate CA. (`x509.load_pem_x509_certificates` discards the private
key.) As a result, `ca = Cert(certs[-1])` must instead be `ca =
Cert(certs[0])`, otherwise the `ca` and `key` won't correspond to each
other and we'll eventually see an error like this when mitmproxy tries
to generate a leaf certificate:

```
Addon error: [('x509 certificate routines', '', 'key values mismatch')]
Traceback (most recent call last):
  File "/Users/manselmi/repos/mitmproxy/mitmproxy/addons/tlsconfig.py", line 208, in tls_start_client
    tls_start.ssl_conn.use_privatekey(
  File "/Users/manselmi/virtualenv/mitmproxy-py312/lib/python3.12/site-packages/OpenSSL/SSL.py", line 1949, in use_privatekey
    self._context._raise_passphrase_exception()
  File "/Users/manselmi/virtualenv/mitmproxy-py312/lib/python3.12/site-packages/OpenSSL/SSL.py", line 1123, in _raise_passphrase_exception
    _raise_current_error()
  File "/Users/manselmi/virtualenv/mitmproxy-py312/lib/python3.12/site-packages/OpenSSL/_util.py", line 57, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('x509 certificate routines', '', 'key values mismatch')]
```
2024-02-16 19:37:24 +00:00
basedBaba 1f2e1a254a
fix: generate wireguard config with correct endpoint when using two or more active nics (#6659)
#### Description

Fix issue #6656

This generates a wireguard config with the correct endpoint when using
two or more active NICs.
 
#### Checklist

 - [x] I have updated tests where applicable.
 - [x] I have added an entry to the CHANGELOG.

---------

Co-authored-by: Maximilian Hils <github@maximilianhils.com>
2024-02-14 09:39:29 +00:00
Sujal Singh d317304268
Remove duplicate answers in DNS queries (#6648)
#### Description

Fixes #6647 by assuming all DNS queries are made over UDP, will need to
be reworked when TCP support is added.

#### Checklist

 - [x] I have updated tests where applicable.
 - [x] I have added an entry to the CHANGELOG.
2024-02-13 10:58:34 +01:00
Maximilian Hils 6bd19a77e2
build macOS arm64 binaries (#6633) 2024-01-31 23:20:19 +00:00
Maximilian Hils 6afa7081e8
fix CHANGELOG 2024-01-31 23:44:02 +01:00
basedBaba ec96b98f7f
fix: save failed connect request urls to har files correctly (#6599) 2024-01-31 23:37:39 +01:00
dq 865e1138da
fix IPv6 matches against the full host:port string (#6614) 2024-01-23 20:29:24 +00:00
mitmproxy release bot d75dc95a1e mitmproxy 10.2.2 2024-01-21 17:09:27 +00:00
Louis 09f4719711
allow_hosts/ignore_hosts option now matches against the full `host:port` string (#6594)
Co-authored-by: Maximilian Hils <git@maximilianhils.com>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2024-01-21 17:07:14 +00:00
Maximilian Hils c6defba77d
websockets: do not process start event before redirecting event handlers (#6609)
fix #6608
2024-01-21 13:53:02 +00:00
Maximilian Hils 08eb515635
fix client playback (#6605) 2024-01-17 17:58:35 +01:00
Maximilian Hils 8fc114a9b2
fix CHANGELOG 2024-01-08 08:54:48 +01:00
Daniel López Guimaraes c2dd0d5a42
savehar: Fix timings being set to null (#6578) 2024-01-07 10:48:50 -08:00
mitmproxy release bot a7edbdf74c mitmproxy 10.2.1 2024-01-06 14:17:25 +00:00
Maximilian Hils 3a3d69eb42
Wireguard mode: bind to 0.0.0.0 (#6587)
fix #6586
2024-01-06 13:26:16 +00:00
Jon Brown ae00e82c3a
remove mention of ctx.log from addon example (#6552)
[#5590](https://github.com/mitmproxy/mitmproxy/pull/5590) made the
switch to standard logging, but this part of the addons documentation
references the previous `ctx.log` approach.
2024-01-04 13:35:17 +00:00
Maximilian Hils a9b8a00b70
fix #6529 (#6581) 2024-01-04 13:29:00 +00:00
mitmproxy release bot fb45c18e01 mitmproxy 10.2.0 2024-01-04 12:00:25 +00:00
Maximilian Hils 5640c53d04 update CHANGELOG 2024-01-04 12:10:45 +01:00
Maximilian Hils 8290c428ca
fix ignore hosts to not drop connections, improve http header handling (#6559)
fixes #6554
2023-12-18 13:55:49 +00:00
mitmproxy release bot d79862c8bb mitmproxy 10.1.6 2023-12-14 09:39:20 +00:00
Maximilian Hils b659ea0101
represent SubjectAltNames as cryptography.x509.GeneralName objects, not strings. (#6537)
This fixes #6536
2023-12-14 10:26:23 +01:00
Llama c2f1aa1600
Change default keybind to export flow to "x" (#6305)
Co-authored-by: Maximilian Hils <git@maximilianhils.com>
2023-12-13 16:45:27 +01:00
Emanuele Micheletti b84a821fd8
make url filters case-insensitive (#6493)
fix #6329
2023-12-12 18:33:19 +00:00
xBZZZZ 13c976de0d
fix `mitmweb` splitter becoming drag and drop (#6492) 2023-12-12 18:23:32 +00:00
Maximilian Hils 9787871e6c
do not include subject key identifier in leaf certificates (#6549)
This fixes #6494: if CA and leaf share the same Subject Key Identifier,
SChannel gets confused. So we just skip the SKI for leafs, which is
still fine with OpenSSL 3.x (this was previously fixed by @mmaxim) and
RFC 5280.

Co-authored-by: Andrius Andrikonis <andrius.andrikonis@toughlex.com>
2023-12-12 19:19:29 +01:00
Maximilian Hils 0a3e016d39
fix a bug where connections would not be fully closed (#6543) 2023-12-12 18:15:19 +00:00
Zanie Blue 1fcd0335d5
Fix response `content-length` when reading from HAR files (#6548)
#### Description

Closes https://github.com/mitmproxy/mitmproxy/issues/6547

Responses in flows constructed from HAR files were using the
`Response.make` utility which resulted in the injection of
`content-length` headers. When a `content-length` header existed
already, this could cause failures during replay.

#### Checklist

 - [x] I have updated tests where applicable.
 - [x] I have added an entry to the CHANGELOG.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2023-12-12 17:24:29 +00:00
Maximilian Hils bda9c4ea3a
fix: log until the very end (#6541) 2023-12-09 19:53:33 +00:00
Fujun Lv 558b0b6642
Fix for --allow-hosts/--ignore-hosts options in WireGuard mode (#5930) (#6513) 2023-12-06 17:00:17 +07:00
rosydawn6 81fc802194
Stream large bodies warn with modify body (#6514)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Co-authored-by: Maximilian Hils <git@maximilianhils.com>
2023-12-05 02:36:54 +00:00
Lucas Ficheux ba84b6be99
make scripts in config.yml relative to config.yml (#4860)
Co-authored-by: Lucas FICHEUX <lficheux@corp.free.fr>
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2023-12-04 17:45:46 +07:00
Lucas Ficheux 43bbcefd1e
Parse set header cookies properly (#5084)
#### Description

Currently when an empty cookie attribute (`Secure`, `HttpOnly` ...) is
encountered while parsing a `Set-Cookie` header it will create a
`CookieAttrs` object containing a (key, value) pair with an empty string
for the attribute value ie:

```python
CookieAttrs[('Secure', ''), ('HttpOnly', ''), ('Path', '/')]
``` 
Resulting in an updated `Set-Cookie` header for the `Response` object
with invalid values for those empty attributes ie:
```python
(b'SetCookie', b'value=XYZ; Secure=; HttpOnly=; Path=/')
``` 
My browser (Firefox 95.0.1) does not pickup these attributes so the
cookie looses them.

______

This fix replaces the empty string attribute for empty cookie attributes
by the value `None` ie:

```python
CookieAttrs[('Secure', None), ('HttpOnly', None), ('Path', '/')]
``` 

So that they can be told apart from attributes with intentional empty
string values when setting the updated header, which results in a
properly formatted header:

```python
(b'SetCookie', b'value=XYZ; Secure; HttpOnly; Path=/')
``` 

#### Checklist

 - [x] I have updated tests where applicable.
 - [x] I have added an entry to the CHANGELOG.

Co-authored-by: Lucas FICHEUX <lficheux@corp.free.fr>
2023-12-02 11:21:38 +07:00
Maximilian Hils 72679e5cf7
CHANGELOG: add a note that PR refs are added automatically (#6521) 2023-12-02 10:55:11 +07:00
mitmproxy release bot 18280927b4 mitmproxy 10.1.5 2023-11-14 12:54:48 +00:00
Maximilian Hils 50012418bc
Remove stray `replay-extra` (#6485)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2023-11-14 12:52:16 +00:00
mitmproxy release bot 11d758e151 mitmproxy 10.1.4 2023-11-13 14:11:21 +00:00
Maximilian Hils 3828373e91
macOS: Use correct entitlements for binary distribution (#6480)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2023-11-13 13:52:55 +01:00
Daniel Karandikar 746537e051
Add option to return various status codes rather than killing during server replay (#6465)
#### Description

Designed to satisfy the requirements of
https://github.com/mitmproxy/mitmproxy/issues/3489

Add `server_replay_404_extra` which behaves similarly to the kill flag,
but returns 404 responses rather than killing

#### Checklist

 - [x] I have updated tests where applicable.
 - [x] I have added an entry to the CHANGELOG.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2023-11-08 11:12:44 +01:00
lain3d 667a098002
Fix case that causes corrupt har files to be written (#6459)
#### Description

* Update savehar addon to fix creating corrupt har files caused by empty
response content

#### Checklist

 - [x] I have updated tests where applicable.
 - [x] I have added an entry to the CHANGELOG.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2023-11-06 07:31:28 -08:00
autofix-ci[bot] 506c0e1013
[autofix.ci] apply automated fixes 2023-11-04 15:16:32 +00:00
pogzyb 2a571a8c09
Update CHANGELOG.md 2023-11-04 11:13:09 -04:00
mitmproxy release bot 23bb301498 mitmproxy 10.1.3 2023-11-04 11:02:01 +00:00
Maximilian Hils e6f8f088d9
Strip double slash from websocket URL, fix #6452 (#6454)
Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
2023-11-04 11:00:22 +00:00
mitmproxy release bot 4c07dc8488 mitmproxy 10.1.2 2023-11-03 11:57:58 +00:00
Maximilian Hils 3470473e4b
build macOS app bundle (#6447) 2023-11-03 11:21:55 +00:00