MemoryError: mitmproxy has crashed

Question

MemoryError：mitmproxy 已崩溃

Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: MemoryError
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: mitmproxy has crashed!
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: Please lodge a bug report at: https://github.com/mitmproxy/mitmproxy
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: 192.168.50.117:60549: Traceback (most recent call last):
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: File "/usr/local/lib/python3.6/site-packages/mitmproxy/proxy/server.py", line 121, in handle
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: root_layer()
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: File "/usr/local/lib/python3.6/site-packages/mitmproxy/proxy/modes/transparent_proxy.py", line 19, in call
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: layer()
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: File "/usr/local/lib/python3.6/site-packages/mitmproxy/proxy/protocol/tls.py", line 285, in call
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: layer()
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: File "/usr/local/lib/python3.6/site-packages/mitmproxy/proxy/protocol/http1.py", line 83, in call
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: layer()
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: File "/usr/local/lib/python3.6/site-packages/mitmproxy/proxy/protocol/http.py", line 190, in call
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: if not self._process_flow(flow):
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: File "/usr/local/lib/python3.6/site-packages/mitmproxy/proxy/protocol/http.py", line 397, in _process_flow
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: self.read_response_body(f.request, f.response)
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: MemoryError
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: Exception in response:[Errno 12] Cannot allocate memory
Jul 16 13:02:40 cubedev-PowerEdge-R330 bash[21256]: message repeated 37 times: [ Exception in response:[Errno 12] Cannot allocate memory]

重现该行为的步骤：

当 2 个客户端连接到 mitmdump.The 时观察到的问题在 10 分钟内观察到崩溃

系统信息

mitmdump --version 中间代理：5.1.1 Python：3.6.0 OpenSSL：OpenSSL 1.1.1g 2020 年 4 月 21 日 平台：Linux-4.15.0-45-generic-x86_64-with-debian-stretch-sid

Answer 1

mitmproxy 在内存中处理所有 request/responses，因此如果您下载大量文件，mitmproxy 至少需要相同数量的 RAM。

您可以使用 option stream_large_bodies:

将 mitmproxy 配置为“流”（直接传递）大型请求和响应主体

mitmproxy --set stream_large_bodies=10m

这会流式传输所有大于 10MB 的主体（AFAIR 流式主体不会被任何过滤器处理，也不会被捕获）。

此外，您应该使用 -w 选项保存收集的 requests/response。