如何在 mitmproxy 中流式传输响应

Question

由于我使用糟糕的互联网连接，我必须使用下载管理器下载大包，然后将它们流式传输到 chocolatey（它仍然缺乏恢复功能）。为了完成 MITM 工作，我使用 mitmproxy 和一个简单的脚本。

脚本

from mitmproxy import http

dlFileLoc = "C:\Users\Pouya\Downloads\flutter_windows_2.0.4-stable.zip"

def responseheaders(flow: http.HTTPFlow):
    flow.response.stream = True

def request(flow: http.HTTPFlow):
    if flow.request.pretty_host.find("storage.googleapis.com") >= 0:
        # flow.intercept()
        with open(dlFileLoc, mode="rb") as fb:
            flow.response = http.HTTPResponse.make(content=fb.read())
            # flow.resume()

正在将脚本加载到 `mitmdump`

mitmdump.exe -s D:\Code\cli\rend\main.py

为 `chocolatey`

设置代理

choco upgrade flutter -y --proxy http://127.0.0.1:8080

mitmproxy 崩溃

127.0.0.1:50395: Traceback (most recent call last):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\server.py", line 121, in handle
    root_layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\modes\http_proxy.py", line 9, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\tls.py", line 285, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http1.py", line 100, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 204, in __call__
    if not self._process_flow(flow):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 277, in _process_flow
    return self.handle_regular_connect(f)
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 222, in handle_regular_connect
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\tls.py", line 285, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http1.py", line 100, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 204, in __call__
    if not self._process_flow(flow):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 457, in _process_flow
    self.send_response_body(f.response, chunks)
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http1.py", line 71, in send_response_body
    for chunk in http1.assemble_body(response.headers, chunks, response.trailers):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\net\http\http1\assemble.py", line 44, in assemble_body
    for chunk in body_chunks:
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\net\http\http1\read.py", line 137, in read_body
    content = rfile.read(chunk_size)
AttributeError: 'NoneType' object has no attribute 'read'
Traceback (most recent call last):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\server.py", line 121, in handle
    root_layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\modes\http_proxy.py", line 9, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\tls.py", line 285, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http1.py", line 100, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 204, in __call__
    if not self._process_flow(flow):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 277, in _process_flow
    return self.handle_regular_connect(f)
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 222, in handle_regular_connect
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\tls.py", line 285, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http1.py", line 100, in __call__
    layer()
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 204, in __call__
    if not self._process_flow(flow):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http.py", line 457, in _process_flow
    self.send_response_body(f.response, chunks)
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\proxy\protocol\http1.py", line 71, in send_response_body
    for chunk in http1.assemble_body(response.headers, chunks, response.trailers):
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\net\http\http1\assemble.py", line 44, in assemble_body
    for chunk in body_chunks:
  File "C:\Users\Pouya\AppData\Roaming\Python\Python39\site-packages\mitmproxy\net\http\http1\read.py", line 137, in read_body
    content = rfile.read(chunk_size)
AttributeError: 'NoneType' object has no attribute 'read'

环境

> mitmdump.exe --version
Mitmproxy: 6.0.2
Python:    3.9.4
OpenSSL:   OpenSSL 1.1.1i  8 Dec 2020
Platform:  Windows-10-10.0.19041-SP0

Answer 1

虽然我之前的脚本应该可以工作，但不知何故，我最终通过如下更改脚本使其工作：

from mitmproxy import http

dlFileLoc = "C:\Users\Pouya\Downloads\flutter_windows_2.0.4-stable.zip"


def request(flow: http.HTTPFlow):
    if flow.request.pretty_host.find("storage.googleapis.com") >= 0:
        with open(dlFileLoc, mode="rb") as fb:
            body = fb.read()
            size = str(len(body))
            flow.response = http.HTTPResponse.make(
                status_code=200,
                content=body,
                headers={
                    "Content-Type": "application/octet-stream",
                    "Content-Length": size,
                },
            )

然后将 --set stream_large_bodies 传递给一个合理的值，例如 10Mb

mitmdump.exe -s D:\Code\cli\rend\main.py --set stream_large_bodies=10m

巧克力必须通过 mitmproxy 请求

choco upgrade flutter -y --proxy http://127.0.0.1:8080

如何在 mitmproxy 中流式传输响应

How to stream response in mitmproxy

mitmproxy

脚本

正在将脚本加载到 mitmdump

为 chocolatey

环境

正在将脚本加载到 `mitmdump`

为 `chocolatey`