如何在 mitmproxy 容器中捕获服务器的响应并将其以 json 格式传递到文件中？

Question

我需要捕获并填写每个通过我的 mitmproxy 容器的响应和请求的单独文件。

Docker文件

FROM mitmproxy/mitmproxy:latest

RUN mkdir url_catching
WORKDIR /home/$USER/url_catching
COPY ./url.py .



EXPOSE 8080:8080

ENTRYPOINT ["/usr/bin/mitmdump","-s","./url.py"]

Docker 运行

sudo docker run --rm -it -p 8080:8080 mitmdump_url:latest

我的pyhton脚本（sry，我是python的新手）

from mitmproxy import http

def response(flow):
    url_request: str = str(flow.request.pretty_url)
    url_request = url_request.replace("/", "_")
    with open(url_request, "ab") as ofile:
        ofile.write(flow.request.pretty_url.encode())
        ofile.write(flow.request.content)
        ofile.write(flow.response.content)

在结果中，我为每个请求获得了单独的文件，它的名称等于请求的 url，但只有 url 是人类可读的格式，其他所有内容都已编码或只是遗漏了。所以我需要将通过我的代理以 json 格式发送到文件的响应过去。

Answer 1

request/responde 有 headers + empty line + body/content

我展示了两种将 headers 转换为 string/bytes 的方法。

作为普通文本行

for key, value in flow.response.headers.items():
    ofile.write('{}: {}\n'.format(key, value).encode())

结果

Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 14 Jan 2020 11:51:49 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 181
Connection: keep-alive

如JSON。我转换成dict()是因为headers不能直接转换成JSON

d = dict(flow.request.headers.items())
d = json.dumps(d, indents=2)
ofile.write(d.encode() + b'\n')

结果

{
  "Host": "httpbin.org",
  "User-Agent": "python-requests/2.22.0",
  "Accept-Encoding": "gzip, deflate",
  "Accept": "*/*",
  "Connection": "keep-alive"
}

我也跳过带有“/static/”的网址

from mitmproxy import http
import json

def response(flow):
    url_request: str = str(flow.request.pretty_url)

    if '/static/' not in url_request:
        url_request = url_request.replace("/", "_")
        with open(url_request + '.txt', "ab") as ofile:

            ofile.write(b'--- url ---\n')
            ofile.write(flow.request.pretty_url.encode() + b'\n')

            ofile.write(b'--- request ---\n')

            ofile.write(b'--- headers ---\n')
            #for key, value in flow.request.headers.items():
            #    ofile.write('{}: {}\n'.format(key, value).encode())
            d = dict(flow.request.headers.items())
            d = json.dumps(d, indents=2)
            ofile.write(d.encode() + b'\n')


            ofile.write(b'--- content ---\n')
            ofile.write(flow.request.content + b'\n')

            ofile.write(b'--- response ---\n')

            ofile.write(b'--- headers ---\n')
            for key, value in flow.response.headers.items():
                ofile.write('{}: {}\n'.format(key, value).encode())

            ofile.write(b'--- content ---\n')
            ofile.write(flow.response.content + b'\n')

要将所有内容整合为一个 JSON，您必须首先创建包含所有元素（headers、body 等）的字典，然后使用 json.dumps(all_elements)

测试代码

import requests

proxy = {
    'http': 'http://localhost:8080',
    'https': 'http://localhost:8080',
}

urls = [
    'https://httpbin.org/get',
    'https://httpbin.org/gzip',
    'https://httpbin.org/brotli',
    'https://httpbin.org/deflate',
    'https://httpbin.org/encoding/utf8',
]

for url in urls:
    print(url)
    r = requests.get(url, proxies=proxy, verify=False)
    print(r.text)

包含结果的文件之一

--- url ---
https://httpbin.org/get
--- request ---
--- headers ---
{
  "Host": "httpbin.org",
  "User-Agent": "python-requests/2.22.0",
  "Accept-Encoding": "gzip, deflate",
  "Accept": "*/*",
  "Connection": "keep-alive"
}
--- content ---

--- response ---
--- headers ---
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 14 Jan 2020 12:06:04 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 181
Connection: keep-alive
--- content ---
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.22.0"
  }, 
  "origin": "83.23.66.224, 83.23.66.224", 
  "url": "https://httpbin.org/get"
}

如何在 mitmproxy 容器中捕获服务器的响应并将其以 json 格式传递到文件中？

How to catch response of server in mitmproxy container and past it into file in json format?

python

reverse-proxy

http-proxy

python-3.x

Docker文件

Docker 运行

我的pyhton脚本（sry，我是python的新手）