从 python 到 gcs 模拟器的 IPv4 请求

Question

我正在尝试从 python 应用程序向 docker 中的 gcs 模拟器发出请求-在 docker 中为 mac 编写桥接网络。当我尝试时，我发现 gcs 客户端库以某种方式尝试使用 IPv6 向 gcs 模拟器发出请求但失败了，因为 docker 对于 mac.

不支持 IPv6

我已经实施了以下答案来纠正 IPv4，但它似乎仍在尝试通过 IPv6 发出请求。

如何在 docker-compose 网络中从 python 向 gcs 模拟器发出成功的请求？

我已经确认从本地 Python 脚本到 gcs 模拟器的请求没有 docker-compose 是成功的。

docker-for-mac 问题：https://github.com/docker/for-mac/issues/1432

参考答案：Force requests to use IPv4 / IPv6

gcs 模拟器：https://github.com/fsouza/fake-gcs-server

样本docker-compose.yaml

version: '3'
services:
  run:
    build: .
    container_name: run
    ports:
      - 9090:8080
    env_file: 
      - ./.env
    environment:
      - PORT=8080
  gcs:
    image: fsouza/fake-gcs-server:latest
    container_name: fake-gcs-server
    ports:
      - 4443:4443
    env_file: 
      - ./.env

实施示例：

from google.cloud import storage
from google.api_core.client_options import ClientOptions
from google.auth.credentials import AnonymousCredentials
from unittest.mock import patch
from multijob_sample import variables as vs
import requests
import urllib3
import urllib3.util.connection
import traceback

import socket
orig_getaddrinfo = socket.getaddrinfo
def getaddrinfoIPv4(host, port, family=0, type=0, proto=0, flags=0):
    print(f'running patched getaddrinfo')
    return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET, type=type, proto=proto, flags=flags)
patcher = patch('socket.getaddrinfo', side_effect=getaddrinfoIPv4)
patcher.start()


# for fake-gcs-emulator
http_ssl_disabled = requests.Session()
http_ssl_disabled.verify = False
urllib3.disable_warnings(
       urllib3.exceptions.InsecureRequestWarning
)  # disable https warnings for https insecure certs

client = storage.Client(
    credentials=AnonymousCredentials(),
    project=vs.project_id,
    client_options=ClientOptions(api_endpoint='https://gcs:4443'), 
    _http=http_ssl_disabled,
)

def put_file(bucket_id: str, file, blobname: str):
    file.seek(0)
    try:
        client.get_bucket(bucket_id).blob(blob_name=blobname).upload_from_file(file)
        print(f'file {blobname} uploaded')
    except Exception as e:
        print(f'failed to put file: {blobname}')
        print(f'error: {e}')
        print(f'trace: {traceback.format_exc()}')


put_file("bucketid", file, "blobname") # do put_file

错误信息：

run              | running patched getaddrinfo
run              | failed to put file: test.csv
run              | error: HTTPSConnectionPool(host='::', port=4443): Max retries exceeded with url: /upload/resumable/efbbcde9c49cda2ff78e8da24371ea03 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8fb0765be0>: Failed to establish a new connection: [Errno -9] Address family for hostname not supported'))
run              | trace: Traceback (most recent call last):
run              |   File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
run              |     conn = connection.create_connection(
run              |   File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
run              |     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
run              |   File "/usr/local/lib/python3.9/unittest/mock.py", line 1093, in __call__
run              |     return self._mock_call(*args, **kwargs)
run              |   File "/usr/local/lib/python3.9/unittest/mock.py", line 1097, in _mock_call
run              |     return self._execute_mock_call(*args, **kwargs)
run              |   File "/usr/local/lib/python3.9/unittest/mock.py", line 1158, in _execute_mock_call
run              |     result = effect(*args, **kwargs)
run              |   File "/app/multijob_sample/storage.py", line 26, in getaddrinfoIPv4
run              |     return orig_getaddrinfo(host=host, port=port, family=socket.AF_INET, type=type, proto=proto, flags=flags)
run              |   File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo
run              |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
run              | socket.gaierror: [Errno -9] Address family for hostname not supported

Answer 1

到目前为止，这是我一段时间以来不得不解决的最烦人的问题之一。解决方案是运行带有 -external-url http://<your docker compose service name>:<port> 选项的模拟器。

此问题仅发生在文件上传时，因为它只发生在可恢复的上传中。对于可恢复上传，GCS 客户端首先“启动”与服务器的可恢复上传，并且在响应中服务器包含一个 URL 以供将来请求转到（不知道为什么，但这似乎是一个复杂的API）。问题是模拟器不知道它自己的url！事实上，如果您查看模拟器的日志，您会看到它打印出类似 server started at http://[::]:4443 的内容。 :: 与您在错误中看到的 :: 相同。所以模拟器用它的 :: URL 响应，然后过了一会儿客户端崩溃试图解析 URL.

我仍然不确定为什么运行在 docker-compose 之外工作，我想在 ""、"localhost" 或周围某处有一些特殊的外壳"::"`.

从 python 到 gcs 模拟器的 IPv4 请求

Request with IPv4 from python to gcs emulator

python

docker

network-programming

google-cloud-storage

docker-for-mac