通过套接字发送包含文件的字典 (python)
Send a dictionary containing a file through a socket (python)
是否可以通过套接字发送包含文件（图像或文档）作为值的字典？
我试过像下面这样的东西但我失败了..
with open("cat.jpeg", "rb") as f:
myFile = f.read(2048)
data = {"id": "1283", "filename": "cat.jpeg", "file": myFile}
dataToSend = json.dumps(data).encode("utf-8")
这给出了一个 json 错误，myFile 是一个字节数组，无法序列化。
我尝试使用 base64 编码将 myFile 转换为字符串，但没有成功。
部分起作用的是将 myFile 转换为字符串，例如 str(myFile)。 json 序列化程序工作，我通过套接字发送它，dict 没问题，但 myFile 数据已损坏，所以我无法重新创建图片。
那么是否可以使用这种方法或者我应该如何通过套接字发送文件和数据以便在另一端轻松解析？
LE:
仍然无法使用 base64 编码，myFile 仍然是 "bytes" 格式并且
json 给出此错误：TypeError: Object of type 'bytes' is not JSON serializable
客户
import os
import base64
import json
import socket
currentPath = os.path.dirname(os.path.abspath(__file__)) + "\downloads\"
with open(currentPath + "cat.png", "rb") as f:
l = f.read()
print(type(l)) #prints <class 'bytes'>
myFile = base64.b64encode(l)
print(type(myFile)) #prints <class 'bytes'>
data = {"id": "12", "filename": "cat.png", "message": "So cute!", "file": myFile}
dataToSend = json.dumps(data).encode("utf-8") #prints TypeError: Object of type 'bytes' is not JSON serializable
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 1234))
s.sendall(dataToSend)
s.close()
和服务器：
import socket
import json
import os
import sys
import time
import base64
currentPath = os.path.dirname(os.path.abspath(__file__)) + "\fileCache\"
tempData = bytearray()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1", 1234))
s.listen(5)
conn, addr = s.accept()
while True:
dataReceived = conn.recv(2048)
if sys.getsizeof(dataReceived) > 17:
tempData = tempData + dataReceived
else:
data = json.loads(tempData.decode("utf-8"))
break
time.sleep(1)
print(data)
myFile = base64.b64decode(data["file"])
with open(currentPath + data["filename"], "wb") as f:
f.write(myFile)
f.close()
你应该可以做到：
data = base64.b64encode(myFile)
dataToSend = json.dumps({"id":"1283","filename":"cat.jpeg", "file":data})
然后通过套接字发送。当您在套接字的另一端收到数据时，只需执行：
jsonDict = json.loads(dataReceived)
data = base64.b64decode(jsonDict["file"])
更好的方法可能是只使用 bson，https://github.com/py-bson/bson。
from gevent import monkey, socket
monkey.patch_all()
import bson
bson.patch_socket()
with open("cat.jpeg", "rb") as f:
myFile = f.read()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 12345))
s.sendobj({u"id": "1283", u"filename": "cat.jpeg", u"file": myFile})
正如我在评论中所说，将二进制数据打包成字符串格式（如 JSON）是一种浪费——如果你使用 base64，你将增加 33% 的数据传输大小，这也使得JSON 解码器很难正确解码 JSON，因为它需要遍历整个结构以提取索引。
最好将它们分开发送 - JSON 作为 JSON，然后文件内容直接作为二进制文件发送。当然，您需要一种区分两者的方法，最简单的方法是在发送 JSON 数据前加上其长度，以便服务器知道要读取多少字节才能获得 JSON，然后读取其余的作为文件内容。这将使它成为一种非常简单的协议，包的形式如下：
[JSON LENGTH][JSON][FILE CONTENTS]
假设 JSON 永远不会大于 4GB（如果是，您将遇到更大的问题，因为解析它会是一场噩梦），[=13 就足够了=] 的固定 4 字节（32 位）作为无符号整数（如果您不希望 JSON 超过 64KB，您甚至可以使用 16 位）所以整个策略将在客户端工作如：
- 创建负载
- 将其编码为 JSON，然后使用 UTF-8 编码将其编码为
bytes
- 获取上述包的长度，作为流的前4个字节发送
- 发送JSON包裹
- 读取并发送文件内容
并且在服务器端执行相同的过程
- 读取接收数据的前 4 个字节以获得 JSON 负载长度
- 读取下一个字节数以匹配此长度
- 使用 UTF-8 将它们解码为字符串，然后解码 JSON 以获得有效负载
- 读取其余的流数据并将其存储到文件
或者在代码中，客户端：
import json
import os
import socket
import struct
BUFFER_SIZE = 4096 # a uniform buffer size to use for our transfers
# pick up an absolute path from the script folder, not necessary tho
file_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "downloads", "cat.png"))
# let's first prepare the payload to send over
payload = {"id": 12, "filename": os.path.basename(file_path), "message": "So cute!"}
# now JSON encode it and then turn it onto a bytes stream by encoding it as UTF-8
json_data = json.dumps(payload).encode("utf-8")
# then connect to the server and send everything
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: # create a socket
print("Connecting...")
s.connect(("127.0.0.1", 1234)) # connect to the server
# first send the JSON payload length
print("Sending `{filename}` with a message: {message}.".format(**payload))
s.sendall(struct.pack(">I", len(json_data))) # pack as BE 32-bit unsigned int
# now send the JSON payload itself
s.sendall(json_data) # let Python deal with the buffer on its own for the JSON...
# finally, open the file and 'stream' it to the socket
with open(file_path, "rb") as f:
chunk = f.read(BUFFER_SIZE)
while chunk:
s.send(chunk)
chunk = f.read(BUFFER_SIZE)
# alternatively, if you're using Python 3.5+ you can just use socket.sendfile() instead
print("Sent.")
和服务器：
import json
import os
import socket
import struct
BUFFER_SIZE = 4096 # a uniform buffer size to use for our transfers
target_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "fileCache"))
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind(("127.0.0.1", 1234)) # bind to the 1234 port on localhost
s.listen(0) # allow only one connection so we don't have to deal with data separation
while True:
print("Waiting for a connection...")
connection, address = s.accept() # wait for and accept the incoming connection
print("Connection from `{}` accepted.".format(address))
# read the starting 32 bits and unpack them into an int to get the JSON length
json_length = struct.unpack(">I", connection.recv(4))[0]
# now read the JSON data of the given size and JSON decode it
json_data = b"" # initiate an empty bytes structure
while len(json_data) < json_length:
chunk = connection.recv(min(BUFFER_SIZE, json_length - len(json_data)))
if not chunk: # no data, possibly broken connection/bad protocol
break # just exit for now, you should deal with this case in production
json_data += chunk
payload = json.loads(json_data.decode("utf-8")) # JSON decode the payload
# now read the rest and store it into a file at the target path
file_path = os.path.join(target_path, payload["filename"])
with open(file_path, "wb") as f: # open the target file for writing...
chunk = connection.recv(BUFFER_SIZE) # and stream the socket data to it...
while chunk:
f.write(chunk)
chunk = connection.recv(BUFFER_SIZE)
# finally, lets print out that we received the data
print("Received `{filename}` with a message: {message}".format(**payload))
注意：请记住，这是 Python 3.x 代码 - 对于 Python 2.x，您必须自己处理上下文管理，而不是让with ... 阻止 open/close 你的套接字。
仅此而已。当然，在真实的环境中，你需要处理断开连接、多个客户端等。但这是底层过程。
感谢大家的帮助，我终于用base64搞定了。
我在堆栈溢出时在这里找到了答案，我忘记了 link 但它在这里。
在使用 json.dumps 之前，我不得不像这样编码和解码文件。
base64_bytes = b64encode(l)
myFile = base64_bytes.decode("utf-8")
这是一个工作示例：
客户：
import os
from base64 import b64encode
import json
import socket
currentPath = os.path.dirname(os.path.abspath(__file__)) + "\downloads\"
with open(currentPath + "cat.png", "rb") as f:
l = f.read()
base64_bytes = b64encode(l)
myFile = base64_bytes.decode("utf-8")
data = {"id": "12", "filename": "cat.png", "message": "So cute!", "file": myFile}
dataToSend = json.dumps(data).encode("utf-8")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 1234))
s.sendall(dataToSend)
s.close()
服务器：
import socket
import json
import os
import sys
import base64
currentPath = os.path.dirname(os.path.abspath(__file__)) + "\fileCache\"
tempData = bytearray()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1", 1234))
s.listen(5)
conn, addr = s.accept()
while True:
dataReceived = conn.recv(4096)
if sys.getsizeof(dataReceived) > 17:
tempData = tempData + dataReceived
else:
data = json.loads(tempData.decode("utf-8"))
break
myFile = base64.b64decode(data["file"])
with open(currentPath + data["filename"], "wb") as f:
f.write(myFile)
f.close()
是否可以通过套接字发送包含文件（图像或文档）作为值的字典？
我试过像下面这样的东西但我失败了..
with open("cat.jpeg", "rb") as f:
myFile = f.read(2048)
data = {"id": "1283", "filename": "cat.jpeg", "file": myFile}
dataToSend = json.dumps(data).encode("utf-8")
这给出了一个 json 错误，myFile 是一个字节数组，无法序列化。
我尝试使用 base64 编码将 myFile 转换为字符串，但没有成功。
部分起作用的是将 myFile 转换为字符串，例如 str(myFile)。 json 序列化程序工作，我通过套接字发送它，dict 没问题，但 myFile 数据已损坏，所以我无法重新创建图片。
那么是否可以使用这种方法或者我应该如何通过套接字发送文件和数据以便在另一端轻松解析？
LE:
仍然无法使用 base64 编码，myFile 仍然是 "bytes" 格式并且 json 给出此错误：TypeError: Object of type 'bytes' is not JSON serializable
客户
import os
import base64
import json
import socket
currentPath = os.path.dirname(os.path.abspath(__file__)) + "\downloads\"
with open(currentPath + "cat.png", "rb") as f:
l = f.read()
print(type(l)) #prints <class 'bytes'>
myFile = base64.b64encode(l)
print(type(myFile)) #prints <class 'bytes'>
data = {"id": "12", "filename": "cat.png", "message": "So cute!", "file": myFile}
dataToSend = json.dumps(data).encode("utf-8") #prints TypeError: Object of type 'bytes' is not JSON serializable
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 1234))
s.sendall(dataToSend)
s.close()
和服务器：
import socket
import json
import os
import sys
import time
import base64
currentPath = os.path.dirname(os.path.abspath(__file__)) + "\fileCache\"
tempData = bytearray()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1", 1234))
s.listen(5)
conn, addr = s.accept()
while True:
dataReceived = conn.recv(2048)
if sys.getsizeof(dataReceived) > 17:
tempData = tempData + dataReceived
else:
data = json.loads(tempData.decode("utf-8"))
break
time.sleep(1)
print(data)
myFile = base64.b64decode(data["file"])
with open(currentPath + data["filename"], "wb") as f:
f.write(myFile)
f.close()
你应该可以做到：
data = base64.b64encode(myFile)
dataToSend = json.dumps({"id":"1283","filename":"cat.jpeg", "file":data})
然后通过套接字发送。当您在套接字的另一端收到数据时，只需执行：
jsonDict = json.loads(dataReceived)
data = base64.b64decode(jsonDict["file"])
更好的方法可能是只使用 bson，https://github.com/py-bson/bson。
from gevent import monkey, socket
monkey.patch_all()
import bson
bson.patch_socket()
with open("cat.jpeg", "rb") as f:
myFile = f.read()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 12345))
s.sendobj({u"id": "1283", u"filename": "cat.jpeg", u"file": myFile})
正如我在评论中所说，将二进制数据打包成字符串格式（如 JSON）是一种浪费——如果你使用 base64，你将增加 33% 的数据传输大小，这也使得JSON 解码器很难正确解码 JSON，因为它需要遍历整个结构以提取索引。
最好将它们分开发送 - JSON 作为 JSON，然后文件内容直接作为二进制文件发送。当然，您需要一种区分两者的方法，最简单的方法是在发送 JSON 数据前加上其长度，以便服务器知道要读取多少字节才能获得 JSON，然后读取其余的作为文件内容。这将使它成为一种非常简单的协议，包的形式如下：
[JSON LENGTH][JSON][FILE CONTENTS]
假设 JSON 永远不会大于 4GB（如果是，您将遇到更大的问题，因为解析它会是一场噩梦），[=13 就足够了=] 的固定 4 字节（32 位）作为无符号整数（如果您不希望 JSON 超过 64KB，您甚至可以使用 16 位）所以整个策略将在客户端工作如：
- 创建负载
- 将其编码为 JSON，然后使用 UTF-8 编码将其编码为
bytes - 获取上述包的长度，作为流的前4个字节发送
- 发送JSON包裹
- 读取并发送文件内容
并且在服务器端执行相同的过程
- 读取接收数据的前 4 个字节以获得 JSON 负载长度
- 读取下一个字节数以匹配此长度
- 使用 UTF-8 将它们解码为字符串，然后解码 JSON 以获得有效负载
- 读取其余的流数据并将其存储到文件
或者在代码中，客户端：
import json
import os
import socket
import struct
BUFFER_SIZE = 4096 # a uniform buffer size to use for our transfers
# pick up an absolute path from the script folder, not necessary tho
file_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "downloads", "cat.png"))
# let's first prepare the payload to send over
payload = {"id": 12, "filename": os.path.basename(file_path), "message": "So cute!"}
# now JSON encode it and then turn it onto a bytes stream by encoding it as UTF-8
json_data = json.dumps(payload).encode("utf-8")
# then connect to the server and send everything
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: # create a socket
print("Connecting...")
s.connect(("127.0.0.1", 1234)) # connect to the server
# first send the JSON payload length
print("Sending `{filename}` with a message: {message}.".format(**payload))
s.sendall(struct.pack(">I", len(json_data))) # pack as BE 32-bit unsigned int
# now send the JSON payload itself
s.sendall(json_data) # let Python deal with the buffer on its own for the JSON...
# finally, open the file and 'stream' it to the socket
with open(file_path, "rb") as f:
chunk = f.read(BUFFER_SIZE)
while chunk:
s.send(chunk)
chunk = f.read(BUFFER_SIZE)
# alternatively, if you're using Python 3.5+ you can just use socket.sendfile() instead
print("Sent.")
和服务器：
import json
import os
import socket
import struct
BUFFER_SIZE = 4096 # a uniform buffer size to use for our transfers
target_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "fileCache"))
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind(("127.0.0.1", 1234)) # bind to the 1234 port on localhost
s.listen(0) # allow only one connection so we don't have to deal with data separation
while True:
print("Waiting for a connection...")
connection, address = s.accept() # wait for and accept the incoming connection
print("Connection from `{}` accepted.".format(address))
# read the starting 32 bits and unpack them into an int to get the JSON length
json_length = struct.unpack(">I", connection.recv(4))[0]
# now read the JSON data of the given size and JSON decode it
json_data = b"" # initiate an empty bytes structure
while len(json_data) < json_length:
chunk = connection.recv(min(BUFFER_SIZE, json_length - len(json_data)))
if not chunk: # no data, possibly broken connection/bad protocol
break # just exit for now, you should deal with this case in production
json_data += chunk
payload = json.loads(json_data.decode("utf-8")) # JSON decode the payload
# now read the rest and store it into a file at the target path
file_path = os.path.join(target_path, payload["filename"])
with open(file_path, "wb") as f: # open the target file for writing...
chunk = connection.recv(BUFFER_SIZE) # and stream the socket data to it...
while chunk:
f.write(chunk)
chunk = connection.recv(BUFFER_SIZE)
# finally, lets print out that we received the data
print("Received `{filename}` with a message: {message}".format(**payload))
注意：请记住，这是 Python 3.x 代码 - 对于 Python 2.x，您必须自己处理上下文管理，而不是让with ... 阻止 open/close 你的套接字。
仅此而已。当然，在真实的环境中，你需要处理断开连接、多个客户端等。但这是底层过程。
感谢大家的帮助，我终于用base64搞定了。 我在堆栈溢出时在这里找到了答案，我忘记了 link 但它在这里。
在使用 json.dumps 之前，我不得不像这样编码和解码文件。
base64_bytes = b64encode(l)
myFile = base64_bytes.decode("utf-8")
这是一个工作示例：
客户：
import os
from base64 import b64encode
import json
import socket
currentPath = os.path.dirname(os.path.abspath(__file__)) + "\downloads\"
with open(currentPath + "cat.png", "rb") as f:
l = f.read()
base64_bytes = b64encode(l)
myFile = base64_bytes.decode("utf-8")
data = {"id": "12", "filename": "cat.png", "message": "So cute!", "file": myFile}
dataToSend = json.dumps(data).encode("utf-8")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 1234))
s.sendall(dataToSend)
s.close()
服务器：
import socket
import json
import os
import sys
import base64
currentPath = os.path.dirname(os.path.abspath(__file__)) + "\fileCache\"
tempData = bytearray()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1", 1234))
s.listen(5)
conn, addr = s.accept()
while True:
dataReceived = conn.recv(4096)
if sys.getsizeof(dataReceived) > 17:
tempData = tempData + dataReceived
else:
data = json.loads(tempData.decode("utf-8"))
break
myFile = base64.b64decode(data["file"])
with open(currentPath + data["filename"], "wb") as f:
f.write(myFile)
f.close()