使用 urllib / sock.sendall(data) OSError 上传大文件（约 3GB）

Question

env: Mac OS X El Capitan / python 3.5.1

我要上传大约 3GB 大小的文件。

def read_in_chunks(file_object, chunk_size=4096):
    while True:
        data = file_object.read(chunk_size)
        if not data:
            break
        yield data

with open('3GB.mov', 'br') as f:
    data = b''.join([chunk for chunk in read_in_chunks(f)])

req = urllib.request.Request(url, data, headers)
response = urllib.request.urlopen(req)
the_page = response.read()

问题是..

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 362, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 913, in _send_output
    self.send(message_body)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/http/client.py", line 885, in send
    self.sock.sendall(data)
OSError: [Errno 22] Invalid argument

你能给我一些建议吗？

Answer 1

import requests
url = 'http://domain.com/api/upload'
with open('3GB.mov', 'br') as f:
    for chunk in read_in_chunks(f):

        offset = index + len(chunk)
        headers['Content-Type'] = 'application/octet-stream'
        headers['Content-length'] = content_size
        headers['Content-Range'] = 'bytes %s-%s/%s' % (index, offset, content_size)
        index = offset
        try:
            r = requests.post(url, data=chunk, headers=headers)
            print "r: %s, Content-Range: %s" % (r, headers['Content-Range'])
        except Exception, e:
            print e

使用 urllib / sock.sendall(data) OSError 上传大文件（约 3GB）

Large file(about 3GB) upload with urllib / sock.sendall(data) OSError

python

urllib

large-file-upload