python3 URLError 未知 url 输入 http

Question

我正在尝试使用 urllib.request.urlretrieve 和多处理模块来下载一些文件并对它们进行一些处理。但是，每次我尝试运行我的程序时，它都会给我错误：

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "./thumb.py", line 13, in download_and_convert
    filename, headers = urlretrieve(url)
  File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 463, in open
    response = self._open(req, data)
  File "/usr/lib/python3.4/urllib/request.py", line 486, in _open
    'unknown_open', req)
  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 1252, in unknown_open
    raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: http>
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./thumb.py", line 27, in <module>
    pool.map(download_and_convert, enumerate(csvr))
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 599, in get
    raise self._value
urllib.error.URLError: <urlopen error unknown url type: http>

好像呛到的url是http://phytoimages.siu.edu/users/vitt/10_27_06_2/Equisetumarvense.JPG。这是我的代码：

#!/usr/bin/env python3

from subprocess import Popen
from sys import argv, stdin
import csv
from multiprocessing import Pool
from urllib.request import urlretrieve

def download_and_convert(args):
    num, url_list = args
    url = url_list[0]
    try:
        filename, headers = urlretrieve(url)
    except:
        print(url)
        raise
    Popen(["convert", filename, "-resize", "250x250",\
           str(num)+'.'+url.split('.')[-1]])

if __name__ == "__main__":
    csvr = csv.reader(open(argv[1]))

    if(len(argv) > 2): nprocs = argv[2]
    else: nprocs = None

    pool = Pool(processes=nprocs)
    pool.map(download_and_convert, enumerate(csvr))

我不知道为什么会出现这个错误。可能是因为我正在使用多处理吗？如果有人能帮助我，我将不胜感激。

编辑：这是它尝试处理的第一个 url，如果我更改它也不会更改错误。

Answer 1

检查这段代码

>>> import urllib.parse
>>> urllib.parse.quote(':')
'%3A'

如您所见，urllib 对“:”字符的解释很奇怪。巧合的是你的程序挂断了你。

试试 urllib.parse.urlencode() 应该会让你走上正轨。

Answer 2

在评论的帮助下，我找到了解决方案。看来问题是 csv 模块在字节顺序标记 (BOM) 上被绊倒了。我能够按照建议 here.

打开文件 encoding='utf-8-sig' 来修复它

python3 URLError 未知 url 输入 http

python3 URLError unknown url type http

python

python-3.x

urllib