TypeError: descriptor 'split' requires a 'str' object but received a 'bytes'

Question

我正在尝试使用 Github 上可用的 python 脚本从 ESPN Cricinfo 抓取数据。代码如下。

import urllib.request as ur
import csv
import sys
import time
import os
import unicodedata
from urllib.parse import urlparse
from bs4 import BeautifulSoup

BASE_URL = 'http://www.espncricinfo.com'
for i in range(0, 6019):
url = 'http://search.espncricinfo.com/ci/content/match/search.html?search=first%20class;all=1;page='
    soupy = BeautifulSoup(ur.urlopen(url + str(i)).read())

    time.sleep(1)
    for new_host in soupy.findAll('a', {'class' : 'srchPlyrNmTxt'}):
        try:
            new_host = new_host['href']
        except:
            continue
        odiurl = BASE_URL + urlparse(new_host).geturl()
        new_host = unicodedata.normalize('NFKD', new_host).encode('ascii','ignore')
        print (new_host)
        print (str.split(new_host, "/"))[4]
        html = urllib2.urlopen(odiurl).read()
        if html:
            with open('espncricinfo-fc/{0!s}'.format(str.split(new_host, "/")[4]), "wb") as f:
                f.write(html)

错误就在这一行。

print (str.split(new_host, "/"))[4]

TypeError：描述符 'split' 需要一个 'str' 对象但收到了一个 'bytes' 我们将不胜感激您的任何帮助。谢谢

Answer 1

使用

str.split(new_host.decode("utf-8"), "/")[4]

.decode("utf-8") 显然是最重要的部分。这会将您的 byte 对象变成一个字符串。

另一方面，请注意 urllib2（顺便说一下，您正在使用但未导入）不再使用（参见 this）。相反，您可以使用 from urllib.request import urlopen.

编辑：这是完整代码，不会给您您在问题中描述的错误。我要强调的是，因为没有以前创建的文件，with open(...) 语句会给你一个 FileNotFoundError.

import urllib.request as ur
import csv
import sys
import time
import os
import unicodedata
from urllib.parse import urlparse
from bs4 import BeautifulSoup
from urllib.request import urlopen

BASE_URL = 'http://www.espncricinfo.com'
for i in range(0, 6019):
    url = 'http://search.espncricinfo.com/ci/content/match/search.html?search=first%20class;all=1;page='
    soupy = BeautifulSoup(ur.urlopen(url + str(i)).read())

    time.sleep(1)
    for new_host in soupy.findAll('a', {'class' : 'srchPlyrNmTxt'}):
        try:
            new_host = new_host['href']
        except:
            continue
        odiurl = BASE_URL + urlparse(new_host).geturl()
        new_host = unicodedata.normalize('NFKD', new_host).encode('ascii','ignore')
        print(new_host)
        print(str.split(new_host.decode("utf-8"), "/")[4])
        html = urlopen(odiurl).read()
        if html:
            with open('espncricinfo-fc/{0!s}'.format(str.split(new_host.decode("utf-8"), "/")[4]), "wb") as f:
                f.write(html)

TypeError: descriptor 'split' requires a 'str' object but received a 'bytes'

TypeError: descriptor 'split' requires a 'str' object but received a 'bytes'

python

csv

beautifulsoup

scrape