BeautifulSoup Returns 空列表导致我的 Python 代码出现 IndexError

Question

我正在尝试使用 BeautifulSoup 进行网络抓取。我写的代码如下：

import requests
from bs4 import BeautifulSoup

response = requests.get("https://whosebug.com/questions")
soup = BeautifulSoup(response.text, "html.parser")

questions = soup.select(".question-summary")
print(type(questions[0]))

当我运行代码时，我收到以下错误消息：

print(type(questions[10]))
IndexError: list index out of range

然后我尝试打印如下列表：

import requests
from bs4 import BeautifulSoup

response = requests.get("https://whosebug.com/questions")
soup = BeautifulSoup(response.text, "html.parser")

questions = soup.select(".question-summary")
print(questions)

然后我得到一个空列表：[]

我做错了什么？

感谢您的回答。

Answer 1

.question-summary 是不正确的定位符，因为它是 id 的一部分，意味着每个 id value 都以 question-summary 开头。现在可以使用了。

import requests
from bs4 import BeautifulSoup

response = requests.get("https://whosebug.com/questions")
soup = BeautifulSoup(response.text, "html.parser")

questions = soup.select('[id^="question-summary"]')
print(questions)

输出：

1" data-post-type-id="1" id="question-summary-71715531">
<div class="s-post-summary--stats js-post-summary-stats">
<div class="s-post-summary--stats-item s-post-summary--stats-item__emphasized" title="Score of 0">
<span class="s-post-summary--stats-item-number">0</span>
<span class="s-post-summary--stats-item-unit">votes</span>
</div>
<div class="s-post-summary--stats-item" title="0 answers">
<span class="s-post-summary--stats-item-number">0</span>
<span class="s-post-summary--stats-item-unit">answers</span>
</div>
<div class="s-post-summary--stats-item" title="5 views">
<span class="s-post-summary--stats-item-number">5</span>
<span class="s-post-summary--stats-item-unit">views</span>
</div>
</div>
<div class="s-post-summary--content">
<h3 class="s-post-summary--content-title">
<a class="s-link" href="/questions/71715531/is-it-possible-to-draw-a-logistic-regression-graph-with-multiple-x-variable">Is it possible to draw a 
logistic regression graph with multiple x variable?</a>
</h3>
<div class="s-post-summary--content-excerpt">
                Currently, this is my X and V value. May I know is it possible to draw a logistic regression curve with X that has multiple column? Or I am required to draw multiple graphs to do so?
X = df1.drop(['...
            </div>
<div class="s-post-summary--meta">
<div class="s-post-summary--meta-tags tags js-tags t-python-3ûx t-machine-learning">
<a class="post-tag flex--item mt0 js-tagname-python-3ûx" href="/questions/tagged/python-3.x" rel="tag" title="show questions tagged 'python-3.x'">python-3.x</a> <a class="post-tag flex--item mt0 js-tagname-machine-learning" href="/questions/tagged/machine-learning" rel="tag" title="show questions tagged 'machine-learning'">machine-learning</a>
</div>
<div class="s-user-card s-user-card__minimal">
<a class="s-avatar s-avatar__16 s-user-card--avatar" href="/users/14128881/christopher-chua"> <div class="gravatar-wrapper-16" data-user-id="14128881">
<img ,="" alt="user avatar" class="s-avatar--image" height="16" src="https://lh6.googleusercontent.com/-Sn3B_E5hiJc/AAAAAAAAAAI/AAAAAAAAAAA/AMZuucl1oyfdhJiXhrx73JLYqzKAK9icag/photo.jpg?sz=32" width="16"/>
</div>
</a>
<div class="s-user-card--info">
<div class="s-user-card--link d-flex gs4">
<a class="flex--item" href="/users/14128881/christopher-chua">Christopher Chua</a>
</div>
<ul class="s-user-card--awards">
<li class="s-user-card--rep"><span class="todo-no-class-here" dir="ltr" title="reputation score ">7</span></li>
</ul>
</div>
<time class="s-user-card--time">asked <span class="relativetime" title="2022-04-02 07:03:06Z">13 mins ago</span></time>

..等等

BeautifulSoup Returns 空列表导致我的 Python 代码出现 IndexError

BeautifulSoup Returns empty list which leads to an IndexError in my Python code

python

screen-scraping

beautifulsoup

web