网页上可以看到评论，但是BeautifulSoup返回的html对象中没有评论部分

Question

我尝试使用 URL link 从网页中提取评论的文本内容，并使用 BeautifulSoup 进行抓取。当我点击URL link时，评论的内容在页面上是可见的，但是BeautifulSoup返回的HTML对象没有包含这些标签和文本。

我使用 BeautifulSoup 和 'html.parser' 进行网页抓取。我成功提取了给定网页中视频的likes/views/comments个数，但HTML文件中没有包含评论部分的信息。我用的浏览器是Chrome，系统是Ubuntu18.04.1 LTS.

这是我使用的代码（在 python 中）：

from urllib.request import urlopen
from bs4 import BeautifulSoup
import os

webpage_link = "https://www.airvuz.com/video/Majestic-Beast-Nanuk?id=59b2a56141ab4823e61ea901"

try:
    page = urlopen(webpage_link)
except urllib.error.HTTPError as err:  # webpage cannot be found
    print("ERROR! %s" %(webpage_link))

soup = BeautifulSoup(page, 'html.parser')

预期的结果是 soup 对象包含网页上可见的所有内容，尤其是评论的文本内容（如 "Not being there I enjoyed a lot seeing the life style of white bear. Thanks to the provider for such documentary." 和 "WOOOW... amazing..."）；但是，我在 soup 对象中找不到相应的节点。如有任何帮助，我们将不胜感激！

Answer 1

评论由 JavasSript 通过 ajax 请求生成。您可以发送相同的请求并从 json 响应中获取评论。您可以使用检查工具中的网络选项卡找到请求。

from urllib.request import urlopen
from bs4 import BeautifulSoup, Comment
import json
webpage_link = "https://www.airvuz.com/api/comments/video/59b2a56141ab4823e61ea901?page=1&limit=20"
page = urlopen(webpage_link).read()
comments_json=data = json.loads(page)
for comment_info in comments_json['data']:
    print(comment_info['comment'].strip())

输出

Not being there I enjoyed a lot seeing the life style of white bear. Thanks to the provider for  such documentary.
WOOOW... amazing...
I've been photographing polar bears for years, but to see this footage from a drones perspective was epic! Well done and congratz on the Nominee! Well deserved.
You are da man Florian!
Absolutely outstanding!
This is incredible
jaw dropping
This is wow amazing, love it.
So cool! Did the bears react to the drone at all?
Congratulations! It's awesome! I am watching in tears....
Awesome!
perfect video awesome
It is very, very beautiful !!! Sincere congratulations
Made my day, exquisite, thank you
Wow
Super!
Marvelous!
Man this is incredible!
Material is good, but  edi is bad. This history about  beer's family...
Muy bueno!

网页上可以看到评论，但是BeautifulSoup返回的html对象中没有评论部分

Comments are visible on the webpage, but the html object returned by BeautifulSoup did not contain the comment parts

python

web-scraping

beautifulsoup

data-extraction