景观跨度，在 DIV 内使用 Python

Question

尝试使用 Python 从以下网站收集 Amazon Prime 可用电影列表：

https://www.amazon.co.uk/s?i=instant-video&bbn=3010085031&rh=n%3A3010085031%2Cp_85%3A3282143031%2Cp_72%3A3289786031&dc&adult-product=0&field-genre=-family&field-review_count=3-&field-ways_to_watch=7448662031&p_n_entity_type=9739952031&qid=1557596014&qs-av_request_type=4&qs-is-prime-customer=0&rnid=3289782031&ref=sr_nr_p_72_4

我试图找到电影的标题，但返回的是 0。这一定是从页面加载开始的 JavaScript，我在 NETWORK 中查看过，但无法弄清楚我在看什么为了。

我尝试了以下代码：

from requests import get
url = 'https://www.amazon.co.uk/s?i=instant-video&bbn=3010085031&rh=n%3A3010085031%2Cp_85%3A3282143031%2Cp_72%3A3289786031&dc&adult-product=0&field-genre=-family&field-review_count=3-&field-ways_to_watch=7448662031&p_n_entity_type=9739952031&qid=1557596014&qs-av_request_type=4&qs-is-prime-customer=0&rnid=3289782031&ref=sr_nr_p_72_4'
response = get(url)
from bs4 import BeautifulSoup
html_soup = BeautifulSoup(response.text, 'lxml')
type(html_soup)
movie_containers = html_soup.find_all('span', class_ = 'a-size-medium     a-color-base a-text-normal')
print(type(movie_containers))
print(len(movie_containers))

然后我尝试循环：

for n in soup.find_all('span', {'class':'a-size-medium     a-color-base a-text-normal'}):
    title.append (n.text)

任何帮助都会很棒。谢谢你。

结果应提取每部电影的标题和 link。

Answer 1

您必须包含 header 获取请求。

import requests
from bs4 import BeautifulSoup

header = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36"}
a = requests.get("https://www.amazon.co.uk/s?i=instant-video&bbn=3010085031&rh=n%3A3010085031%2Cp_85%3A3282143031%2Cp_72%3A3289786031&dc&adult-product=0&field-genre=-family&field-review_count=3-&field-ways_to_watch=7448662031&p_n_entity_type=9739952031&qid=1557596014&qs-av_request_type=4&qs-is-prime-customer=0&rnid=3289782031&ref=sr_nr_p_72_4", headers=header)
b = BeautifulSoup(a.text, "html.parser")
for c in b.find_all("span", class_="a-size-medium"):
    print(c.text)

景观跨度，在 DIV 内使用 Python

Scape span, within DIV using Python

python

scrape