requests.get 没有返回任何东西

Question

我正在尝试使用 requests.get

检索 url

import requests
from bs4 import BeautifulSoup

baseurl = "https://www.olx.com.eg/"

headers = {
'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36'
}

r = requests.get('https://www.olx.com.eg/jobs/')
soup = BeautifulSoup(r.content, 'lxml')


product_list = soup.findAll('div',class_ = 'ads__item')

print(product_list)

但它 returns 是一个空列表，因为它甚至没有打开 URL。

这里有什么问题？

Answer 1

将headers=参数添加到requests.get:

import requests
from bs4 import BeautifulSoup

baseurl = "https://www.olx.com.eg/"

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"
}

r = requests.get("https://www.olx.com.eg/jobs/", headers=headers)
soup = BeautifulSoup(r.content, "lxml")


product_list = soup.findAll("div", class_="ads__item")

print(len(product_list))

打印：

requests.get 没有返回任何东西

requests.get not returning anything

python

user-agent

beautifulsoup