我正在尝试使用 beautfiul soup 抓取多个页面，但代码不断为每个页面返回相同的数据

Question

我正在尝试使用 Python 和漂亮的汤来抓取 Steam 网站上的特价商品。我正在尝试使用 for 循环从多个页面中抓取数据。我附上了下面的 Python 代码。非常感谢任何帮助。提前致谢。

 game_lis = set([])

    for page in range(0,4):
        page_url = "https://store.steampowered.com/specials#p=" +str(page)+"&tab=TopSellers"
        #print(page_url)
        steam_games = requests.get(page_url)
        soup = BeautifulSoup(steam_games.text, 'lxml')
        s_game_offers = soup.findAll('a', class_='tab_item')
        print(page_url)

        for game in s_game_offers:
            title = game.find('div',class_='tab_item_name')
            discount = game.find('div',class_='discount_pct')
            game_lis.add(title.text)
            print(title.text+":"+discount.text)

Answer 1

该页面是通过 JavaScript 从不同的 URL 加载的，因此 beautifulsoup 看不到它。您可以使用下一个示例如何加载不同的页面：

import requests
from bs4 import BeautifulSoup

api_url = "https://store.steampowered.com/contenthub/querypaginated/specials/TopSellers/render/"

params = {
    "query": "",
    "start": "0",
    "count": "15",
    "cc": "SK",  # <-- probably change code here
    "l": "english",
    "v": "4",
    "tag": "",
}

for page in range(0, 4):
    params["start"] = 15 * page

    steam_games = requests.get(api_url, params=params)
    soup = BeautifulSoup(steam_games.json()["results_html"], "lxml")
    s_game_offers = soup.findAll("a", class_="tab_item")

    for game in s_game_offers:
        title = game.find("div", class_="tab_item_name")
        discount = game.find("div", class_="discount_pct")
        print(title.text + ":" + discount.text)

    print("-" * 80)

打印：

F.I.S.T.: Forged In Shadow Torch:-10%
HITMAN 2 - Gold Edition:-85%
NieR:Automata™:-50%
Horizon Zero Dawn™ Complete Edition:-40%
Need for Speed™ Heat:-86%
Middle-earth: Shadow of War Definitive Edition:-80%
Batman: Arkham Collection:-80%
No Man's Sky:-50%
Legion TD 2 - Multiplayer Tower Defense:-20%
NieR Replicant™ ver.1.22474487139...:-35%
Days Gone:-20%
Mortal Kombat 11 Ultimate:-65%
Human: Fall Flat:-66%
Muse Dash - Just as planned:-30%
The Elder Scrolls Online - Blackwood:-50%
--------------------------------------------------------------------------------
The Elder Scrolls Online - Blackwood:-50%
Football Manager 2022:-10%
Age of Empires II: Definitive Edition:-33%
OCTOPATH TRAVELER™:-50%
DRAGON QUEST® XI S: Echoes of an Elusive Age™ - Definitive Edition:-35%
Witch It:-70%
Monster Hunter: World:-34%
NARUTO SHIPPUDEN: Ultimate Ninja STORM 4:-77%
MADNESS: Project Nexus:-10%
Mad Max:-75%
Outer Wilds:-40%
Middle-earth: Shadow of Mordor Game of the Year Edition:-75%
Age of Empires III: Definitive Edition:-33%
Ghostrunner:-60%
The Elder Scrolls® Online:-60%
--------------------------------------------------------------------------------

...

我正在尝试使用 beautfiul soup 抓取多个页面，但代码不断为每个页面返回相同的数据

I am trying to scrape multiple pages using beautfiul soup but the code keeps returning the same data for every page

python

web-scraping

beautifulsoup

data-analysis