抓取网站只得到 20 个结果，但应该有更多

Question

我想了解为什么我的 soup.find_all 只有 returns 20 个结果，尽管应该有更多结果。

#Site
url = 'https://www.fairprice.com.sg/promotions'

#requests
headers = {'user-agent': 'my-agent/1.0.1'}
r = requests.get(url, headers=headers)

#bs4
soup = BeautifulSoup(r.text,'html.parser')
mydivs = soup.find_all("div", {"class": "sc-1plwklf-10 jNMebL"})

len(mydivs)
Output: 40

搜索了各个站点，发现可能是页面向下滚动后加载了产品。有解决方法吗？感谢帮助。

Answer 1

您将获得 20 个项目作为输出，因为其余数据由 JavaScript 作为 GET 方法从 API 动态填充。

import requests
import pandas as pd

experiments = ('searchVariant-B,timerVariant-Z,inlineBanner-A,'
               'substitutionBSVariant-A,gv-A,SPI-Z,SNLI-B,SC-B,'
               'shelflife-C,A')

params = dict(experiments=experiments,
              includeTagDetails='true',
              orderType='DELIVERY',
              pageType='promotion',
              storeId=165,
              )

url = 'https://website-api.omni.fairprice.com.sg/api/product/v2'

result=[]
for params['page'] in range(1, 6):
    data = requests.get(url, params).json()
    for item in data['data']['product']:
        name = item['name']
        result.append({"Product name": name})
df = pd.DataFrame(result)
print(df)

输出：

                                    Product name
0                               Pasar Fresh Eggs
1                     Meiji Fresh Milk - Regular
2     FairPrice DeluxSoft Bathroom Tissue (3ply)
3                              Fresh Blueberries
4         UFC Refresh 100% Natural Coconut Water
..                                           ...
95                Magnum Mini Ice Cream - Almond
96  Clorox Toilet Bowl Cleaner Tablets - Tru Blu
97     Kleenex Facial Tissue Box - Floral (3ply)
98                    Fairprice Scouring Sponge 
99        Prego Pasta Sauce - Traditional Tomato

[100 rows x 1 columns]

screenshot of api url

抓取网站只得到 20 个结果，但应该有更多

Scraping website only gets me 20 results, but there should be more

python

beautifulsoup

web-scraping

python-requests