尝试在 scraping group 上进行练习。为什么 returns 结果少了？

Question

我是 Python 抓取的新手，所以作为练习的一部分，我尝试了一些其他网站，这些网站通常根本不返回数据，但是当我查看 Groupon 时，我发现 urllib 只有 returns 前 8 个结果，而浏览器页面上有 36 个结果。

我正在使用 urllib 和 BS4。下面是代码

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
req = Request('https://www.groupon.com/browse/chicago?category=beauty-and-spas')
req.add_header('User-Agent',
               'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36')
try:
    with urlopen(req) as response:
        htmlcontent = response.read().decode('utf-8')
except:
    htmlcontent = None
soup = BeautifulSoup(htmlcontent, 'lxml')
all_links = soup.find('div', { 'id': 'pull-results' }).select('figure > div > a')

有人可以告诉我，我在代码中缺少什么才能提取所有数据吗？如果这不起作用或不应该起作用，那么我们是否将硒作为唯一的选择？

Answer 1

另外28个链接是动态加载的，所以urllib不支持。但是，您可以通过向以下地址发送 GET 请求来抓取它们：

https://www.groupon.com/partial/browse/get-lazy-loaded-cards?category=beauty-and-spas&_csrf=P6rFPl1o-xDta8uOABKo_9LOiUajyK9bieMg

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen


req = Request(
    "https://www.groupon.com/partial/browse/get-lazy-loaded-cards?category=beauty-and-spas&_csrf=P6rFPl1o-xDta8uOABKo_9LOiUajyK9bieMg"
)
req.add_header(
    "User-Agent",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36",
)

with urlopen(req) as response:
    htmlcontent = response.read().decode("utf-8")
    soup = BeautifulSoup(htmlcontent, "lxml")

    for tag in soup.find_all(class_=r'\"cui-content\"'):
        try:
            link = tag.find("a")["href"]
        except TypeError:
            continue
        print(link.replace('\"', "").replace("\n", ""))

输出：

https://www.groupon.com/deals/yupo-health-2?deal_option=75076cab-9268-4c2d-9459-e79a559dfed6
https://www.groupon.com/deals/infinity-laser-spa-2-4?deal_option=4dc14f7d-29ac-45e1-a664-ea76ddc44718
https://www.groupon.com/deals/dr-laser-nyc-3?deal_option=232325a0-8b3f-4fa9-b42b-7dc668ff7474
...
...

Answer 2

尝试以下方法获取所有项目及其遍历下一页的链接：

import requests
from bs4 import BeautifulSoup

base_link = 'https://www.groupon.com/browse/chicago?category=beauty-and-spas'
url = 'https://www.groupon.com/partial/browse/get-paginated-cards?'

params = {
    'category': 'beauty-and-spas',
    'page': 1
}

with requests.Session() as s:
    s.headers['user-agent'] = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'
    r = s.get(base_link)
    params['_csrf'] = r.cookies['_csrf']

    while True:
        print("current page----------->",params['page'])
        res = s.get(url,params=params)
        soup = BeautifulSoup(res.json()[0]['cardSlotHtml'],"lxml")
        if not soup.select_one("figure[data-pingdom-info='purchasable-deal']"):break
        for item in soup.select("figure[data-pingdom-info='purchasable-deal']"):
            item_title = item.select_one(".cui-udc-details .cui-udc-title").get_text(strip=True)
            item_link = item.select_one(".cui-content > a[href]").get("href")
            print(item_title,item_link)

        params['page']+=1

尝试在 scraping group 上进行练习。为什么 returns 结果少了？

Tried scraping group on for practice only. Why it returns less results?

python

beautifulsoup

web-scraping

urllib3