滚动到下一页并提取数据

Question

我正在尝试从 'https://www.bbc.com/news/coronavirus'

的最新更新中提取所有正文

我已经成功地从第一页中提取正文文本（50 页中的 1 页）。

我想滚动到下一页并再次执行此过程。

这是我写的代码。

from bs4 import BeautifulSoup as soup
import requests

links = []
header = []
body_text = []

r = requests.get('https://www.bbc.com/news/coronavirus')
b = soup(r.content,'lxml')

# Selecting Latest update selection
latest = b.find(class_="gel-layout__item gel-3/5@l")
# Getting title
for news in latest.findAll('h3'):
    header.append(news.text)
    #print(news.text)

# Getting sub-links
for news in latest.findAll('h3',{'class':'lx-stream-post__header-title gel-great-primer-bold qa-post-title gs-u-mt0 gs-u-mb-'}):
    links.append('https://www.bbc.com' + news.a['href'])

# Entering sub-links and extracting texts
for link in links:
    page = requests.get(link)
    bsobj = soup(page.content,'lxml')

    for news in bsobj.findAll('div',{'class':'ssrcss-18snukc-RichTextContainer e5tfeyi1'}):
        body_text.append(news.text.strip())
        #print(news.text.strip())

我应该如何滚动到下一页？

Answer 1

不确定您要查找的是什么文本，但您可以浏览 api。

import requests

url = 'https://push.api.bbci.co.uk/batch'
headers = {'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Mobile Safari/537.36'}

for page in range(1,51):
    payload = '?t=%2Fdata%2Fbbc-morph-lx-commentary-data-paged%2Fabout%2F63b2bbc8-6bea-4a82-9f6b-6ecc470d0c45%2FisUk%2Ffalse%2Flimit%2F20%2FnitroKey%2Flx-nitro%2FpageNumber%2F{page}%2Fversion%2F1.5.4?timeout=5'.format(page=page)

    jsonData = requests.get(url+payload, headers=headers).json()

    results = jsonData['payload'][0]['body']['results']
    for result in results:
        print(result['title'])
        print('\t',result['summary'],'\n')

滚动到下一页并提取数据

Scroll to next page and extract data

web-scraping

beautifulsoup

python-requests

web-crawler