分页不遍历页面

Question

想要遍历此 url ""url = "https://www.iata.org/en/about/members/airline-list/"" 中的所有页面并将结果转储到.csv 文件。

如何将实现一段代码来遍历页面包含在下面的当前代码中？

import requests
import pandas as pd
from bs4 import BeautifulSoup
from urllib.request import Request

url = 'https://www.iata.org/en/about/members/airline-list/'
req = Request(url , headers = {
                            'accept':'*/*',
                            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36'})

data = []

while True:
    print(url)
    html = requests.get(url)
    soup = BeautifulSoup(html.text, 'html.parser')
    data.append(pd.read_html(soup.select_one('table.datatable').prettify())[0])

    if soup.select_one('span.pagination-link.is-active + div a[href]'):
        url = soup.select_one('span.pagination-link.is-active + div a')['href']
    else:
        break
df = pd.concat(data)
df.to_csv('airline-list.csv',encoding='utf-8-sig',index=False)

Answer 1

试试这个方法：

for i in range(1, 30):
    url = f'https://www.iata.org/en/about/members/airline-list/?page={i}&search=&ordering=Alphabetical'
    html = requests.get(url)
    soup = BeautifulSoup(html.text, 'html.parser')
    data.append(pd.read_html(soup.select_one('table.datatable').prettify())[0])

Answer 2

要动态获取数据，请使用：

import pandas as pd
import requests
import bs4

url = 'https://www.iata.org/en/about/members/airline-list/?page={page}&search=&ordering=Alphabetical'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36'}

# Total number of pages
html = requests.get(url.format(page=1), headers=headers)
soup = bs4.BeautifulSoup(html.text)
pages = int(soup.find_all('a', {'class': 'pagination-link'})[-2].text)

data = []
for page in range(1, pages+1):
    html = requests.get(url.format(page=page, headers=headers))
    data.append(pd.read_html(html.text)[0])   
df = pd.concat(data)

输出：

>>> df
             Airline Name IATA Designator 3 digit code ICAO code           Country / Territory
0                 ABX Air              GB          832       ABX                 United States
1         Aegean Airlines              A3          390       AEE                        Greece
2              Aer Lingus              EI           53       EIN                       Ireland
3          Aero Republica              P5          845       RPB                      Colombia
4                Aeroflot              SU          555       AFL            Russian Federation
..                    ...             ...          ...       ...                           ...
3                 WestJet              WS          838       WJA                        Canada
4   White coloured by you              WI           97       WHT                      Portugal
5                 Wideroe              WF          701       WIF                        Norway
6         Xiamen Airlines              MF          731       CXA  China (People's Republic of)
7      YTO Cargo Airlines              YG          860       HYT  China (People's Republic of)

[288 rows x 5 columns]

分页不遍历页面

Pagination not iterating over pages

python

csv

pagination

beautifulsoup

pandas