为单个 ID 运行脚本时正确获取 html 内容,但在尝试循环时,经过几次迭代后,出现错误

Geting html content properly when the script is runned for individual IDs, but when trying on loop, after few iterations, getting error

from bs4 import BeautifulSoup
import requests
import pandas as pd
import time

columns=['ID','Toll Plaza Name','State','Type Of Vehicle','Single Journey','Return Journey','Monthly Pass','Commercial Vehicle Registered within the district of plaza']
df=pd.DataFrame(columns=columns)
row_count=0


URL = "https://tis.nhai.gov.in/TollInformation.aspx?TollPlazaID="


for i in range(0,491):

    URL+=str(i+4)
    print(i+4)
    r = requests.get(URL)
    soup = BeautifulSoup(r.content, 'html.parser')
    div = soup.find('div', attrs = {'id':'DivContaint','class':'content'})
    #print(div)
    name= div.find('lable').text
    #print('Name: ',name)
    state=div.find('div', attrs = {'class':'PA15'}).p.text.split('in')[1].split('Stretch')[0].strip()
    #print('State: ', state)
    table=div.find('table',attrs={'class':'tollinfotbl'})
    rows=table.find_all('tr')
    for j in range(1,14,2):
        df.loc[row_count,'ID']=i+4
        df.loc[row_count,'Toll Plaza Name']=name
        df.loc[row_count,'State']=state
        typeOfVehicle=rows[j].th.text
        df.loc[row_count,'Type Of Vehicle']=typeOfVehicle.strip()
        td=rows[j].find_all('td')
        df.loc[row_count,'Single Journey']=td[0].text
        df.loc[row_count,'Return Journey']=td[1].text
        df.loc[row_count,'Monthly Pass']=td[2].text
        df.loc[row_count,'Commercial Vehicle Registered within the district of plaza']=td[3].text

        row_count+=1
    del r, soup, div, name, state, table, td    
    df.to_excel("output.xlsx")  
    #time.sleep(2)

输出: 4个 5个 6个 7 8

AttributeError Traceback(最后一次调用) 在 15 div = soup.find('div', 属性 = {'id':'DivContaint','class':'content'}) 16#打印(div) ---> 17 名称= div.find('lable').text 18 #print('Name: ',姓名) 19 state=div.find('div', attrs = {'class':'PA15'}).p.text.split('in')[1].split( 'Stretch')[0].strip()

AttributeError: 'NoneType' 对象没有属性 'text'

因此,在第 8 次 id 迭代中,我得到的 div 内容是空的。但是,如果我只尝试那个特定的 id,即 8。获得正确的输出。

按照您的逻辑,它不是遍历每个“广场”,而是将其串联起来。通过那个“第 8 个”广场,它试图去 url 'https://tis.nhai.gov.in/TollInformation.aspx?TollPlazaID=45678',但它不存在。

此外,如果您只是将每个 i 加 4,为什么不将范围从 (0,491) 调整为 (4,495)

我还会考虑使用 pandas 来解析 table。个人喜好而已。

最后,您是否意识到在每次迭代之后,您都在覆盖您的 output.xlsx 文件?我想这很好,因为你只是附加到它,但不确定那是你想要的。

这是第一页第 4-15 页的示例代码(我不会坐下来完成所有 490 多次迭代)。

from bs4 import BeautifulSoup
import requests
import pandas as pd
import time

root_url = "https://tis.nhai.gov.in/TollInformation.aspx?TollPlazaID="

final_df = pd.DataFrame()
for i in range(4,495):
    url = f'{root_url}{i}'
    print(i)
    r = requests.get(url)
    
    df = pd.read_html(r.text)[0].dropna(how='all', axis=0)
    
    soup = BeautifulSoup(r.content, 'html.parser')
    div = soup.find('div', attrs = {'id':'DivContaint','class':'content'})
    #print(div)
    name= div.find('lable').text
    #print('Name: ',name)
    state=div.find('div', attrs = {'class':'PA15'}).p.text.split('in')[1].split('Stretch')[0].strip()
    #print('State: ', state)
    
    df['ID'] = i
    df['Toll Plaza Name'] = name
    df['State'] = state
    
    final_df = final_df.append(df).reset_index(drop=True)

    #time.sleep(2)
    
final_df.to_excel("output.xlsx", index=False)  

输出:

print(final_df.to_string())
        Type of vehicle  Single Journey  Return Journey  Monthly Pass  Commercial Vehicle Registered within the district of plaza  ID                                   Toll Plaza Name                                         State
0          Car/Jeep/Van            95.0           145.0        2860.0                                                         NaN   4   Thakurtola (End of Durg Bypass) ( BOT (Toll) )                                   Chhattisgarh
1                   LCV           165.0           250.0        5005.0                                                         NaN   4   Thakurtola (End of Durg Bypass) ( BOT (Toll) )                                   Chhattisgarh
2             Bus/Truck           335.0           500.0       10010.0                                                         NaN   4   Thakurtola (End of Durg Bypass) ( BOT (Toll) )                                   Chhattisgarh
3   Upto 3 Axle Vehicle           535.0           805.0       16090.0                                                         NaN   4   Thakurtola (End of Durg Bypass) ( BOT (Toll) )                                   Chhattisgarh
4           4 to 6 Axle           535.0           805.0       16090.0                                                         NaN   4   Thakurtola (End of Durg Bypass) ( BOT (Toll) )                                   Chhattisgarh
5               HCM/EME           535.0           805.0       16090.0                                                         NaN   4   Thakurtola (End of Durg Bypass) ( BOT (Toll) )                                   Chhattisgarh
6        7 or more Axle           535.0           805.0       16090.0                                                         NaN   4   Thakurtola (End of Durg Bypass) ( BOT (Toll) )                                   Chhattisgarh
7          Car/Jeep/Van            55.0            80.0        1795.0                                                        25.0   5                       Bankapur ( Public Funded )                                      Karnataka
8                   LCV            85.0           130.0        2905.0                                                        45.0   5                       Bankapur ( Public Funded )                                      Karnataka
9             Bus/Truck           180.0           275.0        6080.0                                                        90.0   5                       Bankapur ( Public Funded )                                      Karnataka
10  Upto 3 Axle Vehicle           200.0           300.0        6635.0                                                       100.0   5                       Bankapur ( Public Funded )                                      Karnataka
11          4 to 6 Axle           285.0           430.0        9540.0                                                       145.0   5                       Bankapur ( Public Funded )                                      Karnataka
12              HCM/EME           285.0           430.0        9540.0                                                       145.0   5                       Bankapur ( Public Funded )                                      Karnataka
13       7 or more Axle           350.0           525.0       11610.0                                                       175.0   5                       Bankapur ( Public Funded )                                      Karnataka
14         Car/Jeep/Van           100.0           145.0        3270.0                                                         NaN   6                      Hirebagewadi ( BOT (Toll) )                                      Karnataka
15                  LCV           160.0           240.0        5280.0                                                         NaN   6                      Hirebagewadi ( BOT (Toll) )                                      Karnataka
16            Bus/Truck           330.0           500.0       11065.0                                                         NaN   6                      Hirebagewadi ( BOT (Toll) )                                      Karnataka
17  Upto 3 Axle Vehicle           520.0           780.0       17350.0                                                         NaN   6                      Hirebagewadi ( BOT (Toll) )                                      Karnataka
18          4 to 6 Axle           520.0           780.0       17350.0                                                         NaN   6                      Hirebagewadi ( BOT (Toll) )                                      Karnataka
19              HCM/EME           520.0           780.0       17350.0                                                         NaN   6                      Hirebagewadi ( BOT (Toll) )                                      Karnataka
20       7 or more Axle           635.0           950.0       21120.0                                                         NaN   6                      Hirebagewadi ( BOT (Toll) )                                      Karnataka
21         Car/Jeep/Van            60.0             NaN           NaN                                                         NaN   7          Durg Bypass (Dhamdanaka) ( BOT (Toll) )                                   Chhattisgarh
22                  LCV           125.0             NaN           NaN                                                         NaN   7          Durg Bypass (Dhamdanaka) ( BOT (Toll) )                                   Chhattisgarh
23            Bus/Truck           185.0             NaN           NaN                                                         NaN   7          Durg Bypass (Dhamdanaka) ( BOT (Toll) )                                   Chhattisgarh
24  Upto 3 Axle Vehicle           280.0             NaN           NaN                                                         NaN   7          Durg Bypass (Dhamdanaka) ( BOT (Toll) )                                   Chhattisgarh
25          4 to 6 Axle           280.0             NaN           NaN                                                         NaN   7          Durg Bypass (Dhamdanaka) ( BOT (Toll) )                                   Chhattisgarh
26              HCM/EME           375.0             NaN           NaN                                                         NaN   7          Durg Bypass (Dhamdanaka) ( BOT (Toll) )                                   Chhattisgarh
27       7 or more Axle           280.0             NaN           NaN                                                         NaN   7          Durg Bypass (Dhamdanaka) ( BOT (Toll) )                                   Chhattisgarh
28         Car/Jeep/Van            30.0            45.0         980.0                                                        15.0   8                       Hattargi ( Public Funded )                                      Karnataka
29                  LCV            45.0            70.0        1580.0                                                        25.0   8                       Hattargi ( Public Funded )                                      Karnataka
30            Bus/Truck           100.0           150.0        3310.0                                                        50.0   8                       Hattargi ( Public Funded )                                      Karnataka
31  Upto 3 Axle Vehicle           110.0           160.0        3610.0                                                        55.0   8                       Hattargi ( Public Funded )                                      Karnataka
32          4 to 6 Axle           155.0           235.0        5190.0                                                        80.0   8                       Hattargi ( Public Funded )                                      Karnataka
33              HCM/EME           155.0           235.0        5190.0                                                        80.0   8                       Hattargi ( Public Funded )                                      Karnataka
34       7 or more Axle           190.0           285.0        6315.0                                                        95.0   8                       Hattargi ( Public Funded )                                      Karnataka
35         Car/Jeep/Van            75.0           115.0        2500.0                                                        40.0   9                        Kognoli ( Public Funded )                                      Karnataka
36                  LCV           120.0           180.0        4040.0                                                        60.0   9                        Kognoli ( Public Funded )                                      Karnataka
37            Bus/Truck           255.0           380.0        8465.0                                                       125.0   9                        Kognoli ( Public Funded )                                      Karnataka
38  Upto 3 Axle Vehicle           275.0           415.0        9235.0                                                       140.0   9                        Kognoli ( Public Funded )                                      Karnataka
39          4 to 6 Axle           400.0           595.0       13275.0                                                       200.0   9                        Kognoli ( Public Funded )                                      Karnataka
40              HCM/EME           400.0           595.0       13275.0                                                       200.0   9                        Kognoli ( Public Funded )                                      Karnataka
41       7 or more Axle           485.0           725.0       16165.0                                                       240.0   9                        Kognoli ( Public Funded )                                      Karnataka
42         Car/Jeep/Van            90.0           135.0        3015.0                                                        45.0  10                            Tundla ( BOT (Toll) )                                  Uttar Pradesh
43                  LCV           145.0           220.0        4870.0                                                        75.0  10                            Tundla ( BOT (Toll) )                                  Uttar Pradesh
44            Bus/Truck           305.0           460.0       10205.0                                                       155.0  10                            Tundla ( BOT (Toll) )                                  Uttar Pradesh
45  Upto 3 Axle Vehicle           335.0           500.0       11130.0                                                       165.0  10                            Tundla ( BOT (Toll) )                                  Uttar Pradesh
46          4 to 6 Axle           480.0           720.0       16000.0                                                       240.0  10                            Tundla ( BOT (Toll) )                                  Uttar Pradesh
47              HCM/EME           480.0           720.0       16000.0                                                       240.0  10                            Tundla ( BOT (Toll) )                                  Uttar Pradesh
48       7 or more Axle           585.0           870.0       19480.0                                                       290.0  10                            Tundla ( BOT (Toll) )                                  Uttar Pradesh
49         Car/Jeep/Van            25.0            40.0         895.0                                                        15.0  11                 Brahamarakotlu ( Public Funded )                                      Karnataka
50                  LCV            45.0            65.0        1445.0                                                        20.0  11                 Brahamarakotlu ( Public Funded )                                      Karnataka
51            Bus/Truck            90.0           135.0        3025.0                                                        45.0  11                 Brahamarakotlu ( Public Funded )                                      Karnataka
52  Upto 3 Axle Vehicle           100.0           150.0        3300.0                                                        50.0  11                 Brahamarakotlu ( Public Funded )                                      Karnataka
53          4 to 6 Axle           140.0           215.0        4745.0                                                        70.0  11                 Brahamarakotlu ( Public Funded )                                      Karnataka
54              HCM/EME           140.0           215.0        4745.0                                                        70.0  11                 Brahamarakotlu ( Public Funded )                                      Karnataka
55       7 or more Axle           175.0           260.0        5775.0                                                        85.0  11                 Brahamarakotlu ( Public Funded )                                      Karnataka
56         Car/Jeep/Van            25.0            35.0         785.0                                                        10.0  12      Cable Stayed Naini Bridge ( Public Funded )   i Bridge ( Public Funded )  Km 1.600 - NH-30
57                  LCV            40.0            55.0        1265.0                                                        20.0  12      Cable Stayed Naini Bridge ( Public Funded )   i Bridge ( Public Funded )  Km 1.600 - NH-30
58            Bus/Truck            80.0           120.0        2650.0                                                        40.0  12      Cable Stayed Naini Bridge ( Public Funded )   i Bridge ( Public Funded )  Km 1.600 - NH-30
59  Upto 3 Axle Vehicle            85.0           130.0        2890.0                                                        45.0  12      Cable Stayed Naini Bridge ( Public Funded )   i Bridge ( Public Funded )  Km 1.600 - NH-30
60          4 to 6 Axle           125.0           185.0        4155.0                                                        60.0  12      Cable Stayed Naini Bridge ( Public Funded )   i Bridge ( Public Funded )  Km 1.600 - NH-30
61              HCM/EME           125.0           185.0        4155.0                                                        60.0  12      Cable Stayed Naini Bridge ( Public Funded )   i Bridge ( Public Funded )  Km 1.600 - NH-30
62       7 or more Axle           150.0           230.0        5055.0                                                        75.0  12      Cable Stayed Naini Bridge ( Public Funded )   i Bridge ( Public Funded )  Km 1.600 - NH-30
63         Car/Jeep/Van            80.0           120.0        2655.0                                                        40.0  13                                    Bassi ( OMT )                                      Rajasthan
64                  LCV           130.0           195.0        4285.0                                                        65.0  13                                    Bassi ( OMT )                                      Rajasthan
65            Bus/Truck           270.0           405.0        8980.0                                                       135.0  13                                    Bassi ( OMT )                                      Rajasthan
66  Upto 3 Axle Vehicle           295.0           440.0        9795.0                                                       145.0  13                                    Bassi ( OMT )                                      Rajasthan
67          4 to 6 Axle           420.0           635.0       14080.0                                                       210.0  13                                    Bassi ( OMT )                                      Rajasthan
68              HCM/EME           420.0           635.0       14080.0                                                       210.0  13                                    Bassi ( OMT )                                      Rajasthan
69       7 or more Axle           515.0           770.0       17145.0                                                       255.0  13                                    Bassi ( OMT )                                      Rajasthan
70         Car/Jeep/Van            70.0           100.0        2260.0                                                        35.0  14                                    Aroli ( OMT )                                      Rajasthan
71                  LCV           110.0           165.0        3645.0                                                        55.0  14                                    Aroli ( OMT )                                      Rajasthan
72            Bus/Truck           230.0           345.0        7640.0                                                       115.0  14                                    Aroli ( OMT )                                      Rajasthan
73  Upto 3 Axle Vehicle           250.0           375.0        8335.0                                                       125.0  14                                    Aroli ( OMT )                                      Rajasthan
74          4 to 6 Axle           360.0           540.0       11985.0                                                       180.0  14                                    Aroli ( OMT )                                      Rajasthan
75              HCM/EME           360.0           540.0       11985.0                                                       180.0  14                                    Aroli ( OMT )                                      Rajasthan
76       7 or more Axle           440.0           655.0       14590.0                                                       220.0  14                                    Aroli ( OMT )                                      Rajasthan