如何使用 urllib 检查页面内容是否加载到 Python？
How to check if a page content is loaded in Python using urllib?
我正在尝试从 url 获取内容并使用 BeautyfulSoup.
解析响应
这个 url 在加载时会检索我最喜欢的关注列表项目，问题是当网站加载时需要几秒钟才能在 table 中显示数据，所以当我 运行 urlopen(my_url) 响应中没有 table，因此我的解析方法失败了。
我正在学习这门语言，所以我尽量保持简单，所以我想使用我已经在我的代码中设置的工具，所以根据我所拥有的，我想知道是否有办法等待，或检查内容何时准备好以便我能够获取数据（table 内容）。
这是我的代码：
from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as ureq
from urllib.error import URLError, HTTPError
URL = 'url route goes here' # In compliance to SO rules I've removed the website path
def get_dom_from_url():
try:
u_client = ureq(URL)
html = u_client.read()
u_client.close()
except HTTPError as e:
print(f'There has been an HTTP ERROR: {e.code}')
except URLError as e:
print(f'There has been a problem reaching the URL. ERROR: {e.code}')
finally:
print('''
DOM loaded!
''')
return html
dom = soup(get_dom_from_url(), 'html.parser')
# Crawl the dom object and get the table thead element
col_names = [col.text for col in dom.table.thead.find_all('th')]
col_names = col_names[1:-2]
col_names
这是错误信息：
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-102-625de133b2e2> in <module>
----> 1 col_names = [col.text for col in dom.table.thead.find_all('th')]
2 col_names = col_names[1:-2]
3 col_names
AttributeError: 'NoneType' object has no attribute 'thead'
当我在没有路由的情况下加载 url 时，上面的代码有效，但我需要它，因为我需要为我正在处理的 ETL 管道存储相同的数据。
如果仅使用 urllib 无法实现此目的，我想听听您的建议。
如评论所述，urllib.request相当古老，Selenium可以处理javascript:
from selenium import webdriver
from bs4 import BeautifulSoup as soup
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',options=options)
wd.get("https://coinmarketcap.com/watchlist/60321ee5b01cab343e1e37d6")
dom = soup(wd.page_source, 'html.parser')
col_names = [col.text for col in dom.table.thead.find_all('th')]
col_names = col_names[1:-2]
col_names
输出：
['Name', 'Price', '24h', '7d', 'Market Cap', 'Volume', 'Circulating Supply']
我不确定你的目的是什么，但你也可以直接将表加载到 pandas 而无需使用 BeautifulSoup:
import pandas as pd
df = pd.read_html(wd.page_source)[0]
df = df.iloc[:, 1:-2] # drop first and last two columns
输出df.head()：
| | Name | Price | 24h | 7d | Market Cap | Volume | Circulating Supply |
|---:|:-----------------|:-----------|:------|:--------|:-------------------|:---------------------------------|:---------------------|
| 0 | Bitcoin1BTC | ,515.75 | 3.67% | 17.00% | ,069,904,656,718 | ,190,502,8631,135,430 BTC | 18,634,643 BTC |
| 1 | Ethereum2ETH | ,972.76 | 1.32% | 7.03% | 5,668,814,800 | ,790,664,03815,658,144 ETH | 114,760,589 ETH |
| 2 | Binance Coin3BNB | 5.07 | 4.20% | 113.31% | ,505,990,437 | ,923,804,82135,249,242 BNB | 154,532,785 BNB |
| 3 | Polkadot4DOT | .69 | 4.23% | 39.72% | ,117,982,640 | ,555,973,830139,996,164 DOT | 910,079,701 DOT |
| 4 | Cardano5ADA | .14 | 8.25% | 31.00% | ,603,496,069 | ,299,401,9179,000,239,285 ADA | 31,112,484,646 ADA |
实际上你不需要在这里使用Selenium。数据以有效的 json 格式嵌入在 <script> 标记中的源 html 中。只需要解析：
import requests
from bs4 import BeautifulSoup
import json
import pandas as pd
url = 'https://coinmarketcap.com/watchlist/60321ee5b01cab343e1e37d6/'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
jsonStr = soup.find('script', {'id':'__NEXT_DATA__'}).text
jsonData = json.loads(jsonStr)
data = jsonData['props']['initialProps']['pageProps']['fetchedWatchlist']['cryptoCurrencies']
rows = []
for each in data:
quotes_row = each.pop('quotes')[0]
each.pop('tags')
if 'platform' in each.keys():
each.pop('platform')
each.update(quotes_row)
rows.append(each)
df = pd.DataFrame(rows)
输出：
print(df.to_string())
id name symbol slug status rank marketPairCount circulatingSupply totalSupply maxSupply lastUpdated dateAdded price volume24h marketCap percentChange1h percentChange24h percentChange7d
0 1 USD BTC bitcoin active 1 9717 1.863544e+07 1.863544e+07 2.100000e+07 2021-02-22T09:37:02.000Z 2013-04-28T00:00:00.000Z 55579.249971 5.656584e+10 1.035744e+12 -1.232746 -1.234765 16.978174
1 1027 USD ETH ethereum active 2 5982 1.147732e+08 1.147732e+08 NaN 2021-02-22T09:37:02.000Z 2015-08-07T00:00:00.000Z 1855.072456 2.450605e+10 2.129125e+11 -1.373583 -4.104364 5.315240
2 1839 USD BNB binance-coin active 3 469 1.545328e+08 1.705328e+08 1.705328e+08 2021-02-22T09:37:11.000Z 2017-07-25T00:00:00.000Z 272.095668 6.811884e+09 4.204770e+10 -2.381284 2.937286 109.533310
3 825 USD USDT tether active 4 10829 3.445054e+10 3.570817e+10 NaN 2021-02-22T09:37:08.000Z 2015-02-25T00:00:00.000Z 0.999576 1.087710e+11 3.443593e+10 -0.061248 -0.023795 -0.074917
4 6636 USD DOT polkadot-new active 5 145 9.103144e+08 1.045967e+09 NaN 2021-02-22T09:36:05.000Z 2020-08-19T00:00:00.000Z 37.503515 3.257901e+09 3.413999e+10 -1.327435 -2.635214 40.263648
5 2010 USD ADA cardano active 6 231 3.111248e+10 4.500000e+10 4.500000e+10 2021-02-22T09:37:09.000Z 2017-10-01T00:00:00.000Z 1.040491 6.621492e+09 3.237226e+10 -1.594681 -7.316003 25.951127
6 52 USD XRP xrp active 7 673 4.540403e+10 9.999083e+10 1.000000e+11 2021-02-22T09:38:03.000Z 2013-08-04T00:00:00.000Z 0.581321 1.102498e+10 2.639430e+10 -1.640063 11.286157 2.731301
7 2 USD LTC litecoin active 8 754 6.653055e+07 6.653055e+07 8.400000e+07 2021-02-22T09:38:02.000Z 2013-04-28T00:00:00.000Z 216.783950 6.530638e+09 1.442276e+10 -2.134667 -3.477237 5.932102
8 1975 USD LINK chainlink active 9 471 4.085096e+08 1.000000e+09 1.000000e+09 2021-02-22T09:37:11.000Z 2017-09-20T00:00:00.000Z 32.145503 1.885830e+09 1.313174e+10 -1.378857 -5.152372 -0.036835
9 1831 USD BCH bitcoin-cash active 10 581 1.866177e+07 1.866177e+07 2.100000e+07 2021-02-22T09:37:07.000Z 2017-07-23T00:00:00.000Z 679.047253 5.800439e+09 1.267222e+10 -1.298651 -0.162108 -1.595937
我正在尝试从 url 获取内容并使用 BeautyfulSoup.
解析响应这个 url 在加载时会检索我最喜欢的关注列表项目，问题是当网站加载时需要几秒钟才能在 table 中显示数据，所以当我 运行 urlopen(my_url) 响应中没有 table，因此我的解析方法失败了。
我正在学习这门语言，所以我尽量保持简单，所以我想使用我已经在我的代码中设置的工具，所以根据我所拥有的，我想知道是否有办法等待，或检查内容何时准备好以便我能够获取数据（table 内容）。
这是我的代码：
from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as ureq
from urllib.error import URLError, HTTPError
URL = 'url route goes here' # In compliance to SO rules I've removed the website path
def get_dom_from_url():
try:
u_client = ureq(URL)
html = u_client.read()
u_client.close()
except HTTPError as e:
print(f'There has been an HTTP ERROR: {e.code}')
except URLError as e:
print(f'There has been a problem reaching the URL. ERROR: {e.code}')
finally:
print('''
DOM loaded!
''')
return html
dom = soup(get_dom_from_url(), 'html.parser')
# Crawl the dom object and get the table thead element
col_names = [col.text for col in dom.table.thead.find_all('th')]
col_names = col_names[1:-2]
col_names
这是错误信息：
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-102-625de133b2e2> in <module>
----> 1 col_names = [col.text for col in dom.table.thead.find_all('th')]
2 col_names = col_names[1:-2]
3 col_names
AttributeError: 'NoneType' object has no attribute 'thead'
当我在没有路由的情况下加载 url 时，上面的代码有效，但我需要它，因为我需要为我正在处理的 ETL 管道存储相同的数据。
如果仅使用 urllib 无法实现此目的，我想听听您的建议。
如评论所述，urllib.request相当古老，Selenium可以处理javascript:
from selenium import webdriver
from bs4 import BeautifulSoup as soup
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',options=options)
wd.get("https://coinmarketcap.com/watchlist/60321ee5b01cab343e1e37d6")
dom = soup(wd.page_source, 'html.parser')
col_names = [col.text for col in dom.table.thead.find_all('th')]
col_names = col_names[1:-2]
col_names
输出：
['Name', 'Price', '24h', '7d', 'Market Cap', 'Volume', 'Circulating Supply']
我不确定你的目的是什么，但你也可以直接将表加载到 pandas 而无需使用 BeautifulSoup:
import pandas as pd
df = pd.read_html(wd.page_source)[0]
df = df.iloc[:, 1:-2] # drop first and last two columns
输出df.head()：
| | Name | Price | 24h | 7d | Market Cap | Volume | Circulating Supply |
|---:|:-----------------|:-----------|:------|:--------|:-------------------|:---------------------------------|:---------------------|
| 0 | Bitcoin1BTC | ,515.75 | 3.67% | 17.00% | ,069,904,656,718 | ,190,502,8631,135,430 BTC | 18,634,643 BTC |
| 1 | Ethereum2ETH | ,972.76 | 1.32% | 7.03% | 5,668,814,800 | ,790,664,03815,658,144 ETH | 114,760,589 ETH |
| 2 | Binance Coin3BNB | 5.07 | 4.20% | 113.31% | ,505,990,437 | ,923,804,82135,249,242 BNB | 154,532,785 BNB |
| 3 | Polkadot4DOT | .69 | 4.23% | 39.72% | ,117,982,640 | ,555,973,830139,996,164 DOT | 910,079,701 DOT |
| 4 | Cardano5ADA | .14 | 8.25% | 31.00% | ,603,496,069 | ,299,401,9179,000,239,285 ADA | 31,112,484,646 ADA |
实际上你不需要在这里使用Selenium。数据以有效的 json 格式嵌入在 <script> 标记中的源 html 中。只需要解析：
import requests
from bs4 import BeautifulSoup
import json
import pandas as pd
url = 'https://coinmarketcap.com/watchlist/60321ee5b01cab343e1e37d6/'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
jsonStr = soup.find('script', {'id':'__NEXT_DATA__'}).text
jsonData = json.loads(jsonStr)
data = jsonData['props']['initialProps']['pageProps']['fetchedWatchlist']['cryptoCurrencies']
rows = []
for each in data:
quotes_row = each.pop('quotes')[0]
each.pop('tags')
if 'platform' in each.keys():
each.pop('platform')
each.update(quotes_row)
rows.append(each)
df = pd.DataFrame(rows)
输出：
print(df.to_string())
id name symbol slug status rank marketPairCount circulatingSupply totalSupply maxSupply lastUpdated dateAdded price volume24h marketCap percentChange1h percentChange24h percentChange7d
0 1 USD BTC bitcoin active 1 9717 1.863544e+07 1.863544e+07 2.100000e+07 2021-02-22T09:37:02.000Z 2013-04-28T00:00:00.000Z 55579.249971 5.656584e+10 1.035744e+12 -1.232746 -1.234765 16.978174
1 1027 USD ETH ethereum active 2 5982 1.147732e+08 1.147732e+08 NaN 2021-02-22T09:37:02.000Z 2015-08-07T00:00:00.000Z 1855.072456 2.450605e+10 2.129125e+11 -1.373583 -4.104364 5.315240
2 1839 USD BNB binance-coin active 3 469 1.545328e+08 1.705328e+08 1.705328e+08 2021-02-22T09:37:11.000Z 2017-07-25T00:00:00.000Z 272.095668 6.811884e+09 4.204770e+10 -2.381284 2.937286 109.533310
3 825 USD USDT tether active 4 10829 3.445054e+10 3.570817e+10 NaN 2021-02-22T09:37:08.000Z 2015-02-25T00:00:00.000Z 0.999576 1.087710e+11 3.443593e+10 -0.061248 -0.023795 -0.074917
4 6636 USD DOT polkadot-new active 5 145 9.103144e+08 1.045967e+09 NaN 2021-02-22T09:36:05.000Z 2020-08-19T00:00:00.000Z 37.503515 3.257901e+09 3.413999e+10 -1.327435 -2.635214 40.263648
5 2010 USD ADA cardano active 6 231 3.111248e+10 4.500000e+10 4.500000e+10 2021-02-22T09:37:09.000Z 2017-10-01T00:00:00.000Z 1.040491 6.621492e+09 3.237226e+10 -1.594681 -7.316003 25.951127
6 52 USD XRP xrp active 7 673 4.540403e+10 9.999083e+10 1.000000e+11 2021-02-22T09:38:03.000Z 2013-08-04T00:00:00.000Z 0.581321 1.102498e+10 2.639430e+10 -1.640063 11.286157 2.731301
7 2 USD LTC litecoin active 8 754 6.653055e+07 6.653055e+07 8.400000e+07 2021-02-22T09:38:02.000Z 2013-04-28T00:00:00.000Z 216.783950 6.530638e+09 1.442276e+10 -2.134667 -3.477237 5.932102
8 1975 USD LINK chainlink active 9 471 4.085096e+08 1.000000e+09 1.000000e+09 2021-02-22T09:37:11.000Z 2017-09-20T00:00:00.000Z 32.145503 1.885830e+09 1.313174e+10 -1.378857 -5.152372 -0.036835
9 1831 USD BCH bitcoin-cash active 10 581 1.866177e+07 1.866177e+07 2.100000e+07 2021-02-22T09:37:07.000Z 2017-07-23T00:00:00.000Z 679.047253 5.800439e+09 1.267222e+10 -1.298651 -0.162108 -1.595937