bs4.FeatureNotFound：找不到具有您请求的功能的树生成器：html-解析器。您需要安装解析器库吗？

Question

我试图通过以下代码Web Scraping：

from bs4 import BeautifulSoup
import requests
import pandas as pd

page = requests.get('https://www.google.com/search?q=phagwara+weather')
soup = BeautifulSoup(page.content, 'html-parser')
day = soup.find(id='wob_wc')

print(day.find_all('span'))

但不断出现以下错误：

 File "C:\Users\myname\Desktop\webscraping.py", line 6, in <module>
    soup = BeautifulSoup(page.content, 'html-parser')
  File "C:\Users\myname\AppData\Local\Programs\Python\Python38-32\lib\site-packages\bs4\__init__.py", line 225, in __init__
    raise FeatureNotFound(
bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: html-parser. Do you need to install a parser library?

我安装了 lxml 和 html5lib 这个问题仍然存在。

Answer 1

您需要将 'html-parser' 更改为 soup = BeautifulSoup(page.content, 'html.parser')

Answer 2

您需要提及标签，因此 soup.find(id="wob_wc") 应该是 soup.find("div", id="wob_wc"))

解析器名称是 html.parser 而不是 html-parser 区别在于点。

同样默认情况下，Google 通常会给你一个 200 的响应，以防止你知道你是否阻止了。通常你必须检查 r.content。

我已经添加了 headers，现在可以使用了。

import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0'}
r = requests.get(
    "https://www.google.com/search?q=phagwara+weather", headers=headers)
soup = BeautifulSoup(r.content, 'html.parser')

print(soup.find("div", id="wob_wc"))

Answer 3

实际上，您不需要遍历整个事情："div #wob_wc"，因为当前位置、天气、日期、温度、降水量、湿度和风由一个元素组成，而不是在其他任何地方重复，您可以使用 select() 或 find() 代替。

如果你想迭代某些东西，那么迭代温度预测是个好主意，例如：

for forecast in soup.select('.wob_df'):
  high_temp = forecast.select_one('.vk_gy .wob_t:nth-child(1)').text
  low_temp = forecast.select_one('.QrNVmd .wob_t:nth-child(1)').text
  print(f'High: {high_temp}, Low: {low_temp}')

'''
High: 67, Low: 55
High: 65, Low: 56
High: 68, Low: 55
'''

看看SelectorGadget Chrome extension where you can grab CSS selectors by clicking on the desired element in your browser. CSS selectors reference。

代码和full example in the online IDE:

from bs4 import BeautifulSoup
import requests, lxml

headers = {
  "User-Agent":
  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

params = {
  "q": "phagwara weather",
  "hl": "en",
  "gl": "us"
}

response = requests.get('https://www.google.com/search', headers=headers, params=params)
soup = BeautifulSoup(response.text, 'lxml')

weather_condition = soup.select_one('#wob_dc').text
tempature = soup.select_one('#wob_tm').text
precipitation = soup.select_one('#wob_pp').text
humidity = soup.select_one('#wob_hm').text
wind = soup.select_one('#wob_ws').text
current_time = soup.select_one('#wob_dts').text

print(f'Weather condition: {weather_condition}\n'
      f'Tempature: {tempature}°F\n'
      f'Precipitation: {precipitation}\n'
      f'Humidity: {humidity}\n'
      f'Wind speed: {wind}\n'
      f'Current time: {current_time}\n')

for forecast in soup.select('.wob_df'):
  day = forecast.select_one('.QrNVmd').text
  weather = forecast.select_one('img.uW5pk')['alt']
  high_temp = forecast.select_one('.vk_gy .wob_t:nth-child(1)').text
  low_temp = forecast.select_one('.QrNVmd .wob_t:nth-child(1)').text
  print(f'Day: {day}\nWeather: {weather}\nHigh: {high_temp}, Low: {low_temp}\n')

---------
'''
Weather condition: Partly cloudy
Temperature: 87°F
Precipitation: 5%
Humidity: 70%
Wind speed: 4 mph
Current time: Tuesday 4:00 PM

Forcast temperature:
Day: Tue
Weather: Partly cloudy
High: 90, Low: 76
...
'''

或者，您可以使用 SerpApi 中的 Google Direct Answer Box API 来实现相同的目的。这是付费 API 和免费计划。

你的例子的主要区别是你只需要迭代已经提取的数据而不是从头开始做所有事情，或者弄清楚如何绕过来自 Google.

的块

要集成的代码：

params = {
  "engine": "google",
  "q": "phagwara weather",
  "api_key": os.getenv("API_KEY"),
  "hl": "en",
  "gl": "us",
}

search = GoogleSearch(params)
results = search.get_dict()

loc = results['answer_box']['location']
weather_date = results['answer_box']['date']
weather = results['answer_box']['weather']
temp = results['answer_box']['temperature']
precipitation = results['answer_box']['precipitation']
humidity = results['answer_box']['humidity']
wind = results['answer_box']['wind']

forecast = results['answer_box']['forecast']

print(f'{loc}\n{weather_date}\n{weather}\n{temp}°F\n{precipitation}\n{humidity}\n{wind}\n')

print(json.dumps(forecast, indent=2))



---------
'''
Phagwara, Punjab, India
Tuesday 4:00 PM
Partly cloudy
87°F
5%
70%
4 mph

[
  {
    "day": "Tuesday",
    "weather": "Partly cloudy",
    "temperature": {
      "high": "90",
      "low": "76"
    },
    "thumbnail": "https://ssl.gstatic.com/onebox/weather/48/partly_cloudy.png"
  }
...
]
'''

Disclaimer, I work for SerpApi.

bs4.FeatureNotFound：找不到具有您请求的功能的树生成器：html-解析器。您需要安装解析器库吗？

bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: html-parser. Do you need to install a parser library?

python

python-3.x

beautifulsoup

html-parser

html-treebuilder