用 python lxml 抓取 airbnb

Question

我正在尝试在爱彼迎房源上查找节点。节点是

< div class="col-md-3 text-muted" data-reactid=".2e7if3twveo.0.0.0.0.1.6.0">< span data-reactid=".2e7if3twveo.0.0.0.0.1.6.0.0">The Space< /span> /div>

import mechanize

br = mechanize.Browser()

url ='https://www.airbnb.com/rooms/5711344'
tree = html.fromstring(br.open(url).get_data())
els = tree.xpath('//div[@class="row"]/div[@class="col-md-3 text-muted"]')
for element in els:
    if element.text.find('The Space') >= 0:

不知何故 'The Space' 无法检索。

Answer 1

这对我有用：我使用 BeautifulSoup 通过它们的 class 属性获取 div，然后循环寻找正确的。

import requests
from bs4 import BeautifulSoup

url = 'https://www.airbnb.com/rooms/5711344'
html = requests.get(url)
soup = BeautifulSoup(html.text, 'html.parser')
divs = soup.find_all('div', attrs={'class': 'col-md-3 text-muted'})
for div in divs:
    space = div.find('span').text.strip()
    if space == "The Space":
        print(space)

用 python lxml 抓取 airbnb

Scraping airbnb with python lxml

python

lxml

beautifulsoup