从 yahoo finance 抓取 python 中的数据

Question

我想从 yahoo finance 抓取特定代码的数据。

我可以抓取 table 格式，但我无法抓取非 table 格式。我用同样的原理在同一个页面中抓取信息，但没有结果。

到目前为止我可以从 https://finance.yahoo.com/quote/AAPL/profile?p=AAPL

我用来抓取 table 的代码是：

import numpy as np
import pandas as pd

import requests
import lxml
from lxml import html

symbol = 'AAPL'

url = 'https://finance.yahoo.com/quote/' + symbol + '/profile?p=' + symbol

page = requests.get(url)
tree = html.fromstring(page.content)

table = tree.xpath('//table') 

assert len(table) == 1 
tstring = lxml.etree.tostring(table[0], method='html')
df = pd.read_html(tstring)[0]

df

我要刮右边的table

Sector: Consumer Goods
Industry: Electronic Equipment
Full Time Employees: 137,000

如果您能帮助获取信息或提供一些提示和建议，我将不胜感激。

Answer 1

您可以使用following-sibling

import requests
from lxml import html

xp = "//span[text()='Sector']/following-sibling::span[1]"

symbol = 'AAPL'

url = 'https://finance.yahoo.com/quote/' + symbol + '/profile?p=' + symbol

page = requests.get(url)
tree = html.fromstring(page.content)

d = {}
for label in ['Sector', 'Industry', 'Full Time Employees']:
    xp = f"//span[text()='{label}']/following-sibling::span[1]"
    s = tree.xpath(xp)[0]
    d[label] = s.text_content()


print(d['Full Time Employees'])
print(d['Industry'])
print(d['Sector'])

从 yahoo finance 抓取 python 中的数据

Scrape data in python from yahoo finance

python

yahoo-finance