RSelenium 抓取雅虎财经新闻头条

Question

我想从雅虎获取一家公司的新闻标题。我使用 RSelenium 启动远程浏览器并接受 cookie。我找到了周围 css class “StretchedBox”，我可以通过浏览器检查从字面上看到标题。如何存储这些标题？接下来，我想使用 RSelenium 向下滚动并保存更多这些元素（比如几天）。

library('RSelenium')

# Start Remote Browser
rD <- rsDriver(port = 4840L, browser = c("firefox")) 
remDr <- rD[["client"]]

# Navigate to Yahoo Finance News for Specific Company
# This takes unusual long time 
remDr$navigate("https://finance.yahoo.com/quote/AAPL/news?p=AAPL")

# Get "accept all cookies" botton
webElems <- remDr$findElements(using = "xpath", "//button[starts-with(@class, 'btn primary')]") 

# We can check if we did get the proper button by checking the text of the element:
unlist(lapply(webElems, function(x) {x$getElementText()}))

# We found the two button, and we want to click the first one:
webElems[[1]]$clickElement()

# wait for page loading
Sys.sleep(5) 

# I am looking for news headline in or after the StretchedBox
boxes <- remDr$findElements(using = "class", "StretchedBox")
boxes[1] # empty

boxes[[1]]$browserName

Answer 1

最后，我找到了一个 xpath，我可以从中 getElementText 新闻文章的标题。

library('RSelenium')

# Start Browser
rD <- rsDriver(port = 4835L, browser = c("firefox")) 
remDr <- rD[["client"]]

# Navigate to Yahoo Financial News
remDr$navigate("https://finance.yahoo.com/quote/AAPL/news?p=AAPL")

# Click Accept Cookies
webElems <- remDr$findElements(using = "xpath", "//button[starts-with(@class, 'btn primary')]")
unlist(lapply(webElems, function(x) {x$getElementText()}))
webElems[[1]]$clickElement()

# extract headlines from html/css by xpath 
headlines <- remDr$findElements(using = "xpath", "//h3[@class = 'Mb(5px)']//a")

# extract headline text
headlines <- sapply(headlines, function(x){x$getElementText()})
headlines[1]

[[1]]
[1] "What Kind Of Investors Own Most Of Apple Inc. (NASDAQ:AAPL)?"

RSelenium 抓取雅虎财经新闻头条

RSelenium To Scrape Yahoo Financial News Headlines

xpath

css-selectors

web-scraping

rselenium