R 在使用 rvest 抓取 table 时将 <p> 元素的 XML 节点集转换为数据帧

Question

我们正在从 https://nbpa.com/agents/directory 中提取联系信息。页面上没有table，而是<div>s，里面有<p>个元素：

我们可以通过以下方式获取此元素：

agents_url <- "https://nbpa.com/agents/directory"
agents_page <- agents_url %>% read_html()
agents_page_elements <- agents_page %>% html_nodes('div.accordion-inner')

agents_page_elements[1]
agents_page_elements[1] %>% html_nodes('p')

我们希望将其转换为 1 行数据框：

Cell             Email                  Professional Credentials:
(123) 456-7890   firstlast@email.com    "NBA Certified Player Agent..."

这可以吗？这个网络抓取的挑战性部分是网站上的每个 accordion-inner div 都有 不同的 p 元素。有些有 Cell: 和 Email:，其他有 Education:、Address: 等。它因 accordion-inner 而异。如果我们可以将每个 individual 节点变成一个 1 行数据帧，然后我们可以使用 plyr::rbind.fill().

将所有数据帧绑定在一起

Answer 1

我们得到后就可以用read.dcf了text

new <- agents_page_elements[1] %>%
     html_nodes('p') %>%
     html_text()
as.data.frame(read.dcf(textConnection(new)))

-输出

    Cell                            Email                                                             Professional Credentials
1 (240) 668-4241 barry.aberdeen@tributesports.com NBA Certified Player Agent, FIBA Certified Player Agent, WNBA Certified Player Agent

对于多个元素，使用map

library(purrr)
library(dplyr)
library(stringr)
out <- map_dfr(agents_page_elements, ~ {
       new <- .x %>% 
               html_nodes('p') %>%
               html_text() %>%
               str_replace_all("\n\s*", " ")
    if(length(new) > 0) {
       as.data.frame(read.dcf(textConnection(new)))
     } else NULL

     })

-输出

> dim(out)
[1] 455   9
> head(out, 2)
            Cell                            Email                                                             Professional Credentials    Title
1 (240) 668-4241 barry.aberdeen@tributesports.com NBA Certified Player Agent, FIBA Certified Player Agent, WNBA Certified Player Agent     <NA>
2 (281) 773-7339                             <NA>                                     Texas Bar No. 24050197|Wisconsin Bar No. 1045470 Attorney
             Company Name                                                                                             Education Address Office International
1                    <NA>                                                                                                  <NA>    <NA>   <NA>          <NA>
2 Adams & Associates, LLC University of Southern California B.S. |University of Houston - M.B.A |University of Wisconsin - J.D.    <NA>   <NA>          <NA>

R 在使用 rvest 抓取 table 时将 <p> 元素的 XML 节点集转换为数据帧

R converting XML nodesets of <p> elements into dataframe when scraping table with rvest

r

web-scraping

rvest