需要优化工作脚本速度的 for-loop 替代方案

Question

我已经有了这个工作，但希望对其进行优化。提取与此相关的文章数据需要很长时间，因为我的方法是使用 for 循环。我需要逐行查看，每行运行需要一秒多一点的时间。然而，在我的实际数据集中，我有大约 10,000 行，这需要很长时间。除了 for 循环之外，还有其他方法可以提取整篇文章吗？我对每一行都使用相同的方法，所以我想知道 R 中是否有一个函数类似于将一列乘以一个超级快的数字。

虚拟数据集的创建：

date<- as.Date(c('2020-06-25', '2020-06-25','2020-06-25','2020-06-25','2020-06-25','2020-06-25','2020-06-25','2020-06-25','2020-06-25','2020-06-25'))

text <- c('Isko cites importance of wearing face mask, gives 10K pieces to 30 barangays', 
      'GMRC now a law; to be integrated in school curriculum',
      'QC to impose stringent measures to screen applicants for PWD ID',
      '‘Baka kalaban ka:’ Cops intimidate dzBB reporter',
      'Is gov’t playing with traditional jeepney drivers? A lawmaker thinks so',
      'PNP records highest single-day COVID-19 tally as cases rise to 579',
      'IBP tells new lawyers: ‘Excel without sacrificing honor’',
      'Senators express concern over DepEd’s preparedness for upcoming school year',
      'Angara calls for probe into reported spread of ‘fake’ PWD IDs',
      'Grab PH eyes new scheme to protect food couriers vs no-show customers')
link<- c('https://newsinfo.inquirer.net/1297621/isko-cites-importance-of-wearing-face-mask-gives-10k-pieces-to-30-barangays',  
     'https://newsinfo.inquirer.net/1297618/gmrc-now-a-law-to-be-integrated-in-school-curriculum',                           
     'https://newsinfo.inquirer.net/1297614/qc-to-impose-stringent-measures-to-screen-applicants-for-pwd-id',                 
     'https://newsinfo.inquirer.net/1297606/baka-kalaban-ka-cops-intimidate-dzbb-reporter',                                  
     'https://newsinfo.inquirer.net/1297582/is-govt-playing-with-traditional-jeepney-drivers-a-party-list-lawmaker-thinks-so',
     'https://newsinfo.inquirer.net/1297577/pnp-records-highest-single-day-covid-19-tally-as-cases-rose-to-579',             
     'https://newsinfo.inquirer.net/1297562/ibp-tells-new-lawyers-excel-without-sacrificing-honor',                         
     'https://newsinfo.inquirer.net/1297559/senators-on-depeds-preparedness-for-upcoming-school-year',                      
     'https://newsinfo.inquirer.net/1297566/angara-calls-for-probe-into-reported-spread-of-fake-pwd-ids',                   
     'https://newsinfo.inquirer.net/1297553/grab-ph-eyes-new-scheme-to-protect-food-couriers-vs-no-show-customers')

df<-data.frame(date, text, link)

虚拟数据集：

df
         date                                                                         text                                                 link
1  2020-06-25 Isko cites importance of wearing face mask, gives 10K pieces to 30 barangays   https://newsinfo.inquirer.net/1297621/isko-cites-importance-of-wearing-face-mask-gives-10k-pieces-to-30-barangays
2  2020-06-25                        GMRC now a law; to be integrated in school curriculum   https://newsinfo.inquirer.net/1297618/gmrc-now-a-law-to-be-integrated-in-school-curriculum
3  2020-06-25              QC to impose stringent measures to screen applicants for PWD ID   https://newsinfo.inquirer.net/1297614/qc-to-impose-stringent-measures-to-screen-applicants-for-pwd-id
4  2020-06-25                             ‘Baka kalaban ka:’ Cops intimidate dzBB reporter   https://newsinfo.inquirer.net/1297606/baka-kalaban-ka-cops-intimidate-dzbb-reporter
5  2020-06-25      Is gov’t playing with traditional jeepney drivers? A lawmaker thinks so   https://newsinfo.inquirer.net/1297582/is-govt-playing-with-traditional-jeepney-drivers-a-party-list-lawmaker-thinks-so
6  2020-06-25           PNP records highest single-day COVID-19 tally as cases rise to 579   https://newsinfo.inquirer.net/1297577/pnp-records-highest-single-day-covid-19-tally-as-cases-rose-to-579
7  2020-06-25                     IBP tells new lawyers: ‘Excel without sacrificing honor’   https://newsinfo.inquirer.net/1297562/ibp-tells-new-lawyers-excel-without-sacrificing-honor
8  2020-06-25  Senators express concern over DepEd’s preparedness for upcoming school year   https://newsinfo.inquirer.net/1297559/senators-on-depeds-preparedness-for-upcoming-school-year
9  2020-06-25                Angara calls for probe into reported spread of ‘fake’ PWD IDs   https://newsinfo.inquirer.net/1297566/angara-calls-for-probe-into-reported-spread-of-fake-pwd-ids
10 2020-06-25        Grab PH eyes new scheme to protect food couriers vs no-show customers   https://newsinfo.inquirer.net/1297553/grab-ph-eyes-new-scheme-to-protect-food-couriers-vs-no-show-customers

为每个link获取文章数据的代码：

now<-Sys.time()
for(i in 1:nrow(df)) {
  test_article<- read_html(df[i, 3]) %>% 
    html_nodes(".article_align div p") %>% 
    html_text() %>%
    toString() 

  text_df <- tibble(test_article)
  df[i,4]<-test_article
  print(paste(i,"/",nrow(df), sep = ""))
}
finish<-Sys.time()
finish-now

所以就10篇文章，用了10秒，我觉得很长。看看是否有更快的方法来做到这一点。

Answer 1

您可以并行化循环：

#setup parallel backend to use many processors
cores=detectCores()
cl <- makeCluster(cores[1]-1) #not to overload your computer
registerDoParallel(cl)
now <- Sys.time()
result <- foreach(i =1:nrow(df),.combine=rbind,.packages=('dplyr','rvest') %dopar% { 
  test_article <- read_html(df[i, 3]) %>% 
    html_nodes(".article_align div p") %>% 
    html_text() %>%
    toString() 
  
  data.frame( test_article = test_article, ID = paste(i,"-",nrow(df), sep = ""))
  }

finish<-Sys.time()
finish-now
#stop cluster
stopCluster(cl)

请注意，您不能从 foreach 循环内部写入原始数据帧，因为每个任务都在单独的环境中运行。

需要优化工作脚本速度的 for-loop 替代方案

Alternative to for-loop needed to optimize speed of working script

for-loop

r

apply

lapply

sapply