curl::curl_fetch_memory(url, handle = handle) 中的错误：发送失败：连接已重置 (RStudio.cloud)

Question

我想从此网页获取 id_product 和 id_parent。昨天还能出结果，今天再试的时候就报错了。不管怎样，我是从 rstudio.cloud.

url <-  paste("https://www.tokopedia.com/zhafranseafood/cumi-asin-1kg-per-pack")

    headers = c('User-Agent' = 'Mozilla/5.0')
    doc <- read_html(httr::GET(url, httr::add_headers(.headers=headers)))%>%
          html_text()
    id_product <- str_match_all(doc,'product_id\s+=\s+(\d+);')[[1]][,2]
    id_parent <- str_match_all(doc,'parent_id\s+=\s+(\d+);')[[1]][,2]

    id_product
    id_parent

Error in curl::curl_fetch_memory(url, handle = handle) : 
  Send failure: Connection was reset

我一直在寻找可能的解释，但仍然无济于事。

Answer 1

服务器需要额外的header

library(httr)
library(stringr)
library(magrittr)

headers = c(
  'User-Agent' = 'Mozilla/5.0',
  'Accept' = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'
)

doc <- read_html(httr::GET(url = 'https://www.tokopedia.com/zhafranseafood/cumi-asin-1kg-per-pack', httr::add_headers(.headers=headers)))%>%
       html_text()

id_product <- str_match_all(doc,'product_id\s+=\s+(\d+);')[[1]][,2]
id_parent <- str_match_all(doc,'parent_id\s+=\s+(\d+);')[[1]][,2]

id_product
id_parent

curl::curl_fetch_memory(url, handle = handle) 中的错误：发送失败：连接已重置 (RStudio.cloud)

Error in curl::curl_fetch_memory(url, handle = handle) : Send failure: Connection was reset (RStudio.cloud)

r

web-scraping

rcurl

httr