使用 R 从 URL 下载多个文件不起作用

Question

我正在尝试从多个 URL 循环下载多个 NetCDF (.nc) 格式文件。但是，当我尝试打开这些文件时，它们似乎已损坏。

您将在下面找到我的代码。我尝试了不同的方法，例如，使用 download.file 或 system.

这是我需要下载的文件示例： http://thredds.met.no/thredds/catalog/metusers/senorge2/seNorge2/provisional_archive/PREC1d/gridded_dataset/201601/catalog.html

但是我需要下载数百个文件，因为每个文件代表一天。

到目前为止，这是我的代码：

year = c("2016","2017")
mon = c("01","02")
day = c("01","02","03","04","05","06","07","08","09","10",
        "11","12","13","14","15","16","17","18","19","20",
        "21","22","23","24","25","26","27","28","29","30","31")
for (y in year){
  for (m in mon){
    for (d in day){
      download.file(paste("http://thredds.met.no/thredds/fileServer/metusers/senorge2/seNorge2/provisional_archive/",
                          "PREC1d/gridded_dataset/",y,m,"/seNorge_v2_0_PREC1d_grid_",y,m,d,"_",y,m,d,".nc",sep=""),
                    destfile=paste("seNorge_v2_0_PREC1d_grid_",y,m,d,"_",y,m,d,".nc",sep=""),method="curl",mode="wb")

      #try(system(paste("wget ",paste("http://thredds.met.no/thredds/fileServer/metusers/senorge2/seNorge2/provisional_archive/",
      #                               "PREC1d/gridded_dataset/",y,m,"/seNorge_v2_0_PREC1d_grid_",y,m,d,"_",y,m,d,".nc",sep=""),sep=""), 
      #           intern = TRUE, ignore.stderr = TRUE, wait=TRUE))
    }
  }
}

感谢任何帮助。

谢谢！

最佳，

米歇尔

Answer 1

文件是'corrupted'是什么意思？您尝试如何阅读 nc 文件？

您的代码似乎有效，我可以读取下载的文件。您可以使用 R 中的 raster 包来读取文件。另请确保您已安装 ncdf4 软件包。

library(raster)
r = raster('seNorge_v2_0_PREC1d_grid_20160101_20160101.nc')

Answer 2

当我尝试你的代码时，我得到了一些文件 503 Service Temporarily Unavailable。在这种情况下要重试下载，请添加 --retry-on-http-error=503。 Maye 还添加 --random-wait。我将 method 从 curl 更改为 wget 并删除了 mode="wb" 正如手册中所说 Not used for methods '"wget"' and '"curl"'。希望以下内容能解决您的问题。

year = c("2016","2017")
mon = c("01","02")
day = c("01","02","03","04","05","06","07","08","09","10",
        "11","12","13","14","15","16","17","18","19","20",
        "21","22","23","24","25","26","27","28","29","30","31")
for (y in year){
  for (m in mon){
    for (d in day){
      download.file(paste("http://thredds.met.no/thredds/fileServer/metusers/senorge2/seNorge2/provisional_archive/",
                          "PREC1d/gridded_dataset/",y,m,"/seNorge_v2_0_PREC1d_grid_",y,m,d,"_",y,m,d,".nc",sep=""),
                    destfile=paste("seNorge_v2_0_PREC1d_grid_",y,m,d,"_",y,m,d,".nc",sep=""),method="wget",extra="--random-wait --retry-on-http-error=503")
    }
  }
}

使用 R 从 URL 下载多个文件不起作用

Downloading multiple files from URL using R not working

url

bash

r

wget

download