当文件夹中的文件数 > 1000 时，Dropbox Files.download 不启动

Question

我在 DropBox 论坛上交叉发布我最初的问题。我认为对于 swiftydropbox 用户来说也有这个很好。

我在通过 swiftyDropbox 将整个文件夹下载到本地设备时遇到问题。我正在执行 ListFolder 和 ListFolderContinue（我观察到它将每个响应分块到 ~500 个文件）并将其附加到本地数组。

之后，我将这个数组传递给 files.download。但是，我发现如果我的文件夹超过 1000 个文件（txt 文件大小约为 0.5-1kb），下载过程将不会开始。

static func downloadMissingFiles(client: DropboxClient, callingProcess: String) {
      let fileManager = FileManager.default
      let localBaseURL = fileManager.urls(for: .cachesDirectory, in: .userDomainMask)[0].appendingPathComponent("Cloud/Dropbox", isDirectory: true)
      
      // Data will be in the form of
      // key   : "/workouts/workout list 1/peye.mrc"
      // value : "//workouts/workout list 1/peye.mrc=_-_=015ca880b135d01000000020cb26de0"
      for dbFiles in Array(dbFileNameRevDict) {
        let dbFilePathLower = dbFiles.key
        let dbFileNameRev = dbFiles.value
        let fullURL = localBaseURL.appendingPathComponent(dbFileNameRev)
        
        if fileManager.fileExists(atPath: fullURL.path) {
          print("  -> FILE EXISTS dbFileNameRev:\(dbFileNameRev)")
          localFileList.append(dbFileNameRev)
        } else {
          let destination : (URL, HTTPURLResponse) -> URL = { temporaryURL, response in
            return fullURL
          }
          
          client.files.download(path:dbFilePathLower, overwrite: true, destination: destination)
            .response { response, error in
              if let (_, url) = response {
                print("====> DOWNLOADED:\(url.lastPathComponent)")
              } else if let error = error {
               print(error)
            }
            /// This gives a progress of every single file on it's own. Hence, useless
            // .progress { progressData in
            //  print(progressData)
            // }
        }
      }
    }
  }

我试过各种方法来下载这些文件，我也试过做一个串行队列来一个一个地迭代文件数组，但它不起作用。

这是我处理 ListFolder 和 ListFolderContinue 的方式，查看 hasMore 属性。

      // 
      if result.hasMore == true {
        processDBMore(client: client, cursor: result.cursor)
      } else {
        // When there is no more files (as indicated by hasMore == false)
        // start downloading the files
        downloadMissingFiles(client: client, callingProcess: "processDBMore-Finish")
        print("PrcessDBMore - dropboxGroup.leave")
        dropboxGroup.leave()
      }

Answer 1

根据 Greg (swiftyDropbox) 的说法

Each 'client.files.download' call downloads one file by making one HTTPS request to the Dropbox API servers. Additionally, these calls run asynchronously, and will not block until the call completes. That is, calling 'client.files.download' will start the HTTPS request, but will itself return before it's complete and the response is fully received. (It just runs the supplied block once the request is done.) That being the case, in the code you showed here, you're actually starting 1000 connections in a row, at almost the same time, so it's likely exhausting your network connection. You should update your code to only submit one (or a few) of these at a time. You mentioned you tried a serial queue, but that may be running in to the same issue, since the actual requests run asynchronously.

所以当我遇到这个 post 时，我正在寻找其他解决方案，这极大地帮助我理解了信号量的工作原理以及实现信号量（除了使用 dispatchGroups 之外）如何能够正确地控制 files.download 个调用。

   static func downloadMissingFiles(client: DropboxClient, callingProcess: String) {
      let fileManager = FileManager.default
      let localBaseURL = fileManager.urls(for: .cachesDirectory, in: .userDomainMask)[0].appendingPathComponent("Cloud/Dropbox", isDirectory: true)
      let semaphore = DispatchSemaphore(value: 1)  // insert desired concurrent downloads value here.

      // Data will be in the form of
      // key   : "/workouts/workout list 1/peye.mrc"
      // value : "//workouts/workout list 1/peye.mrc=_-_=015ca880b135d01000000020cb26de0"
      DispatchQueue.global().async { // Wrap the call within an async block
      for dbFiles in Array(dbFileNameRevDict) {
        semaphore.wait() // Decrement the semaphore counter
        let dbFilePathLower = dbFiles.key
        let dbFileNameRev = dbFiles.value
        let fullURL = localBaseURL.appendingPathComponent(dbFileNameRev)
        
        if fileManager.fileExists(atPath: fullURL.path) {
          print("  -> FILE EXISTS dbFileNameRev:\(dbFileNameRev)")
          localFileList.append(dbFileNameRev)
          semaphore.signal()  // Increment semaphore counter
        } else {
          let destination : (URL, HTTPURLResponse) -> URL = { temporaryURL, response in
            return fullURL
          }
          
          client.files.download(path:dbFilePathLower, overwrite: true, destination: destination)
            .response { response, error in
              if let (_, url) = response {
                print("====> DOWNLOADED:\(url.lastPathComponent)")
                // we've reached here means we've successfully download the file
                // So we can (release)increment semaphore counter
                semaphore.signal() 
              } else if let error = error {
               print(error)
            }
            /// This gives a progress of every single file on it's own. Hence, useless
            // .progress { progressData in
            //  print(progressData)
            // }
        }
      }
    }
   }
  }

当文件夹中的文件数 > 1000 时，Dropbox Files.download 不启动

Dropbox Files.download does not start when number of files in folder is > 1000

arrays

dropbox-api

swift

swiftydropbox