将 200MB 文件上传到 Azure blob 存储时出错

Question

我正在使用 Azure.Storage.Blobs v12.6.0.0 和一个每天将文件上传到 Azure blob 存储的小项目，但我无法上传大文件。

实际上我正在使用下面的代码测试 249MB 的文件，但它失败了。如果文件较小，则上传时没有问题，但如果文件较大，则在开始上传后大约 10 分钟后会失败，并显示错误 Retry failed after 6 tries.

我的网速是 40Mbps

这是我使用的代码（在 Visual Studio 2019 中是 VB.net）：

Using fStream As FileStream = File.OpenRead(fPath)
        Dim credential As New StorageSharedKeyCredential("account", storageAccountKey)
        Dim serviceURI As New Uri("https://name.blob.core.windows.net/")
        Dim blobSC As New BlobServiceClient(serviceURI, credential)

        Dim blobContainer As BlobContainerClient = blobSC.GetBlobContainerClient("files")

        Dim blob As BlobClient = blobContainer.GetBlobClient(fName)

        Dim t As Task = blob.UploadAsync(fStream, True, Threading.CancellationToken.None)

        t.Wait()
End Using

在此先感谢您的帮助。

Answer 1

如果您正在写入大小不超过 200MB 的块 blob，您可以通过一次写入操作将其完整上传，参见 Put Blob。

存储客户端的默认大小是 32 MB 最大单块上传，可以使用 SingleBlobUploadThresholdInBytes 属性.

进行设置

当块 blob 上传大于此属性中的值时，存储客户端将文件分成块。

您可以使用 ParallelOperationThreadCount 属性.

设置用于并行上传块的线程数

 Private Sub SurroundingSub()
    Dim requestoptions As BlobRequestOptions = New BlobRequestOptions() With {
        .SingleBlobUploadThresholdInBytes = 1024 * 1024 * 50,
        .ParallelOperationThreadCount = 12
    }
    Dim account As CloudStorageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"))
    Dim blobclient As CloudBlobClient = account.CreateCloudBlobClient()
    blobclient.DefaultRequestOptions = requestoptions
    Dim blobcontainer As CloudBlobContainer = blobclient.GetContainerReference("uploadfiles")
    blobcontainer.CreateIfNotExists()
    Dim blockblob As CloudBlockBlob = blobcontainer.GetBlockBlobReference("bigfiles")
End Sub

请参阅 Put Blob 和线程以获取更多信息

Answer 2

我已经解决了我的问题，问题是我需要将上传分成块，而且我的帐户中有一些未提交的块具有相同的 blob 名称。

我不得不上传一个同名的小 blob 来创建 azure 存储以清除未提交的 blob 块。

这是我现在使用的代码：

Using fStream As FileStream = File.OpenRead(fPath)
    Dim bytesFile As Byte()

    Using ms As MemoryStream = New MemoryStream
        fStream.CopyTo(ms)
        bytesFile = ms.ToArray()
    End Using

    Dim containerClient As BlobContainerClient = New BlobContainerClient(storageConString, "ContainerName")
    Dim blobClient As BlockBlobClient = containerClient.GetBlockBlobClient("BlobName")

    Dim blockId As Integer = 1000

    Do
        Dim bytesToCopy As Long = Math.Min(bytesRemain, pageSizeInBytes)
        Dim bytesToSend As Byte() = New Byte(bytesToCopy - 1) {}


        Array.Copy(bytesFile, prevLastByte, bytesToSend, 0, bytesToCopy)
        prevLastByte += bytesToCopy
        bytesRemain -= bytesToCopy

        Dim base64BlockId As String = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId))
        Dim stageResponse = blobClient.StageBlock(base64BlockId, New MemoryStream(bytesToSend))
        Dim responseInfo = stageResponse.GetRawResponse()

        blocklist.Add(base64BlockId)

        EventLog1.WriteEntry("Respoonse for blockId: " + blockId.ToString + vbCrLf + vbCrLf + responseInfo.ToString, EventLogEntryType.Warning)

        blockId += 1
    Loop While (bytesRemain > 0)

    Dim contentResponse = blobClient.CommitBlockList(blocklist)
End Using

将 200MB 文件上传到 Azure blob 存储时出错

Error uploading 200MB file to Azure blob storage

azure

visual-studio

azure-blob-storage

azure-storage