"Stream is too long" 在 Azure Blob 存储上上传大文件时

Question

我正在尝试将大文件 (4Gb) 上传到 Azure Blob 存储，但失败了。根据这篇文章 (https://docs.microsoft.com/en-us/azure/storage/storage-dotnet-how-to-use-blobs)，这是我的代码 :

CloudBlobContainer blobContainer = blobClient.GetContainerReference("my-container-name");
blobContainer.CreateIfNotExistsAsync().Wait();
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("blob-name");
await blockBlob.UploadFromFileAsync("C:\test.avi");

但是我得到了这个错误

Message: Stream was too long.
Source: System.Private.CoreLib
StackTrace: at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.d__5.MoveNext() in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Blob\BlobWriteStream.cs:line 144 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.WindowsAzure.Storage.Core.Util.StreamExtensions.d__1`1.MoveNext() in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\Common\Core\Util\StreamExtensions.cs:line 308 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.<>c__DisplayClass20_0.<b__0>d.MoveNext() in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Blob\CloudBlockBlob.cs:line 301 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.<>c__DisplayClass23_0.<b__0>d.MoveNext() in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Blob\CloudBlockBlob.cs:line 397 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at MyCompany.AzureServices.Blob.BlobService.d__7.MoveNext() in C:\MyProjectSource\MyCompany.AzureServices\Blob\BlobService.cs:line 79 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at MyCompany.AzureServices.Blob.MyProject.RecordBlobService.<>c__DisplayClass1_0.<b__0>d.MoveNext() in C:\MyProjectSource\MyCompany.AzureServices\Blob\MyProject\RecordBlobService.cs:line 25

根据这篇文章 (https://www.simple-talk.com/cloud/platform-as-a-service/azure-blob-storage-part-4-uploading-large-blobs/)，我尝试为大文件添加更多选项。 这是我的新密码 :

TimeSpan backOffPeriod = TimeSpan.FromSeconds(2);
int retryCount = 1;
BlobRequestOptions bro = new BlobRequestOptions()
{
    //If the file to upload is more than 67Mo, we send it in multiple parts
    SingleBlobUploadThresholdInBytes = 67108864, //67Mo (maximum)
    //Number of threads used to send data
    ParallelOperationThreadCount = 1,
    //If the block fail, we retry once (retryCount) after 2 seconds (backOffPeriod)
    RetryPolicy = new ExponentialRetry(backOffPeriod, retryCount),
};
blobClient.DefaultRequestOptions = bro;
CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference("blob-name");
//If the file is sended in multiple parts, theses parts size are 4Mo
blockBlob.StreamWriteSizeInBytes = 4194304; //4Mo (maximum)
await blockBlob.UploadFromFileAsync("C:\test.avi");

但我又遇到了同样的错误（流太长了）。

我在 Microsoft.WindowsAzure.Storage 库中发现函数 "UploadFromFileAsync" 使用 "UploadFromStreamAsync"，它使用 MemoryStream。我认为我的错误来自那个 MemoryStream 但它写在 blob 存储文章中，一个 blob 的最大大小是 195Gb。那么我应该如何使用它呢？

我使用Microsoft.WindowsAzure.Storage版本7.2.1

谢谢！

.

更新 1 : 感谢 Tom Sun 和 Zhaoxing Lu，我尝试使用 Microsoft.Azure.Storage.DataMovement.
遗憾的是，我在 "TransferManager.UploadAsync" 函数上遇到错误。我尝试 google 但没有...
有任何想法吗？

这是我的代码：

 string storageConnectionString = "myStorageConnectionString";
   string filePath = @"C:\LargeFile.avi";
   string blobName = "large_file.avi";

    CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);
    CloudBlobClient blobClient = account.CreateCloudBlobClient();
    CloudBlobContainer blobContainer = blobClient.GetContainerReference("mycontainer");
    blobContainer.CreateIfNotExists();

   CloudBlockBlob destBlob = blobContainer.GetBlockBlobReference(blobName);
    // Setup the number of the concurrent operations
    TransferManager.Configurations.ParallelOperations = 64;

    // Setup the transfer context and track the upload progress
    var context = new SingleTransferContext();
    UploadOptions uploadOptions = new UploadOptions
    {
        DestinationAccessCondition = AccessCondition.GenerateIfExistsCondition()
    };
    context.ProgressHandler = new Progress<TransferStatus>(progress =>
    {
        Console.WriteLine("Bytes uploaded: {0}", progress.BytesTransferred);
    });

    // Upload a local blob
    TransferManager.UploadAsync(filePath, destBlob, uploadOptions, context, CancellationToken.None).Wait();

这是错误:
消息：出现一个或多个错误。（传输失败：值'*'的格式无效..）
资料来源：System.Private.CoreLib
堆栈跟踪：

at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at System.Threading.Tasks.Task.Wait() at MyCompany.AzureServices.Blob.BlobService.d__7.MoveNext() in C:\MyProjectSource\MyCompany.AzureServices\Blob\BlobService.cs:line 96 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at MyCompany.AzureServices.Blob.MyProject.RecordBlobService.<>c__DisplayClass1_0.<b__0>d.MoveNext() in C:\MyProjectSource\MyCompany.AzureServices\Blob\MyProject\RecordBlobService.cs:line 25

和内部异常：
消息：传输失败：值“*”的格式无效..
来源：Microsoft.WindowsAzure.Storage.DataMovement
堆栈跟踪：

at Microsoft.WindowsAzure.Storage.DataMovement.TransferScheduler.d__22.MoveNext() in C:\Local\Jenkins\jobs\DM_Hotfix\workspace\lib\TransferScheduler.cs:line 214 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.WindowsAzure.Storage.DataMovement.SingleObjectTransfer.d__7.MoveNext() in C:\Local\Jenkins\jobs\DM_Hotfix\workspace\lib\TransferJobs\SingleObjectTransfer.cs:line 226 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.WindowsAzure.Storage.DataMovement.TransferManager.d__72.MoveNext() in C:\Local\Jenkins\jobs\DM_Hotfix\workspace\lib\TransferManager.cs:line 1263

和下一个内部异常：
消息：值“*”的格式无效。
来源：Microsoft.WindowsAzure.Storage.DataMovement
堆栈跟踪：

at Microsoft.WindowsAzure.Storage.DataMovement.TransferControllers.BlockBlobWriter.HandleFetchAttributesResult(Exception e) in C:\Local\Jenkins\jobs\DM_Hotfix\workspace\lib\TransferControllers\TransferWriters\BlockBlobWriter.cs:line 196 at Microsoft.WindowsAzure.Storage.DataMovement.TransferControllers.BlockBlobWriter.d__18.MoveNext() in C:\Local\Jenkins\jobs\DM_Hotfix\workspace\lib\TransferControllers\TransferWriters\BlockBlobWriter.cs:line 157 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.WindowsAzure.Storage.DataMovement.TransferControllers.BlockBlobWriter.d__16.MoveNext() in C:\Local\Jenkins\jobs\DM_Hotfix\workspace\lib\TransferControllers\TransferWriters\BlockBlobWriter.cs:line 83 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.WindowsAzure.Storage.DataMovement.TransferControllers.SyncTransferController.d__13.MoveNext() in C:\Local\Jenkins\jobs\DM_Hotfix\workspace\lib\TransferControllers\SyncTransferController.cs:line 81 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.WindowsAzure.Storage.DataMovement.TransferControllers.TransferControllerBase.d__33.MoveNext() in C:\Local\Jenkins\jobs\DM_Hotfix\workspace\lib\TransferControllers\TransferControllerBase.cs:line 178 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.WindowsAzure.Storage.DataMovement.TransferScheduler.d__22.MoveNext() in C:\Local\Jenkins\jobs\DM_Hotfix\workspace\lib\TransferScheduler.cs:line 208

最后一个内部异常：
消息：值“*”的格式无效。
资料来源：System.Net.Http
堆栈跟踪：

at System.Net.Http.Headers.HttpHeaderParser.ParseValue(String value, Object storeValue, Int32& index) at System.Net.Http.Headers.EntityTagHeaderValue.Parse(String input) at Microsoft.WindowsAzure.Storage.Shared.Protocol.RequestMessageExtensions.ApplyAccessCondition(StorageRequestMessage request, AccessCondition accessCondition) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Shared\Protocol\RequestMessageExtensions.cs:line 125 at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.<>c__DisplayClass116_0.b__0(RESTCommand1 cmd, Uri uri, UriQueryBuilder builder, HttpContent cnt, Nullable1 serverTimeout, OperationContext ctx) in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Blob\CloudBlob.cs:line 1206 at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.d__4`1.MoveNext() in C:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\WindowsRuntime\Core\Executor\Executor.cs:line 91

最后是我的 project.json :

  {
  "version": "1.0.0-*",

  "dependencies": {
    "Microsoft.Azure.DocumentDB.Core": "0.1.0-preview",
    "Microsoft.Azure.Storage.DataMovement": "0.4.1",
    "Microsoft.IdentityModel.Protocols": "2.0.0",
    "NETStandard.Library": "1.6.1",
    "MyProject.Data.Entities": "1.0.0-*",
    "MyProject.Settings": "1.0.0-*",
    "WindowsAzure.Storage": "7.2.1"
  },

  "frameworks": {   
    "netcoreapp1.0": {
      "imports": [
        "dnxcore50",
        "portable-net451+win8"
      ],
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.0.0-*"
        }
      }
    }
  }
}

感谢您的帮助！

更新 2（有效）

感谢 Tom Sun，这是工作代码

string storageConnectionString = "myStorageConnectionString";
     CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);
      CloudBlobClient blobClient = account.CreateCloudBlobClient();
      CloudBlobContainer blobContainer = blobClient.GetContainerReference("mycontainer");
      blobContainer.CreateIfNotExistsAsync().Wait();
      string sourcePath = @"C:\Tom\TestLargeFile.zip";
      CloudBlockBlob destBlob = blobContainer.GetBlockBlobReference("LargeFile.zip");
      // Setup the number of the concurrent operations
      TransferManager.Configurations.ParallelOperations = 64;
      // Setup the transfer context and track the upoload progress
      var context = new SingleTransferContext
      {
          ProgressHandler =
          new Progress<TransferStatus>(
               progress => { Console.WriteLine("Bytes uploaded: {0}", progress.BytesTransferred); })
       };
      // Upload a local blob
      TransferManager.UploadAsync(sourcePath, destBlob, null, context, CancellationToken.None).Wait();
      Console.WriteLine("Upload finished !");
      Console.ReadKey();

我也加

ShouldOverwriteCallback = (source, destination) =>
               {
                   return true;
               },

在 SingleTransferContext 中覆盖已存在的 blob。

Answer 1

我们正在积极调查 Azure 存储客户端库中的问题。

请注意，由于 UploadFromFileAsync() 对于巨大的 blob 不是可靠且高效的操作，我建议您考虑以下替代方案：

如果你可以接受命令行工具，你可以尝试AzCopy，它可以高性能地传输Azure存储数据，并且可以暂停和恢复传输。

如果您想以编程方式控制传输作业，请使用 Azure Storage Data Movement Library，这是 AzCopy 的核心。

Answer 2

我们可以使用 Azure Storage Data Movement Library easily to upload large files to Azure blob Storage. It works correctly for me, please have a try with the following code. More info about the Azure Storage Data Movement Library please refer to the document :

    string storageConnectionString = "storage connection string";
    CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);
    CloudBlobClient blobClient = account.CreateCloudBlobClient();
    CloudBlobContainer blobContainer = blobClient.GetContainerReference("mycontainer");
    blobContainer.CreateIfNotExists();
    string sourcePath = @"C:\Tom\TestLargeFile.zip";
    CloudBlockBlob destBlob = blobContainer.GetBlockBlobReference("LargeFile.zip");
    // Setup the number of the concurrent operations
    TransferManager.Configurations.ParallelOperations = 64;
    // Setup the transfer context and track the upoload progress

    var context = new SingleTransferContext();

    UploadOptions uploadOptions = new UploadOptions
    {
        DestinationAccessCondition = AccessCondition.GenerateIfExistsCondition()
    };
    context.ProgressHandler = new Progress<TransferStatus>(progress =>
    {
        Console.WriteLine("Bytes uploaded: {0}", progress.BytesTransferred);
    });
    // Upload a local blob
    TransferManager.UploadAsync(sourcePath, destBlob, uploadOptions,context, CancellationToken.None).Wait();

SDK信息请参考package.config文件

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Microsoft.Azure.KeyVault.Core" version="1.0.0" targetFramework="net452" />
  <package id="Microsoft.Azure.Storage.DataMovement" version="0.4.1" targetFramework="net452" />
  <package id="Microsoft.Data.Edm" version="5.6.4" targetFramework="net452" />
  <package id="Microsoft.Data.OData" version="5.6.4" targetFramework="net452" />
  <package id="Microsoft.Data.Services.Client" version="5.6.4" targetFramework="net452" />
  <package id="Microsoft.WindowsAzure.ConfigurationManager" version="1.8.0.0" targetFramework="net452" />
  <package id="Newtonsoft.Json" version="6.0.8" targetFramework="net452" />
  <package id="System.Spatial" version="5.6.4" targetFramework="net452" />
  <package id="WindowsAzure.Storage" version="7.2.1" targetFramework="net452" />
</packages>

从 Azure 门户检查上传的文件

更新：

对于.net核心项目代码：

     string storageConnectionString = "myStorageConnectionString";
     CloudStorageAccount account = CloudStorageAccount.Parse(storageConnectionString);
      CloudBlobClient blobClient = account.CreateCloudBlobClient();
      CloudBlobContainer blobContainer = blobClient.GetContainerReference("mycontainer");
      blobContainer.CreateIfNotExistsAsync().Wait();
      string sourcePath = @"C:\Tom\TestLargeFile.zip";
      CloudBlockBlob destBlob = blobContainer.GetBlockBlobReference("LargeFile.zip");
      // Setup the number of the concurrent operations
      TransferManager.Configurations.ParallelOperations = 64;
      // Setup the transfer context and track the upoload progress
      var context = new SingleTransferContext
      {
          ProgressHandler =
          new Progress<TransferStatus>(
               progress => { Console.WriteLine("Bytes uploaded: {0}", progress.BytesTransferred); })
       };
      // Upload a local blob
      TransferManager.UploadAsync(sourcePath, destBlob, null, context, CancellationToken.None).Wait();
      Console.WriteLine("Upload finished !");
      Console.ReadKey();

Answer 3

原问题已在 WindowsAzure.Storage 的 8.0 版本中修复。

"Stream is too long" 在 Azure Blob 存储上上传大文件时

"Stream is too long" when uploading big files on Azure Blob Storage

c#

.net

azure

azure-blob-storage

blobstorage