在 ASP.NET Core 3.1 中上传和下载大文件？

Question

我正在使用干净的架构开发一个 ASP.NET Core 3.1 API 项目，我有以下类库（层）：

基础架构（安全内容和上传助手等...）
持久性（DA层）
领域（领域模型）
应用程序（用例 - 业务逻辑）
API（API项目作为我的启动项目）

我希望能够上传大文件到服务器（例如 2Gb 的文件大小甚至更多）然后下载它们并想这样做以后不会出现内存溢出等问题。

如有任何帮助，我们将不胜感激。

Answer 1

如果您有那么大的文件，请不要在代码中使用 byte[] 或 MemoryStream。如果您 download/upload 个文件，则仅对流进行操作。

您有两个选择：

如果您同时控制客户端和服务器，请考虑使用 tus 之类的东西。 .NET 有客户端和服务器实现。这可能是最简单和最可靠的选择。
如果您使用 HttpClient 上传大文件，只需使用 StreamContent class 发送它们。同样，不要使用 MemoryStream 作为源，而是像 FileStream.
如果您使用 HttpClient 下载大文件，请务必指定 HttpCompletionOptions，例如 var response = await httpClient.SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead)。否则，HttpClient 将在内存中缓冲整个响应。然后，您可以通过 var stream = response.Content.ReadAsStreamAsync().

ASP.NET 核心具体建议：

如果您想通过 HTTP POST 接收文件，您需要增加请求大小限制：[RequestSizeLimit(10L * 1024L * 1024L * 1024L)] 和 [RequestFormLimits(MultipartBodyLengthLimit = 10L * 1024L * 1024L * 1024L)]。另外，需要禁用表单值绑定，否则整个请求都会缓存到内存中：

   [AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
   public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter
   {
       public void OnResourceExecuting(ResourceExecutingContext context)
       {
           var factories = context.ValueProviderFactories;
           factories.RemoveType<FormValueProviderFactory>();
           factories.RemoveType<FormFileValueProviderFactory>();
           factories.RemoveType<JQueryFormValueProviderFactory>();
       }

       public void OnResourceExecuted(ResourceExecutedContext context)
       {
       }
   }

从控制器return一个文件，通过File方法简单return它，它接受一个流：return File(stream, mimeType, fileName);

示例控制器如下所示（请参阅 https://docs.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-3.1 以了解缺少的助手 classes）：

private const MaxFileSize = 10L * 1024L * 1024L * 1024L; // 10GB, adjust to your need

[DisableFormValueModelBinding]
[RequestSizeLimit(MaxFileSize)]
[RequestFormLimits(MultipartBodyLengthLimit = MaxFileSize)]
public async Task ReceiveFile()
{
    if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
        throw new BadRequestException("Not a multipart request");

    var boundary = MultipartRequestHelper.GetBoundary(MediaTypeHeaderValue.Parse(Request.ContentType));
    var reader = new MultipartReader(boundary, Request.Body);

    // note: this is for a single file, you could also process multiple files
    var section = await reader.ReadNextSectionAsync();

    if (section == null)
        throw new BadRequestException("No sections in multipart defined");

    if (!ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out var contentDisposition))
        throw new BadRequestException("No content disposition in multipart defined");

    var fileName = contentDisposition.FileNameStar.ToString();
    if (string.IsNullOrEmpty(fileName))
    {
        fileName = contentDisposition.FileName.ToString();
    }

    if (string.IsNullOrEmpty(fileName))
        throw new BadRequestException("No filename defined.");

    using var fileStream = section.Body;
    await SendFileSomewhere(fileStream);
}

// This should probably not be inside the controller class
private async Task SendFileSomewhere(Stream stream)
{
    using var request = new HttpRequestMessage()
    {
        Method = HttpMethod.Post,
        RequestUri = new Uri("YOUR_DESTINATION_URI"),
        Content = new StreamContent(stream),
    };
    using var response = await _httpClient.SendAsync(request);
    // TODO check response status etc.
}

在这个例子中，我们将整个文件流式传输到另一个服务。在某些情况下，将文件暂时保存到磁盘中会更好。

Answer 2

我发现这篇文章很有用 - https://www.tugberkugurlu.com/archive/efficiently-streaming-large-http-responses-with-httpclient

这里是用于下载大文件的代码版本：

static public async Task HttpDownloadFileAsync(HttpClient httpClient, string url, string fileToWriteTo) {
  using HttpResponseMessage response = await httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
  using Stream streamToReadFrom = await response.Content.ReadAsStreamAsync(); 
  using Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create); 
  await streamToReadFrom.CopyToAsync(streamToWriteTo);
}

Answer 3

有时问题是我们使用 Nginx 作为 asp.net 部署在 ubuntu/Linux 环境中的 docker 核心应用程序的前端代理。这正是我尝试在 docker 或 .net 核心端进行调试时的情况，但实际的解决方案是将 Nginx 配置配置为

client_max_body_size 50M;

可以在您遇到问题的站点的 Nginx 配置的位置或服务器设置部分添加此行。

可能对某人有帮助。

Answer 4

问题是你必须处理长文件，无论你在哪里使用它们，你都需要大量资源才能阅读。一种可能的解决方案是根据信息将文件分成不同的块，或者在单独的作业或线程中处理它，或者使用 .net 中的并行性来处理它。您还可以指定文件的大小，请阅读以下对您非常有用的博客。

Upload Large Files

Answer 5

我建议阅读 Microsoft Docs 在 ASP.NET Core 中上传文件，@ManuelAllenspach 也链接到。

https://docs.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-6.0

根据您的要求，您应该查看的部分是使用流媒体上传大文件：（例如 2Gb 的文件大小或更多）

https://docs.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-6.0#upload-large-files-with-streaming

public class GenerateAntiforgeryTokenCookieAttribute : ResultFilterAttribute
{
    public override void OnResultExecuting(ResultExecutingContext context)
    {
        var antiforgery = context.HttpContext.RequestServices.GetService<IAntiforgery>();

        // Send the request token as a JavaScript-readable cookie
        var tokens = antiforgery.GetAndStoreTokens(context.HttpContext);

        context.HttpContext.Response.Cookies.Append(
            "RequestVerificationToken",
            tokens.RequestToken,
            new CookieOptions() { HttpOnly = false });
    }

    public override void OnResultExecuted(ResultExecutedContext context)
    {
    }
}

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method)]
public class DisableFormValueModelBindingAttribute : Attribute, IResourceFilter
{
    public void OnResourceExecuting(ResourceExecutingContext context)
    {
        var factories = context.ValueProviderFactories;
        factories.RemoveType<FormValueProviderFactory>();
        factories.RemoveType<FormFileValueProviderFactory>();
        factories.RemoveType<JQueryFormValueProviderFactory>();
    }

    public void OnResourceExecuted(ResourceExecutedContext context)
    {
    }
}

services.AddRazorPages(options =>
{
    options.Conventions
        .AddPageApplicationModelConvention("/StreamedSingleFileUploadDb",
            model =>
            {
                model.Filters.Add(
                    new GenerateAntiforgeryTokenCookieAttribute());
                model.Filters.Add(
                    new DisableFormValueModelBindingAttribute());
            });
    options.Conventions
        .AddPageApplicationModelConvention("/StreamedSingleFileUploadPhysical",
            model =>
            {
                model.Filters.Add(
                    new GenerateAntiforgeryTokenCookieAttribute());
                model.Filters.Add(
                    new DisableFormValueModelBindingAttribute());
            });
});

[HttpPost]
[DisableFormValueModelBinding]
[ValidateAntiForgeryToken]
public async Task<IActionResult> UploadDatabase()
{
    if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
    {
        ModelState.AddModelError("File", 
            $"The request couldn't be processed (Error 1).");
        // Log error

        return BadRequest(ModelState);
    }

    // Accumulate the form data key-value pairs in the request (formAccumulator).
    var formAccumulator = new KeyValueAccumulator();
    var trustedFileNameForDisplay = string.Empty;
    var untrustedFileNameForStorage = string.Empty;
    var streamedFileContent = Array.Empty<byte>();

    var boundary = MultipartRequestHelper.GetBoundary(
        MediaTypeHeaderValue.Parse(Request.ContentType),
        _defaultFormOptions.MultipartBoundaryLengthLimit);
    var reader = new MultipartReader(boundary, HttpContext.Request.Body);

    var section = await reader.ReadNextSectionAsync();

    while (section != null)
    {
        var hasContentDispositionHeader = 
            ContentDispositionHeaderValue.TryParse(
                section.ContentDisposition, out var contentDisposition);

        if (hasContentDispositionHeader)
        {
            if (MultipartRequestHelper
                .HasFileContentDisposition(contentDisposition))
            {
                untrustedFileNameForStorage = contentDisposition.FileName.Value;
                // Don't trust the file name sent by the client. To display
                // the file name, HTML-encode the value.
                trustedFileNameForDisplay = WebUtility.HtmlEncode(
                        contentDisposition.FileName.Value);

                streamedFileContent = 
                    await FileHelpers.ProcessStreamedFile(section, contentDisposition, 
                        ModelState, _permittedExtensions, _fileSizeLimit);

                if (!ModelState.IsValid)
                {
                    return BadRequest(ModelState);
                }
            }
            else if (MultipartRequestHelper
                .HasFormDataContentDisposition(contentDisposition))
            {
                // Don't limit the key name length because the 
                // multipart headers length limit is already in effect.
                var key = HeaderUtilities
                    .RemoveQuotes(contentDisposition.Name).Value;
                var encoding = GetEncoding(section);

                if (encoding == null)
                {
                    ModelState.AddModelError("File", 
                        $"The request couldn't be processed (Error 2).");
                    // Log error

                    return BadRequest(ModelState);
                }

                using (var streamReader = new StreamReader(
                    section.Body,
                    encoding,
                    detectEncodingFromByteOrderMarks: true,
                    bufferSize: 1024,
                    leaveOpen: true))
                {
                    // The value length limit is enforced by 
                    // MultipartBodyLengthLimit
                    var value = await streamReader.ReadToEndAsync();

                    if (string.Equals(value, "undefined", 
                        StringComparison.OrdinalIgnoreCase))
                    {
                        value = string.Empty;
                    }

                    formAccumulator.Append(key, value);

                    if (formAccumulator.ValueCount > 
                        _defaultFormOptions.ValueCountLimit)
                    {
                        // Form key count limit of 
                        // _defaultFormOptions.ValueCountLimit 
                        // is exceeded.
                        ModelState.AddModelError("File", 
                            $"The request couldn't be processed (Error 3).");
                        // Log error

                        return BadRequest(ModelState);
                    }
                }
            }
        }

        // Drain any remaining section body that hasn't been consumed and
        // read the headers for the next section.
        section = await reader.ReadNextSectionAsync();
    }

    // Bind form data to the model
    var formData = new FormData();
    var formValueProvider = new FormValueProvider(
        BindingSource.Form,
        new FormCollection(formAccumulator.GetResults()),
        CultureInfo.CurrentCulture);
    var bindingSuccessful = await TryUpdateModelAsync(formData, prefix: "",
        valueProvider: formValueProvider);

    if (!bindingSuccessful)
    {
        ModelState.AddModelError("File", 
            "The request couldn't be processed (Error 5).");
        // Log error

        return BadRequest(ModelState);
    }

    // **WARNING!**
    // In the following example, the file is saved without
    // scanning the file's contents. In most production
    // scenarios, an anti-virus/anti-malware scanner API
    // is used on the file before making the file available
    // for download or for use by other systems. 
    // For more information, see the topic that accompanies 
    // this sample app.

    var file = new AppFile()
    {
        Content = streamedFileContent,
        UntrustedName = untrustedFileNameForStorage,
        Note = formData.Note,
        Size = streamedFileContent.Length, 
        UploadDT = DateTime.UtcNow
    };

    _context.File.Add(file);
    await _context.SaveChangesAsync();

    return Created(nameof(StreamingController), null);
}

using System;
using System.IO;
using Microsoft.Net.Http.Headers;

namespace SampleApp.Utilities
{
    public static class MultipartRequestHelper
    {
        // Content-Type: multipart/form-data; boundary="----WebKitFormBoundarymx2fSWqWSd0OxQqq"
        // The spec at https://tools.ietf.org/html/rfc2046#section-5.1 states that 70 characters is a reasonable limit.
        public static string GetBoundary(MediaTypeHeaderValue contentType, int lengthLimit)
        {
            var boundary = HeaderUtilities.RemoveQuotes(contentType.Boundary).Value;

            if (string.IsNullOrWhiteSpace(boundary))
            {
                throw new InvalidDataException("Missing content-type boundary.");
            }

            if (boundary.Length > lengthLimit)
            {
                throw new InvalidDataException(
                    $"Multipart boundary length limit {lengthLimit} exceeded.");
            }

            return boundary;
        }

        public static bool IsMultipartContentType(string contentType)
        {
            return !string.IsNullOrEmpty(contentType)
                   && contentType.IndexOf("multipart/", StringComparison.OrdinalIgnoreCase) >= 0;
        }

        public static bool HasFormDataContentDisposition(ContentDispositionHeaderValue contentDisposition)
        {
            // Content-Disposition: form-data; name="key";
            return contentDisposition != null
                && contentDisposition.DispositionType.Equals("form-data")
                && string.IsNullOrEmpty(contentDisposition.FileName.Value)
                && string.IsNullOrEmpty(contentDisposition.FileNameStar.Value);
        }

        public static bool HasFileContentDisposition(ContentDispositionHeaderValue contentDisposition)
        {
            // Content-Disposition: form-data; name="myfile1"; filename="Misc 002.jpg"
            return contentDisposition != null
                && contentDisposition.DispositionType.Equals("form-data")
                && (!string.IsNullOrEmpty(contentDisposition.FileName.Value)
                    || !string.IsNullOrEmpty(contentDisposition.FileNameStar.Value));
        }
    }
}

[HttpPost]
[DisableFormValueModelBinding]
[ValidateAntiForgeryToken]
public async Task<IActionResult> UploadPhysical()
{
    if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
    {
        ModelState.AddModelError("File", 
            $"The request couldn't be processed (Error 1).");
        // Log error

        return BadRequest(ModelState);
    }

    var boundary = MultipartRequestHelper.GetBoundary(
        MediaTypeHeaderValue.Parse(Request.ContentType),
        _defaultFormOptions.MultipartBoundaryLengthLimit);
    var reader = new MultipartReader(boundary, HttpContext.Request.Body);
    var section = await reader.ReadNextSectionAsync();

    while (section != null)
    {
        var hasContentDispositionHeader = 
            ContentDispositionHeaderValue.TryParse(
                section.ContentDisposition, out var contentDisposition);

        if (hasContentDispositionHeader)
        {
            // This check assumes that there's a file
            // present without form data. If form data
            // is present, this method immediately fails
            // and returns the model error.
            if (!MultipartRequestHelper
                .HasFileContentDisposition(contentDisposition))
            {
                ModelState.AddModelError("File", 
                    $"The request couldn't be processed (Error 2).");
                // Log error

                return BadRequest(ModelState);
            }
            else
            {
                // Don't trust the file name sent by the client. To display
                // the file name, HTML-encode the value.
                var trustedFileNameForDisplay = WebUtility.HtmlEncode(
                        contentDisposition.FileName.Value);
                var trustedFileNameForFileStorage = Path.GetRandomFileName();

                // **WARNING!**
                // In the following example, the file is saved without
                // scanning the file's contents. In most production
                // scenarios, an anti-virus/anti-malware scanner API
                // is used on the file before making the file available
                // for download or for use by other systems. 
                // For more information, see the topic that accompanies 
                // this sample.

                var streamedFileContent = await FileHelpers.ProcessStreamedFile(
                    section, contentDisposition, ModelState, 
                    _permittedExtensions, _fileSizeLimit);

                if (!ModelState.IsValid)
                {
                    return BadRequest(ModelState);
                }

                using (var targetStream = System.IO.File.Create(
                    Path.Combine(_targetFilePath, trustedFileNameForFileStorage)))
                {
                    await targetStream.WriteAsync(streamedFileContent);

                    _logger.LogInformation(
                        "Uploaded file '{TrustedFileNameForDisplay}' saved to " +
                        "'{TargetFilePath}' as {TrustedFileNameForFileStorage}", 
                        trustedFileNameForDisplay, _targetFilePath, 
                        trustedFileNameForFileStorage);
                }
            }
        }

        // Drain any remaining section body that hasn't been consumed and
        // read the headers for the next section.
        section = await reader.ReadNextSectionAsync();
    }

    return Created(nameof(StreamingController), null);
}

在 ASP.NET Core 3.1 中上传和下载大文件？

Uploading and Downloading large files in ASP.NET Core 3.1?

c#

file-io

large-file-upload

asp.net-core

asp.net-core-webapi