C#以更少的内存消耗从服务器下载大文件

Question

我有一个内存大小为 42 MB 的大文件。我想下载内存占用少的文件。
控制器代码

public ActionResult Download()
{
    var filePath = "file path in server";
    FileInfo file = new FileInfo(filePath);
    Response.ContentType = "application/zip";                        
    Response.AppendHeader("Content-Disposition", "attachment; filename=folder.zip");                   
    Response.TransmitFile(file.FullName);
    Response.End(); 
}

尝试使用 Stream

的替代方法

public ActionResult Download()
{           
    string failure = string.Empty;
    Stream stream = null;
    int bytesToRead = 10000;


    long LengthToRead;
    try
    {
        var path = "file path from server";
        FileWebRequest fileRequest = (FileWebRequest)FileWebRequest.Create(path);
        FileWebResponse fileResponse = (FileWebResponse)fileRequest.GetResponse();

        if (fileRequest.ContentLength > 0)
            fileResponse.ContentLength = fileRequest.ContentLength;

        //Get the Stream returned from the response
        stream = fileResponse.GetResponseStream();

        LengthToRead = stream.Length;

        //Indicate the type of data being sent
        Response.ContentType = "application/octet-stream";

        //Name the file 
        Response.AddHeader("Content-Disposition", "attachment; filename=SolutionWizardDesktopClient.zip");
        Response.AddHeader("Content-Length", fileResponse.ContentLength.ToString());

        int length;
        do
        {
            // Verify that the client is connected.
            if (Response.IsClientConnected)
            {
                byte[] buffer = new Byte[bytesToRead];

                // Read data into the buffer.
                length = stream.Read(buffer, 0, bytesToRead);

                // and write it out to the response's output stream
                Response.OutputStream.Write(buffer, 0, length);

                // Flush the data
                Response.Flush();

                //Clear the buffer
                LengthToRead = LengthToRead - length;
            }
            else
            {
                // cancel the download if client has disconnected
                LengthToRead = -1;
            }
        } while (LengthToRead > 0); //Repeat until no data is read

    }
    finally
    {
        if (stream != null)
        {
            //Close the input stream                   
            stream.Close();
        }
        Response.End();
        Response.Close();
    }
    return View("Failed");
}

由于文件的大小，它正在消耗更多的内存，从而导致性能问题。
检查 iis 日志后，下载过程分别占用 42 MB 和 64 MB。
提前致谢

Answer 1

更好的选择是使用 FileResult 而不是 ActionResult:

使用此方法意味着您不必在服务前将 file/bytes 加载到内存中。

public FileResult Download()
{
     var filePath = "file path in server";
     return new FilePathResult(Server.MapPath(filePath), "application/zip");
}

编辑：对于较大的文件，FilePathResult 也会失败。

那么您最好的选择可能是 Response.TransmitFile()。我在较大的文件 (GB) 上使用过它并且之前没有问题

public ActionResult Download()
{
   var filePath = @"file path from server";

    Response.Clear();
    Response.ContentType = "application/octet-stream";
    Response.AppendHeader("Content-Disposition", "filename=" + filePath);

    Response.TransmitFile(filePath);

    Response.End();

    return Index();
}

来自 MSDN：

Writes the specified file directly to an HTTP response output stream, without buffering it in memory.

Answer 2

你只需要使用 IIS 启用 HTTP 下载看看这个 link

而且您只需要 return 文件的 HTTP 路径，它就会下载得如此快速和容易。

Answer 3

尝试将 Transfer-Encoding header 设置为分块，并 return 一个带有 PushStreamContent 的 HttpResponseMessage。 Transfer-Encoding of chunked 意味着 HTTP 响应不会有 Content-Length header，因此客户端必须将 HTTP 响应的块解析为流。请注意，我从来没有运行跨越不处理分块传输编码的客户端（浏览器等）。您可以在下面的 link 阅读更多内容。

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding

    [HttpGet]
    public async Task<HttpResponseMessage> Download(CancellationToken token)
    {
        var response = new HttpResponseMessage(System.Net.HttpStatusCode.OK)
        {
            Content = new PushStreamContent(async (stream, context, transportContext) =>
            {
                try
                {
                    using (var fileStream = System.IO.File.OpenRead("some path to MyBigDownload.zip"))
                    {
                        await fileStream.CopyToAsync(stream);
                    }
                }
                finally
                {
                    stream.Close();
                }
            }, "application/octet-stream"),
        };
        response.Headers.TransferEncodingChunked = true;
        response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment")
        {
            FileName = "MyBigDownload.zip"
        };
        return response;
    }

Answer 4

我遇到了类似的问题，但本地磁盘上没有文件，我不得不从 API 下载它（我的 MVC 就像一个代理）。关键是在您的 MVC 操作上设置 Response.Buffer=false;。我认为@JanusPienaar 的第一个解决方案应该适用于此。我的 MVC 操作是：

public class HomeController : Controller
{
    public async Task<FileStreamResult> Streaming(long RecordCount)
    {
        HttpClient Client;
        System.IO.Stream Stream;

        //This is the key thing
        Response.Buffer=false;

        Client = new HttpClient() { BaseAddress=new Uri("http://MyApi", };
        Stream = await Client.GetStreamAsync("api/Streaming?RecordCount="+RecordCount);
        return new FileStreamResult(Stream, "text/csv");
    }
}

我的测试 WebApi（生成文件）是：

public class StreamingController : ApiController
{
    // GET: api/Streaming/5
    public HttpResponseMessage Get(long RecordCount)
    {
        var response = Request.CreateResponse();

        response.Content=new PushStreamContent((stream, http, transport) =>
        {
            RecordsGenerator Generator = new RecordsGenerator();
            long i;

            using(var writer = new System.IO.StreamWriter(stream, System.Text.Encoding.UTF8))
            {
                for(i=0; i<RecordCount; i++)
                {
                    writer.Write(Generator.GetRecordString(i));

                    if(0==(i&0xFFFFF))
                        System.Diagnostics.Debug.WriteLine($"Record no: {i:N0}");
                    }
                }
            });

            return response;
        }

        class RecordsGenerator
        {
            const string abc = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
            char[] Chars = new char[14];//Ceiling(log26(2^63))

            public string GetRecordString(long Record)
            {
                int iLength = 0;
                long Div = Record, Mod;

                do
                {
                    iLength++;
                    Div=Math.DivRem(Div, abc.Length, out Mod);
                    //Save from backwards
                    Chars[Chars.Length-iLength]=abc[(int)Mod];
                }
                while(Div!=0);

                return $"{Record} {new string(Chars, Chars.Length-iLength, iLength)}\r\n";
            }
        }
    }
}

如果RecordCount 为100000000，则TestApi 生成的文件为1.56 GB。 WebApi 和 MVC 都不会消耗这么多内存。

Answer 5

有一个 Rizwan Ansari post 对我有用：

There are situation when you need to provide download option for a big file located somewhere on server or generated at runtime. Below function could be used to download files of any size. Sometimes downloading big file throws exception OutOfMemoryException showing “Insufficient memory to continue execution of the program”. So this function also handle this situation by breaking down file in 1 MB chunks (can be customized by changing bufferSize variable).

用法：

DownloadLargeFile("A big file.pdf", "D:\Big Files\Big File.pdf", "application/pdf", System.Web.HttpContext.Current.Response);

你可以改"application/pdf"右边Mime type

下载功能：

public static void DownloadLargeFile(string DownloadFileName, string FilePath, string ContentType, HttpResponse response)
    {
        Stream stream = null;

        // read buffer in 1 MB chunks
        // change this if you want a different buffer size
        int bufferSize = 1048576;

        byte[] buffer = new Byte[bufferSize];

        // buffer read length
        int length;
        // Total length of file
        long lengthToRead;

        try
        {
            // Open the file in read only mode 
            stream = new FileStream(FilePath, FileMode.Open, FileAccess.Read, FileShare.Read);

            // Total length of file
            lengthToRead = stream.Length;
            response.ContentType = ContentType;
            response.AddHeader("Content-Disposition", "attachment; filename=" + HttpUtility.UrlEncode(DownloadFileName, System.Text.Encoding.UTF8));

            while (lengthToRead > 0)
            {
                // Verify that the client is connected.
                if (response.IsClientConnected)
                {
                    // Read the data in buffer
                    length = stream.Read(buffer, 0, bufferSize);

                    // Write the data to output stream.
                    response.OutputStream.Write(buffer, 0, length);

                    // Flush the data 
                    response.Flush();

                    //buffer = new Byte[10000];
                    lengthToRead = lengthToRead - length;
                }
                else
                {
                    // if user disconnects stop the loop
                    lengthToRead = -1;
                }
            }
        }
        catch (Exception exp)
        {
            // handle exception
            response.ContentType = "text/html";
            response.Write("Error : " + exp.Message);
        }
        finally
        {
            if (stream != null)
            {
                stream.Close();
            }
            response.End();
            response.Close();
        }
    }

C#以更少的内存消耗从服务器下载大文件

C# Download big file from Server with less memory consumption

c#

asp.net-mvc

stream

filestream