ASP.NET return 动态生成的二进制文件，不将整个内容存储在内存中

Question

互联网上有很多地方展示了如何 return 文件，但我发现 none 将 return 动态生成二进制数据而不存储全部内容记忆。也许我应该使用 Json 而不是 protobufers 来序列化我的数据。

感谢 this question 我能够创建这样的东西：

[HttpGet]
public ActionResult DownloadItems()
{
    // get 100K items from database as IEnumerable.
    IEnumerable<SomeObject> items = myDatabase.query("my query that returns 100K objects");

    // create memory stream where to place serialized items
    MemoryStream ms = new ();

    // write all serialized items to stream
    foreach(var item in items)
    {
         byte[] itemSerialized = item.BinarySerialize();
         ms.Write(itemSerialized,0,itemSerialized.Length);
    }

    // set position to the begining of memory stream
    ms.Position = 0;

    return File(ms, "application /octet-stream", "foo.bin");
}

这很好用，但我正在将 100K 项加载到内存中。我的问题是我如何 return 相同的动态生成文件而不必将所有项目加载到内存中？

我记得当 returning 二进制文件时，HTTP 协议 return 是这样的：


HTTP response headers
...

---------SomeGUID--------------

.. binary data

---------SomeGUID--------------

因此我相信有这样的东西会让它工作（它有伪代码）：

[HttpGet]
public ActionResult DownloadItems()
{
    // get 100K items from database as IEnumerable.
    IEnumerable<SomeObject> items = myDatabase.query("my query that returns 100K objects");

    // write the begining of file (PSEUDO code)
    this.response.body.writeString("-----------------SomeGuid------------");

    // write all serialized items to stream
    foreach(var item in items)
    {
         byte[] itemSerialized = item.BinarySerialize();
         this.response.body.write(itemSerialized,0,itemSerialized.Length);
    }

    // set position to the begining of memory stream
    ms.Position = 0;

    this.response.body.writeString("-----------------SomeGuid------------");
}

我可以安装 fiddler 或任何其他代理来查看文件的真实二进制传输情况。但是有没有一种构建方法可以让我不必经历所有这些麻烦？

Answer 1

我刚刚为此创建了自己的假文件流：

public class FakeFileStream : Stream
{
    private readonly IEnumerator<object> _enumerator;
    private bool _completed;

    public FakeFileStream(IEnumerable<object> items)
    {
        if (items is null)
            throw new ArgumentNullException();

        _enumerator = items.GetEnumerator();
    }


    public override int Read(byte[] buffer, int offset, int count)
    {
        if (_enumerator.MoveNext())
        {
            var currentItem = _enumerator.Current;

            // deserialize item.  
            byte[] itemSerialized = currentItem.SerializeUsingDotNetProtoBuf();

            // this will probably not happen but it is a good idea to have it implemented.
            // if this is the case store data on memory and return it on the next read
            if (itemSerialized.Length > buffer.Length)
                throw new NotImplementedException();

            // copy data to buffer
            Buffer.BlockCopy(itemSerialized, 0, buffer, 0, itemSerialized.Length);
            return itemSerialized.Length;
        }
        else
        {
            _completed = true;
            return 0;
        }
    }

    // unused methods
    public override void Flush() => throw new Exception("Unused method");
    public override long Seek(long offset, SeekOrigin origin) => throw new Exception("Unused method");
    public override void SetLength(long value) => throw new Exception("Unused method");
    public override void Write(byte[] buffer, int offset, int count) => throw new Exception("Unused method");

    // Properties
    public override bool CanRead => !_completed;
    public override bool CanSeek => false;
    public override bool CanWrite => false;
    public override long Length => throw new NotImplementedException("Not needed");
    public override long Position
    {
        get => throw new Exception("Unused property");
        set => throw new Exception("Unused property");
    }

    // Implement IDisposable
    public override ValueTask DisposeAsync()
    {
        _enumerator.Dispose();
        return base.DisposeAsync();
    }
}

我的端点如下所示：

[HttpGet]
[ProducesResponseType(StatusCodes.Status200OK, Type = typeof(FileStreamResult))]
public IActionResult GetBinary()
{         
    // get some IEnumerable collection 
    IEnumerable<Foo> items = MyDatabase.MyTable.Find("my query");

    // create a fake file stream
    var fs = new FakeFileStream(items);

    return File(fs, "application/octet-stream"); 
}

编辑

不要使用建议 FakeFileStream。出于某种原因，它给了我一些问题。可能是因为它写入流的字节很少。

无论如何，我无法通过控制器执行此操作。但是我能够使用中间件来做到这一点。我不得不这样做：

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.UseHttpsRedirection();

// any previous middleware you may have

app.Use(async (context, next) =>
{
    var downloadFile = context.Request.Query["downloadFile"];
    if (!string.IsNullOrWhiteSpace(downloadFile))
    {
        context.Response.ContentType = "application/octet-stream";

        // get 100K items from database as IEnumerable.
        IEnumerable<SomeObject> items = myDatabase.query("my query that returns 100K objects");

    

        // write all serialized items to stream
        foreach(var item in items)
        {
             byte[] itemSerialized = item.BinarySerialize();
             await context.WriteAsync(itemSerialized, context.CancelationToken);
        }

        return;

  
    }


    // Call the next delegate/middleware in the pipeline.
    await next(context);
});

// etc rest of your middleware

app.Run();

Answer 2

与其尝试重用 File() / FileStreamResult，我建议实施您自己的 ActionResult 并将内容呈现到那里的响应流。

public class ByteStreamResult : ActionResult
{
    private readonly IEnumerable<byte[]> blobs;

    public ByteStreamResult(IEnumerable<byte[]> blobs)
    {
        this.blobs = blobs;
    }

    public override async Task ExecuteResultAsync(ActionContext context)
    {
        context.HttpContext.Response.ContentType = "application/octet-stream";
        foreach (var item in blobs)
            await context.HttpContext.Response.Body.WriteAsync(item, context.HttpContext.RequestAborted);
    }
}

return new ByteStreamResult(items.Select(i => i.BinarySerialize()));

或者您可以更进一步，实施 custom formatter。

ASP.NET return 动态生成的二进制文件，不将整个内容存储在内存中

ASP.NET return dynamic generated binary file without storing entire content in memory

c#

asp.net

binary-data

编辑