'AsyncEnumerableReader' 枚举值时达到配置的最大缓冲区大小
'AsyncEnumerableReader' reached the configured maximum size of the buffer when enumerating a value
我正在使用返回 IAsyncEnumerable<> 的 async/await 方法从 SQL 服务器数据库检索行并通过 Web API .Net Core 3.0 接口提供它们。它工作正常，直到我超过 8192 行。对于超出该点的行，它会失败。我尝试了 3 个不同的表，每个表的行大小都非常不同。每次在生成 8192 行后发生故障。这是我看到的错误：
An unhandled exception occurred while processing the request.
InvalidOperationException: 'AsyncEnumerableReader' reached the configured maximum size of the buffer when enumerating a value of type 'Microsoft.EntityFrameworkCore.TestDatabase.TestDatabaseDbSet`1[TestNamespace.TestClass]'. This limit is in place to prevent infinite streams of 'IAsyncEnumerable<>' from continuing indefinitely. If this is not a programming mistake, consider ways to reduce the collection size, or consider manually converting 'Microsoft.EntityFrameworkCore.TestDatabase.TestDatabaseDbSet`1[TestNamespace.TestClass]' into a list rather than increasing the limit.
Microsoft.AspNetCore.Mvc.Infrastructure.AsyncEnumerableReader.ReadTestDatabase<T>(IAsyncEnumerable<object> value)
Stack Query Cookies Headers Routing
InvalidOperationException: 'AsyncEnumerableReader' reached the configured maximum size of the buffer when enumerating a value of type 'Microsoft.EntityFrameworkCore.TestDatabase.TestDatabaseDbSet`1[TestNamespace.TestClass]'. This limit is in place to prevent infinite streams of 'IAsyncEnumerable<>' from continuing indefinitely. If this is not a programming mistake, consider ways to reduce the collection size, or consider manually converting 'Microsoft.EntityFrameworkCore.TestDatabase.TestDatabaseDbSet`1[TestNamespace.TestClass]' into a list rather than increasing the limit.
Microsoft.AspNetCore.Mvc.Infrastructure.AsyncEnumerableReader.ReadTestDatabase<T>(IAsyncEnumerable<object> value)
Microsoft.AspNetCore.Mvc.Infrastructure.AsyncEnumerableReader.ReadTestDatabase<T>(IAsyncEnumerable<object> value)
Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor.ExecuteAsyncEnumerable(ActionContext context, ObjectResult result, IAsyncEnumerable<object> asyncEnumerable)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultFilters>g__Awaited|27_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
Show raw exception details
System.InvalidOperationException: 'AsyncEnumerableReader' reached the configured maximum size of the buffer when enumerating a value of type 'Microsoft.EntityFrameworkCore.TestDatabase.TestDatabaseDbSet`1[TestNamespace.TestClass]'. This limit is in place to prevent infinite streams of 'IAsyncEnumerable<>' from continuing indefinitely. If this is not a programming mistake, consider ways to reduce the collection size, or consider manually converting 'Microsoft.EntityFrameworkCore.TestDatabase.TestDatabaseDbSet`1[TestNamespace.TestClass]' into a list rather than increasing the limit.
at Microsoft.AspNetCore.Mvc.Infrastructure.AsyncEnumerableReader.ReadTestDatabase[T](IAsyncEnumerable`1 value)
at Microsoft.AspNetCore.Mvc.Infrastructure.AsyncEnumerableReader.ReadTestDatabase[T](IAsyncEnumerable`1 value)
at Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor.ExecuteAsyncEnumerable(ActionContext context, ObjectResult result, IAsyncEnumerable`1 asyncEnumerable)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultFilters>g__Awaited|27_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
reached the configured maximum size of the buffer when enumerating a value of type Microsoft.EntityFrameworkCore DbSet This limit is in place to prevent infinite streams from continuing indefinitely. If this is not a programming mistake, consider ways to reduce the collection size, or consider manually converting Microsoft.EntityFrameworkCore DbSet into a list rather than increasing the limit.
Microsoft.EntityFrameworkCore.DbSet AsyncEnumerable
发生了什么事
限制来自MvcOptions.MaxIAsyncEnumerableBufferLimit Property :
Gets or sets the most number of entries of an IAsyncEnumerable that that ObjectResultExecutor will buffer.
When Value is an instance of IAsyncEnumerable, ObjectResultExecutor will eagerly read the enumeration and add to a synchronous collection prior to invoking the selected formatter. This property determines the most number of entries that the executor is allowed to buffer.
(...)
Defaults to 8192.
修复
在我看来，从最好到最差。
一个。 (首选)不要return那么多记录
我建议不要 return 来自网络的这么多记录 api。
分页是处理大量记录的一种方法。你可以查看 Tutorial: Add sorting, filtering, and paging - ASP.NET MVC with EF Core.
乙。调整上限
MvcOptions.MaxIAsyncEnumerableBufferLimit 可以设置为其他值。
摄氏度。不要使用 IAsyncEnumerable
您可以 return IEnumerable - 它将能够处理更多的记录。
异常消息和描述完美地描述了情况。您正在将（异常大量的）数据读入缓冲区，并且框架警告您您可能正在做一些您不期望的事情，或者查询了太多信息。
由于您使用的是 WEBAPI，因此您会收到警告，因为 WEBAPI 和网站的目标通常是缩短响应时间，而必须收集 8K+ 记录的数据通常无法实现这一目标。话虽这么说，我们不知道您的应用程序的实际用例，您可能不在乎，但无论如何，该框架正在努力帮助您完成 "right thing".
无论您的经验水平如何，reading/interpreting 错误消息都是容易掉入的陷阱。相反，想想它们的意思，如果您不理解它们，请重新阅读和研究，直到您理解为止。自 guru meditations 以来，我们已经取得了长足的进步，所以不要将这些有用的信息视为理所当然。
TL;DR 您一次读取的数据过多。找到一种方法将数据分成更小的部分以 return 发送给客户端。较小的页面尺寸是一个很好的起点。
就我而言，我在将 odata-endpoints 从 .net core 2.2 迁移到 .net core 3.1 的上下文中遇到了这个问题。作为解决方案，我遇到了 article with provided solution。长话短说，我们应该创建自己的扩展方法 ToQueryable()，它不实现 IAsyncEnumerable 接口，并使用它代替现有的 AsQueryable()。
无需通过查询字符串进行额外过滤，它就像一个魅力。但是当我们试图在过滤查询中发送 $top=3 时，我们得到了错误：'InvalidCastException: Unable to cast object of type Queryable to type System.Linq.IOrderedQueryable'。
解决方案是更新上面引用的文章中提供的代码，将 IQueryable 替换为 IOrderedQueryable。
结果我得到了：
public static partial class CustomOrderedQueryable
{
public static IEnumerable<T> ToEnumerable<T>(this IEnumerable<T> source)
{
foreach (var item in source)
yield return item;
}
}
public static partial class CustomOrderedQueryable
{
public static IOrderedQueryable<T> ToQueryable<T>(this IQueryable<T> source)
=> new OrderedQueryable<T>(new QueryProvider(source.Provider), source.Expression);
class OrderedQueryable<T> : IOrderedQueryable<T>
{
internal OrderedQueryable(IQueryProvider provider, Expression expression)
{
Provider = provider;
Expression = expression;
}
public Type ElementType => typeof(T);
public Expression Expression { get; }
public IQueryProvider Provider { get; }
public IEnumerator<T> GetEnumerator() => Provider.Execute<IEnumerable<T>>(Expression)
.ToEnumerable().GetEnumerator();
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
class QueryProvider : IQueryProvider
{
private readonly IQueryProvider source;
internal QueryProvider(IQueryProvider source) => this.source = source;
public IQueryable CreateQuery(Expression expression)
{
var query = source.CreateQuery(expression);
return (IQueryable)Activator.CreateInstance(
typeof(OrderedQueryable<>).MakeGenericType(query.ElementType),
this, query.Expression);
}
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
=> new OrderedQueryable<TElement>(this, expression);
public object Execute(Expression expression) => source.Execute(expression);
public TResult Execute<TResult>(Expression expression) => source.Execute<TResult>(expression);
}
}
在 ASP.NET Core 5 中，IAsyncEnumerable 类型的实例确实已通过在内存中缓冲序列并一次性格式化缓冲集合来处理。这解释了为什么您会看到该异常。
但是，有了ASP.NET Core 6.0，这种情况就不会再发生了！
In ASP.NET Core 6, when formatting using System.Text.Json, MVC no longer buffers IAsyncEnumerable instances. Instead, MVC relies on the support that System.Text.Json added for these types ([reference][1])
有关如何使用 IAsyncEnumerable 的更多详细信息，请参阅。
我正在使用返回 IAsyncEnumerable<> 的 async/await 方法从 SQL 服务器数据库检索行并通过 Web API .Net Core 3.0 接口提供它们。它工作正常，直到我超过 8192 行。对于超出该点的行，它会失败。我尝试了 3 个不同的表，每个表的行大小都非常不同。每次在生成 8192 行后发生故障。这是我看到的错误：
An unhandled exception occurred while processing the request.
InvalidOperationException: 'AsyncEnumerableReader' reached the configured maximum size of the buffer when enumerating a value of type 'Microsoft.EntityFrameworkCore.TestDatabase.TestDatabaseDbSet`1[TestNamespace.TestClass]'. This limit is in place to prevent infinite streams of 'IAsyncEnumerable<>' from continuing indefinitely. If this is not a programming mistake, consider ways to reduce the collection size, or consider manually converting 'Microsoft.EntityFrameworkCore.TestDatabase.TestDatabaseDbSet`1[TestNamespace.TestClass]' into a list rather than increasing the limit.
Microsoft.AspNetCore.Mvc.Infrastructure.AsyncEnumerableReader.ReadTestDatabase<T>(IAsyncEnumerable<object> value)
Stack Query Cookies Headers Routing
InvalidOperationException: 'AsyncEnumerableReader' reached the configured maximum size of the buffer when enumerating a value of type 'Microsoft.EntityFrameworkCore.TestDatabase.TestDatabaseDbSet`1[TestNamespace.TestClass]'. This limit is in place to prevent infinite streams of 'IAsyncEnumerable<>' from continuing indefinitely. If this is not a programming mistake, consider ways to reduce the collection size, or consider manually converting 'Microsoft.EntityFrameworkCore.TestDatabase.TestDatabaseDbSet`1[TestNamespace.TestClass]' into a list rather than increasing the limit.
Microsoft.AspNetCore.Mvc.Infrastructure.AsyncEnumerableReader.ReadTestDatabase<T>(IAsyncEnumerable<object> value)
Microsoft.AspNetCore.Mvc.Infrastructure.AsyncEnumerableReader.ReadTestDatabase<T>(IAsyncEnumerable<object> value)
Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor.ExecuteAsyncEnumerable(ActionContext context, ObjectResult result, IAsyncEnumerable<object> asyncEnumerable)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultFilters>g__Awaited|27_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
Show raw exception details
System.InvalidOperationException: 'AsyncEnumerableReader' reached the configured maximum size of the buffer when enumerating a value of type 'Microsoft.EntityFrameworkCore.TestDatabase.TestDatabaseDbSet`1[TestNamespace.TestClass]'. This limit is in place to prevent infinite streams of 'IAsyncEnumerable<>' from continuing indefinitely. If this is not a programming mistake, consider ways to reduce the collection size, or consider manually converting 'Microsoft.EntityFrameworkCore.TestDatabase.TestDatabaseDbSet`1[TestNamespace.TestClass]' into a list rather than increasing the limit.
at Microsoft.AspNetCore.Mvc.Infrastructure.AsyncEnumerableReader.ReadTestDatabase[T](IAsyncEnumerable`1 value)
at Microsoft.AspNetCore.Mvc.Infrastructure.AsyncEnumerableReader.ReadTestDatabase[T](IAsyncEnumerable`1 value)
at Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor.ExecuteAsyncEnumerable(ActionContext context, ObjectResult result, IAsyncEnumerable`1 asyncEnumerable)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultFilters>g__Awaited|27_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
reached the configured maximum size of the buffer when enumerating a value of type Microsoft.EntityFrameworkCore DbSet This limit is in place to prevent infinite streams from continuing indefinitely. If this is not a programming mistake, consider ways to reduce the collection size, or consider manually converting Microsoft.EntityFrameworkCore DbSet into a list rather than increasing the limit.
Microsoft.EntityFrameworkCore.DbSet AsyncEnumerable
发生了什么事
限制来自MvcOptions.MaxIAsyncEnumerableBufferLimit Property :
Gets or sets the most number of entries of an IAsyncEnumerable that that ObjectResultExecutor will buffer.
When Value is an instance of IAsyncEnumerable, ObjectResultExecutor will eagerly read the enumeration and add to a synchronous collection prior to invoking the selected formatter. This property determines the most number of entries that the executor is allowed to buffer.
(...)
Defaults to 8192.
修复
在我看来，从最好到最差。
一个。 (首选)不要return那么多记录
我建议不要 return 来自网络的这么多记录 api。
分页是处理大量记录的一种方法。你可以查看 Tutorial: Add sorting, filtering, and paging - ASP.NET MVC with EF Core.
乙。调整上限
MvcOptions.MaxIAsyncEnumerableBufferLimit 可以设置为其他值。
摄氏度。不要使用 IAsyncEnumerable
您可以 return IEnumerable - 它将能够处理更多的记录。
异常消息和描述完美地描述了情况。您正在将（异常大量的）数据读入缓冲区，并且框架警告您您可能正在做一些您不期望的事情，或者查询了太多信息。
由于您使用的是 WEBAPI，因此您会收到警告，因为 WEBAPI 和网站的目标通常是缩短响应时间，而必须收集 8K+ 记录的数据通常无法实现这一目标。话虽这么说，我们不知道您的应用程序的实际用例，您可能不在乎，但无论如何，该框架正在努力帮助您完成 "right thing".
无论您的经验水平如何，reading/interpreting 错误消息都是容易掉入的陷阱。相反，想想它们的意思，如果您不理解它们，请重新阅读和研究，直到您理解为止。自 guru meditations 以来，我们已经取得了长足的进步，所以不要将这些有用的信息视为理所当然。
TL;DR 您一次读取的数据过多。找到一种方法将数据分成更小的部分以 return 发送给客户端。较小的页面尺寸是一个很好的起点。
就我而言，我在将 odata-endpoints 从 .net core 2.2 迁移到 .net core 3.1 的上下文中遇到了这个问题。作为解决方案，我遇到了 article with provided solution。长话短说，我们应该创建自己的扩展方法 ToQueryable()，它不实现 IAsyncEnumerable 接口，并使用它代替现有的 AsQueryable()。
无需通过查询字符串进行额外过滤，它就像一个魅力。但是当我们试图在过滤查询中发送 $top=3 时，我们得到了错误：'InvalidCastException: Unable to cast object of type Queryable to type System.Linq.IOrderedQueryable'。
解决方案是更新上面引用的文章中提供的代码，将 IQueryable 替换为 IOrderedQueryable。
结果我得到了：
public static partial class CustomOrderedQueryable
{
public static IEnumerable<T> ToEnumerable<T>(this IEnumerable<T> source)
{
foreach (var item in source)
yield return item;
}
}
public static partial class CustomOrderedQueryable
{
public static IOrderedQueryable<T> ToQueryable<T>(this IQueryable<T> source)
=> new OrderedQueryable<T>(new QueryProvider(source.Provider), source.Expression);
class OrderedQueryable<T> : IOrderedQueryable<T>
{
internal OrderedQueryable(IQueryProvider provider, Expression expression)
{
Provider = provider;
Expression = expression;
}
public Type ElementType => typeof(T);
public Expression Expression { get; }
public IQueryProvider Provider { get; }
public IEnumerator<T> GetEnumerator() => Provider.Execute<IEnumerable<T>>(Expression)
.ToEnumerable().GetEnumerator();
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
class QueryProvider : IQueryProvider
{
private readonly IQueryProvider source;
internal QueryProvider(IQueryProvider source) => this.source = source;
public IQueryable CreateQuery(Expression expression)
{
var query = source.CreateQuery(expression);
return (IQueryable)Activator.CreateInstance(
typeof(OrderedQueryable<>).MakeGenericType(query.ElementType),
this, query.Expression);
}
public IQueryable<TElement> CreateQuery<TElement>(Expression expression)
=> new OrderedQueryable<TElement>(this, expression);
public object Execute(Expression expression) => source.Execute(expression);
public TResult Execute<TResult>(Expression expression) => source.Execute<TResult>(expression);
}
}
在 ASP.NET Core 5 中，IAsyncEnumerable 类型的实例确实已通过在内存中缓冲序列并一次性格式化缓冲集合来处理。这解释了为什么您会看到该异常。
但是，有了ASP.NET Core 6.0，这种情况就不会再发生了！
In ASP.NET Core 6, when formatting using System.Text.Json, MVC no longer buffers IAsyncEnumerable instances. Instead, MVC relies on the support that System.Text.Json added for these types ([reference][1])
有关如何使用 IAsyncEnumerable 的更多详细信息，请参阅