Entity Framework：Count() 在大型 DbSet 和复杂的 WHERE 子句上非常慢

Question

我需要使用相对复杂的表达式作为 WHERE 子句对这个 Entity Framework (EF6) 数据集执行计数操作，并期望它达到 return 大约 100k 条记录。

计数操作显然是具体化记录的地方，因此是最慢的操作。在我们的生产环境中，计数操作大约需要 10 秒，即 unacceptable.

请注意，操作是直接在 DbSet 上执行的（db 是上下文 class），因此不应发生延迟加载。

如何进一步优化此查询以加快处理速度？

主要用例是显示具有多个过滤条件的索引页面，但该函数还用于根据服务中其他操作的需要将通用查询写入 ParcelOrderstable classes 这可能是一个坏主意，导致由于懒惰而导致非常复杂的查询，并且可能成为未来的问题。该计数稍后用于分页，实际显示的记录数（例如 500）要少得多。这是一个使用 SQL 服务器的数据库优先项目。

ParcelOrderSearchModel 是一个 C#-class，用于封装查询参数，并且仅供服务 classes 使用，以便调用 GetMatchingOrders 函数。请注意，在大多数调用中，ParcelOrderSearchModel 的大部分参数将为空。

public List<ParcelOrderDto> GetMatchingOrders(ParcelOrderSearchModel searchModel)
{
        // cryptic id known --> allow public access without login
        if (String.IsNullOrEmpty(searchModel.KeyApplicationUserId) && searchModel.ExactKey_CrypticID == null)
            throw new UnableToCheckPrivilegesException();

        Func<ParcelOrder, bool> userPrivilegeValidation = (x => false);

        if (searchModel.ExactKey_CrypticID != null)
        {
            userPrivilegeValidation = (x => true);
        }
        else if (searchModel.KeyApplicationUserId != null)
            userPrivilegeValidation = privilegeService.UserPrivilegeValdationExpression(searchModel.KeyApplicationUserId);

        var criteriaMatchValidation = CriteriaMatchValidationExpression(searchModel);
    
        var parcelOrdersWithNoteHistoryPoints = db.HistoryPoint.Where(hp => hp.Type == (int)HistoryPointType.Note)
            .Select(hp => hp.ParcelOrderID)
            .Distinct();

        Func<ParcelOrder, bool> completeExpression = order => userPrivilegeValidation(order) && criteriaMatchValidation(order);
        searchModel.PaginationTotalCount = db.ParcelOrder.Count(completeExpression);
       
        // todo: use this count for pagination
}


public Func<ParcelOrder, bool> CriteriaMatchValidationExpression(ParcelOrderSearchModel searchModel)
{
        Func<ParcelOrder, bool> expression =
            po => po.ID == 1;

        expression =
           po =>
           (searchModel.KeyUploadID == null || po.UploadID == searchModel.KeyUploadID)
       && (searchModel.KeyCustomerID == null || po.CustomerID == searchModel.KeyCustomerID)
       && (searchModel.KeyContainingVendorProvidedId == null || (po.VendorProvidedID != null && searchModel.KeyContainingVendorProvidedId.Contains(po.VendorProvidedID)))
       && (searchModel.ExactKeyReferenceNumber == null || (po.CustomerID + "-" + po.ReferenceNumber) == searchModel.ExactKeyReferenceNumber)
       && (searchModel.ExactKey_CrypticID == null || po.CrypticID == searchModel.ExactKey_CrypticID)
       && (searchModel.ContainsKey_ReferenceNumber == null || (po.CustomerID + "-" + po.ReferenceNumber).Contains(searchModel.ContainsKey_ReferenceNumber))
       && (searchModel.OrKey_Referencenumber_ConsignmentID == null ||
               ((po.CustomerID + "-" + po.ReferenceNumber).Contains(searchModel.OrKey_Referencenumber_ConsignmentID)
               || (po.VendorProvidedID != null && po.VendorProvidedID.Contains(searchModel.OrKey_Referencenumber_ConsignmentID))))
       && (searchModel.KeyClientName == null || po.Parcel.Name.ToUpper().Contains(searchModel.KeyClientName.ToUpper()))
       && (searchModel.KeyCountries == null || searchModel.KeyCountries.Contains(po.Parcel.City.Country))
       && (searchModel.KeyOrderStates == null || searchModel.KeyOrderStates.Contains(po.State.Value))
       && (searchModel.KeyFromDateRegisteredToOTS == null || po.DateRegisteredToOTS > searchModel.KeyFromDateRegisteredToOTS)
       && (searchModel.KeyToDateRegisteredToOTS == null || po.DateRegisteredToOTS < searchModel.KeyToDateRegisteredToOTS)
       && (searchModel.KeyFromDateDeliveredToVendor == null || po.DateRegisteredToVendor > searchModel.KeyFromDateDeliveredToVendor)
       && (searchModel.KeyToDateDeliveredToVendor == null || po.DateRegisteredToVendor < searchModel.KeyToDateDeliveredToVendor);
        return expression;
}

public Func<ParcelOrder, bool> UserPrivilegeValdationExpression(string userId)
{
        var roles = GetRolesForUser(userId);

        Func<ParcelOrder, bool> expression =
            po => po.ID == 1;
        if (roles != null)
        {
            if (roles.Contains("ParcelAdministrator"))
                expression =
                    po => true;

            else if (roles.Contains("RegionalAdministrator"))
            {
                var user = db.AspNetUsers.First(u => u.Id == userId);
                if (user.RegionalAdministrator != null)
                {
                    expression =
                        po => po.HubID == user.RegionalAdministrator.HubID;
                }
            }
            else if (roles.Contains("Customer"))
            {
                var customerID = db.AspNetUsers.First(u => u.Id == userId).CustomerID;
                expression =
                    po => po.CustomerID == customerID;
            }
            else
            {
                expression =
                    po => false;
            }
        }

        return expression;
}

Answer 1

如果可以避免的话，不要算分页。只是 return 第一页。计算起来总是很昂贵，对用户体验的提升也很少。

无论如何，您构建的动态搜索都是错误的。

您正在调用 IEnumerable.Count(Func<ParcelOrder,bool>)，这将强制在您应该调用 IQueryable.Count(Expression<Func<ParcelOrder,bool>>) 的位置进行客户端评估。这里：

    Func<ParcelOrder, bool> completeExpression = order => userPrivilegeValidation(order) && criteriaMatchValidation(order);
    searchModel.PaginationTotalCount = db.ParcelOrder.Count(completeExpression);

但是在 EF 中有一个更简单、更好的模式：只需有条件地向 IQueryable 添加条件。

例如，像这样在您的 DbContext 上放置一个方法：

public IQueryable<ParcelOrder> SearchParcels(ParcelOrderSearchModel searchModel)
{
        var q = this.ParcelOrders();
        if (searchModel.KeyUploadID != null)
        {
          q = q.Where( po => po.UploadID == searchModel.KeyUploadID );
        }
        if (searchModel.KeyCustomerID != null)
        {
          q = q.Where( po.CustomerID == searchModel.KeyCustomerID );
        }
        //. . .
        return q;
}

Entity Framework：Count() 在大型 DbSet 和复杂的 WHERE 子句上非常慢

Entity Framework: Count() very slow on large DbSet and complex WHERE clause

c#

sql-server

asp.net

entity-framework

entity-framework-6