在每次迭代中将对象放置在内存中

Question

我正在为现有的 webjob 编写一个增强功能，它从一个 table（外环）获取记录并将其推送到另一个（内环）。外环 table 大约有 7500 万条记录，我想非常有选择性地选择要推送到内环的数据 table。我正在使用一个存储过程来获取数据，将其分页，然后 return 将其推回内环。

    public async Task ExecuteAsync(
        AMS360DbContext outerRingDb
        , MiddleTierCoreDbContext innerRingDb
        , SyncSupportDbContext syncDb
        , ILogger logger
        , CancellationToken cancellationToken)
    {
        var log = logger.CoverageSync(this);

        TelemetryClient telemetry = new TelemetryClient();
        log.SyncStarted("Coverages");
        var sw = Stopwatch.StartNew();

        var transformer = new AmsPolicyCoveragesTransformer();


        const string innerRingTable = InnerRingTables.PolicyCoverages;
        const string sourceSchema = "ams360";
        const string sourceStoredProcedure = "GetPolicyCoverages";

        try
        {
            const int pageSize = 50_000;
            var page = 0;
            bool hasMoreRecords;

            var rowVersion = await syncDb.GetInnerRingRowVersionAsync(innerRingTable, sourceSchema, sourceStoredProcedure);

            do
            {
                if (cancellationToken.IsCancellationRequested)
                    return;

                var index = page * pageSize;

                log.SyncInProgress(index, index + pageSize - 1);

                var rowVersionParam = new SqlParameter()
                {
                    ParameterName = "@RowVersion",
                    SqlDbType = SqlDbType.Timestamp,
                    Direction = ParameterDirection.Input,
                    Value = rowVersion != null ? BitConverter.GetBytes(Convert.ToUInt64(rowVersion)).Reverse().ToArray() : (object)DBNull.Value
                };

                var prms = new SqlParameter[]
                {
                    new SqlParameter("@PageStart", index),
                    new SqlParameter("@PageSize", pageSize),
                    rowVersionParam
                };

                var outerRingCoverages = await outerRingDb.Set<SpPolicyCoverages>()
                                                    .FromSqlRaw("EXEC ams360.GetPolicyCoverages @PageStart, @PageSize, @RowVersion", prms)
                                                    .ToListAsync(cancellationToken);

                page++;

                var transformed = transformer.Transform(outerRingCoverages).ToList();

                if (transformed.Any())
                {
                    await MergeToInnerRingAsync(innerRingDb, transformed, cancellationToken);

                    var latestVersion = outerRingCoverages.Max(x => x.RowVersion);

                    telemetry.TrackEvent("Inner Ring Sync - Coverages Success");

                    await syncDb.UpdateInnerRingSyncRowVersionAsync(innerRingTable, sourceSchema, sourceStoredProcedure, latestVersion.Value, cancellationToken);

                }

                hasMoreRecords = (outerRingCoverages.Count == pageSize);

            } while (hasMoreRecords);
        }
        catch (Exception ex)
        {
            log.SyncError("Coverages", ex);
            telemetry.TrackEvent("Inner Ring Sync - Coverages Error");
            throw;
        }

        log.SyncFinished("Coverages", sw.Elapsed);


    }

我在使用此代码时遇到的问题是，在迭代过程中的某个时刻，我运行遇到了 OutofMemory 异常。所以我正在查看诊断工具，我惊讶地发现当我从 table 中获得 SpPolicyCoverages 的 50000 条记录时，在内存堆中创建了 50,000 个对象，在下一次迭代中，创建了 100,000 个对象在内存中，并且随着它继续在内存中累积，直到它运行进入 OutofMemory 异常。在每次迭代中处理对象 (SpPolicyCoverages) 的最佳方法是什么，这样我就不会运行进入 OutofMemory 异常？请指教..提前致谢。

Answer 1

SpPolicyCoverages 是实体类型吗？如果是这样，这可能是一个问题，因为 ChangeTracking 未在数据库上下文中禁用。更改跟踪没有任何不同只是因为您使用的是 FromSqlRaw。尝试在上下文中禁用更改跟踪或使用 .AsNoTracking():

var outerRingCoverages = await outerRingDb.Set<SpPolicyCoverages>()
    .FromSqlRaw("EXEC ams360.GetPolicyCoverages @PageStart, @PageSize, @RowVersion", prms)
    .AsNoTracking() // <--
    .ToListAsync(cancellationToken);

在每次迭代中将对象放置在内存中

Disposing objects in memory in each iteration

c#

out-of-memory