NPGSQL 开始文本导入在较大的 CSV 文件上引发异常

Question

编辑：在进行更多测试后，此错误与 HAPROXY 直接相关。当我从堆栈中取出它时，它按预期工作。

几天来我一直在为这个问题苦苦挣扎。此代码适用于较小的 CSV 文件 (25 KB)，但一旦您尝试处理较大的文件 (5000 KB)，它就会抛出异常。

我的堆栈是以下代码，从 asp.net 核心 3.0、NPGSQL 4.1.1 到 HAPROXY，再到 PostgreSQL 12 实例。

代码如下：

      string query = "COPY " + seedingDataApi.tableName + " FROM STDIN CSV ENCODING 'UTF8' QUOTE '\"'; ";
                using (var conn = postgresContext.GetDBConnectionPG())
                {

                    conn.Open();
                    using (var writer = conn.BeginTextImport(query))
                    {
                        _logger.LogInformation("Text import started");
                        using (StreamReader sr = new StreamReader(seedingDataApi.tableName + ".csv"))
                        {

                            _logger.LogInformation("Read stream started.");
                            while (!sr.EndOfStream)
                            {
                                string testString = "";

                                testString = sr.ReadLine();
                                while ("<EOL>" != testString.Substring(Math.Max(0, testString.Length - 5)))
                                {
                                    testString += sr.ReadLine();
                                }
                                testString = testString.Remove(testString.Length - 5);
                                writer.WriteLine(testString);
                            }

                        }
                    }
                }

应用抛出的异常为：

 System.Exception: Npgsql.NpgsqlException (0x80004005): Exception while reading from stream
 ---> System.IO.EndOfStreamException: Attempted to read past the end of the stream.
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.<>c__DisplayClass160_0.<<DoReadMessage>g__ReadMessageLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.<>c__DisplayClass160_0.<<DoReadMessage>g__ReadMessageLong|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Npgsql.NpgsqlConnector.ReadMessage(DataRowLoadingMode dataRowLoadingMode)
   at Npgsql.NpgsqlRawCopyStream..ctor(NpgsqlConnector connector, String copyCommand)
   at Npgsql.NpgsqlConnection.BeginTextImport(String copyFromCommand)
   at MiniAppApiServer.Repositories.TableSeedRepo.TableSeedRepository.SeedTableDataAsync(TableSeedDataParams seedingDataApi, CancellationToken ct) in /source/MiniAppApiServer/Repositories/TableSeedRepo/TableSeedRepository.cs:line 306System.IO.EndOfStreamException: Attempted to read past the end of the stream.
   at Npgsql.NpgsqlReadBuffer.<>c__DisplayClass34_0.<<Ensure>g__EnsureLong|0>d.MoveNext()
   at MiniAppApiServer.Repositories.TableSeedRepo.TableSeedRepository.SeedTableDataAsync(TableSeedDataParams seedingDataApi, CancellationToken ct) in /source/MiniAppApiServer/Repositories/TableSeedRepo/TableSeedRepository.cs:line 339
   at MiniAppApiServer.Supervisors.TableSeedSupervisor.TableSeedSupervisor.SeedTableDataAsync(Stream data, String tablename, CancellationToken ct) in /source/MiniAppApiServer/Supervisors/TableSeedSupervisor/TableSeedSupervisor.cs:line 86
   at MiniAppApiServer.Controllers.TableSeedingController.PostTableDataSeed(IFormFile file, CancellationToken ct) in /source/MiniAppApiServer/Controllers/TableSeedingController.cs:line 114

非常感谢！

Answer 1

已解决！问题出在 HAProxy 配置上。我更改了以下值：

timeout client          1m
timeout server          1m

对此：

timeout client          60m
timeout server          60m

NPGSQL 开始文本导入在较大的 CSV 文件上引发异常

NPGSQL Begin Text Import throws exception on larger CSV files

c#

postgresql

haproxy

npgsql

asp.net-core