在 Apache Ignite 中插入大量数据

Question

我正在使用 Apache Ignite .NET 瘦客户端 2.8.1 在 Apache Ignite 节点中插入大量数据。 Ignite 托管在 Amazon Linux AMI 上。我正在尝试使用 PutAllAsync 方法插入 500000 条记录：

await cacheClient.PutAllAsync(entities); // ICacheClient<int, T>

之后我在客户端日志中看到以下异常：

2021-02-03 07:23:37.9917 - NO_TRACE - ****************** - Error: Could not get data from cache
Exception has been thrown by the target of an invocation. System.Reflection.TargetInvocationException System.Object InvokeMethod(System.Object, System.Object[], System.Signature, Boolean) System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at Apache.Ignite.Core.Impl.Client.ClientSocket.SocketRead(Byte[] buf, Int32 pos, Int32 len)
   at Apache.Ignite.Core.Impl.Client.ClientSocket.ReceiveBytes(Int32 size)
   at Apache.Ignite.Core.Impl.Client.ClientSocket.ReceiveMessage()
   at Apache.Ignite.Core.Impl.Client.ClientSocket.SendRequest(RequestMessage& reqMsg)
   at Apache.Ignite.Core.Impl.Client.ClientSocket.DoOutInOp[T](ClientOp opId, Action`1 writeAction, Func`2 readFunc, Func`3 errorFunc)
   at Apache.Ignite.Core.Impl.Client.Cache.CacheClient`2.DoOutInOp[T](ClientOp opId, Action`1 writeAction, Func`2 readFunc)
   at Apache.Ignite.Core.Impl.Client.Cache.CacheClient`2.DoOutOp(ClientOp opId, Action`1 writeAction)

并且在 ignite 日志中出现以下错误：

[07:44:53,529][WARNING][grid-timeout-worker-#22][ClientListenerNioListener] Unable to perform handshake within timeout [timeout=10000, remoteAddr=/172.31.56.14:52631]

有没有关于如何在 ignite 缓存中插入超过 5000 条记录的最佳实践？在事务中插入批次？

Answer 1

也许您确实需要使用某种流媒体模式。
如果你有持久性，考虑事先禁用WAL。
也许您遇到了一些具体问题，例如 IGNITE-14076。尝试使用较小的批次是否可以解决这个问题。
尝试在 ClientConnectorConfiguration 和客户端配置中为瘦客户端调整超时。

Answer 2

Ignite is hosted on Amazon Linux AMI

你在哪里运行瘦客户端？我怀疑问题可能只是服务器和客户端之间的低连接速度。

你能测试连接速度吗？
数据的大小是多少，以兆字节为单位？
PutAll 处理 10、100、1000 个条目需要多少时间？

瘦客户端中还没有DataStreamer，所以我的建议是：

将一个大的 PutAll 分成多个较小的（例如一次 100 个条目）
增加 server and client

在 Apache Ignite 中插入大量数据

Inserting large amount of data in Apache Ignite

c#

ignite