由于大量 "IsStillRunningService" 个对象，Hazelcast 集群成员内存不足

Question

我们有一个在 3.5 版本上使用 Hazelcast IExecutor Service 和 IMap 的系统。我们最近遇到 Hazelcast 集群成员在生产中内存不足，一个接一个，最后所有节点都因 OOM 而崩溃。
在进行因果分析时，我们发现下面有数千条日志条目，并且日志文件的大小呈指数级增长。此外，存在日志的存储 space 也有运行个 space。

WARNING: [10.7.90.189]:30103 [FB] [3.5] Asking if operation execution has been started: com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$InvokeIsStillRunningOperationRunnable@48b3ac3b
Mar 30, 2016 11:09:29 AM com.hazelcast.spi.impl.operationservice.impl.Invocation
WARNING: [10.7.90.189]:30103 [FB] [3.5] While asking 'is-executing': Invocation{ serviceName='hz:core:partitionService', op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{serviceName='hz:core:partition
Service', partitionId=-1, callId=59834, invocationTime=1459349279980, waitTimeout=-1, callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, tryPauseMillis=0, invokeCount=1, callTimeout=5000, target=Address[1.2.3.4]:30102, b
ackupsExpected=0, backupsCompleted=0}
com.hazelcast.core.OperationTimeoutException: No response for 10000 ms. Aborting invocation! Invocation{ serviceName='hz:core:partitionService', op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{servic
eName='hz:core:partitionService', partitionId=-1, callId=268177, invocationTime=1459349295209, waitTimeout=-1, callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, tryPauseMillis=0, invokeCount=1, callTimeout=5000, target=Addr
ess[10.7.90.190]:30102, backupsExpected=0, backupsCompleted=0} No response has been received!  backups-expected:0 backups-completed: 0
        at com.hazelcast.spi.impl.operationservice.impl.Invocation.newOperationTimeoutException(Invocation.java:491)
        at com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback.setOperationTimeout(IsStillRunningService.java:224)
        at com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$IsOperationStillRunningCallback.onFailure(IsStillRunningService.java:219)
        at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.run(InvocationFuture.java:137)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)

据我了解，集群成员会不断发出心跳以确保所有成员都存活，我相信默认值为 10 秒。现在的问题是，如果任何成员进入无响应或休状态，其余成员将继续进行正在执行的调用。查看堆转储后，了解到 >73% 的堆充满了 "IsStillRunningService" 个对象。

问题：

如何知道究竟出了什么问题？
运行 out of storage space 只是巧合还是可能有任何关联？我们怀疑一个可能导致另一个，因为它在一周内发生了两次。

Hazelcast XML 配置：

<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config http://www.hazelcast.com/schema/config/hazelcast-config-3.5.xsd"
           xmlns="http://www.hazelcast.com/schema/config"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <map name="myMap">
        <backup-count>0</backup-count>
        <time-to-live-seconds>43200</time-to-live-seconds>
        <eviction-policy>LRU</eviction-policy>
        <max-size policy="USED_HEAP_PERCENTAGE">75</max-size>
        <eviction-percentage>10</eviction-percentage>
        <in-memory-format>OBJECT</in-memory-format>
    </map>

    <executor-service name="calculation">
            <pool-size>10</pool-size>
        <queue-capacity>400</queue-capacity>
    </executor-service>

    <executor-service name="loader">
            <pool-size>5</pool-size>
        <queue-capacity>400</queue-capacity>
    </executor-service>

    <properties>
        <property name="hazelcast.icmp.timeout">5000</property>
        <property name="hazelcast.initial.wait.seconds">10</property>
        <property name="hazelcast.connection.monitor.interval">5000</property>
    </properties>

    <network>
        <port auto-increment="true" port-count="100">30101</port>
        <join>
            <multicast enabled="false">
                <multicast-group>224.2.2.3</multicast-group>
                <multicast-port>54327</multicast-port>
            </multicast>
            <tcp-ip enabled="true">
                <interface>1.2.3.4</interface>
                <interface>1.2.3.5</interface>
                <interface>1.2.3.6</interface>
            </tcp-ip>
            <aws enabled="false"/>
        </join>
        <interfaces enabled="false">
            <interface>127.0.0.1</interface>
        </interfaces>
    </network>
</hazelcast>

StackTrace
LinkedBlockingQueue which holds IsStillRunningService Objects

Answer 1

你能升级到3.6吗？添加了修复以防止运行使用 is-still-运行进入 OOME。在 3.7 中，整个机制将被移除，并被问题较少的方法所取代。

https://github.com/hazelcast/hazelcast/pull/7719

由于大量 "IsStillRunningService" 个对象，Hazelcast 集群成员内存不足

Hazelcast Cluster members going out of memory due to huge number of "IsStillRunningService" objects

hazelcast

hazelcast-imap