节点在关闭后不会重新加入集群

Question

我正在使用 Akka.NET 的集群 (1.0.5) 功能来实现一个服务，该服务由一个主节点组成，该节点通过 HTTP 接收请求并将工作分配给已加入集群的工作节点群集。

想法是能够轻松完成以下任务：

在需求高时向集群添加工作节点（检查）
能够重启主节点或使其脱机 (maintentance/misbehaviour/whatever) 并在可用时让工作节点重新连接（检查）
upgrade/reboot 一个行为不端的 worker 并让它重新连接到主节点（失败！）

第一点如您所料：启动一个新实例（Azure 云服务工作者角色），并加入主节点 - 这也是种子节点。

对于第二点，所有工作节点都有一个侦听集群八卦的参与者，它确定主节点是否已死亡。如果是这种情况，工作节点参与者系统将重新启动。

最后一点是我卡住的地方。主节点还侦听集群八卦以确定工作人员何时变得无法访问（ClusterEvent.UnreachableMember）或正在关闭（退出状态）并决定是否应该关闭它。根据我从文档中了解到的内容，让同一节点的 "new" 版本重新加入集群的唯一方法是先关闭旧版本。

不幸的是，这似乎并没有发生。在测试场景中，我运行在计算模拟器中本地重现问题，步骤如下：

启动master节点（8090端口）
启动worker节点（9090端口）
做一些工作
突然杀死工作节点
启动工作节点备份

以下是我在此测试期间为两个节点收集的日志中的相关片段：

师傅：

工作人员变得无法访问：

[WARNING][07/12/2015 20:39:35][Thread 0023][[akka://InventoryService/system/cluster/core/daemon]] Cluster Node [akka.tcp://InventoryService@127.0.0.1:8090] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://InventoryService@0.0.0.0:9090, status = Up]

主节点调用 Cluster.Leave() 和 Cluster.Down() worker 的地址：

[DEBUG][07/12/2015 20:39:35][Thread 0021][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.ClusterUserAction+Leave
[INFO][07/12/2015 20:39:35][Thread 0021][[akka://InventoryService/system/cluster/core/daemon]] Marked address [akka.tcp://InventoryService@0.0.0.0:9090] as Leaving]
[DEBUG][07/12/2015 20:39:35][Thread 0021][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.ClusterUserAction+Down
[INFO][07/12/2015 20:39:35][Thread 0021][[akka://InventoryService/system/cluster/core/daemon]] Marking unreachable node [akka.tcp://InventoryService@0.0.0.0:9090] as Down
[DEBUG][07/12/2015 20:39:35][Thread 0020][[akka://InventoryService/system/cluster/core/daemon/heartbeatSender]] Cluster Node [akka.tcp://InventoryService@127.0.0.1:8090] - Heartbeat to [akka.tcp://InventoryService@0.0.0.0:9090]
[INFO][07/12/2015 20:39:36][Thread 0021][[akka://InventoryService/system/cluster/core/daemon]] Leader is removing unreachable node [akka.tcp://InventoryService@0.0.0.0:9090]

Master 确认旧节点将不再被允许加入（不过似乎有一个错误，请看第一行 - gated instead for akka.tcp://InventoryService@0.0.0.0:9090 ms，我想这应该是它应该被门控的时间） :

[WARNING][07/12/2015 20:39:36][Thread 0013][remoting] Association to [akka.tcp://InventoryService@0.0.0.0:9090] with unknown UID is reported as quarantined, but address cannot be quarantined without knowing the UID, gated instead for akka.tcp://InventoryService@0.0.0.0:9090 ms
[DEBUG][07/12/2015 20:39:36][Thread 0015][[akka://InventoryService/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fInventoryService%400.0.0.0%3a9090-2/endpointWriter]] Disassociated [akka.tcp://InventoryService@127.0.0.1:8090] -> akka.tcp://InventoryService@0.0.0.0:9090
[DEBUG][07/12/2015 20:39:36][Thread 0020][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Association to [akka.tcp://InventoryService@0.0.0.0:9090] having UID [1198519768] is irrecoverably failed. UID is now quarantined and all messages to this UID will be delivered to dead letters. Remote actorsystem must be restarted to recover from this situation.
[WARNING][07/12/2015 20:39:36][Thread 0013][remoting] Association to [akka.tcp://InventoryService@0.0.0.0:9090] having UID [1198519768] is irrecoverably failed. UID is now quarantined and all messages to this UID will be delivered to dead letters. Remote actorsystem must be restarted to recover from this situation.

Worker 启动并尝试连接到 master：

[DEBUG][07/12/2015 20:40:20][Thread 0013][remoting] Associated [akka.tcp://InventoryService@127.0.0.1:8090] <- akka.tcp://InventoryService@0.0.0.0:9090
[DEBUG][07/12/2015 20:40:21][Thread 0023][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:40:21][Thread 0020][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:40:23][Thread 0021][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:40:28][Thread 0020][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:40:33][Thread 0023][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:40:38][Thread 0022][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:40:43][Thread 0021][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:40:48][Thread 0020][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:40:53][Thread 0023][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:40:58][Thread 0022][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:41:03][Thread 0021][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:41:08][Thread 0021][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:41:13][Thread 0020][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin
[DEBUG][07/12/2015 20:41:18][Thread 0023][[akka://InventoryService/system/cluster/core/daemon]] [Initialized] Received Akka.Cluster.InternalClusterAction+InitJoin

这里发生了什么？

工人：

被杀死后重新启动：

[DEBUG][07/12/2015 20:40:18][Thread 0020][[akka://InventoryService/system/cluster/core/daemon]] [Uninitialized] Received Akka.Cluster.InternalClusterAction+JoinSeedNodes
[DEBUG][07/12/2015 20:40:18][Thread 0020][[akka://InventoryService/system/cluster/core/daemon]] [Uninitialized] Received Akka.Cluster.InternalClusterAction+Subscribe
[DEBUG][07/12/2015 20:40:18][Thread 0020][[akka://InventoryService/system/cluster/core/daemon]] [Uninitialized] Received Akka.Cluster.InternalClusterAction+JoinSeedNodes
[DEBUG][07/12/2015 20:40:20][Thread 0021][[akka://InventoryService/system/cluster/core/daemon]] [Uninitialized] Received Akka.Cluster.InternalClusterAction+Subscribe
[DEBUG][07/12/2015 20:40:18][Thread 0020][[akka://InventoryService/system/cluster/core/daemon]] [Uninitialized] Received Akka.Cluster.InternalClusterAction+Subscribe
[DEBUG][07/12/2015 20:40:21][Thread 0015][[akka://InventoryService/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fInventoryService%40127.0.0.1%3a8090-1/endpointWriter]] Drained buffer with maxWriteCount: 50, fullBackoffCount: 1,smallBackoffCount: 0, noBackoffCount: 0,adaptiveBackoff: 10000

就是这样...没有其他内容写入日志！

完整日志文件：

大师：http://pastebin.com/raw.php?i=WtjEhV1V
工人：http://pastebin.com/raw.php?i=QGPxkqEd

主集群配置：

cluster {
    seed-nodes = ["master's address here"]
    roles = [ InventoryServiceMaster, InventoryServiceWorker ]
    failure-detector {
        acceptable-heartbeat-pause = 5s
        threshold = 10.0
    }
}

Worker 的配置相同，但只有 InventoryServiceWorker 角色。

我在这里错过了什么？这是配置问题吗？（我希望这不是一个错误 - 我在 Github 上看到其他人 report a similar problem）。

编辑：

需要说明的是，我没有使用 Nuget 的 Akka.dll，因为它包含一个序列化错误 - 我检查了当前的主控是否应用了修复并进行了发布构建。日志中有调试信息，因为我从构建中保留了 PDB。

编辑 2：

在worker日志中，重启后，事件Akka.Cluster.InternalClusterAction+JoinSeedNodes出现了两次，因为我原来手动调用了Cluster.JoinSeedNodes()。我已经删除了它，但结果仍然相同。

Answer 1

此问题已在 Akka.NET 1.1 中得到解决 - 我们的 UID 系统在此之前未正确实施（1.0.5，在此 post 时），但现在可以正常工作.

节点在关闭后不会重新加入集群

Node doesn't rejoin cluster after being downed

akka.net

akka.net-cluster