无法重新注册 mesos 代理

Question

更新 mesos-slave 的属性 (isolation) 后，无法重新注册：

6868 status_update_manager.cpp:177] Pausing sending status updates
6877 slave.cpp:915] New master detected at master@192.168.1.1:5050
6867 status_update_manager.cpp:177] Pausing sending status updates
6877 slave.cpp:936] No credentials provided. Attempting to register without authentication
6877 slave.cpp:947] Detecting new master
6869 slave.cpp:1217] Re-registered with master master@192.168.1.1:5050
6866 status_update_manager.cpp:184] Resuming sending status updates
6869 slave.cpp:1253] Forwarding total oversubscribed resources {}
6874 slave.cpp:4141] Master marked the agent as disconnected but the agent considers itself registered! Forcing re-registration.
6874 slave.cpp:904] Re-detecting master
6874 slave.cpp:947] Detecting new master
6874 status_update_manager.cpp:177] Pausing sending status updates
6869 status_update_manager.cpp:177] Pausing sending status updates
6871 slave.cpp:915] New master detected at master@192.168.1.1:5050
6871 slave.cpp:936] No credentials provided. Attempting to register without authentication
6871 slave.cpp:947] Detecting new master
6872 slave.cpp:1217] Re-registered with master master@192.168.1.1:5050
6872 slave.cpp:1253] Forwarding total oversubscribed resources {}
6871 status_update_manager.cpp:184] Resuming sending status updates
6871 slave.cpp:4141] Master marked the agent as disconnected but the agent considers itself registered! Forcing re-registration.

好像陷入了死循环。知道如何开始新的奴隶吗？我尝试删除 work_dir 并重新启动 mesos-slave 进程，但没有成功。

这种情况是由于 work_dir 的意外重命名造成的。重新启动 mesos-slave 后，它无法重新连接，也无法终止运行任务。我试过在奴隶上使用 cleanup：

echo 'cleanup' > /etc/mesos-slave/recover
service mesos-slave restart
# after recovery finishes
rm /etc/mesos-slave/recover
service mesos-slave restart

这部分有帮助，但 Marathon 中仍然有许多僵尸任务，因为 Mesos 管理节点无法检索有关该任务的任何信息。当我查看指标时，我发现一些奴隶被标记为 "inactive".

更新：主日志中的如下：

Cannot kill task service_mesos-kafka_kafka.e0e3e128-ef0e-11e6-af93-fead7f32c37c 
of framework ecd3a4be-d34c-46f3-b358-c4e26ac0d131-0000 (marathon) at
scheduler-e76665b1-de85-48a3-b9fd-5e736b64a9d8@192.168.1.10:52192
because the agent cac09818-0d75-46a9-acb1-4e17fdb9e328-S10 at 
slave(1)@192.168.1.1:5051 (w10.example.net) is disconnected. 
Kill will be retried if the agent re-registers

重启电流后mesos-master:

Cannot kill task service_mesos-kafka_kafka.e0e3e128-ef0e-11e6-af93-fead7f32c37c 
of framework ecd3a4be-d34c-46f3-b358-c4e26ac0d131-0000 (marathon)
at scheduler-9e9753be-99ae-40a6-ab2f-ad7834126c33@192.168.1.10:39972 
because it is unknown; performing reconciliation

Performing explicit task state reconciliation for 1 tasks 
of framework ecd3a4be-d34c-46f3-b358-c4e26ac0d131-0000 (marathon) 
at scheduler-9e9753be-99ae-40a6-ab2f-ad7834126c33@192.168.1.10:39972

Dropping reconciliation of task service_mesos-kafka_kafka.e0e3e128-ef0e-11e6-af93-fead7f32c37c 
for framework ecd3a4be-d34c-46f3-b358-c4e26ac0d131-0000 (marathon) 
at scheduler-9e9753be-99ae-40a6-ab2f-ad7834126c33@192.168.1.10:39972 
because there are transitional agents

Answer 1

split-brain 的情况是因为有多个 work_dir。在大多数情况下，从不正确的 work_dir:

移动数据可能就足够了

 mv /tmp/mesos/slaves/* /var/lib/mesos/slaves/

然后强制re-registration:

rm -rf /var/lib/mesos/meta/slaves/latest
service mesos-slave restart

目前运行任务将无法生存（无法恢复）。来自旧执行者的任务应标记为 TASK_LOST 并安排清理。这将避免僵尸任务的问题，Mesos 无法杀死（因为它们运行在不同的 work_dir 中）。

如果 mesos-slave 仍然注册为非活动状态，请重新启动当前的 Mesos 管理节点。

无法重新注册 mesos 代理

Can not re-register mesos agent

mesos