Kubernetes 1.7 calico 日志消息 calico-node -felix-live -bird-live] 和超时 1 (s)

Question

我是运行 Kubernetes 1.17 集群，在 containerd 上使用 Calico CNI 运行。我的服务器是运行 CentOS 8，我的所有节点都收到此日志消息。如何停止这些消息？

Mar  8 21:15:48 node03 systemd[1]: Removed slice libcontainer_14289_systemd_test_default.slice.
Mar  8 21:15:48 node03 containerd[1400]: time="2020-03-08T21:15:48.641993033-04:00" level=info msg="ExecSync for "a5e553ee2fd117b0724f56fd9606031f187f4ca0aa468c31f90690f305f49a89" returns with exit code 0"
Mar  8 21:15:50 node03 containerd[1400]: time="2020-03-08T21:15:50.606510411-04:00" level=info msg="ExecSync for "a5e553ee2fd117b0724f56fd9606031f187f4ca0aa468c31f90690f305f49a89" with command [/bin/calico-node -felix-live -bird-live] and timeout 1 (s)"
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14399-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14399-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: Created slice libcontainer_14399_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: Removed slice libcontainer_14399_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14405-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14405-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: Created slice libcontainer_14405_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: Removed slice libcontainer_14405_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14411-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14411-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: Created slice libcontainer_14411_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: Removed slice libcontainer_14411_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14417-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14417-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: Created slice libcontainer_14417_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: Removed slice libcontainer_14417_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14437-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: libcontainer-14437-systemd-test-default-dependencies.scope: Scope has no PIDs. Refusing.
Mar  8 21:15:50 node03 systemd[1]: Created slice libcontainer_14437_systemd_test_default.slice.
Mar  8 21:15:50 node03 systemd[1]: Removed slice libcontainer_14437_systemd_test_default.slice.
Mar  8 21:15:50 node03 containerd[1400]: time="2020-03-08T21:15:50.775530563-04:00" level=info msg="Exec process "1f990c60ac0163d4a69c3ac6d2c1c2c53458d872e3d7e950cb89bcf2365c4601" exits with exit code 0 and error <nil>"
Mar  8 21:15:50 node03 containerd[1400]: time="2020-03-08T21:15:50.775615176-04:00" level=info msg="Finish piping "stdout" of container exec "1f990c60ac0163d4a69c3ac6d2c1c2c53458d872e3d7e950cb89bcf2365c4601""

Answer 1

已经在 Github 上打开了问题，可以找到更多详细信息 here。

此问题与 cgroup-driver 有关。在最新的 Kubernetes 版本中设置为 systemd 而不是 cgroupfs.

一种解决方法是配置 rsyslog。 this site 上详细描述了步骤。短版以下：

$ cd /etc/rsyslog.d

创建文件 ignore-systemd-session-slice.conf 内容为：

if ($programname == "systemd") and ($msg contains "_systemd_test_default.slice" or $msg contains "systemd-test-default-dependencies.scope") then {
  stop
}

之后必须重新启动服务 rsyslog。

$ service rsyslog restart

另一种方法是将cgroupfs用作cgroup-driver。这也在 this Github thread.

上进行了讨论

添加这些以忽略日志

cat /etc/rsyslog.d/ignore-systemd-session-slice.conf 文件

if ($programname == "systemd") and ($msg contains "_systemd_test_default.slice" or $msg contains "systemd-test-default-dependencies.scope" or $msg contains " Failed to get system container stats for") then {
  stop
}
if ($programname == "kubelet") and ($msg contains " Failed to get system container stats for" or $msg contains "test_default.slice") then {
  stop
}

if ($programname == "systemd") and ($msg contains "Starting Session" or $msg contains "Started Session" or $msg contains "Created slice" or $msg contains "Starting user-" or $msg contains "Starting User Slice of" or $msg contains "Removed session" or $msg contains "Removed slice User Slice of" or $msg contains "Stopping User Slice of") then {
        stop
}
if ($programname == "containerd") and ($msg regex "Exec process.*with exit code 0" or $msg regex "ExecSync for.*with exit code 0" or $msg contains "Finish piping") then {
  stop
}

Kubernetes 1.7 calico 日志消息 calico-node -felix-live -bird-live] 和超时 1 (s)

Kubernetes 1.7 calico log message calico-node -felix-live -bird-live] and timeout 1 (s)

kubernetes

project-calico