容器优化 OS k8s 节点丢失 google-ip-forwarding-daemon.service IP
Container-Optimized OS k8s nodes losing google-ip-forwarding-daemon.service IPs
版本： GCE 中的 Kubernetes v1.9.2 运行 (kube-up.sh)
容器优化 OS 10032.88.0
症状： 我们的 COS 节点在其 google-ip-forwarding-daemon.service 日志中显示以下内容：
Mar 07 13:43:28 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.154.243.135'].
Mar 07 13:43:45 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.198.28.188'].
Mar 07 13:44:01 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.171.xxx'].
Mar 07 13:44:17 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'130.211.120.8'].
Mar 07 13:44:37 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.198.52.xxx'].
Mar 07 13:44:53 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.17.xx'].
Mar 07 13:45:10 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.155.181.xx'].
Mar 07 13:45:26 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.255.xxx'].
Mar 07 13:45:43 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.166.xxx'].
Mar 07 13:45:59 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] by adding None and removing [u'146.148.61.xx'].
Mar 07 13:46:19 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.197.104.xxx'].
Mar 07 13:46:41 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.198.172.xxx'].
Mar 07 13:46:58 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.198.162.xxx'].
Mar 07 13:47:14 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.155.131.xxx'].
Mar 07 13:47:34 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'130.211.238.xxx'] by adding None and removing [u'104.198.131.xxx'].
Mar 07 13:47:53 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'130.211.238.xxx'] by adding None and removing [u'104.154.72.xxx'].
Mar 07 13:48:10 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'130.211.238.xxx'] to [u'130.211.238.xxx'] by adding None and removing [u'104.154.20.xx'].
Mar 07 13:48:26 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'130.211.238.xxx'] to None by adding None and removing [u'130.211.238.xxx'].
发生这种情况时，运行 PODs 通常会继续 运行 但有时会变得未知并重新安排。任何新安排的 pods 都卡在 ContainerCreating 中并出现以下错误：
Failed create pod sandbox. & Error response from daemon: network none not found
除了重新启动实例之外，我们还没有找到解决此症状的方法。
Kubernetes 1.2 和 1.9.2 之间的版本太多，你为什么要创建一个很旧的版本然后升级它？我遵循了相同的 Kubernetes 文档并得到了一个集群 运行 1.9.3 没有问题。您应该尝试相同的操作并验证您描述的问题是否仍然存在。
此外，图像 cos-stable-63-10032-88-0 is not the latest release. You should also try running with cos-stable-64-10176-62-0 代替。
Mar 12 13:21:04 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from None to [u'104.155.131.xxx'] by adding [u'104.155.131.xxx'] and removing None.
Mar 12 13:21:21 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.155.131.xxx'] to [u'104.154.72.xxx', u'104.155.131.xxx'] by adding [u'104.154.72.xxx'] and removing None.
Mar 12 13:21:37 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.72.xxx', u'104.155.131.xxx'] to [u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx'] by adding [u'104.198.131.xxx'] and removing None.
Mar 12 13:21:53 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx'] by adding [u'104.154.20.xx'] and removing None.
Mar 12 13:22:10 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'130.211.238.xxx'] and removing None.
Mar 12 13:22:26 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'104.154.243.xxx'] and removing None.
Mar 12 13:22:43 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'104.198.52.xxx'] and removing None.
Mar 12 13:22:59 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'104.198.28.xxx'] and removing None.
Mar 12 13:23:15 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'104.197.171.xxx'] and removing None.
Mar 12 13:23:28 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'130.211.120.x'] and removing None.
Mar 12 13:23:42 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'104.197.255.xxx'] and removing None.
Mar 12 13:23:58 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.17.xx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'104.197.17.xx'] and removing None.
Mar 12 13:24:15 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.17.xx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'104.155.181.xx'] and removing None.
Mar 12 13:24:31 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'104.197.166.xxx'] and removing None.
Mar 12 13:24:48 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] by adding [u'146.148.61.xx'] and removing None.
Mar 12 13:25:01 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] by adding [u'104.197.104.xxx'] and removing None.
Mar 12 13:25:17 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.172.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] by adding [u'104.198.172.xxx'] and removing None.
Mar 12 13:25:34 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.172.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] by adding [u'104.198.162.xxx'] and removing None.
在这个问题的评论中，Yu-Ju Hong suggested disabling docker live-restore。这个有见地的建议有助于缓解症状。阅读 live-restore 文档后，任何人都可以确认启用 docker 的 live-restore（在 kubernetes 生态系统中）所获得的好处之一是否正在保留 运行 pods 在实例重启之间？如果是这样，这 protection-between-restarts-behavior 是在 kube 中启用 live-restore 的主要好处吗？我们 运行 一些节点 live-restore 被禁用作为测试，但是我正在尝试发现禁用此功能的全部影响。
版本： GCE 中的 Kubernetes v1.9.2 运行 (kube-up.sh)
容器优化 OS 10032.88.0
症状： 我们的 COS 节点在其 google-ip-forwarding-daemon.service 日志中显示以下内容：
Mar 07 13:43:28 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.154.243.135'].
Mar 07 13:43:45 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.188', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.198.28.188'].
Mar 07 13:44:01 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.171.xxx'].
Mar 07 13:44:17 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.8', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'130.211.120.8'].
Mar 07 13:44:37 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.198.52.xxx'].
Mar 07 13:44:53 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.17.xx'].
Mar 07 13:45:10 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.155.181.xx'].
Mar 07 13:45:26 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.255.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.255.xxx'].
Mar 07 13:45:43 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] by adding None and removing [u'104.197.166.xxx'].
Mar 07 13:45:59 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] by adding None and removing [u'146.148.61.xx'].
Mar 07 13:46:19 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.197.104.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.197.104.xxx'].
Mar 07 13:46:41 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.198.172.xxx'].
Mar 07 13:46:58 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.198.162.xxx'].
Mar 07 13:47:14 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding None and removing [u'104.155.131.xxx'].
Mar 07 13:47:34 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xx', u'130.211.238.xxx'] by adding None and removing [u'104.198.131.xxx'].
Mar 07 13:47:53 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'130.211.238.xxx'] by adding None and removing [u'104.154.72.xxx'].
Mar 07 13:48:10 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'130.211.238.xxx'] to [u'130.211.238.xxx'] by adding None and removing [u'104.154.20.xx'].
Mar 07 13:48:26 kube-minion-0fqw google-ip-forwarding[935]: INFO Changing eth0 IPs from [u'130.211.238.xxx'] to None by adding None and removing [u'130.211.238.xxx'].
发生这种情况时，运行 PODs 通常会继续 运行 但有时会变得未知并重新安排。任何新安排的 pods 都卡在 ContainerCreating 中并出现以下错误：
Failed create pod sandbox. & Error response from daemon: network none not found
除了重新启动实例之外，我们还没有找到解决此症状的方法。
Kubernetes 1.2 和 1.9.2 之间的版本太多，你为什么要创建一个很旧的版本然后升级它？我遵循了相同的 Kubernetes 文档并得到了一个集群 运行 1.9.3 没有问题。您应该尝试相同的操作并验证您描述的问题是否仍然存在。
此外，图像 cos-stable-63-10032-88-0 is not the latest release. You should also try running with cos-stable-64-10176-62-0 代替。
Mar 12 13:21:04 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from None to [u'104.155.131.xxx'] by adding [u'104.155.131.xxx'] and removing None.
Mar 12 13:21:21 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.155.131.xxx'] to [u'104.154.72.xxx', u'104.155.131.xxx'] by adding [u'104.154.72.xxx'] and removing None.
Mar 12 13:21:37 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.72.xxx', u'104.155.131.xxx'] to [u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx'] by adding [u'104.198.131.xxx'] and removing None.
Mar 12 13:21:53 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx'] by adding [u'104.154.20.xx'] and removing None.
Mar 12 13:22:10 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'130.211.238.xxx'] and removing None.
Mar 12 13:22:26 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'104.154.243.xxx'] and removing None.
Mar 12 13:22:43 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'104.198.52.xxx'] and removing None.
Mar 12 13:22:59 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'104.198.28.xxx'] and removing None.
Mar 12 13:23:15 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] by adding [u'104.197.171.xxx'] and removing None.
Mar 12 13:23:28 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'130.211.120.x'] and removing None.
Mar 12 13:23:42 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'104.197.255.xxx'] and removing None.
Mar 12 13:23:58 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.17.xx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'104.197.17.xx'] and removing None.
Mar 12 13:24:15 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.197.17.xx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'104.155.181.xx'] and removing None.
Mar 12 13:24:31 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] by adding [u'104.197.166.xxx'] and removing None.
Mar 12 13:24:48 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] by adding [u'146.148.61.xx'] and removing None.
Mar 12 13:25:01 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] by adding [u'104.197.104.xxx'] and removing None.
Mar 12 13:25:17 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.172.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] by adding [u'104.198.172.xxx'] and removing None.
Mar 12 13:25:34 kube-minion-0fqw google-ip-forwarding[955]: INFO Changing eth0 IPs from [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.172.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] to [u'104.154.20.xx', u'104.154.72.xxx', u'104.154.243.xxx', u'104.155.131.xxx', u'104.155.181.xx', u'104.197.17.xx', u'104.197.104.xxx', u'104.197.166.xxx', u'104.197.171.xxx', u'104.197.255.xxx', u'104.198.28.xxx', u'104.198.52.xxx', u'104.198.131.xxx', u'104.198.162.xxx', u'104.198.172.xxx', u'130.211.120.x', u'130.211.238.xxx', u'146.148.61.xx'] by adding [u'104.198.162.xxx'] and removing None.
在这个问题的评论中，Yu-Ju Hong suggested disabling docker live-restore。这个有见地的建议有助于缓解症状。阅读 live-restore 文档后，任何人都可以确认启用 docker 的 live-restore（在 kubernetes 生态系统中）所获得的好处之一是否正在保留 运行 pods 在实例重启之间？如果是这样，这 protection-between-restarts-behavior 是在 kube 中启用 live-restore 的主要好处吗？我们 运行 一些节点 live-restore 被禁用作为测试，但是我正在尝试发现禁用此功能的全部影响。