如何限制 kubelet 日志大小或在 kubernetes 中轮换

Question

今天，当我使用 duc 命令检查 kubernetes(v1.21) 集群主机磁盘使用情况时，如下所示：

[root@k8smasterone log]# duc ls -Fg /var/log/
  5.4G messages-20220515             [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++]
  3.9G messages                      [++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++                                                   ]
571.1M messages-20220508             [++++++++++++++++++

kubelet输出的文件太大，如何限制kubelet日志大小？我已经阅读了这个问题来讨论 kubelet 应该处理的日志：https://github.com/containerd/containerd/issues/4830。但我没有找到任何可能的解决方案来解决 kubelet 日志轮换问题。 PS：我现在使用的是 containerd 而不是 docker。这是我的日志配置：

[root@k8smasterone log]# cat /etc/systemd/journald.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
SystemMaxUse=50M
#SystemKeepFree=
SystemMaxFileSize=20M
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
MaxRetentionSec=1week
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K

Answer 1

您可以在 kubelet config 中调整以下两个参数：

containerLogMaxSize string A quantity defines the maximum size of the container log file before it is rotated. For example "5Mi" or "256Ki". Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may trigger log rotation. Default: "10Mi"

containerLogMaxFiles int32 Maximum number of container log files that can be present for a container. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that lowering it may cause log files to be deleted. Default: 5

示例：

sudo vi /etc/kubernetes/kubelet-config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
nodeStatusUpdateFrequency: "10s"
failSwapOn: True
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: True
  x509:
    clientCAFile: /etc/kubernetes/ssl/ca.crt
authorization:
  mode: Webhook
staticPodPath: /etc/kubernetes/manifests
cgroupDriver: systemd
containerLogMaxFiles: 5    # Maximum number of container logs to retain. 
containerLogMaxSize: 1Mi   # Change the size of /var/log/containers/<pod-name>/log files size to 1M max.
maxPods: 110
address: 192.168.22.5
readOnlyPort: 0
healthzPort: 10248
healthzBindAddress: 127.0.0.1
kubeletCgroups: /systemd/system.slice
clusterDomain: cluster.local
protectKernelDefaults: true
rotateCertificates: true
clusterDNS:
- 8.8.8.8
kubeReserved:
  cpu: 200m
  memory: 512Mi
resolvConf: "/run/systemd/resolve/resolv.conf"
eventRecordQPS: 5
shutdownGracePeriod: 60s
shutdownGracePeriodCriticalPods: 20s

重新加载 kubelet 配置：

sudo systemctl daemon-reload && sudo systemctl restart kubelet

如何限制 kubelet 日志大小或在 kubernetes 中轮换

how to limit the kubelet log size or rotate in kubernetes

kubernetes