kube-controller-manager 没有记录细节

Question

我在裸机 kubernetes 集群上为 gitlab 设置持久卷时遇到问题：

Operation for "provision-gitlab/repo-data-gitlab-gitaly-0[3f758288-290c-4d9c-a084-5506f58a22d7]" failed. No retries permitted until 2020-11-28 11:55:56.533202624 +0000 UTC m=+305.008238514 (durationBeforeRetry 4s). Error: "failed to create volume: failed to create volume: see kube-controller-manager.log for details"

问题是：这个文件在任何地方都不存在，我无法获得有关该问题的更多详细信息，即使通过调整配置也是如此：

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-cidr=192.168.0.0/16
    - --cluster-name=kubernetes
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --node-cidr-mask-size=24
    - --port=0
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --use-service-account-credentials=true
    - --log-dir=/var/log/
    - --log-file=kube-controller-manager.log
    - --logtostderr=false
    image: k8s.gcr.io/kube-controller-manager:v1.19.4
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10257
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-controller-manager
    resources:
      requests:
        cpu: 200m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10257
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /var/log/kube-controller-manager.log
      name: logfile
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
      name: flexvolume-dir
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/kubernetes/controller-manager.conf
      name: kubeconfig
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
  - hostPath:
      path: /var/log/kube-controller-manager.log
    name: logfile
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
      type: DirectoryOrCreate
    name: flexvolume-dir
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/kubernetes/controller-manager.conf
      type: FileOrCreate
    name: kubeconfig
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
status: {}

我尝试手动创建它，更改它的权限，但是 pod 仍然没有登录这个文件

Answer 1

Control Plane 组件使用 klog 库进行日志记录，目前，记录相当糟糕。
实际上 --log-dir 和 --log-file 是 mutually exclusive.

## it should be either --log-dir
--log-dir=/var/log/kube
...
volumeMounts:
- mountPath: /var/log/kube
  name: log
...
volumes:
- hostPath:
    path: /var/log/kube
    type: DirectoryOrCreate
  name: log

## or --log-file
--log-file=/var/log/kube-controller-manager.log
...
volumeMounts:
- mountPath: /var/log/kube-controller-manager.log
  name: log
...
volumes:
- hostPath:
    path: /var/log/kube-controller-manager.log
    type: FileOrCreate
  name: log

使用 --log-dir 组件会将每个日志级别写入给定目录中的单独文件。
因此，您将拥有一组名称类似于 kube-controller-manager.INFO.log

的文件

使用 --log-file，您将获得预期的单个文件。
不要忘记在您的卷定义中指定 FileOrCreate，否则将默认创建一个目录。

kube-controller-manager 没有记录细节

kube-controller-manager is not logging details

glusterfs

kubernetes

kubectl

kube-controller-manager