kube-dns 保留在 "ContainerCreating"

Question

我手动安装了 k8s-1.6.6 并在裸机上部署了 calico-2.3（使用 etcd-3.0.17 和 kube-apiserver）和 kube-dns（ubuntu 16.04）。

没有RBAC就没有任何问题。

但是，在通过向 kube-apiserver 添加“--authorization-mode=RBAC”来应用 RBAC 之后。我无法应用状态保持在 "ContainerCreating".

的 kube-dns

我查了"kubectl describe pod kube-dns.."

Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----            -------------   --------    ------      -------
  10m       10m     1   default-scheduler           Normal      Scheduled   Successfully assigned kube-dns-1759312207-t35t3 to work01
  9m        9m      1   kubelet, work01             Warning     FailedSync  Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: 8c2585b1b3170f220247a6abffb1a431af58060f2bcc715fe29e7c2144d19074
  8m        8m      1   kubelet, work01             Warning     FailedSync  Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: c6962db6c5a17533fbee563162c499631a647604f9bffe6bc71026b09a2a0d4f
  7m        7m      1   kubelet, work01             Warning     FailedSync  Error syncing pod, skipping: failed to "KillPodSandbox" for "f693931a-7335-11e7-aaa2-525400aa8825" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"kube-dns-1759312207-t35t3_kube-system\" network: CNI failed to retrieve network namespace path: Error: No such container: 9adc41d07a80db44099460c6cc56612c6fbcd53176abcc3e7bbf843fca8b7532"

  5m    5m  1   kubelet, work01     Warning FailedSync  Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: 4c2d450186cbec73ea28d2eb4c51497f6d8c175b92d3e61b13deeba1087e9a40
  4m    4m  1   kubelet, work01     Warning FailedSync  Error syncing pod, skipping: failed to "KillPodSandbox" for "f693931a-7335-11e7-aaa2-525400aa8825" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"kube-dns-1759312207-t35t3_kube-system\" network: CNI failed to retrieve network namespace path: Error: No such container: 12df544137939d2b8af8d70964e46b49f5ddec1228da711c084ff493443df465"

  3m    3m  1   kubelet, work01     Warning FailedSync  Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: c51c9d50dcd62160ffe68d891967d118a0f594885e99df3286d0c4f8f4986970
  2m    2m  1   kubelet, work01     Warning FailedSync  Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: 94533f19952c7d5f32e919c03d9ec5147ef63d4c1f35dd4fcfea34306b9fbb71
  1m    1m  1   kubelet, work01     Warning FailedSync  Error syncing pod, skipping: rpc error: code = 2 desc = Error: No such container: 166a89916c1e6d63e80b237e5061fd657f091f3c6d430b7cee34586ba8777b37
  16s   12s 2   kubelet, work01     Warning FailedSync  (events with common reason combined)
  10m   2s  207 kubelet, work01     Warning FailedSync  Error syncing pod, skipping: failed to "CreatePodSandbox" for "kube-dns-1759312207-t35t3_kube-system(f693931a-7335-11e7-aaa2-525400aa8825)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-dns-1759312207-t35t3_kube-system(f693931a-7335-11e7-aaa2-525400aa8825)\" failed: rpc error: code = 2 desc = NetworkPlugin cni failed to set up pod \"kube-dns-1759312207-t35t3_kube-system\" network: the server does not allow access to the requested resource (get pods kube-dns-1759312207-t35t3)"

  10m   1s  210 kubelet, work01     Normal  SandboxChanged  Pod sandbox changed, it will be killed and re-created.

我的 kubelet

[Unit] 
Description=Kubernetes Kubelet 
Documentation=https://github.com/kubernetes/kubernetes 

[Service] 
ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=/bin/mkdir -p /var/log/containers
ExecStartPre=/bin/mkdir -p /etc/cni/net.d
ExecStartPre=/bin/mkdir -p /opt/cni/bin
ExecStart=/usr/local/bin/kubelet \
          --api-servers=http://127.0.0.1:8080 \
          --allow-privileged=true \
          --pod-manifest-path=/etc/kubernetes/manifests \
          --kubeconfig=/var/lib/kubelet/kubeconfig \
          --cluster-dns=10.3.0.10 \
          --cluster-domain=cluster.local \
          --register-node=true \
          --network-plugin=cni \
          --cni-conf-dir=/etc/cni/net.d \
          --cni-bin-dir=/opt/cni/bin \
          --container-runtime=docker

我的 kube-apiserver

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: kube-apiserver
    image: kube-apiserver:v1.6.6
    command:
    - kube-apiserver
    - --bind-address=0.0.0.0
    - --etcd-servers=http://127.0.0.1:2379
    - --allow-privileged=true
    - --service-cluster-ip-range=10.3.0.0/16
    - --secure-port=6443
    - --advertise-address=172.30.1.10
    - --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota
    - --tls-cert-file=/srv/kubernetes/apiserver.pem
    - --tls-private-key-file=/srv/kubernetes/apiserver-key.pem
    - --client-ca-file=/srv/kubernetes/ca.pem
    - --service-account-key-file=/srv/kubernetes/apiserver-key.pem
    - --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP
    - --anonymous-auth=false
    - --authorization-mode=RBAC
    - --token-auth-file=/srv/kubernetes/known_tokens.csv
    - --basic-auth-file=/srv/kubernetes/basic_auth.csv
    - --storage-backend=etcd3
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        port: 8080
        path: /healthz
        scheme: HTTP
      initialDelaySeconds: 15
      timeoutSeconds: 15
    ports:
    - name: https
      hostPort: 6443
      containerPort: 6443
    - name: local
      hostPort: 8080
      containerPort: 8080
    volumeMounts:
    - name: srvkube
      mountPath: "/srv/kubernetes"
      readOnly: true
    - name: etcssl
      mountPath: "/etc/ssl"
      readOnly: true
  volumes:
  - name: srvkube
    hostPath:
      path: "/srv/kubernetes"
  - name: etcssl
    hostPath:
      path: "/etc/ssl"

Answer 1

我找到了原因。此问题与 kube-dns 无关。在部署 calico

之前，我只是错过了应用 ClusterRole/ClusterRoleBinding

kube-dns 保留在 "ContainerCreating"

kube-dns remains in "ContainerCreating"

rbac

calico

kubernetes

kube-dns