基于 Calico 的 pod 不会出现
Calico based pod is not coming up
我正在使用 calico 设置一个 pod，但它一直因某些授权错误而失败。默认情况下，以下是我系统的节点 cidr：
[root@k8master-1 ~]# kubeadm config view | grep Subnet
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
我已经使用以下步骤设置了 ippools：
https://docs.projectcalico.org/getting-started/kubernetes/flannel/flannel
IP 池创建
- apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: rack-ip-pool
spec:
blockSize: 26
cidr: 10.244.1.0/24
ipipMode: Never
natOutgoing: true
nodeSelector: all()
vxlanMode: Never
IP 池列表
[root@k8master-1 ~]# calicoctl get ippool -o wide
NAME CIDR NAT IPIPMODE VXLANMODE DISABLED SELECTOR
rack-ip-pool 10.244.1.0/24 true Never Never false all()
Pod Yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: testcalico
labels:
cracklerack: "1"
spec:
serviceName: testcalico-svc
selector:
matchLabels:
cracklerack: "1"
template:
metadata:
labels:
cracklerack: "1"
annotations:
cni.projectcalico.org/ipv4pools: "[\"rack-ip-pool\"]"
spec:
runtimeClassName: kata-containers
containers:
- name: testcalico
image: cracklelinux:7
ports:
- containerPort: 80
command: [/usr/sbin/init]
securityContext:
privileged: true
---
apiVersion: v1
kind: Service
metadata:
name: testcalico-svc
spec:
clusterIP: None
selector:
cracklerack: "1"
当我创建 pod 时，它抛出以下错误：
错误
Warning FailedCreatePodSandBox 112s kubelet, k8worker-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_xxxxx-0_default_45357eab-bf40-4fe7-a470-da42c9668116_0(579e2c258154fcdc2e85df4a1e35264ea9550b0dd1c4384331abc471f552456d): connection is unauthorized: ipamconfigs.crd.projectcalico.org "default" is forbidden: User "system:serviceaccount:kube-system:canal" cannot get resource "ipamconfigs" in API group "crd.projectcalico.org" at the cluster scope
看起来你有一个 RBAC 问题，你的 pod 无法读取 Kubernetes IPAMConfig CRD。
我查看了 https://docs.projectcalico.org/manifests/canal.yaml 的清单，发现它在几个 RBAC ClusterRoles 中丢失了 ipamconfigs。所以你可以继续尝试添加它们。
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-kube-controllers
rules:
# Nodes are watched to monitor for deletions.
- apiGroups: [""]
resources:
- nodes
verbs:
- watch
- list
- get
# Pods are queried to check for existence.
- apiGroups: [""]
resources:
- pods
verbs:
- get
# IPAM resources are manipulated when nodes are deleted.
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
verbs:
- list
- apiGroups: ["crd.projectcalico.org"]
resources:
- blockaffinities
- ipamblocks
- ipamhandles
- ipamconfigs add here
...
然后是另一个ClusterRole：
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-node
rules:
...
# Calico monitors various CRDs for config.
- apiGroups: ["crd.projectcalico.org"]
resources:
- globalfelixconfigs
- felixconfigurations
- bgppeers
- globalbgpconfigs
- bgpconfigurations
- ippools
- ipamblocks
- ipamconfigs add here
- globalnetworkpolicies
- globalnetworksets
- networkpolicies
- networksets
- clusterinformations
- hostendpoints
- blockaffinities
verbs:
- get
- list
- watch
# Calico must create and update some CRDs on startup.
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
- ipamconfigs just in case
- felixconfigurations
- clusterinformations
verbs:
- create
- update
...
然后运行:
kubectl apply -f canal.yaml
应用此后，您可能需要重新启动集群（至少在我的 minikube 上需要）。
我使用了以下 conf 文件并且有效：
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-node
rules:
# The CNI plugin needs to get pods, nodes, and namespaces.
- apiGroups: [""]
resources:
- pods
- nodes
- namespaces
verbs:
- get
- apiGroups: [""]
resources:
- endpoints
- services
verbs:
# Used to discover service IPs for advertisement.
- watch
- list
# Used to discover Typhas.
- get
# Pod CIDR auto-detection on kubeadm needs access to config maps.
- apiGroups: [""]
resources:
- configmaps
verbs:
- get
- apiGroups: [""]
resources:
- nodes/status
verbs:
# Needed for clearing NodeNetworkUnavailable flag.
- patch
# Calico stores some configuration information in node annotations.
- update
# Watch for changes to Kubernetes NetworkPolicies.
- apiGroups: ["networking.k8s.io"]
resources:
- networkpolicies
verbs:
- watch
- list
# Used by Calico for policy information.
- apiGroups: [""]
resources:
- pods
- namespaces
- serviceaccounts
verbs:
- list
- watch
# The CNI plugin patches pods/status.
- apiGroups: [""]
resources:
- pods/status
verbs:
- patch
# Calico monitors various CRDs for config.
- apiGroups: ["crd.projectcalico.org"]
resources:
- globalfelixconfigs
- felixconfigurations
- bgppeers
- globalbgpconfigs
- bgpconfigurations
- ippools
- ipamblocks
- ipamconfigs
- globalnetworkpolicies
- globalnetworksets
- networkpolicies
- networksets
- clusterinformations
- hostendpoints
- blockaffinities
verbs:
- get
- list
- watch
# Calico must create and update some CRDs on startup.
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
- ipamblocks
- ipamconfigs
- blockaffinities
- felixconfigurations
- clusterinformations
verbs:
- create
- update
# Calico stores some configuration information on the node.
- apiGroups: [""]
resources:
- nodes
verbs:
- get
- list
- watch
# These permissions are only required for upgrade from v2.6, and can
# be removed after upgrade or on fresh installations.
- apiGroups: ["crd.projectcalico.org"]
resources:
- bgpconfigurations
- bgppeers
verbs:
- create
- update
同一文件中的另一个块：
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-kube-controllers
rules:
# Nodes are watched to monitor for deletions.
- apiGroups: [""]
resources:
- nodes
verbs:
- watch
- list
- get
# Pods are queried to check for existence.
- apiGroups: [""]
resources:
- pods
verbs:
- get
# IPAM resources are manipulated when nodes are deleted.
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
verbs:
- list
- apiGroups: ["crd.projectcalico.org"]
resources:
- blockaffinities
- ipamblocks
- ipamhandles
- ipamconfigs
verbs:
- get
- list
- create
- update
- delete
# kube-controllers manages hostendpoints.
- apiGroups: ["crd.projectcalico.org"]
resources:
- hostendpoints
verbs:
- get
- list
- create
- update
- delete
# Needs access to update clusterinformations.
- apiGroups: ["crd.projectcalico.org"]
resources:
- clusterinformations
verbs:
- get
- create
- update
# KubeControllersConfiguration is where it gets its config
- apiGroups: ["crd.projectcalico.org"]
resources:
- kubecontrollersconfigurations
verbs:
# read its own config
- get
# create a default if none exists
- create
# update status
- update
# watch for changes
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-kube-controllers
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: calico-kube-controllers
subjects:
- kind: ServiceAccount
name: calico-kube-controllers
namespace: kube-system
---
我正在使用 calico 设置一个 pod，但它一直因某些授权错误而失败。默认情况下，以下是我系统的节点 cidr：
[root@k8master-1 ~]# kubeadm config view | grep Subnet
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
我已经使用以下步骤设置了 ippools：
https://docs.projectcalico.org/getting-started/kubernetes/flannel/flannel
IP 池创建
- apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
name: rack-ip-pool
spec:
blockSize: 26
cidr: 10.244.1.0/24
ipipMode: Never
natOutgoing: true
nodeSelector: all()
vxlanMode: Never
IP 池列表
[root@k8master-1 ~]# calicoctl get ippool -o wide
NAME CIDR NAT IPIPMODE VXLANMODE DISABLED SELECTOR
rack-ip-pool 10.244.1.0/24 true Never Never false all()
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: testcalico
labels:
cracklerack: "1"
spec:
serviceName: testcalico-svc
selector:
matchLabels:
cracklerack: "1"
template:
metadata:
labels:
cracklerack: "1"
annotations:
cni.projectcalico.org/ipv4pools: "[\"rack-ip-pool\"]"
spec:
runtimeClassName: kata-containers
containers:
- name: testcalico
image: cracklelinux:7
ports:
- containerPort: 80
command: [/usr/sbin/init]
securityContext:
privileged: true
---
apiVersion: v1
kind: Service
metadata:
name: testcalico-svc
spec:
clusterIP: None
selector:
cracklerack: "1"
当我创建 pod 时，它抛出以下错误：
错误
Warning FailedCreatePodSandBox 112s kubelet, k8worker-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_xxxxx-0_default_45357eab-bf40-4fe7-a470-da42c9668116_0(579e2c258154fcdc2e85df4a1e35264ea9550b0dd1c4384331abc471f552456d): connection is unauthorized: ipamconfigs.crd.projectcalico.org "default" is forbidden: User "system:serviceaccount:kube-system:canal" cannot get resource "ipamconfigs" in API group "crd.projectcalico.org" at the cluster scope
看起来你有一个 RBAC 问题，你的 pod 无法读取 Kubernetes IPAMConfig CRD。
我查看了 https://docs.projectcalico.org/manifests/canal.yaml 的清单，发现它在几个 RBAC ClusterRoles 中丢失了 ipamconfigs。所以你可以继续尝试添加它们。
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-kube-controllers
rules:
# Nodes are watched to monitor for deletions.
- apiGroups: [""]
resources:
- nodes
verbs:
- watch
- list
- get
# Pods are queried to check for existence.
- apiGroups: [""]
resources:
- pods
verbs:
- get
# IPAM resources are manipulated when nodes are deleted.
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
verbs:
- list
- apiGroups: ["crd.projectcalico.org"]
resources:
- blockaffinities
- ipamblocks
- ipamhandles
- ipamconfigs add here
...
然后是另一个ClusterRole：
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-node
rules:
...
# Calico monitors various CRDs for config.
- apiGroups: ["crd.projectcalico.org"]
resources:
- globalfelixconfigs
- felixconfigurations
- bgppeers
- globalbgpconfigs
- bgpconfigurations
- ippools
- ipamblocks
- ipamconfigs add here
- globalnetworkpolicies
- globalnetworksets
- networkpolicies
- networksets
- clusterinformations
- hostendpoints
- blockaffinities
verbs:
- get
- list
- watch
# Calico must create and update some CRDs on startup.
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
- ipamconfigs just in case
- felixconfigurations
- clusterinformations
verbs:
- create
- update
...
然后运行:
kubectl apply -f canal.yaml
应用此后，您可能需要重新启动集群（至少在我的 minikube 上需要）。
我使用了以下 conf 文件并且有效：
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-node
rules:
# The CNI plugin needs to get pods, nodes, and namespaces.
- apiGroups: [""]
resources:
- pods
- nodes
- namespaces
verbs:
- get
- apiGroups: [""]
resources:
- endpoints
- services
verbs:
# Used to discover service IPs for advertisement.
- watch
- list
# Used to discover Typhas.
- get
# Pod CIDR auto-detection on kubeadm needs access to config maps.
- apiGroups: [""]
resources:
- configmaps
verbs:
- get
- apiGroups: [""]
resources:
- nodes/status
verbs:
# Needed for clearing NodeNetworkUnavailable flag.
- patch
# Calico stores some configuration information in node annotations.
- update
# Watch for changes to Kubernetes NetworkPolicies.
- apiGroups: ["networking.k8s.io"]
resources:
- networkpolicies
verbs:
- watch
- list
# Used by Calico for policy information.
- apiGroups: [""]
resources:
- pods
- namespaces
- serviceaccounts
verbs:
- list
- watch
# The CNI plugin patches pods/status.
- apiGroups: [""]
resources:
- pods/status
verbs:
- patch
# Calico monitors various CRDs for config.
- apiGroups: ["crd.projectcalico.org"]
resources:
- globalfelixconfigs
- felixconfigurations
- bgppeers
- globalbgpconfigs
- bgpconfigurations
- ippools
- ipamblocks
- ipamconfigs
- globalnetworkpolicies
- globalnetworksets
- networkpolicies
- networksets
- clusterinformations
- hostendpoints
- blockaffinities
verbs:
- get
- list
- watch
# Calico must create and update some CRDs on startup.
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
- ipamblocks
- ipamconfigs
- blockaffinities
- felixconfigurations
- clusterinformations
verbs:
- create
- update
# Calico stores some configuration information on the node.
- apiGroups: [""]
resources:
- nodes
verbs:
- get
- list
- watch
# These permissions are only required for upgrade from v2.6, and can
# be removed after upgrade or on fresh installations.
- apiGroups: ["crd.projectcalico.org"]
resources:
- bgpconfigurations
- bgppeers
verbs:
- create
- update
同一文件中的另一个块：
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-kube-controllers
rules:
# Nodes are watched to monitor for deletions.
- apiGroups: [""]
resources:
- nodes
verbs:
- watch
- list
- get
# Pods are queried to check for existence.
- apiGroups: [""]
resources:
- pods
verbs:
- get
# IPAM resources are manipulated when nodes are deleted.
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
verbs:
- list
- apiGroups: ["crd.projectcalico.org"]
resources:
- blockaffinities
- ipamblocks
- ipamhandles
- ipamconfigs
verbs:
- get
- list
- create
- update
- delete
# kube-controllers manages hostendpoints.
- apiGroups: ["crd.projectcalico.org"]
resources:
- hostendpoints
verbs:
- get
- list
- create
- update
- delete
# Needs access to update clusterinformations.
- apiGroups: ["crd.projectcalico.org"]
resources:
- clusterinformations
verbs:
- get
- create
- update
# KubeControllersConfiguration is where it gets its config
- apiGroups: ["crd.projectcalico.org"]
resources:
- kubecontrollersconfigurations
verbs:
# read its own config
- get
# create a default if none exists
- create
# update status
- update
# watch for changes
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-kube-controllers
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: calico-kube-controllers
subjects:
- kind: ServiceAccount
name: calico-kube-controllers
namespace: kube-system
---