服务 "kube-dns" 无效：spec.clusterIP：无效值：“10.10.0.10”：字段不可变
Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.10.0.10": field is immutable
我通过 kubeadm 设置我的集群。在最后一步，我执行 kubeadm init --config kubeadm.conf --v=5。我收到有关 clusterIp 值的错误。这是输出的一部分：
I0220 00:16:27.625920 31630 clusterinfo.go:79] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I0220 00:16:27.947941 31630 kubeletfinalize.go:88] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0220 00:16:27.949398 31630 kubeletfinalize.go:132] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
[addons]: Migrating CoreDNS Corefile
I0220 00:16:28.447420 31630 dns.go:381] the CoreDNS configuration has been migrated and applied: .:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
.
I0220 00:16:28.447465 31630 dns.go:382] the old migration has been saved in the CoreDNS ConfigMap under the name [Corefile-backup]
I0220 00:16:28.447486 31630 dns.go:383] The changes in the new CoreDNS Configuration are as follows:
Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.10.0.10": field is immutable
unable to create/update the DNS service
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.createDNSService
/workspace/anago-v1.17.0-rc.2.10+70132b0f130acc/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns/dns.go:323
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.createCoreDNSAddon
/workspace/anago-v1.17.0-rc.2.10+70132b0f130acc/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns/dns.go:305
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.coreDNSAddon
我的配置文件是这样的：
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 172.16.5.151
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: master02
# taints:
# - effect: NoSchedule
# key: node-role.kubernetes.io/master
---
apiServer:
certSANs:
- "172.16.5.150"
- "172.16.5.151"
- "172.16.5.152"
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
external:
endpoints:
- "https://172.16.5.150:2379"
- "https://172.16.5.151:2379"
- "https://172.16.5.152:2379"
caFile: /etc/k8s/pki/ca.pem
certFile: /etc/k8s/pki/etcd.pem
keyFile: /etc/k8s/pki/etcd.key
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.17.0
networking:
dnsDomain: cluster.local
serviceSubnet: 10.10.0.0/16
podSubnet: 192.168.0.0/16
scheduler: {}
我检查了 kubeadm 生成的 kube-apiserver.yaml。
--service-cluster-ip-range=10.10.0.0/16 设置包含 10.10.0.10
你可以在下面看到：
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=172.16.5.151
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
- --etcd-cafile=/etc/k8s/pki/ca.pem
- --etcd-certfile=/etc/k8s/pki/etcd.pem
- --etcd-keyfile=/etc/k8s/pki/etcd.key
- --etcd-servers=https://172.16.5.150:2379,https://172.16.5.151:2379,https://172.16.5.152:2379
- --insecure-port=0
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
- --requestheader-allowed-names=front-proxy-client
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --secure-port=6443
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-cluster-ip-range=10.10.0.0/16
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.17.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 172.16.5.151
path: /healthz
port: 6443
scheme: HTTPS
initialDelaySeconds: 15
timeoutSeconds: 15
name: kube-apiserver
resources:
requests:
cpu: 250m
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/pki
name: etc-pki
readOnly: true
- mountPath: /etc/k8s/pki
name: etcd-certs-0
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
hostNetwork: true
priorityClassName: system-cluster-critical
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/pki
type: DirectoryOrCreate
name: etc-pki
- hostPath:
path: /etc/k8s/pki
type: DirectoryOrCreate
name: etcd-certs-0
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
status: {}
如你所见。所有的 service-ip-range 都设置为 10.10.0.0/16。奇怪的是，当我执行 "kubectl get svc" 时，我得到的 kubernetes clusterip 是 10.96.0.1
[root@master02 manifests]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d3h
这意味着默认的服务IP范围是：10.96.0.0/16。我修改的内容不起作用。有谁知道如何自定义 service-ip-range 范围。以及如何解决我的问题？
因为这个节点我之前作为一个节点加入了集群因为这个节点我作为一个节点加入了集群before.Later我用"kubeadm reset "重置了这个command.After重置了，我加入了它作为集群的主角色。所以我在上面的问题中得到了错误。
报错是因为我重置前的clusterip范围已经记录在etcd集群中了。而且"kubeadm reset"命令没有清理etcd.So中的数据 新定义的clusterip与original.So冲突 解决方法是清理etcd中的数据，重新设置。 （由于我搭建的集群是测试集群，直接清理了etcd，生产环境请慎重）
将此答案发布为社区 Wiki 以扩展和解释根本原因。
当 kubeadm 启动时，我们没有指定任何标志，$ kubeadm init 它将使用默认值创建 kubeadm 集群。您可以签入 Kubernetes docs 标志，这些标志可以在初始化期间指定并且是默认值。
--service-cidr string Default: "10.96.0.0/12"
Use alternative range of IP address for service VIPs.
这就是默认 kubernetes 服务使用 10.96.0.1 作为 ClusterIP 的原因。
这里OP也想用自己的配置
--config string Path to a kubeadm configuration file.
可以找到整个初始化工作流程here。
作为 Kubernetes 文档示例 Kubeadm reset
Performs a best effort revert of changes made by kubeadm init or kubeadm join.
有时取决于我们的配置，一些配置保留在集群上。
问题，提到了遇到的 OP here - External etcd clean up
kubeadm reset will not delete any etcd data if external etcd is used. This means that if you run kubeadm init again using the same etcd endpoints, you will see state from previous clusters.
关于不可变字段：Service “kube-dns” is invalid: spec.clusterIP: Invalid value: “10.10.0.10”: field is immutable。
在 Kubernetes 中，一些字段是安全的，以防止可能中断集群工作的更改。
如果任何字段是 immutable 但我们必须更改它，则必须删除此对象并重新添加。
我通过 kubeadm 设置我的集群。在最后一步，我执行 kubeadm init --config kubeadm.conf --v=5。我收到有关 clusterIp 值的错误。这是输出的一部分：
I0220 00:16:27.625920 31630 clusterinfo.go:79] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I0220 00:16:27.947941 31630 kubeletfinalize.go:88] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0220 00:16:27.949398 31630 kubeletfinalize.go:132] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
[addons]: Migrating CoreDNS Corefile
I0220 00:16:28.447420 31630 dns.go:381] the CoreDNS configuration has been migrated and applied: .:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
.
I0220 00:16:28.447465 31630 dns.go:382] the old migration has been saved in the CoreDNS ConfigMap under the name [Corefile-backup]
I0220 00:16:28.447486 31630 dns.go:383] The changes in the new CoreDNS Configuration are as follows:
Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.10.0.10": field is immutable
unable to create/update the DNS service
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.createDNSService
/workspace/anago-v1.17.0-rc.2.10+70132b0f130acc/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns/dns.go:323
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.createCoreDNSAddon
/workspace/anago-v1.17.0-rc.2.10+70132b0f130acc/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns/dns.go:305
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.coreDNSAddon
我的配置文件是这样的：
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 172.16.5.151
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: master02
# taints:
# - effect: NoSchedule
# key: node-role.kubernetes.io/master
---
apiServer:
certSANs:
- "172.16.5.150"
- "172.16.5.151"
- "172.16.5.152"
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
external:
endpoints:
- "https://172.16.5.150:2379"
- "https://172.16.5.151:2379"
- "https://172.16.5.152:2379"
caFile: /etc/k8s/pki/ca.pem
certFile: /etc/k8s/pki/etcd.pem
keyFile: /etc/k8s/pki/etcd.key
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.17.0
networking:
dnsDomain: cluster.local
serviceSubnet: 10.10.0.0/16
podSubnet: 192.168.0.0/16
scheduler: {}
我检查了 kubeadm 生成的 kube-apiserver.yaml。 --service-cluster-ip-range=10.10.0.0/16 设置包含 10.10.0.10 你可以在下面看到：
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
component: kube-apiserver
tier: control-plane
name: kube-apiserver
namespace: kube-system
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=172.16.5.151
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
- --etcd-cafile=/etc/k8s/pki/ca.pem
- --etcd-certfile=/etc/k8s/pki/etcd.pem
- --etcd-keyfile=/etc/k8s/pki/etcd.key
- --etcd-servers=https://172.16.5.150:2379,https://172.16.5.151:2379,https://172.16.5.152:2379
- --insecure-port=0
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
- --requestheader-allowed-names=front-proxy-client
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --secure-port=6443
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-cluster-ip-range=10.10.0.0/16
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.17.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 172.16.5.151
path: /healthz
port: 6443
scheme: HTTPS
initialDelaySeconds: 15
timeoutSeconds: 15
name: kube-apiserver
resources:
requests:
cpu: 250m
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/pki
name: etc-pki
readOnly: true
- mountPath: /etc/k8s/pki
name: etcd-certs-0
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
hostNetwork: true
priorityClassName: system-cluster-critical
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/pki
type: DirectoryOrCreate
name: etc-pki
- hostPath:
path: /etc/k8s/pki
type: DirectoryOrCreate
name: etcd-certs-0
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
status: {}
如你所见。所有的 service-ip-range 都设置为 10.10.0.0/16。奇怪的是，当我执行 "kubectl get svc" 时，我得到的 kubernetes clusterip 是 10.96.0.1
[root@master02 manifests]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d3h
这意味着默认的服务IP范围是：10.96.0.0/16。我修改的内容不起作用。有谁知道如何自定义 service-ip-range 范围。以及如何解决我的问题？
因为这个节点我之前作为一个节点加入了集群因为这个节点我作为一个节点加入了集群before.Later我用"kubeadm reset "重置了这个command.After重置了，我加入了它作为集群的主角色。所以我在上面的问题中得到了错误。 报错是因为我重置前的clusterip范围已经记录在etcd集群中了。而且"kubeadm reset"命令没有清理etcd.So中的数据 新定义的clusterip与original.So冲突 解决方法是清理etcd中的数据，重新设置。 （由于我搭建的集群是测试集群，直接清理了etcd，生产环境请慎重）
将此答案发布为社区 Wiki 以扩展和解释根本原因。
当 kubeadm 启动时，我们没有指定任何标志，$ kubeadm init 它将使用默认值创建 kubeadm 集群。您可以签入 Kubernetes docs 标志，这些标志可以在初始化期间指定并且是默认值。
--service-cidrstring Default: "10.96.0.0/12" Use alternative range of IP address for service VIPs.
这就是默认 kubernetes 服务使用 10.96.0.1 作为 ClusterIP 的原因。
这里OP也想用自己的配置
--configstring Path to a kubeadm configuration file.
可以找到整个初始化工作流程here。
作为 Kubernetes 文档示例 Kubeadm reset
Performs a best effort revert of changes made by kubeadm init or kubeadm join.
有时取决于我们的配置，一些配置保留在集群上。
问题，提到了遇到的 OP here - External etcd clean up
kubeadm resetwill not delete any etcd data if external etcd is used. This means that if you run kubeadm init again using the same etcd endpoints, you will see state from previous clusters.
关于不可变字段：Service “kube-dns” is invalid: spec.clusterIP: Invalid value: “10.10.0.10”: field is immutable。
在 Kubernetes 中，一些字段是安全的，以防止可能中断集群工作的更改。
如果任何字段是 immutable 但我们必须更改它，则必须删除此对象并重新添加。