使用 kubeadm 将 Kubernetes 从 1.11 升级到 1.12
Upgrading Kubernetes from 1.11 to 1.12 stuck using kubeadm
我正在尝试将 Kubernetes 集群从 1.11 升级到 1.12。我已按照正确的步骤到达此处：
[root@ip-10-0-1-124 a10-harmony-controller-5.0.0]# kubeadm upgrade apply v1.12.3 --force --config=/tmp/a10_setup/multi_master/config.yaml
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration options from a file: /tmp/a10_setup/multi_master/config.yaml
[upgrade/apply] Respecting the --cri-socket flag that is set with higher priority than the config file.
[upgrade/version] You have chosen to change the cluster version to "v1.12.3"
[upgrade/versions] Cluster version: v1.11.0
[upgrade/versions] kubeadm version: v1.12.3
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component etcd.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.12.3"...
卡在这里，没有前进。卡在这里的原因，我可以使用日志级别v9找到。在 kube-system 命名空间中找不到 kubeapiserver pod。
I0902 09:46:51.194839 616837 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"kube-apiserver-ip-10-0-1-124.ec2.internal\" not found","reason":"NotFound","details":{"name":"kube-apiserver-ip-10-0-1-124.ec2.internal","kind":"pods"},"code":404}
I0902 09:46:51.692437 616837 round_trippers.go:386] curl -k -v -XGET -H "User-Agent: kubeadm/v1.12.3 (linux/amd64) kubernetes/435f92c" -H "Accept: application/json, */*" 'https://10.0.1.124:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-ip-10-0-1-124.ec2.internal'
我的 kube-system 中的 kube-apiserver 名称为 kube-apiserver-10.0.1.124 而升级脚本正在搜索名称 kube-apiserver-ip-10-0-1-124.ec2.internal 。该脚本试图在 kube-apiserver 之后附加主机名，而我用名称 10.0.1.124
定义了 nodeName
这是我用于升级的配置：
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
api:
advertiseAddress: 10.0.1.124
controlPlaneEndpoint: 10.0.1.124
etcd:
endpoints:
- https://10.0.1.124:2379
- https://10.0.1.231:2379
- https://10.0.1.30:2379
caFile: /etc/kubernetes/pki/etcd/ca.pem
certFile: /etc/kubernetes/pki/etcd/client.pem
keyFile: /etc/kubernetes/pki/etcd/client-key.pem
networking:
podSubnet: 192.168.12.0/24
kubernetesVersion: 1.12.3
apiServerCertSANs:
- 10.0.1.124
apiServerExtraArgs:
endpoint-reconciler-type: lease
nodeName: 10.0.1.124
我可以使用一些参数让升级脚本搜索正确的名称吗？我该如何解决这个问题？
有点过时了。
请尊重次要版本，因此对于 v1.11.0 请尝试升级到 v1.12.0。您应该考虑并使用最新的受支持版本之一 Kubernetes version and version skew support policy and Supported Versions of the Kubernetes Documentation
对于最新版本，有更多具体信息：
All containers are restarted after upgrade, because the container spec hash value is changed.
You only can upgrade from one MINOR version to the next MINOR version, or between PATCH versions of the same MINOR. That is, you cannot skip MINOR versions when you upgrade. For example, you can upgrade from 1.y to 1.y+1, but not from 1.y to 1.y+2.
Additional outdated information
请尝试将 +1 次要版本从 1.11.0 升级到 1.12.0
其他帮助完整的命令：
kubeadm upgrade plan.
kubeadm upgrade --force [in order to Recovery from a failure state]
我不确定在 upgrade apply 期间使用 --force 选项是好的做法:
Force upgrading although some requirements might not be met. This also implies non-interactive mode.
您还可以使用：
--dry-run Do not change any state, just output what actions would be performed
--diff Show what differences would be applied to existing static pod manifests. See also: kubeadm upgrade apply –dry-run
希望对您有所帮助。
问题是当您的 kubernetes 主节点名不等于主机名时，kubeadm 升级脚本无法找到您的节点名。它正在尝试使用默认的主机名来搜索它。作为解决方法，您可以在 kubeadm 升级期间提供自定义配置，如下所示：
kubectl -n kube-system get cm kubeadm-config -o jsonpath={.data.MasterConfiguration} > config.yaml
在 config.yaml 的末尾添加以下块：
nodeRegistration:
name: <node-name>
现在，当您尝试使用上述配置文件进行升级时，您的升级经历了：
root@ip-10-0-1-124 centos]# kubeadm upgrade apply v1.12.3 --config config.yaml
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration options from a file: config.yaml
[upgrade/apply] Respecting the --cri-socket flag that is set with higher priority than the config file.
[upgrade/version] You have chosen to change the cluster version to "v1.12.3"
[upgrade/versions] Cluster version: v1.11.0
[upgrade/versions] kubeadm version: v1.12.3
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "10.0.1.124" as an annotation
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.12.3". Enjoy!
如果其他人也面临同样的问题，请将此作为答案发布。
我正在尝试将 Kubernetes 集群从 1.11 升级到 1.12。我已按照正确的步骤到达此处：
[root@ip-10-0-1-124 a10-harmony-controller-5.0.0]# kubeadm upgrade apply v1.12.3 --force --config=/tmp/a10_setup/multi_master/config.yaml
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration options from a file: /tmp/a10_setup/multi_master/config.yaml
[upgrade/apply] Respecting the --cri-socket flag that is set with higher priority than the config file.
[upgrade/version] You have chosen to change the cluster version to "v1.12.3"
[upgrade/versions] Cluster version: v1.11.0
[upgrade/versions] kubeadm version: v1.12.3
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Prepulled image for component etcd.
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.12.3"...
卡在这里，没有前进。卡在这里的原因，我可以使用日志级别v9找到。在 kube-system 命名空间中找不到 kubeapiserver pod。
I0902 09:46:51.194839 616837 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"kube-apiserver-ip-10-0-1-124.ec2.internal\" not found","reason":"NotFound","details":{"name":"kube-apiserver-ip-10-0-1-124.ec2.internal","kind":"pods"},"code":404}
I0902 09:46:51.692437 616837 round_trippers.go:386] curl -k -v -XGET -H "User-Agent: kubeadm/v1.12.3 (linux/amd64) kubernetes/435f92c" -H "Accept: application/json, */*" 'https://10.0.1.124:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-ip-10-0-1-124.ec2.internal'
我的 kube-system 中的 kube-apiserver 名称为 kube-apiserver-10.0.1.124 而升级脚本正在搜索名称 kube-apiserver-ip-10-0-1-124.ec2.internal 。该脚本试图在 kube-apiserver 之后附加主机名，而我用名称 10.0.1.124
这是我用于升级的配置：
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
api:
advertiseAddress: 10.0.1.124
controlPlaneEndpoint: 10.0.1.124
etcd:
endpoints:
- https://10.0.1.124:2379
- https://10.0.1.231:2379
- https://10.0.1.30:2379
caFile: /etc/kubernetes/pki/etcd/ca.pem
certFile: /etc/kubernetes/pki/etcd/client.pem
keyFile: /etc/kubernetes/pki/etcd/client-key.pem
networking:
podSubnet: 192.168.12.0/24
kubernetesVersion: 1.12.3
apiServerCertSANs:
- 10.0.1.124
apiServerExtraArgs:
endpoint-reconciler-type: lease
nodeName: 10.0.1.124
我可以使用一些参数让升级脚本搜索正确的名称吗？我该如何解决这个问题？
有点过时了。
请尊重次要版本，因此对于 v1.11.0 请尝试升级到 v1.12.0。您应该考虑并使用最新的受支持版本之一 Kubernetes version and version skew support policy and Supported Versions of the Kubernetes Documentation
对于最新版本，有更多具体信息：
All containers are restarted after upgrade, because the container spec hash value is changed. You only can upgrade from one MINOR version to the next MINOR version, or between PATCH versions of the same MINOR. That is, you cannot skip MINOR versions when you upgrade. For example, you can upgrade from 1.y to 1.y+1, but not from 1.y to 1.y+2.
Additional outdated information 请尝试将 +1 次要版本从 1.11.0 升级到 1.12.0 其他帮助完整的命令：
kubeadm upgrade plan.
kubeadm upgrade --force [in order to Recovery from a failure state]
我不确定在 upgrade apply 期间使用 --force 选项是好的做法:
Force upgrading although some requirements might not be met. This also implies non-interactive mode.
您还可以使用：
--dry-run Do not change any state, just output what actions would be performed
--diff Show what differences would be applied to existing static pod manifests. See also: kubeadm upgrade apply –dry-run
希望对您有所帮助。
问题是当您的 kubernetes 主节点名不等于主机名时，kubeadm 升级脚本无法找到您的节点名。它正在尝试使用默认的主机名来搜索它。作为解决方法，您可以在 kubeadm 升级期间提供自定义配置，如下所示：
kubectl -n kube-system get cm kubeadm-config -o jsonpath={.data.MasterConfiguration} > config.yaml
在 config.yaml 的末尾添加以下块：
nodeRegistration:
name: <node-name>
现在，当您尝试使用上述配置文件进行升级时，您的升级经历了：
root@ip-10-0-1-124 centos]# kubeadm upgrade apply v1.12.3 --config config.yaml
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration options from a file: config.yaml
[upgrade/apply] Respecting the --cri-socket flag that is set with higher priority than the config file.
[upgrade/version] You have chosen to change the cluster version to "v1.12.3"
[upgrade/versions] Cluster version: v1.11.0
[upgrade/versions] kubeadm version: v1.12.3
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "10.0.1.124" as an annotation
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.12.3". Enjoy!
如果其他人也面临同样的问题，请将此作为答案发布。