如何在Suse Linux enterprise server 15台虚拟机中安装kubernetes？
How to install kubernetes in Suse Linux enterprize server 15 virtual machines?
我们正在尝试在 SUSE enterprize linux 服务器 v15 中安装 kubernetes。我们发现没有办法使用kubeadm来安装k8s。 SUSE提供Container as a service Platform(CaasP)来安装k8s。
我们只有少量虚拟机和 suse 订阅。我们可以在其中安装 CaasP 吗？
我们找不到在虚拟机中安装它的任何文档。
有什么方法可以在虚拟机中逐步安装 CaasP 吗？
SLES 上的 Kubeadm
可以使用 kubeadm.
在 SUSE Linux Enterprise Server 15 上安装 Kubernetes
您可以在下面找到分步示例。
该示例在以下云 VM 映像上进行了测试：
GCP :
- SUSE Linux Enterprise Server 15 SP1 x86_x64
AWS :
- openSUSE-Leap-15.2-v20200710-HVM-x86_64-548f7b74-f1d6-437e-b650-f6315f6d8aa3-ami-0f5745b812a5b7654.4 - ami-023643495f15f104b
- suse-sles-15-sp1-v20200615-hvm-ssd-x86_64 - ami-0044ae6906d786f4b
蔚蓝:
- SUSE Enterprise Linux 15 SP1 +补丁
因此，它很有可能与其他图像一起使用，只需进行少量更改。
它也在 Vagrant box trueability/sles-15-sp1 上进行了测试，由于订阅密钥已过期，因此需要一些额外的步骤。我使用了 OSS repositories 并忽略了过期错误：
# add OSS repository for software installation
$ zypper addrepo http://download.opensuse.org/distribution/leap/15.2/repo/oss/ public
# add repository for installing newer Docker version
$ zypper addrepo https://download.opensuse.org/repositories/Virtualization:containers/openSUSE_Leap_15.0/Virtualization:containers.repo virt
# install symbols required by Docker:
$ zypper install libseccomp-devel
# turn off all swap partitions. Comment appropriate /etc/fstab entry as well.
$ swapoff -a
# Rest of the steps is similar except additional argument during cluster initialization.
# This box is using btrfs for /var/lib/docker and kubeadm complains about it.
# I've just asked kubeadm to ignore that fact.
# Even with btrfs it can start and run pods, but there might be some problems with Persistent Volumes usage,
# so consider using additional xfs or ext4 partition for /var/lib/docker
$ kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all
云虚拟机：
Cloud SLES 15 SP1 镜像使用 xfs 作为他们的 / 文件系统并且不使用开箱即用的交换，并且 kubeadm 顺利通过了所有飞行前检查。
# become root
$ sudo -s
# install docker
$ zypper refresh
$ zypper install docker
# configure sysctl for Kubernetes
$ cat <<EOF >> /etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv4.conf.all.forwarding=1
net.bridge.bridge-nf-call-iptables=1
EOF
# add Google repository for installing Kubernetes packages
#$ zypper addrepo --type yum --gpgcheck-strict --refresh https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 google-k8s
#or
$ cat <<EOF > /etc/zypp/repos.d/google-k8s.repo
[google-k8s]
name=google-k8s
enabled=1
autorefresh=1
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
type=rpm-md
gpgcheck=1
repo_gpgcheck=1
pkg_gpgcheck=1
EOF
# import Google repository keys
$ rpm --import https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
$ rpm --import https://packages.cloud.google.com/yum/doc/yum-key.gpg
$ rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n'
# the following repository was needed only for GCP image
# other images was able successfully install conntrack-tools using existing repository
$ zypper addrepo https://download.opensuse.org/repositories/security:netfilter/SLE_12/security:netfilter.repo conntrack
$ zypper refresh conntrack
# conntrack presence is checked during kubeadm pre-flight checks
# but zypper unable to find appropriate dependency for kubelet,
# so let's install it manually
$ zypper install conntrack-tools
# refresh Google repository cache and check if we see several versions of Kubernetes packages to choose from
$ zypper refresh google-k8s
$ zypper packages --repo google-k8s
# install latest available kubelet package
# ignore conntrack dependency and install kubelet (Solution 2 in my case)
$ zypper install kubelet
# install kubeadm package. kubectl and cri-tools are installed as kubeadm dependency
$ zypper install kubeadm
# force docker to use systemd cgroup driver and overlay2 storage driver.
# Check the links in the end of the answer for details.
# BTW, kubelet would work even with default content of the file.
$ cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
# Not sure if it's necessary it was taken from the Kubernetes documentation
$ mkdir -p /etc/systemd/system/docker.service.d
# lets start and enable docker and kubelet services
$ systemctl start docker.service
$ systemctl enable docker.service
$ systemctl enable kubelet.service
# apply configured earlier sysctl settings.
# net.bridge.bridge-nf-call-iptables becomes available after successfully starting
# Docker service
$ sysctl -p
# Now it's time to initialize Kubernetes master node.
# Ignore pre-flight checks for Vagrant box.
$ kubeadm init --pod-network-cidr=10.244.0.0/16
# prepare kubectl configuration to connect the cluster
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
# Check if api-server responds to our requests.
# At this moment it's fine to see master node in NotReady state.
$ kubectl get nodes
# Deploy Flannel network addon
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# remove taint from the master node.
# It allows master node to run application pods.
# At least one worker node is required if this step is skipped.
$ kubectl taint nodes --all node-role.kubernetes.io/master-
# run test pod to check if everything works fine
$ kubectl run nginx1 --image=nginx
# after some time... ~ 3-5 minutes
# check the pods' state
$ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default nginx1 1/1 Running 0 74s 10.244.0.4 suse-test <none> <none>
kube-system coredns-66bff467f8-vc2x4 1/1 Running 0 2m26s 10.244.0.2 suse-test <none> <none>
kube-system coredns-66bff467f8-w4jvq 1/1 Running 0 2m26s 10.244.0.3 suse-test <none> <none>
kube-system etcd-suse-test 1/1 Running 0 2m41s 10.4.0.4 suse-test <none> <none>
kube-system kube-apiserver-suse-test 1/1 Running 0 2m41s 10.4.0.4 suse-test <none> <none>
kube-system kube-controller-manager-suse-test 1/1 Running 0 2m41s 10.4.0.4 suse-test <none> <none>
kube-system kube-flannel-ds-amd64-mbfxp 1/1 Running 0 2m12s 10.4.0.4 suse-test <none> <none>
kube-system kube-proxy-cw5xm 1/1 Running 0 2m26s 10.4.0.4 suse-test <none> <none>
kube-system kube-scheduler-suse-test 1/1 Running 0 2m41s 10.4.0.4 suse-test <none> <none>
# check if the test pod is working fine
# curl 10.244.0.4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...skipped...
# basic Kubernetes installation is done
其他材料：
Container runtimes（Kubernetes 文档页面）
Giving error saying “unsupported graph driver: btrfs” in SLES when try to Kubeadm init
（如果你有 btrfs / 分区，你可以挂载额外的 xfs 或 ext4 分区供 /var/lib/docker 使用 overlay2 Docker存储驱动）
OverlayFS support for SLES 12（我希望 SLES 15 也支持它。）
-
OS Distribution (x86_64): SLES 15
Enterprise Engine: 19.03.x
UCP: 3.2.x
DTR: 2.7.x
Storage Driver: overlay2,btrfs
Orchestration: Swarm mode, Kubernetes
DTR Storage Backend: NFSv4, NFSv3, Amazon S3, S3 Compliant Alternatives,
Azure Storage (Blob), Google Cloud Storage, OpenStack Swift,
Local Filesystem
-
-
Install Linux Kernel 4.12 in openSUSE（如果你想在 Linux 内核中添加 AUFS 支持）
关于 SUSE CaaSP 的资料
我们正在尝试在 SUSE enterprize linux 服务器 v15 中安装 kubernetes。我们发现没有办法使用kubeadm来安装k8s。 SUSE提供Container as a service Platform(CaasP)来安装k8s。
我们只有少量虚拟机和 suse 订阅。我们可以在其中安装 CaasP 吗？ 我们找不到在虚拟机中安装它的任何文档。
有什么方法可以在虚拟机中逐步安装 CaasP 吗？
SLES 上的 Kubeadm
可以使用 kubeadm.
在 SUSE Linux Enterprise Server 15 上安装 Kubernetes
您可以在下面找到分步示例。
该示例在以下云 VM 映像上进行了测试：
GCP :
- SUSE Linux Enterprise Server 15 SP1 x86_x64
AWS :
- openSUSE-Leap-15.2-v20200710-HVM-x86_64-548f7b74-f1d6-437e-b650-f6315f6d8aa3-ami-0f5745b812a5b7654.4 - ami-023643495f15f104b
- suse-sles-15-sp1-v20200615-hvm-ssd-x86_64 - ami-0044ae6906d786f4b
蔚蓝:
- SUSE Enterprise Linux 15 SP1 +补丁
因此，它很有可能与其他图像一起使用，只需进行少量更改。
它也在 Vagrant box trueability/sles-15-sp1 上进行了测试，由于订阅密钥已过期，因此需要一些额外的步骤。我使用了 OSS repositories 并忽略了过期错误：
# add OSS repository for software installation
$ zypper addrepo http://download.opensuse.org/distribution/leap/15.2/repo/oss/ public
# add repository for installing newer Docker version
$ zypper addrepo https://download.opensuse.org/repositories/Virtualization:containers/openSUSE_Leap_15.0/Virtualization:containers.repo virt
# install symbols required by Docker:
$ zypper install libseccomp-devel
# turn off all swap partitions. Comment appropriate /etc/fstab entry as well.
$ swapoff -a
# Rest of the steps is similar except additional argument during cluster initialization.
# This box is using btrfs for /var/lib/docker and kubeadm complains about it.
# I've just asked kubeadm to ignore that fact.
# Even with btrfs it can start and run pods, but there might be some problems with Persistent Volumes usage,
# so consider using additional xfs or ext4 partition for /var/lib/docker
$ kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all
云虚拟机：
Cloud SLES 15 SP1 镜像使用 xfs 作为他们的 / 文件系统并且不使用开箱即用的交换，并且 kubeadm 顺利通过了所有飞行前检查。
# become root
$ sudo -s
# install docker
$ zypper refresh
$ zypper install docker
# configure sysctl for Kubernetes
$ cat <<EOF >> /etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv4.conf.all.forwarding=1
net.bridge.bridge-nf-call-iptables=1
EOF
# add Google repository for installing Kubernetes packages
#$ zypper addrepo --type yum --gpgcheck-strict --refresh https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 google-k8s
#or
$ cat <<EOF > /etc/zypp/repos.d/google-k8s.repo
[google-k8s]
name=google-k8s
enabled=1
autorefresh=1
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
type=rpm-md
gpgcheck=1
repo_gpgcheck=1
pkg_gpgcheck=1
EOF
# import Google repository keys
$ rpm --import https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
$ rpm --import https://packages.cloud.google.com/yum/doc/yum-key.gpg
$ rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n'
# the following repository was needed only for GCP image
# other images was able successfully install conntrack-tools using existing repository
$ zypper addrepo https://download.opensuse.org/repositories/security:netfilter/SLE_12/security:netfilter.repo conntrack
$ zypper refresh conntrack
# conntrack presence is checked during kubeadm pre-flight checks
# but zypper unable to find appropriate dependency for kubelet,
# so let's install it manually
$ zypper install conntrack-tools
# refresh Google repository cache and check if we see several versions of Kubernetes packages to choose from
$ zypper refresh google-k8s
$ zypper packages --repo google-k8s
# install latest available kubelet package
# ignore conntrack dependency and install kubelet (Solution 2 in my case)
$ zypper install kubelet
# install kubeadm package. kubectl and cri-tools are installed as kubeadm dependency
$ zypper install kubeadm
# force docker to use systemd cgroup driver and overlay2 storage driver.
# Check the links in the end of the answer for details.
# BTW, kubelet would work even with default content of the file.
$ cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
# Not sure if it's necessary it was taken from the Kubernetes documentation
$ mkdir -p /etc/systemd/system/docker.service.d
# lets start and enable docker and kubelet services
$ systemctl start docker.service
$ systemctl enable docker.service
$ systemctl enable kubelet.service
# apply configured earlier sysctl settings.
# net.bridge.bridge-nf-call-iptables becomes available after successfully starting
# Docker service
$ sysctl -p
# Now it's time to initialize Kubernetes master node.
# Ignore pre-flight checks for Vagrant box.
$ kubeadm init --pod-network-cidr=10.244.0.0/16
# prepare kubectl configuration to connect the cluster
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
# Check if api-server responds to our requests.
# At this moment it's fine to see master node in NotReady state.
$ kubectl get nodes
# Deploy Flannel network addon
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# remove taint from the master node.
# It allows master node to run application pods.
# At least one worker node is required if this step is skipped.
$ kubectl taint nodes --all node-role.kubernetes.io/master-
# run test pod to check if everything works fine
$ kubectl run nginx1 --image=nginx
# after some time... ~ 3-5 minutes
# check the pods' state
$ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default nginx1 1/1 Running 0 74s 10.244.0.4 suse-test <none> <none>
kube-system coredns-66bff467f8-vc2x4 1/1 Running 0 2m26s 10.244.0.2 suse-test <none> <none>
kube-system coredns-66bff467f8-w4jvq 1/1 Running 0 2m26s 10.244.0.3 suse-test <none> <none>
kube-system etcd-suse-test 1/1 Running 0 2m41s 10.4.0.4 suse-test <none> <none>
kube-system kube-apiserver-suse-test 1/1 Running 0 2m41s 10.4.0.4 suse-test <none> <none>
kube-system kube-controller-manager-suse-test 1/1 Running 0 2m41s 10.4.0.4 suse-test <none> <none>
kube-system kube-flannel-ds-amd64-mbfxp 1/1 Running 0 2m12s 10.4.0.4 suse-test <none> <none>
kube-system kube-proxy-cw5xm 1/1 Running 0 2m26s 10.4.0.4 suse-test <none> <none>
kube-system kube-scheduler-suse-test 1/1 Running 0 2m41s 10.4.0.4 suse-test <none> <none>
# check if the test pod is working fine
# curl 10.244.0.4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...skipped...
# basic Kubernetes installation is done
其他材料：
Container runtimes（Kubernetes 文档页面）
Giving error saying “unsupported graph driver: btrfs” in SLES when try to Kubeadm init
（如果你有btrfs/ 分区，你可以挂载额外的xfs或ext4分区供/var/lib/docker使用overlay2Docker存储驱动）OverlayFS support for SLES 12（我希望 SLES 15 也支持它。）
-
OS Distribution (x86_64): SLES 15 Enterprise Engine: 19.03.x UCP: 3.2.x DTR: 2.7.x Storage Driver: overlay2,btrfs Orchestration: Swarm mode, Kubernetes DTR Storage Backend: NFSv4, NFSv3, Amazon S3, S3 Compliant Alternatives, Azure Storage (Blob), Google Cloud Storage, OpenStack Swift, Local Filesystem Install Linux Kernel 4.12 in openSUSE（如果你想在 Linux 内核中添加 AUFS 支持）