如何在 BareMetal 部署中通过 Ingress 访问 kubernetes 集群？

Question

我弄了两天了，还是想不通

整个部署在裸机上。

为简单起见，我将集群从 HA 最小化为 1 个主节点和 2 个工作节点。

$ kubectl get nodes
NAME                          STATUS   ROLES    AGE    VERSION
worker1   Ready    <none>   99m    v1.19.2
worker2   Ready    <none>   99m    v1.19.2
master    Ready    master   127m   v1.19.2

我是运行 Nginx-ingress 但我认为这无关紧要，因为相同的配置也应该适用于例如 HaProxy。

$ kubectl -n ingress-nginx get pod
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-g645g        0/1     Completed   0          129m
ingress-nginx-admission-patch-ftg7p         0/1     Completed   2          129m
ingress-nginx-controller-587cd59444-cxm7z   1/1     Running     0          129m

我可以看到集群上没有外部IP：

$ kubectl get service -A
NAMESPACE                NAME                                 TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)                      AGE
cri-o-metrics-exporter   cri-o-metrics-exporter               ClusterIP   192.168.11.163    <none>        80/TCP                       129m
default                  kubernetes                           ClusterIP   192.168.0.1       <none>        443/TCP                      130m
ingress-nginx            ingress-nginx-controller             NodePort    192.168.30.224    <none>        80:32647/TCP,443:31706/TCP   130m
ingress-nginx            ingress-nginx-controller-admission   ClusterIP   192.168.212.9     <none>        443/TCP                      130m
kube-system              kube-dns                             ClusterIP   192.168.0.10      <none>        53/UDP,53/TCP,9153/TCP       130m
kube-system              metrics-server                       ClusterIP   192.168.178.171   <none>        443/TCP                      129m
kubernetes-dashboard     dashboard-metrics-scraper            ClusterIP   192.168.140.142   <none>        8000/TCP                     129m
kubernetes-dashboard     kubernetes-dashboard                 ClusterIP   192.168.100.126   <none>        443/TCP                      129m

ConfigMap 示例：

apiVersion: v1
kind: ConfigMap
metadata:
  name: dashboard-ingress-nginx
  namespace: kubernetes-dashboard
data:
  ssl-certificate: my-cert

Ingress 配置示例：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard-ingress-ssl
  namespace: kubernetes-dashboard
  annotations:
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/whitelist-source-range: 10.96.0.0/16  #the IP to be allowed
  spec:
    tls:
      - hosts:
        - kube.my.domain.internal
        secretName: my-cert
    rules:
      - host: kube.my.domain.internal
        http:
          paths:
          - path: /
            backend:
              serviceName: kubernetes-dashboard
              servicePort: 443

如果将我的浏览器重定向到域，例如https://kube.my.domain.internal 我看到 403 禁止。是否可能是由于 RBAC 规则导致我无法查看仪表板？

我发现了相关问题，但尽管这似乎配置对其他用户有效，因为他们没有 . I also tried to whitelist a big range of IPs as described here Restricting Access By IP (Allow/Block Listing) Using NGINX-Ingress Controller in Kubernetes 但结果仍然相同。

然而我也无法理解为什么 Nginx-ingress 只在一个节点上启动，而我希望在两个节点（工作节点）上启动。我在任何节点上都没有标签。

我还阅读了有关 MetalLB Bare-metal considerations 的信息，但就我而言，我并没有尝试访问专用网络之外的 Web，我只是试图将集群外部的节点连接到集群中。我可能是错的，但我认为此时不需要这样做。

更新： 我已经按照官方页面 Web UI (Dashboard) 中的记录成功地使用 kubectl 代理启动了仪表板，但是因为我想将我的集群升级到 HA，所以这是不是最好的解决方案。如果代理运行所在的节点出现故障，则仪表板将无法访问。

Update2: 在遵循 metallb/Layer 2 Configuration 的文档之后，我得到了以下几点：

$ kubectl get pods -A -o wide
NAMESPACE                NAME                                           READY   STATUS      RESTARTS   AGE     IP              NODE                          NOMINATED NODE   READINESS GATES
cri-o-metrics-exporter   cri-o-metrics-exporter-77c9cf9746-5xw4d        1/1     Running     0          30m     172.16.9.131    workerNode   <none>           <none>
ingress-nginx            ingress-nginx-admission-create-cz9h9           0/1     Completed   0          31m     172.16.9.132    workerNode   <none>           <none>
ingress-nginx            ingress-nginx-admission-patch-8fkhk            0/1     Completed   2          31m     172.16.9.129    workerNode   <none>           <none>
ingress-nginx            ingress-nginx-controller-8679c5678d-fmc2q      1/1     Running     0          31m     172.16.9.134    workerNode   <none>           <none>
kube-system              calico-kube-controllers-574d679d8c-7jt87       1/1     Running     0          32m     172.16.25.193   masterNode          <none>           <none>
kube-system              calico-node-sf2cn                              1/1     Running     0          9m11s   10.96.95.52     workerNode   <none>           <none>
kube-system              calico-node-zq9vf                              1/1     Running     0          32m     10.96.96.98     masterNode          <none>           <none>
kube-system              coredns-7588b55795-5pg6m                       1/1     Running     0          32m     172.16.25.195   masterNode          <none>           <none>
kube-system              coredns-7588b55795-n8z2p                       1/1     Running     0          32m     172.16.25.194   masterNode          <none>           <none>
kube-system              etcd-masterNode                      1/1     Running     0          32m     10.96.96.98     masterNode          <none>           <none>
kube-system              kube-apiserver-masterNode            1/1     Running     0          32m     10.96.96.98     masterNode          <none>           <none>
kube-system              kube-controller-manager-masterNode   1/1     Running     0          32m     10.96.96.98     masterNode          <none>           <none>
kube-system              kube-proxy-6d5sj                               1/1     Running     0          9m11s   10.96.95.52     workerNode   <none>           <none>
kube-system              kube-proxy-9dfbk                               1/1     Running     0          32m     10.96.96.98     masterNode          <none>           <none>
kube-system              kube-scheduler-masterNode            1/1     Running     0          32m     10.96.96.98     masterNode          <none>           <none>
kube-system              metrics-server-76bb4cfc9f-5tzfh                1/1     Running     0          31m     172.16.9.130    workerNode   <none>           <none>
kubernetes-dashboard     dashboard-metrics-scraper-5f644f6df-8sjsx      1/1     Running     0          31m     172.16.25.197   masterNode          <none>           <none>
kubernetes-dashboard     kubernetes-dashboard-85b6486959-thhnl          1/1     Running     0          31m     172.16.25.196   masterNode          <none>           <none>
metallb-system           controller-56f5f66c6f-5vvhf                    1/1     Running     0          31m     172.16.9.133    workerNode   <none>           <none>
metallb-system           speaker-n5gxx                                  1/1     Running     0          31m     10.96.96.98     masterNode          <none>           <none>
metallb-system           speaker-n9x9v                                  1/1     Running     0          8m51s   10.96.95.52     workerNode   <none>           <none>
$ kubectl get service -A
NAMESPACE                NAME                                 TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)                      AGE
cri-o-metrics-exporter   cri-o-metrics-exporter               ClusterIP   192.168.74.27     <none>        80/TCP                       31m
default                  kubernetes                           ClusterIP   192.168.0.1       <none>        443/TCP                      33m
ingress-nginx            ingress-nginx-controller             NodePort    192.168.201.230   <none>        80:30509/TCP,443:31554/TCP   32m
ingress-nginx            ingress-nginx-controller-admission   ClusterIP   192.168.166.218   <none>        443/TCP                      32m
kube-system              kube-dns                             ClusterIP   192.168.0.10      <none>        53/UDP,53/TCP,9153/TCP       32m
kube-system              metrics-server                       ClusterIP   192.168.7.75      <none>        443/TCP                      31m
kubernetes-dashboard     dashboard-metrics-scraper            ClusterIP   192.168.51.178    <none>        8000/TCP                     31m
kubernetes-dashboard     kubernetes-dashboard                 ClusterIP   192.168.50.70     <none>        443/TCP                      31m

但是我看不到 public IP，所以我可以通过 NAT 访问集群。

Answer 1

我进行 BareMetal 设置的方法是安装 METALLB 和 ingress-nxginx 并使用 NAT 将我的主机（端口 80 和 443）上收到的流量转发到 ingress-nginx.

# MetalLB installation

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml

# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
kubectl apply -f $YAML_FILES/0_cluster_setup_metallb_conf.yaml

0_cluster_setup_metallb_conf.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.204.61.240-10.204.61.250

Answer 2

经过这么多时间，我终于弄明白了。我是 k8s 的初学者，所以该解决方案可能会帮助其他初学者。

我决定在仪表板运行的同名 space 上启动入口。用户可以选择不同的名称 space 只需确保使用 connect your name space with kubernetes-dashboard 服务。可以在此处找到文档 Understanding namespaces and DNS.

将与 NGINX ingress 一起使用的示例的完整代码：

apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: kubernetes-dashboard-ingress
        namespace: kubernetes-dashboard
        annotations:
          kubernetes.io/ingress.class: "nginx"
          nginx.ingress.kubernetes.io/ssl-passthrough: "true"
          nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      spec:
        tls:
        - hosts:
            - "dashboard.example.com"
          secretName: kubernetes-dashboard-secret
        rules:
        - host: "dashboard.example.com"
          http:
            paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: kubernetes-dashboard
                  port:
                    number: 443

请记住遵循入口控制器的注释（在示例 NGINX 上），因为它们可能会在以后的版本中更新。例如，从版本 image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1 开始，注释 secure-backends 被 backend-protocol.

取代

此外，在提供的示例中，外部负载平衡器配置必须为 SSL Pass-Through。

更新： 如果其他人是 k8s 的初学者，我不太清楚的一个小输入是，如果用户决定使用 MetalLB 您需要在入口文件中指定 type: LoadBalancer。

如何在 BareMetal 部署中通过 Ingress 访问 kubernetes 集群？

How to access kubernetes cluster through Ingress on BareMetal deployments?

kubernetes

kubernetes-dashboard

kubernetes-ingress

nginx-ingress