通过 Ingress 访问 K8s 服务

Question

我们已经配置了 MetalLB，因为我们的 K8s 集群托管在裸机基础设施上。它似乎运行很好，所有 pods 和运行。

[~]# kubectl get all -n metallb-system
NAME                             READY   STATUS    RESTARTS   AGE
pod/controller-b78574c59-47qfv   1/1     Running   0          24h
pod/speaker-4q2vm                1/1     Running   0          24h
pod/speaker-m8kwk                1/1     Running   0          24h
pod/speaker-t4rvs                1/1     Running   0          24h

NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/speaker   3         3         3       3            3           kubernetes.io/os=linux   24h

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/controller   1/1     1            1           24h

NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/controller-b78574c59   1         1         1       24h

我们已经通过 helm 从 https://github.com/kubernetes/ingress-nginx/releases/tag/helm-chart-3.29.0 配置了入口控制器，并将 hostNetwork、ingressClass、kind 更新为 true、ingress-nginx， DaemonSet 分别在文件 values.yaml 中。 helm 安装似乎与所有 daemonset pods 运行和为创建的入口控制器服务提供的 LB ip 一起正常工作。

[~]# kubectl get all -n ingress-nginx
NAME                                            READY   STATUS    RESTARTS   AGE
pod/devingress-ingress-nginx-controller-c2x42   1/1     Running   0          18h
pod/devingress-ingress-nginx-controller-wtmgw   1/1     Running   0          18h

NAME                                                    TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                      AGE
service/devingress-ingress-nginx-controller             LoadBalancer     x.x.x.x         1.2.3.40     80:32386/TCP,443:30020/TCP   18h
service/devingress-ingress-nginx-controller-admission   ClusterIP        x.x.x.x           <none>        443/TCP                      18h

NAME                                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/devingress-ingress-nginx-controller   2         2         2       2            2           kubernetes.io/os=linux   18h

现在我们已经部署了两个 pods，即具有 LoadBalancer 服务类型的 nginx 和具有 ClusterIP 服务类型的 nginx-deploy-main。

[~]# kubectl get all -n default
NAME                                     READY   STATUS    RESTARTS   AGE
pod/nginx-854cf6b4d7-lv5ss               1/1     Running   0          18h
pod/nginx-deploy-main-6b5457fbb5-7tg9z   1/1     Running   0          18h

NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
service/nginx                  LoadBalancer   x.x.x.x        1.2.3.41       8080:31101/TCP    18h
service/nginx-deploy-main      ClusterIP      x.x.x.x          <none>           80/TCP          18h

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx               1/1     1            1           18h
deployment.apps/nginx-deploy-main   1/1     1            1           18h

NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-854cf6b4d7               1         1         1       18h
replicaset.apps/nginx-deploy-main-6b5457fbb5   1         1         1       18h

下面是访问 nginx-deploy-main.

的入口资源设置

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  ingressClassName: nginx
  rules:
  - host: nginx-main.int.org.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-deploy-main
          servicePort: 80

并且入口资源似乎已正确创建并指向 nginx-deploy-main 服务。

[~]# kubectl get ing -n default
NAME                 CLASS   HOSTS                           ADDRESS   PORTS   AGE
ingress-resource     nginx   nginx-main.int.org.com                    80      19h

[~]# kubectl describe ing/ingress-resource -n default
Name:             ingress-resource
Namespace:        default
Address:
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host                           Path  Backends
  ----                           ----  --------
  nginx-main.int.org.com
                                 /   nginx-deploy-main:80 (x.x.x.x:80)
Annotations:                     kubernetes.io/ingress.class: nginx
Events:                          <none>

在 K8s 集群之外，我们将 nginx 设置为使用域 int.org.com 解析的反向代理。

下面是 nginx 配置，应该可以帮助我点击 url http://nginx-main.int.org.com 并获得响应，但返回的响应是 404。

upstream nginx-main.int.org.com {
  server 1.2.3.40:80;     ## Ingress Controller Service IP
}

server {
  listen 80;
  server_name nginx-main.int.org.com;
  location / {
    proxy_pass http://nginx-main.int.org.com;
  }
}

现在，当我尝试使用具有以下配置的 LoadBalancer 服务 IP 访问 nginx pod（不是 nginx-main）时，它能够提供响应并正常工作很好

upstream nginx.int.org.com {
  server 1.2.3.41:8080;
}

server {
  listen 80;
  server_name nginx.int.org.com;
  location / {
    proxy_pass http://nginx.int.org.com;
  }
}

关于 Ingress Controller 或 Resource，我在这里遗漏了什么吗？端口转发工作正常，并且能够使用相同的服务访问服务。

这确实是一个障碍，任何帮助或文档参考都会非常有用。

Answer 1

我们尝试使用另一个 Ingress Controller，即 https://github.com/nginxinc/kubernetes-ingress 并且能够使其正常工作。

以下是完成的步骤。

[~] git clone https://github.com/nginxinc/kubernetes-ingress/
[~] cd kubernetes-ingress/deployments
[~] git checkout v1.11.1
[~] kubectl apply -f common/ns-and-sa.yaml
[~] kubectl apply -f rbac/rbac.yaml
[~] kubectl apply -f common/default-server-secret.yaml
[~] kubectl apply -f common/nginx-config.yaml
[~] kubectl apply -f common/ingress-class.yaml

创建了带有额外环境参数的守护进程集 pods，即由于控制器日志中的以下问题 --enable-custom-resources=false 添加到 yaml

参考：

[~] kubectl apply -f daemon-set/nginx-ingress.yaml
[~] kubectl get all -n nginx-ingress -o wide
NAME                      READY   STATUS    RESTARTS   AGE     IP            NODE         NOMINATED NODE   READINESS GATES
pod/nginx-ingress-gd8gw   1/1     Running   0          3h55m   x.x.x.x      worker1          <none>           <none>
pod/nginx-ingress-kr9lx   1/1     Running   0          3h55m   x.x.x.x      worker2          <none>           <none>
 
NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE     CONTAINERS     IMAGES                                                  SELECTOR
daemonset.apps/nginx-ingress   2         2         2       2            2           <none>          5h14m   nginx-ingress   nginx/nginx-ingress:1.11.1   app=nginx-ingress

在端口 80 命中相应的工作节点，404 响应意味着它工作正常。

使用 github link https://github.com/vipin-k/Ingress-Controller-v1.9.0/blob/main/hotel.yml 部署了示例应用程序，并将 Ingress 对象中的主机条目更新为 hotel.int.org.com

[~] kubectl create -f hotel.yaml
[~] kubectl get all -n hotel -o wide
NAME                         READY   STATUS    RESTARTS   AGE     IP            NODE         NOMINATED NODE   READINESS GATES
pod/hotel-65d644c8f7-bj597   1/1     Running   0          3h51m   x.x.x.x     worker1          <none>           <none>
pod/hotel-65d644c8f7-csvgp   1/1     Running   0          3h51m   x.x.x.x     worker2          <none>           <none>
 
NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE     SELECTOR
service/hotel-svc   ClusterIP   x.x.x.x   <none>        80/TCP    3h51m   app=hotel
 
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                    SELECTOR
deployment.apps/hotel   2/2     2            2           3h51m   hotel        nginxdemos/hello:plain-text   app=hotel
 
NAME                               DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES                                                    SELECTOR
replicaset.apps/hotel-65d644c8f7   2         2         2       3h51m   hotel        nginxdemos/hello:plain-text   app=hotel,pod-template-hash=65d644c8f7

[~] kubectl get ing -n hotel
NAME            CLASS   HOSTS                       ADDRESS   PORTS   AGE
hotel-ingress   nginx   hotel.int.org.com            80      3h52m
[~] kubectl describe ing hotel-ingress -n hotel
Name:             hotel-ingress
Namespace:        hotel
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>) 
Rules:
  Host                       Path  Backends
  ----                       ----  --------
  hotel.int.org.com
                             /        hotel-svc:80 (x.x.x.x:80,x.x.x.x:80)
Annotations:                 Events:  <none>

更新了启用域解析的外部 nginx 配置。

upstream hotel.int.org.com {
  server 1.2.3.41:80;  #worker1
  server 1.2.3.42:80;  #worker2
}

server {
  listen 80;
  server_name hotel.int.org.com;
  location / {
    proxy_pass http://hotel.int.org.com;
  }
}

重新启动 nginx 并验证能够通过浏览器访问其来自相应运行酒店名称空间守护进程 pods.

的服务响应

[~]# curl hotel.int.org.com
Server address: x.x.x.x:80
Server name: hotel-65d644c8f7-bj597
Date: 28/Apr/2021:05:47:15 +0000
URI: /
Request ID: 28d5cfab4ea28beea49879422b7e8f4c

[~]# curl hotel.int.org.com
Server address: x.x.x.x:80
Server name: hotel-65d644c8f7-csvgp
Date: 28/Apr/2021:05:52:06 +0000
URI: /
Request ID: 4135cacf83f8bf41c9677104500e610b

也在用 MetalLB 进行探索，一旦它起作用，就会 post 解决方案

通过 Ingress 访问 K8s 服务

Access K8s Services via Ingress

nginx

jenkins

kubernetes

kubernetes-ingress

ingress-controller