Kubernetes 入口 502

Question

我在我的服务器上设置了一个 k8s 集群（全部在一个节点中）并安装了 ingress-nginx。现在我想部署一个 frps 服务到集群。这是我的代码：

apiVersion: v1
kind: Namespace
metadata:
  name: helloworld
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: frps-configmap
  namespace: helloworld
data:
  frps.ini: |
    [common]
    bind_port = 7000
    vhost_http_port = 8080
    custom_404_page = /etc/frp/frps.html
  frps.html: |
    <!DOCTYPE html>
    <html lang="en">
      // ...
    </html>
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frps-deployment
  namespace: helloworld
  labels:
    app: frps
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frps
  template:
    metadata:
      labels:
        app: frps
    spec:
      containers:
        - name: frps
          image: 'snowdreamtech/frps:0.34.3'
          ports:
            - containerPort: 8080
            - containerPort: 7000
          volumeMounts:
            - name: frps-etc
              mountPath: '/etc/frp'
              readOnly: true
      volumes:
        - name: frps-etc
          configMap:
            name: frps-configmap
            items:
              - key: frps.ini
                path: frps.ini
              - key: frps.html
                path: frps.html
---
apiVersion: v1
kind: Service
metadata:
  name: frps
  namespace: helloworld
spec:
  selector:
    app: frps
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      name: vhost
    - protocol: TCP
      port: 7000
      targetPort: 7000
      name: frps
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frps-ingress
  namespace: helloworld
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: 'nginx'
  rules:
    - host: dev.helloworld.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frps
                port: 
                  name: vhost

这里是应用之前的yaml后的一些集群信息：

kubectl get deploy -A
NAMESPACE         NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
helloworld        frps-deployment            1/1     1            1           12m
ingress-nginx     ingress-nginx-controller   1/1     1            1           23d
...

kubectl get svc -A
NAMESPACE       NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
helloworld      frps                                 ClusterIP   10.97.67.98      <none>        8080/TCP,7000/TCP            6m18s
ingress-nginx   ingress-nginx-controller             NodePort    10.101.128.200   <none>        80:32038/TCP,443:31363/TCP   23d
ingress-nginx   ingress-nginx-controller-admission   ClusterIP   10.97.36.119     <none>        443/TCP                      23d
...

kubectl describe ingress --namespace=helloworld frps-ingress
Name:             frps-ingress
Namespace:        helloworld
Address:          192.168.0.176
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                Path  Backends
  ----                ----  --------
  dev.helloworld.com
                      /   frps:vhost (192.168.103.147:8080)
Annotations:          nginx.ingress.kubernetes.io/rewrite-target: /
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    13m (x2 over 13m)  nginx-ingress-controller  Scheduled for sync

但是当我运行curl -H 'HOST: dev.helloworld.com' http://localhost:32038/时，它return502:

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

我在nginx pod中执行了以下命令，得到了正确的内容：

curl http://frps.helloworld:8080
<!DOCTYPE html>
<html lang="en">
  // ...
</html>

所以我确定 frps 服务是正确的运行。我不仅尝试使用 header，还尝试更改云 dns 记录（在另一个域下），但得到了相同的结果。发生了什么，为什么 ingress-nginx return 502?

我看错了，好像是nginx发现upstream不对。 frps服务的cluster ip是10.97.67.98，而ingress-nginx使用pod ip 192.168.103.147。 pod ip不可达，service cluster ip可达。我不知道为什么以及如何解决它。

Answer 1

没想到这个问题可能是宿主机的防火墙守护进程引起的。在启用伪装并打开 kubernetes 端口后，一切正常:) 类似于：

firewall-cmd --add-masquerade --permanent
firewall-cmd --permanent --zone=public --add-port=10250-10252/tcp
firewall-cmd --permanent --zone=public --add-port=10255/tcp
firewall-cmd --permanent --zone=public --add-port=30000-32767/tcp
firewall-cmd --reload

Kubernetes 入口 502

Kubernetes Ingress 502

kubernetes

nginx-ingress