GKE 上的 nginx-ingress 无法路由到已配置服务的路径
nginx-ingress on GKE fails to route to path for configured service
我正在尝试为 GKE 集群配置 nginx 入口并在配置的子域上定义路径。似乎即使我能够成功 ping 主机，并且域绑定正确完成，但每当我尝试访问配置的路径时，我都会收到 404 返回。
我的目标是能够为我的入口控制器配置一个静态 IP，并在不同路径上公开多个服务。
您可以在下面找到我的部署文件 - 我要补充的另一件事是我使用 Terraform 来自动配置和部署 GCP 和 Kubernetes 资源。
成功配置 GKE 集群后，我首先从 here 部署官方 nginx-ingress 控制器 - 在我的 Terraform 脚本下方，该脚本使用我在 GCP 上配置的自定义静态 IP 配置和部署控制器。
resource "helm_release" "nginx" {
name = "nginx"
chart = "nginx-stable/nginx-ingress"
timeout = 900
set {
name = "controller.stats.enabled"
value = true
}
set {
name = "controller.service.type"
value = "LoadBalancer"
}
set {
name = "controller.service.loadBalancerIP"
value = "<MY_STATIC_IP_ADDRESS>"
}
}
在我也通过 Terraform 部署的入口配置下方：
resource "kubernetes_ingress" "ingress" {
wait_for_load_balancer = true
metadata {
name = "app-ingress"
annotations = {
"kubernetes.io/ingress.class": "nginx"
"nginx.ingress.kubernetes.io/rewrite-target": "/"
"kubernetes.io/ingress.global-static-ip-name": <MY_STATIC_IP_ADDRESS>
}
}
spec {
rule {
host = custom.my_domain.com
http {
path {
backend {
service_name = "app-service"
service_port = 5000
}
path = "/app"
}
}
}
}
}
以及从 GCP 获取的结果入口配置：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.global-static-ip-name: static-ip-name
nginx.ingress.kubernetes.io/rewrite-target: /
creationTimestamp: "2021-04-14T20:28:41Z"
generation: 7
name: app-ingress
namespace: default
resourceVersion: HIDDEN
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/app-ingress
uid: HIDDEN
spec:
rules:
- host: custom.my_domain.com
http:
paths:
- backend:
serviceName: app-service
servicePort: 5000
path: /app
status:
loadBalancer:
ingress:
- ip: <MY_STATIC_IP_ADDRESS>
kubectl describe ingress app-ingress 命令的输出：
Name: app-ingress
Namespace: default
Address: <MY_STATIC_IP_ADDRESS>
Default backend: default-http-backend:80 (192.168.10.8:8080)
Rules:
Host Path Backends
---- ---- --------
custom.my_domain.com
/app app-service:5000 (192.168.10.11:5000)
Annotations: kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.global-static-ip-name: static-ip-name
nginx.ingress.kubernetes.io/rewrite-target: /
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 16m (x6 over 32m) nginx-ingress-controller Configuration for default/app-ingress was added or updated
我使用以下配置文件部署了我试图公开的应用程序：
pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: app-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: default
service.yaml
apiVersion: v1
kind: Service
metadata:
name: app-service
spec:
type: NodePort
ports:
- port: 5000
targetPort: 5000
protocol: TCP
name: http
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-deployment
spec:
replicas: 1
template:
spec:
restartPolicy: Always
volumes:
- name: app-pvc
persistentVolumeClaim:
claimName: app-pvc
containers:
- name: app-container
image: "eu.gcr.io/<PROJECT_ID>/<IMAGE_NAME>:VERSION_TAG"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5000
livenessProbe:
tcpSocket:
port: 5000
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
tcpSocket:
port: 5000
initialDelaySeconds: 15
periodSeconds: 20
volumeMounts:
- mountPath: "/data"
name: app-pvc
一切都已成功部署，因为我可以通过 运行 以下命令通过配置的服务在本地直接连接到应用程序：
kubectl port-forward service/app-service 5000:5000
这让我可以在我的浏览器中访问该应用程序，并且一切正常。
为了确保我配置的 <MY_STATIC_IP_ADDRESS> 正确绑定到 custom.my_domain.com，我尝试对主机执行 ping 操作，但确实得到了正确的响应：
ping custom.my_domain.com
Pinging custom.my_domain.com [<MY_STATIC_IP_ADDRESS>] with 32 bytes of data:
Reply from <MY_STATIC_IP_ADDRESS>: bytes=32 time=36ms TTL=113
Reply from <MY_STATIC_IP_ADDRESS>: bytes=32 time=37ms TTL=113
Reply from <MY_STATIC_IP_ADDRESS>: bytes=32 time=36ms TTL=113
Reply from <MY_STATIC_IP_ADDRESS>: bytes=32 time=45ms TTL=113
Ping statistics for <MY_STATIC_IP_ADDRESS>:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 36ms, Maximum = 45ms, Average = 38ms
即使一切似乎都按预期工作，但每当我尝试在浏览器中导航到 custom.my_domain.com/app 时，即使在等待超过 30 米以确保入口配置已在 GCP 上正确注册：
这是显示在我的 nginx-controller pod 日志中的条目：
<HIDDEN_LOCAL_IP> - - [14/Apr/2021:21:18:10 +0000] "GET /app/ HTTP/1.1" 404 232 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0" "-"
更新 #1
看来，如果我更新入口以直接在 / 路径上公开目标服务，它就会按预期工作。在更新的配置下方。尽管如此，如果我尝试设置任何其他路径，它似乎不起作用。
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.global-static-ip-name: static-ip-name
nginx.ingress.kubernetes.io/rewrite-target: /
creationTimestamp: "2021-04-14T20:28:41Z"
generation: 7
name: app-ingress
namespace: default
resourceVersion: HIDDEN
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/app-ingress
uid: HIDDEN
spec:
rules:
- host: custom.my_domain.com
http:
paths:
- backend:
serviceName: app-service
servicePort: 5000
path: /
status:
loadBalancer:
ingress:
- ip: <MY_STATIC_IP_ADDRESS>
更新 #2
看完评论区@jccampanero 分享的资料后，我得到了一个工作配置。
而不是使用 nginx-stable which is referenced on the official nginx website, I used the one here 并相应地更新了我的 Terraform 脚本以使用与我拥有的配置完全相同的脚本。
之后，我不得不按照文档 here 更新我的入口 - 在更新的配置下方：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.global-static-ip-name: static-ip-name
nginx.ingress.kubernetes.io/rewrite-target: /
creationTimestamp: "2021-04-14T20:28:41Z"
generation: 7
name: app-ingress
namespace: default
resourceVersion: HIDDEN
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/app-ingress
uid: HIDDEN
spec:
rules:
- host: custom.my_domain.com
http:
paths:
- backend:
serviceName: app-service
servicePort: 5000
path: /app(/|$)(.*)
status:
loadBalancer:
ingress:
- ip: <MY_STATIC_IP_ADDRESS>
正如问题评论和问题本身所指出的那样，@vladzam 对此进行了很好的记录，有两个是问题的原因。
一方面，通过 Helm stable 通道可用的 nginx 入口控制器似乎是 deprecated in favor of the new ingress-nginx controller - please, see the Github repo and the official documentation。
另一方面，这似乎是与 Rewrite target 注释的定义有关的问题。根据 docs:
Starting in Version 0.22.0, ingress definitions using the annotation nginx.ingress.kubernetes.io/rewrite-target are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group.
因此，有必要修改入口资源的定义以考虑到这一变化。例如：
$ echo '
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
name: rewrite
namespace: default
spec:
rules:
- host: rewrite.bar.com
http:
paths:
- backend:
serviceName: http-svc
servicePort: 80
path: /something(/|$)(.*)
' | kubectl create -f -
问题本身提供了准确的入口资源定义。
我正在尝试为 GKE 集群配置 nginx 入口并在配置的子域上定义路径。似乎即使我能够成功 ping 主机，并且域绑定正确完成，但每当我尝试访问配置的路径时，我都会收到 404 返回。
我的目标是能够为我的入口控制器配置一个静态 IP，并在不同路径上公开多个服务。
您可以在下面找到我的部署文件 - 我要补充的另一件事是我使用 Terraform 来自动配置和部署 GCP 和 Kubernetes 资源。
成功配置 GKE 集群后，我首先从 here 部署官方 nginx-ingress 控制器 - 在我的 Terraform 脚本下方，该脚本使用我在 GCP 上配置的自定义静态 IP 配置和部署控制器。
resource "helm_release" "nginx" {
name = "nginx"
chart = "nginx-stable/nginx-ingress"
timeout = 900
set {
name = "controller.stats.enabled"
value = true
}
set {
name = "controller.service.type"
value = "LoadBalancer"
}
set {
name = "controller.service.loadBalancerIP"
value = "<MY_STATIC_IP_ADDRESS>"
}
}
在我也通过 Terraform 部署的入口配置下方：
resource "kubernetes_ingress" "ingress" {
wait_for_load_balancer = true
metadata {
name = "app-ingress"
annotations = {
"kubernetes.io/ingress.class": "nginx"
"nginx.ingress.kubernetes.io/rewrite-target": "/"
"kubernetes.io/ingress.global-static-ip-name": <MY_STATIC_IP_ADDRESS>
}
}
spec {
rule {
host = custom.my_domain.com
http {
path {
backend {
service_name = "app-service"
service_port = 5000
}
path = "/app"
}
}
}
}
}
以及从 GCP 获取的结果入口配置：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.global-static-ip-name: static-ip-name
nginx.ingress.kubernetes.io/rewrite-target: /
creationTimestamp: "2021-04-14T20:28:41Z"
generation: 7
name: app-ingress
namespace: default
resourceVersion: HIDDEN
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/app-ingress
uid: HIDDEN
spec:
rules:
- host: custom.my_domain.com
http:
paths:
- backend:
serviceName: app-service
servicePort: 5000
path: /app
status:
loadBalancer:
ingress:
- ip: <MY_STATIC_IP_ADDRESS>
kubectl describe ingress app-ingress 命令的输出：
Name: app-ingress
Namespace: default
Address: <MY_STATIC_IP_ADDRESS>
Default backend: default-http-backend:80 (192.168.10.8:8080)
Rules:
Host Path Backends
---- ---- --------
custom.my_domain.com
/app app-service:5000 (192.168.10.11:5000)
Annotations: kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.global-static-ip-name: static-ip-name
nginx.ingress.kubernetes.io/rewrite-target: /
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 16m (x6 over 32m) nginx-ingress-controller Configuration for default/app-ingress was added or updated
我使用以下配置文件部署了我试图公开的应用程序：
pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: app-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: default
service.yaml
apiVersion: v1
kind: Service
metadata:
name: app-service
spec:
type: NodePort
ports:
- port: 5000
targetPort: 5000
protocol: TCP
name: http
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-deployment
spec:
replicas: 1
template:
spec:
restartPolicy: Always
volumes:
- name: app-pvc
persistentVolumeClaim:
claimName: app-pvc
containers:
- name: app-container
image: "eu.gcr.io/<PROJECT_ID>/<IMAGE_NAME>:VERSION_TAG"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5000
livenessProbe:
tcpSocket:
port: 5000
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
tcpSocket:
port: 5000
initialDelaySeconds: 15
periodSeconds: 20
volumeMounts:
- mountPath: "/data"
name: app-pvc
一切都已成功部署，因为我可以通过 运行 以下命令通过配置的服务在本地直接连接到应用程序：
kubectl port-forward service/app-service 5000:5000
这让我可以在我的浏览器中访问该应用程序，并且一切正常。
为了确保我配置的 <MY_STATIC_IP_ADDRESS> 正确绑定到 custom.my_domain.com，我尝试对主机执行 ping 操作，但确实得到了正确的响应：
ping custom.my_domain.com
Pinging custom.my_domain.com [<MY_STATIC_IP_ADDRESS>] with 32 bytes of data:
Reply from <MY_STATIC_IP_ADDRESS>: bytes=32 time=36ms TTL=113
Reply from <MY_STATIC_IP_ADDRESS>: bytes=32 time=37ms TTL=113
Reply from <MY_STATIC_IP_ADDRESS>: bytes=32 time=36ms TTL=113
Reply from <MY_STATIC_IP_ADDRESS>: bytes=32 time=45ms TTL=113
Ping statistics for <MY_STATIC_IP_ADDRESS>:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 36ms, Maximum = 45ms, Average = 38ms
即使一切似乎都按预期工作，但每当我尝试在浏览器中导航到 custom.my_domain.com/app 时，即使在等待超过 30 米以确保入口配置已在 GCP 上正确注册：
这是显示在我的 nginx-controller pod 日志中的条目：
<HIDDEN_LOCAL_IP> - - [14/Apr/2021:21:18:10 +0000] "GET /app/ HTTP/1.1" 404 232 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0" "-"
更新 #1
看来，如果我更新入口以直接在 / 路径上公开目标服务，它就会按预期工作。在更新的配置下方。尽管如此，如果我尝试设置任何其他路径，它似乎不起作用。
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.global-static-ip-name: static-ip-name
nginx.ingress.kubernetes.io/rewrite-target: /
creationTimestamp: "2021-04-14T20:28:41Z"
generation: 7
name: app-ingress
namespace: default
resourceVersion: HIDDEN
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/app-ingress
uid: HIDDEN
spec:
rules:
- host: custom.my_domain.com
http:
paths:
- backend:
serviceName: app-service
servicePort: 5000
path: /
status:
loadBalancer:
ingress:
- ip: <MY_STATIC_IP_ADDRESS>
更新 #2
看完评论区@jccampanero 分享的资料后，我得到了一个工作配置。
而不是使用 nginx-stable which is referenced on the official nginx website, I used the one here 并相应地更新了我的 Terraform 脚本以使用与我拥有的配置完全相同的脚本。
之后，我不得不按照文档 here 更新我的入口 - 在更新的配置下方：
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.global-static-ip-name: static-ip-name
nginx.ingress.kubernetes.io/rewrite-target: /
creationTimestamp: "2021-04-14T20:28:41Z"
generation: 7
name: app-ingress
namespace: default
resourceVersion: HIDDEN
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/app-ingress
uid: HIDDEN
spec:
rules:
- host: custom.my_domain.com
http:
paths:
- backend:
serviceName: app-service
servicePort: 5000
path: /app(/|$)(.*)
status:
loadBalancer:
ingress:
- ip: <MY_STATIC_IP_ADDRESS>
正如问题评论和问题本身所指出的那样，@vladzam 对此进行了很好的记录，有两个是问题的原因。
一方面，通过 Helm stable 通道可用的 nginx 入口控制器似乎是 deprecated in favor of the new ingress-nginx controller - please, see the Github repo and the official documentation。
另一方面，这似乎是与 Rewrite target 注释的定义有关的问题。根据 docs:
Starting in Version 0.22.0, ingress definitions using the annotation
nginx.ingress.kubernetes.io/rewrite-targetare not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group.
因此，有必要修改入口资源的定义以考虑到这一变化。例如：
$ echo '
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
name: rewrite
namespace: default
spec:
rules:
- host: rewrite.bar.com
http:
paths:
- backend:
serviceName: http-svc
servicePort: 80
path: /something(/|$)(.*)
' | kubectl create -f -
问题本身提供了准确的入口资源定义。