Kubernetes 中的 Nginx 控制器：与上游的握手 - SSL 握手中的对等关闭连接

Question

在使用 kubeadm 构建的本地环境中。集群由主节点和 2 个工作节点组成。

它失败了什么：

通过服务类型 LoadBalancer 将 Nginx-ingress 控制器暴露给外部，尝试 TLS 终止到 Kubernetes 集群。

此处公开的服务：

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      
nginx-ingress   LoadBalancer   10.101.5.75     192.168.1.82   80:30745/TCP,443:30092/TCP   
web-service     ClusterIP      10.101.26.176   <none>         8080/TCP

工作原理：

能够从外部通过 HTTP 端口 80 访问 Web 应用程序。

这里是集群中的pods：

NAME                              READY   STATUS    RESTARTS   AGE   IP          NODE    
nginx-ingress-7c5588544d-4mw9d    1/1     Running   0          38m   10.44.0.2   node1   
web-deployment-7d84778bc6-52pq7   1/1     Running   1          19h   10.44.0.1   node1   
web-deployment-7d84778bc6-9wwmn   1/1     Running   1          19h   10.36.0.2   node2

TLS 终止测试结果

客户端：

curl -k https://example.com -v
* Rebuilt URL to https://example.com/
*   Trying 192.168.1.82...
* Connected to example.com (192.168.1.82) port 443 (#0)
* found 127 certificates in /etc/ssl/certs/ca-certificates.crt
* found 513 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*    server certificate verification SKIPPED
*    server certificate status verification SKIPPED
*    common name: example.com (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #1
*    subject: CN=example.com
*    start date: Thu, 22 Oct 2020 16:33:49 GMT
*    expire date: Fri, 22 Oct 2021 16:33:49 GMT
*    issuer: CN=My Cert Authority
*    compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 502 Bad Gateway
< Server: nginx/1.19.3

服务器端（来自 nginx-ingress pod 的日志）：

GET / HTTP/1.1
Connection: close
Host: example.com
X-Real-IP: 10.32.0.1
X-Forwarded-For: 10.32.0.1
X-Forwarded-Host: example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
User-Agent: curl/7.47.0
Accept: */*

2020/10/22 18:38:59 [error] 23#23: *12 peer closed connection in SSL handshake while SSL handshaking  to upstream, client: 10.32.0.1, server: example.com, request: "GET / HTTP/1.1", upstream: "https://10.44.0.1:8081/", host: "example.com"

2020/10/22 18:38:59 [warn] 23#23: *12 upstream server temporarily disabled while SSL handshaking to upstream, client: 10.32.0.1, server: example.com, request: "GET / HTTP/1.1", upstream: "https://10.44.0.1:8081/", host: "example.com"

HTTP/1.1 502 Bad Gateway
Server: nginx/1.19.3

我检查了什么

在 link 之后生成了 CA 和服务器证书： https://kubernetes.github.io/ingress-nginx/examples/PREREQUISITES/#tls-certificates

检查了 nginx-ingress Pod /etc/nginx/secrets/default 下的服务器证书和服务器密钥，它们看起来是正确的。这里是资源 VirtualServer 的输出：

NAME         STATE   HOST          IP             PORTS      AGE
vs-example   Valid   example.com   192.168.1.82   [80,443]   121m

虚拟服务器:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: vs-example
  namespace: nginx-ingress
spec:
  host: example.com
  tls:
    secret: example-tls
  upstreams:
  - name: example
    service: web-service
    port: 8080
    tls:
      enable: true
  routes:
  - path: /v2
    action:
      pass: example
  routes:
  - path: /
    action:
      pass: example

秘密

apiVersion: v1
kind: Secret
metadata:
  name: example-tls
  namespace: nginx-ingress
data:
  tls.crt: (..omitted..)
  tls.key: (..omitted..)
type: kubernetes.io/tls

Nginx.conf

这里是从运行连播中摘录的 nginx.conf：

server {
    # required to support the Websocket protocol in VirtualServer/VirtualServerRoutes
    set $default_connection_header "";
    listen 80 default_server;
    listen 443 ssl default_server;
    ssl_certificate /etc/nginx/secrets/default;
    ssl_certificate_key /etc/nginx/secrets/default;
    server_name _;
    server_tokens "on";

尝试使用 HTTPS 访问 Web 应用程序时找不到正在发生的事情。

Answer 1

VirtualServer 中的几个错误，通过这些更改进行编辑解决了问题：

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: vs-example
  namespace: nginx-ingress
spec:
  host: example.com
  tls:
    secret: example-tls

  upstreams:
  - name: example
    service: web-service
    port: 8080
#    tls:            --> this caused the SSL issue in the upstream
#      enable: true

  routes:
  - path: /
    action:
      pass: example
  - path: /v1
    action:
      pass: example

现在可以使用 HTTPS 访问 Web 应用程序

Kubernetes 中的 Nginx 控制器：与上游的握手 - SSL 握手中的对等关闭连接

Nginx Controller in Kubernetes: Handshaking to upstream - peer closed connection in SSL handshake

kubernetes

nginx-config

nginx-ingress