Nginx pod 在自签名证书示例中使用其侦听端口进行响应

Nginx pod responds with its listening port in self-signed certificate examples

环境:

❯ sw_vers
ProductName:    macOS
ProductVersion: 11.6.1
BuildVersion:   20G224

❯ minikube version
minikube version: v1.24.0
commit: 76b94fb3c4e8ac5062daf70d60cf03ddcc0a741b

我在 NGINX pod 上制作了一个自签名证书示例。省略创建证书和密钥,因为它们在我的本地 mac 上工作,文件如下:

❯ ll rootCA.*
-rw-r--r--@ 1 hansuk  staff  1383  1 17 12:37 rootCA.crt
-rw-------  1 hansuk  staff  1874  1 17 12:02 rootCA.key

❯ ll localhost.*
-rw-------  1 hansuk  staff  1704  1 17 12:09 localhost.key
-rw-r--r--  1 hansuk  staff  1383  1 17 12:37 localhost.pem

在 minikube(kubectl apply -f nginx.yml -n cert) 上启动以下 kubernetes 定义:

apiVersion: v1
kind: Service
metadata:
  name: nginx-cert
  labels:
    app: nginx-cert
spec:
  type: NodePort
  ports:
  - port: 80
    protocol: TCP
    name: http
    nodePort: 30080
  - port: 443
    protocol: TCP
    name: https
    nodePort: 30443
  selector:
    app: nginx-cert
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: nginx-cert
  name: nginx-cert
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-cert
  template:
    metadata:
      labels:
        app: nginx-cert
    spec:
      volumes:
      - name: secret-volume
        secret:
          secretName: nginxsecret
      - name: configmap-volume
        configMap:
          name: nginxconfmap
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
        - containerPort: 443
        volumeMounts:
        - mountPath: /etc/nginx/ssl
          name: secret-volume
        - mountPath: /etc/nginx/conf.d
          name: configmap-volume

分别为 nginx config 和 TLS 路径创建 configmap 和 secret:

❯ cat default.conf 
server {
        listen 80 default_server;
        listen [::]:80 default_server ipv6only=on;

        listen 443 ssl;

        root /usr/share/nginx/html;
        index index.html;

        server_name locahost;
        ssl_certificate /etc/nginx/ssl/tls.crt;
        ssl_certificate_key /etc/nginx/ssl/tls.key;

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        location / {
                try_files  / =404;
        }
}

❯ kubectl create configmap nginxconfmap --from-file=default.conf -n cert

❯ kubectl create secret tls nginxsecret --key localhost.key --cert localhost.pem -n cert

部署和服务的所有状态以及事件日志都正常。无失败:

❯ kubectl get all -n cert
NAME                              READY   STATUS    RESTARTS   AGE
pod/nginx-cert-76f7f8748f-q2nvl   1/1     Running   0          21m

NAME                 TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/nginx-cert   NodePort   10.110.115.36   <none>        80:30080/TCP,443:30443/TCP   21m

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-cert   1/1     1            1           21m

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-cert-76f7f8748f   1         1         1       21m

❯ kubectl get events -n cert
22m         Normal    Scheduled           pod/nginx-cert-76f7f8748f-q2nvl    Successfully assigned cert/nginx-cert-76f7f8748f-q2nvl to minikube
22m         Normal    Pulling             pod/nginx-cert-76f7f8748f-q2nvl    Pulling image "nginx"
22m         Normal    Pulled              pod/nginx-cert-76f7f8748f-q2nvl    Successfully pulled image "nginx" in 4.345505365s
22m         Normal    Created             pod/nginx-cert-76f7f8748f-q2nvl    Created container nginx
22m         Normal    Started             pod/nginx-cert-76f7f8748f-q2nvl    Started container nginx
22m         Normal    SuccessfulCreate    replicaset/nginx-cert-76f7f8748f   Created pod: nginx-cert-76f7f8748f-q2nvl
22m         Normal    ScalingReplicaSet   deployment/nginx-cert              Scaled up replica set nginx-cert-76f7f8748f to 

然后,SSL 握手正在使用 minukube 服务 IP:

❯ minikube service --url nginx-cert --namespace cert
http://192.168.64.2:30080
http://192.168.64.2:30443

❯ openssl s_client -CAfile rootCA.crt -connect 192.168.64.2:30443 -showcerts 2>/dev/null < /dev/null 
CONNECTED(00000003)
---
Certificate chain
 0 s:C = KR, ST = Seoul, L = Seocho-gu, O = Localhost, CN = localhost
   i:C = KR, ST = RootState, L = RootCity, O = Root Inc., OU = Root CA, CN = Self-signed Root CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 17 03:37:15 2022 GMT; NotAfter: Jan 17 03:37:15 2023 GMT
-----BEGIN CERTIFICATE-----
MIIDzzCCAregAwIBAgIUYMe6nRgsZwq9UPMKFgj9dt9z9FIwDQYJKoZIhvcNAQEL
BQAweDELMAkGA1UEBhMCS1IxEjAQBgNVBAgMCVJvb3RTdGF0ZTERMA8GA1UEBwwI
Um9vdENpdHkxEjAQBgNVBAoMCVJvb3QgSW5jLjEQMA4GA1UECwwHUm9vdCBDQTEc
MBoGA1UEAwwTU2VsZi1zaWduZWQgUm9vdCBDQTAeFw0yMjAxMTcwMzM3MTVaFw0y
MzAxMTcwMzM3MTVaMFkxCzAJBgNVBAYTAktSMQ4wDAYDVQQIDAVTZW91bDESMBAG
A1UEBwwJU2VvY2hvLWd1MRIwEAYDVQQKDAlMb2NhbGhvc3QxEjAQBgNVBAMMCWxv
Y2FsaG9zdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALc9retjBorw
RKbuyC1SNx1U9L5LJPPbBkBh4kg98saQxtRX0Wqs5mgswWMZYL3E6yRl0gfwBkdq
t8GVQ49dgg0QO5MbG9ylfCLS9xR3WWjAgxaDJ0W96PyvTzmg295aqqHFKPSaG/nM
JyZgFJDuGoRRgwoWNqZ1pRCDLMIENDx4qgjOnQch529pM9ZRwFQSswKpn4BVkY00
/u8jIvax67kFOg70QGY16paGEg7YfSNle7BFZY0VJ8rIiBoqwRmPH6hbF/djxe5b
yzkI9eqts9bqw8eDLC28S36x62FxkdqkK8pI/rzWAKSV43TWML1zq4vM2bI+vp0k
a06GhSsS1bUCAwEAAaNwMG4wHwYDVR0jBBgwFoAUURHNpOE9zTgXgVYAGvLt94Ym
P+8wCQYDVR0TBAIwADALBgNVHQ8EBAMCBPAwFAYDVR0RBA0wC4IJbG9jYWxob3N0
MB0GA1UdDgQWBBSS1ZHHT6OHTomYIRsmhz6hMJLGnDANBgkqhkiG9w0BAQsFAAOC
AQEAWA23pCdAXtAbdSRy/p8XURCjUDdhkp3MYA+1gIDeGAQBKNipU/KEo5wO+aVk
AG6FryPZLOiwiP8nYAebUxOAqKG3fNbgT9t95BEGCip7Cxjp96KNYt73Kl/OTPjJ
KZUkHQ7MXN4vc5gmca8q+OqwCCQ/daMkzLabPQWNk3R/Hzo/mT42v8ht9/nVh1Ml
u3Dow5QPp8LESrJABLIRyRs0+Tfp+WodgekgDX5hnkkSk77+oXB49r2tZUeG/CVv
Fg8PuUNi+DWpdxX8fE/gIbSzSsamOf29+0sCIoJEPvk7lEVLt9ca0SoJ7rKn/ai4
HxwTiYo9pNcoLwhH3xdXjvbuGA==
-----END CERTIFICATE-----
---
Server certificate
subject=C = KR, ST = Seoul, L = Seocho-gu, O = Localhost, CN = localhost
issuer=C = KR, ST = RootState, L = RootCity, O = Root Inc., OU = Root CA, CN = Self-signed Root CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1620 bytes and written 390 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: EED06A09B8971ADD25F352BF55298096581A490020C88BB457AB9864B9844778
    Session-ID-ctx: 
    Master-Key: 71C686180017B4DB5D681CCFC2C8741A7A70F7364572811AE548556A1DCAC078ABAF34B9F53885C6177C7024991B98FF
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 8b 7f 76 5a c3 4a 1f 40-43 8e 00 e7 ad 35 ae 24   ..vZ.J.@C....5.$
    0010 - 5c 63 0b 0c 91 86 d0 74-ef 39 94 8a 07 fa 96 51   \c.....t.9.....Q
    0020 - 58 cd 61 99 7d ae 47 87-7b 36 c1 22 89 fa 8e ca   X.a.}.G.{6."....
    0030 - 52 c2 04 6e 7a 9f 2d 3e-42 25 fc 1f 87 11 5f 02   R..nz.->B%...._.
    0040 - 37 b3 26 d4 1f 10 97 a3-29 e8 d1 37 cd 9a a3 8e   7.&.....)..7....
    0050 - 61 52 15 63 89 99 8e a8-95 58 a8 e0 12 03 c4 15   aR.c.....X......
    0060 - 95 bf 1e b7 48 dc 4e fb-c4 8c 1a 17 eb 19 88 ca   ....H.N.........
    0070 - eb 16 b0 17 83 97 04 0d-79 ca d9 7d 80 5b 96 8d   ........y..}.[..
    0080 - d3 bf 6f 4f 55 6d 2f ce-0b b9 24 a9 a2 d0 5b 28   ..oOUm/...$...[(
    0090 - 06 10 1d 72 52 a3 ef f1-5c e3 2a 35 83 93 a1 91   ...rR...\.*5....
    00a0 - cb 94 6c 4f 3e f7 2e 8d-87 76 a5 46 29 6f 0e 5f   ..lO>....v.F)o._

    Start Time: 1643011123
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

但是它无法在 Chrome 浏览器或 curl 上连接,重定向到它的监听端口 each(30080 -> 80, 30443 -> 443):

# for convenience ignore root CA now, the problem is not in there.
❯ curl -k https://192.168.64.2:30443
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.21.5</center>
</body>
</html>

❯ curl -kL https://192.168.64.2:30443
curl: (7) Failed to connect to 192.168.64.2 port 443: Connection refused

❯ curl  http://192.168.64.2:30080
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.21.5</center>
</body>
</html>

❯ curl -L http://192.168.64.2:30080
curl: (7) Failed to connect to 192.168.64.2 port 80: Connection refused

❯ kubectl logs nginx-cert-76f7f8748f-q2nvl  -n cert
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/01/24 07:33:25 [notice] 1#1: using the "epoll" event method
2022/01/24 07:33:25 [notice] 1#1: nginx/1.21.5
2022/01/24 07:33:25 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
2022/01/24 07:33:25 [notice] 1#1: OS: Linux 4.19.202
2022/01/24 07:33:25 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/01/24 07:33:25 [notice] 1#1: start worker processes
2022/01/24 07:33:25 [notice] 1#1: start worker process 24
2022/01/24 07:33:25 [notice] 1#1: start worker process 25
172.17.0.1 - - [24/Jan/2022:07:44:36 +0000] "\x16\x03\x01\x01$\x01\x00\x01 \x03\x03rM&\xF2\xDD\xA3\x04(\xB0\xB2\xBF\x1CTS`\xDC\x90\x86\xF1\xEC\xBD9\x9Cz1c4\x0B\x8F\x13\xC2" 400 157 "-" "-"
172.17.0.1 - - [24/Jan/2022:07:44:48 +0000] "\x16\x03\x01\x01$\x01\x00\x01 \x03\x03'Y\xECP\x15\xD1\xE6\x1C\xC4\xB1v\xC1\x97\xEE\x04\xEBu\xDE\xF9\x04\x95\xC2V\x14\xB5\x7F\x91\x86V\x8F\x05\x83 \xBFtL\xDB\xF6\xC2\xD8\xD4\x1E]\xAE4\xCA\x03xw\x92D&\x1E\x8D\x97c\xB3,\xFD\xCD\xF47\xC4:\xF8\x00>\x13\x02\x13\x03\x13\x01\xC0,\xC00\x00\x9F\xCC\xA9\xCC\xA8\xCC\xAA\xC0+\xC0/\x00\x9E\xC0$\xC0(\x00k\xC0#\xC0'\x00g\xC0" 400 157 "-" "-"
172.17.0.1 - - [24/Jan/2022:07:45:05 +0000] "\x16\x03\x01\x01$\x01\x00\x01 \x03\x03;J\xA7\xD0\xC2\xC3\x1A\xF9LK\xC7\xA8l\xBD>*\x80A$\xA4\xFCw\x19\xE7(\xFAGc\xF6]\xF3I \xFF\x83\x84I\xC2\x8D\xD5}\xEA\x95\x8F\xDB\x8Cfq\xC6\xBA\xCF\xDDyn\xC6v\xBA\xCC\xDC\xCC\xCC/\xAF\xBC\xB2\x00>\x13\x02\x13\x03\x13\x01\xC0,\xC00\x00\x9F\xCC\xA9\xCC\xA8\xCC\xAA\xC0+\xC0/\x00\x9E\xC0$\xC0(\x00k\xC0#\xC0'\x00g\xC0" 400 157 "-" "-"
172.17.0.1 - - [24/Jan/2022:07:49:08 +0000] "GET / HTTP/1.1" 301 169 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"
172.17.0.1 - - [24/Jan/2022:07:49:08 +0000] "GET / HTTP/1.1" 301 169 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"
172.17.0.1 - - [24/Jan/2022:08:00:24 +0000] "GET / HTTP/1.1" 400 255 "-" "curl/7.64.1"
172.17.0.1 - - [24/Jan/2022:08:01:46 +0000] "GET / HTTP/1.1" 301 169 "-" "curl/7.64.1"
172.17.0.1 - - [24/Jan/2022:08:01:50 +0000] "GET / HTTP/1.1" 301 169 "-" "curl/7.64.1"
172.17.0.1 - - [24/Jan/2022:08:03:04 +0000] "GET / HTTP/1.1" 301 169 "-" "curl/7.64.1"
172.17.0.1 - - [24/Jan/2022:08:03:07 +0000] "GET / HTTP/1.1" 301 169 "-" "curl/7.64.1"

实际上,起初,pod 响应请求的端口 30080 和 30443,但现在它响应 80 和 443。我不知道它什么时候变了,我确实变了。

我已将 nginx 配置上的 server_namelocalhost 更改为 192.168.64.2 但这没关系。

我在 Linux 上完全重新创建了您的 minikube 配置。您的 Kubernetes 配置没问题。我得到了相同的回复 - 301 Moved Permanently.

之后,我在 default.conf 文件中更改了这些行:

location / {
                    try_files $uri $uri/ =404;
            }

现在一切正常(使用 curl 和浏览器可以访问 pod 中的 nginx 网页)。