启用 mTLS 时,使用 istio ingress 访问服务会出现 503 错误

Accessing service using istio ingress gives 503 error when mTLS is enabled

我有一个启用了双向 TLS 的 Istio 网格。我的设置如下

  1. pod 内的服务 运行(服务容器 + envoy)
  2. 位于上述服务前面的特使网关。附加到此的 Istio 网关和虚拟服务。它路由 /info/ 路由到上述服务。
  3. 另一个 Istio 网关使用默认的 istio 入口 pod 配置入口。这也有网关+虚拟服务组合。虚拟服务将 /info/ 路径指向 2
  4. 中描述的服务

我正在尝试使用以下 curl 命令从入口网关访问服务:

$ curl -X GET http://istio-ingressgateway.istio-system:80/info/ -H "Authorization: Bearer $token" -v

但我收到如下 503 未找到错误:

$ curl -X GET http://istio-ingressgateway.istio-system:80/info/ -H "Authorization: Bearer $token" -v
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 10.105.138.94...
* Connected to istio-ingressgateway.istio-system (10.105.138.94) port 80 (#0)
> GET /info/ HTTP/1.1
> Host: istio-ingressgateway.istio-system
> User-Agent: curl/7.47.0
> Accept: */*
> Authorization: Bearer ...
>
< HTTP/1.1 503 Service Unavailable
< content-length: 57
< content-type: text/plain
< date: Sat, 12 Jan 2019 13:30:13 GMT
< server: envoy
<
* Connection #0 to host istio-ingressgateway.istio-system left intact

我检查了 istio-ingressgateway pod 的日志,下面的行记录在那里

[2019-01-13T05:40:16.517Z] "GET /info/ HTTP/1.1" 503 UH 0 19 6 - "10.244.0.5" "curl/7.47.0" "da02fdce-8bb5-90fe-b422-5c74fe28759b" "istio-ingressgateway.istio-system" "-"

如果我登录到 istio ingress pod 并尝试使用 curl 发送请求,我会收到成功的 200 OK。

# curl hr--gateway-service.default/info/ -H "Authorization: Bearer $token" -v

此外,当在 mTLS 禁用模式下创建网格时,我成功地获得了对同一 curl 命令的响应。 mTLS 设置中没有显示冲突。

这是我的服务网格的配置详细信息,以备不时之需。

Pods

$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                                       READY     STATUS      RESTARTS   AGE
default         hr--gateway-deployment-688986c87c-z9nkh                    1/1       Running     0          37m
default         hr--hr-deployment-596946948d-c89bn                         2/2       Running     0          37m
default         hr--sts-deployment-694d7cff97-gjwdk                        1/1       Running     0          37m
ingress-nginx   default-http-backend-6586bc58b6-8qss6                      1/1       Running     0          42m
ingress-nginx   nginx-ingress-controller-6bd7c597cb-t4rwq                  1/1       Running     0          42m
istio-system    grafana-85dbf49c94-lfpbr                                   1/1       Running     0          42m
istio-system    istio-citadel-545f49c58b-dq5lq                             1/1       Running     0          42m
istio-system    istio-cleanup-secrets-bh5ws                                0/1       Completed   0          42m
istio-system    istio-egressgateway-7d59954f4-qcnxm                        1/1       Running     0          42m
istio-system    istio-galley-5b6449c48f-72vkb                              1/1       Running     0          42m
istio-system    istio-grafana-post-install-lwmsf                           0/1       Completed   0          42m
istio-system    istio-ingressgateway-8455c8c6f7-5khtk                      1/1       Running     0          42m
istio-system    istio-pilot-58ff4d6647-bct4b                               2/2       Running     0          42m
istio-system    istio-policy-59685fd869-h7v94                              2/2       Running     0          42m
istio-system    istio-security-post-install-cqj6k                          0/1       Completed   0          42m
istio-system    istio-sidecar-injector-75b9866679-qg88s                    1/1       Running     0          42m
istio-system    istio-statsd-prom-bridge-549d687fd9-bspj2                  1/1       Running     0          42m
istio-system    istio-telemetry-6ccf9ddb96-hxnwv                           2/2       Running     0          42m
istio-system    istio-tracing-7596597bd7-m5pk8                             1/1       Running     0          42m
istio-system    prometheus-6ffc56584f-4cm5v                                1/1       Running     0          42m
istio-system    servicegraph-5d64b457b4-jttl9                              1/1       Running     0          42m
kube-system     coredns-78fcdf6894-rxw57                                   1/1       Running     0          50m
kube-system     coredns-78fcdf6894-s4bg2                                   1/1       Running     0          50m
kube-system     etcd-ubuntu                                                1/1       Running     0          49m
kube-system     kube-apiserver-ubuntu                                      1/1       Running     0          49m
kube-system     kube-controller-manager-ubuntu                             1/1       Running     0          49m
kube-system     kube-flannel-ds-9nvf9                                      1/1       Running     0          49m
kube-system     kube-proxy-r868m                                           1/1       Running     0          50m
kube-system     kube-scheduler-ubuntu                                      1/1       Running     0          49m

服务

$ kubectl get svc --all-namespaces
NAMESPACE       NAME                                    TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                                                                                                   AGE

default         hr--gateway-service                     ClusterIP      10.100.238.144   <none>         80/TCP,443/TCP                                                                                                            39m
default         hr--hr-service                          ClusterIP      10.96.193.43     <none>         80/TCP                                                                                                                    39m
default         hr--sts-service                         ClusterIP      10.99.54.137     <none>         8080/TCP,8081/TCP,8090/TCP                                                                                                39m
default         kubernetes                              ClusterIP      10.96.0.1        <none>         443/TCP                                                                                                                   52m
ingress-nginx   default-http-backend                    ClusterIP      10.109.166.229   <none>         80/TCP                                                                                                                    44m
ingress-nginx   ingress-nginx                           NodePort       10.108.9.180     192.168.60.3   80:31001/TCP,443:32315/TCP                                                                                                44m
istio-system    grafana                                 ClusterIP      10.102.141.231   <none>         3000/TCP                                                                                                                  44m
istio-system    istio-citadel                           ClusterIP      10.101.128.187   <none>         8060/TCP,9093/TCP                                                                                                         44m
istio-system    istio-egressgateway                     ClusterIP      10.102.157.204   <none>         80/TCP,443/TCP                                                                                                            44m
istio-system    istio-galley                            ClusterIP      10.96.31.251     <none>         443/TCP,9093/TCP                                                                                                          44m
istio-system    istio-ingressgateway                    LoadBalancer   10.105.138.94    <pending>      80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:31219/TCP,8060:31482/TCP,853:30034/TCP,15030:31544/TCP,15031:32652/TCP   44m
istio-system    istio-pilot                             ClusterIP      10.100.170.73    <none>         15010/TCP,15011/TCP,8080/TCP,9093/TCP                                                                                     44m
istio-system    istio-policy                            ClusterIP      10.104.77.184    <none>         9091/TCP,15004/TCP,9093/TCP                                                                                               44m
istio-system    istio-sidecar-injector                  ClusterIP      10.100.180.152   <none>         443/TCP                                                                                                                   44m
istio-system    istio-statsd-prom-bridge                ClusterIP      10.107.39.50     <none>         9102/TCP,9125/UDP                                                                                                         44m
istio-system    istio-telemetry                         ClusterIP      10.110.55.232    <none>         9091/TCP,15004/TCP,9093/TCP,42422/TCP                                                                                     44m
istio-system    jaeger-agent                            ClusterIP      None             <none>         5775/UDP,6831/UDP,6832/UDP                                                                                                44m
istio-system    jaeger-collector                        ClusterIP      10.102.43.21     <none>         14267/TCP,14268/TCP                                                                                                       44m
istio-system    jaeger-query                            ClusterIP      10.104.182.189   <none>         16686/TCP                                                                                                                 44m
istio-system    prometheus                              ClusterIP      10.100.0.70      <none>         9090/TCP                                                                                                                  44m
istio-system    servicegraph                            ClusterIP      10.97.65.37      <none>         8088/TCP                                                                                                                  44m
istio-system    tracing                                 ClusterIP      10.109.87.118    <none>         80/TCP                                                                                                                    44m
kube-system     kube-dns                                ClusterIP      10.96.0.10       <none>         53/UDP,53/TCP                                                                                                             52m

第 2 点中描述的网关和虚拟服务

$ kubectl describe gateways.networking.istio.io hr--gateway
Name:         hr--gateway
Namespace:    default
API Version:  networking.istio.io/v1alpha3
Kind:         Gateway
Metadata:
  ...
Spec:
  Selector:
    App:                           hr--gateway
  Servers:
    Hosts:
      *
    Port:
      Name:      http2
      Number:    80
      Protocol:  HTTP2
    Hosts:
      *
    Port:
      Name:      https
      Number:    443
      Protocol:  HTTPS
    Tls:
      Mode:  PASSTHROUGH

$ kubectl describe virtualservices.networking.istio.io hr--gateway
Name:         hr--gateway
Namespace:    default
Labels:       app=hr--gateway
              Annotations:  <none>
API Version:  networking.istio.io/v1alpha3
Kind:         VirtualService
Metadata:
  ...
Spec:
  Gateways:
    hr--gateway
  Hosts:
    *
  Http:
    Match:
      Uri:
        Prefix:  /info/
    Rewrite:
      Uri:  /
    Route:
      Destination:
        Host:  hr--hr-service

第 3 点中描述的网关和虚拟服务

$ kubectl describe gateways.networking.istio.io ingress-gateway
Name:         ingress-gateway
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"ingress-gateway","namespace":"default"},"spec":{"sel...
API Version:  networking.istio.io/v1alpha3
Kind:         Gateway
Metadata:
  ...
Spec:
  Selector:
    Istio:  ingressgateway
  Servers:
    Hosts:
      *
    Port:
      Name:      http2
      Number:    80
      Protocol:  HTTP2

$ kubectl describe virtualservices.networking.istio.io hr--gateway-ingress-vs
Name:         hr--gateway-ingress-vs
Namespace:    default
Labels:       app=hr--gateway
              API Version:  networking.istio.io/v1alpha3
Kind:         VirtualService
Metadata:
Spec:
  Gateways:
    ingress-gateway
  Hosts:
    *
  Http:
    Match:
      Uri:
        Prefix:  /info/
    Route:
      Destination:
        Host:  hr--gateway-service
Events:        <none>

问题大概如下:istio-ingressgateway在80端口发起mTLS到hr--gateway-service,但是hr--gateway-service 需要纯 HTTP 连接。

有多种解决方案:

  1. 定义 DestinationRule 以指示客户端在调用 hr--gateway-service
  2. 时禁用 mTLS
   apiVersion: networking.istio.io/v1alpha3
   kind: DestinationRule
   metadata:
     name: hr--gateway-service-disable-mtls
   spec:
     host: hr--gateway-service.default.svc.cluster.local
     trafficPolicy:
       tls:
         mode: DISABLE
  1. 指示 hr-gateway-service 接受 mTLS 连接。为此,将端口 80 上的 server TLS options 配置为 MUTUAL 并使用 Istio 证书和私钥。将serverCertificatecaCertificatesprivateKey分别指定为/etc/certs/cert-chain.pem/etc/certs/root-cert.pem/etc/certs/key.pem