使用 Istio 在 Kubernetes 上出现 503 UH 错误，但服务正常

Question

我在安装了 Istio 的 Kubernetes(minikube) 上正确配置我的服务之间的通信时遇到问题。

我正在尝试从我的服务向 elasticsearch 发送 POST 请求，但我一直收到：

POST /_bulk?timeout=1m HTTP/1.1" 503 UH "-" "-" 0 19 0 - "-" "Apache-HttpAsyncClient/4.1.4 (Java/11.0.9.1)" "1a290357-7b18-9692-9392-d0298ed3276c" "elasticsearch:9200" "-" - - 10.102.10.19:9200 172.18.0.12:39194 - default

Istioctl analyze 没有显示任何问题。我也禁用了 mtls。

你知道哪里出了问题吗？不明白为什么会出现UH（unhealthy），因为elasticsearch可以运行，而且Kiali dashboard也显示为healthy

我的部署 + 服务：

弹性搜索

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  labels:
    app: elasticsearch
    tier: database
spec:
  selector:
    app: elasticsearch
  ports:
  - name: "http-9200"
    port: 9200
    targetPort: 9200
  - name: "tcp-9300"
    port: 9300
    targetPort: 9300
  selector:
    app: elasticsearch
    tier: database

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  labels:
    service: elasticsearch
spec:
  serviceName: elasticsearch
  replicas: 1
  selector:
    matchLabels:
      service: elasticsearch
  template:
    metadata:
      labels:
        service: elasticsearch
    spec:
      terminationGracePeriodSeconds: 300
      initContainers:
      - name: fix-the-volume-permission
        image: busybox
        command:
        - sh
        - -c
        - chown -R 1000:1000 /usr/share/elasticsearch/data
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-the-vm-max-map-count
        image: busybox
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      - name: increase-the-ulimit
        image: busybox
        command:
        - sh
        - -c
        - ulimit -n 65536
        securityContext:
          privileged: true
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.4
        ports:
        - containerPort: 9200
          name: "http-9200"
        - containerPort: 9300
          name: "tcp-9300"
        env:
          - name: cluster.name
            value: elasticsearch-cluster
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: ES_JAVA_OPTS
            value: -Xms4g -Xmx4g
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
  volumeClaimTemplates:
  - metadata:
      name: data
      annotations:
        volume.beta.kubernetes.io/storage-class: "standard"
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

我的服务

apiVersion: v1
kind: Service
metadata:
  name: scrappers-service
  labels:
    name: scrappers-service
spec:
  ports:
  - nodePort: 30164
    name: "http-8080"
    port: 8080
    targetPort: 8080
  selector:
    app: scrappers-service
  type: NodePort
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: scrappers-service
  labels:
    name: scrappers-service
spec:
  selector:
    matchLabels:
      app: scrappers-service
  replicas: 1
  template:
    metadata:
      labels:
        app: scrappers-service
    spec:
      containers:
      - image: example/scrappers:master
        imagePullPolicy: Never
        name: scrappers-service
        ports: 
        - containerPort: 8080

Answer 1

如前所述

I decided to use the solution described by elasticsearch. I mean elasticseach-operator. I applied all steps and it just works without any bigger problems.

所以解决方案是遵循 elasticsearch documentation，它使用下面的注释使其工作。

annotations:
  traffic.sidecar.istio.io/excludeOutboundPorts: "" 
  traffic.sidecar.istio.io/excludeInboundPorts: ""

To make the validating webhook work under Istio, you need to exclude the inbound port 9443 from being proxied. This can be done by editing the template definition of the elastic-operator StatefulSet to add the following annotations to the operator Pod:

[...]
spec:
  template:
    metadata:
      annotations:
        traffic.sidecar.istio.io/excludeInboundPorts: "9443"
        traffic.sidecar.istio.io/includeInboundPorts: '*'
[...]

If you have configured Istio in permissive mode, examples defined elsewhere in the ECK documentation will continue to work without requiring any modifications. However, if you have enabled strict mutual TLS authentication between services either via global (MeshPolicy) or namespace-level (Policy) configuration, the following modifications to the resource manifests are necessary for correct operation.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic-istio
spec:
  version: 7.10.0
  http:
    tls: 
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - name: default
    count: 3
    podTemplate:
      metadata:
        annotations:
          traffic.sidecar.istio.io/includeInboundPorts: "*"
          traffic.sidecar.istio.io/excludeOutboundPorts: "9300" 
          traffic.sidecar.istio.io/excludeInboundPorts: "9300"
      spec:
        automountServiceAccountToken: true

If you do not have automatic mutual TLS enabled, you may need to create a Destination Rule to allow the operator to communicate with the Elasticsearch cluster. A communication issue between the operator and the managed Elasticsearch cluster can be detected by looking at the operator logs to see if there are any errors reported with the text 503 Service Unavailable.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: elastic-istio
spec:
  host: "elastic-istio-es-http.default.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

存在相关的 github 个问题：

使用 Istio 在 Kubernetes 上出现 503 UH 错误，但服务正常

503 UH error on Kubernetes with Istio, but service is working

kubernetes

istio