kuberntes init 容器中的 iptables 不起作用

Question

背景：

我正在尝试使用 goreplay 将流量镜像到其他目的地。我发现k8s服务是4层的负载均衡导致流量无法被goreplay抓取，所以我决定像istio一样在pod里面加一个reverse-proxy sidecar

这是我的 pod yaml：

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx
    ports:
    - containerPort: 80
      name: http
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: proxy
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 10m
        memory: 40Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File  
    volumeMounts:
      - mountPath: /etc/nginx/conf.d
        name: default   
  initContainers:
    - command: 
      - iptables
      args:
      - -t 
      - nat
      - -A
      - PREROUTING
      - -p
      - tcp
      - --dport
      - "80"
      - -j
      - REDIRECT
      - --to-ports
      - "15001"
      image: soarinferret/iptablesproxy
      imagePullPolicy: IfNotPresent
      name: istio-init
      resources:
        limits:
          cpu: 100m
          memory: 50Mi
        requests:
          cpu: 10m
          memory: 10Mi
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          add:
          - NET_ADMIN
          - NET_RAW
          drop:
          - ALL
        privileged: false
        readOnlyRootFilesystem: false
        runAsGroup: 0
        runAsNonRoot: false
        runAsUser: 0
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File        
  dnsPolicy: ClusterFirst
  terminationGracePeriodSeconds: 30
  volumes:
    - configMap:
        defaultMode: 256
        name: default
        optional: false
      name: default
---

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

--- 

apiVersion: v1
data:
  default.conf: |
    server {
        listen       15001;
        server_name  localhost;
        access_log  /var/log/nginx/host.access.log  main;
        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }
kind: ConfigMap
metadata:
  name: default
  namespace: default

我使用 kubectl port-forward service/nginx 8080:80 然后 curl http://localhost:8080，流量直接发送到 nginx 而不是我的代理。

我想要什么：

一种让goreplay抓取k8s服务负载均衡流量的方法
更正 iptables 规则，让流量成功路由到我的代理 sideCar。

感谢您的帮助！

Answer 1

正如@Jonyhy96 在评论中提到的，这里唯一需要更改的是 securityContext field of initContainer.

中的特权值设置为 true

Privileged - determines if any container in a pod can enable privileged mode. By default a container is not allowed to access any devices on the host, but a "privileged" container is given access to all devices on the host. This allows the container nearly all the same access as processes running on the host. This is useful for containers that want to use linux capabilities like manipulating the network stack and accessing devices.

所以 initContainer 看起来像这样

initContainers:
    - command: 
      - iptables
      args:
      - -t 
      - nat
      - -A
      - PREROUTING
      - -p
      - tcp
      - --dport
      - "80"
      - -j
      - REDIRECT
      - --to-ports
      - "15001"
      image: soarinferret/iptablesproxy
      imagePullPolicy: IfNotPresent
      name: istio-init
      resources:
        limits:
          cpu: 100m
          memory: 50Mi
        requests:
          cpu: 10m
          memory: 10Mi
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          add:
          - NET_ADMIN
          - NET_RAW
          drop:
          - ALL
        privileged: true   <---- changed from false
        readOnlyRootFilesystem: false
        runAsGroup: 0
        runAsNonRoot: false
        runAsUser: 0
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File

这方面tutorial非常好，不完全在 nginx 上，但解释了如何实际构建代理。

Answer 2

除了需要更改为 allowPrivilegeEscalation: true

以下精简版本也适用于 GKE (Google Kubernetes Engine)：

    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        drop:
        - ALL
      privileged: true

kuberntes init 容器中的 iptables 不起作用

iptables in kuberntes init container does't work

reverse-proxy

iptables

kubernetes

istio