如何使用nodeport服务类型在k8s环境下搭建redis集群？

Question

我尝试使用“NodePort”类型的服务在 k8s 环境中创建一个 redis 集群。更具体地说，我想跨两个不同的 k8s 集群组成一个 redis 集群。

当我使用LoadBalancer(External IP)作为服务类型时，集群创建成功。问题是 NodePort。

在我命令 redis-cli --cluster create 后，它卡在“等待集群加入”

下面是集群创建命令的日志。我部署了 4 个 leader pods 和 4 个 slave pods 以及单独的 nodeport 服务。

root@redis-leader00-5fc546c4bd-28x8w:/data# redis-cli -a mypassword --cluster create --cluster-replicas 1 \
> 192.168.9.194:30030 192.168.9.199:30031 192.168.9.194:30032 192.168.9.199:30033 \
> 192.168.9.199:30030 192.168.9.194:30031 192.168.9.199:30032 192.168.9.194:30033
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 8 nodes...
Master[0] -> Slots 0 - 4095
Master[1] -> Slots 4096 - 8191
Master[2] -> Slots 8192 - 12287
Master[3] -> Slots 12288 - 16383
Adding replica 192.168.9.199:30030 to 192.168.9.194:30030
Adding replica 192.168.9.194:30033 to 192.168.9.199:30031
Adding replica 192.168.9.199:30032 to 192.168.9.194:30032
Adding replica 192.168.9.194:30031 to 192.168.9.199:30033
M: 94bf3c6760e6b3a91c408eda497822b4961e8d82 192.168.9.194:30030
   slots:[0-4095] (4096 slots) master
M: 31f4a9604b15109316f91956aa4a32b0c6952a4d 192.168.9.199:30031
   slots:[4096-8191] (4096 slots) master
M: 0738d1e1a677352fc3b0b3600a67d837b795fa8a 192.168.9.194:30032
   slots:[8192-12287] (4096 slots) master
M: 7dd7edbfab6952273460778d1f140b0716118042 192.168.9.199:30033
   slots:[12288-16383] (4096 slots) master
S: 17e044681319d0a05bd92deeb4ead31c0cd468e2 192.168.9.199:30030
   replicates 94bf3c6760e6b3a91c408eda497822b4961e8d82
S: 9c9e47ec566ac781e8e3dcb51398a27d1da71004 192.168.9.194:30031
   replicates 7dd7edbfab6952273460778d1f140b0716118042
S: b8f7028b56f96565a91fdb442c94fbedcee088c2 192.168.9.199:30032
   replicates 0738d1e1a677352fc3b0b3600a67d837b795fa8a
S: e4c9ffdf67e2b2ef9f840618110738358bde52d5 192.168.9.194:30033
   replicates 31f4a9604b15109316f91956aa4a32b0c6952a4d
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...........................................

奇怪的是其他redis容器收到了信号但是没有任何进展。下面是其他redis容器的日志。

.... // Some logs for initializing redis
1:M 20 Jan 2022 06:09:12.055 * Ready to accept connections
1:M 20 Jan 2022 06:13:41.263 # configEpoch set to 5 via CLUSTER SET-CONFIG-EPOCH

我以为通信成功了，但是gossip 端口没有正常工作。所以，我修改了 redis.conf 并设置了 cluster-announce-bus-port 但它也没有用。

如何使用 nodeport 类型的服务组成 redis 集群？

请参考其中一个.yaml文件

apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-cluster-leader00
  namespace: redis
  labels:
    app: redis-cluster
    leader: "00"
data:
  fix-ip.sh: |
    #!/bin/sh
    CLUSTER_CONFIG="/data/nodes.conf"
    if [ -f ${CLUSTER_CONFIG} ]; then
      if [ -z "${HOST_IP}" ]; then
        echo "Unable to determine Pod IP address!"
        exit 1
      fi
      echo "Updating my IP to ${HOST_IP} in ${CLUSTER_CONFIG}"
      sed -i.bak -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${HOST_IP}/" ${CLUSTER_CONFIG}
    fi
    exec "$@"
  redis.conf: |+
    bind 0.0.0.0
    cluster-enabled yes
    cluster-require-full-coverage no
    cluster-node-timeout 15000
    cluster-config-file /data/nodes.conf
    cluster-migration-barrier 1
    appendonly no
    save ""
    protected-mode no
    requirepass "mypassword"
    masterauth "mypassword"
    cluster-announce-ip 192.168.9.194
    cluster-announce-port 30030
    cluster-announce-bus-port 31030
---
apiVersion: v1
kind: Service
metadata:
  name: redis-leader00
  namespace: redis
  labels:
    app: redis
    role: leader
    tier: backend
    leader: "00"
spec:
  ports:
  - port: 6379
    targetPort: 6379
    nodePort: 30030
    name: client
  - port: 16379
    targetPort: 16379
    nodePort: 31030
    name: gossip
  selector:
    app: redis
    role: leader
    tier: backend
    leader: "00"
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-leader00
  namespace: redis
  labels:
    app: redis
    role: leader
    tier: backend
    leader: "00"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
      leader: "00"
  template:
    metadata:
      labels:
        app: redis
        role: leader
        tier: backend
        leader: "00"
    spec:
      containers:
      - name: leader
        image: docker.io/redis:6.0.5
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379
          name: client
        - containerPort: 16379
          name: gossip
        volumeMounts:
        - name: conf
          mountPath: /conf
          readOnly: false
        args: ["--requirepass", "mypassword"]
        command: ["/conf/fix-ip.sh", "redis-server", "/conf/redis.conf"]
        env:
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
      volumes:
      - name: conf
        configMap:
          name: redis-cluster-leader00
          defaultMode: 0755

在我命令 redis-cli create

后，还可以看到容器中的 nodes.conf 文件

root@redis-leader01-87ccb466-bsnq4:/data# cat nodes.conf
31f4a9604b15109316f91956aa4a32b0c6952a4d 192.168.9.199:30031@31031 myself,master - 0 0 2 connected 4096-8191
vars currentEpoch 2 lastVoteEpoch 0

Answer 1

我不确定您创建 Redis 集群所遵循的实际过程，但我建议您查看 helm 图表以在 K8s 上部署 Redis 集群。

使用helm chart可以很容易地在K8s上管理和部署Redis集群。

https://github.com/bitnami/charts/tree/master/bitnami/redis

要部署图表，您只需运行命令：

helm install my-release bitnami/redis

一旦部署了 helm chart，您就可以在节点端口端编辑服务类型，您还可以更新第一个 helm chart，然后将这些更改应用到 K8s。

这将在 K8s 上为 Redis 服务创建节点端口。

您可以在这里找到服务模板：https://github.com/bitnami/charts/tree/master/bitnami/redis/templates

Answer 2

已经有人发现问题了。这个link就是他做的issue。 https://github.com/redis/redis/issues/6432
问题的原因是 CLUSTER MEET 没有使用 cluster-announce-bus-port。实际上，在 clusterManagerNode 结构中甚至没有存储 cluster-announce-bus-port 的变量。

这个问题已经有 PR。 https://github.com/redis/redis/pull/6442

它尚未合并或关闭，所以这是持续存在的问题...

如何使用nodeport服务类型在k8s环境下搭建redis集群？

How to make a redis cluster in k8s environment using nodeport service type?

kubernetes

redis

cri-o