在 minikube 上部署 Kafka

Question

嗨，我是 kubernets 的新手，我正在使用 minikube 单节点集群进行本地开发和测试。

Host: Ubuntu 16.04 LTS.
Minikube: Virtual box running minikube cluster

我的要求是我需要在 minikube 上部署 kafka 和 zookeeper，并且应该用于生产或消费消息。我关注 this link 并成功将其部署在 minikube 上，详情如下

$ kubectl get services
NAME            CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
kafka-service   10.0.0.15    <pending>     9092:30244/TCP               46m
kubernetes      10.0.0.1     <none>        443/TCP                      53m
zoo1            10.0.0.43    <none>        2181/TCP,2888/TCP,3888/TCP   50m
zoo2            10.0.0.226   <none>        2181/TCP,2888/TCP,3888/TCP   50m
zoo3            10.0.0.6     <none>        2181/TCP,2888/TCP,3888/TCP   50m

$ kubectl get pods
NAME                                      READY     STATUS    RESTARTS   AGE
kafka-deployment-3583985961-f2301         1/1       Running   0          48m
zookeeper-deployment-1-1598963595-vgx1l   1/1       Running   0          52m
zookeeper-deployment-2-2038841231-tdsff   1/1       Running   0          52m
zookeeper-deployment-3-2478718867-5vjcj   1/1       Running   0          52m

$ kubectl describe service kafka-service
Name:           kafka-service
Namespace:      default
Labels:         app=kafka
Annotations:    <none>
Selector:       app=kafka
Type:           LoadBalancer
IP:             10.0.0.15
Port:           kafka-port  9092/TCP
NodePort:       kafka-port  30244/TCP
Endpoints:      172.17.0.7:9092
Session Affinity:   None
Events:         <none>

并且我已将 KAFKA_ADVERTISED_HOST_NAME 设置为 minikube ip(192.168.99.100)。现在对于消息生成器，我正在使用 $cat textfile.log | kafkacat -b $(minikube ip):30244 -t mytopic 它不会发布消息

% Auto-selecting Producer mode (use -P or -C to override)
% Delivery failed for message: Local: Message timed out

谁能帮忙发布和消费消息。

Answer 1

您使用了类型为 LoadBalancer 的服务，该服务用于云提供商（您可以看到该服务正在等待外部 IP 地址..挂起状态...这永远不会发生）。在您的情况下，您应该尝试使用 NodePort。

Answer 2

离开，您在此处的评论是使用 type=Nodeport 的 Kafka 服务的非常基本的示例代码。

apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  ports:
  - port: 9092
    name: kafka-port
    protocol: TCP
  selector:
    app: kafka
  type: NodePort

Answer 3

我知道这已经很老了 post。您是否能够在 minikube 中解决运行 kafka + zookeeper？我能够使用 minikube v0.17.1 成功地运行一个简单的单集群 kafka 和 zookeeper 部署，并分别使用 kafkacat 生产者和消费者生产和消费消息。我能够在 uBuntu 和 Mac OSX 上成功运行这些。部署和服务 yaml 如下：

zookeeper-deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: zookeeper
  name: zookeeper
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
      - image: wurstmeister/zookeeper
        imagePullPolicy: IfNotPresent
        name: zookeeper
        ports:
        - containerPort: 2181

zookeeper-service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: zookeeper-service
  name: zookeeper-service
spec:
  type: NodePort
  ports:
  - name: zookeeper-port
    port: 2181
    nodePort: 30181
    targetPort: 2181
  selector:
    app: zookeeper

kafka-deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: kafka
  name: kafka
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - env:
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: "192.168.99.100"
        - name: KAFKA_ADVERTISED_PORT
          value: "30092"
        - name: KAFKA_BROKER_ID
          value: "1"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: 192.168.99.100:30181
        - name: KAFKA_CREATE_TOPICS
          value: "test-topic:1:1"
        image: wurstmeister/kafka
        imagePullPolicy: IfNotPresent
        name: kafka
        ports:
        - containerPort: 9092

kafka-service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: kafka-service
  name: kafka-service
spec:
  type: NodePort
  ports:
  - name: kafka-port
    port: 9092
    nodePort: 30092
    targetPort: 9092
  selector:
    app: kafka

您可以通过安装 kafkacat 客户端和运行在单独的终端上执行以下命令来测试您的部署 windows:

echo "Am I receiving this message?" | kafkacat -P -b 192.168.99.100:30092 -t test-topic

kafkacat -C -b 192.168.99.100:30092 -t test-topic

% Reached end of topic test-topic [0] at offset 0

Am I receiving this message?

我能够在 minikube 版本 v0.17.1 和 v0.19.0 上成功运行。如果你想在 minikube 版本 v0.21.1 和 v0.23.0 上运行这个，请参考我在这里对 post 的回复：Kafka inaccessible once inside Kubernetes/Minikube

谢谢。

Answer 4

我按照下图进行了关注。

$ git clone https://github.com/d1egoaz/minikube-kafka-cluster

$ cd minikube-kafka-cluster

$ kubectl apply -f 00-namespace/

$ kubectl apply -f 01-zookeeper/

$ kubectl apply -f 02-kafka/

$ kubectl apply -f 03-yahoo-kafka-manager/

$ kubectl get svc -n kafka-ca1 (get the port of kafka manager like 31445)

$ minikube ip

在浏览器中输入http://minikube-ip:port得到kafka manager的ui页面。

来源https://technology.amis.nl/2019/03/24/running-apache-kafka-on-minikube/

在 minikube 上部署 Kafka

Kafka deployment on minikube

apache-kafka

kubernetes

kafkacat

kubectl

minikube