Gitlab 中的 Kafka 和 Zookeeper CI

Question

我正在尝试运行一个简单的测试，看看我的应用程序是否运行正常，没有任何问题。我的问题是 faust 在初始化时需要连接到 kafka - 所以我正在尝试运行 kafka 与 zookeeper 作为服务，但我无法正确连接它们。

错误：

2021-12-16T13:53:51.385341793Z [2021-12-16 13:53:51,385] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
2021-12-16T13:53:51.391012666Z [2021-12-16 13:53:51,390] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn)
2021-12-16T13:53:51.395158219Z [2021-12-16 13:53:51,395] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
2021-12-16T13:53:51.399485772Z [2021-12-16 13:53:51,397] ERROR Unable to resolve address: zookeeper:2181 (org.apache.zookeeper.client.StaticHostProvider)
2021-12-16T13:53:51.399499707Z java.net.UnknownHostException: zookeeper: Name or service not known
2021-12-16T13:53:51.399503169Z  at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
2021-12-16T13:53:51.399506400Z  at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
2021-12-16T13:53:51.399509510Z  at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519)
2021-12-16T13:53:51.399512353Z  at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
2021-12-16T13:53:51.399531020Z  at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
2021-12-16T13:53:51.399534098Z  at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
2021-12-16T13:53:51.399537044Z  at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
2021-12-16T13:53:51.399540881Z  at org.apache.zookeeper.client.StaticHostProvider.getAllByName(StaticHostProvider.java:88)
2021-12-16T13:53:51.399544771Z  at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2021-12-16T13:53:51.399548877Z  at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2021-12-16T13:53:51.399553025Z  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1207)
2021-12-16T13:53:51.406655054Z [2021-12-16 13:53:51,406] WARN Session 0x0 for sever zookeeper:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. (org.apache.zookeeper.ClientCnxn)
2021-12-16T13:53:51.406696302Z java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
2021-12-16T13:53:51.406703099Z  at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:78)
2021-12-16T13:53:51.406707676Z  at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41)
2021-12-16T13:53:51.406711700Z  at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1161)
2021-12-16T13:53:51.406715631Z  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1210)
2021-12-16T13:53:52.508636206Z [2021-12-16 13:53:52,508] ERROR Unable to resolve address: zookeeper:2181 (org.apache.zookeeper.client.StaticHostProvider)
2021-12-16T13:53:52.508665462Z java.net.UnknownHostException: zookeeper

.gitlab-ci.yml:

.zoo_service: &zoo_service
  name: zookeeper:latest
  alias: zookeeper

.kafka_service: &kafka_service
  name: bitnami/kafka:latest
  alias: kafka


faust:
  variables:
    ALLOW_ANONYMOUS_LOGIN: "yes"
    KAFKA_BROKER_ID: 1
    KAFKA_CFG_LISTENERS: "PLAINTEXT://:9092"
    KAFKA_CFG_ADVERTISED_LISTENERS: "PLAINTEXT://127.0.0.1:9092"
    KAFKA_CFG_ZOOKEEPER_CONNECT: "zookeeper:2181"
    ALLOW_PLAINTEXT_LISTENER: "yes"
  stage: test
  <<: *python_image
  services:
    - *zoo_service
    - *kafka_service
  before_script:
    - *setup_venv_script
  script:
    - faust -A runner worker -l info & sleep 15; kill -HUP $!
  <<: *load_env
  except:
    - schedules

我希望我的做法是正确的 - 遗憾的是，关于这个问题我可以阅读的资源不多。我知道问题出在 kafka 和 zookeeper 之间，但我不确定如何解决（认为这是正确的方法）。 CI 中甚至有 2 个服务可以相互通信吗？

谢谢！

Answer 1

浏览了关于连接不同服务的GitLab CI docs，它提到了允许跨服务通信的功能标志，所以尝试

faust:
  variables:
    FF_NETWORK_PER_BUILD: 1
    ...
  services:
    ...

另外，对于Kafka通信，它需要通告它的别名而不是localhost，所以改变

KAFKA_CFG_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092"

Gitlab 中的 Kafka 和 Zookeeper CI

Kafka & Zookeeper in Gitlab CI

apache-kafka

gitlab-ci

apache-zookeeper