从 Docker (ksqlDB) 连接到主机上的 Kafka
Connect to Kafka on host from Docker (ksqlDB)
我是 运行 来自 docker-compor 的 ksqldb-server https://ksqldb.io/quickstart.html#quickstart-content
我的 kafka bootstrap 服务器 运行 在同一个 VM 上,处于标准单独模式。
我可以通过控制台消费者看到一个主题中的消息:
sudo kafka-avro-console-consumer --from-beginning --bootstrap-server localhost:9092 --topic source-air-input --property print.key=true --max-messages 2
不幸的是 运行 来自 docker 的 ksql 给了我这个错误。
ksqldb-server | [2021-07-15 23:12:58,772] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:66)
ksqldb-server | java.lang.RuntimeException: Failed to get Kafka cluster information
ksqldb-server | at io.confluent.ksql.services.KafkaClusterUtil.getKafkaClusterId(KafkaClusterUtil.java:107)
ksqldb-server | at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:624)
ksqldb-server | at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:152)
ksqldb-server | at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:59)
ksqldb-server | Caused by: java.util.concurrent.TimeoutException
ksqldb-server | at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:108)
ksqldb-server | at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
ksqldb-server | at io.confluent.ksql.services.KafkaClusterUtil.getKafkaClusterId(KafkaClusterUtil.java:105)
我的docker-compose.yml如下。
---
version: '3.9'
services:
ksqldb-server:
image: confluentinc/ksqldb-server:0.18.0
hostname: ksqldb-server
container_name: ksqldb-server
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- "8088:8088"
environment:
KSQL_LISTENERS: http://0.0.0.0:8088
KSQL_BOOTSTRAP_SERVERS: host.docker.internal:9092
KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
ksqldb-cli:
image: confluentinc/ksqldb-cli:0.18.0
container_name: ksqldb-cli
depends_on:
- ksqldb-server
entrypoint: /bin/sh
tty: true
我尝试了多种可能的地址配置,但均未成功。
可能出了什么问题?
我尝试了这个问题 From inside of a Docker container, how do I connect to the localhost of the machine? 的建议,但没有成功。
修改Kafka的server.properties
listeners=PLAINTEXT_DOCKER://:29092,PLAINTEXT_LOCAL://localhost:9092
advertised.listeners=PLAINTEXT_DOCKER://host.docker.internal:29092,PLAINTEXT_LOCAL://localhost:9092
listener.security.protocol.map=PLAINTEXT_DOCKER:PLAINTEXT,PLAINTEXT_LOCAL:PLAINTEXT
inter.broker.listener.name=PLAINTEXT_LOCAL
像这样更新您的 Compose to point at the host 而不是它本身
version: '3.9'
services:
# TODO: add schema-registry
ksqldb-server:
image: confluentinc/ksqldb-server:0.18.0
hostname: ksqldb-server
container_name: ksqldb-server
ports:
- "8088:8088"
extra_hosts:
- "host.docker.internal:host-gateway"
environment:
KSQL_BOOTSTRAP_SERVERS: host.docker.internal:29092
...
(在 Mac 上测试),正在获取 KSQL
的 /info
端点
http :8088/info
HTTP/1.1 200 OK
content-length: 133
content-type: application/json
{
"KsqlServerInfo": {
"kafkaClusterId": "ZH2-h1W_SaivCW0qa8DQGA",
"ksqlServiceId": "default_",
"serverStatus": "RUNNING",
"version": "0.18.0"
}
}
将上面所有host.docker.internal
替换为本机外部hostname/IP,如果Kafka是远程服务器
我是 运行 来自 docker-compor 的 ksqldb-server https://ksqldb.io/quickstart.html#quickstart-content
我的 kafka bootstrap 服务器 运行 在同一个 VM 上,处于标准单独模式。 我可以通过控制台消费者看到一个主题中的消息:
sudo kafka-avro-console-consumer --from-beginning --bootstrap-server localhost:9092 --topic source-air-input --property print.key=true --max-messages 2
不幸的是 运行 来自 docker 的 ksql 给了我这个错误。
ksqldb-server | [2021-07-15 23:12:58,772] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:66)
ksqldb-server | java.lang.RuntimeException: Failed to get Kafka cluster information
ksqldb-server | at io.confluent.ksql.services.KafkaClusterUtil.getKafkaClusterId(KafkaClusterUtil.java:107)
ksqldb-server | at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:624)
ksqldb-server | at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:152)
ksqldb-server | at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:59)
ksqldb-server | Caused by: java.util.concurrent.TimeoutException
ksqldb-server | at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:108)
ksqldb-server | at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
ksqldb-server | at io.confluent.ksql.services.KafkaClusterUtil.getKafkaClusterId(KafkaClusterUtil.java:105)
我的docker-compose.yml如下。
---
version: '3.9'
services:
ksqldb-server:
image: confluentinc/ksqldb-server:0.18.0
hostname: ksqldb-server
container_name: ksqldb-server
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- "8088:8088"
environment:
KSQL_LISTENERS: http://0.0.0.0:8088
KSQL_BOOTSTRAP_SERVERS: host.docker.internal:9092
KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
ksqldb-cli:
image: confluentinc/ksqldb-cli:0.18.0
container_name: ksqldb-cli
depends_on:
- ksqldb-server
entrypoint: /bin/sh
tty: true
我尝试了多种可能的地址配置,但均未成功。 可能出了什么问题? 我尝试了这个问题 From inside of a Docker container, how do I connect to the localhost of the machine? 的建议,但没有成功。
修改Kafka的server.properties
listeners=PLAINTEXT_DOCKER://:29092,PLAINTEXT_LOCAL://localhost:9092
advertised.listeners=PLAINTEXT_DOCKER://host.docker.internal:29092,PLAINTEXT_LOCAL://localhost:9092
listener.security.protocol.map=PLAINTEXT_DOCKER:PLAINTEXT,PLAINTEXT_LOCAL:PLAINTEXT
inter.broker.listener.name=PLAINTEXT_LOCAL
像这样更新您的 Compose to point at the host 而不是它本身
version: '3.9'
services:
# TODO: add schema-registry
ksqldb-server:
image: confluentinc/ksqldb-server:0.18.0
hostname: ksqldb-server
container_name: ksqldb-server
ports:
- "8088:8088"
extra_hosts:
- "host.docker.internal:host-gateway"
environment:
KSQL_BOOTSTRAP_SERVERS: host.docker.internal:29092
...
(在 Mac 上测试),正在获取 KSQL
的/info
端点
http :8088/info
HTTP/1.1 200 OK
content-length: 133
content-type: application/json
{
"KsqlServerInfo": {
"kafkaClusterId": "ZH2-h1W_SaivCW0qa8DQGA",
"ksqlServiceId": "default_",
"serverStatus": "RUNNING",
"version": "0.18.0"
}
}
将上面所有host.docker.internal
替换为本机外部hostname/IP,如果Kafka是远程服务器