如何在 Docker (fig.yml) 中设置 Zookeeper dataDir
How to set Zookeeper dataDir in Docker (fig.yml)
我已经在 fig.yml 文件中为 Docker 配置了 Zookeeper 和 Kafka 容器。两个容器都开始正常。但是在发送了一些消息之后,我的应用程序 /zk-client 挂起了。在检查 zookeeper 日志时,我看到错误:
Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers
我的fig.yml如下:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
environment:
ZK_ADVERTISED_HOST_NAME: xx.xx.x.xxx
ZK_CONNECTION_TIMEOUT_MS: 6000
ZK_SYNC_TIME_MS: 2000
ZK_DATADIR: /path/to/data/zk/data/dir
kafka:
image: wurstmeister/kafka:0.8.2.0
ports:
- "xx.xx.x.xxx:9092:9092"
links:
- zookeeper:zk
environment:
KAFKA_ADVERTISED_HOST_NAME: xx.xx.x.xxx
KAFKA_LOG_DIRS: /home/svc_cis4/dl
volumes:
- /var/run/docker.sock:/var/run/docker.sock
我已经搜索了好久了,但我还没有找到解决方案。我还尝试使用 ZK_DATADIR: '/path/to/zk/data/dir' 在 fig.yml 中设置数据目录,但它似乎没有帮助。如有任何帮助,我们将不胜感激。
更新
/opt/kafka_2.10-0.8.2.0/config/server.properties
的内容:
broker.id=0
port=9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
您遇到的问题与zookeeper的数据目录无关。错误 Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers
是由于您的应用程序无法在 zookeeper 的数据中找到任何代理 znode。发生这种情况可能是因为 kafka 容器未与 zookeeper 正确连接,并且查看 wurstmeister 的图像我认为问题可能与变量 KAFKA_ADVERTISED_HOST_NAME
可能是错误的有关。我不知道是否有理由通过必须传递的环境变量来分配该变量,但从我的角度来看,这不是一个好方法。有多种配置kafka的方法(实际上不需要设置advertised.host.name
,你可以将其注释掉,kafka将采用默认主机名,可以使用docker设置),但是一个快速的解决方案使用它会编辑 start-kafka.sh
并重建图像:
#!/bin/bash
if [[ -z "$KAFKA_ADVERTISED_PORT" ]]; then
export KAFKA_ADVERTISED_PORT=$(docker port `hostname` 9092 | sed -r "s/.*:(.*)//g")
fi
if [[ -z "$KAFKA_BROKER_ID" ]]; then
export KAFKA_BROKER_ID=$KAFKA_ADVERTISED_PORT
fi
if [[ -z "$KAFKA_LOG_DIRS" ]]; then
export KAFKA_LOG_DIRS="/kafka/kafka-logs-$KAFKA_BROKER_ID"
fi
if [[ -z "$KAFKA_ZOOKEEPER_CONNECT" ]]; then
export KAFKA_ZOOKEEPER_CONNECT=$(env | grep ZK.*PORT_2181_TCP= | sed -e 's|.*tcp://||' | paste -sd ,)
fi
if [[ -n "$KAFKA_HEAP_OPTS" ]]; then
sed -r -i "s/^(export KAFKA_HEAP_OPTS)=\"(.*)\"/=\"$KAFKA_HEAP_OPTS\"/g" $KAFKA_HOME/bin/kafka-server-start.sh
unset KAFKA_HEAP_OPTS
fi
for VAR in `env`
do
if [[ $VAR =~ ^KAFKA_ && ! $VAR =~ ^KAFKA_HOME ]]; then
kafka_name=`echo "$VAR" | sed -r "s/KAFKA_(.*)=.*//g" | tr '[:upper:]' '[:lower:]' | tr _ .`
env_var=`echo "$VAR" | sed -r "s/(.*)=.*//g"`
if egrep -q "(^|^#)$kafka_name=" $KAFKA_HOME/config/server.properties; then
sed -r -i "s@(^|^#)($kafka_name)=(.*)@=${!env_var}@g" $KAFKA_HOME/config/server.properties #note that no config values may contain an '@' char
else
echo "$kafka_name=${!env_var}" >> $KAFKA_HOME/config/server.properties
fi
fi
done
###NEW###
IP=$(hostname --ip-address)
sed -i -e "s/^advertised.host.name.*/advertised.host.name=$IP/" $KAFKA_HOME/config/server.properties
###END###
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
如果这不能解决您的问题,您可以获得更多信息,在容器内启动会话(即:docker exec -it kafkadocker_kafka_1 /bin/bash
用于 kafka,docker exec -it kafkadocker_zookeeper_1 /bin/bash
用于 zookeeper),然后检查 kafka 日志,或动物园管理员控制台 (/opt/zookeeper-3.4.6/bin/zkCli.sh
)
过去两天对我有效的配置包括为 Zookeeper 和 Kafka 指定主机地址。我的fig.yml
内容是:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "xx.xx.x.xxx:2181:2181"
kafka:
image: wurstmeister/kafka:0.8.2.0
ports:
- "9092:9092"
links:
- zookeeper:zk
environment:
KAFKA_ADVERTISED_HOST_NAME: xx.xx.x.xxx
KAFKA_NUM_REPLICA_FETCHERS: 4
...other env variables...
volumes:
- /var/run/docker.sock:/var/run/docker.sock
validator:
build: .
volumes:
- .:/host
entrypoint: /bin/bash
command: -c 'java -jar /host/app1.jar'
links:
- zookeeper:zk
- kafka
analytics:
build: .
volumes:
- .:/host
entrypoint: /bin/bash
command: -c 'java -jar /host/app2.jar'
links:
- zookeeper:zk
- kafka
loader:
build: .
volumes:
- .:/host
entrypoint: /bin/bash
command: -c 'java -jar /host/app3.jar'
links:
- zookeeper:zk
- kafka
以及附带的Dockerfile
内容:
FROM ubuntu:trusty
MAINTAINER Wurstmeister
RUN apt-get update; apt-get install -y unzip openjdk-7-jdk wget git docker.io
RUN wget -q http://apache.mirrors.lucidnetworks.net/kafka/0.8.2.0/kafka_2.10-0.8.2.0.tgz -O /tmp/kafka_2.10-0.8.2.0.tgz
RUN tar xfz /tmp/kafka_2.10-0.8.2.0.tgz -C /opt
VOLUME ["/kafka"]
ENV KAFKA_HOME /opt/kafka_2.10-0.8.2.0
ADD start-kafka.sh /usr/bin/start-kafka.sh
ADD broker-list.sh /usr/bin/broker-list.sh
CMD start-kafka.sh
我已经在 fig.yml 文件中为 Docker 配置了 Zookeeper 和 Kafka 容器。两个容器都开始正常。但是在发送了一些消息之后,我的应用程序 /zk-client 挂起了。在检查 zookeeper 日志时,我看到错误:
Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers
我的fig.yml如下:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
environment:
ZK_ADVERTISED_HOST_NAME: xx.xx.x.xxx
ZK_CONNECTION_TIMEOUT_MS: 6000
ZK_SYNC_TIME_MS: 2000
ZK_DATADIR: /path/to/data/zk/data/dir
kafka:
image: wurstmeister/kafka:0.8.2.0
ports:
- "xx.xx.x.xxx:9092:9092"
links:
- zookeeper:zk
environment:
KAFKA_ADVERTISED_HOST_NAME: xx.xx.x.xxx
KAFKA_LOG_DIRS: /home/svc_cis4/dl
volumes:
- /var/run/docker.sock:/var/run/docker.sock
我已经搜索了好久了,但我还没有找到解决方案。我还尝试使用 ZK_DATADIR: '/path/to/zk/data/dir' 在 fig.yml 中设置数据目录,但它似乎没有帮助。如有任何帮助,我们将不胜感激。
更新
/opt/kafka_2.10-0.8.2.0/config/server.properties
的内容:
broker.id=0
port=9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
您遇到的问题与zookeeper的数据目录无关。错误 Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers
是由于您的应用程序无法在 zookeeper 的数据中找到任何代理 znode。发生这种情况可能是因为 kafka 容器未与 zookeeper 正确连接,并且查看 wurstmeister 的图像我认为问题可能与变量 KAFKA_ADVERTISED_HOST_NAME
可能是错误的有关。我不知道是否有理由通过必须传递的环境变量来分配该变量,但从我的角度来看,这不是一个好方法。有多种配置kafka的方法(实际上不需要设置advertised.host.name
,你可以将其注释掉,kafka将采用默认主机名,可以使用docker设置),但是一个快速的解决方案使用它会编辑 start-kafka.sh
并重建图像:
#!/bin/bash
if [[ -z "$KAFKA_ADVERTISED_PORT" ]]; then
export KAFKA_ADVERTISED_PORT=$(docker port `hostname` 9092 | sed -r "s/.*:(.*)//g")
fi
if [[ -z "$KAFKA_BROKER_ID" ]]; then
export KAFKA_BROKER_ID=$KAFKA_ADVERTISED_PORT
fi
if [[ -z "$KAFKA_LOG_DIRS" ]]; then
export KAFKA_LOG_DIRS="/kafka/kafka-logs-$KAFKA_BROKER_ID"
fi
if [[ -z "$KAFKA_ZOOKEEPER_CONNECT" ]]; then
export KAFKA_ZOOKEEPER_CONNECT=$(env | grep ZK.*PORT_2181_TCP= | sed -e 's|.*tcp://||' | paste -sd ,)
fi
if [[ -n "$KAFKA_HEAP_OPTS" ]]; then
sed -r -i "s/^(export KAFKA_HEAP_OPTS)=\"(.*)\"/=\"$KAFKA_HEAP_OPTS\"/g" $KAFKA_HOME/bin/kafka-server-start.sh
unset KAFKA_HEAP_OPTS
fi
for VAR in `env`
do
if [[ $VAR =~ ^KAFKA_ && ! $VAR =~ ^KAFKA_HOME ]]; then
kafka_name=`echo "$VAR" | sed -r "s/KAFKA_(.*)=.*//g" | tr '[:upper:]' '[:lower:]' | tr _ .`
env_var=`echo "$VAR" | sed -r "s/(.*)=.*//g"`
if egrep -q "(^|^#)$kafka_name=" $KAFKA_HOME/config/server.properties; then
sed -r -i "s@(^|^#)($kafka_name)=(.*)@=${!env_var}@g" $KAFKA_HOME/config/server.properties #note that no config values may contain an '@' char
else
echo "$kafka_name=${!env_var}" >> $KAFKA_HOME/config/server.properties
fi
fi
done
###NEW###
IP=$(hostname --ip-address)
sed -i -e "s/^advertised.host.name.*/advertised.host.name=$IP/" $KAFKA_HOME/config/server.properties
###END###
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties
如果这不能解决您的问题,您可以获得更多信息,在容器内启动会话(即:docker exec -it kafkadocker_kafka_1 /bin/bash
用于 kafka,docker exec -it kafkadocker_zookeeper_1 /bin/bash
用于 zookeeper),然后检查 kafka 日志,或动物园管理员控制台 (/opt/zookeeper-3.4.6/bin/zkCli.sh
)
过去两天对我有效的配置包括为 Zookeeper 和 Kafka 指定主机地址。我的fig.yml
内容是:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "xx.xx.x.xxx:2181:2181"
kafka:
image: wurstmeister/kafka:0.8.2.0
ports:
- "9092:9092"
links:
- zookeeper:zk
environment:
KAFKA_ADVERTISED_HOST_NAME: xx.xx.x.xxx
KAFKA_NUM_REPLICA_FETCHERS: 4
...other env variables...
volumes:
- /var/run/docker.sock:/var/run/docker.sock
validator:
build: .
volumes:
- .:/host
entrypoint: /bin/bash
command: -c 'java -jar /host/app1.jar'
links:
- zookeeper:zk
- kafka
analytics:
build: .
volumes:
- .:/host
entrypoint: /bin/bash
command: -c 'java -jar /host/app2.jar'
links:
- zookeeper:zk
- kafka
loader:
build: .
volumes:
- .:/host
entrypoint: /bin/bash
command: -c 'java -jar /host/app3.jar'
links:
- zookeeper:zk
- kafka
以及附带的Dockerfile
内容:
FROM ubuntu:trusty
MAINTAINER Wurstmeister
RUN apt-get update; apt-get install -y unzip openjdk-7-jdk wget git docker.io
RUN wget -q http://apache.mirrors.lucidnetworks.net/kafka/0.8.2.0/kafka_2.10-0.8.2.0.tgz -O /tmp/kafka_2.10-0.8.2.0.tgz
RUN tar xfz /tmp/kafka_2.10-0.8.2.0.tgz -C /opt
VOLUME ["/kafka"]
ENV KAFKA_HOME /opt/kafka_2.10-0.8.2.0
ADD start-kafka.sh /usr/bin/start-kafka.sh
ADD broker-list.sh /usr/bin/broker-list.sh
CMD start-kafka.sh