无法设置 kerberized kafka 代理:使用 Zookeeper 仲裁成员进行身份验证时出错:仲裁成员的 saslToken 为空
Cannot setup kerberized kafka broker: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null
所以我一直在尝试在启用了 kerberos 的单个节点上设置 kafka 代理和 zookeeper。
大部分基于本教程:https://qiita.com/visualskyrim/items/8f48ff107232f0befa5a
系统:Ubuntu18.04
设置:一个 EC2 盒子中的一个 zoopeeker 实例和一个 kafka 代理进程,另一个 EC2 盒子中的一个 KDC。两者都在 UDP 88 上具有开放端口的同一安全组中。
这是我到目前为止所做的。
- 从这里下载了 kafka 代理:https://kafka.apache.org/downloads
- 创建了一个 KDC 并正确生成了密钥表(通过 kinit -t 验证)。然后在 /etc/hosts 文件中定义 krb5_config 文件和主机条目到 kdc。
- 创建了两个 jaas 配置
猫zookeeper_jaas.conf
Server {
com.sun.security.auth.module.Krb5LoginModule required debug=true
useKeyTab=true
keyTab="/etc/kafka/zookeeper.keytab"
storeKey=true
useTicketCache=false
principal="zookeeper";
};
猫kafka_jaas.conf
cat /etc/kafka/kafka_jaas.conf
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required debug=true
useKeyTab=true
useTicketCache=false
storeKey=true
keyTab="/etc/kafka/kafka.keytab"
principal="kafka";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required debug=true
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/kafka/kafka.keytab"
principal="kafka";
};
- 在 kafka 代理配置中添加了一些行。
config/zookeeper 文件添加了这些额外的行
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
kerberos.removeHostFromPrincipal=true
kerberos.removeRealmFromPrincipal=true
和config/server.properties(代理的配置文件)添加了这些额外的行
listeners=SASL_PLAINTEXT://kafka.com:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
在一个屏幕会话中,我这样做
5. export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/etc/kafka/zookeeper_jaas.conf -Dsun.security.krb5.debug=true"
然后 运行
bin/zookeeper-server-start.sh config/zookeeper.properties
正确 运行s,zookeeper 启动。
我在另一个屏幕会话中做
6. export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/etc/kafka/kafka_jaas.conf -Dsun.security.krb5.debug=true"
然后 运行
bin/kafka-server-start.sh config/server.properties
但是这个失败了,除了这个例外
[2020-02-27 22:56:04,724] ERROR SASL authentication failed using login context 'Client' with
exception: {} (org.apache.zookeeper.client.ZooKeeperSaslClient)
javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member:
the quorum member's saslToken is null.
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:279)
at org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:242)
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:805)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2020-02-27 22:56:04,726] ERROR [ZooKeeperClient Kafka server] Auth failed.
(kafka.zookeeper.ZooKeeperClient)
[2020-02-27 22:56:04,842] ERROR Fatal error during KafkaServer startup. Prepare to shutdown
(kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for
/consumers
at org.apache.zookeeper.KeeperException.create(KeeperException.java:126)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:560)
at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1610)
at kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1532)
at kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths(KafkaZkClient.scala:1524)
at kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths$adapted(KafkaZkClient.scala:1524)
at scala.collection.immutable.List.foreach(List.scala:392)
at kafka.zk.KafkaZkClient.createTopLevelPaths(KafkaZkClient.scala:1524)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:388)
at kafka.server.KafkaServer.startup(KafkaServer.scala:207)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
我还启用了 kerberos 调试日志
这是 kerberos 的凭据日志
DEBUG: ----Credentials----
client: kafka@VISUALSKYRIM
server: zookeeper/localhost@VISUALSKYRIM
ticket: sname: zookeeper/localhost@VISUALSKYRIM
endTime: 1582881662000
----Credentials end----
这意味着客户端 jaas 配置在某种程度上是一个问题,问题出自这行代码:https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/client/ZooKeeperSaslClient.java#L310, but I cannot for the life of me figure out why. I cross referenced it with confluent docs and https://docs.confluent.io/2.0.0/kafka/sasl.html 看来我做对了。那么给出了什么?
谁能帮我解决这个问题?谢谢。
事实证明 kafka 暗中相信 zookeeper 的委托人是
zookeeper/localhost
为了进步我
- 在 KDC 中创建了 zookeeper/localhost 主体。
- 为此名为 zookeeper 创建了一个密钥表-server.keyta
将 zookeeper jaas 配置更新为
服务器{
com.sun.security.auth.module.Krb5LoginModule 需要 debug=true
使用KeyTab=真
keyTab="/etc/kafka/zookeeper-server.keytab"
存储键=真
useTicketCache=false
校长="zookeeper/localhost";
};
现在不再显示此错误。
kafka 生产者似乎正在根据我的 /etc/hosts 配置获取 SPN
# Replace there keberos KDC server IP with the appropriate IP addresses
172.31.40.220 kerberos.com
127.0.0.1 localhost
也许尝试查看 KAFKA_HOME/config/server.properties
并将默认值 localhost
更改为 your-host
zookeeper.connect=localhost:2181
作为校长 cname
与 sname
不同。示例:
cname zk/myhost@REALM.MY
sname zookeeper/localhost@REALM.MY
我也最终使用 EXTRA_ARGS 选项 -Dzookeeper.sasl.client.username=zk
,如 docs 中所述。
所以我一直在尝试在启用了 kerberos 的单个节点上设置 kafka 代理和 zookeeper。
大部分基于本教程:https://qiita.com/visualskyrim/items/8f48ff107232f0befa5a
系统:Ubuntu18.04 设置:一个 EC2 盒子中的一个 zoopeeker 实例和一个 kafka 代理进程,另一个 EC2 盒子中的一个 KDC。两者都在 UDP 88 上具有开放端口的同一安全组中。
这是我到目前为止所做的。
- 从这里下载了 kafka 代理:https://kafka.apache.org/downloads
- 创建了一个 KDC 并正确生成了密钥表(通过 kinit -t 验证)。然后在 /etc/hosts 文件中定义 krb5_config 文件和主机条目到 kdc。
- 创建了两个 jaas 配置
猫zookeeper_jaas.conf
Server {
com.sun.security.auth.module.Krb5LoginModule required debug=true
useKeyTab=true
keyTab="/etc/kafka/zookeeper.keytab"
storeKey=true
useTicketCache=false
principal="zookeeper";
};
猫kafka_jaas.conf
cat /etc/kafka/kafka_jaas.conf
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required debug=true
useKeyTab=true
useTicketCache=false
storeKey=true
keyTab="/etc/kafka/kafka.keytab"
principal="kafka";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required debug=true
useKeyTab=true
storeKey=true
useTicketCache=false
keyTab="/etc/kafka/kafka.keytab"
principal="kafka";
};
- 在 kafka 代理配置中添加了一些行。
config/zookeeper 文件添加了这些额外的行
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
jaasLoginRenew=3600000
kerberos.removeHostFromPrincipal=true
kerberos.removeRealmFromPrincipal=true
和config/server.properties(代理的配置文件)添加了这些额外的行
listeners=SASL_PLAINTEXT://kafka.com:9092
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
在一个屏幕会话中,我这样做
5. export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/etc/kafka/zookeeper_jaas.conf -Dsun.security.krb5.debug=true"
然后 运行
bin/zookeeper-server-start.sh config/zookeeper.properties
正确 运行s,zookeeper 启动。
我在另一个屏幕会话中做
6. export KAFKA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.auth.login.config=/etc/kafka/kafka_jaas.conf -Dsun.security.krb5.debug=true"
然后 运行
bin/kafka-server-start.sh config/server.properties
但是这个失败了,除了这个例外
[2020-02-27 22:56:04,724] ERROR SASL authentication failed using login context 'Client' with
exception: {} (org.apache.zookeeper.client.ZooKeeperSaslClient)
javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member:
the quorum member's saslToken is null.
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:279)
at org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:242)
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:805)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2020-02-27 22:56:04,726] ERROR [ZooKeeperClient Kafka server] Auth failed.
(kafka.zookeeper.ZooKeeperClient)
[2020-02-27 22:56:04,842] ERROR Fatal error during KafkaServer startup. Prepare to shutdown
(kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for
/consumers
at org.apache.zookeeper.KeeperException.create(KeeperException.java:126)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:560)
at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1610)
at kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1532)
at kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths(KafkaZkClient.scala:1524)
at kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths$adapted(KafkaZkClient.scala:1524)
at scala.collection.immutable.List.foreach(List.scala:392)
at kafka.zk.KafkaZkClient.createTopLevelPaths(KafkaZkClient.scala:1524)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:388)
at kafka.server.KafkaServer.startup(KafkaServer.scala:207)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
我还启用了 kerberos 调试日志
这是 kerberos 的凭据日志
DEBUG: ----Credentials----
client: kafka@VISUALSKYRIM
server: zookeeper/localhost@VISUALSKYRIM
ticket: sname: zookeeper/localhost@VISUALSKYRIM
endTime: 1582881662000
----Credentials end----
这意味着客户端 jaas 配置在某种程度上是一个问题,问题出自这行代码:https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/client/ZooKeeperSaslClient.java#L310, but I cannot for the life of me figure out why. I cross referenced it with confluent docs and https://docs.confluent.io/2.0.0/kafka/sasl.html 看来我做对了。那么给出了什么?
谁能帮我解决这个问题?谢谢。
事实证明 kafka 暗中相信 zookeeper 的委托人是
zookeeper/localhost
为了进步我
- 在 KDC 中创建了 zookeeper/localhost 主体。
- 为此名为 zookeeper 创建了一个密钥表-server.keyta
将 zookeeper jaas 配置更新为
服务器{ com.sun.security.auth.module.Krb5LoginModule 需要 debug=true 使用KeyTab=真 keyTab="/etc/kafka/zookeeper-server.keytab" 存储键=真 useTicketCache=false 校长="zookeeper/localhost"; };
现在不再显示此错误。
kafka 生产者似乎正在根据我的 /etc/hosts 配置获取 SPN
# Replace there keberos KDC server IP with the appropriate IP addresses
172.31.40.220 kerberos.com
127.0.0.1 localhost
也许尝试查看 KAFKA_HOME/config/server.properties
并将默认值 localhost
更改为 your-host
zookeeper.connect=localhost:2181
作为校长 cname
与 sname
不同。示例:
cname zk/myhost@REALM.MY
sname zookeeper/localhost@REALM.MY
我也最终使用 EXTRA_ARGS 选项 -Dzookeeper.sasl.client.username=zk
,如 docs 中所述。