offsets.storage=kafka 的 Kafka 0.8.2.1 怎么可能仍然需要 ZooKeeper?

How could Kafka 0.8.2.1 with offsets.storage=kafka still require ZooKeeper?

我们正在使用 0.8.2.1 建立一个新的 Kafka 项目,并希望专门将消费者偏移量写入 Kafka。所以我们在消费者配置中设置了 offsets.storage=kafkadual.commit.enabled=false。然而,当我们创建消费者连接器时,它仍然想要连接到 ZooKeeper:

kafka.consumer.Consumer.createJavaConsumerConnector(config);
// fails with:
// Caused by: java.lang.IllegalArgumentException: requirement failed: 
//      Missing required property 'zookeeper.connect'

我想也许我们只需要指定 zookeeper.connect 并且它会被忽略,所以我指定了一个无效的主机名,但仍然失败,因为它确实尝试连接。如果可以避免,我们真的不希望我们的消费者必须连接到 ZooKeeper。那么给出了什么?

ZooKeeper 似乎在 0.8.2 中仍然是必需的,但他们希望在 0.8.3 中解决这个问题。来自邮件列表中的 Gwen Shapira:

The existing consumer uses Zookeeper both to commit offsets and to assign partitions to different consumers and threads in the same consumer group. While offsets can be committed to Kafka in 0.8.2 releases, which greatly reduces the load on Zookeeper, the consumer still requires Zookeeper to manage group membership and partition ownership. The new consumer (which we hope to have ready for 0.8.3 release) will completely remove the Zookeeper dependency, managing both offsets and partition ownership within Kafka itself.