Spring 使用 Confluent Kafka Avro Serializer 的 Kafka 测试找不到 ZKClientConfig
Spring Kafka Test with Confluent Kafka Avro Serializer cannot find ZKClientConfig
我很难启动嵌入式 Kafka 进行集成测试。显示此错误:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'embeddedKafka': Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/zookeeper/client/ZKClientConfig
这是我的最小样本。只要我需要进一步自动装配 Spring 个 beans,我就需要使用 @ExtendWith(SpringExtension.class)
。
@ExtendWith(SpringExtension.class)
@EmbeddedKafka(
partitions = 1,
bootstrapServersProperty = "spring.kafka.bootstrap-servers")
public class KafkaTest {
@Test
public void test(EmbeddedKafkaBroker broker) {
var brokerList = broker.getBrokersAsString();
log.debug("Embedded Kakfa topics {}", broker.getTopics());
}
}
我使用:
- Java11
- Spring 启动 2.3
- jUnit 5
我猜依赖性不一致发生在这里,但是,这很奇怪,因为我相信Spring Boot parent 可以实现依赖性兼容性。
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.3.0.RELEASE</version>
<relativePath/>
</parent>
当然,我在下面的pom.xml中列出了相关依赖项:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>5.2.3</version>
</dependency>
<!-- TEST SCOPE -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.6.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
我试图在 org.springframework.kafka:spring-kafka-test
和 io.confluent:kafka-avro-serializer
中排除 org.apache.kafka:kafka-clients
但没有成功,并且发生了同样的错误。
<exclusion>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
</exclusion>
简答:
这些依赖项发生冲突:
io.confluent:kafka-avro-serializer:5.2.3
org.springframework.kafka:spring-kafka-test:jar:2.5.0.RELEASE
- ... 来自父级
org.springframework.boot:spring-boot-starter-parent:2.3.0.RELEASE
前者使用org.apache.zookeeper:zookeeper:jar:3.4.13:compile
,应排除:
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>5.2.3</version>
<exclusions>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
</exclusions>
</dependency>
长答案:
之前检查 mvn dependency:tree
显示 kafka-avro-serializer
令人惊讶地包含 org.apache.zookeeper
依赖项,而 spring-kafka-test
则没有:
为 kafka-avro-serializer
提取
[INFO] +- io.confluent:kafka-avro-serializer:jar:5.2.3:compile
[INFO] | +- io.confluent:kafka-schema-registry-client:jar:5.2.3:compile
[INFO] | +- io.confluent:common-config:jar:5.2.3:compile
[INFO] | \- io.confluent:common-utils:jar:5.2.3:compile
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.13:compile
[INFO] | | +- jline:jline:jar:0.9.94:compile
[INFO] | | +- org.apache.yetus:audience-annotations:jar:0.5.0:compile
[INFO] | | \- io.netty:netty:jar:3.10.6.Final:compile
[INFO] | \- com.101tec:zkclient:jar:0.10:compile
为spring-kafka-test
提取
[INFO] \- org.springframework.kafka:spring-kafka-test:jar:2.5.0.RELEASE:test
[INFO] +- org.apache.kafka:kafka-clients:jar:test:2.5.0:test
[INFO] +- org.apache.kafka:kafka-streams:jar:2.5.0:test
[INFO] | +- org.apache.kafka:connect-json:jar:2.5.0:test
[INFO] | | \- org.apache.kafka:connect-api:jar:2.5.0:test
[INFO] | \- org.rocksdb:rocksdbjni:jar:5.18.3:test
[INFO] +- org.apache.kafka:kafka-streams-test-utils:jar:2.5.0:test
[INFO] +- org.apache.kafka:kafka_2.12:jar:2.5.0:test
[INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.11.0:test
[INFO] | | \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.11.0:test
[INFO] | | \- com.thoughtworks.paranamer:paranamer:jar:2.8:test
[INFO] | +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.11.0:test
[INFO] | +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:test
[INFO] | +- com.yammer.metrics:metrics-core:jar:2.2.0:test
[INFO] | +- org.scala-lang.modules:scala-collection-compat_2.12:jar:2.1.3:test
[INFO] | +- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.9.0:test
[INFO] | +- org.scala-lang:scala-library:jar:2.12.10:test
[INFO] | +- org.scala-lang:scala-reflect:jar:2.12.10:test
[INFO] | +- com.typesafe.scala-logging:scala-logging_2.12:jar:3.9.2:test
[INFO] | \- commons-cli:commons-cli:jar:1.4:test
[INFO] +- org.apache.kafka:kafka_2.12:jar:test:2.5.0:test
[INFO] \- org.junit.jupiter:junit-jupiter-api:jar:5.6.2:test
[INFO] +- org.apiguardian:apiguardian-api:jar:1.1.0:test
[INFO] +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] \- org.junit.platform:junit-platform-commons:jar:1.6.2:test
缺少 spring-kafka-test
org.apache.zookeeper:zookeeper
因此测试无法连接到 Zookeeper 并显示错误:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'embeddedKafka': Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/zookeeper/client/ZKClientConfig
从 kafka-avro-serializer
中排除这种依赖关系修复了问题(参见 简答)。 mvn dependency:tree
的输出确认 org.apache.zookeeper:zookeeper
已从 kafka-avro-serializer
中正确排除并包含在 spring-kafka-test
.
中
为 kafka-avro-serializer 提取:
+- io.confluent:kafka-avro-serializer:jar:5.2.3:compile
[INFO] | +- io.confluent:kafka-schema-registry-client:jar:5.2.3:compile
[INFO] | +- io.confluent:common-config:jar:5.2.3:compile
[INFO] | \- io.confluent:common-utils:jar:5.2.3:compile
[INFO] | \- com.101tec:zkclient:jar:0.10:compile
为spring-kafka-test提取:
\- org.springframework.kafka:spring-kafka-test:jar:2.5.0.RELEASE:test
[INFO] +- org.apache.kafka:kafka-clients:jar:test:2.5.0:test
[INFO] +- org.apache.kafka:kafka-streams:jar:2.5.0:test
[INFO] | +- org.apache.kafka:connect-json:jar:2.5.0:test
[INFO] | | \- org.apache.kafka:connect-api:jar:2.5.0:test
[INFO] | \- org.rocksdb:rocksdbjni:jar:5.18.3:test
[INFO] +- org.apache.kafka:kafka-streams-test-utils:jar:2.5.0:test
[INFO] +- org.apache.kafka:kafka_2.12:jar:2.5.0:test
[INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.11.0:test
[INFO] | | \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.11.0:test
[INFO] | | \- com.thoughtworks.paranamer:paranamer:jar:2.8:test
[INFO] | +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.11.0:test
[INFO] | +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:test
[INFO] | +- com.yammer.metrics:metrics-core:jar:2.2.0:test
[INFO] | +- org.scala-lang.modules:scala-collection-compat_2.12:jar:2.1.3:test
[INFO] | +- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.9.0:test
[INFO] | +- org.scala-lang:scala-library:jar:2.12.10:test
[INFO] | +- org.scala-lang:scala-reflect:jar:2.12.10:test
[INFO] | +- com.typesafe.scala-logging:scala-logging_2.12:jar:3.9.2:test
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.5.7:test
[INFO] | | +- org.apache.zookeeper:zookeeper-jute:jar:3.5.7:test
[INFO] | | +- org.apache.yetus:audience-annotations:jar:0.5.0:test
[INFO] | | +- io.netty:netty-handler:jar:4.1.49.Final:test
[INFO] | | | +- io.netty:netty-common:jar:4.1.49.Final:test
[INFO] | | | +- io.netty:netty-resolver:jar:4.1.49.Final:test
[INFO] | | | +- io.netty:netty-buffer:jar:4.1.49.Final:test
[INFO] | | | +- io.netty:netty-transport:jar:4.1.49.Final:test
[INFO] | | | \- io.netty:netty-codec:jar:4.1.49.Final:test
[INFO] | | \- io.netty:netty-transport-native-epoll:jar:4.1.49.Final:test
[INFO] | | \- io.netty:netty-transport-native-unix-common:jar:4.1.49.Final:test
[INFO] | \- commons-cli:commons-cli:jar:1.4:test
[INFO] +- org.apache.kafka:kafka_2.12:jar:test:2.5.0:test
[INFO] \- org.junit.jupiter:junit-jupiter-api:jar:5.6.2:test
[INFO] +- org.apiguardian:apiguardian-api:jar:1.1.0:test
[INFO] +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] \- org.junit.platform:junit-platform-commons:jar:1.6.2:test
我想在之前的回答中补充一些信息,因为我今天 运行 遇到了类似的问题。
主要问题是 kafka-client 2.5.0 依赖于 Zookeeper 3.5 而不再是 Zookeeper 3.4(请参阅 https://spring.io/projects/spring-kafka 了解兼容性矩阵)
ZkClientConfig class 已在 Zookeeper 3.5.2 中引入
因此,低于此版本的任何 Zookeeper 版本都会引发此错误。
出于测试目的,您需要删除任何已存在且低于 3.5.2 的 zookeeper 依赖项,并将 post 3.5.2 添加到您的测试依赖项中。
我很难启动嵌入式 Kafka 进行集成测试。显示此错误:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'embeddedKafka': Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/zookeeper/client/ZKClientConfig
这是我的最小样本。只要我需要进一步自动装配 Spring 个 beans,我就需要使用 @ExtendWith(SpringExtension.class)
。
@ExtendWith(SpringExtension.class)
@EmbeddedKafka(
partitions = 1,
bootstrapServersProperty = "spring.kafka.bootstrap-servers")
public class KafkaTest {
@Test
public void test(EmbeddedKafkaBroker broker) {
var brokerList = broker.getBrokersAsString();
log.debug("Embedded Kakfa topics {}", broker.getTopics());
}
}
我使用:
- Java11
- Spring 启动 2.3
- jUnit 5
我猜依赖性不一致发生在这里,但是,这很奇怪,因为我相信Spring Boot parent 可以实现依赖性兼容性。
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.3.0.RELEASE</version>
<relativePath/>
</parent>
当然,我在下面的pom.xml中列出了相关依赖项:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>5.2.3</version>
</dependency>
<!-- TEST SCOPE -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.6.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
我试图在 org.springframework.kafka:spring-kafka-test
和 io.confluent:kafka-avro-serializer
中排除 org.apache.kafka:kafka-clients
但没有成功,并且发生了同样的错误。
<exclusion>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
</exclusion>
简答:
这些依赖项发生冲突:
io.confluent:kafka-avro-serializer:5.2.3
org.springframework.kafka:spring-kafka-test:jar:2.5.0.RELEASE
- ... 来自父级
org.springframework.boot:spring-boot-starter-parent:2.3.0.RELEASE
前者使用org.apache.zookeeper:zookeeper:jar:3.4.13:compile
,应排除:
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>5.2.3</version>
<exclusions>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
</exclusions>
</dependency>
长答案:
之前检查 mvn dependency:tree
显示 kafka-avro-serializer
令人惊讶地包含 org.apache.zookeeper
依赖项,而 spring-kafka-test
则没有:
为 kafka-avro-serializer
提取[INFO] +- io.confluent:kafka-avro-serializer:jar:5.2.3:compile
[INFO] | +- io.confluent:kafka-schema-registry-client:jar:5.2.3:compile
[INFO] | +- io.confluent:common-config:jar:5.2.3:compile
[INFO] | \- io.confluent:common-utils:jar:5.2.3:compile
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.13:compile
[INFO] | | +- jline:jline:jar:0.9.94:compile
[INFO] | | +- org.apache.yetus:audience-annotations:jar:0.5.0:compile
[INFO] | | \- io.netty:netty:jar:3.10.6.Final:compile
[INFO] | \- com.101tec:zkclient:jar:0.10:compile
为spring-kafka-test
提取[INFO] \- org.springframework.kafka:spring-kafka-test:jar:2.5.0.RELEASE:test
[INFO] +- org.apache.kafka:kafka-clients:jar:test:2.5.0:test
[INFO] +- org.apache.kafka:kafka-streams:jar:2.5.0:test
[INFO] | +- org.apache.kafka:connect-json:jar:2.5.0:test
[INFO] | | \- org.apache.kafka:connect-api:jar:2.5.0:test
[INFO] | \- org.rocksdb:rocksdbjni:jar:5.18.3:test
[INFO] +- org.apache.kafka:kafka-streams-test-utils:jar:2.5.0:test
[INFO] +- org.apache.kafka:kafka_2.12:jar:2.5.0:test
[INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.11.0:test
[INFO] | | \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.11.0:test
[INFO] | | \- com.thoughtworks.paranamer:paranamer:jar:2.8:test
[INFO] | +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.11.0:test
[INFO] | +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:test
[INFO] | +- com.yammer.metrics:metrics-core:jar:2.2.0:test
[INFO] | +- org.scala-lang.modules:scala-collection-compat_2.12:jar:2.1.3:test
[INFO] | +- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.9.0:test
[INFO] | +- org.scala-lang:scala-library:jar:2.12.10:test
[INFO] | +- org.scala-lang:scala-reflect:jar:2.12.10:test
[INFO] | +- com.typesafe.scala-logging:scala-logging_2.12:jar:3.9.2:test
[INFO] | \- commons-cli:commons-cli:jar:1.4:test
[INFO] +- org.apache.kafka:kafka_2.12:jar:test:2.5.0:test
[INFO] \- org.junit.jupiter:junit-jupiter-api:jar:5.6.2:test
[INFO] +- org.apiguardian:apiguardian-api:jar:1.1.0:test
[INFO] +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] \- org.junit.platform:junit-platform-commons:jar:1.6.2:test
缺少 spring-kafka-test
org.apache.zookeeper:zookeeper
因此测试无法连接到 Zookeeper 并显示错误:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'embeddedKafka': Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/zookeeper/client/ZKClientConfig
从 kafka-avro-serializer
中排除这种依赖关系修复了问题(参见 简答)。 mvn dependency:tree
的输出确认 org.apache.zookeeper:zookeeper
已从 kafka-avro-serializer
中正确排除并包含在 spring-kafka-test
.
为 kafka-avro-serializer 提取:
+- io.confluent:kafka-avro-serializer:jar:5.2.3:compile
[INFO] | +- io.confluent:kafka-schema-registry-client:jar:5.2.3:compile
[INFO] | +- io.confluent:common-config:jar:5.2.3:compile
[INFO] | \- io.confluent:common-utils:jar:5.2.3:compile
[INFO] | \- com.101tec:zkclient:jar:0.10:compile
为spring-kafka-test提取:
\- org.springframework.kafka:spring-kafka-test:jar:2.5.0.RELEASE:test
[INFO] +- org.apache.kafka:kafka-clients:jar:test:2.5.0:test
[INFO] +- org.apache.kafka:kafka-streams:jar:2.5.0:test
[INFO] | +- org.apache.kafka:connect-json:jar:2.5.0:test
[INFO] | | \- org.apache.kafka:connect-api:jar:2.5.0:test
[INFO] | \- org.rocksdb:rocksdbjni:jar:5.18.3:test
[INFO] +- org.apache.kafka:kafka-streams-test-utils:jar:2.5.0:test
[INFO] +- org.apache.kafka:kafka_2.12:jar:2.5.0:test
[INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.11.0:test
[INFO] | | \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.11.0:test
[INFO] | | \- com.thoughtworks.paranamer:paranamer:jar:2.8:test
[INFO] | +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.11.0:test
[INFO] | +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:test
[INFO] | +- com.yammer.metrics:metrics-core:jar:2.2.0:test
[INFO] | +- org.scala-lang.modules:scala-collection-compat_2.12:jar:2.1.3:test
[INFO] | +- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.9.0:test
[INFO] | +- org.scala-lang:scala-library:jar:2.12.10:test
[INFO] | +- org.scala-lang:scala-reflect:jar:2.12.10:test
[INFO] | +- com.typesafe.scala-logging:scala-logging_2.12:jar:3.9.2:test
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.5.7:test
[INFO] | | +- org.apache.zookeeper:zookeeper-jute:jar:3.5.7:test
[INFO] | | +- org.apache.yetus:audience-annotations:jar:0.5.0:test
[INFO] | | +- io.netty:netty-handler:jar:4.1.49.Final:test
[INFO] | | | +- io.netty:netty-common:jar:4.1.49.Final:test
[INFO] | | | +- io.netty:netty-resolver:jar:4.1.49.Final:test
[INFO] | | | +- io.netty:netty-buffer:jar:4.1.49.Final:test
[INFO] | | | +- io.netty:netty-transport:jar:4.1.49.Final:test
[INFO] | | | \- io.netty:netty-codec:jar:4.1.49.Final:test
[INFO] | | \- io.netty:netty-transport-native-epoll:jar:4.1.49.Final:test
[INFO] | | \- io.netty:netty-transport-native-unix-common:jar:4.1.49.Final:test
[INFO] | \- commons-cli:commons-cli:jar:1.4:test
[INFO] +- org.apache.kafka:kafka_2.12:jar:test:2.5.0:test
[INFO] \- org.junit.jupiter:junit-jupiter-api:jar:5.6.2:test
[INFO] +- org.apiguardian:apiguardian-api:jar:1.1.0:test
[INFO] +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] \- org.junit.platform:junit-platform-commons:jar:1.6.2:test
我想在之前的回答中补充一些信息,因为我今天 运行 遇到了类似的问题。
主要问题是 kafka-client 2.5.0 依赖于 Zookeeper 3.5 而不再是 Zookeeper 3.4(请参阅 https://spring.io/projects/spring-kafka 了解兼容性矩阵)
ZkClientConfig class 已在 Zookeeper 3.5.2 中引入 因此,低于此版本的任何 Zookeeper 版本都会引发此错误。 出于测试目的,您需要删除任何已存在且低于 3.5.2 的 zookeeper 依赖项,并将 post 3.5.2 添加到您的测试依赖项中。