KafkaSpout 为 log4j 抛出 NoClassDefFoundError

Question

出于某种原因，当我尝试运行我在 Storm 集群上的拓扑时出现以下错误：

java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.Log4jLoggerFactory
  at org.apache.log4j.Logger.getLogger(Logger.java:39)
  at kafka.utils.Logging$class.logger(Logging.scala:24)
  at kafka.consumer.SimpleConsumer.logger$lzycompute(SimpleConsumer.scala:30)
  at kafka.consumer.SimpleConsumer.logger(SimpleConsumer.scala:30)
  at kafka.utils.Logging$class.info(Logging.scala:67)
  at kafka.consumer.SimpleConsumer.info(SimpleConsumer.scala:30)
  at kafka.consumer.SimpleConsumer.liftedTree1(SimpleConsumer.scala:75)
  at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:69)
  at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:128)
  at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
  at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:77)
  at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:67)
  at storm.kafka.PartitionManager.<init>(PartitionManager.java:83)
  at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98)
  at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69)
  at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:135)
  at backtype.storm.daemon.executor$fn__3373$fn__3388$fn__3417.invoke(executor.clj:565)
  at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463) at clojure.lang.AFn.run(AFn.java:24)
  at java.lang.Thread.run(Thread.java:745)cg

问题是什么以及如何解决？

以下是我包含的依赖项：

 <dependencies>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-core</artifactId>
        <version>0.9.5</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <version>0.8.2-beta</version>
    </dependency>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-kafka</artifactId>
        <version>0.9.5</version>
    </dependency>
    <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-mapper-asl</artifactId>
        <version>1.9.11</version>
    </dependency>
    <dependency>
        <groupId>org.java-websocket</groupId>
        <artifactId>Java-WebSocket</artifactId>
        <version>1.3.0</version>
    </dependency>
    <dependency>
        <groupId>org.twitter4j</groupId>
        <artifactId>twitter4j-core</artifactId>
        <version>[3.0,)</version>
    </dependency>
    <dependency>
        <groupId>org.twitter4j</groupId>
        <artifactId>twitter4j-stream</artifactId>
        <version>[3.0,)</version>
    </dependency>
</dependencies>

Answer 1

您使用 "right" 日志框架吗？

来自https://storm.apache.org/2013/12/08/storm090-released.html

Logging Changes

Another important change in 0.9.0 has to do with logging. Storm has largely switched over to the slf4j API (backed by a logback logger implementation). Some Storm dependencies rely on the log4j API, so Storm currently depends on log4j-over-slf4j.

These changes have implications for existing topologies and topology components that use the log4j API.

In general, and when possible, Storm topologies and topology components should use the slf4j API for logging.

如果您不使用与 Storm 相同的日志记录框架，则需要将使用的库与拓扑代码一起包含到您的 jar 文件中。

Answer 2

关键是你应该通过以下方式包含 Kafka：

   <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <version>0.8.1.1</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
            </exclusion>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

原因如下：

Note that the ZooKeeper and log4j dependencies are excluded to prevent version conflicts with Storm's dependencies.

Answer 3

我和你一样在引入kafka的时候也出现了同样的问题，所以我根据抛出的异常用另一种方式引入kafka:

SLF4J: Detected both log4j-over-slf4j.jar AND slf4j-log4j12.jar on the class path, preempting WhosebugError. SLF4J: See also http://www.slf4j.org/codes.html#log4jDelegationLoop for more details.

您可以通过以下方式包含kafka：

<dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <version>${kafka.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
        </exclusion>
    </exclusions>
</dependency>

KafkaSpout 为 log4j 抛出 NoClassDefFoundError

KafkaSpout throws NoClassDefFoundError for log4j

apache-kafka

apache-storm