5

For some reason I get the following error when I try to run my topology on a Storm cluster:

java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.Log4jLoggerFactory
  at org.apache.log4j.Logger.getLogger(Logger.java:39)
  at kafka.utils.Logging$class.logger(Logging.scala:24)
  at kafka.consumer.SimpleConsumer.logger$lzycompute(SimpleConsumer.scala:30)
  at kafka.consumer.SimpleConsumer.logger(SimpleConsumer.scala:30)
  at kafka.utils.Logging$class.info(Logging.scala:67)
  at kafka.consumer.SimpleConsumer.info(SimpleConsumer.scala:30)
  at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:75)
  at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:69)
  at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:128)
  at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
  at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:77)
  at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:67)
  at storm.kafka.PartitionManager.<init>(PartitionManager.java:83)
  at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98)
  at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69)
  at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:135)
  at backtype.storm.daemon.executor$fn__3373$fn__3388$fn__3417.invoke(executor.clj:565)
  at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463) at clojure.lang.AFn.run(AFn.java:24)
  at java.lang.Thread.run(Thread.java:745)cg

What is the problem and how to solve it?

Here are the dependencies that I include:

 <dependencies>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-core</artifactId>
        <version>0.9.5</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <version>0.8.2-beta</version>
    </dependency>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-kafka</artifactId>
        <version>0.9.5</version>
    </dependency>
    <dependency>
        <groupId>org.codehaus.jackson</groupId>
        <artifactId>jackson-mapper-asl</artifactId>
        <version>1.9.11</version>
    </dependency>
    <dependency>
        <groupId>org.java-websocket</groupId>
        <artifactId>Java-WebSocket</artifactId>
        <version>1.3.0</version>
    </dependency>
    <dependency>
        <groupId>org.twitter4j</groupId>
        <artifactId>twitter4j-core</artifactId>
        <version>[3.0,)</version>
    </dependency>
    <dependency>
        <groupId>org.twitter4j</groupId>
        <artifactId>twitter4j-stream</artifactId>
        <version>[3.0,)</version>
    </dependency>
</dependencies>

3 Answers 3

7

The thing is that you should include Kafka in the following way:

   <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <version>0.8.1.1</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
            </exclusion>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

For the following reason:

Note that the ZooKeeper and log4j dependencies are excluded to prevent version conflicts with Storm's dependencies.

4
  • Yes, I agree with you. :) So you'll get an up-vote from me, since my answer is more concrete for this Kafka-related dependency problem, therefore should be more notable for users. Oct 10, 2015 at 21:45
  • Thank you, I will, as soon as possible (when the time limitation expires). Oct 10, 2015 at 21:50
  • Recent versions of kafka also import org.slf4j/slf4j-log4j12; You may want to add that to your exclusion list
    – Ravi
    Jan 29, 2016 at 1:39
  • With Kafka 0.9.0.0, I found I also have to <exclusion><groupId>org.slf4j</groupId><artifactId>slf4j-log4j12</artifactId></exclusion> Feb 16, 2016 at 23:06
2

Do you use the "right" logging framework?

From https://storm.apache.org/2013/12/08/storm090-released.html

Logging Changes

Another important change in 0.9.0 has to do with logging. Storm has largely switched over to the slf4j API (backed by a logback logger implementation). Some Storm dependencies rely on the log4j API, so Storm currently depends on log4j-over-slf4j.

These changes have implications for existing topologies and topology components that use the log4j API.

In general, and when possible, Storm topologies and topology components should use the slf4j API for logging.

If you do not use the same logging framework as Storm, you need to include the used libraries into your jar file together with your topology code.

1
1

I have occured to the same problem when I included kafka like you, so I included kafka by another way according to the throw exception:

SLF4J: Detected both log4j-over-slf4j.jar AND slf4j-log4j12.jar on the class path, preempting StackOverflowError. SLF4J: See also http://www.slf4j.org/codes.html#log4jDelegationLoop for more details.

You can include kafka in the following way:

<dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <version>${kafka.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.