7

I am having bit of an issue in logging to a file using log4j in storm .

Before submitting my topology , i.e in my main method I wrote some log statements and configured the logger using :

PropertyConfigurator.configure(myLog4jProperties)
  • Now when I run my topology using my executable jar in eclipse - its working fine and log files are being created as supposed.
    OR
    When i run my executable jar using "java -jar MyJarFile someOtherOptions", i can see log4j being configured and the files are formed correctly and logging is done on both files and console (as defined in my log4j.properties)

  • BUT when i run the same jar using "storm jar MyJarFile MyMainClass someOtherOptions" it is not being able to create and log into any of the files except on console.

I am talking about the logs I am printing BEFORE submitting my topology.

Is there any way to log my statements in a file while using storm ? I am not bound to use org.apache.log4j.

2
  • You are using a local cluster as a jar exec? When you use "storm jar ..." you are using the cluster mode. So did you check the file storm.log.properties on your storm install dir / log4j in your server ? When you submit a topology in cluster mode, storm is already running and writing logs, maybe you can't overwrite log4j at runtime, so modifying the storm log4j file before should work in all cases.
    – zenbeni
    Sep 5, 2014 at 8:54
  • @zenbeni I am actually running in local_mode. I can't find any log4j related properties in storm install directory. I am using apache-storm-0.9.2-incubating.
    – holmes840
    Sep 5, 2014 at 9:12

2 Answers 2

4

The storm framework uses its own logging. Your logs most likely will end up in the logs dir where storm is installed({Storm DIR}/logs). You can find the storm log properties in {Storm DIR}/logback/cluster.xml. It uses logback not log4j

2
  • This file defines properties for files in apache-storm-incubating-0.9.2.jar. Sure I can change the properties for that. But I want to change the logging properties for MyJarFile.jar. cluster.xml won't affect its logging. And also I am able to define properties for logs which are printed AFTER submitting my topology, i.e on supervisor nodes. I just can't get the logs (which are present in MyJarFile.jar) to print in a file BEFORE submitting my topology.
    – holmes840
    Sep 8, 2014 at 9:19
  • how can i make use of log4j over slf4j jar?
    – Mani
    Jun 17, 2015 at 10:44
4

I would recommend using SLF4J for your logging within storm. You probably could get LOG4J working but you will have additional setup to do on each node in the cluster. Since storm does this for you already, I don't really see the point.

In your project, include these maven dependencies (slf4j-simple for your unit tests!):

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-api</artifactId>
  <version>1.7.5</version>
</dependency>
<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-simple</artifactId>
  <version>1.7.5</version>
  <scope>test</scope>
</dependency>

Then, inside your topologies/spouts/bolts you just get the Logger:

private static final Logger LOG = LoggerFactory.getLogger(MySpout.class);

In my environment, anything INFO and above is logged to file and, more importantly, visible in the Storm UI. You just need to click on your spout/bolt and then click on the port number to go to the log viewer page.

If you want to get to the actual files then you can gather them off of each node (mine are in /var/log/storm/).

4
  • So when u say " get to the actual files" are u saying about worker-<port>.log files which by default goes into $STORM_HOME/logs directory . or u mean to say u configured ur log4j file with path as /var/log/storm ? Also , is there any way for Storm to not use Cluster.xml and use ur log file which u sent in the jar u deployed to the cluster.
    – Ankur Garg
    Oct 28, 2015 at 8:06
  • @AnkurGarg I meant the worker-port.log files. Anytime you call LOG.info or above inside your topology it will be logged to those worker logs and visible in the Storm UI.
    – Kit Menke
    Oct 28, 2015 at 14:21
  • Ok. Thanks for replying Kit :)
    – Ankur Garg
    Oct 28, 2015 at 14:23
  • my problem is I have about 10 applications running on master. Every times I want to grab log file for an application I have to go to each node , search for port in order to get worker-xxx.log. Is there any way to configure the name of log for each application ?
    – hoangpx
    Jul 26, 2017 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.