2

In our fancy ESB, logging of each request is done via a common infrastructure based on JMS based logging. Here is what happens in a nutshell:

  1. service gets a request service
  2. prepares some data in a LogData
  3. object service calls database
  4. time taken for db interaction is captured in LogData object
  5. service is ready to send response
  6. LogData object is sent to a messaging destination
  7. service sends response

Very rosey! yes for paper architects. Here is the actual issue: The JMS service provider sometimes becomes unavailable - due to a system level error or the software crashes. Then the service waits at the step (step no. 6) where it has to make JMS connection to send LogData object. Resulting in delayed response, thus leading to bad performance and user experience.

So that is the biggest shortcoming of "Distributed logging using JMS" touted by a lot of developer websites. Also note that the presance of LogData is kind of critical non-functional requirement. That means the messages are sent in persistent mode, leading to a wait until the JMS provider confirms receipt of message to the sender (the service in this case) - what to be blamed? immature design? Are there any success stories of implementing something like this?

1

2 Answers 2

3

Doing any logging in db/jms/socket/etc synchronously is asking for problems and a lot of them.

Implement it with logging into memory and async. dump to file/jms (depending if the JMS becomes available). A single background thread should do the trick for you. Having sync. logging could cause a lot of trouble in totally unexpected and innocent portions of the application.

I can't think of any possible success story of a sync. logging.

Edit. preferrably use the like of ConcurrentLinkedQueue to keep the LogData (I mean avoid any blocking, if possible to improve the perforamce/throughput)

2

Well, the main problem in your scenario is that your JMS message broker (service provider) has stability issues. Distributed logging via JMS will certainly work if it's treated as a critical part of your infrastructure that shouldn't be down but rather should be monitored and running in a clustered configuration that ensures that at least one instance is always up.

Async logging will not be much help if JMS broker is down. e.g. if you use log4j AsyncAppender, it will either become a blocking appender after the appender's buffer is filled up, or will throw away logging messages (if you have blocking=false).

That said, in my experience logging via JMS has almost always been an overkill, especially if it's the only reason you've set up a JMS broker.

If you want to "distribute" your logs across several boxes, there are several simple ways of doing it:

  • log to a file rolling over every hour or so (using rolling file appender in log4j or backlog) and then just rsync every hour to other boxes
  • write logs to an NFS diriectory or to a shared SAN directory mounted on all boxes (each log file will have to use instance specific name to avoid overriding each other)
  • log to a DB directly from the app - will probably want to pipe it through AsyncAppender though to avoid blocking. Hopefully your DB has higher availability than JMS provider

  • Ed Y.

7
  • Hey Ed. Thanks for commenting. So here is the scenario the software we use currently is Oracle's (sun's) Java JMS mq, which even when clustered acts weirdly when one of the brokers in the cluster goes down. Feb 25, 2011 at 15:23
  • @Ed Y, Logging must be fast and reliable. JMS and anything running out of the process is inherently unreliable to call. It depends on how you implement the async logger. I mention especially logging to a file if the JMS is unavailable, then dump the temporary storage (file) when JMS is available again. The latter could be due to router/switch issue, some software bugs, etc. Logging to a file is significantly more reliable then network based one. The only issues are running out of hard disk space or hard disk failure.
    – bestsss
    Feb 26, 2011 at 10:13
  • @ring bearer: I'm not familiar with Sun's JMS implementation, and to be honest haven't heard of anybody using it. However, from what I've seen, all JMS implementations provided with J2EE servers (like weblogic, WebSphere) have been either limited or flaky and are generally meant for "lightweight" messaging. If you're intent on using JMS for logging, look into getting a proper dedicated message broker like ActiveMQ, RabbitMQ or HornetQ. However, as I mentioned, I think it's an overkill and logging to a file locally and then replicating in some simple way is more sensible, less brittle approach.
    – edyavno
    Feb 26, 2011 at 23:48
  • @bestsss: as you saw, I actually think logging to a local file is a better approach anyway. I just wouldn't call it "async logger", more like file based queing if you're still sending it via JMS later.
    – edyavno
    Feb 26, 2011 at 23:50
  • @Ed Y, if you have some reason to log to JMS, it's a specification requirement or anything alike, the file logging just doesn't run the demands. For the record, I'd never log to JMS. What I offer is absolute async logger. The file based happens only if the JMS is down and the memory queue reaches some size. Otherwise file logging is not used either. It's a safety net for long unavailble JMS. Usually memory can take few gigas before it becomes an issue.
    – bestsss
    Feb 26, 2011 at 23:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.