5

We are building a web app, and my boss insists on using lucene, because the logs will be indexed and the admin can search and query it after.

But, IMHO, lucene is an indexing API, so, I think that the correct is use it to index things, not to store logs. The admin usually is a admin, not a dumb user. He could perfectly open the log files and search inside it with a CTRL+F like shortcut.

log4j is an log API, so, it means that it should be used to log things. I think that it is the correct choice.

Am I right? What do you think about it? Exists a API that could be used to query a log file?

thanks.

5
  • 1
    You wouldn't log with lucene. You'd log with a logger. You could index log files later, but I'm not convinced lucerne would add a lot of value unless your logs contain a lot of prose. DB logging, or post-processing, would likely be a better option. Jan 27, 2012 at 12:56
  • but, then the database will not slow down?
    – caarlos0
    Jan 28, 2012 at 0:27
  • 1
    If you log to a DB, sure, there will be the overhead of DB logging, just like anything else that writes to a DB. I have no way of knowing if your app/infrastructure can support it or not. The DB portion can also be loaded from log files, or handled as a separate task from the existing files, or... etc. Jan 28, 2012 at 0:39
  • it was what I was thinking. Thanks.
    – caarlos0
    Jan 30, 2012 at 12:33
  • 1
    You would use both solutions, use a logging framework to log, and then if you want to search the logs, index the logs and search them with Lucene. Think of it like a processing pipeline.
    – casperOne
    Jan 30, 2012 at 15:33

6 Answers 6

7

The Lucene Homepage states: "Apache Lucene(TM) is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform. "

It is definitely not a logging framework and I wont recommed it for this issue. Use log4j for logging or distributed loggig with Syslog4j.

6

I also agree that one should use log4j, generates log files and search using the ctrl + f, Lucene is not a good choice for this situation

2

Lucene is a text search engine API, not a logging framework. I would prefer slf4j. You could use a DB appender and then query your results as needed.

2

Log4j has a JDBC appender. This obviously would not update the index. If it is ok to update the index manually or scheduled, i.e. not having real time capability, then this is a quite easy setup. If this is not acceptable you could write a custom appender that uses Lucene or Hibernate Search. So it is possible and depending on the amount of search needed within the logs it may even be useful.

But: You add additional logging overhead and possible problems (database and index problems) with this approach. If searching through logfiles is a task so common that the log files have to be indexed something else smells.

2

Your boss is probably looking for something like splunk (centralized indexing and a nice query interface).

Disclaimer: it'sa comercial product and I don't work for them nor am I associated in any way with the product :)

0

I think the use case would be the same lucene to facilitate the search and query. In an application that has several different logs, for example, login logs, logs of actions for an module, logs of the action in another module, logs privileged actions.

And the query would be possible for users with permission within the system in the form of reports, so they can inspect the occurrences.

Still do not know Splunk, might solve this problem

Not the answer you're looking for? Browse other questions tagged or ask your own question.