25

My team has inherited support for 100+ applications. The applications don't have any kind of common architecture, so the ones that do logging usually do it with custom code to local files or a local database, and it's all unmanaged. We want to change that.

We're slowly migrating the applications over to using log4net and standardising the types of things that are logged. The next question becomes: where should we send the logs?

I was thinking that it would be good to use a central SQL Server dedicated to receiving all the logs, which would provide easy maintenance (one place for backups/archiving) and provide the future possibility of some data mining and trend analysis.

Is that the best practice for this kind of thing, or is there some dedicated application logging server we should be looking at instead?

Update: I should have been more clear than just casually mentioning log4net and SQL Server: we're a Microsoft house, with most things written in .NET. UNIX solutions are no good for us.

2
  • Accepted answer to this question has wisdom but is badly out of date. RDBMS is a sub-optimal place for logs, as the benefit you gain in a degree of queryability is offset by poor performance/flexibility/cost. Try adding a new column or index to a table with a billion rows in it. The best solution I've come across is to use an inverted index against the original data. Solr or Elasticsearch are much better solutions than SQL for any non-trivial use. Look up the Logstash and Kibana plugins for Elasticsearch. Jul 29, 2014 at 9:00
  • agree wit richard, the answers provided are legacy little bit. it is better to use Application insights feature provided by MS. it can be used any language not just Microsoft languages. Application insights is diveded into 2 parts. SDk for instrumenting telemetry(logs data) and visualizing these logs in azure dash boards. SDK is opensource and you need to pay for azure visualization tools. if you don't want to pay then use Application insgists in your code and send these logs to elastic stack which is open source Aug 30, 2017 at 19:35

9 Answers 9

23

One world of caution: at 100+ apps in a big shop, with hundreds perhaps thousands of hosts running those apps, steer clear of anything that induces a tight coupling. This pretty much rules out connect directly to SQL Server or any database solution, because your application logging will be dependent on the availability of the log repository.

Availability of the central repository is a little more complicated than just 'if you can't connect, don't log it' because usually the most interesting events occur when there are problems, not when things go smooth. If your logging drops entries exactly when things turn interesting, it will never be trusted to solve incidents and as such will fail to gain traction and support for other stake holders (ie. the application owners).
If you decide that you can implement retention and retry failed log info delivery on your own, you are facing an uphill battle: it is not a trivial task and is much more complex than it sounds, starting from eficient and reliable storage of the retained information and ending with putting in place good retry and inteligent fallback logic.

You also must have an answer to the problems of authentication and security. Large orgs have multiple domains with various trust relations, employees venture in via VPN or Direct Access from home, some applications run unattended, some services are configured to run as local users, some machines are not joined to the domain etc etc. You better have an asnwer to the question how is the logging module of each application, everywhere is deployed, going to authenticate with the central repository (and what situations are going to be unsuported).

Ideally you would use an out-of-the box delivery mechanism for your logging module. MSMQ is probably the most appropiate fit: robust asynchronous reliable delivery (at least to the extent of most use cases), available on every Windows host when is installed (optional). Which is the major pain point, your applications will take a dependency on a non-default OS component.

The central repository storage has to be able to deliver the information requested, perhaps:

  • the application developers investigating incidents
  • customer support team investigating a lost transaction reported by a customer complaint
  • the security org doing forensics
  • the business managers demanding statistics, trends and aggregated info (BI).

The only storage capable of delivering this for any serious org (size, lifetime) is a relational engine, so probably SQL Server. Doing analysis over text files is really not going to go the distance.

So I would recommend a messaging based log transport/delivery (MSMQ) and a relational central repository (SQL Server) perhaps with aanalitycal component on top of it (Analysis Services Data Mining). as you see, this is clearly no small feat and it covers slightly more than just configuring log4net.

As for what to log, you say you already give a thought but I'd like to chime in my extra 2c: often times, specially on incident investigation, you will like the ability to request extra information. This means you would like to know certain files content from the incident machine, or some registry keys, or some performance counter values, or a full process dump. It is very useful to be able to request this information from the central repository interface, but is impractical to always collect this information, just in case is needed. Which implies there has to be some sort of bidirectional communication between the applictaion and the central repository, when the application reports an incident it can be asked to add extra information (eg a dump of the process at fault). There has to be a lot of infrastructure in place for something like this to occur, from the protocol between application logging and the central repository, to the ability of the central repository to recognize an incident repeat, to the capacity of the loggin library to collect the extra information required and not least the ability of an operator to mark incidents as needing extra information on next occurence.

I understand this answer goes probably seems overkill at the moment, but I was involved with this problem space for quite a while, I had looked at many online crash reports from Dr. Watson back in the day when I was with MS, and I can tell you that these requirement exists, they are valid concerns and when implemented the solution helps tremendously. Ultimately, you can't fix what you cannot measure. A large organisation depends on good management and monitoring of its application stock, including logging and auditing.

There are some third party vendors that offer solutions, some even integrated with log4net, like bugcollect.com (Full disclosure: that's my own company), Error Traffic Controller or Exceptioneer and other.

3
  • 3
    Was your 'One world of caution..' a deliberate pun? I mean, it clearly is more of a world than a word. ;-)
    – Wim
    Jan 19, 2010 at 12:33
  • 1
    @Wim: honest typo, but I'll leav it as it, makes more fun Jan 19, 2010 at 16:36
  • Exceptioneer has been down for a year or so. exceptron.com is a nice alternative. Disclaimer: I work on exceptron :D
    – kay.one
    Aug 22, 2012 at 23:54
9

Logstash + Elasticsearch + Kibana + Redis or RabbitMQ + NLog or Log4net

Storage + Search & Analytics: Elasticsearch
Collecting & Parsing : Logstash
Visualize: Kibana
Queue&Buffer: Redis
In Application: NLog

2
  • In terms of scalability, I really tend to move away from SQL in favour of ES. Redis doesn't really have a production-ready client for Windows, but I'm sure you'd be able to find another data structure server to act as a broker.
    – Lino Silva
    Feb 7, 2014 at 12:24
  • 1
    After few years, have you changed your logging infrastructe from this schema? As it combines 5 systems, have you elaborated somewhere the installation setup? Feb 22, 2017 at 12:40
5

The 1024 byte Syslog message length limit mentioned so far is misleading and incorrectly biases against Syslog-based solutions to the problem.

The limit for the obsolete "BSD Syslog Protocol" is indeed 1024 bytes.

The BSD syslog Protocol - 4.1 syslog Message Parts

The limit for the modern "Syslog Protocol" is implementation-dependent but MUST be at least 480 bytes, SHOULD be at least 2048 bytes, and MAY be even higher.

The BSD syslog Protocol - 6.1. Message Length

As an example, Rsyslog's configuration setting is called MaxMessageSize, which the documentation suggests can be set at least as high as 64kb.

rsyslog - Configuration Directives

That the asker's organisation is "a Microsoft house" where "UNIX solutions are no good" should not prevent less discriminatory readers from getting accurate information.

3

SQL would work, but I've used Splunk to aggregate logs. I was able to find some surprising information based on the way Splunk allows you to set up indexes on your data, and then use their query tools to make some nice graphs. You can download a basic version of it for free too.

2

As the other responses have pointed out, the closest thing to an industry standard is syslog. But don't despair because you're living in a Windows world. Kiwi have a syslog daemaon which runs on Windows, and it is free. Find out more.

update
As @MichaelFreidgeim points out, Kiwi now charge for their syslog daemon. However there are other free alternatives available. This other SO answer links to a couple of them.

2
  • syslog protocol has a limit 1024 bytes, which is not sufficient for general application logging. Also Kiwi is not free anymore. Aug 13, 2013 at 21:59
  • @MichaelFreidgeim - boy you must write some long messages. Personally I find I rarely need more than 256 characters for a single message.
    – APC
    Aug 14, 2013 at 6:22
1

On Unix, there's syslog.
Also, you might want to check out this case study.

1

If you have log4net log to the local EventViewer, you can mine these logs on a Windows 2008 box, see this centralized auditing article.

On that box, you can then easily import these events and provide some management and mining tools on top of it.

1
  • Alas, we're a government shop so everything is Windows 2003. :-/ Thanks anyway. Nov 15, 2009 at 14:56
1

As others already pointed out, directing logs from magnitude of apps and hosts directly to the database isn't a good idea. I just wanted to add one more advantage in favor of using dedicated centralized log server - it's decoupling of your apps from the log infrastructure. Since you're in .Net, there are couple of good choices - log4net and NLog. Both are very good products, but I particularly like the NLog, it proved to be much better performer with heavier loads, has much better configuration options and being actively maintained. Log4Net as far as I know hasn't been changed for a few years and have some issues, but still very robust solution as well. So, once you use such framework, you control on app level as to how, what and when it transmits its logs to centralized server. If at all.

Have a look at logFaces which was built specifically for a situations you describe - to aggregate logs from magnitude of apps and hosts providing centralized storage and source for analysis and monitoring. And doing all this not intrusively with zero changes in your existing code base. It will handle massive load of apps and hosts and let you specify what you want to do with the data. On the other hand, you've got very nice GUI for monitoring in real-time or digging into the data. You don't have to deal with databases directly at all. There are many databases to chose from - both SQL and NoSQL. BTW, RDBS are not the best performers with very large data stores. logFaces can work with MongoDB - this setup normally outperforms best traditional RDBS brands ten fold or so. Particularly when used with capped collections.

(for the disclosure, I am the author of logFaces)

0

If your running on *nix machines, the traditional solution is syslog.

1
  • 1
    syslog protocol has a limit 1024 bytes, which is not sufficient for general application logging Aug 13, 2013 at 21:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.