6

We are using a basic python log server based on BaseHTTPServer to aggregate our python logs on an ubunutu server. This solution has fulfilled our needs... until now. The number of programs dumping to this log server has grown and now the logger is crippling the system.

Now that we are back to the drawing board, we are considering using syslog.

Would it be advantageous to use syslog over other logging facilites.

Thanks for the help

3 Answers 3

5

Using syslog might be simple and fast, but it cannot give you full control over how your logs are aggregated.

Your main problem now is using BaseHTTPServer, which was never meant to be used on a production server, or for anything needing high performance.

I see two options:

  1. use a better http server with wsgi support, together with mini web framework (we are using gevent+bottle, but http://nichol.as/benchmark-of-python-web-servers is a well written comparison of solutions)
  2. use a message queue. this will mean more changes in your code but is a dedicated solution for your problem (and more efficient). (we use rabbitmq but check google or http://www.darkcoding.net/software/choosing-a-message-queue-for-python-on-ubuntu-on-a-vps for comparisons)

Edit: A dedicated solution that supports message queues is logbook. It can also be used as direct replacement of the standard library's logging module.

3
  • excellent links, I've slowly been beginning to look into both of those topics. I understand what you are saying about syslog being limited... but aren't there other flavors of syslog that open up better control? The logger we will using is in a distributed environment so control over aggregation and message distribution is a must. I have considered message services, but I am really looking some something more fully baked for logging. Have you any experience with scribe?
    – sbartell
    Oct 14, 2011 at 6:12
  • I knew the limitations of BaseHTTPServer, but threw caution to the wind thinking I would never see these limits. ha!
    – sbartell
    Oct 19, 2011 at 15:51
  • It depends on what is more important, pure speed or flexibility. If you only need speed than syslog-ng or scribe should be the best option. I personally prefer flexibility. Logbook is a newer project that support both python logging interface and message queues. I have edited the answer with a link
    – Mihai Stan
    Oct 19, 2011 at 17:36
4

The advantages of using syslog where available (all modern *nix systems, including Linux, FreeBSD, OS-X etc.) are numerous:

  • Performance is better: syslog is compiled C and most importantly it works as a separate process so all your logging operations become non-blocking to the applications, processes, and threads that make them
  • You can log from multiple processes/threads concurrently without worrying about locking. All logging is safely serialized for you so you don't lose data
  • You get standard sortable time-stamps on all logged lines for free
  • You get log rotation for free
  • You get severity level support for free (see man syslog)
  • You can call logging from any language with a C binding, which is virtually any language
  • You can trivially log from shell scripts or command line (via logger)
  • You don't need to reinvent the (how to log) wheel

The only disadvantage I can think of is that syslog is non portable (to non *nix systems), but if you're on any modern *nix, any alternative is more complicated and likely less reliable.

The concern of losing packets because syslog is using UDP may be valid, but in practice on a LAN, I've never found it to be an issue.

1

Not sure why HTTP is needed but this surely adds an overhead. Not to mention that the python implementation might not be up to the task. I can recommend syslog , but beware that using UDP for transport can result in message loss so TCP is highly recommended. syslog-ng and rsyslog can both handle TCP syslog. nxlog can too, and is not even bound to the syslog protocol.

1
  • Agreed, HTTP is unnecessary. It was used in the logging package I chose some time ago before logging in this project was a huge concern.
    – sbartell
    Oct 19, 2011 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.