0

I have a legacy app running that sends its output to log files. Rather than tail and manage those log files, I want to capture the output before it hits the disk and send it off elsewhere on the network (using syslog, or fluentd, or logstash, etc.).

Is there any way I can capture the output without changing the application code itself? I do have the option of changing the process startup, so I can "wrap" it.

I thought of a few options:

  • Named pipe - it would work, but subject to pipe buffer and broken pipe limitations.
  • Standard redirect - really only works for known file descriptors, i.e. stdin, stdout, stderr / 0,1,2

Is there any sane way to interject my program so that the app thinks it is still writing to /var/log/myapp.log but instead is being sent to my log controller which can then do as it wishes?

18
  • 1
    "pipe buffer and broken pipe limitations" - I don't see any problems with that. could you elaborate on it? May 14, 2015 at 16:35
  • @KarolyHorvath you are not concerned with it? Think of a system pumping out thousands of log messages per second. With disk-based log file, it causes significant disk i/o and space usage. If I can capture it and send it off-host immediately, I solve both of those. But thousands of messages per second will overwhelm /dev/log or any other pipe, no?
    – deitch
    May 14, 2015 at 16:39
  • 1
    it shouldn't. if in doubt, test it. note: it's a pipe, not a disk based file. May 14, 2015 at 16:40
  • Of course, that is the idea!
    – deitch
    May 14, 2015 at 16:41
  • If the disk is not able to keep up with the logger, then the network will never be able to do so too.
    – Mr.Me
    May 14, 2015 at 16:41

3 Answers 3

1

Use named pipes. The pipe buffer size isn't something you should need to worry about and you shouldn't get SIGPIPES if the reader end doesn't die (or close the connection early).

Your reader could be as simple as

<named_pipe nc some_address some_port &

If you were to kill it while your writer was writing to it, the writer would get a SIGPIPE.

As for the buffer size concern, as long as ncing to some_address someport is faster or as fast as the rate at which data is being written to the named pipe, the writer process won't get blocked. If the writer is faster, it will when the buffer fills up. (But then, writing to disk or the network will get you blocked too).

You don't have to worry about the system-determined pipe buffer size; simply modify buffering at the reader end of the pipe (e.g., you can set the TCP output buffer size of nc, which will have the same effect as increasing the pipe buffer size) if the pipe buffer size is too small for you.

If you're launching the programs from bash and your app takes the log file as an argument, you can simply give it >( nc some_address some_port), which will create an anonymous named pipe for you.

2
  • Oooh. Like. I need to check if the logfile is hardcoded in. But if not, then the best thing to do is just use it in the argument. Would that work for env vars as well?
    – deitch
    May 14, 2015 at 17:19
  • The anonymous named pipes are only open for the current pipeline foo=>(cat) echo hi > $foo could theoretically work, but in practice it doesn't; you'll need to use real named fifos if you're passing args via the environment. May 14, 2015 at 19:44
0

On linux you can create a chroot environment for your process. All file access is now local to your given root path which enables you to prepare pipes/fifos to all files you want to replace. This may take a while to get the whole environment running but it makes it possible to run the process in a fully controlled environment.

0

As PSkocik said, you could try using 'nc' to send off log right away, but keep in mind 'nc' stops itself when it receives EOF character. You could try using tail to follow log and send it to remote host

tail -f named-pipe-log-file|nc remote-host port

On remote host, have 'nc' running in listen mode,writing into local log using 'nc -lkd port > log'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.