2

I am having an issue with filtering a log file that is being written and writing the output to another file (if possible using tee, so I can see it working as it goes).

I can get it to output on stdout, but not write to a file, either using tee or >>. I can also get it to write to the file, but only if I drop the -f options from tail, which I need.

So, here is an overview of the commands:

  1. tail -f without writing to file: tail -f test.log | sed 's/a/b/' works
  2. tail writing to file: tail test.log | sed 's/a/b/' | tee -a a.txt works
  3. tail -f writing to file: tail -f test.log | sed 's/a/b/' | tee -a a.txt doesn't output on stdout nor writes to file.

I would like 3. to work.

6
  • 2
    How much data are you letting run through before you look at the output file? If it's not enough to flush the buffers... well, there's your problem. Once there's been enough input, or the tail process exits, sed will finish writing. Feb 13, 2018 at 14:08
  • 1
    This is BashFAQ #9. Feb 13, 2018 at 14:08
  • See also this duplicative question on Superuser. Feb 13, 2018 at 14:09
  • ...so, (2) works because your tail test.log hits EOF, so sed flushes its output when there's no more input. And (1) works because sed operates in line-buffered mode by default if its output is to the TTY. Feb 13, 2018 at 14:13
  • That said, if you try to "fix" the problem by forcing line-buffered or unbuffered mode, you're creating a performance penalty in return for that latency improvement (smaller writes == more context-switch operations from userspace to the kernel). Which may make sense for your workload -- just note that it the latency improvement is not free-of-charge. Feb 13, 2018 at 14:17

1 Answer 1

5

It's the sed buffering. Use sed -u. man sed:

-u, --unbuffered

          load  minimal amounts of data from the input files and flush the
          output buffers more often

And here's a test for it (creates files fooand bar):

$ for i in {1..3} ; do echo a $i ; sleep 1; done >> foo &
[1] 12218
$ tail -f foo | sed -u 's/a/b/' | tee -a bar
b 1
b 2
b 3

Be quick or increase the {1..3} to suit your skillz.

0

Not the answer you're looking for? Browse other questions tagged or ask your own question.