98

Is there a way to run shell commands without output buffering?

For example, hexdump file | ./my_script will only pass input from hexdump to my_script in buffered chunks, not line by line.

Actually I want to know a general solution how to make any command unbuffered?

2

5 Answers 5

173

Try stdbuf, included in GNU coreutils and thus virtually any Linux distro. This sets the buffer length for input, output and error to zero:

stdbuf -i0 -o0 -e0 command
7
  • 9
    This worked better than unbuffer for me. stdbuf passed any signals (SIGUSR2 in my case) I sent to it to the command (which is what I wanted to happen), while unbuffer didn't seem to want to.
    – ElDog
    Apr 11, 2015 at 13:13
  • 7
    stdbuf uses one the aforementioned LD_PRELOAD tricks to do this and hence does not work with statically linked or setuid executables. See this question for a discussion: stackoverflow.com/questions/13644024/… Jul 4, 2016 at 20:45
  • 11
    I found this very useful, but limited, as it does not run scripts implicitly. However, stdbuf -o0 bash runs a complete session with all my scripts and aliases available, and with no output buffering on any of the commands, which is exactly what I wanted. Buffering was of course restored on exit from this bash instance. I'm using Ubuntu 16.04.
    – AFH
    Jun 20, 2017 at 12:13
  • 2
    I liked this so much, I put this first in my .bash_profile. if [[ "$0" == "-bash" ]]; then exec /usr/bin/stdbuf -i0 -oL -eL /bin/bash -l; fi
    – IcarusNM
    Aug 24, 2019 at 20:19
  • 2
    Worked for me in an exec pattern for sending output to a logfile, with all output line buffered rather than buffering fully disabled: exec > >(stdbuf -i0 -oL -eL awk '{print strftime("%Y-%m-%d %H:%M:%S"), $0 }' | stdbuf -i0 -oL -eL tee "$LOGFILE") 2>&1
    – JinnKo
    Sep 6, 2019 at 12:48
37

The command unbuffer from the expect package disables the output buffering:
Ubuntu Manpage: unbuffer - unbuffer output

Example usage:

unbuffer hexdump file | ./my_script
2
  • 3
    unbuffer seems to merge stdout and stderr of the command though... (!)
    – olejorgenb
    May 10, 2016 at 19:57
  • 3
    @olejorgenb yes it merges stdout and stderr, that's by design: unbuffer uses expect, and expect is there to emulate the human in human-program interaction, and humans don't get to see whether an output is stdout or stderr.
    – toolforger
    Mar 31, 2019 at 9:26
25

AFAIK, you can't do it without ugly hacks. Writing to a pipe (or reading from it) automatically turns on full buffering and there is nothing you can do about it :-(. "Line buffering" (which is what you want) is only used when reading/writing a terminal. The ugly hacks exactly do this: They connect a program to a pseudo-terminal, so that the other tools in the pipe read/write from that terminal in line buffering mode. The whole problem is described here:

The page has also some suggestions (the aforementioned "ugly hacks") what to do, i.e. using unbuffer or pulling some tricks with LD_PRELOAD.

1
  • Thanks. I had never heard about pseudo terminals before.
    – bodacydo
    Aug 12, 2010 at 16:07
23

You could also use the script command to make the output of hexdump line-buffered (hexdump will be run in a pseudo terminal which tricks hexdump into thinking its writing its stdout to a terminal, and not to a pipe).

# cf. http://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe/
stty -echo -onlcr
script -q /dev/null hexdump file | ./my_script         # FreeBSD, Mac OS X
script -q -c "hexdump file" /dev/null | ./my_script    # Linux
stty echo onlcr
4
  • I used the -f parameter to flush, not sure if necessary but it worked. Oct 24, 2013 at 1:43
  • In my case, stdbuf failed, but script works like a charm. The program buffers the output when it is being piped to tee, but not when running in terminal. So, script works! Thanks! May 18, 2016 at 12:11
  • Very clever solution, and it works!! Thank you. :smile:
    – xmnboy
    Mar 1, 2017 at 1:26
  • Only one problem with that: per script's man page as referenced by this, "Certain interactive commands, such as vi(1), create garbage in the typescript file. Script works best with commands that do not manipulate the screen, the results are meant to emulate a hardcopy terminal." Nov 2, 2017 at 0:01
1

One should use grep or egrep "--line-buffered" options to solve this. no other tools needed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.