135

Is there any way, in bash, to pipe STDERR through a filter before unifying it with STDOUT? That is, I want

STDOUT ────────────────┐
                       ├─────> terminal/file/whatever
STDERR ── [ filter ] ──┘

rather than

STDOUT ────┐
           ├────[ filter ]───> terminal/file/whatever
STDERR ────┘
2

7 Answers 7

80

TL;DR:

$ cmd 2> >(stderr-filter >&2)

Example:

% cat /non-existant 2> >(tr o X >&2)
cat: /nXn-existant: NX such file Xr directXry
%

This will work in both bash and zsh. Bash is pretty much ubiquitous these days, however, if you really do need a (really gnarly) solution for POSIX sh, then see here.


Explanation

By far, the easiest way to do this is to redirect STDERR via process substitution:

Process substitution allows a process’s input or output to be referred to using a filename. It takes the form of

>(list)

The process list is run asynchronously, and its input or output appears as a filename.

So what you get with process substituion is a filename.

Just like you could do:

$ cmd 2> filename

you can do

$ cmd 2> >(filter >&2)

The >&2 redirect's filter's STDOUT back to the original STDERR.

3
  • 7
    There is a caveat that goes along this answer: bash does not wait for the substituted process to complete, while the FD juggling to swap 1 and 2 does. This may be important. I got burned by this doing exec 2> >(while .. read .. echo) in a script running as a systemd service. journald captures fd2 of the service, and infers log level from an '<N>' prefix: prepend <2> to output lines, and you get ERROR level assigned to journal records. But sometimes systemd was swifter to kill the whole process group upon exit of the main one before it could log its last, the most important message! Jan 12, 2020 at 4:06
  • 3
    Nice simple solution. Caveat: be careful using this as part of a cron job; process substitution is not available in some shells used by cron. Jun 1, 2020 at 17:32
  • I've been using unix/linux command line for many decades and did not know this one. Thanks. Here's a example of ignoring stdout and filtering stderr cat file 1>/dev/null 2> >(grep -v <search-term> >&2) Mar 23 at 21:28
71

Here's an example, modeled after how to swap file descriptors in bash . The output of a.out is the following, without the 'STDXXX: ' prefix.

STDERR: stderr output
STDOUT: more regular

./a.out 3>&1 1>&2 2>&3 3>&- | sed 's/e/E/g'
more regular
stdErr output

Quoting from the above link:

  1. First save stdout as &3 (&1 is duped into 3)
  2. Next send stdout to stderr (&2 is duped into 1)
  3. Send stderr to &3 (stdout) (&3 is duped into 2)
  4. close &3 (&- is duped into 3)
2
  • 2
    It doesn't work for me. Finally I make it work with 3>&2 2>&1 1>&3-.
    – aleung
    Jan 25, 2018 at 10:37
  • 7
    Caution: this assumes FD 3 is not in use, and doesn't undo the swapping of file descriptors 1 and 2, so you can't go on to pipe this to yet another command. See this answer for further detail and work-around. For a much cleaner syntax for {ba,z}sh, see this answer.
    – Tom Hale
    Sep 30, 2018 at 6:24
31

TL;DR: (bash and zsh)

$ cmd 2> >(stderr-filter >&2)

Example:

% cat /non-existant 2> >(tr o X >&2)
cat: /nXn-existant: NX such file Xr directXry
%

Many answers on the StackExchange network have the form:

cat /non-existant 3>&1 1>&2 2>&3 3>&- | sed 's/e/E/g'

This has a built-in assumption: that file descriptor 3 isn't being used for something else.

Instead, use a named file descriptor, and {ba,z}sh will allocate the next available file descriptor >= 10:

cat /non-existant {tmp}>&1 1>&2 2>&$tmp {tmp}>&- | sed 's/e/E/g'

Note that named file descriptors aren't supported by POSIX sh.

The other issue with the above is that the command cannot be piped to further commands without again swapping STDOUT and STDERR back to their original values.

To allow onward piping in POSIX sh, (and still assuming FD 3 is not it use) it gets complicated:

(cmd 2>&1 >&3 3>&- | stderr-filter >&2 3>&-) 3>&1

So, Given the assumption and gnarly syntax of this, you're likely to be better off using the simpler bash/zsh syntax shown in the TL;DR above, and explained here.


practical demonstration, grepping only stderr:

$ ls -l . noexistABC noexistXYZ
ls: cannot access 'noexistABC': No such file or directory
ls: cannot access 'noexistXYZ': No such file or directory
.:
total 4
-rw-rw-r-- 1 frank frank    0 Aug 19 12:26 bar.txt
-rw-rw-r-- 1 frank frank    0 Aug 19 12:26 foo.txt
drwxrwxr-x 2 frank frank 4096 Aug 19 12:26 someFolder


$ ( ls -l . noexistABC noexistXYZ 2>&1 >&3 3>&- | grep ABC >&2 3>&-) 3>&1
.:
ls: cannot access 'noexistABC': No such file or directory
total 4
-rw-rw-r-- 1 frank frank    0 Aug 19 12:26 bar.txt
-rw-rw-r-- 1 frank frank    0 Aug 19 12:26 foo.txt
drwxrwxr-x 2 frank frank 4096 Aug 19 12:26 someFolder
1
  • Why post the same answer twice?
    – Rucent88
    Aug 21, 2021 at 16:24
25

A naive use of process substitution seems to allow filtering of stderr separately from stdout:

:; ( echo out ; echo err >&2 ) 2> >( sed s/^/e:/ >&2 )
out
e:err

Note that stderr comes out on stderr and stdout on stdout, which we can see by wrapping the whole thing in another subshell and redirecting to files o and e

( ( echo out ; echo err >&2 ) 2> >( sed s/^/e:/ >&2 ) ) 1>o 2>e
3
  • why the :; at the beginning? I tried the magic line without, and doesn't seem to make a difference.
    – Koshmaar
    Feb 8, 2016 at 17:29
  • 2
    Some people use $ as a command prompt. Then they often write shell examples like: $ cat /var/log/syslog | fgrep .... However, this line is not copy-pastable because of the $. :; looks like a prompt but is basically a shell no-op; so you can select and paste the whole line safely.
    – solidsnack
    Feb 12, 2016 at 9:44
  • 27
    Ok, but you could just omit the :; and the line would be also copy-pastable :) To me :; doesn't look like a prompt, it was not making the example clearer (separating commands from output) but confusing. Though I understand your point of view and let's not discuss syntax/conventions.
    – Koshmaar
    Feb 19, 2016 at 16:31
8

I find the use of bash process substitution easier to remember and use as it reflects the original intention almost verbatim. For example:

$ cat ./p
echo stdout
echo stderr >&2
$ ./p 2> >(sed -e 's/s/S/') | sed 's/t/T/'
sTdout
STderr

uses the first sed command as a filter on stderr only and the second sed command to modify the joined output.

Note that the white space after 2> is mandatory for the command to be parsed correctly.

7

The last part of this page of the Advanced Bash Scripting Guide is "redirecting only stderr to a pipe".

# Redirecting only stderr to a pipe.

exec 3>&1                              # Save current "value" of stdout.
ls -l 2>&1 >&3 3>&- | grep bad 3>&-    # Close fd 3 for 'grep' (but not 'ls').
#              ^^^^   ^^^^
exec 3>&-                              # Now close it for the remainder of the script.

# Thanks, S.C.

This may be what you want. If not, some other part of the ABSG should be able to help you, it is excellent.

1
  • We are somewhat hesitant to recommend the ABSG as a reference, as it mixes documentation, prescription, and opinion without clearly marking the differences. Some sections also have dubious content, although the one you link to seems fine.
    – tripleee
    Mar 7, 2017 at 7:59
2

Take a look at named pipes:

$ mkfifo err
$ cmd1 2>err |cat - err |cmd2
3
  • won't cat - err break the interspersing of stdout and stderr? Sep 1, 2010 at 14:15
  • @Martin - it depends. If cmd1, cat, or cmd2 buffers output, then you could see output out of sequence.
    – mob
    Sep 1, 2010 at 22:31
  • This won't run. The shell can't finish opening err for writing until cat opens it for reading, but cat won't open it until it finishes reading stdin, which it can't until cmd1 dies, but cmd1 can't start until it finishes opening err. This is a deadlock. fifo(7) documents this: "Normally, opening the FIFO blocks until the other end is opened also."
    – JoL
    Dec 6, 2021 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.