13

Inside a BASH script we can have multiple processes running in background which intercommunicate using named pipes, FIFOs registered on the filesystem. An example of this could be:

#!/bin/bash
mkfifo FIFO

# BG process 1
while :; do echo x; done & >FIFO

# BG process 2
while :; do read; done & <FIFO

exit

I wonder if it's possible to do the same intercommunication between background processes of a script without using a FIFO on filesystem, maybe with some kind of file-descriptor redirection.

5
  • 1
    You can use mktemp to come up with a unique name Commented Jun 2, 2012 at 14:14
  • I'd prefer not to have to manage the filesystem at all. Also, with a fast paced creation/deletion of files or FIFOs, the filesystem interaction drops performance.
    – davide
    Commented Jun 2, 2012 at 14:37
  • Sadly, bourne shell doesn't have very flexible handling of these things. IPC is definitely possible without FIFOs though, since they're pretty recent: the underlying calls are to socketpair/pipe. Commented Jun 2, 2012 at 15:38
  • @NicholasWilson: This sounds interesting. Could you point me to a doc reference or make a very-brief example?
    – davide
    Commented Jun 2, 2012 at 15:48
  • 1
    Sorry, rubbish wording there. What I was trying to say: FIFOs are recent, so IPC is clearly possible without them. If you're not using FIFOs, the underlying calls will be to socketpair (or pipe). Bash has a lot of funny limitations on file descriptors, so it's usually better the moment these issues crop up to go straight to something like python or (my preference) a C wrapper that does the calls you want to achieve directly. Commented Jun 2, 2012 at 15:55

5 Answers 5

18

Here's an example that runs two subprocesses implemented as functions of the same shell-script... One subprocess generates numbers 1...5 (sleeps in between prints), the second one reads from a fixed filedescriptor (5, to which STDOUT of the first FD is redirected to), multiplies by 2 and prints again. The main process redirects STDOUT of that second process to another fixed filedescriptor (6) and later on reads from that one in the loop.

It works basically the same as you'd do in C-code with fd pairs created by the pipe(2) system call. To understand what's happening run the script under strace -f!

Bash Version is 4.2.24(1) running on Ubuntu/x86.

[ubuntu /home/chris]
$ bash --version
GNU bash, version 4.2.24(1)-release (i686-pc-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Output of script:

[ubuntu /home/chris]
$ ./read_from_fd.sh
Got number 2.
Got number 4.
Got number 6.
Got number 8.
Got number 10.

Source code:

#!/bin/bash

# Generate data, output to STDOUT.
generate_five_numbers() {
        for n in `seq 5` ; do
                echo $n
                sleep 2
        done
}

# Read data from FD#5, multiply by two, output to STDOUT.
multiply_number_from_fd5_by_two() {
        while read n <&5 ; do
                echo "$(( $n * 2 ))"
        done
}

# choose your FD number wisely ;-)

# run generator with its output dup'ed to FD #5
exec 5< <( generate_five_numbers )

# run multiplyier (reading from fd 5) with output dup'ed to FD #6
exec 6< <( multiply_number_from_fd5_by_two )

# read numbers from fd 6
while read n <&6 ; do
        echo "Got number $n."
done

Process tree while running:

──read_from_fd.sh(8118)─┬─read_from_fd.sh(8119)───sleep(8123)
                        └─read_from_fd.sh(8120)
1
  • I suppose this technically answers the question. Though I do not believe this method can support bi-directional communication, so this is little better than using a standard bash pipe e.g. generate_five_numbers | 0<&5 multiply_number_from_fd5_by_two.
    – DBear
    Commented Jan 20, 2022 at 3:15
6

Bash 4 has coprocesses.

You can also use anonymous named pipes, aka process substitution in Bash 2, 3 or 4.

3
  • Correct me if I'm wrong, but I think by that way you can't redirect both STDIN and STDOUT of a process at the same time. Also, Bash supports only one coprocess per istance; what if we have many background processes that need intercommunication?
    – davide
    Commented Jun 2, 2012 at 14:55
  • @davide: Then I would say you should use something other than Bash. By the way, the reference manual says "with a two-way pipe established between the executing shell and the coprocess" [emphasis mine]. Commented Jun 2, 2012 at 15:27
  • Sorry Dennis I wasn't clear. I mean that I'm not sure if a two-ways redirection is possible with process substitution. Of course coproc can, but unfortunately Bash limits its usage to only one per instance.
    – davide
    Commented Jun 2, 2012 at 15:44
3

You could use nc (aka netcat) which allows connecting a script's standard streams to a network socket. Of course it also works on localhost, so you can use it for IPC between scripts. The bonus is the possibility to have scripts running on different hosts, which is not possible with FIFOs (OK, maybe on NFS it is, but that would be rather cumbersome to set up unless you already have the NFS in place).

4
  • Yes that would work too, but wouldn't it require me (or the end user) to pay attention not to use network ports (they may be hundreds) already taken by other services running on the same computer?
    – davide
    Commented Jun 4, 2012 at 17:42
  • Possibly yes, but if you choose a random port from the safe range (not an ephemeral port ) it's not very likely. Additionally, if you tell nc to listen on a port which is already taken, it will immediately return with an error, after which you can try the next port and so on. Of course you would then have to communicate the port to the other app via another channel, like through a file. Probably you would only need to this once on a particular machine, though. Either solution should work well enough to be practical. Commented Jun 4, 2012 at 21:11
  • Also, firewall settings could complicate this method of communication. Commented Dec 23, 2016 at 12:49
  • 1
    While the firewall could technically mess things up, it is extremely unlikely. Firewalls generally leave the loopback interface open because a lot of programs will break if the loopback is blocked.
    – DBear
    Commented Jan 20, 2022 at 3:20
2

I just want to point out that ugly hacks didn't wish to be born that way.

Part which receives data:

node -e "require('net').createServer(function(s){s.pipe(process.stdout)}).listen(1337)"

Part which sends data:

echo "write clean code they said" > /dev/tcp/localhost/1337
echo "it will pay off they said" > /dev/tcp/localhost/1337

Works even in MSysGit's Bash for Windows, to my surprise.

1

Have you considered the use of signals? If the only thing you need is to trigger an event (without passing arguments), using kill and trap works perfectly (be careful of the semantics though, use SIGUSR1 for instance).

You might need to rework the logic though, as in the example below:

subprocess_finished()
{
    np=$( jobs -p | wc -l )
}

start_processing()
{
    myfile="$1"
    # DO SOMETHING HERE!!
    kill -SIGUSR1 $2
}

CPUS=$( lscpu | grep "^CPU(s):" | rev | cut -f 1 -d ' ' | rev )
POLLPERIOD=5  # 5s between each poll
np=0
trap subprocess_finished SIGUSR1

for myfile in *
do 
        start_processing "$myfile" $$ &
        np=$( jobs -p | wc -l )
        echo "$( date +'%Y-%m-%d %H:%M:%S' ) [$!] Starting #$np on $CPUS: $myfile"

        if [ $np -eq $CPUS ] 
        then
            # Wait for one CPU to be free
            trap subprocess_finished SIGUSR1
            while [ $np -eq $CPUS ]
            do
                sleep $POLLPERIOD
            done
        fi
    done
done

# wait for the last subprocesses
while [ ! -z "$( jobs -rp )" ]
do
    sleep $POLLPERIOD
done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.