1

I want to tee and get the results from multiple shell commands connected in the pipeline. I made a simple example to explain the point. Suppose I wanna count the numbers of 'a', 'b' and 'c'.

echo "abcaabbcabc" | tee >(tr -dc 'a' | wc -m) >(tr -dc 'b' | wc -m) >(tr -dc 'c' | wc -m) > /dev/null

Then I tried to assign the result from each count to a shell variable, but they all end up empty.

echo "abcaabbcabc" | tee >(A=$(tr -dc 'a' | wc -m)) >(B=$(tr -dc 'b' | wc -m)) >(C=$(tr -dc 'c' | wc -m)) > /dev/null && echo $A $B $C

What is the right way to do it?

5
  • The variables are being created in subshells, they don't exist in the original shell.
    – Barmar
    Sep 5, 2019 at 21:33
  • Instead of using multiple process substitutions, why don't you just use a single awk script that counts each character and prints all the counts?
    – Barmar
    Sep 5, 2019 at 21:34
  • I see. But is there a way to retrieve those variables?
    – LEo
    Sep 5, 2019 at 21:39
  • it was just a simple example... what I'm doing is not counting the characters ;-)
    – LEo
    Sep 5, 2019 at 21:40
  • Whatever you're doing, you can probably put it into a single script, rather than lots of separate tee process substitutions.
    – Barmar
    Sep 5, 2019 at 21:43

2 Answers 2

2

Use files. They are the single most reliable solution. Any of the commands may need different time to run. There is no easy way to synchronize command redirections. Then most reliable way is to use a separate "entity" to collect all the data:

tmpa=$(mktemp) tmpb=$(mktemp) tmpc=$(mktemp)
trap 'rm "$tmpa" "$tmpb" "$tmpc"' EXIT

echo "abcaabbcabc" | 
     tee >(tr -dc 'a' | wc -m > "$tmpa") >(tr -dc 'b' | wc -m > "$tmpb") | 
     tr -dc 'c' | wc -m > "$tmpc"
A=$(<"$tmpa")
B=$(<"$tmpb")
C=$(<"$tmpc")

rm "$tmpa" "$tmpb" "$tmpc"
trap '' EXIT

Second way:

You can prepend the data from each stream with a custom prefix. Then sort all lines (basically, buffer them) on the prefix and then read them. The example script will generate only a single number from each process substitution, so it's easy to do:

read -r A B C < <(
  echo "abcaabbcabc" | 
  tee >(
    tr -dc 'a' | wc -m | sed 's/^/A /'
  ) >(
    tr -dc 'b' | wc -m | sed 's/^/B /'
  ) >(
    tr -dc 'c' | wc -m | sed 's/^/C /'
  ) >/dev/null |
  sort |
  cut -d' ' -f2 |
  paste -sd' '
)
echo A="$A" B="$B" C="$C"

Using temporary files with flock to synchronize the output of child processes could look like this:

tmpa=$(mktemp) tmpb=$(mktemp) tmpc=$(mktemp)
trap 'rm "$tmpa" "$tmpb" "$tmpc"' EXIT

echo "abcaabbcabc" | 
(
  flock 3
  flock 4
  flock 5

  tee >(
    tr -dc 'a' | wc -m | 
    { sleep 0.1; cat; } > "$tmpa"

    # unblock main thread
    flock -u 3
  ) >(
    tr -dc 'b' | wc -m | 
    { sleep 0.2; cat; } > "$tmpb"

    # unblock main thread
    flock -u 4
  ) >(
    tr -dc 'c' | wc -m | 
    { sleep 0.3; cat; } > "$tmpc"

    # unblock main thread
    flock -u 5
  ) >/dev/null

  # wait for subprocesses to finish
  # need to re-open the files to block on them
  (
    flock 3
    flock 4
    flock 5
  ) 3<"$tmpa" 4<"$tmpb" 5<"$tmpc"
) 3<"$tmpa" 4<"$tmpb" 5<"$tmpc"

A=$(<"$tmpa")
B=$(<"$tmpb")
C=$(<"$tmpc")

declare -p A B C
12
  • Curious, why set the trap only to manually remove the files and the trap later rather than just letting the trap handle removal. Sep 5, 2019 at 21:51
  • When the sub-processes take longer to compute, the tmp files are still empty when trying to retrieve the values.
    – LEo
    Sep 5, 2019 at 22:05
  • Indeed. For that you have to synchronize them. You could use flock. flock no temporary files and unlock them when finished. The "main" process wait in flock on all 3 files.
    – KamilCuk
    Sep 5, 2019 at 22:07
  • The solution with flock does not give any result. I guess it got trapped somewhere.
    – LEo
    Sep 5, 2019 at 23:08
  • 1
    Right. It just don't work on newer version of bash. Something changed or is it just a bug?
    – LEo
    Sep 6, 2019 at 17:49
0

You can use this featured letter frequency analysis

#!/usr/bin/env bash

declare -A letter_frequency
while read -r v k; do
    letter_frequency[$k]="$v"
done < <(
  grep -o '[[:alnum:]]' <<<"abcaabbcabc" |
    sort |
      uniq -c
)
for k in "${!letter_frequency[@]}"; do
  printf '%c = %d\n' "$k" "${letter_frequency[$k]}"
done

Output:

c = 3
b = 4
a = 4

Or to only assign $A, $B and $C as in your example:

#!/usr/bin/env bash

{
  read -r A _
  read -r B _
  read -r C _
}< <(
  grep -o '[[:alnum:]]' <<<"abcaabbcabc" |
    sort |
      uniq -c
)
printf 'a=%d\nb=%d\nc=%d\n' "$A" "$B" "$C"
  • grep -o '[[:alnum:]]': split each alphanumeric character on its own line
  • sort: sort lines of characters
  • uniq -c: count each instance and output count and character for each
  • < <( command group; ): the output of this command group is for stdin of the command group before

If you need to count occurrence of non-printable characters, newlines, spaces, tabs, you have to make all these commands output and deal with null delimited lists. It can sure be done with the GNU versions of these tools. I let it to you as an exercise.


Solution to the count arbitrary characters except null:

As demonstrated, works also with Unicode.

#!/usr/bin/env bash
declare -A character_frequency
declare -i v
while read -d '' -r -N 8 v && read -r -d '' -N 1 k; do
    character_frequency[$k]="$v"
done < <(
  grep --only-matching --null-data . <<<$'a¹bc✓   ✓\n\t\t\u263A☺ ☺   aabbcabc' |
    head --bytes -2 | # trim the newline added by grep
      sort --zero-terminated | # sort null delimited list
        uniq --count --zero-terminated # count occurences of char (null delim)
)
for k in "${!character_frequency[@]}"; do
  printf '%q = %d\n' "$k" "${character_frequency[$k]}"
done

Output:

$'\n' = 1
$'\t' = 2
☺ = 3
\  = 7
✓ = 2
¹ = 1
c = 3
b = 4
a = 4

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.