125

I currently have the current script.

#!/bin/bash
# script.sh

for i in {0..99}; do
   script-to-run.sh input/ output/ $i
done

I wish to run it in parallel using xargs. I have tried

script.sh | xargs -P8

But doing the above only executed once at the time. No luck with -n8 as well. Adding & at the end of the line to be executed in the script for loop would try to run the script 99 times at once. How do I execute the loop only 8 at the time, up to 100 total.

4
  • That is what I initially wanted to do, but had to resort to xargs because I am on Windows. I was not able to get GNU Parallel running on Windows
    – Olivier
    Feb 6, 2015 at 3:21
  • Is that script calling itself or did you just confuse the names when you asked here? Feb 6, 2015 at 3:24
  • Sorry, it should call another script. I will fix it
    – Olivier
    Feb 6, 2015 at 3:26
  • The answer to stackoverflow.com/questions/3321738/… is relevant here. Feb 6, 2015 at 3:28

3 Answers 3

185

From the xargs man page:

This manual page documents the GNU version of xargs. xargs reads items from the standard input, delimited by blanks (which can be protected with double or single quotes or a backslash) or newlines, and executes the command (default is /bin/echo) one or more times with any initial- arguments followed by items read from standard input. Blank lines on the standard input are ignored.

Which means that for your example xargs is waiting and collecting all of the output from your script and then running echo <that output>. Not exactly all that useful nor what you wanted.

The -n argument is how many items from the input to use with each command that gets run (nothing, by itself, about parallelism here).

To do what you want with xargs you would need to do something more like this (untested):

printf %s\\n {0..99} | xargs -n 1 -P 8 script-to-run.sh input/ output/

Which breaks down like this.

  • printf %s\\n {0..99} - Print one number per-line from 0 to 99.
  • Run xargs
    • taking at most one argument per run command line
    • and run up to eight processes at a time
6
  • 10
    Actually you don't need to put the arguments on separate lines; xargs word-splits. So echo {0..99} | would work just as well. <<<{0..99} doesn't seem to work; although <<<word is documented as brace-expanding word, it doesn't do so with any version of bash I have handy.
    – rici
    Feb 6, 2015 at 3:41
  • 1
    @rici Looks like a documentation bug then especially since the documentation for Here Documents doesn't mention brace expansion (and it doesn't happen there either in a quick test) though they also don't mention tilde expansion (which doesn't happen for << but does for <<< so *shrug*). The expansions that do and don't happen in here docs and here strings are a bit odd to my mind. Feb 6, 2015 at 3:49
  • 1
    How can you separate results from different runs with e.g. newlines? Oct 8, 2017 at 0:48
  • 8
    Demo: time head -12 <(yes "1") | xargs -n1 -P4 sleep will run 12 sleep 1 commands, 4 parallel. The command will take 3 seconds.
    – Walter A
    Oct 16, 2019 at 10:13
  • 3
    It's probably worth noting that -P 0 will use the number of cpus on the system
    – slf
    Dec 3, 2021 at 17:35
13

Here's an example running commands in parallel in conjuction with find:

find -name "*.wav" -print0 | xargs -0 -t -I % -P $(nproc) flac %

-print0 terminates filenames with a null byte rather than a newline so we can use -0 in xargs to prevent filenames with spaces being treated as two seperate arguments.

-t means verbose, makes xargs print every command it's executing, can be useful, remove if not needed.

-I % means replace occurrences of % in the command with arguments read from standard input.

-P $(nproc) means run a maximum of nproc instances of our command in parallel (nproc prints the number of available processing units).

flac % is our command, the -I % from earlier means this will become flac foo.wav

See also: Manual for xargs(1)

3
  • This answer was very helpful for me in that it explained how to use xargs rather than another tool 👍 Aug 9, 2022 at 17:59
  • Small footnote if you want to run multiple commands, you'll need to use bash -c. So for example maybe you wanted to rm the original file, you could do bash -c "flac \"%\" && rm \"%\"" Aug 10, 2022 at 21:25
  • It’s a bit easier to understand if you use the long options: for example --verbose instead of -t and --max-procs= instead of -P.
    – bfontaine
    Feb 9 at 14:54
8

You can use this simple 1 line command

seq 1 500 | xargs -n 1 -P 8 script-to-run.sh input/ output/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.