85

I have a source input, input.txt

a.txt
b.txt
c.txt

I want to feed these input into a program as the following:

my-program --file=a.txt --file=b.txt --file=c.txt

So I try to use xargs, but with no luck.

cat input.txt | xargs -i echo "my-program --file"{}

It gives

my-program --file=a.txt
my-program --file=b.txt
my-program --file=c.txt

But I want

my-program --file=a.txt --file=b.txt --file=c.txt

Any idea?

1
  • seq 1996 2022 | xargs calls for this when I'm trying to stripe my files by year on my file system. Jul 30, 2022 at 17:02

17 Answers 17

180

Don't listen to all of them. :) Just look at this example:

echo argument1 argument2 argument3 | xargs -l bash -c 'echo this is first:$0 second:$1 third:$2'

Output will be:

this is first:argument1 second:argument2 third:argument3
10
  • 20
    If you're trying to do this on OSX, use -L1 e.g. echo argument1 argument2 argument3 | xargs -L1 bash -c 'echo this is first:$0 second:$1 third:$2' | xargs Feb 1, 2016 at 18:14
  • 4
    But I think this answer is incorrect. The file has 3 lines but in this answer there are 3 argument in 1 line! Nov 15, 2017 at 14:15
  • 11
    why is the pipe to xargs for the second time needed in the end?
    – np20
    Jan 16, 2019 at 21:59
  • You have to use double $ signs in a makefile. Example: echo zero one | xargs -L1 -n 4 sh -c 'echo $$0' Feb 26, 2019 at 20:21
  • 1
    @LondonRob -l means "use at most MAX-LINES non-blank input lines per command line". default to 1 if not specified. bash has nothing to do with -l. We need bash for separating the one line string argument1 argument2 argument3 to 3 different arguments by space. xargs doesn't do it automatically. Jul 24, 2022 at 15:36
41

None of the solutions given so far deals correctly with file names containing space. Some even fail if the file names contain ' or ". If your input files are generated by users, you should be prepared for surprising file names.

GNU Parallel deals nicely with these file names and gives you (at least) 3 different solutions. If your program takes 3 and only 3 arguments then this will work:

(echo a1.txt; echo b1.txt; echo c1.txt;
 echo a2.txt; echo b2.txt; echo c2.txt;) |
parallel -N 3 my-program --file={1} --file={2} --file={3}

Or:

(echo a1.txt; echo b1.txt; echo c1.txt;
 echo a2.txt; echo b2.txt; echo c2.txt;) |
parallel -X -N 3 my-program --file={}

If, however, your program takes as many arguments as will fit on the command line:

(echo a1.txt; echo b1.txt; echo c1.txt;
 echo d1.txt; echo e1.txt; echo f1.txt;) |
parallel -X my-program --file={}

Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

4
  • 2
    Interesting - didn't know about GNU Parallel. Bash is a sane shell; you don't need to put a backslash after a pipe to tell it that the rest of the command is on the next line. Sep 25, 2010 at 19:03
  • 1
    GNU parallel is great. One word of warning: On some systems you need to give parallel the "--gnu" option to make it work sensibly. I never use xargs anymore. If you really don't want to parallelize you can always give it the "-j 1" option (1 job).
    – travc
    Oct 13, 2014 at 16:52
  • 2
    Utilite parallel is not always available, especially in plain unix and need to be installed if possible. There interesting solution below available with standart unix tools.
    – oklas
    Jan 13, 2017 at 10:28
23

How about:

echo $'a.txt\nb.txt\nc.txt' | xargs -n 3 sh -c '
   echo my-program --file="$1" --file="$2" --file="$3"
' argv0
12

It's simpler if you use two xargs invocations: 1st to transform each line into --file=..., 2nd to actually do the xargs thing ->

$ cat input.txt | xargs -I@ echo --file=@ | xargs echo my-program
my-program --file=a.txt --file=b.txt --file=c.txt
7

You can use sed to prefix --file= to each line and then call xargs:

sed -e 's/^/--file=/' input.txt | xargs my-program
1
  • 1
    After Google'ing this (four years later), I came up with my own (similar) solution... specifically, for this question, it would be more like: echo a.txt b.txt c.txt | xargs | sed 's/ / --file=/g' | xargs echo my-program --file My complicated solution was more to pipe filenames (such as Apache logs) in to a long command line. For example: ls /var/log/apache2/*access.log | xargs | sed 's/ / -f /g' | xargs echo myprogram -f
    – RVT
    Nov 24, 2014 at 1:59
5

Here is a solution using sed for three arguments, but is limited in that it applies the same transform to each argument:

cat input.txt | sed 's/^/--file=/g' | xargs -n3 my-program

Here's a method that will work for two args, but allows more flexibility:

cat input.txt | xargs -n 2 | xargs -I{} sh -c 'V="{}"; my-program -file=${V% *} -file=${V#* }'
5

I stumbled on a similar problem and found a solution which I think is nicer and cleaner than those presented so far.

The syntax for xargs that I have ended with would be (for your example):

xargs -I X echo --file=X

with a full command line being:

my-program $(cat input.txt | xargs -I X echo --file=X)

which will work as if

my-program --file=a.txt --file=b.txt --file=c.txt

was done (providing input.txt contains data from your example).


Actually, in my case I needed to first find the files and also needed them sorted so my command line looks like this:

my-program $(find base/path -name "some*pattern" -print0 | sort -z | xargs -0 -I X echo --files=X)

Few details that might not be clear (they were not for me):

  • some*pattern must be quoted since otherwise shell would expand it before passing to find.
  • -print0, then -z and finally -0 use null-separation to ensure proper handling of files with spaces or other wired names.

Note however that I didn't test it deeply yet. Though it seems to be working.

2
  • Thanks ! Using the $(...) to execute the command inline is much cleaner !
    – Didi Bear
    Jul 21 at 21:14
  • @DidiBear never use some_command $(...), it's subject to glob expansion and word splitting; check this: printf '%s\n' $(echo 'filename with * .txt'), does it output filename with * .txt?
    – Fravadona
    Aug 22 at 8:27
2

xargs doesn't work that way. Try:

  myprogram $(sed -e 's/^/--file=/' input.txt)
1

It's because echo prints a newline. Try something like

echo my-program `xargs --arg-file input.txt -i echo -n " --file "{}`
2
  • 1
    What happens when the input gets to 30,000 file names? Sep 25, 2010 at 19:05
  • 1
    Use subshell expansion, like echo my-program $(xargs --arg-file input.txt -i echo -n " --file "{})
    – thiagowfx
    Jan 9, 2014 at 16:05
1

I was looking for a solution for this exact problem and came to the conclution of coding a script in the midle.

to transform the standard output for the next example use the -n '\n' delimeter

example:

 user@mybox:~$ echo "file1.txt file2.txt" | xargs -n1 ScriptInTheMiddle.sh

 inside the ScriptInTheMidle.sh:
 !#/bin/bash
 var1=`echo $1 | cut -d ' ' -f1 `
 var2=`echo $1 | cut -d ' ' -f2 `
 myprogram  "--file1="$var1 "--file2="$var2 

For this solution to work you need to have a space between those arguments file1.txt and file2.txt, or whatever delimeter you choose, one more thing, inside the script make sure you check -f1 and -f2 as they mean "take the first word and take the second word" depending on the first delimeter's position found (delimeters could be ' ' ';' '.' whatever you wish between single quotes . Add as many parameters as you wish.

Problem solved using xargs, cut , and some bash scripting.

Cheers!

if you wanna pass by I have some useful tips http://hongouru.blogspot.com

1

Actually, it's relatively easy:

... | sed 's/^/--prefix=/g' | xargs echo | xargs -I PARAMS your_cmd PARAMS

The sed 's/^/--prefix=/g' is optional, in case you need to prefix each param with some --prefix=.

The xargs echo turns the list of param lines (one param in each line) into a list of params in a single line and the xargs -I PARAMS your_cmd PARAMS allows you to run a command, placing the params where ever you want.

So cat input.txt | sed 's/^/--file=/g' | xargs echo | xargs -I PARAMS my-program PARAMS does what you need (assuming all lines within input.txt are simple and qualify as a single param value each).

1

There is another nice way of doing this, if you do not know the number of files upront:

my-program $(find . -name '*.txt' -printf "--file=%p ")
0

Nobody has mentioned echoing out from a loop yet, so I'll put that in for completeness sake (it would be my second approach, the sed one being the first):

for line in $(< input.txt) ; do echo --file=$line ; done | xargs echo my-program
0

Old but this is a better answer:

cat input.txt | gsed "s/\(.*\)/\-\-file=\1/g" | tr '\n' ' ' | xargs my_program

# i like clean one liners gsed is just gnu sed to ensure syntax matches version brew install gsed or just sed if your on gnu linux already...

test it:

cat input.txt | gsed "s/\(.*\)/\-\-file=\1/g" | tr '\n' ' ' | xargs echo my_program
0

I discovered a more straightforward way. You can even customize the order of positional parameters to your liking, like in the demonstration below.

set $(cat input.txt)
my-program --file=$1 --file=$3 --file=$2

This works well in scenarios where you know the order and magnitude of arguments that are in the input.

1
  • $(cat input.txt) is subject to glob expansion and word-splitting. Don't use unquoted $(...) in a script!
    – Fravadona
    Aug 23 at 8:45
0

check this out too :

while IFS= read -r file; do
  my-program --file="$file"
done < input.txt

the while loop reads each line from the input.txt file and passes it as an argument to the my-program command with the --file flag!

0

Most answers don't take into account that some characters in the input lines can break xargs and unquoted $(...). Here are a few solutions that do the job correctly (i.e. robustly).


1. PURE BASH

You can load the input lines into a bash array and prepend --file= to each element with a parameter expansion. WARNING! You can't use this method for generating more than one argument per array element (for e.g. ‑f xxx.txt).

readarray -t files < input.txt

my-program "${files[@]/#/--file=}"

note: if your bash doesn't have readarray/mapfile then you can use IFS=$'\n' read -r -d '' -a files < input.txt instead.


2. AWK | XARGS -0

You can use awk for prepending --file= to each line and generate a NUL-delimited stream that'll be safely processed by xargs -0:

awk '{printf("--file=%s%c",$0,0)}' input.txt |
xargs -0 my-program

3. SED | XARGS

If you ever need to be POSIX compliant:

sed -e 's/"/"\\""/g' -e 's/.*/--file="&"/' input.txt |
xargs my-program

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.