473

How can I make xargs execute the command exactly once for each line of input given? It's default behavior is to chunk the lines and execute the command once, passing multiple lines to each instance.

From http://en.wikipedia.org/wiki/Xargs:

find /path -type f -print0 | xargs -0 rm

In this example, find feeds the input of xargs with a long list of file names. xargs then splits this list into sublists and calls rm once for every sublist. This is more efficient than this functionally equivalent version:

find /path -type f -exec rm '{}' \;

I know that find has the "exec" flag. I am just quoting an illustrative example from another resource.

4
  • 6
    In the example you provide, find /path -type f -delete would be even more efficient :)
    – tzot
    Dec 14, 2011 at 8:34
  • 1
    try not to use xargs...
    – Naib
    Mar 2, 2015 at 10:04
  • 7
    OP, I know this question is very old, but it still comes up on Google and IMHO the accepted answer is wrong. See my longer answer below.
    – Tobia
    Mar 2, 2015 at 10:11
  • 1
    Please consider switching your accept to @Tobia's answer which is much better. The accepted answer does not handle spaces in names and doesn't allow multiple arguments to the xargs command which is one of the main features of xargs.
    – Gray
    May 8, 2015 at 17:35

13 Answers 13

494

The following will only work if you do not have spaces in your input:

xargs -L 1
xargs --max-lines=1 # synonym for the -L option

from the man page:

-L max-lines
          Use at most max-lines nonblank input lines per command line.
          Trailing blanks cause an input line to be logically continued  on
          the next input line.  Implies -x.
12
  • 15
    For me it can out as xargs -n 1 as the one you gave showed "argument list too long".
    – Wernight
    Sep 20, 2011 at 9:14
  • 24
    If MAX-LINES is omitted it defaults to 1, thus xargs -l is enough. See info xargs.
    – Thor
    Jun 30, 2012 at 0:50
  • 3
    @Wernight: "-n1" does not give 1 invocation per input line. maybe your input line was too long. demo: echo "foo bar" | xargs -n1 echo. hence if you pipe in stuff like 'ls', it won't handle spaces well. Aug 5, 2014 at 5:44
  • 11
    This is wrong. -L 1 does not answer the original question, and -n 1 does so only in one of the possible interpretations. See my long answer below.
    – Tobia
    Mar 2, 2015 at 10:05
  • 3
    @Tobia: It answers the original question, which was quite specifically about lines of input. That's exactly what -L 1 does. To me, the OP seemed to be clearly trying to avoid the default chunking behaviour, and since this was accepted I assume I was right. Your answer addresses a slightly different use-case where you want the chunking behaviour as well.
    – Draemon
    Mar 2, 2015 at 20:02
329

It seems to me all existing answers on this page are wrong, including the one marked as correct. That stems from the fact that the question is ambiguously worded.

Summary:   If you want to execute the command "exactly once for each line of input," passing the entire line (without newline) to the command as a single argument, then this is the best UNIX-compatible way to do it:

... | tr '\n' '\0' | xargs -0 -n1 ...

If you are using GNU xargs and don't need to be compatible with all other UNIX's (FreeBSD, Mac OS X, etc.) then you can use the GNU-specific option -d:

... | xargs -d\\n -n1 ...

Now for the long explanation…


There are two issues to take into account when using xargs:

  1. how does it split the input into "arguments"; and
  2. how many arguments to pass the child command at a time.

To test xargs' behavior, we need an utility that shows how many times it's being executed and with how many arguments. I don't know if there is a standard utility to do that, but we can code it quite easily in bash:

#!/bin/bash
echo -n "-> "; for a in "$@"; do echo -n "\"$a\" "; done; echo

Assuming you save it as show in your current directory and make it executable, here is how it works:

$ ./show one two 'three and four'
-> "one" "two" "three and four" 

Now, if the original question is really about point 2. above (as I think it is, after reading it a few times over) and it is to be read like this (changes in bold):

How can I make xargs execute the command exactly once for each argument of input given? Its default behavior is to chunk the input into arguments and execute the command as few times as possible, passing multiple arguments to each instance.

then the answer is -n 1.

Let's compare xargs' default behavior, which splits the input around whitespace and calls the command as few times as possible:

$ echo one two 'three and four' | xargs ./show 
-> "one" "two" "three" "and" "four" 

and its behavior with -n 1:

$ echo one two 'three and four' | xargs -n 1 ./show 
-> "one" 
-> "two" 
-> "three" 
-> "and" 
-> "four" 

If, on the other hand, the original question was about point 1. input splitting and it was to be read like this (many people coming here seem to think that's the case, or are confusing the two issues):

How can I make xargs execute the command with exactly one argument for each line of input given? Its default behavior is to chunk the lines around whitespace.

then the answer is more subtle.

One would think that -L 1 could be of help, but it turns out it doesn't change argument parsing. It only executes the command once for each input line, with as many arguments as were there on that input line:

$ echo $'one\ntwo\nthree and four' | xargs -L 1 ./show 
-> "one" 
-> "two" 
-> "three" "and" "four" 

Not only that, but if a line ends with whitespace, it is appended to the next:

$ echo $'one \ntwo\nthree and four' | xargs -L 1 ./show 
-> "one" "two" 
-> "three" "and" "four" 

Clearly, -L is not about changing the way xargs splits the input into arguments.

The only argument that does so in a cross-platform fashion (excluding GNU extensions) is -0, which splits the input around NUL bytes.

Then, it's just a matter of translating newlines to NUL with the help of tr:

$ echo $'one \ntwo\nthree and four' | tr '\n' '\0' | xargs -0 ./show 
-> "one " "two" "three and four" 

Now the argument parsing looks all right, including the trailing whitespace.

Finally, if you combine this technique with -n 1, you get exactly one command execution per input line, whatever input you have, which may be yet another way to look at the original question (possibly the most intuitive, given the title):

$ echo $'one \ntwo\nthree and four' | tr '\n' '\0' | xargs -0 -n1 ./show 
-> "one " 
-> "two" 
-> "three and four" 

As mentioned above, if you are using GNU xargs you can replace the tr with the GNU-specific option -d:

$ echo $'one \ntwo\nthree and four' | xargs -d\\n -n1 ./show 
-> "one " 
-> "two" 
-> "three and four" 
12
  • looks like this is the better answer. however, i still don't quite understand what is the difference between -L and -n... can you explain a bit more?
    – olala
    Jan 19, 2016 at 21:22
  • 5
    @olala -L executes the command once per input line (but a space at the end of a line joins it to the next line, and the line is still split into arguments according to whitespace); while -n executes the command once per input argument. If you count the number of -> in the output examples, those are the number of times the script ./show is executed.
    – Tobia
    Jan 20, 2016 at 11:05
  • 5
    GNU xargs may or may not have useful extensions that allow you to do away with tr It has such a very useful extension; from xargs --help-d, --delimiter=CHARACTER items in input stream are separated by CHARACTER, not by whitespace; disables quote and backslash processing and logical EOF processing Mar 24, 2016 at 21:39
  • 1
    @Tobia: Do you mind if I edit your answer to include xargs -d '\n' as a shortcut for tr '\n' '\0' | xargs -0 that works on "some" systems?
    – krlmlr
    Nov 15, 2019 at 6:36
  • 1
    @krlmlr I added it.
    – Tobia
    Oct 8, 2020 at 14:44
25

If you want to run the command for every line (i.e. result) coming from find, then what do you need the xargs for?

Try:

find path -type f -exec your-command {} \;

where the literal {} gets substituted by the filename and the literal \; is needed for find to know that the custom command ends there.

EDIT:

(after the edit of your question clarifying that you know about -exec)

From man xargs:

-L max-lines
Use at most max-lines nonblank input lines per command line. Trailing blanks cause an input line to be logically continued on the next input line. Implies -x.

Note that filenames ending in blanks would cause you trouble if you use xargs:

$ mkdir /tmp/bax; cd /tmp/bax
$ touch a\  b c\  c
$ find . -type f -print | xargs -L1 wc -l
0 ./c
0 ./c
0 total
0 ./b
wc: ./a: No such file or directory

So if you don't care about the -exec option, you better use -print0 and -0:

$ find . -type f -print0 | xargs -0L1 wc -l
0 ./c
0 ./c
0 ./b
0 ./a
1
  • I found it didactically interesting to prepend the command with echo 'find . -type f -print0 | xargs -0L1 echo wc -l'. You can then easyly preview what command xargs will generate for example see the difference if you use '-0L2'
    – grenix
    Oct 27, 2021 at 17:01
23

How can I make xargs execute the command exactly once for each line of input given?

-L 1 is the simple solution but it does not work if any of the files contain spaces in them. This is a key function of find's -print0 argument – to separate the arguments by '\0' character instead of whitespace. Here's an example:

echo "file with space.txt" | xargs -L 1 ls
ls: file: No such file or directory
ls: with: No such file or directory
ls: space.txt: No such file or directory

A better solution is to use tr to convert newlines to null (\0) characters, and then use the xargs -0 argument. Here's an example:

echo "file with space.txt" | tr '\n' '\0' | xargs -0 ls
file with space.txt

If you then need to limit the number of calls you can use the -n 1 argument to make one call to the program for each input:

echo "file with space.txt" | tr '\n' '\0' | xargs -0 -n 1 ls

This also allows you to filter the output of find before converting the breaks into nulls.

find . -name \*.xml | grep -v /target/ | tr '\n' '\0' | xargs -0 tar -cf xml.tar
3
  • 1
    There is a syntax error in the second code block tr '\n' '\0\ => tr '\n' '\0', I tried to fix this but "Edits must be at least 6 characters" (this seems as stupid as git refusing to commit because my change was less than 6 chars)
    – htaccess
    Apr 12, 2018 at 21:52
  • 1
    What does this mean: "Another problem with using -L also is that it doesn't allow multiple arguments for each xargs command call. " ?
    – Moberg
    Feb 4, 2019 at 7:37
  • I've improved my answer to remove that extraneous info @Moberg.
    – Gray
    Feb 4, 2019 at 13:45
21

These two ways also work, and will work for other commands that are not using find!

xargs -I '{}' rm '{}'
xargs -i rm '{}'

example use case:

find . -name "*.pyc" | xargs -i rm '{}'

will delete all pyc files under this directory even if the pyc files contain spaces.

5
  • This issues one utility call for every element which isn't optimal.
    – Gray
    May 7, 2015 at 15:06
  • @Gray: What's a "utility call"? May 23, 2022 at 21:36
  • This answer works better than the top-voted answers. May 23, 2022 at 21:36
  • One call to the application (rm in this example) per file as opposed to a number of arguments. See my answer for a different solution.
    – Gray
    May 27, 2022 at 21:56
  • is this portable (Posix)?
    – 3dGrabber
    Feb 26 at 11:15
12

Another alternative...

find /path -type f | while read ln; do echo "processing $ln"; done
0
9
find path -type f | xargs -L1 command 

is all you need.

0
4

The following command will find all the files (-type f) in /path and then copy them using cp to the current folder. Note the use if -I % to specify a placeholder character in the cp command line so that arguments can be placed after the file name.

find /path -type f -print0 | xargs -0 -I % cp % .

Tested with xargs (GNU findutils) 4.4.0

3

You can limit the number of lines, or arguments (if there are spaces between each argument) using the --max-lines or --max-args flags, respectively.

  -L max-lines
         Use at most max-lines nonblank input lines per command line.  Trailing blanks cause an input line to be logically continued on the next  input
         line.  Implies -x.

  --max-lines[=max-lines], -l[max-lines]
         Synonym  for  the -L option.  Unlike -L, the max-lines argument is optional.  If max-args is not specified, it defaults to one.  The -l option
         is deprecated since the POSIX standard specifies -L instead.

  --max-args=max-args, -n max-args
         Use at most max-args arguments per command line.  Fewer than max-args arguments will be used if the size (see  the  -s  option)  is  exceeded,
         unless the -x option is given, in which case xargs will exit.
1

@Draemon answers seems to be right with "-0" even with space in the file.

I was trying the xargs command and I found that "-0" works perfectly with "-L". even the spaces are treated (if input was null terminated ). the following is an example :

#touch "file with space"
#touch "file1"
#touch "file2"

The following will split the nulls and execute the command on each argument in the list :

 #find . -name 'file*' -print0 | xargs -0 -L1
./file with space
./file1
./file2

so -L1 will execute the argument on each null terminated character if used with "-0". To see the difference try :

 #find . -name 'file*' -print0 | xargs -0 | xargs -L1
 ./file with space ./file1 ./file2

even this will execute once :

 #find . -name 'file*' -print0  | xargs -0  | xargs -0 -L1
./file with space ./file1 ./file2

The command will execute once as the "-L" now doesn't split on null byte. you need to provide both "-0" and "-L" to work.

0

It seems I don't have enough reputation to add a comment to Tobia's answer above, so I am adding this "answer" to help those of us wanting to experiment with xargs the same way on the Windows platforms.

Here is a windows batch file that does the same thing as Tobia's quickly coded "show" script:

@echo off
REM
REM  cool trick of using "set" to echo without new line
REM  (from:  http://www.psteiner.com/2012/05/windows-batch-echo-without-new-line.html)
REM
if "%~1" == "" (
    exit /b
)

<nul set /p=Args:  "%~1"
shift

:start
if not "%~1" == "" (
    <nul set /p=, "%~1"
    shift
    goto start
)
echo.
-3

In your example, the point of piping the output of find to xargs is that the standard behavior of find's -exec option is to execute the command once for each found file. If you're using find, and you want its standard behavior, then the answer is simple - don't use xargs to begin with.

1
  • Actually, what I can imply from the OP's edits is that the input data have nothing to do with find, and that is why they don't prefer the -exec option.
    – tzot
    Dec 24, 2009 at 16:11
-3

execute ant task clean-all on every build.xml on current or sub-folder.

find . -name 'build.xml' -exec ant -f {} clean-all \;
1
  • Not everyone has ant installed.
    – Gray
    May 7, 2015 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.