57

I am new to bash and I am trying to understand the use of xargs, which is still not clear for me. For example:

history | grep ls

Here I am searching for the command ls in my history. In this command, I did not use xargs and it worked fine.

find /etc - name "*.txt" | xargs ls -l

I this one, I had to use xargs but I still can not understand the difference and I am not able to decide correctly when to use xargs and when not.

1
  • 2
    The bottom line is there is a limit to the size of the command line commands can take. (it is a hardcoded number usually 128KiB) see What defines the maximum size for a command single argument?. When your command line exceeds that size, your alternative is to use xargs which will break that up and process the command for you. Commented Feb 24, 2016 at 20:44

6 Answers 6

59

xargs can be used when you need to take the output from one command and use it as an argument to another. A pipe alone takes the output from one command and sends it to the input stream of another.

So in your first example, grep can accept data from standard input, rather than as an argument and xargs is not needed.

xargs takes data from standard input and executes a command. By default, the data is appended to the end of the command as an argument. It can be inserted anywhere however, using a placeholder for the input. The traditional placeholder is {}; using that, your second command might then be written as:

find /etc -name "*.txt" | xargs -I {} ls -l {}

If you have 3 text files in /etc you'll get a full directory listing of each. Of course, you could just have easily written ls -l /etc/*.txt and saved the trouble.

Another example lets you rename those files, and shows when the placeholder would be required. {} has to be used twice to specify the source and destination names.

find /etc -name "*.txt" | xargs -I {} mv {} {}.bak

These are both bad examples, and will break as soon as you have a filename containing whitespace. You can work around that by telling find to separate filenames with a null character.

find /etc -print0 -name "*.txt" | xargs -I {} -0 mv {} {}.bak

My personal opinion is that there are almost always alternatives to using xargs (such as the -exec argument to find, or the process substitution operator) and you will be better served by learning those.

5
  • In case you find it useful, I use this: gitlab.com/es20490446e/args Commented Jun 18, 2020 at 15:54
  • can you explain what -I {} is in xargs -I {}? Commented Nov 9, 2021 at 15:26
  • @MohammadFaisal did you read the second paragraph of the answer? It specifies the placeholder.
    – miken32
    Commented Nov 9, 2021 at 17:52
  • when do we need the -0 in xargs -I {} -0 ?
    – thinkdeep
    Commented Jul 15 at 22:28
  • 1
    @thinkdeep when the input is separated by null characters, as in the find example in the answer.
    – miken32
    Commented Jul 16 at 21:45
21

When you use piping without xargs, the actual data is fed into the next command. On the other hand, when using piping with xargs, the actual data is viewed as a parameter to the next command. To give a concrete example, say you have a folder with a.txt and b.txt. a.txt contains just a single line 'hello world!', and b.txt is just empty.

If you do

ls | grep txt

you would end up getting the output:

a.txt
b.txt

Yet, if you do

ls | xargs grep txt

you would get nothing since neither file a.txt nor b.txt contains the word txt.

If the command is

ls | xargs grep hello

you would get:

hello world!

That's because with xargs, the two filenames given by ls are passed to grep as arguments, rather than the actual content.

1
  • In ls | grep txt, grep still gets a filename (something like "/proc/1551/fd/0") as its argument, and then grep search in that file.
    – Youjun Hu
    Commented May 24, 2023 at 1:16
6

Understanding the data flow

While the pipe operator | redirects the stdout from the last command to the stdin of the next command, the xargs command builds and executes a command from the stdin.

To put it simply:

  • | stdout -> stdin
  • xargs stdin -> command arguments

As xargs happens to expect the result of the pipe operator, a combination of both comes of in handy for converting stdout into command arguments:

  • | xargs stdout -> stdin (pipe) -> command arguments (xargs)

When to use | and | xargs?

To put it simply, it depends on the command you want to execute, as different commands handle the stdin or command arguments in different ways, depending on it's implementation.

In other words, even commands which accept both stdin and arguments can behaviour differently depending on which one of them you provided, as you can see in the examples bellow.

Still, if you want some kind of rule, you can think that:

  • if the data you want a command to process is already in the stdout, you should use the pipe operator |;
  • if the stdout contains either names of files whose content you want to process or even arguments for changing a command's default behaviour, you should use the combination | xargs;

However, be prepared for exceptions.


Examples

Setting up the scenario for the next examples:

echo "hello_world_content" > "hello_world.txt"
# Creates a file named "hello_world.txt" and
# writes "hello_world_content" inside it

The grep command:

ls | grep "hello"
# Grep searches for 'hello' in the stdout of the ls command
# Outputs: hello_world.txt


ls | xargs grep "hello"
# Corresponds to: grep 'hello_world.txt'
# Outputs: hello_world_content

The echo command:

ls | echo
# Outputs a blank line, as echo seems to ignore the stdin
# Outputs:


ls | xargs echo
# Corresponds to: echo 'hello_world.txt'
# Outputs: hello_world.txt

The cat command:

ls | cat
# Reads and prints the stdout of ls
# Outputs: hello_world.txt


ls | xargs cat
# Corresponds to: cat 'hello_world.txt'
# Outputs: hello_world_content

The cat command [Part II]:

echo "this_file_does_not_exists.txt" | cat
# Reads and prints the stdout of echo
# Outputs: this_file_does_not_exists.txt


echo "this_file_does_not_exists.txt" | xargs cat
# Corresponds to: cat 'this_file_does_not_exists.txt'
# Outputs error: cat: this_file_does_not_exists.txt: No such file or directory
3

xargs(1) is dangerous (broken, exploitable, etc.) when reading non-NUL-delimited input.

If you're working with filenames, use find's -exec [command] {} + instead. If you can get NUL-delimited output, use xargs -0.

7
  • 1
    @SaraHamad : you may be able to get null terminated strings from your find by adding -print0 to your cmd-line. Check you man page. (most new versions have it)
    – shellter
    Commented Feb 23, 2016 at 22:23
  • Wouldn't that read better be IFS= read -r -d '' line to ensure reading null separated input? Commented Feb 23, 2016 at 22:24
  • @DirkHerrmann You're correct. Though I wrote it for the general case. I removed this part according to your comment. Commented Feb 23, 2016 at 22:26
  • 1
    Being "dangerous (broken, exploitable, etc.)" doesn't mean that it is not super useful for one-shot scripts to grind dozens of files :)
    – Jakub M.
    Commented Feb 23, 2016 at 22:50
  • @JakubM. Grrrrind to dust! Commented Feb 23, 2016 at 22:53
3

Short answer: Avoid xargs for now. Return to xargs when you have written dozens or hundreds of scripts.

Commands can get their input from parameters (like rm bad_example) or can get the input from stdin (not just the y on the question after rm -i is_this_bad_too, but also read answer). Other commands like grep and sed will look for parameters and when the parameters don't show the input, switch to the input.
Your grep example works fine reading from stdin, nothing special needed.
Your ls needs the output of find as a parameter. xargs is just one way to turn things around. Use man xargs for more about xargs. Alternatives:

find /etc -name "*.txt" -exec ls -l {} \;
find /etc -name "*.txt" -ls
ls -l $(find /etc -name "*.txt" )
ls /etc/*.txt

First try to see which of this commands is best when you have a nasty filename with spaces.txt in /etc.

1

GNU Parallel can do the same as xargs, but does not have the broken and exploitable "features".

You can learn GNU Parallel by looking at examples http://www.gnu.org/software/parallel/man.html#EXAMPLE:-Working-as-xargs--n1.-Argument-appending and walking through the tutorial http://www.gnu.org/software/parallel/parallel_tutorial.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.