Xargs

xargs is a command on Unix and most Unix-like operating systems used to build and execute command lines from standard input. Commands like grep and awk can accept the standard input as a parameter, or argument by using a pipe. However, others like cp and echo disregard the standard input stream and rely solely on the arguments found after the command. Additionally, under the Linux kernel before version 2.6.23, arbitrarily long lists of parameters could not be passed to a command,^[1] so xargs breaks the list of arguments into sublists small enough to be acceptable.

For example, commands like:

rm /path/*

or

rm `find /path -type f`

will fail with an error message of "Argument list too long" if there are too many files in /path.

However, the version below (functionally equivalent to rm `find /path -type f`) will not fail:

find /path -type f -print0 | xargs -0 rm

In the above example, the find utility feeds the input of xargs with a long list of file names. xargs then splits this list into sublists and calls rm once for every sublist.

The previous example is more efficient than this functionally equivalent version which calls rm once for every single file:

find /path -type f -exec rm '{}' \;

Note, however, that with modern versions of find, the following variant does the same thing as the xargs version:

find /path -type f -exec rm '{}' +

xargs often covers the same functionality as the backquote (`) feature of many shells, but is more flexible and often also safer, especially if there are blanks or special characters in the input. It is a good companion for commands that output long lists of files like find, locate and grep, but only if you use -0, since xargs without -0 deals badly with file names containing ', " and space. GNU Parallel is the perfect companion to find, locate and grep if file names may contain ', " and space (newline still requires -0).

The separator problem

Many Unix utilities are line oriented. These may work with xargs as long as the lines do not contain ', " or space. Some of the Unix utilities can use NUL as record separator (e.g. Perl (requires -0 and \0 instead of \n), locate (requires using -0), find (requires using -print0), grep (requires -z or -Z), sort (requires using -z)). Using -0 for xargs deals with the problem, but many Unix utilities cannot use NUL as separator (e.g. head, tail, ls, echo, sed, tar -v, wc, which).

But often people forget this and assume xargs is also line oriented, which is not the case (per default xargs separates on newlines and blanks within lines, substrings with blanks must be single or double-quoted).

The separator problem is illustrated here:

touch important_file
touch 'not important_file'
find . -name not\* | tail -1 | xargs rm
mkdir -p '12" records'
find \! -name . -type d | tail -1 | xargs rmdir

Running the above will cause important_file to be removed but will remove neither the directory called 12" records, nor the file called not important_file.

The proper fix is to use find -print0, but tail (and other tools) do not support NUL-terminated strings:

touch important_file
touch 'not important_file'
find . -name not\* -print0 | xargs -0 rm
mkdir -p '12" records'
find \! -name . -type d -print0 | xargs -0 rmdir

When using the syntax find -print0, entries are separated by a null character instead of a end-of-line. This is equivalent to the more verbose command:

find . -name not\* | tr \\n \\0 | xargs -0 rm

or shorter, by switching xargs to line oriented mode with the -d (delimiter) option:

find . -name not\* | xargs -d '\n' rm

but in general using the -0 option should be preferred, since newlines in filenames are still a problem.

GNU Parallel is an alternative to xargs that is designed to have the same options, but be line oriented. Thus, using GNU Parallel instead, the above would work as expected.^[2]

For Unix environments where xargs does not support the -0 option (e.g. Solaris, AIX), the following can not be used as it does not deal with ' and " (GNU Parallel would work on Solaris, though):

find . -name not\* | sed 's/ /\\ /g' | xargs rm

For Solaris, do not use these examples to fix file perms as they do not deal correctly with names like 12" records (GNU Parallel instead of xargs would work, though):

find . -type d -print | sed -e 's/^/"/' -e 's/$/"/' | xargs chmod 755
find . -type f -print | sed -e 's/^/"/' -e 's/$/"/' | xargs chmod 644

References

External links

• xargs: construct argument lists and invoke utility – Commands & Utilities Reference, The Single UNIX® Specification, Issue 7 from The Open Group

Manual pages

• xargs(1) – GNU Findutils reference
• xargs(1): construct argument list(s) and execute utility – FreeBSD General Commands Manual
• xargs(1): construct argument list(s) and execute utility – NetBSD General Commands Manual
• xargs(1): construct argument list(s) and execute utility – OpenBSD General Commands Manual
• xargs(1): construct argument lists and invoke utility – Solaris 10 User Commands Reference Manual