460

Could somebody please provide the code to do the following: Assume there is a directory of files, all of which need to be run through a program. The program outputs the results to standard out. I need a script that will go into a directory, execute the command on each file, and concat the output into one big output file.

For instance, to run the command on 1 file:

$ cmd [option] [filename] > results.out
2
  • 3
    I would like to add to the question. Can it be done using xargs? e.g., ls <directory> | xargs cmd [options] {filenames put in here automatically by xargs} [more arguments] > results.out Commented May 9, 2012 at 20:16
  • 3
    It can, but you probably don't want to use ls to drive xargs. If cmd is at all competently written, perhaps you can simply do cmd <wildcard>.
    – tripleee
    Commented Oct 31, 2017 at 4:42

11 Answers 11

635

The following bash code will pass $file to command where $file will represent every file in /dir

for file in /dir/*
do
  cmd [option] "$file" >> results.out
done

Example

el@defiant ~/foo $ touch foo.txt bar.txt baz.txt
el@defiant ~/foo $ for i in *.txt; do echo "hello $i"; done
hello bar.txt
hello baz.txt
hello foo.txt
9
  • 39
    If no files exist in /dir/, then the loop still runs once with a value of '*' for $file, which may be undesirable. To avoid this, enable nullglob for the duration of the loop. Add this line before the loop shopt -s nullglob and this line after the loop shopt -u nullglob #revert nullglob back to it's normal default state.
    – Stew-au
    Commented Sep 19, 2012 at 7:38
  • If the output file is the same inside the loop, it's much more efficient to redirect outside the loop done >results.out (and probably then you can overwrite instead of append, like I have assumed here).
    – tripleee
    Commented Oct 31, 2017 at 4:37
  • 3
    be carefull by using this command for huge amount of files in dir. Use find -exec instead.
    – kolisko
    Commented Feb 28, 2019 at 14:22
  • 2
    "be carefull by using this command for huge amount of files in dir. Use find -exec instead". But why? Commented Oct 16, 2020 at 23:35
  • 1
    @Timo The xargs solutions on this page typically do that pretty well.
    – tripleee
    Commented Dec 13, 2023 at 15:50
269

How about this:

find /some/directory -maxdepth 1 -type f -exec cmd option {} \; > results.out
  • -maxdepth 1 argument prevents find from recursively descending into any subdirectories. (If you want such nested directories to get processed, you can omit this.)
  • -type -f specifies that only plain files will be processed.
  • -exec cmd option {} tells it to run cmd with the specified option for each file found, with the filename substituted for {}
  • \; denotes the end of the command.
  • Finally, the output from all the individual cmd executions is redirected to results.out

However, if you care about the order in which the files are processed, you might be better off writing a loop. I think find processes the files in inode order (though I could be wrong about that), which may not be what you want.

8
  • 2
    This is the correct way to process files. Using a for loop is error-prone due to many reasons. Also sorting can be done by using other commands such as stat and sort, which of-course dependes on what is sorting criteria.
    – tuxdna
    Commented Dec 25, 2013 at 8:10
  • 3
    if I wanted to run two commands how would I link them after the -exec option? Do i have to wrap them in single quotes or something?
    – frei
    Commented Nov 20, 2017 at 8:28
  • 1
    find is always the best option cause you can filter by file name pattern with option -name and you can do it in a single command. Commented Dec 7, 2017 at 18:56
  • 9
    @frei the answer to your question is here: stackoverflow.com/a/6043896/1243247 but basically just add -exec options: find . -name "*.txt" -exec echo {} \; -exec grep banana {} \; Commented Dec 7, 2017 at 19:02
  • 4
    how can you reference the file name as option?
    – Toskan
    Commented Apr 10, 2019 at 16:24
140

I'm doing this on my Raspberry Pi from the commandline by running:

for i in ./*; do cmd "$i"; done
4
  • 7
    While this answer is probably the "right" way to do this in a production environment, for day-to-day usage convenience, this one-liner wins!
    – rinogo
    Commented Apr 9, 2021 at 15:31
  • If one wants to use the modified filename as an argument (e.g. for the name of the output file), you can add anything after the $i part, and you will have a new string. Example of an imaginary command ppp -i raw.txt -o processed.txt would be: for i in *; do ppp -i "$i" -o "$i changed"; done - this will do the ppp command on every file and the resulting file for each execution will be named like the input file, with addition of " changed" at the end.
    – Aleksandar
    Commented Jan 9, 2023 at 10:43
  • 1
    You want for i in ./* to avoid the pesky corner case where * expands to a string which looks like a command option (which it will if you have files whose name starts with a dash; this is uncommon, but not as uncommon as you'd hope).
    – tripleee
    Commented Dec 13, 2023 at 15:47
  • @tripleee Good catch, modified original post.
    – robgraves
    Commented Mar 22 at 0:53
24

You can use xarg:

ls | xargs -L 1 -d '\n' your-desired-command 
  • -L 1 causes pass 1 item at a time

  • -d '\n' splits the output of ls based on new line.

3
  • 3
    Using xargs is nice because it allows you to run your-desired-command in parallel if you add the -P 8 flag (up to 8 processes at the same time).
    – Nick Crews
    Commented Aug 22, 2022 at 21:05
  • 4
    For macOS, the -d option isn't available. You can fix it by brew install findutils first and then use gxargs instead of xargs
    – Wit
    Commented Nov 16, 2022 at 8:39
  • As noted earlier in my comment on the question, you almost certainly don't want to use ls to drive xargs. If you have GNU xargs, you can do printf '%s\0' * | xargs -r0 cmd but the -0 option is not in POSIX, and not avaliable e.g. on Mac. See also mywiki.wooledge.org/BashFAQ/020
    – tripleee
    Commented Mar 22 at 5:00
21

The accepted/high-voted answers are great, but they are lacking a few nitty-gritty details. This post covers the cases on how to better handle when the shell path-name expansion (glob) fails, when filenames contain embedded newlines/dash symbols and moving the command output re-direction out of the for-loop when writing the results to a file.

When running the shell glob expansion using * there is a possibility for the expansion to fail if there are no files present in the directory and an un-expanded glob string will be passed to the command to be run, which could have undesirable results. The bash shell provides an extended shell option for this using nullglob. So the loop basically becomes as follows inside the directory containing your files

 shopt -s nullglob

 for file in ./*; do
     cmdToRun [option] -- "$file"
 done

This lets you safely exit the for loop when the expression ./* doesn't return any files (if the directory is empty)

or in a POSIX compliant way (nullglob is bash specific)

 for file in ./*; do
     [ -f "$file" ] || continue
     cmdToRun [option] -- "$file"
 done

This lets you go inside the loop when the expression fails for once and the condition [ -f "$file" ] check if the un-expanded string ./* is a valid filename in that directory, which wouldn't be. So on this condition failure, using continue we resume back to the for loop which won't run subsequently.

Also note the usage of -- just before passing the file name argument. This is needed because as noted previously, the shell filenames can contain dashes anywhere in the filename. Some of the shell commands interpret that and treat them as a command option when the name are not quoted properly and executes the command thinking if the flag is provided.

The -- signals the end of command line options in that case which means, the command shouldn't parse any strings beyond this point as command flags but only as filenames.


Double-quoting the filenames properly solves the cases when the names contain glob characters or white-spaces. But *nix filenames can also contain newlines in them. So we de-limit filenames with the only character that cannot be part of a valid filename - the null byte (\0). Since bash internally uses C style strings in which the null bytes are used to indicate the end of string, it is the right candidate for this.

So using the printf option of shell to delimit files with this NULL byte using the -d option of read command, we can do below

( shopt -s nullglob; printf '%s\0' ./* ) | while read -rd '' file; do
    cmdToRun [option] -- "$file"
done

The nullglob and the printf are wrapped around (..) which means they are basically run in a sub-shell (child shell), because to avoid the nullglob option to reflect on the parent shell, once the command exits. The -d '' option of read command is not POSIX compliant, so needs a bash shell for this to be done. Using find command this can be done as

while IFS= read -r -d '' file; do
    cmdToRun [option] -- "$file"
done < <(find -maxdepth 1 -type f -print0)

For find implementations that don't support -print0 (other than the GNU and the FreeBSD implementations), this can be emulated using printf

find . -maxdepth 1 -type f -exec printf '%s\0' {} \; | xargs -0 cmdToRun [option] --

Another important fix is to move the re-direction out of the for-loop to reduce a high number of file I/O. When used inside the loop, the shell has to execute system-calls twice for each iteration of the for-loop, once for opening and once for closing the file descriptor associated with the file. This will become a bottle-neck on your performance for running large iterations. Recommended suggestion would be to move it outside the loop.

Extending the above code with this fixes, you could do

( shopt -s nullglob; printf '%s\0' ./* ) | while read -rd '' file; do
    cmdToRun [option] -- "$file"
done > results.out

which will basically put the contents of your command for each iteration of your file input to stdout and when the loop ends, open the target file once for writing the contents of the stdout and saving it. The equivalent find version of the same would be

while IFS= read -r -d '' file; do
    cmdToRun [option] -- "$file"
done < <(find -maxdepth 1 -type f -print0) > results.out
1
  • 2
    +1 for checking that the file exists. If searching in a non existing dir, $file contains the regex string "/invald_dir/*" not a valid filename.
    – cdalxndr
    Commented Feb 19, 2020 at 18:14
6

One quick and dirty way which gets the job done sometimes is:

find directory/ | xargs  Command 

For example to find number of lines in all files in the current directory, you can do:

find . | xargs wc -l
5
  • 8
    @Hubert Why do you have newlines in your filenames?!
    – musicin3d
    Commented Dec 1, 2018 at 20:37
  • 4
    it's not a question of "why", it's a question of correctness – filenames don't have to include printable characters, they don't even have to be valid UTF-8 sequences. Also, what's a newline is very much encoding dependent, one encodings ♀ is another's newline. See code page 437 Commented Dec 7, 2018 at 12:23
  • 3
    cmon, really? this does work 99.9% of the time, and he did say "quick and dirty"
    – Edoardo
    Commented Jan 24, 2019 at 14:22
  • 1
    I am not fan of "quick and dirty" (AKA "broken") Bash scripts. Sooner or later it ends in things like famous "Moved ~/.local/share/steam. Ran steam. It deleted everything on system owned by user." bug report. Commented Jan 26, 2019 at 18:31
  • This also won't work with files that have spaces in the name.
    – Shamas S
    Commented Sep 12, 2019 at 7:21
2

Based on @Jim Lewis's approach:

Here is a quick solution using find and also sorting files by their modification date:

$ find  directory/ -maxdepth 1 -type f -print0 | \
  xargs -r0 stat -c "%y %n" | \
  sort | cut -d' ' -f4- | \
  xargs -d "\n" -I{} cmd -op1 {} 

For sorting see:

https://www.commandlinefu.com/commands/view/5720/find-files-and-list-them-sorted-by-modification-time

3
  • this will not work if the files have newlines in their names Commented Nov 12, 2018 at 12:04
  • 2
    @HubertKario You may want to read more about -print0 for find and -0 for xargs which use null character instead of any whitespace ( including newlines ).
    – tuxdna
    Commented Nov 15, 2018 at 4:35
  • yes, using -print0 is something that helps, but the whole pipeline needs to use something like this, and sort isn't Commented Nov 15, 2018 at 18:02
1

I needed to copy all .md files from one directory into another, so here is what I did.

for i in **/*.md;do mkdir -p ../docs/"$i" && rm -r ../docs/"$i" && cp "$i" "../docs/$i" && echo "$i -> ../docs/$i"; done

Which is pretty hard to read, so lets break it down.

first cd into the directory with your files,

for i in **/*.md; for each file in your pattern

mkdir -p ../docs/"$i"make that directory in a docs folder outside of folder containing your files. Which creates an extra folder with the same name as that file.

rm -r ../docs/"$i" remove the extra folder that is created as a result of mkdir -p

cp "$i" "../docs/$i" Copy the actual file

echo "$i -> ../docs/$i" Echo what you did

; done Live happily ever after

1
  • Note: for ** to work, the globstar shell option needs to be set: shopt -s globstar Commented Nov 12, 2018 at 12:07
1

Maxdepth

I found it works nicely with Jim Lewis's answer just add a bit like this:

$ export DIR=/path/dir && cd $DIR && chmod -R +x *
$ find . -maxdepth 1 -type f -name '*.sh' -exec {} \; > results.out

Sort Order

If you want to execute in sort order, modify it like this:

$ export DIR=/path/dir && cd $DIR && chmod -R +x *
find . -maxdepth 2 -type f -name '*.sh' | sort | bash > results.out

Just for an example, this will execute with following order:

bash: 1: ./assets/main.sh
bash: 2: ./builder/clean.sh
bash: 3: ./builder/concept/compose.sh
bash: 4: ./builder/concept/market.sh
bash: 5: ./builder/concept/services.sh
bash: 6: ./builder/curl.sh
bash: 7: ./builder/identity.sh
bash: 8: ./concept/compose.sh
bash: 9: ./concept/market.sh
bash: 10: ./concept/services.sh
bash: 11: ./product/compose.sh
bash: 12: ./product/market.sh
bash: 13: ./product/services.sh
bash: 14: ./xferlog.sh

Unlimited Depth

If you want to execute in unlimited depth by certain condition, you can use this:

export DIR=/path/dir && cd $DIR && chmod -R +x *
find . -type f -name '*.sh' | sort | bash > results.out

then put on top of each files in the child directories like this:

#!/bin/bash
[[ "$(dirname `pwd`)" == $DIR ]] && echo "Executing `realpath $0`.." || return

and somewhere in the body of parent file:

if <a condition is matched>
then
    #execute child files
    export DIR=`pwd`
fi
0

If you hate bash more than YAML configuration files, you can try the tool I've written.

  1. Write a config:
---
version: '1.2'
name: files
processors:
  - name: cmd_each
    operations:
      - name: filesystem_input_read
        params:
          target:
            sourceType: env_var
            source: INPUT_READ_TARGET
      - name: command_exec
        maxPacketSize: 10
        params:
          commandName:
            sourceType: value
            source: cmd
          commandArgs:
            sourceType: value
            source: ["{{.AbsolutePath}}"]
  1. Run:
INPUT_READ_TARGET=/dir/* capycmd -c -f files.pipeline.yml files:cmd_each

Advantages:

  • zero code written on bash
  • can do such stuff concurrently

Disadvantages:

  • you won't see the stdout (can be fixed thought)
1
  • It would be more weird to find someone who doesn't hate YAML more than they hate Bash.
    – tripleee
    Commented Mar 22 at 4:29
-2

i think the simple solution is:

sh /dir/* > ./result.txt
1
  • 3
    Did you understand the question correctly? This will merely try to run each file in the directory through the shell - as if it were a script.
    – rdas
    Commented Apr 16, 2019 at 8:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.