441

What is the quickest and most pragmatic way to combine all *.txt file in a directory into one large text file?

Currently I'm using windows with cygwin so I have access to BASH.

Windows shell command would be nice too but I doubt there is one.

12 Answers 12

773

This appends the output to all.txt

cat *.txt >> all.txt

This overwrites all.txt

cat *.txt > all.txt
11
  • 44
    you may run into a problem where it cats all.txt into all.txt... I have this problem with grep sometimes, not sure if cat has the same behavior.
    – rmeador
    Jan 27, 2010 at 23:54
  • 13
    @rmeador yes, that is true, if all.txt already exists you will have this problem. This problem is solved by providing the output file with a different extension, or moving all.txt to a different folder. Jan 28, 2010 at 1:11
  • 6
    cat *.txt >> tmp; mv tmp all.txt (and make sure that all.txt does not exist beforehand)
    – Renaud
    Feb 14, 2013 at 10:16
  • 19
    I get "Argument list too long" -- guess it can't handle 40,000+ files.
    – Matt
    Sep 16, 2013 at 15:51
  • 40
    Avoid argument list too long with: echo *.txt | xargs cat > all.txt
    – 5heikki
    Sep 22, 2014 at 8:45
212

Just remember, for all the solutions given so far, the shell decides the order in which the files are concatenated. For Bash, IIRC, that's alphabetical order. If the order is important, you should either name the files appropriately (01file.txt, 02file.txt, etc...) or specify each file in the order you want it concatenated.

$ cat file1 file2 file3 file4 file5 file6 > out.txt
0
33

The Windows shell command type can do this:

type *.txt > outputfile.txt

Type type command also writes file names to stderr, which are not captured by the > redirect operator (but will show up on the console).

2
  • 3
    Just be aware that if you put the output file in the same directory as the original file it will cause a duplication because it will also combine the new output file twice.
    – CathalMF
    May 14, 2013 at 10:15
  • 8
    user requested for bash
    – capdragon
    Nov 24, 2021 at 17:47
30

You can use Windows shell copy to concatenate files.

C:\> copy *.txt outputfile

From the help:

To append files, specify a single file for destination, but multiple files for source (using wildcards or file1+file2+file3 format).

4
  • This as the IMHO cleanest solution with basically no side effects that beginners could trip over unfortunately does not get appreciated enough :-(
    – Grmpfhmbl
    Jun 20, 2017 at 6:39
  • OP asked for Bash.
    – Big Rich
    May 3, 2018 at 11:39
  • 5
    Did you read the question? "Windows shell command would be nice too..."
    – Carl Norum
    May 3, 2018 at 20:40
  • Worked pretty well, except at the very end of my file I got a weird SUB special unicode character. Deleting it is pretty easy programmatically but not sure why that happened.
    – abelito
    Oct 12, 2021 at 11:44
19

Be careful, because none of these methods work with a large number of files. Personally, I used this line:

for i in $(ls | grep ".txt");do cat $i >> output.txt;done

EDIT: As someone said in the comments, you can replace $(ls | grep ".txt") with $(ls *.txt)

EDIT: thanks to @gnourf_gnourf expertise, the use of glob is the correct way to iterate over files in a directory. Consequently, blasphemous expressions like $(ls | grep ".txt") must be replaced by *.txt (see the article here).

Good Solution

for i in *.txt;do cat $i >> output.txt;done
5
  • 2
    Why not for i in $(ls *.txt);do cat $i >> output.txt;done? Nov 12, 2018 at 21:49
  • 3
    Mandatory ParsingLs link, together with a downvote (and you deserve more than one downvote, because ls | grep is a seriously bad antipattern). Jan 25, 2019 at 9:23
  • Got an upvote from me because it allows for arbitrary testing/ operations by file name prior to output and it's quick and easy and good for practice. (In my case I wanted: for i in *; do echo -e "\n$i:\n"; cat $1; done ) Feb 7, 2019 at 22:09
  • 1
    Wouldn't the ls *.txt fail if there are too many files (Argument list too long error)? Mar 21, 2019 at 15:25
  • @Mandatory: ls .txt | grep *.txt | awk '/.txt/' LOL
    – runlevel0
    Feb 8, 2022 at 15:24
18

How about this approach?

find . -type f -name '*.txt' -exec cat {} + >> output.txt
5
  • Since OP says the files are in the same directory, you may need to add -maxdepth 1 to the find command. Jul 25, 2017 at 2:52
  • 3
    Works great with a big number of files, where the accepted reply's approach fails
    – amine
    Sep 21, 2017 at 12:21
  • ah i wish i knew what this plus and double redirect signify... Mar 27, 2020 at 21:24
  • This should be the correct answer. It will work properly in a shell script. Here is a similar method if you want output sorted: sort -u --output="$OUTPUT_FILE" --files0-from=- < <(find "$DIRECTORY_NAME" -maxdepth 1 -type f -name '*.txt' -print0)
    – steveH
    Apr 27, 2020 at 13:41
  • This is a very flexible approach relying on all the strengths of the find. My favourite! Surely cat *.txt > all.txt does the job within the same directory (as pointed out above). To me, however, becoming comfortably fluent in using find has been a very good habit. Today they're all in one folder, tomorrow they have multiple file-endings across nested directory hierarchies. Don't overcomplicate, but also, do make friends with find. :)
    – nJGL
    Nov 16, 2021 at 11:32
8

the most pragmatic way with the shell is the cat command. other ways include,

awk '1' *.txt > all.txt
perl -ne 'print;' *.txt > all.txt
1
  • 2
    This should be the correct answer for most circumstances. If any text file without an empty new line, using all the above cat method will concatenate last line and first line from adjacent files.
    – mootmoot
    Oct 25, 2016 at 10:56
3
type [source folder]\*.[File extension] > [destination folder]\[file name].[File extension]

For Example:

type C:\*.txt > C:\1\all.txt

That will Take all the txt files in the C:\ Folder and save it in C:\1 Folder by the name of all.txt

Or

type [source folder]\* > [destination folder]\[file name].[File extension]

For Example:

type C:\* > C:\1\all.txt

That will take all the files that are present in the folder and put there Content in C:\1\all.txt

1

You can do like this: cat [directory_path]/**/*.[h,m] > test.txt

if you use {} to include the extension of the files you want to find, there is a sequencing problem.

1

The most upvoted answers will fail if the file list is too long.

A more portable solution would be using fd

fd -e txt -d 1 -X awk 1 > combined.txt

-d 1 limits the search to the current directory. If you omit this option then it will recursively find all .txt files from the current directory.
-X (otherwise known as --exec-batch) executes a command (awk 1 in this case) for all the search results at once.

Note, fd is not a "standard" Unix program, so you will likely need to install it

2
  • bash: fd: command not found It should be mentioned that fd is a package you must install yourself. Oct 3, 2021 at 0:56
  • I did link to it in the answer, but I will be even more explicit. Oct 3, 2021 at 4:03
0

When you run into a problem where it cats all.txt into all.txt, You can try check all.txt is existing or not, if exists, remove

Like this:

[ -e $"all.txt" ] && rm $"all.txt"

1
  • cat *.txt > all.txt > command overwrites all.txt if it exists >> adds data to existing file Jun 12, 2019 at 15:15
-5

all of that is nasty....

ls | grep *.txt | while read file; do cat $file >> ./output.txt; done;

easy stuff.

1
  • 7
    Eeek! Don't do that. Do find . -iname "*.txt" -maxdepth 1 -exec cat {} >> out.txt \; Jan 28, 2010 at 11:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.