111

I have a file temp.txt, that I want to sort with the sort command in bash.

I want the sorted results to replace the original file.

This doesn't work for example (I get an empty file):

sortx temp.txt > temp.txt

Can this be done in one line without resorting to copying to temporary files?


EDIT: The -o option is very cool for sort. I used sort in my question as an example. I run into the same problem with other commands:

uniq temp.txt > temp.txt.

Is there a better general solution?

1

14 Answers 14

169
sort temp.txt -o temp.txt
5
  • 3
    This is an answer. I was actually wondering if there is a generic solution to this problem. For example if I want to find all UNIQ lines in a file "in place", I can't do -o
    – jm.
    Sep 28, 2008 at 21:45
  • 1
    It's not generic, but you can use -u with GNU sort to find unique lines
    – James
    Apr 24, 2010 at 21:45
  • Has anyone solved the problem to allow e.g. sort --inplace *.txt? That would be crazy cool
    – sehe
    Apr 24, 2014 at 13:32
  • @sehe Try this: find . -name \*.txt -exec sort {} -o {} \; Jan 15, 2015 at 17:50
  • 1
    To type filename only once, brace expansion can be used: sort -o temp.txt{,}
    – Shevchuk
    May 29, 2022 at 6:11
29

A sort needs to see all input before it can start to output. For this reason, the sort program can easily offer an option to modify a file in-place:

sort temp.txt -o temp.txt

Specifically, the documentation of GNU sort says:

Normally, sort reads all input before opening output-file, so you can safely sort a file in place by using commands like sort -o F F and cat F | sort -o F. However, sort with --merge (-m) can open the output file before reading all input, so a command like cat F | sort -m -o F - G is not safe as sort might start writing F before cat is done reading it.

While the documentation of BSD sort says:

If [the] output-file is one of the input files, sort copies it to a temporary file before sorting and writing the output to [the] output-file.

Commands such as uniq can start writing output before they finish reading the input. These commands typically do not support in-place editing (and it would be harder for them to support this feature).

You typically work around this with a temporary file, or if you absolutely want to avoid having an intermediate file, you could use a buffer to store the complete result before writing it out. For example, with perl:

uniq temp.txt | perl -e 'undef $/; $_ = <>; open(OUT,">temp.txt"); print OUT;'

Here, the perl part reads the complete output from uniq in variable $_ and then overwrites the original file with this data. You could do the same in the scripting language of your choice, perhaps even in Bash. But note that it will need enough memory to store the entire file, this is not advisable when working with large files.

20

Here's a more general approach, works with uniq, sort and whatnot.

{ rm file && uniq > file; } < file
9
  • 15
    Another generic approach, with sponge from the moreutils: cat file |frobnicate |sponge file.
    – Tobu
    Mar 16, 2011 at 20:04
  • 3
    @Tobu: why not submit that as a separate answer?
    – Flimm
    Aug 31, 2011 at 17:03
  • 1
    It's probably good to note that this doesn't necessarily preserve the files permissions. Your umask dictates what the new permissions will be.
    – wor
    Nov 23, 2014 at 9:42
  • 1
    Tricky one. Can you explain how does it exactly works? Feb 19, 2015 at 21:51
  • 2
    @patryk.beza: In order: The input FD is opened from the original file; the original directory entry is deleted; the redirection is processed, creating a new empty file with the same name the old one used to have; then the command runs. Apr 17, 2015 at 22:19
11

Tobu's comment on sponge warrants being an answer in its own right.

To quote from the moreutils homepage:

Probably the most general purpose tool in moreutils so far is sponge(1), which lets you do things like this:

% sed "s/root/toor/" /etc/passwd | grep -v joey | sponge /etc/passwd

However, sponge suffers from the same problem Steve Jessop comments on here. If any of the commands in the pipeline before sponge fail, then the original file will be written over.

$ mistyped_command my-important-file | sponge my-important-file
mistyped-command: command not found

Uh-oh, my-important-file is gone.

2
  • 1
    Sponge knows that it will be used to replace the input file and it initially creates a temp file to avoid a race condition. In order for this to work, sponge must be the last element in the pipeline and it must be allowed to create the output file itself (as opposed to shell-level output redirection, for example). BTW: It seems like an easy source-code fix for the 'fail' case would be to not rename the temp file in the case of a pipefail (don't know why sponge doesn't have that option). Mar 6, 2014 at 17:08
  • I think if you add set -o pipefail at the beginning of your script, the error on mistyped_command my-important-file would make the script exit immediately, before executing sponge, thus preserving the important file. Nov 29, 2019 at 14:59
7

Here you go, one line:

sort temp.txt > temp.txt.sort && mv temp.txt.sort temp.txt

Technically there's no copying to a temporary file, and the 'mv' command should be instant.

9
  • 6
    Hm. I'd still call temp.txt.sort a temporary file.
    – JesperE
    Sep 28, 2008 at 18:39
  • 5
    This code is risky, because if sort fails for whatever reason without completing its job, the original is overwritten. Sep 28, 2008 at 20:19
  • 1
    Lack of disk space being a plausible cause, or a signal (user hits CTRL-C). Sep 28, 2008 at 20:25
  • 5
    if you want to use something like this use && (logical and) instead of ; because using that will make sure that if a command fails next one will not be executed. for example: cp backup.tar /root/backup.tar && rm backup.tar if you don't have rights to copy you'll be safe as the file won't be deleted
    – daniels
    Sep 28, 2008 at 20:46
  • 1
    changed my answer to take your suggestions into account, thanks
    – davr
    Sep 29, 2008 at 14:58
5

I like the sort file -o file answer but don't want to type the same file name twice.

Using BASH history expansion:

$ sort file -o !#^

grabs the current line's first arg when you press enter.

A unique sort in-place:

$ sort -u -o file !#$

grabs the last arg in the current line.

4

Many have mentioned the -o option. Here is the man page part.

From the man page:

   -o output-file
          Write output to output-file instead of to the  standard  output.
          If  output-file  is  one of the input files, sort copies it to a
          temporary file before sorting and writing the output to  output-
          file.
3

This would be highly memory constrained, but you could use awk to store the intermediate data in memory, and then write it back out.

uniq temp.txt | awk '{line[i++] = $0}END{for(j=0;j<i;j++){print line[j]}}' > temp.txt
1
  • I think it’s possible the > truncates the file before the command (uniq in this case) reads it.
    – Martin
    Feb 18, 2019 at 16:35
3

An alternative to sponge with the more common sed:

sed -ni r<(command file) file

It works for any command (sort, uniq, tac, ...) and uses the very well known sed's -i option (edit files in-place).

Warning: Try command file first because editing files in-place is not safe by nature.


Explanation

Firstly, you're telling sed not to print the (original) lines (-n option), and with the help of the sed's r command and bash's Process Substitution, the generated content by <(command file) will be the output saved in place.


Making things even easier

You can wrap this solution into a function:

ip_cmd() { # in place command
    CMD=${1:?You must specify a command}
    FILE=${2:?You must specify a file}
    sed -ni r<("$CMD" "$FILE") "$FILE"
}

Example

$ cat file
d
b
c
b
a

$ ip_cmd sort file
$ cat file
a
b
b
c
d

$ ip_cmd uniq file
$ cat file
a
b
c
d

$ ip_cmd tac file
$ cat file
d
c
b
a

$ ip_cmd
bash: 1: You must specify a command
$ ip_cmd uniq
bash: 2: You must specify a file
2

Read up on the non-interactive editor, ex.

1
  • heh -- that's a totally evil idea. I like it. Aug 29, 2014 at 12:28
1

Use the argument --output= or -o

Just tried on FreeBSD:

sort temp.txt -otemp.txt
1
  • Though correct, it's simply a duplicate of this answer
    – whoan
    Jan 17, 2015 at 15:57
1

To add the uniq capability, what are the downsides to:

sort inputfile | uniq | sort -o inputfile
0

If you insist on using the sort program, you have to use a intermediate file -- I don't think sort has an option for sorting in memory. Any other trick with stdin/stdout will fail unless you can guarantee that the buffer size for sort's stdin is big enough to fit the entire file.

Edit: shame on me. sort temp.txt -o temp.txt works excellent.

1
  • I read the Q also as being "in-place" but the second read made me believe he wasn't really asking for it
    – epatel
    Sep 28, 2008 at 20:03
0

Another solution:

uniq file 1<> file
1
  • It should be noted though that the <> trick only works in this case because uniq is special in that it only copies input lines to output lines, dropping some on the way. If other command (e.g. sed) was used which would change the input (e.g. would change every a into aa), then it can override file in ways which don't make any sense and even loop infinitely, providing that the input is sufficiently large (more than a single read buffer).
    – David
    Feb 24, 2019 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.