71

I wish to copy the top 1000 lines in a text file containing more than 50 million entries, to another new file, and also delete these lines from the original file.

Is there some way to do the same with a single shell command in Unix?

4
  • 1
    @gagneet: What is the high-level task you're trying to accomplish? Are you splitting the file into chunks, or what? There is probably an alternative that avoids rewriting the entire file as these answers all do.
    – Phil H
    Commented Apr 29, 2009 at 6:14
  • @Phil my answer doesn't rewrite the entire file
    – moinudin
    Commented Apr 29, 2009 at 6:44
  • @Phil: I wish to convert some lines in the original file to another format. For the same, I need to first derive the lines from this BIG file, and then convert and copy them to another file. But parsing through this file is taking a long time and the machine gets hung, as almost 11G of memory is being used.
    – gagneet
    Commented Apr 29, 2009 at 10:49
  • You might want to profile with a subset of your file. In particular, using 'time' to test whether sed and other core utils on your system, as well as whatever else you're using to parse your files, are operating on the filesystem or in memory. Commented Apr 29, 2009 at 16:16

6 Answers 6

97
head -1000 input > output && sed -i '1,+999d' input

For example:

$ cat input 
1
2
3
4
5
6
$ head -3 input > output && sed -i '1,+2d' input
$ cat input 
4
5
6
$ cat output 
1
2
3
4
  • 1
    @Alex, do you have a file named 'input'? Commented Apr 29, 2009 at 6:03
  • I think you mean "cat > input". I still get the same error message: $ cat > input 1 2 3 4 5 6 $ head -3 input > output && sed -i '1,+2d' input sed: 1: "input": command i expects \ followed by text Commented Apr 29, 2009 at 6:04
  • This does not work. Or if it does, it works with a specific version of sed. Commented Apr 29, 2009 at 6:18
  • Okay, I'm using FreeBSD, which does not have a GNU version of sed. I've added an answer that includes a test run of sed vs tail that suggests tail is faster. It is, however, only one test. Nonetheless, head/tail/cp/rm seem to have standard implementations across UNIXes and, if faster, may seem preferable to sed. Commented Apr 29, 2009 at 7:09
39
head -1000 file.txt > first100lines.txt
tail --lines=+1001 file.txt > restoffile.txt
4
  • Upvoted, until I noticed the "and also delete these lines from the original file" requirement. Commented Apr 29, 2009 at 5:42
  • This does not delete lines from the original file. Commented Apr 29, 2009 at 5:44
  • 3
    Have patience. Removing the first 1000 lines and writing it back out takes a long, long time.
    – dkretz
    Commented Apr 29, 2009 at 6:02
  • I only needed to copy the first x number of rows in a file and store them in another. The first line does the job.
    – Jakob
    Commented Jan 20, 2021 at 13:37
11

Out of curiosity, I found a box with a GNU version of sed (v4.1.5) and tested the (uncached) performance of two approaches suggested so far, using an 11M line text file:

$ wc -l input
11771722 input

$ time head -1000 input > output; time tail -n +1000 input > input.tmp; time cp input.tmp input; time rm input.tmp

real    0m1.165s
user    0m0.030s
sys     0m1.130s

real    0m1.256s
user    0m0.062s
sys     0m1.162s

real    0m4.433s
user    0m0.033s
sys     0m1.282s

real    0m6.897s
user    0m0.000s
sys     0m0.159s

$ time head -1000 input > output && time sed -i '1,+999d' input

real    0m0.121s
user    0m0.000s
sys     0m0.121s

real    0m26.944s
user    0m0.227s
sys     0m26.624s

This is the Linux I was working with:

$ uname -a
Linux hostname 2.6.18-128.1.1.el5 #1 SMP Mon Jan 26 13:58:24 EST 2009 x86_64 x86_64 x86_64 GNU/Linux

For this test, at least, it looks like sed is slower than the tail approach (27 sec vs ~14 sec).

7

This is a one-liner but uses four atomic commands:

head -1000 file.txt > newfile.txt; tail +1000 file.txt > file.txt.tmp; cp file.txt.tmp file.txt; rm file.txt.tmp
4
  • He wants to move the first 1000 lines from one file to another. This deletes all but the first 1000 lines, i.e. is wrong.
    – moinudin
    Commented Apr 29, 2009 at 5:53
  • With "more than 50 million entries" that tail will be quite slow.
    – moinudin
    Commented Apr 29, 2009 at 6:00
  • Why are you doing "cp file.txt.tmp file.txt; rm file.txt.tmp" instead of "mv file.txt.tmp file.txt"?
    – Espo
    Commented Apr 29, 2009 at 6:20
  • 3
    cp and rm are atomic filesystem operations. mv is not. Commented Apr 29, 2009 at 6:26
4

Perl approach:

perl -ne 'if($i<1000) { print; } else { print STDERR;}; $i++;' in 1> in.new 2> out && mv in.new in
0

Using pipe:

cat en-tl.100.en | head -10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.