116

I tried opening a huge (~2GB) file in VIM but it choked. I don't actually need to edit the file, just jump around efficiently.

How can I go about working with very large files in VIM?

8
  • 1
    Here is a similar question.
    – GeoAvila
    Oct 19, 2009 at 23:40
  • 5
    Vim should be okay as long as you :set binary first...
    – ephemient
    Oct 19, 2009 at 23:44
  • 1
    This is a good target for a new fuse filesystem! splitfs or something like that... I'm into it!
    – rodrigo
    Jul 29, 2011 at 16:58
  • 1
    Too late... this already exists: sourceforge.net/projects/joinsplitfs
    – rodrigo
    Jul 29, 2011 at 16:59
  • 5
    You need a pager, not an editor sir! See Jim's answer below. Aug 16, 2013 at 3:55

10 Answers 10

93

I had a 12GB file to edit today. The vim LargeFile plugin did not work for me. It still used up all my memory and then printed an error message :-(. I could not use hexedit for either, as it cannot insert anything, just overwrite. Here is an alternative approach:

You split the file, edit the parts and then recombine it. You still need twice the disk space though.

  • Grep for something surrounding the line you would like to edit:

    grep -n 'something' HUGEFILE | head -n 1
    
  • Extract that range of the file. Say the lines you want to edit are at line 4 and 5. Then do:

    sed -n -e '4,5p' -e '5q' HUGEFILE > SMALLPART
    
    • The -n option is required to suppress the default behaviour of sed to print everything
    • 4,5p prints lines 4 and 5
    • 5q aborts sed after processing line 5
  • Edit SMALLPART using your favourite editor.

  • Combine the file:

    (head -n 3 HUGEFILE; cat SMALLPART; sed -e '1,5d' HUGEFILE) > HUGEFILE.new 
    
    • i.e: pick all the lines before the edited lines from the HUGEFILE (which in this case is the top 3 lines), combine it with the edited lines (in this case lines 4 and 5) and use this combined set of lines to replace the equivalent (in this case the top 5 lines) in the HUGEFILE and write it all to a new file.

    HUGEFILE.new will now be your edited file, you can delete the original HUGEFILE.

0
31

This has been a recurring question for many years. (The numbers keep changing, but the concept is the same: how do I view or edit files that are larger than memory?)

Obviously more or less are good approaches to merely reading the files --- less even offers vi like keybindings for scrolling and searching.

A Freshmeat search on "large files" suggests that two editors would be particularly suited to your needs.

One would be: lfhex ... a large file hex editor (which depends on Qt). That one, obviously, entails using a GUI.

Another would seem to be suited to console use: hed ... and it claims to have a vim-like interface (including an ex mode?).

I'm sure I've seen other editors for Linux/UNIX that were able to page through files without loading their entirety into memory. However, I don't recall any of their names. I'm making this response a "wiki" entry to encourage others to add their links to such editors. (Yes, I am familiar with ways to work around the issue using split and cat; but I'm thinking of editors, especially console/curses editors which can dispense with that and save us the time/latencies and disk space overhead that such approaches entail).

0
25

Since you don't need to actually edit the file:

  1. view (or vim -R) should work reasonably well on large files.
  2. Or you can use more or less
2
  • By "chokes" you mean takes a while to open? Or actually crashes? It takes a bit over 4 minutes on my not-so-recent Linux box to open 2.7GB file in view (just tried and timed). Granted, that's not exactly instant, but it does work.
    – ChssPly76
    Oct 20, 2009 at 0:05
  • Yeah it stalls. I am sure if I waited it would open eventually. I've gone with less because it opens immediately and I'm used to the navigation.
    – hoju
    Oct 20, 2009 at 1:14
10

I wrote a little script based on Florian's answer that uses nano (my favorite editor):

#!/bin/sh

if [ "$#" -ne 3 ]; then
  echo "Usage: $0 hugeFilePath startLine endLine" >&2
  exit 1
fi

sed -n -e $2','$3'p' -e $3'q' $1 > hfnano_temporary_file
nano hfnano_temporary_file
(head -n `expr $2 - 1` $1; cat hfnano_temporary_file; sed -e '1,'$3'd' $1) > hfnano_temporary_file2
cat hfnano_temporary_file2 > $1
rm hfnano_temporary_file hfnano_temporary_file2

Use it like this:

sh hfnano yourHugeFile 3 8

In that example, nano will open up lines 3 through 8, you can edit them, and when you save and quit, those lines in the hugefile will automatically be overwritten with your saved lines.

3

I had the same problem, but it was a 300GB mysql dump and I wanted to get rid of the DROP and change CREATE TABLE to CREATE TABLE IF NOT EXISTS so didn't want to run two invocations of sed. I wrote this quick Ruby script to dupe the file with those changes:

#!/usr/bin/env ruby

matchers={
    %q/^CREATE TABLE `foo`/ => %q/CREATE TABLE IF NOT EXISTS `foo`/,
    %q/^DROP TABLE IF EXISTS `foo`;.*$/ => "-- DROP TABLE IF EXISTS `foo`;"
}

matchers.each_pair { |m,r|
    STDERR.puts "%s: %s" % [ m, r ]
}

STDIN.each { |line|
    #STDERR.puts "line=#{line}"
    line.chomp!
    unless matchers.length == 0
        matchers.each_pair { |m,r|
            re=/#{m}/
            next if line[re].nil?
            line.sub!(re,r)
            STDERR.puts "Matched: #{m} -> #{r}"
            matchers.delete(m)
            break
        }
    end
    puts line
}

Invoked like

./mreplace.rb < foo.sql > foo_two.sql
2
  • Just to note for running, to run it as an exe requires chmod +x mreplace.rb first, you could also just ruby mreplace.rb ..
    – Smar
    Sep 11, 2015 at 11:41
  • Thanks @Steeve McCauley! Nice work. Exactly what I was looking for when searching for the answer to this question. Jul 7, 2016 at 4:37
3

For huge one-liners (prints characters from 1 to 99):

cut -c 1-99 filename
1

emacs works very well with files into the 100's of megabytes, I've used it on log files without too much trouble.

But generally when I have some kind of analysis task, I find writing a perl script a better choice.

1

It's already late but if you just want to navigate through the file without editing it, cat can do the job too.

% cat filename | less

or alternatively simple:

% less filename
1
  • 11
    Note that catting the file first is insanely stupid, as it either means the file would be fully in memory (so less can seek the file) or it can’t be seeked at all; cat just gives static output stream.
    – Smar
    Sep 11, 2015 at 11:38
0

Old thread. But nevertheless( pun :) ).

 $less filename

less works efficiently if you don't want to edit and just look around which is the case for examining huge log files.

Search in less works like vi

Best part, it's available by default on most distros. So won't be problem for production environment as well.

2
  • Searching in 650MB text file with less proved to be a PITA. Using vim with LargeFile works like a charm.
    – MariusCC
    Apr 20, 2015 at 14:02
  • 2
    @MariusCC Then you haven't worked with more than 2 GB files, your charm will fade with crash!
    – deepdive
    Apr 21, 2015 at 7:17
-15

this is old but, use nano, vim or gvim

2
  • 5
    These tools do nothing to address the problem. Mar 8, 2013 at 17:08
  • 1
    nano fills memory and dies on me. May 29, 2015 at 17:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.