Are there any editors that can edit multi-gigabyte text files, perhaps by only loading small portions into memory at once? It doesn't seem like Vim can handle it =(
-
I've loaded really big data acquisition files in vim, and it handled them without problem.– RookMay 26, 2009 at 10:57
-
Depending on your editing needs, you may just be able to pipe it through something like sed or perl to do a search and replace.– El YoboJun 1, 2010 at 1:57
-
35Actually it's not off-topic, many programmers use vim, sometimes as a complement to UI editor. Topic question is about real problem. We all know only two such good swiss army tools for this kind of task, so please do not treat vim as too exotic or off-site. SO is for people.– Sławomir LenartSep 6, 2016 at 14:16
-
2Instead of closing it, why not move it to SuperUser or Linux/Unix, or VIM ?– NikeJan 25, 2020 at 15:41
15 Answers
Ctrl-C will stop file load. If the file is small enough you may have been lucky to have loaded all the contents and just killed any post load steps. Verify that the whole file has been loaded when using this tip.
Vim can handle large files pretty well. I just edited a 3.4GB file, deleting lines, etc. Three things to keep in mind:
- Press Ctrl-C: Vim tries to read in the whole file initially, to do things like syntax highlighting and number of lines in file, etc. Ctrl-C will cancel this enumeration (and the syntax highlighting), and it will only load what's needed to display on your screen. Do NOT save at this point, as it will probably lose data since it didn't load the whole file.
- Readonly: Vim will likely start read-only when the file is too big for it to make a . file copy to perform the edits on. I had to w! to save the file, and that's when it took the most time.
- Go to line: Typing
:115355
will take you directly to line 115355, which is much faster going in those large files. Vim seems to start scanning from the beginning every time it loads a buffer of lines, and holding down Ctrl-F to scan through the file seems to get really slow near the end of it.
Note - If your Vim instance is in readonly because you hit Ctrl-C, it is possible that Vim did not load the entire file into the buffer. If that happens, saving it will only save what is in the buffer, not the entire file. You might quickly check with a G
to skip to the end to make sure all the lines in your file are there.
-
20Was able to deal with 44 gigabyte wikipedia xml dump in vim using this advice. (ctrl-c). Nov 11, 2013 at 1:57
-
1Tried to read the end of 2.5GB log file on windows. Opening in gvim resulted in out of memory error when it exceeded 2GB memory allocated. When trying ctrl-c trick, it did stop loading the file into memory but only allowed to see the part of the file that gvim was able to load. So the longer I waited before pressing ctrl-c the more of the file I could see. Navigating to the end of file or loading rest of the file was impossible (or I didn't know how). Kinda disappointing that vim wasn't up to the task : ( In the end I used some free dedicated tool to split the file into 100MB files.– slawekJan 7, 2014 at 14:12
-
16Doesn't work for me. I load a 3GB file, press ctrl-c and then the contents show up. I can edit scroll etc but when I get to the end of loaded part (let's say 5%) it won't load anymore (I am stuck with the part of file that loaded initially up to the point that I pressed ctrl-c)– PatrykDec 16, 2014 at 12:02
-
1Confirmed, user3338098. If you press Ctrl-C and it doesn't load the whole file (as others have talked about), saving it only saves what you have loaded. That's probably why it goes into readonly in the first place. I'll update my Readonly point to note that.– Aaron R.May 5, 2015 at 13:37
-
20Following these instructions caused me to destroy a huge file I had just downloaded. You need to completely remove point 2 as it basically gives instructions that cause data loss, which you don't mention until the end of the post.– NeobyteAug 22, 2015 at 0:24
If you are on *nix (and assuming you have to modify only parts of file (and rarely)), you may split the files (using the split
command), edit them individually (using awk
, sed
, or something similar) and concatenate them after you are done.
cat file2 file3 >> file1
-
11Great tip. I had a 13GB (152.000.000 lines) sql-file, and just using "split -l 1000000" then editing the one million line files where I wanted with vim worked great. Took 10 minutes just to split them. (I tried to open the original file with vim and that worked, but it was too slow to be usable.) Jan 28, 2011 at 11:51
-
I've made the mistake and used
split -n 16
. This creates 16 files, which are of the same size. But the SQL syntax was broken after i usedcat
.split -n
was adding a line feed somewhere in a row. But you don't want this. Usesplit -l
instead, like suggested by Claes Mogren. Apr 13 at 6:28
It may be plugins that are causing it to choke. (syntax highlighting, folds etc.)
You can run vim without plugins.
vim -u "NONE" hugefile.log
It's minimalist but it will at least give you the vi motions you are used to.
syntax off
is another obvious one. Prune your install down and source what you need. You'll find out what it's capable of and if you need to accomplish a task via other means.
-
9
-
@Totor yeah I would split the file first but that setting would quickly give you the best vim performance by turning off random autocommands. That was my point. Workstations with decent memory should be able to handle files approaching a gig.– michaelMar 13, 2013 at 23:48
-
3so vim/vi is useless when the file is 10 times the size of virtual memory? May 4, 2015 at 16:06
-
1I used this command to open a 250MB file in under 2 seconds. Amazing Jun 18, 2015 at 19:31
-
1If the file being opened has a line over 10-100 megabytes on a single line, then vanilla vim with the
vim -u NONE hugefile.log
will still hang for 30 or 90 seconds as the C under the hood inflictsset wrap
upon the buffer. Remedy this by settingset nowrap
before you open the file. Command:help wrap
gives clues. Nov 10, 2020 at 3:26
A slight improvement on the answer given by @Al pachio with the split + vim solution you can read the files in with a glob, effectively using file chunks as a buffer e.g
$ split -l 5000 myBigFile
xaa
xab
xac
...
$ vim xa*
#edit the files
:nw #skip forward and write
:n! #skip forward and don't save
:Nw #skip back and write
:N! #skip back and don't save
You might want to check out this VIM plugin which disables certain vim features in the interest of speed when loading large files.
I've tried to do that, mostly with files around 1 GB when I needed to make some small change to an SQL dump. I'm on Windows, which makes it a major pain. It's seriously difficult.
The obvious question is "why do you need to?" I can tell you from experience having to try this more than once, you probably really want to try to find another way.
So how do you do it? There are a few ways I've done it. Sometimes I can get vim or nano to open the file, and I can use them. That's a really tough pain, but it works.
When that doesn't work (as in your case) you only have a few options. You can write a little program to make the changes you need (for example, search & replaces). You could use a command line program that may be able to do it (maybe it could be accomplished with sed/awk/grep/etc?)
If those don't work, you can always split the file into chunks (something like split being the obvious choice, but you could use head/tail to get the part you want) and then edit the part(s) that need it, and recombine later.
Trust me though, try to find another way.
-
3Usually sed is your friend in cases like this. Your editor really doesn't like the thought of inserting a few characters at the top of a file and figuring out how to push everything else down.– dkretzMay 26, 2009 at 1:45
-
@le dorfier: Yep. I used sed when I had to do a search / replace. When I had to delete a few lines from a file like that (a few insanely long lines) I managed to do it in vim, but as you can guess moving between lines (as well as the actual deletion) took quite a bit of time (seconds+ to respond and redraw). I wouldn't want to attempt adding even a few letters to one of those lines.– MBCookMay 26, 2009 at 1:49
-
Exact same issue.... a "using" statement at the top of a SQL script for a large table, or a file group that doesn't exist in the target system. I use Free File Splitter to bust them up, the the command line below to rejoin.– EBarrFeb 12, 2015 at 0:58
I think it is reasonably common for hex editors to handle huge files. On Windows, I use HxD, which claims to handle files up to 8 EB (8 billion gigabytes).
-
14
-
I'm using vim 7.3.3 on Win7 x64 with the LargeFile plugin by Charles Campbell to handle multi-gigabyte plain text files. It works really well.
I hope you come right.
-
How can you disable the plugin? Eg getting all other extensions such as highlighting to work again when a file is open in Vim?– hhhOct 9, 2017 at 10:33
Wow, never managed to get vim to choke, even with a GB or two. I've heard that UltraEdit (on Windows) and BBEdit (on Macs) are even more suitable for even-larger files, but I have no personal experience.
The only thing I've been able to use for something like that is my favorite Mac hex editor, 0XED. However, that was with files that I considered large at tens of megabytes. I'm not sure how far it will go. I'm pretty sure it only loads parts of the file into memory at once, though.
In the past I've successfully used a split/edit/join approach when files get very large. For this to work you have to know about where the to-be-edited text is, in the original file.