1

I'm trying to substitute only the first match of one string in a huge file with only one line (2.1 GB), this substitution will occur in a shell script job. The big problem is that the machine that will run this script has only 1GB memory (approximately 300MB free), so i need a buffered strategy that don't overflow my memory. I already tried sed, perl and a python approach, but all of them return me out of memory errors. Here are my attemps (discovered in other questions):

# With perl
perl -pi -e '!$x && s/FROM_STRING/TO_STRING/ && ($x=1)' file.txt

# With sed
sed '0,/FROM_STRING/s//TO_STRING/' file.txt > file.txt.bak

# With python (in a custom script.py file)
for line in fileinput.input('file.txt', inplace=True):
    print line.replace(FROM_STRING, TO_STRING, 1)
    break

One good point is that the FROM_STRING that i'm searching is always in the beggining of this huge 1-line file, in the first 100 characters. Other good thing is that the execution time is not a problem, it can take time without problems.

EDIT (SOLUTION):

I tested three solutions of the answers, all them solved the problem, thanks for all of you. I tested the performance with Linux time and all of them take about the same time too, up to 10 seconds approximately... But i choose the @Miller solution because it's simpler (just uses perl).

2
  • @AvinashRaj Because the entire file is 1 line, so that will still attempt to load 2 gigs into memory.
    – Miller
    Sep 1, 2014 at 21:31
  • How long is the match string? Is replacement string the same length?
    – AnFi
    Sep 1, 2014 at 21:35

5 Answers 5

6

Since you know that your string is always in the first chunk of the file, you should use dd for this. You'll also need a temporary file to work with, as in tmpfile="$(mktemp)"

First, copy the first block of the file to a new, temporary location: dd bs=32k count=1 if=file.txt of="$tmpfile"

Then, do your substitution on that block: sed -i 's/FROM_STRING/TO_STRING/' "$tmpfile"

Next, concatenate the new first block with the rest of the old file, again using dd: dd bs=32k if=file.txt of="$tmpfile" seek=1 skip=1


EDIT: As per Mark Setchell's suggestion, I have added a specification of bs=32k to these commands to speed up the pace of the dd operations. This is tunable, per your needs, but if tuning separate commands distinctly, you may need to be careful about the changes in semantics between different input and output block sizes.

1
  • 2
    +1 A good approach, but I would take maybe 32kB off the start and then put the file back together using 32kB chunks because dd is often much faster with bigger blocks. Specifically, I mean to add bs=32k to your dd commands. Sep 1, 2014 at 21:20
2

If you're certain the string you're trying to replace is just in the first 100 characters, then the following perl one-liner should work:

perl -i -pe 'BEGIN {$/ = \1024} s/FROM_STRING/TO_STRING/ .. undef' file.txt

Explanation:

Switches:

  • -i: Edit <> files in place (makes backup if extension supplied)
  • -p: Creates a while(<>){...; print} loop for each “line” in your input file.
  • -e: Tells perl to execute the code on command line.

Code:

  • BEGIN {$/ = \1024}: Set the $INPUT_RECORD_SEPARATOR to the number of characters to read for each “line”
  • s/FROM/TO/ .. undef: Use a flip-flop to perform the regex only once. Could also have used if $. == 1.
3
  • triggers a harmless warning if you enable warnings (which you should, even on oneliners)
    – ysth
    Sep 1, 2014 at 21:09
  • AFAIK -i doesn't really edit the file inplace, it makes a copy of it, so will that really work with a 2G file and 1G of memory?
    – Ed Morton
    Sep 2, 2014 at 3:17
  • 1
    @Ed Morton, You'll need 4GB of disk space.
    – ikegami
    Sep 2, 2014 at 4:09
1
  • Given that then string to replace is in the first 100 bytes,
  • Given that Perl IO is slow unless you start using sysread to read large blocks,
  • Assuming that the substitution changes the size of the file[1], and
  • Assuming that binmode isn't needed[2],

I'd use

( head -c 100 | perl -0777pe's/.../.../' && cat ) <file.old >file.new

  1. A faster solution for that exists.
  2. Though it's easy to add if needed.
4
  • nice; I'd assumed trying to do something like that would result in head discarding a partial block, but it doesn't seem to have that problem.
    – ysth
    Sep 1, 2014 at 22:34
  • @ysth, The original version of my code (as seen in the answer's edit history) would work on systems such as the one you describe.
    – ikegami
    Sep 2, 2014 at 4:08
  • FWIW, I saw no speed difference between this answer and mine, though I was doing other stuff on the system at the time.
    – ysth
    Sep 2, 2014 at 5:39
  • Could be my second "given" isn't really given.
    – ikegami
    Sep 2, 2014 at 6:52
0

Untested, but I would do:

perl -pi -we 'BEGIN{$/=\65536} s/FROM_STRING/TO_STRING/ if 1..1' file.txt

to read in 64k chunks.

2
  • 1
    Note: That still read from the disk in 8K chunks (or 4K on the older Perls that most people use). Far better performance can be obtained by using sysread to actually read large blocks.
    – ikegami
    Sep 1, 2014 at 22:21
  • makes almost no difference that I can tell.
    – ysth
    Sep 1, 2014 at 22:49
0

A practical (not very compact but efficient) would be to split the file, do the search-replace and join: eg:

head -c 100 myfile | sed 's/FROM/TO/' > output.1
tail -c +101 myfile > output.2
cat output.1 output.2 > output && /bin/rm output.1 output.2

Or, in one line:

( ( head -c 100 myfile | sed 's/FROM/TO/' ) && (tail -c +101 myfile ) ) > output
1
  • I wrote an answer very similar to yours before noticing yours. However, mine does far less work to achieve the same result.
    – ikegami
    Sep 1, 2014 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.