4

I have looked around StackOverflow and couldn't find an answer to my specific question so forgive me if I have missed something.

import re

target = open('output.txt', 'w')

for line in open('input.txt', 'r'):
    match = re.search(r'Stuff', line)
    if match:
        match_text = match.group()
        target.write(match_text + '\n')
    else:
        continue
target.close()

The file I am parsing is huge so need to process it line by line.

This (of course) leaves an additional newline at the end of the file.

How should I best change this code so that on the final iteration of the 'if match' loop it doesn't put the extra newline character at the end of the file. Should it look through the file again at the end and remove the last line (seems a bit inefficient though)?

The existing StackOverflow questions I have found cover removing all new lines from a file.

If there is a more pythonic / efficient way to write this code I would welcome suggestions for my own learning also.

Thanks for the help!

3
  • 4
    The newline at the end of the file is not "additional", it is the newline for the last line!
    – kindall
    Aug 9, 2013 at 3:02
  • People normally keep the final new line. That's why you don't see many examples otherwise.
    – tdelaney
    Aug 9, 2013 at 3:11
  • That is an interesting point @tdelaney. I had not thought of that, and I guess it is not really a big concern in a lot of circumstances. Aug 9, 2013 at 4:52

4 Answers 4

9

Another thing you can do, is to truncate the file. .tell() gives us the current byte number in the file. We then subtract one, and truncate it there to remove the trailing newline.

with open('a.txt', 'w') as f:
    f.write('abc\n')
    f.write('def\n')
    f.truncate(f.tell()-1)

On Linux and MacOS, the -1 is correct, but on Windows it needs to be -2. A more Pythonic method of determining which is to check os.linesep.

import os
remove_chars = len(os.linesep)

with open('a.txt', 'w') as f:
    f.write('abc\n')
    f.write('def\n')
    f.truncate(f.tell() - remove_chars)

kindal's answer is also valid, with the exception that you said it's a large file. This method will let you handle a terabyte sized file on a gigabyte of RAM.

8

Write the newline of each line at the beginning of the next line. To avoid writing a newline at the beginning of the first line, use a variable that is initialized to an empty string and then set to a newline in the loop.

import re

with open('input.txt') as source, open('output.txt', 'w') as target:

    newline = ''
    for line in source:
        match = re.search(r'Stuff', line)
        if match:
            target.write(newline + match.group())
            newline = '\n'

I also restructured your code a bit (the else: continue is not needed, because what else is the loop going to do?) and changed it to use the with statement so the files are automatically closed.

0
1

The shortest path from what you have to what you want is probably to store the results in a list, then join the list with newlines and write that to the file.

import re

target = open('output.txt', 'w')
results = []

for line in open('input.txt', 'r'):
    match = re.search(r'Stuff', line)
    if match:
        results.append(match.group())
target.write("\n".join(results))
target.close()

Voilà, no extra newline at the beginning or end. Might not scale very well of the resulting list is huge. (And like kindall I left out the else)

2
  • The file I am parsing is huge so need to process it line by line. Aug 9, 2013 at 4:04
  • I had considered this initially but the file is too big to fit in memory on the computer I am using Aug 9, 2013 at 4:51
1

Since you're performing the same regex over and over, you'd probably want to compile it beforehand.

import re
prog = re.compile(r'Stuff')

I tend to input from and output to stdin and stdout for simplicity. But that's a matter of taste (and specs).

from sys import stdin, stdout

Ignoring the specific requirement about removing the final EOL[1], and just addressing the bit about your own learning, the whole thing could be written like this:

from itertools import imap
stdout.writelines(match.group() for match in imap(prog.match, stdin) if match)

[1] As others have commented, this is a Bad Thing, and it's extremely annoying when someone does this.

2
  • I wrote a small script that loops 10,000 times through doing a re search on a string. The second version precompiled the re. The first called re.py line 230 10,000 times, the second refered to it 10,001 times. Plus one time for line 192 of the same file. The second version had 40k function calls and took twice as long as the first example (30 k function calls). Feb 21, 2017 at 8:31
  • the second refered to [_compile() on re.py line 230] 10,001 times: @ChristoferOhlsson It appears you're compiling your regex 10,001 times instead of compiling it just once. Did you use your precompiled regex correctly like in the example here?
    – antak
    Feb 22, 2017 at 1:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.