0

I'm new to Python, and I need to do a parsing exercise. I got a file, and I need to parse it (just the headers), but after the process, i need to keep the file the same format, the same extension, and at the same place in disk, but only with the differences of new headers..

I tried this code...

for line in open ('/home/name/db/str/dir/numbers/str.phy'):
    if line.startswith('ENS'):
        linepars = re.sub ('ENS([A-Z]+)0+([0-9]{6})','\\1\\2',line)
        print linepars

..and it does the job, but I don't know how to "overwrite" the file with the new parsing.

2
  • You can't output to a temp file, and then delete the original and replace with the temp?
    – Dan
    May 19, 2011 at 17:52
  • You may wish to say what size / how many lines you expect such files to be.
    – ninjagecko
    May 19, 2011 at 19:21

5 Answers 5

4

The easiest way, but not the most efficient (by far, and especially for long files) would be to rewrite the complete file.

You could do this by opening a second file handle and rewriting each line, except in the case of the header, you'd write the parsed header. For example,

fr = open('/home/name/db/str/dir/numbers/str.phy')
fw = open('/home/name/db/str/dir/numbers/str.phy.parsed', 'w') # Name this whatever makes sense

for line in fr:
    if line.startswith('ENS'):
        linepars = re.sub ('ENS([A-Z]+)0+([0-9]{6})','\\1\\2',line)
        fw.write(linepars)
    else:
        fw.write(line)

fw.close()
fr.close()

EDIT: Note that this does not use readlines(), so its more memory efficient. It also does not store every output line, but only one at a time, writing it to file immediately.

Just as a cool trick, you could use the with statement on the input file to avoid having to close it (Python 2.5+):

fw = open('/home/name/db/str/dir/numbers/str.phy.parsed', 'w') # Name this whatever makes sense

with open('/home/name/db/str/dir/numbers/str.phy') as fr:
    for line in fr:
        if line.startswith('ENS'):
            linepars = re.sub ('ENS([A-Z]+)0+([0-9]{6})','\\1\\2',line)
            fw.write(linepars)
        else:
             fw.write(line)

fw.close()

P.S. Welcome :-)

7
  • Not that there's anything wrong with the cool trick, but I generally prefer the explicit file closure.
    – Greg
    May 19, 2011 at 19:55
  • 4
    with is not just a cool trick, it is a new idiom for Python, from version 2.5 onward. If exceptions are raised and there's no handler or finally clause, your explicit file closure may never get called. with takes care of all that. And its not just for files - this pattern, called a context manager, can be used for thread locking, database transaction commit/rollback, or any other resource you need to be sure returns to a base state when your code is done. Time to learn a new trick.
    – PaulMcG
    May 19, 2011 at 20:10
  • I'd imagine it would have great use in sockets as well -- thanks for the note.
    – jedwards
    May 19, 2011 at 20:18
  • Thanks guys for all your advices...!! I'll use this solution.. :D
    – peixe
    May 20, 2011 at 7:54
  • @jedwards "The easiest way would be to rewrite the complete file" It isn't the easiest way, it is the only one, since it isn't possible to cut bits in a hard disk to let holes replacing the characters to be "removed". Removing characters means the file must be rewritten while putting apart the characters to remove. Then the point is the manner the rewritting is performed. You choosed to "do this by opening a second file handle and rewriting each line" , but that's not what the OP wanted: he asked for the file to be rewritten "at the same place in disk". See my answer that does overwrite
    – eyquem
    Dec 26, 2011 at 14:25
2

As others are saying here, you want to open a file and use that file object's .write() method.

The best approach would be to open an additional file for writing:

import os

current_cfg = open(...)
parsed_cfg  = open(..., 'w')
for line in current_cfg:
    new_line = parse(line)
    print new_line
    parsed.cfg.write(new_line + '\n')
current_cfg.close()
parsed_cfg.close()

os.rename(....) # Rename old file to backup name
os.rename(....) # Rename new file into place

Additionally I'd suggest looking at the tempfile module and use one of its methods for either naming your new file or opening/creating it. Personally I'd favor putting the new file in the same directory as the existing file to ensure that os.rename will work atomically (the configuration file named will be guaranteed to either point at the old file or the new file; in no case would it point at a partially written/copied file).

0

The following code DOES the job.
I mean it DOES overwrite the file ON ONESELF; that's what the OP asked for. That's possible because the transformations are only removing characters, so the file's pointer fo that writes is always BEHIND the file's pointer fi that reads.

import re

regx = re.compile('\AENS([A-Z]+)0+([0-9]{6})')

with open('bomo.phy','rb+') as fi, open('bomo.phy','rb+') as fo:
    fo.writelines(regx.sub('\\1\\2',line) for line in fi)

I think that the writing isn't performed by the operating system one line at a time but through a buffer. So several lines are read before a pool of transformed lines are written. That's what I think.

-1
newlines = []
for line in open ('/home/name/db/str/dir/numbers/str.phy').readlines():
    if line.startswith('ENS'):
        linepars = re.sub ('ENS([A-Z]+)0+([0-9]{6})','\\1\\2',line)
        newlines.append( linepars )
open ('/home/name/db/str/dir/numbers/str.phy', 'w').write('\n'.join(newlines))
6
  • 1
    This should work, but for long files it requires you to have two copies of the file in memory, which might be too much.
    – jedwards
    May 19, 2011 at 17:58
  • I don't think one would care about a text file that is over 1-4GB in size, unless you're dealing with textual database dumps (of which this may be a case), but which seem silly and which I have never seen above a hundred megabytes. Never optimize before you need to.
    – ninjagecko
    May 19, 2011 at 18:10
  • Specificially, for long files, readlines() and join() are not memory efficient. You'd be better off reading one line at a time and writing one line at a time.
    – jedwards
    May 19, 2011 at 18:11
  • "Never optimize before you need to." -- Is that (c) Microsoft? :-)
    – jedwards
    May 19, 2011 at 18:13
  • 1
    @jedwards - no it was Donald Knuth, although you are paraphrasing.
    – PaulMcG
    May 19, 2011 at 20:12
-1

(sidenote: Of course if you are working with large files, you should be aware that the level of optimization required may depend on your situation. Python by nature is very non-lazily-evaluated. The following solution is not a good choice if you are parsing large files, such as database dumps or logs, but a few tweaks such as nesting the with clauses and using lazy generators or a line-by-line algorithm can allow O(1)-memory behavior.)

targetFile = '/home/name/db/str/dir/numbers/str.phy'

def replaceIfHeader(line):
    if line.startswith('ENS'):
        return re.sub('ENS([A-Z]+)0+([0-9]{6})','\\1\\2',line)
    else:
        return line

with open(targetFile, 'r') as f:
    newText = '\n'.join(replaceIfHeader(line) for line in f)

try:
    # make backup of targetFile
    with open(targetFile, 'w') as f:
        f.write(newText)
except:
    # error encountered, do something to inform user where backup of targetFile is

edit: thanks to Jeff for suggestion

10
  • For long files, readlines() and join() are not memory efficient.
    – jedwards
    May 19, 2011 at 18:10
  • That is an absolutely preposterous downvote, when was the last time one saw a text file over 1GB in size? Maybe if you're dealing with textual database dumps, but those are so rare (and one is typically doing something wrong anyway) why bother. Never optimize before you need to, at least not in cases where you definitely don't need to. Else one would write everything in C. Code readability, elegance, and clearness take precedence unless the program runs much slower or hogs too much memory. Besides, only one file will be parsed at a time. I could say your solution uses unnecessary disk space.
    – ninjagecko
    May 19, 2011 at 18:14
  • 2
    Actually, I often deal with text files of 500 MB+ in my work. They're usually output from scripts that run over a few days. And the problem isn't so much that you're using all the memory needlessly, but rather the time it takes. I downvoted because your solution is needlessly inefficient, you downvoted my answer out of spite...
    – jedwards
    May 19, 2011 at 18:17
  • No, I downvoted your answer for the same reason you downvoted mine: inefficiency. You are also rewriting the entire file and using (as you allege we care about) an extra hypothetical 500MB of unnecessary disk-writing-time to replace a few lines of header files; if we really cared we'd modify it in-place with filesystem tricks if the inode structure allows in-place shrinking. (Also you did not use the with statement which I found quite inelegant, but it seems you have added it. This is not nearly as bad as not throwing an error if some bad occurs to prevent corrupting the data.)
    – ninjagecko
    May 19, 2011 at 18:50
  • 1
    Rather than reading the whole file in to a string only to later split it up again then rejoin them, why not save a few steps by building the newText as you read the file in? i.e., newText = '\n'.join(replaceIfHeader(line) for line in f) May 19, 2011 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.