1

I've got a file with around 6MB of data. All of the data are written in a single line. Why is the following command taking more than 15 minutes to finish? Is it normal?

infile = open('file.txt')
outfile = open('out.txt', 'w')
for line in infile.readlines():
    outfile.write(line);

Details:

I'm using Python 2.7.

Output from wc:

  • newline count: 2
  • word count: 3475246
  • byte count: 6951140

Evaluation 1:

Reference code using in file (suggestted by Ahsanul Haque and Daewon Lee):

for line in infile:
    output.write(line):

Time: 959.487 secs.

11
  • this definitely seems fishy ... but outfile isn't opened in the correct mode for writing ...
    – mgilson
    Commented Aug 31, 2016 at 7:52
  • Sorry. I've mistyped.
    – rph
    Commented Aug 31, 2016 at 7:52
  • Can you please calculate number of characters in that line and put it in question? Will be helpful. Commented Aug 31, 2016 at 7:53
  • 1
    for line in infile:output.write(line) takes how much time? It should be faster. Commented Aug 31, 2016 at 8:01
  • 1
    I think your environment seems to have a problem.
    – Daewon Lee
    Commented Aug 31, 2016 at 8:47

2 Answers 2

1

Try the following code snippet. The readlines() loads all data on memory, which seems to cause a long time.

infile = open('file.txt', 'r')
outfile = open('out.txt', 'w')
for line in infile:
    outfile.write(line)

With my python 3.5 (64bit) on Windows 10 OS, the following code snippet finished within a few seconds.

import time

start = time.time()

with open("huge_text.txt", "w") as fout:
    for i in range(1737623):
        fout.write("ABCD ")
    fout.write('\n')   
    for i in range(1737623):
        fout.write("EFGH ")
    fout.write('\n')
# end of with

infile = open('huge_text.txt', 'r')
outfile = open('out.txt', 'w')
for line in infile:
    outfile.write(line)

outfile.close()
infile.close()

end = time.time()
print("Time elapsed: ", end - start)

"""
<Output>
Time elapsed:  1.557690143585205
"""
2
  • Thanks for the suggestion. However, seems not to have a major effect. So far it already took 959 secs (16 minutes).
    – rph
    Commented Aug 31, 2016 at 8:24
  • hardware matters, if you have old pc components with many limitations then i will execute slowly. Commented Aug 31, 2016 at 9:42
0

try reading file in chunks

infile = open('file.txt')
outfile = open('out.txt', 'w')
while True:
    text = infile.read(100): # or any other size
    if not text:
        break
    outfile.write(line);

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.