334

I want to read a large file (>5GB), line by line, without loading its entire contents into memory. I cannot use readlines() since it creates a very large list in memory.

1

14 Answers 14

446

Use a for loop on a file object to read it line-by-line. Use with open(...) to let a context manager ensure that the file is closed after reading:

with open("log.txt") as infile:
    for line in infile:
        print(line)
9
  • 50
    the question still is, "for line in infile" will load my 5GB of lines in to the memory? and, How can I read from tail? Jun 25, 2011 at 2:31
  • 108
    @rochacbruno, it only reads one line at a time. When the next line is read, the previous one will be garbage collected unless you have stored a reference to it somewhere else Jun 25, 2011 at 2:33
  • 2
    @rochacbruno, Reading the lines in reverse order is not as easy to do efficiently unfortunately. Generally you would want to read from the end of the file in sensible sized chunks (kilobytes to megabytes say) and split on newline characters ( or whatever the line ending char is on your platform) Jun 25, 2011 at 2:36
  • 4
    Thanks! I found the tail solution stackoverflow.com/questions/5896079/… Jun 25, 2011 at 3:09
  • 4
    @bawejakunal, Do you mean if a line is too long to load into memory at once? That is unusual for a text file. Instead of using for loop which iterates over the lines, you can use chunk = infile.read(chunksize) to read limited size chunks regardless of their content. You'll have to search inside the chunks for newlines yourself. Jan 9, 2018 at 21:50
80

All you need to do is use the file object as an iterator.

for line in open("log.txt"):
    do_something_with(line)

Even better is using context manager in recent Python versions.

with open("log.txt") as fileobject:
    for line in fileobject:
        do_something_with(line)

This will automatically close the file as well.

3
  • 4
    That is not loading whole file in to the memory? Jun 25, 2011 at 2:10
  • Shouldn't you close the file after the loop in the 1st example?
    – maciejwww
    Jun 9, 2021 at 10:13
  • 1
    @maciejwww yes, but I didn't to make it look more like the OP example. The second example uses the with statement is a "context manager" that automatically closes the fileobject.
    – Keith
    Jun 10, 2021 at 6:07
21

Please try this:

with open('filename','r',buffering=100000) as f:
    for line in f:
        print line
2
  • 8
    From Python's official docmunets: link The optional buffering argument specifies the file’s desired buffer size: 0 means unbuffered, 1 means line buffered, any other positive value means use a buffer of (approximately) that size (in bytes). A negative buffering means to use the system default, which is usually line buffered for tty devices and fully buffered for other files. If omitted, the system default is used
    – jyoti das
    Apr 19, 2018 at 5:26
  • @jyotidas While I like this method, you run the risk of having line in your text broken into chunks. I saw this personally, which means that if you are searching for sstring in the file like I was, I'd miss some because the line they were at were broken into chunks. Is there a way to get around this? Using readlines didn't work well as i got miscounts
    – edo101
    Jun 3, 2020 at 2:37
21

You are better off using an iterator instead.
Relevant: fileinput — Iterate over lines from multiple input streams.

From the docs:

import fileinput
for line in fileinput.input("filename", encoding="utf-8"):
    process(line)

This will avoid copying the whole file into memory at once.

1
  • Although the docs show the snippet as "typical use", using it does not call the close() method of the returned FileInput class object when the loop finishes -- so I would avoid using it this way. In Python 3.2 they've finally made fileinput compatible with the context manager protocol which addresses this issue (but the code still wouldn't be written quite the way shown).
    – martineau
    Jul 24, 2012 at 3:50
19

An old school approach:

fh = open(file_name, 'rt')
line = fh.readline()
while line:
    # do stuff with line
    line = fh.readline()
fh.close()
2
  • 2
    minor remark: for exception safety it is recommended to use 'with' statement, in your case "with open(filename, 'rt') as fh:"
    – prokher
    Jan 15, 2015 at 14:44
  • 23
    @prokher: Yeah, but I did call this "old school".
    – PTBNL
    Jan 16, 2015 at 13:40
15

Here's what you do if you dont have newlines in the file:

with open('large_text.txt') as f:
  while True:
    c = f.read(1024)
    if not c:
      break
    print(c,end='')
2
  • While I like this method, you run the risk of having line in your text broken into chunks. I saw this personally, which means that if you are searching for sstring in the file like I was, I'd miss some because the line they were at were broken into chunks. Is there a way to get around this? Using readlines didn't work well as i got miscounts @Ariel Cabib
    – edo101
    Jun 3, 2020 at 2:37
  • You could also use sys.stdout.write(c) instead of print()
    – sivann
    May 10, 2022 at 8:46
7

I couldn't believe that it could be as easy as @john-la-rooy's answer made it seem. So, I recreated the cp command using line by line reading and writing. It's CRAZY FAST.

#!/usr/bin/env python3.6

import sys

with open(sys.argv[2], 'w') as outfile:
    with open(sys.argv[1]) as infile:
        for line in infile:
            outfile.write(line)
1
  • 4
    NOTE: Because python's readline standardizes line endings, this has the side effect of converting documents with DOS line endings of \r\n to Unix line endings of \n. My whole reason for searching out this topic was that I needed to convert a log file that receives a jumble of line endings (because the developer blindly used various .NET libraries). I was shocked to find that after my initial speed test, I didn't need to go back and rstrip the lines. It was already perfect! Aug 11, 2017 at 13:13
5

The blaze project has come a long way over the last 6 years. It has a simple API covering a useful subset of pandas features.

dask.dataframe takes care of chunking internally, supports many parallelisable operations and allows you to export slices back to pandas easily for in-memory operations.

import dask.dataframe as dd

df = dd.read_csv('filename.csv')
df.head(10)  # return first 10 rows
df.tail(10)  # return last 10 rows

# iterate rows
for idx, row in df.iterrows():
    ...

# group by my_field and return mean
df.groupby(df.my_field).value.mean().compute()

# slice by column
df[df.my_field=='XYZ'].compute()
2

Heres the code for loading text files of any size without causing memory issues. It support gigabytes sized files

https://gist.github.com/iyvinjose/e6c1cb2821abd5f01fd1b9065cbc759d

download the file data_loading_utils.py and import it into your code

usage

import data_loading_utils.py.py
file_name = 'file_name.ext'
CHUNK_SIZE = 1000000


def process_lines(data, eof, file_name):

    # check if end of file reached
    if not eof:
         # process data, data is one single line of the file

    else:
         # end of file reached

data_loading_utils.read_lines_from_file_as_data_chunks(file_name, chunk_size=CHUNK_SIZE, callback=self.process_lines)

process_lines method is the callback function. It will be called for all the lines, with parameter data representing one single line of the file at a time.

You can configure the variable CHUNK_SIZE depending on your machine hardware configurations.

1
  • 1
    While I like this method, you run the risk of having line in your text broken into chunks. I saw this personally, which means that if you are searching for sstring in the file like I was, I'd miss some because the line they were at were broken into chunks. Is there a way to get around this? Using readlines didn't work well as i got miscounts
    – edo101
    Jun 3, 2020 at 2:38
1

I realise this has been answered quite some time ago, but here is a way of doing it in parallel without killing your memory overhead (which would be the case if you tried to fire each line into the pool). Obviously swap the readJSON_line2 function out for something sensible - its just to illustrate the point here!

Speedup will depend on filesize and what you are doing with each line - but worst case scenario for a small file and just reading it with the JSON reader, I'm seeing similar performance to the ST with the settings below.

Hopefully useful to someone out there:

def readJSON_line2(linesIn):
  #Function for reading a chunk of json lines
   '''
   Note, this function is nonsensical. A user would never use the approach suggested 
   for reading in a JSON file, 
   its role is to evaluate the MT approach for full line by line processing to both 
   increase speed and reduce memory overhead
   '''
   import json

   linesRtn = []
   for lineIn in linesIn:

       if lineIn.strip() != 0:
           lineRtn = json.loads(lineIn)
       else:
           lineRtn = ""
        
       linesRtn.append(lineRtn)

   return linesRtn




# -------------------------------------------------------------------
if __name__ == "__main__":
   import multiprocessing as mp

   path1 = "C:\\user\\Documents\\"
   file1 = "someBigJson.json"

   nBuffer = 20*nCPUs  # How many chunks are queued up (so cpus aren't waiting on processes spawning)
   nChunk = 1000 # How many lines are in each chunk
   #Both of the above will require balancing speed against memory overhead

   iJob = 0  #Tracker for SMP jobs submitted into pool
   iiJob = 0  #Tracker for SMP jobs extracted back out of pool

   jobs = []  #SMP job holder
   MTres3 = []  #Final result holder
   chunk = []  
   iBuffer = 0 # Buffer line count
   with open(path1+file1) as f:
      for line in f:
            
          #Send to the chunk
          if len(chunk) < nChunk:
              chunk.append(line)
          else:
              #Chunk full
              #Don't forget to add the current line to chunk
              chunk.append(line)
                
              #Then add the chunk to the buffer (submit to SMP pool)                  
              jobs.append(pool.apply_async(readJSON_line2, args=(chunk,)))
              iJob +=1
              iBuffer +=1
              #Clear the chunk for the next batch of entries
              chunk = []
                            
          #Buffer is full, any more chunks submitted would cause undue memory overhead
          #(Partially) empty the buffer
          if iBuffer >= nBuffer:
              temp1 = jobs[iiJob].get()
              for rtnLine1 in temp1:
                  MTres3.append(rtnLine1)
              iBuffer -=1
              iiJob+=1
            
      #Submit the last chunk if it exists (as it would not have been submitted to SMP buffer)
      if chunk:
          jobs.append(pool.apply_async(readJSON_line2, args=(chunk,)))
          iJob +=1
          iBuffer +=1

      #And gather up the last of the buffer, including the final chunk
      while iiJob < iJob:
          temp1 = jobs[iiJob].get()
          for rtnLine1 in temp1:
              MTres3.append(rtnLine1)
          iiJob+=1

   #Cleanup
   del chunk, jobs, temp1
   pool.close()
0

How about this? Divide your file into chunks and then read it line by line, because when you read a file, your operating system will cache the next line. If you are reading the file line by line, you are not making efficient use of the cached information.

Instead, divide the file into chunks and load the whole chunk into memory and then do your processing.

def chunks(file,size=1024):
    while 1:

        startat=fh.tell()
        print startat #file's object current position from the start
        fh.seek(size,1) #offset from current postion -->1
        data=fh.readline()
        yield startat,fh.tell()-startat #doesnt store whole list in memory
        if not data:
            break
if os.path.isfile(fname):
    try:
        fh=open(fname,'rb') 
    except IOError as e: #file --> permission denied
        print "I/O error({0}): {1}".format(e.errno, e.strerror)
    except Exception as e1: #handle other exceptions such as attribute errors
        print "Unexpected error: {0}".format(e1)
    for ele in chunks(fh):
        fh.seek(ele[0])#startat
        data=fh.read(ele[1])#endat
        print data
1
  • 1
    This looks promising. Is this loading by bytes or by lines? I'm afraid of lines being broken if it's by bytes.. how can we load say 1000 lines at a time and process that?
    – Nikhil VJ
    Mar 31, 2018 at 3:59
0

Thank you! I have recently converted to python 3 and have been frustrated by using readlines(0) to read large files. This solved the problem. But to get each line, I had to do a couple extra steps. Each line was preceded by a "b'" which I guess that it was in binary format. Using "decode(utf-8)" changed it ascii.

Then I had to remove a "=\n" in the middle of each line.

Then I split the lines at the new line.

b_data=(fh.read(ele[1]))#endat This is one chunk of ascii data in binary format
        a_data=((binascii.b2a_qp(b_data)).decode('utf-8')) #Data chunk in 'split' ascii format
        data_chunk = (a_data.replace('=\n','').strip()) #Splitting characters removed
        data_list = data_chunk.split('\n')  #List containing lines in chunk
        #print(data_list,'\n')
        #time.sleep(1)
        for j in range(len(data_list)): #iterate through data_list to get each item 
            i += 1
            line_of_data = data_list[j]
            print(line_of_data)

Here is the code starting just above "print data" in Arohi's code.

0

The best solution I found regarding this, and I tried it on 330 MB file.

lineno = 500
line_length = 8
with open('catfour.txt', 'r') as file:
    file.seek(lineno * (line_length + 2))
    print(file.readline(), end='')

Where line_length is the number of characters in a single line. For example "abcd" has line length 4.

I have added 2 in line length to skip the '\n' character and move to the next character.

-1

This might be useful when you want to work in parallel and read only chunks of data but keep it clean with new lines.

def readInChunks(fileObj, chunkSize=1024):
    while True:
        data = fileObj.read(chunkSize)
        if not data:
            break
        while data[-1:] != '\n':
            data+=fileObj.read(1)
        yield data

Not the answer you're looking for? Browse other questions tagged or ask your own question.