4

Let's say I want to combine several massive files into one and then uniq! the one (THAT alone might take a hot second)

It's my understanding that File.readlines() loads ALL the lines into memory. Is there a way to read it line by line, sort of like how node.js pipe() system works?

1
  • Do you definitely want to do it line by line (as opposed to say 1Mb at a time) ? Aug 19, 2016 at 8:46

4 Answers 4

6

One of the great things about Ruby is that you can do file IO in a block:

File.open("test.txt", "r").each_line do |row|
  puts row
end               # file closed here

so things get cleaned up automatically. Maybe it doesn't matter on a little script but it's always nice to know you can get it for free.

3

you aren't operating on the entire file contents at once, and you don't need to store the entirety of each line either if you use readline.

file = File.open("sample.txt", 'r')
while !file.eof?
   line = file.readline
   puts line
end
2
  • If this is the case, why does loading a 350Mb file with "readlines" takes like 5 seconds? I just assumed it is "preloading" the array
    – dsp_099
    Aug 19, 2016 at 5:23
  • Yeah so I had the same problem but the readline saved a lot of my time. If this works please accept answer.
    – Muaaz Rafi
    Aug 19, 2016 at 5:25
1

Large files are best read by streaming methods like each_line as shown in the other answer or with foreach which opens the file and reads line by line. So if the process doesn't request to have the whole file in memory you should use the streaming methods. While using streaming the required memory won't increase even if the file size increases opposing to non-streaming methods like readlines.

File.foreach("name.txt") { |line| puts line }

1

uniq! is defined on Array, so you'll have to read the files into an Array anyway. You cannot process the file line-by-line because you don't want to process a file, you want to process an Array, and an Array is a strict in-memory data structure.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.