I'm writing an import script that processes a file that has potentially hundreds of thousands of lines (log file). Using a very simple approach (below) took enough time and memory that I felt like it would take out my MBP at any moment, so I killed the process.
#...
File.open(file, 'r') do |f|
f.each_line do |line|
# do stuff here to line
end
end
This file in particular has 642,868 lines:
$ wc -l nginx.log /code/src/myimport
642868 ../nginx.log
Does anyone know of a more efficient (memory/cpu) way to process each line in this file?
UPDATE
The code inside of the f.each_line
from above is simply matching a regex against the line. If the match fails, I add the line to a @skipped
array. If it passes, I format the matches into a hash (keyed by the "fields" of the match) and append it to a @results
array.
# regex built in `def initialize` (not on each line iteration)
@regex = /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - (.{0})- \[([^\]]+?)\] "(GET|POST|PUT|DELETE) ([^\s]+?) (HTTP\/1\.1)" (\d+) (\d+) "-" "(.*)"/
#... loop lines
match = line.match(@regex)
if match.nil?
@skipped << line
else
@results << convert_to_hash(match)
end
I'm completely open to this being an inefficient process. I could make the code inside of convert_to_hash
use a precomputed lambda instead of figuring out the computation each time. I guess I just assumed it was the line iteration itself that was the problem, not the per-line code.
each_line
. You could read the file in blocks which is faster, then useString#lines
to grab individual lines along with rejoining any partially loaded lines that crossed the block boundaries. It becomes a wash having to split out the lines and rejoin broken ones.