780

I am trying to read a large file one line at a time. I found a question on Quora that dealt with the subject but I'm missing some connections to make the whole thing fit together.

 var Lazy=require("lazy");
 new Lazy(process.stdin)
     .lines
     .forEach(
          function(line) { 
              console.log(line.toString()); 
          }
 );
 process.stdin.resume();

The bit that I'd like to figure out is how I might read one line at a time from a file instead of STDIN as in this sample.

I tried:

 fs.open('./VeryBigFile.csv', 'r', '0666', Process);

 function Process(err, fd) {
    if (err) throw err;
    // DO lazy read 
 }

but it's not working. I know that in a pinch I could fall back to using something like PHP, but I would like to figure this out.

I don't think the other answer would work as the file is much larger than the server I'm running it on has memory for.

8
  • 3
    This turns out to be quite difficult using just low-level fs.readSync(). You can read binary octets into a buffer but there's no easy way to deal with partial UTF-8 or UTF-16 characters without inspecting the buffer before translating it to JavaScript strings and scanning for EOLs. The Buffer() type doesn't have as rich set of functions to operate on its instances as native strings, but native strings cannot contain binary data. It seems to me that lacking a built-in way to read text lines from arbitrary filehandles is a real gap in node.js. Commented Jan 2, 2013 at 1:52
  • 5
    Empty lines read in by this method get converted to a line with a single 0 (actual character code for 0) in them. I had to hack this line in there: if (line.length==1 && line[0] == 48) special(line);
    – Thabo
    Commented Aug 9, 2013 at 15:28
  • 2
    One might also use the 'line-by-line' package which does the job perfectly.
    – Patrice
    Commented Feb 7, 2014 at 8:49
  • 1
    Please update the question to say that the solution is to use a transform stream Commented Jun 8, 2014 at 6:33
  • 2
    @DanDascalescu if you like you can add this to the list: your example landed slightly modified in node's API docs github.com/nodejs/node/pull/4609 Commented Jan 11, 2016 at 19:47

31 Answers 31

1144

Since Node.js v0.12 and as of Node.js v4.0.0, there is a stable readline core module. Here's the easiest way to read lines from a file, without any external modules:

const fs = require('fs');
const readline = require('readline');

async function processLineByLine() {
  const fileStream = fs.createReadStream('input.txt');

  const rl = readline.createInterface({
    input: fileStream,
    crlfDelay: Infinity
  });
  // Note: we use the crlfDelay option to recognize all instances of CR LF
  // ('\r\n') in input.txt as a single line break.

  for await (const line of rl) {
    // Each line in input.txt will be successively available here as `line`.
    console.log(`Line from file: ${line}`);
  }
}

processLineByLine();

Or alternatively:

var lineReader = require('readline').createInterface({
  input: require('fs').createReadStream('file.in')
});

lineReader.on('line', function (line) {
  console.log('Line from file:', line);
});

lineReader.on('close', function () {
    console.log('all done, son');
});

The last line is read correctly (as of Node v0.12 or later), even if there is no final \n.

UPDATE: this example has been added to Node's API official documentation.

23
  • 8
    you need a terminal:false in the createInterface definition
    – glasspill
    Commented Sep 17, 2015 at 13:06
  • 76
    How to determine the last line? By catching a "close" event: rl.on('close', cb)
    – Green
    Commented Sep 27, 2015 at 16:04
  • 42
    Readline is for a similar purpose as GNU Readline, not for reading files line by line. There are several caveats in using it to read files and this is not a best practice.
    – Nakedible
    Commented Oct 10, 2015 at 11:42
  • 11
    @Nakedible: interesting. Could you post an answer with a better method? Commented Oct 14, 2015 at 23:23
  • 10
    I consider github.com/jahewson/node-byline to be the best implementation of line-by-line reading, but opinions may vary.
    – Nakedible
    Commented Oct 15, 2015 at 10:42
180

For such a simple operation there shouldn't be any dependency on third-party modules. Go easy.

var fs = require('fs'),
    readline = require('readline');

var rd = readline.createInterface({
    input: fs.createReadStream('/path/to/file'),
    output: process.stdout,
    console: false
});

rd.on('line', function(line) {
    console.log(line);
});

rd.on('close', function() {
    console.log('all done, son');
});
7
  • 41
    sadly, this attractive solution does'nt work correctly—line events come only after hitting \n, ie, all the alternatives are missed (see unicode.org/reports/tr18/#Line_Boundaries). #2, data after the last \n is silently ignored (see stackoverflow.com/questions/18450197/…). i'd call this solution dangerous cause it works for 99% of all files and for 99% of the data but fails silently for the rest. whenever you do fs.writeFileSync( path, lines.join('\n')) you've written a file that will only be partly read by above solution.
    – flow
    Commented Aug 27, 2013 at 15:56
  • 4
    There is a problem with this solution. If you use your.js <lines.txt you don't get last line. If it don't have a '\n' in the end of course.
    – zag2art
    Commented Jan 11, 2014 at 12:31
  • 1
    The readline package behaves in truly bizarre ways to an experienced Unix/Linux programmer.
    – Pointy
    Commented Oct 21, 2014 at 23:41
  • 13
    rd.on("close", ..); can be used as a callback (occurrs when all lines are read)
    – Luca Steeb
    Commented Feb 16, 2015 at 23:39
  • 8
    The "data after the last \n" issue seems to be resolved in my version of node (0.12.7). So I prefer this answer, which seems the simplest and most elegant.
    – Myk Melez
    Commented Sep 1, 2015 at 22:23
73

Update in 2019

An awesome example is already posted on official Nodejs documentation. here

This requires the latest Nodejs is installed on your machine. >11.4

const fs = require('fs');
const readline = require('readline');

async function processLineByLine() {
  const fileStream = fs.createReadStream('input.txt');

  const rl = readline.createInterface({
    input: fileStream,
    crlfDelay: Infinity
  });
  // Note: we use the crlfDelay option to recognize all instances of CR LF
  // ('\r\n') in input.txt as a single line break.

  for await (const line of rl) {
    // Each line in input.txt will be successively available here as `line`.
    console.log(`Line from file: ${line}`);
  }
}

processLineByLine();
5
  • 3
    this answer is much better than anything above thanks to its promise-based behaviour, distinctively indicating the EOF.
    – phil294
    Commented Sep 4, 2019 at 21:39
  • Thanks, that's sweet. Commented Oct 23, 2019 at 8:42
  • 15
    Maybe this is obvious to others, but it took me a while to debug: if you have any awaits between the createInterface() call and the start of the for await loop, you will mysteriously lose lines from the start of the file. createInterface() immediately starts emitting lines behind the scenes, and the async iterator implicitly created with const line of rl can’t start listening for those lines until it is created.
    – andrewdotn
    Commented Nov 9, 2019 at 16:09
  • how do you pause? rl.pause() && fileStream.pause() is not working. is there any way to stop the stream while processing the lines, I do not want them to pile up in memory.
    – Loebre
    Commented Dec 3, 2023 at 11:17
  • Unfortunately AirBnB eslint seems to dislike for...of... Commented Aug 7 at 15:44
61
require('fs').readFileSync('file.txt', 'utf-8').split(/\r?\n/).forEach(function(line){
  console.log(line);
})
3
  • 75
    This will read the entire file in memory, then split it into lines. It's not what the questions asks. The point is to be able to read large files sequentially, on demand. Commented Oct 25, 2013 at 10:50
  • 8
    This fits my use case, I was looking for a simply way to convert input from one script into another format. Thanks!
    – Callat
    Commented Aug 12, 2019 at 15:56
  • 4
    This might not answer the original question, but is still useful if it fits your memory constraints. Commented Jan 29, 2021 at 16:32
60

You don't have to open the file, but instead, you have to create a ReadStream.

fs.createReadStream

Then pass that stream to Lazy

8
  • 2
    Is there something like an end event for Lazy? When all lines have been read in?
    – BMBM
    Commented Jun 15, 2011 at 8:08
  • 1
    @Max, Try: new lazy(fs.createReadStream('...')).lines.forEach(function(l) { /* ... */ }).join(function() { /* Done */ })
    – Cecchi
    Commented Sep 20, 2012 at 19:39
  • 6
    @Cecchi and @Max, don't use join because it will buffer the entire file in memory. Instead, just listen to the 'end' event: new lazy(...).lines.forEach(...).on('end', function() {...})
    – Corin
    Commented Oct 17, 2012 at 12:33
  • 3
    @Cecchi, @Corin, and @Max: For what it's worth, I drove myself crazy chaining .on('end'... after .forEach(...), when in fact everything behaved as expected when I bound the event first.
    – crowjonah
    Commented Sep 6, 2013 at 19:05
  • 60
    This result is very high on search results, so it is worth noting that Lazy looks abandoned. It has been 7 months without any changes, and has some horrifying bugs (last line ignored, massive memory leaks, etc).
    – blu
    Commented Nov 20, 2013 at 21:21
43

there is a very nice module for reading a file line by line, it's called line-reader

with it you simply just write:

var lineReader = require('line-reader');

lineReader.eachLine('file.txt', function(line, last) {
  console.log(line);
  // do whatever you want with line...
  if(last){
    // or check if it's the last one
  }
});

you can even iterate the file with a "java-style" interface, if you need more control:

lineReader.open('file.txt', function(reader) {
  if (reader.hasNextLine()) {
    reader.nextLine(function(line) {
      console.log(line);
    });
  }
});
5
  • 4
    This works well. It even reads the last line (!). It is worth mentionning that it keeps the \r if it is a windows style text file. line.trim() does the trick of removing the extra \r. Commented Mar 4, 2014 at 18:41
  • It's sub-optimal in that input can only be from a named file, and not (for an obvious and extremely important example, process/stdin). At least, if it can, it's certainly not obvious from reading the code and attempting it.
    – Pointy
    Commented Oct 21, 2014 at 23:40
  • 2
    In the meantime there's a built-in way to read lines from a file, using the readline core module. Commented Sep 16, 2015 at 3:04
  • This is old, but in case anyone stumbles upon it: function(reader) and function(line) should be: function(err,reader) and function(err,line).
    – jallmer
    Commented Nov 27, 2018 at 13:50
  • 3
    Just for the record, line-reader reads the file asynchronously. The synchronous alternative to it is line-reader-sync
    – Prajwal
    Commented Mar 25, 2019 at 6:50
19

You can always roll your own line reader. I have'nt benchmarked this snippet yet, but it correctly splits the incoming stream of chunks into lines without the trailing '\n'

var last = "";

process.stdin.on('data', function(chunk) {
    var lines, i;

    lines = (last+chunk).split("\n");
    for(i = 0; i < lines.length - 1; i++) {
        console.log("line: " + lines[i]);
    }
    last = lines[i];
});

process.stdin.on('end', function() {
    console.log("line: " + last);
});

process.stdin.resume();

I did come up with this when working on a quick log parsing script that needed to accumulate data during the log parsing and I felt that it would nice to try doing this using js and node instead of using perl or bash.

Anyway, I do feel that small nodejs scripts should be self contained and not rely on third party modules so after reading all the answers to this question, each using various modules to handle line parsing, a 13 SLOC native nodejs solution might be of interest .

6
  • There doesn't seem to be any trivial way to extend this to work with arbitrary files besides just stdin... unless I'm missing somethign. Commented Jan 2, 2013 at 1:55
  • 3
    @hippietrail you can create a ReadStream with fs.createReadStream('./myBigFile.csv') and use it instead of stdin
    – nolith
    Commented May 8, 2013 at 12:57
  • 2
    Is each chunk guaranteed to contain only complete lines? Are multi-byte UTF-8 characters guaranteed not to be split at chunk boundaries? Commented May 8, 2013 at 22:38
  • 1
    @hippietrail I dont think that multibyte characters is handled correctly by this implementation. For that, one must first correctly convert the buffers to strings and keep track of characters that is split between two buffers. To do that properly, one can use the built in StringDecoder
    – Ernelli
    Commented Dec 5, 2013 at 9:07
  • In the meantime there's a built-in way to read lines from a file, using the readline core module. Commented Sep 16, 2015 at 3:04
19

A new function was added in Node.js v18.11.0 to read files line by line

  • filehandle.readLines([options])

This is how you use this with a text file you want to read

import { open } from 'node:fs/promises';
myFileReader();
async function myFileReader() {
    const file = await open('./TextFileName.txt');
    for await (const line of file.readLines()) {
        console.log(line)
    }
}

To understand more read Node.js documentation here is the link for file system readlines(): https://nodejs.org/api/fs.html#filehandlereadlinesoptions

1
  • 4
    Good find, this has saved me from the pitfalls of some of the other answers (e.g. CRLF line endings, reading the last line, properly closing read streams early). Thanks!! Commented Oct 17, 2023 at 2:38
18

Old topic, but this works:

var rl = readline.createInterface({
      input : fs.createReadStream('/path/file.txt'),
      output: process.stdout,
      terminal: false
})
rl.on('line',function(line){
     console.log(line) //or parse line
})

Simple. No need for an external module.

3
11

I ended up with a massive, massive memory leak using Lazy to read line by line when trying to then process those lines and write them to another stream due to the way drain/pause/resume in node works (see: http://elegantcode.com/2011/04/06/taking-baby-steps-with-node-js-pumping-data-between-streams/ (i love this guy btw)). I haven't looked closely enough at Lazy to understand exactly why, but I couldn't pause my read stream to allow for a drain without Lazy exiting.

I wrote the code to process massive csv files into xml docs, you can see the code here: https://github.com/j03m/node-csv2xml

If you run the previous revisions with Lazy line it leaks. The latest revision doesn't leak at all and you can probably use it as the basis for a reader/processor. Though I have some custom stuff in there.

Edit: I guess I should also note that my code with Lazy worked fine until I found myself writing large enough xml fragments that drain/pause/resume because a necessity. For smaller chunks it was fine.

2
  • In the meantime there's a much simpler way to read lines from a file, using the readline core module. Commented Sep 16, 2015 at 3:01
  • yup. That is the correct way now. But this was from 2011. :)
    – j03m
    Commented Sep 16, 2015 at 14:25
11

With the carrier module:

var carrier = require('carrier');

process.stdin.resume();
carrier.carry(process.stdin, function(line) {
    console.log('got one line: ' + line);
});
3
  • Nice. This is also works for any input file: var inStream = fs.createReadStream('input.txt', {flags:'r'}); But your syntax is cleaner than the documented method of using .on(): carrier.carry(inStream).on('line', function(line) { ... Commented Jan 1, 2012 at 22:34
  • carrier seems to only handle \r\n and \n line endings. If you ever need to deal with MacOS-style test files from before OS X, they used \r and carrier does not handle this. Surprisingly, there are still such files floating about in the wild. You might also need to handle the Unicode BOM (byte order mark) explicitly, this is used at the beginning of text files in the MS Windows sphere of influence. Commented Dec 31, 2012 at 3:31
  • In the meantime there's a built-in way to read lines from a file, using the readline core module. Commented Sep 16, 2015 at 3:04
9

In most cases this should be enough:

const fs = require("fs")

fs.readFile('./file', 'utf-8', (err, file) => {
  const lines = file.split('\n')

  for (let line of lines)
    console.log(line)
});
0
8

Edit:

Use a transform stream.


With a BufferedReader you can read lines.

new BufferedReader ("lorem ipsum", { encoding: "utf8" })
    .on ("error", function (error){
        console.log ("error: " + error);
    })
    .on ("line", function (line){
        console.log ("line: " + line);
    })
    .on ("end", function (){
        console.log ("EOF");
    })
    .read ();
1
6

I was frustrated by the lack of a comprehensive solution for this, so I put together my own attempt (git / npm). Copy-pasted list of features:

  • Interactive line processing (callback-based, no loading the entire file into RAM)
  • Optionally, return all lines in an array (detailed or raw mode)
  • Interactively interrupt streaming, or perform map/filter like processing
  • Detect any newline convention (PC/Mac/Linux)
  • Correct eof / last line treatment
  • Correct handling of multi-byte UTF-8 characters
  • Retrieve byte offset and byte length information on per-line basis
  • Random access, using line-based or byte-based offsets
  • Automatically map line-offset information, to speed up random access
  • Zero dependencies
  • Tests

NIH? You decide :-)

6

Since posting my original answer, I found that split is a very easy to use node module for line reading in a file; Which also accepts optional parameters.

var split = require('split');
fs.createReadStream(file)
    .pipe(split())
    .on('data', function (line) {
      //each chunk now is a seperate line! 
    });

Haven't tested on very large files. Let us know if you do.

5
function createLineReader(fileName){
    var EM = require("events").EventEmitter
    var ev = new EM()
    var stream = require("fs").createReadStream(fileName)
    var remainder = null;
    stream.on("data",function(data){
        if(remainder != null){//append newly received data chunk
            var tmp = new Buffer(remainder.length+data.length)
            remainder.copy(tmp)
            data.copy(tmp,remainder.length)
            data = tmp;
        }
        var start = 0;
        for(var i=0; i<data.length; i++){
            if(data[i] == 10){ //\n new line
                var line = data.slice(start,i)
                ev.emit("line", line)
                start = i+1;
            }
        }
        if(start<data.length){
            remainder = data.slice(start);
        }else{
            remainder = null;
        }
    })

    stream.on("end",function(){
        if(null!=remainder) ev.emit("line",remainder)
    })

    return ev
}


//---------main---------------
fileName = process.argv[2]

lineReader = createLineReader(fileName)
lineReader.on("line",function(line){
    console.log(line.toString())
    //console.log("++++++++++++++++++++")
})
6
  • I will test this, but can you tell me, is it guaranteed never to break multibyte characters? (UTF-8 / UTF-16) Commented Jan 2, 2013 at 1:54
  • 2
    @hippietrail: The answer is no for UTF-8, even though it is working on a byte stream rather than a character stream. It breaks on newlines (0x0a). In UTF-8, all bytes of a multibyte character have their hi-order bit set. Thus, no multibyte character can include an embedded newline or other common ASCII character. UTF-16 and UTF-32 are another matter, however.
    – George
    Commented May 10, 2013 at 4:40
  • @George: I think we misunderstand each other. As CR and LF are both within the ASCII range and UTF-8 preserves the 128 ASCII characters unchanged, neither CR nor LF can ever be part of a multibyte UTF-8 character. What I was asking is whether the data in the call to stream.on("data") might ever begin or end with only part of a multibyte UTF-8 character such as which is U+10D0, made up of the three bytes e1 83 90 Commented May 10, 2013 at 6:19
  • 1
    This still loads the whole file contents into memory before making it a "new line". This does not READ one line at a time, it instead takes ALL the lines and then breaks them up according to the "new line" buffer length. This method defeats the purpose of creating a stream.
    – Justin
    Commented Jul 29, 2015 at 17:13
  • In the meantime there's a much simpler way to read lines from a file, using the readline core module. Commented Sep 16, 2015 at 3:01
5

I wanted to tackle this same problem, basically what in Perl would be:

while (<>) {
    process_line($_);
}

My use case was just a standalone script, not a server, so synchronous was fine. These were my criteria:

  • The minimal synchronous code that could reuse in many projects.
  • No limits on file size or number of lines.
  • No limits on length of lines.
  • Able to handle full Unicode in UTF-8, including characters beyond the BMP.
  • Able to handle *nix and Windows line endings (old-style Mac not needed for me).
  • Line endings character(s) to be included in lines.
  • Able to handle last line with or without end-of-line characters.
  • Not use any external libraries not included in the node.js distribution.

This is a project for me to get a feel for low-level scripting type code in node.js and decide how viable it is as a replacement for other scripting languages like Perl.

After a surprising amount of effort and a couple of false starts this is the code I came up with. It's pretty fast but less trivial than I would've expected: (fork it on GitHub)

var fs            = require('fs'),
    StringDecoder = require('string_decoder').StringDecoder,
    util          = require('util');

function lineByLine(fd) {
  var blob = '';
  var blobStart = 0;
  var blobEnd = 0;

  var decoder = new StringDecoder('utf8');

  var CHUNK_SIZE = 16384;
  var chunk = new Buffer(CHUNK_SIZE);

  var eolPos = -1;
  var lastChunk = false;

  var moreLines = true;
  var readMore = true;

  // each line
  while (moreLines) {

    readMore = true;
    // append more chunks from the file onto the end of our blob of text until we have an EOL or EOF
    while (readMore) {

      // do we have a whole line? (with LF)
      eolPos = blob.indexOf('\n', blobStart);

      if (eolPos !== -1) {
        blobEnd = eolPos;
        readMore = false;

      // do we have the last line? (no LF)
      } else if (lastChunk) {
        blobEnd = blob.length;
        readMore = false;

      // otherwise read more
      } else {
        var bytesRead = fs.readSync(fd, chunk, 0, CHUNK_SIZE, null);

        lastChunk = bytesRead !== CHUNK_SIZE;

        blob += decoder.write(chunk.slice(0, bytesRead));
      }
    }

    if (blobStart < blob.length) {
      processLine(blob.substring(blobStart, blobEnd + 1));

      blobStart = blobEnd + 1;

      if (blobStart >= CHUNK_SIZE) {
        // blobStart is in characters, CHUNK_SIZE is in octets
        var freeable = blobStart / CHUNK_SIZE;

        // keep blob from growing indefinitely, not as deterministic as I'd like
        blob = blob.substring(CHUNK_SIZE);
        blobStart -= CHUNK_SIZE;
        blobEnd -= CHUNK_SIZE;
      }
    } else {
      moreLines = false;
    }
  }
}

It could probably be cleaned up further, it was the result of trial and error.

0
4

This is my favorite way of going through a file, a simple native solution for a progressive (as in not a "slurp" or all-in-memory way) file read with modern async/await. It's a solution that I find "natural" when processing large text files without having to resort to the readline package or any non-core dependency.

let buf = '';
for await ( const chunk of fs.createReadStream('myfile') ) {
    const lines = buf.concat(chunk).split(/\r?\n/);
    buf = lines.pop() ?? '';
    for( const line of lines ) {
        console.log(line);
    }
}
if(buf.length) console.log(buf);  // last line, if file does not end with newline

You can adjust encoding in the fs.createReadStream or use chunk.toString(<arg>). Also this let's you better fine-tune the line splitting to your taste, ie. use .split(/\n+/) to skip empty lines and control the chunk size with fs.createReadStream('myfile', { highWaterMark: <chunkSize> }).

Don't forget to create a function like processLine(line) to avoid repeating the line processing code twice due to the ending buf leftover. Unfortunately, the ReadStream instance does not update its end-of-file flags in this setup, so there's no way, afaik, to detect within the loop that we're in the last iteration without some more verbose tricks like comparing the file size from a fs.Stats() with .bytesRead. Hence the final buf processing solution, unless you're absolutely sure your file ends with a newline \n, in which case the for await loop should suffice.

Performance Considerations

Chunk sizes are important for performance, the default is 64k for text files and, for multi MB files, larger chunks can improve speed by an order of magnitude.

The above snippet runs at least the same speed (or even 5% faster sometimes) as code based on NodeJS v18's fs.readLine() or based on the readline module (the accepted answer), once you tune highWaterMark to something that your machine can handle, ie. setting it to the same size as the file, if your available memory allows it, is the fastest.

In any case, any of NodeJS line-reading answers here are an order of magnitude slower than the Perl or native *Nix solutions.

Similar alternatives

★ If you prefer the evented asynchronous version, this would be it:

let buf = '';
fs.createReadStream('myfile')
.on('data', chunk => {
    const lines = buf.concat(chunk).split(/\r?\n/);
    buf = lines.pop();
    for( const line of lines ) {
        console.log(line);
    }
})
.on('end', () => buf.length && console.log(buf) );

★ Now if you don't mind importing the stream core package, then this is the equivalent piped stream version, which allows for chaining transforms like gzip decompression:

const { Writable } = require('stream');
let buf = '';
fs.createReadStream('myfile').pipe(
    new Writable({
        write: (chunk, enc, next) => {
            const lines = buf.concat(chunk).split(/\r?\n/);
            buf = lines.pop();
            for (const line of lines) {
                console.log(line);
            }
            next();
        }
    })
).on('finish', () => buf.length && console.log(buf) );
2
  • I would add buf = lines.pop() ?? '' - as Array.pop() may return undefined, forcing me to trace back and find if lines could be empty, it also makes Typescript happy. Commented Feb 17, 2023 at 21:32
  • I don't think pop ever will return undefined actually. ''.split(/\r?\n/) is [''], so even if the chunk is empty, the array will contain 1 element. I'd just slap a ! on it to make TS happy.
    – mpen
    Commented Mar 7 at 8:13
3

Generator based line reader: https://github.com/neurosnap/gen-readlines

var fs = require('fs');
var readlines = require('gen-readlines');

fs.open('./file.txt', 'r', function(err, fd) {
  if (err) throw err;
  fs.fstat(fd, function(err, stats) {
    if (err) throw err;

    for (var line of readlines(fd, stats.size)) {
      console.log(line.toString());
    }

  });
});
2

If you want to read a file line by line and writing this in another:

var fs = require('fs');
var readline = require('readline');
var Stream = require('stream');

function readFileLineByLine(inputFile, outputFile) {

   var instream = fs.createReadStream(inputFile);
   var outstream = new Stream();
   outstream.readable = true;
   outstream.writable = true;

   var rl = readline.createInterface({
      input: instream,
      output: outstream,
      terminal: false
   });

   rl.on('line', function (line) {
        fs.appendFileSync(outputFile, line + '\n');
   });
};
1
  • What's the difference between yours and kofrasa's answer?
    – Buffalo
    Commented Apr 18, 2017 at 6:08
2
var fs = require('fs');

function readfile(name,online,onend,encoding) {
    var bufsize = 1024;
    var buffer = new Buffer(bufsize);
    var bufread = 0;
    var fd = fs.openSync(name,'r');
    var position = 0;
    var eof = false;
    var data = "";
    var lines = 0;

    encoding = encoding || "utf8";

    function readbuf() {
        bufread = fs.readSync(fd,buffer,0,bufsize,position);
        position += bufread;
        eof = bufread ? false : true;
        data += buffer.toString(encoding,0,bufread);
    }

    function getLine() {
        var nl = data.indexOf("\r"), hasnl = nl !== -1;
        if (!hasnl && eof) return fs.closeSync(fd), online(data,++lines), onend(lines); 
        if (!hasnl && !eof) readbuf(), nl = data.indexOf("\r"), hasnl = nl !== -1;
        if (!hasnl) return process.nextTick(getLine);
        var line = data.substr(0,nl);
        data = data.substr(nl+1);
        if (data[0] === "\n") data = data.substr(1);
        online(line,++lines);
        process.nextTick(getLine);
    }
    getLine();
}

I had the same problem and came up with above solution looks simular to others but is aSync and can read large files very quickly

Hopes this helps

2

Two questions we must ask ourselves while doing such operations are:

  1. What's the amount of memory used to perform it?
  2. Is the memory consumption increasing drastically with the file size?

Solutions like require('fs').readFileSync() loads the whole file into memory. That means that the amount of memory required to perform operations will be almost equivalent to the file size. We should avoid these for anything larger than 50mbs

We can easily track the amount of memory used by a function by placing these lines of code after the function invocation :

    const used = process.memoryUsage().heapUsed / 1024 / 1024;
    console.log(
      `The script uses approximately ${Math.round(used * 100) / 100} MB`
    );

Right now the best way to read particular lines from a large file is using node's readline. The documentation has amazing examples.

2
  • The npm module and the github repo are gone. Commented Nov 14, 2021 at 2:33
  • @JamesMoore The links have since been updated and replaced with references to the readline lib.
    – oligofren
    Commented Feb 18, 2022 at 10:34
1

I have a little module which does this well and is used by quite a few other projects npm readline Note thay in node v10 there is a native readline module so I republished my module as linebyline https://www.npmjs.com/package/linebyline

if you dont want to use the module the function is very simple:

var fs = require('fs'),
EventEmitter = require('events').EventEmitter,
util = require('util'),
newlines = [
  13, // \r
  10  // \n
];
var readLine = module.exports = function(file, opts) {
if (!(this instanceof readLine)) return new readLine(file);

EventEmitter.call(this);
opts = opts || {};
var self = this,
  line = [],
  lineCount = 0,
  emit = function(line, count) {
    self.emit('line', new Buffer(line).toString(), count);
  };
  this.input = fs.createReadStream(file);
  this.input.on('open', function(fd) {
    self.emit('open', fd);
  })
  .on('data', function(data) {
   for (var i = 0; i < data.length; i++) {
    if (0 <= newlines.indexOf(data[i])) { // Newline char was found.
      lineCount++;
      if (line.length) emit(line, lineCount);
      line = []; // Empty buffer.
     } else {
      line.push(data[i]); // Buffer new line data.
     }
   }
 }).on('error', function(err) {
   self.emit('error', err);
 }).on('end', function() {
  // Emit last line if anything left over since EOF won't trigger it.
  if (line.length){
     lineCount++;
     emit(line, lineCount);
  }
  self.emit('end');
 }).on('close', function() {
   self.emit('close');
 });
};
util.inherits(readLine, EventEmitter);
1

Another solution is to run logic via sequential executor nsynjs. It reads file line-by-line using node readline module, and it doesn't use promises or recursion, therefore not going to fail on large files. Here is how the code will looks like:

var nsynjs = require('nsynjs');
var textFile = require('./wrappers/nodeReadline').textFile; // this file is part of nsynjs

function process(textFile) {

    var fh = new textFile();
    fh.open('path/to/file');
    var s;
    while (typeof(s = fh.readLine(nsynjsCtx).data) != 'undefined')
        console.log(s);
    fh.close();
}

var ctx = nsynjs.run(process,{},textFile,function () {
    console.log('done');
});

Code above is based on this exampe: https://github.com/amaksr/nsynjs/blob/master/examples/node-readline/index.js

1

Here's a copy-pasteable, streaming solution:

import {createReadStream} from 'fs'

async function* readLines(filePath: string) {
    const stream = createReadStream(filePath, {
        encoding: 'utf-8',
    })

    let remainder = ''
    for await (const chunk of stream) {
        let lines = (remainder + chunk).split(/\r?\n/)
        remainder = lines.pop()!

        for(const line of lines) {
            yield line
        }
    }

    if(remainder) {
        yield remainder
    }
}

Usage:

for await(const line of readLines('file.txt')) {
    console.log(line);
}

I'm using it to iterate a 14.8 GiB file right now. Which sure takes awhile to finish, but it starts quickly and doesn't run out of memory.

0

i use this:

function emitLines(stream, re){
    re = re && /\n/;
    var buffer = '';

    stream.on('data', stream_data);
    stream.on('end', stream_end);

    function stream_data(data){
        buffer += data;
        flush();
    }//stream_data

    function stream_end(){
        if(buffer) stream.emmit('line', buffer);
    }//stream_end


    function flush(){
        var re = /\n/;
        var match;
        while(match = re.exec(buffer)){
            var index = match.index + match[0].length;
            stream.emit('line', buffer.substring(0, index));
            buffer = buffer.substring(index);
            re.lastIndex = 0;
        }
    }//flush

}//emitLines

use this function on a stream and listen to the line events that is will emit.

gr-

0

While you should probably use the readline module as the top answer suggests, readline appears to be oriented toward command line interfaces rather than line reading. It's also a little bit more opaque regarding buffering. (Anyone who needs a streaming line oriented reader probably will want to tweak buffer sizes). The readline module is ~1000 lines while this, with stats and tests, is 34.

const EventEmitter = require('events').EventEmitter;
class LineReader extends EventEmitter{
    constructor(f, delim='\n'){
        super();
        this.totalChars = 0;
        this.totalLines = 0;
        this.leftover = '';

        f.on('data', (chunk)=>{
            this.totalChars += chunk.length;
            let lines = chunk.split(delim);
            if (lines.length === 1){
                this.leftover += chunk;
                return;
            }
            lines[0] = this.leftover + lines[0];
            this.leftover = lines[lines.length-1];
            if (this.leftover) lines.pop();
            this.totalLines += lines.length;
            for (let l of lines) this.onLine(l);
        });
        // f.on('error', ()=>{});
        f.on('end', ()=>{console.log('chars', this.totalChars, 'lines', this.totalLines)});
    }
    onLine(l){
        this.emit('line', l);
    }
}
//Command line test
const f = require('fs').createReadStream(process.argv[2], 'utf8');
const delim = process.argv[3];
const lineReader = new LineReader(f, delim);
lineReader.on('line', (line)=> console.log(line));

Here's an even shorter version, without the stats, at 19 lines:

class LineReader extends require('events').EventEmitter{
    constructor(f, delim='\n'){
        super();
        this.leftover = '';
        f.on('data', (chunk)=>{
            let lines = chunk.split(delim);
            if (lines.length === 1){
                this.leftover += chunk;
                return;
            }
            lines[0] = this.leftover + lines[0];
            this.leftover = lines[lines.length-1];
            if (this.leftover) 
                lines.pop();
            for (let l of lines)
                this.emit('line', l);
        });
    }
}
0
const fs = require("fs")

fs.readFile('./file', 'utf-8', (err, data) => {
var innerContent;
    console.log("Asynchronous read: " + data.toString());
    const lines = data.toString().split('\n')
    for (let line of lines)
        innerContent += line + '<br>';


});
0
0

I wrap the whole logic of daily line processing as a npm module: line-kit https://www.npmjs.com/package/line-kit

// example
var count = 0
require('line-kit')(require('fs').createReadStream('/etc/issue'),
                    (line) => { count++; },
                    () => {console.log(`seen ${count} lines`)})

-1

I use below code the read lines after verify that its not a directory and its not included in the list of files need not to be check.

(function () {
  var fs = require('fs');
  var glob = require('glob-fs')();
  var path = require('path');
  var result = 0;
  var exclude = ['LICENSE',
    path.join('e2e', 'util', 'db-ca', 'someother-file'),
    path.join('src', 'favicon.ico')];
  var files = [];
  files = glob.readdirSync('**');

  var allFiles = [];

  var patternString = [
    'trade',
    'order',
    'market',
    'securities'
  ];

  files.map((file) => {
    try {
      if (!fs.lstatSync(file).isDirectory() && exclude.indexOf(file) === -1) {
        fs.readFileSync(file).toString().split(/\r?\n/).forEach(function(line){
          patternString.map((pattern) => {
            if (line.indexOf(pattern) !== -1) {
              console.log(file + ' contain `' + pattern + '` in in line "' + line +'";');
              result = 1;
            }
          });
        });
      }
    } catch (e) {
      console.log('Error:', e.stack);
    }
  });
  process.exit(result);

})();

Not the answer you're looking for? Browse other questions tagged or ask your own question.