49

I have large text files, which range between 30MB and 10GB. How can I count the number of lines in a file using Node.js?

I have these limitations:

  • The entire file does not need to be written to memory
  • A child process is not required to perform the task
8
  • 10
    wc -l file ...
    – zerkms
    Commented Sep 17, 2012 at 4:18
  • "using NodeJS" -- any real technical reason behind this requirement?
    – zerkms
    Commented Sep 17, 2012 at 4:20
  • 4
    I'm sure that wc will be faster that any "native" nodejs solution
    – zerkms
    Commented Sep 17, 2012 at 4:21
  • You could just count the lines-- stackoverflow.com/questions/6156501/…
    – JoshRagem
    Commented Sep 17, 2012 at 4:41
  • @zerkms Which shell scripting language are you using? Commented Oct 3, 2012 at 22:03

11 Answers 11

47

solution without using wc:

var i;
var count = 0;
require('fs').createReadStream(process.argv[2])
  .on('data', function(chunk) {
    for (i=0; i < chunk.length; ++i)
      if (chunk[i] == 10) count++;
  })
  .on('end', function() {
    console.log(count);
  });

it's slower, but not that much you might expect - 0.6s for 140M+ file including node.js loading & startup time

>time node countlines.js video.mp4 
619643

real    0m0.614s
user    0m0.489s
sys 0m0.132s

>time wc -l video.mp4 
619643 video.mp4
real    0m0.133s
user    0m0.108s
sys 0m0.024s

>wc -c video.mp4
144681406  video.mp4
8
  • 3
    Your benchmark isn't very convincing since you're running it on a file that is not structured into lines and as such is not representative of the sort of file the OP wants to process. The line if (chunk[i] == 10) count++; will be executed far more often during the analysis of a text file than during the analysis of a binary video file.
    – ebohlman
    Commented Sep 18, 2012 at 7:14
  • 1
    I don't have 100mb text file :) And I don't expect any difference even in the case of similar 100mb text file but with 10x number of newlines - it's same linear search iterating every byte in each of Buffer chunks Commented Sep 18, 2012 at 7:43
  • 2
    Excuse my innocence but what does "chunk[i] == 10" mean ? I guess that if the chunk is equals to 10 it's a new line, but why compare to the number 10 ?
    – Ashbay
    Commented Sep 28, 2017 at 19:18
  • 9
    10 is ascii code for "New Line" character. For better readability you could have few lines earlier const LINE_FEED = '\n'.charCodeAt(0) and then if (chunk[i] == LINE_FEED) count++ Commented Oct 2, 2017 at 0:26
  • 3
    Your implementation is off by one. For example, if your file has 2 lines, then it only has 1 newline, so your script will log 1.
    – Benjamin
    Commented Mar 7, 2019 at 23:42
34

We can use indexOf to let the VM find the newlines:

function countFileLines(filePath){
  return new Promise((resolve, reject) => {
  let lineCount = 0;
  fs.createReadStream(filePath)
    .on("data", (buffer) => {
      let idx = -1;
      lineCount--; // Because the loop will run once for idx=-1
      do {
        idx = buffer.indexOf(10, idx+1);
        lineCount++;
      } while (idx !== -1);
    }).on("end", () => {
      resolve(lineCount);
    }).on("error", reject);
  });
};

What this solution does is that it finds the position of the first newline using .indexOf. It increments lineCount, then it finds the next position. The second parameter to .indexOf tells where to start looking for newlines. This way we are jumping over large chunks of the buffer. The while loop will run once for every newline, plus one.

We are letting the Node runtime do the searching for us which is implemented on a lower level and should be faster.

On my system this is about twice as fast as running a for loop over the buffer length on a large file (111 MB).

5
  • 3
    This is the best solution compared to others showed here! Commented Oct 3, 2017 at 14:36
  • This answer is just amazing. Should be on the top. Took only 200 milliseconds to count lines on a 200MB file.
    – m4heshd
    Commented Apr 7, 2018 at 1:10
  • I can gladly confirm that this approach performs very well also with very large files (16GB). Commented Jun 3, 2020 at 8:30
  • If someone could explain how this works starting at line let idx = -1 that would be awesome. Thank you! Commented Nov 21, 2020 at 11:42
  • @philosopher It does idx+1 when being passed into .indexOf(), so the first value that one will see is 0, not -1. Commented Nov 3, 2022 at 23:01
31

You could do this as the comments suggest using wc

var exec = require('child_process').exec;

exec('wc -l /path/to/file', function (error, results) {
    console.log(results);
});
6
  • 16
    wc is a bash specific command and might not work in a windows environment for example
    – Renaud
    Commented Nov 21, 2014 at 13:43
  • 2
    wc -l to only count the number of lines
    – Yves M.
    Commented Mar 30, 2015 at 8:33
  • 1
    wc -l path/to/file will give number of lines along with filename. To get only number of lines use wc -l < path/to/file
    – Sarita
    Commented Oct 21, 2015 at 7:36
  • 3
    If you like to do it this way, try sed -n '$=' /path/to/file It will only return the number of lines, you can apply the function parseInt on it to get your number. Commented Dec 23, 2015 at 16:31
  • 2
    parseInt(execSync('wc -l < /path/to/file').toString().trim()) Commented Oct 2, 2018 at 2:32
7
var fs=require('fs');
filename=process.argv[2];
var data=fs.readFileSync(filename);
var res=data.toString().split('\n').length;
console.log(res-1);`
5
  • 6
    While this code snippet may solve the question, including an explanation really helps to improve the quality of your post. Remember that you are answering the question for readers in the future, and those people might not know the reasons for your code suggestion. Please also try not to crowd your code with explanatory comments, this reduces the readability of both the code and the explanations! Commented Jun 8, 2016 at 3:48
  • 3
    This solution requires loading the file in memory. I would advise against it. The answer using wc doesn't because wc is optimized to stream the file.
    – thalisk
    Commented Jun 27, 2017 at 14:01
  • 2
    The answer also doesn't add anything valuable compared to Alan Viars who posted the same thing a year before. Commented Dec 5, 2017 at 11:03
  • 2
    The question specifically states that the files range from 30MB to 10GB. This solution reads the entire file into memory before processing. This would likely cause the code to crash because JavaScript would run out of memory
    – thekenobe
    Commented May 13, 2019 at 18:23
  • your code would not be able to handle larger file size. Commented Jan 17, 2023 at 6:13
5

since iojs 1.5.0 there is Buffer#indexOf() method, using it to compare to Andrey Sidorov' answer:

ubuntu@server:~$ wc logs
  7342500  27548750 427155000 logs
ubuntu@server:~$ time wc -l logs 
7342500 logs

real    0m0.180s
user    0m0.088s
sys 0m0.084s
ubuntu@server:~$ nvm use node
Now using node v0.12.1
ubuntu@server:~$ time node countlines.js logs 
7342500

real    0m2.559s
user    0m2.200s
sys 0m0.340s
ubuntu@server:~$ nvm use iojs
Now using node iojs-v1.6.2
ubuntu@server:~$ time iojs countlines2.js logs 
7342500

real    0m1.363s
user    0m0.920s
sys 0m0.424s
ubuntu@server:~$ cat countlines.js 
var i;
var count = 0;
require('fs').createReadStream(process.argv[2])
  .on('data', function(chunk) {
    for (i=0; i < chunk.length; ++i)
      if (chunk[i] == 10) count++;
  })
  .on('end', function() {
    console.log(count);
  });
ubuntu@server:~$ cat countlines2.js 
var i;
var count = 0;
require('fs').createReadStream(process.argv[2])
  .on('data', function(chunk) {
    var index = -1;
    while((index = chunk.indexOf(10, index + 1)) > -1) count++
  })
  .on('end', function() {
    console.log(count);
  });
ubuntu@server:~$ 
4

If you use Node 8 and above, you can use this async/await pattern

const util = require('util');
const exec = util.promisify(require('child_process').exec);

async function fileLineCount({ fileLocation }) {
  const { stdout } = await exec(`cat ${fileLocation} | wc -l`);
  return parseInt(stdout);
};

// Usage

async someFunction() {
  const lineCount = await fileLineCount({ fileLocation: 'some/file.json' });
}
0
3

Here is another way without so much nesting.

var fs = require('fs');
filePath = process.argv[2];
fileBuffer =  fs.readFileSync(filePath);
to_string = fileBuffer.toString();
split_lines = to_string.split("\n");
console.log(split_lines.length-1);
2
  • 8
    For a 10gb file, this is not very performant, to say the least. Commented Feb 5, 2018 at 2:35
  • This is a simple and good, but only for small files! If files is a 10 GB file, Script will die. Commented Jun 29, 2018 at 6:52
3

Best solution I've found is using promises, async, and await. This is also an example of how await for the fulfillment of a promise:

#!/usr/bin/env node
const fs = require('fs');
const readline = require('readline');
function main() {
    function doRead() {
        return new Promise(resolve => {
            var inf = readline.createInterface({
                input: fs.createReadStream('async.js'),
                crlfDelay: Infinity
            });
            var count = 0;
            inf.on('line', (line) => {
                console.log(count + ' ' + line);
                count += 1;
            });
            inf.on('close', () => resolve(count));
        });
    }
    async function showRead() {
        var x = await doRead();
        console.log('line count: ' + x);
    }
    showRead();
}
main();
2
  • 1
    It's incorrect to say that you can turn an async function into a synchronous function. Your top-level main function needs to be async so that it can call await on showRead(). The only reason you get an apparent confirmation of your statement is because the NodeJs event loop is waiting for the IO phase to complete, and the program won't terminate until then. If you add a logging statement right below showRead() it would execute immediately
    – Felipe
    Commented Mar 20, 2019 at 19:15
  • 1
    Correct. This was more an example of how to use await to wait for the fulfillment of a promise. Poor choice of words on my part. I will fix that. Commented Mar 22, 2019 at 14:48
1

You can also use indexOf():

var index = -1;
var count = 0;
while ((index = chunk.indexOf(10, index + 1)) > -1) count++;
1

Simple solution using readline:

import readline from 'node:readline';

export default async function countLines(input) {
    let lineCount = 0;

    for await (const _ of readline.createInterface({input, crlfDelay: Infinity})) {
        lineCount++;
    }

    return lineCount;
}
import fs from 'node:fs';

console.log(await countLines(fs.createReadStream('file.txt')));
//=> <number>
0

You can try this solution for getting number of lines containing in a file.

const fs = require('fs'); 
const path = require('path');
        
        
        const filePath = path.join(__dirname, 'data.js');;
        
        function countLines(filePath) {
        
            const fileContents = fs.readFileSync(filePath, 'utf8');
        
            const lines = fileContents.split('\n');
        
            return lines.length; }
        
        const lineCount = countLines(filePath);
        
        
        
    console.log("filePath", filePath); 
    console.log("File contains no of line: ", lineCount);

output:

File contains no of line: 11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.