5

I'm trying to figure out the fastest way without affecting performance much while moving either synchronously or asynchronously 100.000+ files using Node.js

I made sync and async tests with 3 different looping forEach async.each and for. The loop iterations didn't impact timing results much. And the heaviest logic affecting performance and speed is moving files either synchronously or asynchronously.

The test cases showed, that synchronous renameSync() is slower at 20% then asynchronous rename(). In the same time asynchronous rename() using 3 times more CPU power, then synchronous renameSync().


Am I right to assume that it would be wiser to use synchronous renameSync() because speed boost in asynchronous rename() is not so significant and equal to 20%. While CPU utilization overhead in async version seems to be huge - 300%?

Or did I miss anything? Maybe there are better workarounds in terms of speed/performance.

The timings:

eachAsyncRenameAsync()

node rename.js  1.60s user 11.20s system 280% cpu 4.559 total
node rename.js  1.65s user 11.82s system 284% cpu 4.732 total
node rename.js  1.64s user 11.84s system 292% cpu 4.606 total


eachSyncRenameSync()

node rename.js  0.69s user 5.01s system 97% cpu 5.851 total
node rename.js  0.67s user 4.88s system 97% cpu 5.687 total
node rename.js  0.68s user 5.01s system 99% cpu 5.734 total


eachAsyncRenameSync()

node rename.js  0.67s user 4.99s system 97% cpu 5.797 total
node rename.js  0.68s user 4.95s system 97% cpu 5.754 total
node rename.js  0.66s user 4.89s system 97% cpu 5.690 total


eachSyncRenameAsync()

node rename.js  1.63s user 11.12s system 274% cpu 4.638 total
node rename.js  1.66s user 12.29s system 286% cpu 4.874 total
node rename.js  1.66s user 12.23s system 289% cpu 4.795 total


forSyncRenameAsync()

node rename.js  1.72s user 12.04s system 283% cpu 4.862 total
node rename.js  1.69s user 11.88s system 276% cpu 4.904 total
node rename.js  1.64s user 11.89s system 287% cpu 4.712 total


forSyncRenameSync()

node rename.js  0.64s user 4.94s system 97% cpu 5.715 total
node rename.js  0.66s user 5.01s system 97% cpu 5.807 total
node rename.js  0.65s user 4.93s system 99% cpu 5.616 total

The code:

const fs = require('fs')
const path = require('path')
const { each } = require('async')

let dir = '/opt/bin/test/Random 100000'
let dir2 = '/opt/bin/test/Random 100000 2'

function eachAsyncRenameAsync(files) {
        each(files, file => {
              fs.rename(path.join(dir,file), path.join(dir2,file), err => {
                if(err) { console.log(err) }
              })
        })
}

function eachSyncRenameSync(files) {
    files.forEach(file => {
            fs.renameSync(path.join(dir,file), path.join(dir2,file))
    })
}

function eachAsyncRenameSync(files) {
    each(files, file => {
            fs.renameSync(path.join(dir,file), path.join(dir2,file))
    })
}

function eachSyncRenameAsync(files) {
    files.forEach(file => {
        fs.rename(path.join(dir,file), path.join(dir2,file), err => {
                if(err) { console.log(err) }
        })
    })
}

function forSyncRenameAsync(files) {
    for (i=0; i<files.length; i++) {
        fs.rename(path.join(dir, files[i]), path.join(dir2, files[i]), err => {
                if(err) { console.log(err) }
        })
    }

}

function forSyncRenameSync(files) {
    for (i=0; i<files.length; i++) {
        fs.renameSync(path.join(dir, files[i]), path.join(dir2, files[i]))
    }
}

// Reading dir asynchronously and moving files

fs.readdir(dir, (err, files) => {
    if (err) { console.log(err) }
    else {
        console.log('eachAsyncRenameAsync()')
        eachAsyncRenameAsync(files)
    }
})

1 Answer 1

-2

for asynchronous performans comparison-optimization,making a pool to limit the number of concurrent operations will let you find the optimal result.

the benchmark you made shows that the optimal result is between 1 and n async concurent operations.

6
  • I don't think this is the place to explain mathematical theories... My answer needs more explanation for those who is not familiar with math... I don't think that kind of explanation is what i should make here... Still , using pool(s) to deal with over using async callbacks , and making the best optimizations for the jobs , will result the best . Here , n is the number of async concurent operations. My answer is clear enough for especially nodejs programmers , who already deal with callbacks.
    – murat aka
    Commented Jun 10, 2020 at 9:56
  • ---- How does this answer OP's question?--- It does not solve it. it is the explanationof the result of the bench . What would you expect ? generate a pool library as an answer doing the whole job optimally ? ----
    – murat aka
    Commented Jun 10, 2020 at 9:58
  • other commenter gone , just leaving those comments for someone not to repeat that behaviour. thank you.
    – murat aka
    Commented Jun 11, 2020 at 0:45
  • "it is the explanationof the result of the bench" - but it doesn't explain anything either. Commented Jun 11, 2020 at 1:12
  • @user2357112supportsMonica , why don't you let the op to see the solution, although this is an old question, the problem is a general case , which many of us nodejs developers face. This , what you are trying to do is just so offensive. Using pooling method , is already suggested in official documentary of NodeJS. The implementation is optimal , for just the target platform , and using pooling is not any difficult . The answer which i wrote , is clearly helpful for anyone, was a solution for me , and will be a solution for many if you let it.
    – murat aka
    Commented Jun 11, 2020 at 2:09

Not the answer you're looking for? Browse other questions tagged or ask your own question.