5

I have a webpage with file upload functionality. The upload is performed in 5MB chunks. I want to calculate hash for each chunk before sending it to the server. The chunks are represented by Blob objects. In order to calculate the hash I am reading such blob into an ArrayBuffer using a native FileReader. Here is the code:

var reader = new FileReader();

var getHash = function (blob, callback) {
    reader.onloadend = function (e) {
        var hash = util.hash(e.target.result);
        callback(hash);
    };
    reader.readAsArrayBuffer(blob);
}

var processChunk = function (chunk) {
    if (chunk) {
        getHash(chunk, function (hash) {
            util.sendToServer(chunk, hash, function() {
                // this callback is called when chunk upload is finished
                processChunk(chunks.shift());
            });
        });
    }
}

var chunks = file.splitIntoChunks(); // gets an array of blobs
processChunk(chunks.shift());

The problem: using the FileReader.readAsArrayBuffer seems to eat up a lot of memory which is not released. So far I tested with a 5GB file on following browsers:

  • Chrome 55.0.2883.87 m (64-bit): the memory goes up to 1-2GB quickly and oscillates around that. Sometimes it goes all the way up and browser tab crashes. It can use more memory than the size of read chunks. E.g. after reading 500MB of chunks the process already uses 700MB of memory.

  • Firefox 50.1.0: memory usage oscillates around 300-600MB

Code adjustments I have tried - all to no avail:

  • re-using the same FileReader instance for all chunks (as suggested in this question)
  • creating new FileReader for each chunk
  • adding timeout before starting new chunk
  • setting the FileReader and the ArrayBuffer to null after each read

The question: is there a way to fix the problem? Is this a bug in the FileReader implementations or am I doing something wrong?

EDIT: Here is a JSFiddle https://jsfiddle.net/andy250/pjt9udeu/

1
  • I tried with Chrome 55.0.2883.87 m (64-bit) as well - getting about the same consumption of memory when running your JsFiddle. See video clip: screencast.com/t/e7eQkVMWI68a. Also tested Internet Explorer 11 - it only uses about 130 MB RAM. Commented Dec 16, 2016 at 12:37

1 Answer 1

3

This is a bug in Chrome on Windows. It is reported here: https://bugs.chromium.org/p/chromium/issues/detail?id=674903

1
  • 1
    Is there any solution 3 years after the original post? Commented Jan 21, 2020 at 5:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.