4

How can I read any large file(greater than 1 gigabytes) locally by chunks(2kb or more),and then convert the chunk to a string, process the string and then get the next chunk and so on until the end of the file?

I'm only able to read small files and convert it to string, as you can see from the code I don't know how to read the file by chunks. The browser freezes if I try it with a file greater than 10mb.

<html>
  <head>
    <title>Read File</title>
  </head>

  <body>
    <input type="file" id="myFile">
    <hr>
    <textarea style="width:500px;height: 400px" id="output"></textarea>

    <script>
      var input = document.getElementById("myFile");
      var output = document.getElementById("output");
      input.addEventListener("change", function () {
        if (this.files && this.files[0]) {
          var myFile = this.files[0];
          var reader = new FileReader();
          reader.addEventListener('load', function (e) {
            output.textContent = e.target.result;
          });
          reader.readAsBinaryString(myFile);
        }
      });
    </script>

  </body>
</html>

Below are the links and answers I found on StackOverflow whilst researching on how to accomplish it, but it didn't solve my question.

1: This question was asking about how to do it using UniversalXPConnect, and only in Firefox, which is why i found the answer there to be irrelevant, because I use Chrome and don't know what UniversalXPConnect is. How to read a local file by chunks in JavaScript

2: This question was asking about how to read text files only, but I want to be able to read any file not just text, and also by chunks, which makes the answers there irrelevant, but i liked how short the code of the answer was. Reading local text file into a JavaScript array [duplicate]

3: This also is about text files and doesn't show how to read files by chunks How to read a local text file.

I know a little bit of Java, which you can easily do it by;

char[] myBuffer = new char[512];
int bytesRead = 0;
BufferedReader in = new BufferedReader(new FileReader("foo.mp4"));
while ((bytesRead = in.read(myBuffer,0,512)) != -1){
...
}

but I'm new to javascript

5
  • Are you trying to perform this in the browser? Or are you using Node.js? May 9, 2018 at 15:18
  • Yes please, I'm trying to use a browser to achieve it. Thank you.
    – king amada
    May 9, 2018 at 16:28
  • FileReader is supposed to be asynchronous, and you aren't actually looking to do anything in between these chunked reads, so I think it might not be what you are looking for. Put it another way: why would you expect reading it in chunks to be any better than what you are already seeing? I would also, for testing purposes, see if the browser is freezing on the reading or on the outputting. Change the 'load' callback to just something like console.log("Done Reading!"); If there is no freezing up before you see that, it is the output to the browser.
    – Anthony
    May 9, 2018 at 17:32
  • Just tested. Definitely seeing the browser freeze, and it is definitely after the file is read. So you need a way to cycle through the result in chunks, not the file itself. Probably readAsArrayBuffer is a good place to start.
    – Anthony
    May 9, 2018 at 17:39
  • stackoverflow.com/a/48726871/1693593
    – user1693593
    May 21, 2018 at 0:40

2 Answers 2

3

I was able to solve it by slicing the file by specifying attributes of where to begin the slice and where to end which will be the chunk, I then enclosed it in a while loop so that for each loop chunk position will shift according to the desired chunk size until the end of the file.

But after running it, I end up getting the last value of the chunk in the text area, so to display all the binary string i concatenate the output on each iteration.

<html>
<head>
  <title>Read File</title>
</head>

<body>
  <input type="file" id="myFile">
  <hr>
  <textarea style="width:500px;height: 400px" id="output"></textarea>

  <script>
    var input = document.getElementById("myFile");
    var output = document.getElementById("output");
    var chunk_size = 2048;
    var offset = 0;
    input.addEventListener("change", function () {
      if (this.files && this.files[0]) {
        var myFile = this.files[0];
        var size = myFile.size; //getting the file size so that we can use it for loop statement
        var i=0;
        while( i<size){
        var blob = myFile.slice(offset, offset + chunk_size); //slice the file by specifying the index(chunk size)
        var reader = new FileReader();
        reader.addEventListener('load', function (e) {
          output.textContent += e.target.result; //concatenate the output on each iteration.
        });
        reader.readAsBinaryString(blob);
        offset += chunk_size; // Increment the index position(chunk) 
        i += chunk_size; // Keeping track of when to exit, by incrementing till we reach file size(end of file).
        }
      }
    });
  </script>

</body>
</html>
1

So the issue isn't with FileReader, it's with :

output.textContent = e.target.result;

Because you are trying to dump 10MB+ worth of string into that textarea all at once. I'm not even sure there is a "right" way to do what you are wanting, since even if you did have it in chunks, it would still have to concat the previous value of output.textContent on each loop through those chunks, so that as it gets closer to the end, it would start slowing down in the same way (worse, really, because it would be doing the slow memory hogging business on every loop). So I think part of the looping process is going to have to be adding a new element (like a new textarea to push the current chunk to (so it doesn't have to do any concatenation to preserve what has already been output). I haven't worked that part out yet, but here's what I've got so far:

  var input = document.getElementById("myFile");
  var output = document.getElementById("output");
  var chunk_length = 2048; //2KB as you mentioned
  var chunker = new RegExp('[^]{1,' + chunk_length + '}', 'g');
  var chunked_results;

  input.addEventListener("change", function () {
    if (this.files && this.files[0]) {
      var myFile = this.files[0];
      var reader = new FileReader();
      reader.addEventListener('load', function (e) {
        chunked_results = e.target.result.match(chunker);
        output.textContent = chunked_results[0];
      });
      reader.readAsBinaryString(myFile);
    }
  });

This is just outputting the first string in the array of 2KB chunks. You would want to do your thing as far as adding a new element/node in the DOM document for outputting all the other chunks.

Using RegExp and match for the actual chunking was lifted from a clever gist I found.

5
  • Thanks Anthony, but i just finish solving it on my own, though similar to your own. I don't understand "RegExp('.{1,' + chunk_length + '}', 'g');" though, I will be posting my solution for anyone with same problem. Though it seems slow for large files, maybe you can help with that?
    – king amada
    May 9, 2018 at 18:42
  • Great. Glad I could help (or not).
    – Anthony
    May 9, 2018 at 18:47
  • I just tested my solution with a 68MB file and it choked up the browser pretty bad. And that was with var content = document.createTextNode(chunk); output.appendChild(content); So presumably no concatenation (or at least none on the surface level). I'll try again with adding an actual new textarea for each chunk and compare, but if that hangs up the browser too, I would imagine any solutions are going to involve Workers of some kind.
    – Anthony
    May 9, 2018 at 18:51
  • Check my answer, though is working as i wanted, but I will have to work on the performance with large files, any improvement from your side is welcome.
    – king amada
    May 9, 2018 at 19:05
  • I think you are essentially doing the same thing I did, you just are slicing the file before passing it to FileReader, while I'm splitting the file into an array after it gets done with FileReader. Neither one has a ton of overhead, it is the output to the browser that is killing it. I've tried adding a new textarea for each chunk, and it still breaks everything with a 20MB file. So the question isn't "how do I break this up into smaller chunks?" it is "how do I output 10MB+ worth of data to the browser window?"
    – Anthony
    May 9, 2018 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.