11

I'm developing a FileShare application with webRTC. I want to implement the client in JavaScript/HTML. The code should be run on the clients browser. I need to save them when downloaded via webRTC. The files can be quite big and I can't completely donwload them and save them in a array or blob before saving them to disk as a file.

Is there any API that allows me to save the file in chunks as I recieve them?

I have found so far Downloadify, FileSave.js and html5 FileWriterApi so far. While the first two are not chunked and require me to first download the complete file to memory before saving, the FileWriterAPI is not available on most browsers.

3
  • For reasonably-sized files, my first instinct is to store each chunk in its own Blob, join them all together into one big Blob at the end (using the new constructor) and use createObjectURL to save the final Blob. IIRC, JS Blobs are stored to disk where necessary, so unless the files are really huge (or I'm really wrong ;) ), it shouldn't be too horrific? Commented Oct 17, 2013 at 15:34
  • @JordanGray Might be a good idea - I assumed the downloaded chunks need to be persisted for continuous download between browser sessions, but if not, this might be the cleanest idea (if the browsers do implement persisting the blob to file when needed). Commented Oct 20, 2013 at 1:03

3 Answers 3

11
+100

As @jordan-gray suggested, saving the chunks in blobs and joining them to a larger blob might be a solution if:

  • Persistence of chunks is not needed (i.e. closing the browser will delete all chunks)
  • The file is persisted only by the user saving it to his own filesystem. The web application will not have access to the file once it is closed, unless the user gave access to the saved file again.
  • Possibly, if the file sizes are not too big (you'll have to benchmark to find that out). Chrome was behaving quite nice for me for chunks totaling at 1GB.

I've created a simple test for using blobs as chunks. You can play around with the different size and chunk numbers parameters:

var chunkSize = 500000;
var totalChunks = 200;
var currentChunk = 0;
var mime = 'application/octet-binary';
var waitBetweenChunks = 50;

var finalBlob = null;
var chunkBlobs =[];

function addChunk() {
    var typedArray = new Int8Array(chunkSize);
    chunkBlobs[currentChunk] = new Blob([typedArray], {type: mime});
    console.log('added chunk', currentChunk);
    currentChunk++;
    if (currentChunk == totalChunks) {
        console.log('all chunks completed');
        finalBlob = new Blob(chunkBlobs, {type: mime});
        document.getElementById('completedFileLink').href = URL.createObjectURL(finalBlob);
    } else {
        window.setTimeout(addChunk, waitBetweenChunks);
    }
}
addChunk();

If you do need that persistence, the W3C File System API should support what you need. You can use it to write the chunks to separate files, and then when all the chunks are completed you can read them all and append them to a single file, and remove the chunks.

Note that it works by assigning a sandboxed filesystem for your application (for a given quota), and the files are only accessible to that application. If the files are meant to use outside of the web application, you might need the function for the use to save the file from the application filesystem to his "normal" filesystem. You can do something like that using the createObjectURL() method.

You are right about current state of browser support. A Filesystem API polyfill is available, which is based on IndexedDB (which is more widely supported) as a filesystem emulation backend. I did not test the polyfill on large files. You might run into size limits or performance limitations.

4
  • Yup, this is pretty much what I had in mind. Based on my understanding of the spec it should be able to scale reasonably well, with chunks being persisted to underlying storage where necessary. :) Commented Oct 21, 2013 at 8:24
  • It looks like this is still being saved to memory first (currentChunk == totalChunks) before saving to disk, correct?
    – dman
    Commented Nov 30, 2015 at 15:50
  • No, think of new Blob() as if it were returning a pointer to an object on disk. developer.mozilla.org/en/docs/Web/API/Blob
    – dave.zap
    Commented Nov 28, 2016 at 9:04
  • @AmitayDobo How could it show the download process from the first chunk, I would like it to show the Browsers download behavior. Could it be done?
    – hackp0int
    Commented Aug 14, 2018 at 14:22
3

Did you check https://github.com/Peer5/Sharefest out ? It should cover your requirements

1
-1
while (currentChunk < totalChunks) {
                var offset = currentChunk * chunkSize;
        var currentFilePart = rr.files.item(0).slice(offset, (offset+chunkSize));
        var cFilePart = new FileReader();
        var ar1 = new Uint8Array(currentFilePart.arrayBuffer());
        for (var i = 0; i < chunkSize; i++) 
        {
                st = ar1[i] ^ bb;
                st = ar1[i];
                Rajar2[Rajtop] = iii;
                Rajtop++;
        }
                currentChunk++;
        iii = iii  + currentChunk;
        }
    {
        {
            var blob1 = new Blob([Rajar2], {type: "application/octet-stream"});
            var fileName = 'a.txt';
            var url = window.URL.createObjectURL(blob1);
            {
                var element = document.createElement('a');
                element.style.display = 'none';
                    element.href = url;
                    element.download = fileName;
                document.body.appendChild(element);
                element.click();
            }
        }
    }
1
  • 1
    Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Commented Nov 12, 2023 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.