3

I am trying to create a Zip from a list of files in parallel and stream it to client.

I have a working code where I iterate over files sequentially, but I want it instead to be zipped in parallel (multiple files with >100mb each).

using ZipArchive zipArchive = new(Response.BodyWriter.AsStream(), ZipArchiveMode.Create, leaveOpen: false);

for (int i = 0; i < arrLocalFilesPath.Length; i++) // iterate over files
{
    string strFilePath = arrLocalFilesPath[i]; // list of files path
    string strFileName = Path.GetFileName(strFilePath);

    ZipArchiveEntry zipEntry = zipArchive.CreateEntry(strFileName, CompressionLevel.Optimal);
    using Stream zipStream = zipEntry.Open();

    using FileStream fileStream = System.IO.File.Open(strFilePath, FileMode.Open, FileAccess.Read);
    fileStream.CopyTo(zipStream);
}

return new EmptyResult();

Parallel.For and Parallel.ForEach do not work with ZipArchive

Since ZipArchive is not thread safe, I am trying to use DotNetZip to accomplish this task.

I looked at the docs and here's what I have so far using DotNetZip

using Stream streamResponseBody = Response.BodyWriter.AsStream();

Parallel.For(0, arrLocalFilesPath.Length, i =>
{
    string strFilePath = arrLocalFilesPath[i]; // list of files path
    string strFileName = Path.GetFileName(strFilePath);

    string strCompressedOutputFile = strFilePath + ".compressed";

    byte[] arrBuffer = new byte[8192]; //[4096];
    int n = -1;

    using FileStream input = System.IO.File.OpenRead(strFilePath);
    using FileStream raw = new(strCompressedOutputFile, FileMode.Create, FileAccess.ReadWrite);

    using Stream compressor = new ParallelDeflateOutputStream(raw);
    while ((n = input.Read(arrBuffer, 0, arrBuffer.Length)) != 0)
    {
        compressor.Write(arrBuffer, 0, n);
    }

    input.CopyTo(streamResponseBody);
});

return new EmptyResult();

However, this doesn't zip files and send to client (it only creates local zip files on the server).

Using MemoryStream or creating a local zip file is out of the question and not what I am looking for.

The server should seamlessly stream read bytes of a file, zip it on the fly and send it to client as chunks (like in my ZipArchive), but with the added benefits of reading those files in parallel and creating a zip of them.

I know that parallelism is usually not optimal for I/O (sometimes a bit worse), but parallel zipping multiple big files should be faster for this case.

I also tried to use SharpZipLib without success.

Usage of any other libraries is fine as long as it read and stream files to client seamlessly without impacting memory.

Any help is appreciated.

8
  • I am still pretty sure that using parallel for in this case is wrong. Why do you think you will benefit from it? have you tried using single thread? Also this code smells, I'll let you know later what should be changed.
    – szamil
    Commented Jan 26, 2023 at 22:46
  • I am searching for a way to read files and zip them in parallel and stream the zip data to the client. The DotNetZip code is broken and doesn't work as intended, but I cannot use ZipArchive (which works fine in single thread) in multiple parallel threads.
    – Codingwiz
    Commented Jan 26, 2023 at 22:56
  • I would suggest you lean on the OS ability to "read ahead". Try to open a few files in advance and read their first block, then pass them over to a single threaded zip method. Right now you are only telling the OS about the new file you want at the last moment, blocking your thread from making any progress until the OS can find that file on disk and read it. Commented Jan 27, 2023 at 0:54
  • @JeremyLakeman any code snippets on how to "read ahead" files and send them to a stream ?
    – Codingwiz
    Commented Jan 27, 2023 at 1:04
  • @JeremyLakeman but os won't overcome serial IO read. What you suggested, as I understand, is a kind of caching files by OS in memory before zipping them. I would not rely on this kind of optimisation.
    – szamil
    Commented Jan 27, 2023 at 1:05

1 Answer 1

1

If these files are on the same drive there won't be any speed up. The parallelization is used to compress/decompress data, but the disk IO operation cannot be done in parallel.

Assuming that files are not on the same drive and there is a chance to speed up this process...

Are you sure the Stream.CopyTo() is thread safe? Either check the docs or use single thread or set lock on it.

EDIT:

I've checked my old codes, where I was packing huge amount of data into a zip file using ZipArchive. I did it in parallel, but there was no IO read there.

You can use ZipArchive with Parallel.For but you need to use lock:

//create zip into stream
using (ZipArchive zipArchive = new ZipArchive(zipFS, ZipArchiveMode.Update, false))
{    
    //use parallel foreach instead of parallel, but not for IO read operation!
    Parallel.ForEach(listOfFiles, filename =>
    {
        //create a file entry
        ZipArchiveEntry zipFileEntry = zipArchive.CreateEntry(filename);

        //prepare memory for the entry
        MemoryStream ms = new MemoryStream();

        /*fill the memory stream here - I did another packing with BZip2OutputStream, because the zip was packed without compression to speed up random decompression */

        //only one thread can write to zip!
        lock (zipFileEntry)
        {
            //open stream for writing 
            using (Stream zipEntryStream = zipFileEntry.Open())
            {
                ms.Position = 0; // rewind the stream
                StreamUtils.Copy(ms, zipEntryStream, new byte[4096]); //from  ICSharpCode.SharpZipLib.Core, copy memory stream data into zip entry with packing.            
            }
        }
    }
}

Anyway, if you need to read the files first, it's your performance bottleneck. You won't gain a lot (if anything) from parallel approach here.

5
  • There's no mention of Stream.CopyTo() being thread safe in the docs, but PipeStream is thread safe so I may use that in the future.
    – Codingwiz
    Commented Jan 26, 2023 at 15:40
  • Changing input.CopyTo() to compressor.CopyTo() throws an Exception System.NotSupportedException: Stream does not support reading.
    – Codingwiz
    Commented Jan 26, 2023 at 15:41
  • Thanks for the edit, but what is the advantage of using lock on Parallel.ForEach ? Doesn't it mean that it is essentially the same as normal foreach ? (I know that ZipArchive will throw an Exception when using it with Parallel.ForEach: System.IO.IOException: Entries in create mode may only be written to once, and only one entry may be held open at a time.)
    – Codingwiz
    Commented Jan 27, 2023 at 0:21
  • 1
    The advantage of using lock is that this is the only way it will work. You cannot write to file/stream in parallel. That is simply impossible. What you can do is to perform parallel packing in memory, as I pointed in comment inside the code, then make files from that. Lock creates critical section to ensure only one thread writes to the stream.
    – szamil
    Commented Jan 27, 2023 at 0:26
  • Read the error carefully: only one entry open at a time - ergo you must lock before openning new entry in zip.
    – szamil
    Commented Jan 27, 2023 at 0:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.