I am trying to create a Zip from a list of files in parallel and stream it to client.
I have a working code where I iterate over files sequentially, but I want it instead to be zipped in parallel (multiple files with >100mb each).
using ZipArchive zipArchive = new(Response.BodyWriter.AsStream(), ZipArchiveMode.Create, leaveOpen: false);
for (int i = 0; i < arrLocalFilesPath.Length; i++) // iterate over files
{
string strFilePath = arrLocalFilesPath[i]; // list of files path
string strFileName = Path.GetFileName(strFilePath);
ZipArchiveEntry zipEntry = zipArchive.CreateEntry(strFileName, CompressionLevel.Optimal);
using Stream zipStream = zipEntry.Open();
using FileStream fileStream = System.IO.File.Open(strFilePath, FileMode.Open, FileAccess.Read);
fileStream.CopyTo(zipStream);
}
return new EmptyResult();
Parallel.For
and Parallel.ForEach
do not work with ZipArchive
Since ZipArchive
is not thread safe, I am trying to use DotNetZip
to accomplish this task.
I looked at the docs and here's what I have so far using DotNetZip
using Stream streamResponseBody = Response.BodyWriter.AsStream();
Parallel.For(0, arrLocalFilesPath.Length, i =>
{
string strFilePath = arrLocalFilesPath[i]; // list of files path
string strFileName = Path.GetFileName(strFilePath);
string strCompressedOutputFile = strFilePath + ".compressed";
byte[] arrBuffer = new byte[8192]; //[4096];
int n = -1;
using FileStream input = System.IO.File.OpenRead(strFilePath);
using FileStream raw = new(strCompressedOutputFile, FileMode.Create, FileAccess.ReadWrite);
using Stream compressor = new ParallelDeflateOutputStream(raw);
while ((n = input.Read(arrBuffer, 0, arrBuffer.Length)) != 0)
{
compressor.Write(arrBuffer, 0, n);
}
input.CopyTo(streamResponseBody);
});
return new EmptyResult();
However, this doesn't zip files and send to client (it only creates local zip files on the server).
Using MemoryStream
or creating a local zip file is out of the question and not what I am looking for.
The server should seamlessly stream read bytes of a file, zip it on the fly and send it to client as chunks (like in my ZipArchive
), but with the added benefits of reading those files in parallel and creating a zip of them.
I know that parallelism is usually not optimal for I/O (sometimes a bit worse), but parallel zipping multiple big files should be faster for this case.
I also tried to use SharpZipLib
without success.
Usage of any other libraries is fine as long as it read and stream files to client seamlessly without impacting memory.
Any help is appreciated.
parallel for
in this case is wrong. Why do you think you will benefit from it? have you tried using single thread? Also this code smells, I'll let you know later what should be changed.DotNetZip
code is broken and doesn't work as intended, but I cannot useZipArchive
(which works fine in single thread) in multiple parallel threads.