Compress any size documents by applying parallelism

Parallel Archive: Unleashing the Power of Multi-core Processing

Overwiew

Most compression algorithms were designed for single-core processors. However, in the era of multi-core CPUs, it is beneficial to utilize all your CPU cores to speed up archive creation. Aspose.ZIP provides this capability for some archive formats.

Parallel methods on BZIP2, LZIP and XZ archives

Parallelization is possible due to block nature of these algorithms. On the program level these archives use common approach of multi-core compression. User can set the number of threads via CompressionThreads property. So if this property is more than one several CPU cores would be used.

Sample 1 – Parallel compression to BZIP2    using (Bzip2Archive archive = new Bzip2Archive())
    {
        archive.SetSource("data.bin");
        archive.Save("archive.bz2", new Bzip2SaveOptions() { CompressionThreads = 4 });
    }

Sample 2 - Parallel compression to LZIP    var settings = new LzipArchiveSettings(16777216) { CompressionThreads = 4 }
    using (LzipArchive archive = new LzipArchive(settings))
    {
        archive.SetSource("data.bin");
        archive.Save("archive.lz");
    }

Sample 3 - Parallel compression to XZ    using (Bzip2Archive archive = new Bzip2Archive())
    {
        archive.SetSource("data.bin");
        archive.Save("archive.bz2", new Bzip2SaveOptions() { CompressionThreads = 4 });
    }

How to Zip a Folder with .NET

ZIP archives serve the purpose of compressing and consolidating one or multiple files or folders into a unified container. In .NET , you can use the ZipFile class to work with ZIP files and compression. The primary objective of archiving is typically to diminish the file size for storage or transmission, while also incorporating encryption measures for enhanced security. Beyond traditional file compression tools, automated compression/extraction features find application in diverse desktop and web applications, facilitating tasks such as uploading, downloading, sharing, or encrypting files.

7Z LZMA2 archives

One of 7Z format compression methods is LZMA2 , which can be compressed in several threads. Similarly to previous kind of archives you can set the number of threads via CompressionThreads property.

Sample 1 – Parallel compression to 7Z LZMA2    var settings = new SevenZipEntrySettings(new SevenZipLZMA2CompressionSettings(){ CompressionThreads = 4 });
    using (SevenZipArchive archive = new SevenZipArchive(settings))
    {
        archive.CreateEntry("first.bin", "data.bin");
        archive.Save("result.7z");
    }

Parallel Method on ZIP archives

For previous archive classes parallelization was applied either on block or algorithm level. For the ZIP archive it is possible on entry level. For doing it instantiate ParallelOptions on archive saving.

Sample 1 – Parallel compression to zip    using (Archive archive = new Archive())
    {
        archive.CreateEntries(@"C:\Data");
        archive.Save(zipFile, new ArchiveSaveOptions()
        {
            ParallelOptions = new ParallelOptions() 
            { ParallelCompressInMemory = ParallelCompressionMode.Auto }
        });
    }

The Parallel-Compress-In-Memory setup strategy is used to multitask

Setting ParallelCompressInMemory indicates the strategy we choose to multitask. Here are three options :
ParallelCompressionMode.Never: compression of all entries is sequential. Only one CPU core works on compression and flushes compressed data as it comes.
ParallelCompressionMode.Always: It forces compression of entries in different threads regardless of entry size, available memory, and other factors. Each CPU core simultaneously compresses a file keeping its compressed data in RAM. Upon the entry is compressed it flushes to the result stream. If your RAM amount is small and the total size of some N entries (where N is the number of CPU cores) is huge it may happen that all RAM available for CLR will exhaust and OutOfMemoryExcepton arises.
ParallelCompressionMode.Auto: It estimates CPU cores, sizes of entries, available memory and chooses whether to compress entries in parallel or sequentially. In this mode some smaller entries to be compressed in parallel while others sequentially. LZMA and PPMd entries are not compressed in parallel because of high memory consumption. Generally, it is safe to go with this option; Aspose.ZIP is wary with estimations and switches to sequential compression as a fallback. There is one more property of ParallelOptions for this mode - AvailableMemorySize. It is pointless for any other mode. Roughly speaking, it is the high limit of allocated memory while compressing entries with all CPU cores, in megabytes. Aspose.ZIP uses that number to estimate the biggest size of entry which is safe to be compressed in parallel. Entries above the threshold to be compressed sequentially. AvailableMemorySize is a double-edged sword: being set too high with huge entries, it can produce RAM exhaustion, intense swap, and even might be out of memory exception. Being set too low, most of the entries will be compressed in a sequential way without much speed-up. So, sophisticated users can assign it considering trade-off.
We encourage you to play with different modes of parallel compression on your typical data to determine what the best settings in your case are.