Compress any size documents by applying parallelism via Python.Net

Overview

Parallelizing BZIP2, LZIP, and XZ Archives

The number of compression threads can be set through the "compression threads" property of the corresponding settings class. The default value is 1, and compression is thus performed on a single thread. However, if you set a value greater than one, the algorithm will use the corresponding number of processor cores

Parallel compression to BZIP2

    save_options =  zp.bzip2.Bzip2SaveOptions()
    save_options.compression_threads = 4
    with zp.bzip2.Bzip2Archive() as archive:
        archive.set_source("data.bin")
        archive.save("archive.bz2", save_options)

Parallel compression to LZIP

    # Will use 16Mb for dictionary:
    settings = zp.lzip.LzipArchiveSettings(0x1000000)
    settings.compression_threads = 4
    with zp.lzip.LzipArchive(settings) as archive:
        archive.set_source("data.bin")
        archive.save("archive.lz")

Parallel compression to XZ

    settings = zp.xz.settings.XzArchiveSettings()
    settings.compression_threads = 4
    with zp.xz.XzArchive(settings) as archive:
        archive.set_source("data.bin")
        archive.save("archive.xz")

In these examples, the Python code snippets perform similar actions for creating archives, differing only in the compression formats used. They typically follow these steps:

• Create a settings object specific to the archive format.
• Set the number of parallelism streams for compression.
• An archive object is created in the context with, which ensures automatic closing of the archive after operations are completed.
• Specify the data source to be added to the archive.
• Save the archive in the appropriate format using the configured settings.

LZMA2 Compression and Multithreading in 7Z Archives

This example demonstrates how to create a 7z archive using the data.bin file. We use LZMA2 compression algorithm with four streams to efficiently compress data.

Parallel compression to 7Z LZMA2

    compression_settings = zp.saving.SevenZipLZMA2CompressionSettings()
    compression_settings.compression_threads = 4
    entry_settings = zp.saving.SevenZipEntrySettings(compression_settings)
    with zp.sevenzip.SevenZipArchive(entry_settings) as archive:
        archive.create_entry("data.bin", "data.bin")
        archive.save("result.7z")

This code creates a 7z archive by reading the data from the data.bin file, using SevenZipLZMA2CompressionSettings instead of the settings for Bzip2, Lzip, or XZ, and the LZMA2 compression algorithm with four threads for compression. It uses the create_entry method of SevenZipArchive instead of the set_source method.

Speed Up ZIP Archiving with Parallel Compression

The creation of a ZIP archive can be parallelized at the entry level. To do this, use the ParallelOptions instance and its properties

    parallel_opt = zp.saving.ParallelOptions()
    parallel_opt.parallel_compress_in_memory =  zp.saving.ParallelCompressionMode.AUTO
    save_opt =  zp.saving.ArchiveSaveOptions()
    save_opt.parallel_options = parallel_opt
    with zp.Archive as archive:
        archive.create_entries("~/Data")
        archive.Save("archive.zip", save_opt)

The main features are:

• Using ParallelOptions to configure parallel in-memory compression.
• Using ArchiveSaveOptions for general save settings.
• Employing the create_entries method to add files from the ~/Data directory to the archive.