# [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees

Bob Friesenhahn bfriesen at simple.dallas.tx.us
Sat Aug 17 14:33:29 UTC 2013

- Previous message: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees
- Next message: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

    On Sat, 17 Aug 2013, Valrhona wrote:

    > Thanks for the tip. If I understand correctly, lzop is basically a
    > drop-in replacement for gzip that is much faster, but single-threaded.
    > Is there a comparison anywhere between (single-threaded) lzop and
    > (multithreaded) pigz or pbzip2? I have a hex-core xeon and 72 GB of
    > RAM so I expect the compression to be IO-bound rather than CPU-bound,
    > but that is just a guess. And for ZFS streams, is there a reason to
    > prefer any one of these programs over another? Thanks!

    Lzop uses a completely different compression format.  Its default
    compression is a bit less compression than 'gzip -3' but it is much
    more CPU efficient so it is able to achieve "wire speed" level
    compression rates on modern CPUs, and without relying on threading.
    There are some other compressors which are even faster than lzop but
    their compression formats are often not stable.

    A problem with most threaded compressors is that more effective
    compression algorithms (e.g. lzma as used by 'xz' and 'lzip') require
    very large data chunk sizes so that the compression algorithm works
    effectively.  This means that the input data size needs to be large
    (hundreds of MB or even gigabytes) in order for the multi-thread chunk
    size to be large enough.  Zfs send streams or tar/cpio archives of
    large directory trees are likely to be large enough but many/most
    ordinary files don't qualify.  As a result, using threading may result
    in much less compression.

    Bob
    --
    Bob Friesenhahn
    bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
    GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

- Previous message: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees
- Next message: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the OmniOS-discuss mailing list