# [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees

Valrhona valrhona at gmail.com
Sat Aug 17 03:47:24 UTC 2013

- Previous message: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees
- Next message: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

    y> I am not sure about repositories, but there are projects such as
    > pigz and pbzip2 which are parallelized interfaces to the same
    > compression libraries, and easily compilable
    Thanks! Looks like pigz and pbzip2 are in the libraries; thanks Eric
    for the tip!

    > In some versions of pigz there was a problem with compression of
    > multiple filename arguments (some state was not cleared, so they
    > were processed as if concatenated), so for predictable results
    > better script up loops and call each time for one argument :)
    At the moment, I am just interested in a single file, piped in from
    zfs send. Would the problem you mention affect me?

    > You can look into examples on cpio manpage - it takes results of
    > the find command as input, so you can "find" piles according to
    > the name pattern(s) you need, and as part of their path they would
    > include the directories which contain them. If you need the whole
    > directory structure, try "find -type d" - though you might need
    > to somehow tell cpio to only archive the directories and not their
    > contents recursively. Or if you're doing this locally - just create
    > the structure at new root node with "mkdir -p". For remote sending,
    > you might do the same and then archive this empty structure ;)
    So I found an easier solution: rsync

    for files with extension .xyz

    rsync -avgR --include='*/' --include='*.xyz' --exclude='*'
    --remove-source-files /src /dest

    This recreates the whole subdirectory structure, including (many)
    directories which have no files of type .xyz.

    To remove them, in /dest

    find . -type d -empty -exec rmdir {} \;

    You have to run this a few times because only one level of empty
    directory gets removed each time, so if you have empty directories
    that are subdirectories that then become empty, etc. etc.

    Thanks!

    Peter

- Previous message: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees
- Next message: [OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the OmniOS-discuss mailing list