# [OmniOS-discuss] Hung ZFS Pool

Brian Hechinger wonko at 4amlunch.net
Thu Dec 17 19:05:21 UTC 2015

- Previous message: [OmniOS-discuss] Hung ZFS Pool
- Next message: [OmniOS-discuss] Hung ZFS Pool
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

    Ok, let’s add to the weirdness.

    I destroyed the degraded pool.

    I re-created it.

    I then re-ran iozone.

    It completed with zero errors on the pool. iozone did have some issues at the end, but the FS seems ok:

      pool: zoom
     state: ONLINE
      scan: none requested
    config:

            NAME          STATE     READ WRITE CKSUM
            zoom          ONLINE       0     0     0
              mirror-0    ONLINE       0     0     0
                c4t1d0s1  ONLINE       0     0     0
                c5t1d0s1  ONLINE       0     0     0

    errors: No known data errors

            Iozone: Performance Test of File I/O
                    Version $Revision: 3.434 $
                    Compiled for 64 bit mode.
                    Build: Solaris10gcc-64

            Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                         Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                         Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                         Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                         Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                         Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                         Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
                         Vangel Bojaxhi, Ben England, Vikentsi Lapa,
                         Alexey Skidanov.

            Run began: Thu Dec 17 13:21:59 2015

            Multi_buffer. Work area 16777216 bytes
            OPS Mode. Output is in operations per second.
            Record Size 8 kB
            SYNC Mode.
            File size set to 2097152 kB
            Command line used: /usr/local/bin/iozone -m -t 16 -T -O -r 8k -o -s 2G
            Time Resolution = 0.000001 seconds.
            Processor cache size set to 1024 kBytes.
            Processor cache line size set to 32 bytes.
            File stride size set to 17 * record size.
            Throughput test with 16 threads
            Each thread writes a 2097152 kByte file in 8 kByte records

            Children see throughput for 16 initial writers  =   29558.13 ops/sec
            Parent sees throughput for 16 initial writers   =   29467.57 ops/sec
            Min throughput per thread                       =    1845.28 ops/sec
            Max throughput per thread                       =    1853.28 ops/sec
            Avg throughput per thread                       =    1847.38 ops/sec
            Min xfer                                        =  261012.00 ops

            Children see throughput for 16 rewriters        =   26802.94 ops/sec
            Parent sees throughput for 16 rewriters         =   26801.51 ops/sec
            Min throughput per thread                       =    1671.70 ops/sec
            Max throughput per thread                       =    1679.40 ops/sec
            Avg throughput per thread                       =    1675.18 ops/sec
            Min xfer                                        =  260942.00 ops

            Children see throughput for 16 readers          =  305525.26 ops/sec
            Parent sees throughput for 16 readers           =  304910.58 ops/sec
            Min throughput per thread                       =   16371.37 ops/sec
            Max throughput per thread                       =   20084.48 ops/sec
            Avg throughput per thread                       =   19095.33 ops/sec
            Min xfer                                        =  213905.00 ops

            Children see throughput for 16 re-readers       =  301510.86 ops/sec
            Parent sees throughput for 16 re-readers        =  301021.85 ops/sec
            Min throughput per thread                       =   16066.28 ops/sec
            Max throughput per thread                       =   19850.40 ops/sec
            Avg throughput per thread                       =   18844.43 ops/sec
            Min xfer                                        =  212289.00 ops

            Children see throughput for 16 reverse readers  =  520691.82 ops/sec
            Parent sees throughput for 16 reverse readers   =  520026.68 ops/sec
            Min throughput per thread                       =   30897.40 ops/sec
            Max throughput per thread                       =   33412.20 ops/sec
            Avg throughput per thread                       =   32543.24 ops/sec
            Min xfer                                        =  242448.00 ops

            Children see throughput for 16 stride readers   =   27067.77 ops/sec
            Parent sees throughput for 16 stride readers    =   27064.74 ops/sec
            Min throughput per thread                       =    1549.09 ops/sec
            Max throughput per thread                       =    3205.10 ops/sec
            Avg throughput per thread                       =    1691.74 ops/sec
            Min xfer                                        =  126699.00 ops

            Children see throughput for 16 random readers   =  215258.98 ops/sec
            Parent sees throughput for 16 random readers    =  214461.71 ops/sec
            Min throughput per thread                       =    2759.80 ops/sec
            Max throughput per thread                       =  169551.89 ops/sec
            Avg throughput per thread                       =   13453.69 ops/sec
            Min xfer                                        =    4281.00 ops

            Children see throughput for 16 mixed workload   =    8673.89 ops/sec
            Parent sees throughput for 16 mixed workload    =    6341.03 ops/sec
            Min throughput per thread                       =     442.73 ops/sec
            Max throughput per thread                       =     641.36 ops/sec
            Avg throughput per thread                       =     542.12 ops/sec
            Min xfer                                        =  180991.00 ops

            Children see throughput for 16 random writers   =    4008.54 ops/sec
            Parent sees throughput for 16 random writers    =    3972.48 ops/sec
            Min throughput per thread                       =     248.54 ops/sec
            Max throughput per thread                       =     252.76 ops/sec
            Avg throughput per thread                       =     250.53 ops/sec
            Min xfer                                        =  257769.00 ops

            Children see throughput for 16 fwriters         =   70222.20 ops/sec
            Parent sees throughput for 16 fwriters          =   65632.32 ops/sec
            Min throughput per thread                       =    4132.12 ops/sec
            Max throughput per thread                       =    4686.85 ops/sec
            Avg throughput per thread                       =    4388.89 ops/sec
            Min xfer                                        =  262144.00 ops

    Error in file: Found ?0? Expecting ?7979797979797979? addr 29f6770
    Error in file: Found ?0? Expecting ?7979797979797979? addr 29f6770
    Error in file: Position 0
    Error in file: Position 0
    Record # 0 Record size 8 kb
    Record # 0 Record size 8 kb
    where 29f6770x loop 0
    where 29f6770x loop 0

    I can delete and create files just fine.

    Grrrr.

    -brian

    > On Dec 9, 2015, at 11:27 AM, Brian Hechinger <wonko at 4amlunch.net> wrote:
    >
    >
    >> On Dec 9, 2015, at 11:22 AM, Dan McDonald <danmcd at omniti.com> wrote:
    >>
    >>
    >>> On Dec 9, 2015, at 11:18 AM, Brian Hechinger <wonko at 4amlunch.net> wrote:
    >>>
    >>> It’s brand new!!
    >>
    >> Sometimes you get flaky HW that's new.  I've had to return new spinning-rust disks, for example.
    >
    > Bah. :(
    >
    >>
    >>> Also, I would expect the other slice to be affected as well?  It’s been humming along just fine as SLOG with no errors:
    >>>
    >>>      logs
    >>>        mirror-3    ONLINE       0     0     0
    >>>          c4t1d0s0  ONLINE       0     0     0
    >>>          c5t1d0s0  ONLINE       0     0     0
    >>
    >> Could just be bad luck your slog hasn't encountered the bad portion of this drive.
    >
    > I suppose. You think there is a maybe a good way to test this device before I try to get it RMA-ed?
    >
    >> Also, what OmniOS revision are you running? If you're not up to the latest November r151014 update, you may be missing some NVMe fixes.
    >
    > Oh right, totally forgot to do that for you:
    >
    > wonko at basket1:/var/adm$ head /etc/release ; uname -a
    >  OmniOS v11 r151016
    >  Copyright 2015 OmniTI Computer Consulting, Inc. All rights reserved.
    >  Use is subject to license terms.
    > SunOS basket1 5.11 omnios-073d8c0 i86pc i386 i86pc
    >

- Previous message: [OmniOS-discuss] Hung ZFS Pool
- Next message: [OmniOS-discuss] Hung ZFS Pool
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the OmniOS-discuss mailing list