`<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Nov 8, 2013, at 8:20 AM, Jim Klimov <<a href="mailto:jimklimov@cos.ru">jimklimov@cos.ru</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">The logs specify that your IDE devices (I believe, these are the rpool<br>SSDs in legacy mode) return errors on reads and timeout on retries or<br>resets. This may mean a few things:<br><br>1) Imminent device death i.e. due to wear over lifetime, try to get<br>these replaced with new units (especially if their age or some actual<br>diagnostics results from "smartctl" or vendor tools also indicate the<br>possibility of such scenario)<br></blockquote><div><br></div><div>I vote for this one. The X-25E are well-known for behaving this way as a</div><div>failure mode. The only recourse is to replace the disk.</div><div><br></div><br><blockquote type="cite"><br>2) Bad diagnostics, perhaps due to IDE protocol limitations - try to<br>switch the controller into SATA mode and use some illumos live media<br>(OI LiveCD/LiveUSB or OmniOS equivalents) to boot the server with the<br>rpool disks in SATA mode and run:<br></blockquote><div><br></div><div>This isn't the cause or solution for the disk's woes, but I recommend going</div><div>to AHCI mode at your convenience. You might be able to replace the disk </div><div>without an outage, but this step will require an outage.</div><div> -- richard</div><br><blockquote type="cite"><br>zpool import -N -R /a -f rpool<br>mount -F zfs rpool/ROOT/your_BE_name /a && \<br> touch /a/reconfigure<br>zpool export rpool<br><br>Depending on your OS setup, the BE mounting may require some other<br>command (like "zfs mount rpool/ROOT/your_BE_name").<br><br>This routine mounts the pool, indicates to the BE that it should make<br>new device nodes (so it runs "devfsadm" early in the boot), and exports<br>the pool. In the process, the rpool ZFS labels begin referencing the new<br>hard-disk device node names which is what the rootfs procedure relies<br>on. In some more difficult cases it might help to also copy (rsync) the<br>/dev/ and /devices/ from the live environment into the on-disk BE so<br>that these device names saved into the pool labels would match those<br>discovered by the kernel upon boot.<br><br>Do have backups; it might make sense to complete this experiment with<br>one of the mirror halves removed, so that if nothing works (even rolling<br>back to an IDE-only setup) you can destroy this half's content and boot<br>in IDE mode from the other half and re-attach the mirrored part to it.<br><br>As a variant, it might make sense (if you'd also refresh the hardware)<br>to attach the new device(s) to the rpool as a 3/4-way mirror, and then<br>completing the switcheroo to SATA with only the new couple plugged in -<br>you'd be able to fall back on the old and tested set if all goes wrong<br>somehow.<br><br>Good luck,<br>//Jim<br><br><br>On 2013-11-08 13:35, Hafiz Rafibeyli wrote:<br><blockquote type="cite">log on monitor when system hangs was like this:(can send actuall taken screenshot to individual mail adresses)<br><br>scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,2/ide@0 (ata0):<br> timeout: reset bus, target=0 lun=0<br>scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,2/ide@0 (ata0):<br> timeout: early timeout, target=0 lun=0<br>gda: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,2/ide@0/cmdk@0,0 (Disk0):<br> Error for command 'read sector' Error Level: Informational<br>gda: [ID 107833 kern.notice] Sense Key: aborted command<br>gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' error code: 0x3<br>gda: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,2/ide@0/cmdk@0,0 (Disk0):<br> Error for command 'read sector' Error Level: Informational<br>gda: [ID 107833 kern.notice] Sense Key: aborted command<br>gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' error code: 0x3<br>scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,2/ide@0 (ata0):<br> timeout: abort request, target=0 lun=0<br>scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,2/ide@0 (ata0):<br> timeout: abort device, target=0 lun=0<br>scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,2/ide@0 (ata0):<br> timeout: reset target, target=0 lun=0<br>scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,2/ide@0 (ata0):<br> timeout: reset bus, target=0 lun=0<br>scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,2/ide@0 (ata0):<br> timeout: early timeout, target=0 lun=0<br>gda: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,2/ide@0/cmdk@0,0 (Disk0):<br> Error for command 'read sector' Error Level: Informational<br>gda: [ID 107833 kern.notice] Sense Key: aborted command<br>gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' error code: 0x3<br>gda: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,2/ide@0/cmdk@0,0 (Disk0):<br><br><br>Hello,<br><br>Omnios version:SunOS 5.11 omnios-b281e50<br>Server:Supermicro X8DAH (24x storage chassis)<br><br>we are using omnios as a production nfs server for Esxi hosts.<br><br>everything was ok,but last 20 days system hangs 3 times.Nothing changed on hardware side.<br><br>for OS disks we are using two SSDSA2SH032G1GN(32 Gb Intel X25-E SSD) in zfs mirror attached onboard sata ports of motherboard.<br><br>I captured monitor screenshot when system hangs,and sending as attachment.<br><br><br>My pools info:<br><br>pool: rpool<br> state: ONLINE<br> scan: resilvered 20.0G in 0h3m with 0 errors on Sun Oct 20 14:01:01 2013<br>config:<br><br><span class="Apple-tab-span" style="white-space:pre"> </span>NAME STATE READ WRITE CKSUM<br><span class="Apple-tab-span" style="white-space:pre"> </span>rpool ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> mirror-0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> c4d0s0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> c3d1s0 ONLINE 0 0 0<br><br>errors: No known data errors<br><br><br> pool: zpool1<br> state: ONLINE<br>status: Some supported features are not enabled on the pool. The pool can<br><span class="Apple-tab-span" style="white-space:pre"> </span>still be used, but some features are unavailable.<br>action: Enable all features using 'zpool upgrade'. Once this is done,<br><span class="Apple-tab-span" style="white-space:pre"> </span>the pool may no longer be accessible by software that does not support<br><span class="Apple-tab-span" style="white-space:pre"> </span>the features. See zpool-features(5) for details.<br> scan: scrub repaired 0 in 5h0m with 0 errors on Sat Oct 12 19:00:53 2013<br>config:<br><br><span class="Apple-tab-span" style="white-space:pre"> </span>NAME STATE READ WRITE CKSUM<br><span class="Apple-tab-span" style="white-space:pre"> </span>zpool1 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> raidz1-0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> c1t5000C50041E9D9A7d0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> c1t5000C50041F1A5EFd0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> c1t5000C5004253FF87d0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> c1t5000C50055A607E3d0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> c1t5000C50055A628EFd0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> c1t5000C50055A62F57d0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span>logs<br><span class="Apple-tab-span" style="white-space:pre"> </span> mirror-1 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> c1t5001517959627219d0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> c1t5001517BB2747BE7d0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span>cache<br><span class="Apple-tab-span" style="white-space:pre"> </span> c1t5001517803D007D8d0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span> c1t5001517BB2AFB592d0 ONLINE 0 0 0<br><span class="Apple-tab-span" style="white-space:pre"> </span>spares<br><span class="Apple-tab-span" style="white-space:pre"> </span> c1t5000C5005600A6B3d0 AVAIL<br><span class="Apple-tab-span" style="white-space:pre"> </span> c1t5000C5005600B43Bd0 AVAIL<br><br>errors: No known data errors<br><br><br><br><br><br>_______________________________________________<br>OmniOS-discuss mailing list<br><a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br>http://lists.omniti.com/mailman/listinfo/omnios-discuss<br><br></blockquote><br>_______________________________________________<br>OmniOS-discuss mailing list<br><a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br>http://lists.omniti.com/mailman/listinfo/omnios-discuss<br></blockquote></div><br><div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">--</div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br></div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><a href="mailto:Richard.Elling@RichardElling.com">Richard.Elling@RichardElling.com</a><br>+1-760-896-4422<br><br><br></div></span></div></span></div></span></div></span></div>
</div>
<br></body></html>`