# [OmniOS-discuss] System hangs randomly

Hafiz Rafibeyli rafibeyli at gmail.com
Mon Nov 11 07:45:28 UTC 2013

- Previous message: [OmniOS-discuss] system hangs randomly
- Next message: [OmniOS-discuss] From Stable to Bloody and vice-versa
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

    Helllo,

    Jim and Richard thank you for your quick answers.

    I will control and  change controller mode to AHCI,but I think Richard right.

    My OS disks are new,but I have some suspicions on Intel X-25E series.

    regards

    Hafiz.

    ----- Orijinal Mesaj -----
    Kimden: omnios-discuss-request at lists.omniti.com
    Kime: omnios-discuss at lists.omniti.com
    Gönderilenler: 9 Kasım Cumartesi 2013 0:02:32
    Konu: OmniOS-discuss Digest, Vol 20, Issue 7

    Send OmniOS-discuss mailing list submissions to
    	omnios-discuss at lists.omniti.com

    To subscribe or unsubscribe via the World Wide Web, visit
    	http://lists.omniti.com/mailman/listinfo/omnios-discuss
    or, via email, send a message with subject or body 'help' to
    	omnios-discuss-request at lists.omniti.com

    You can reach the person managing the list at
    	omnios-discuss-owner at lists.omniti.com

    When replying, please edit your Subject line so it is more specific
    than "Re: Contents of OmniOS-discuss digest..."

    Today's Topics:

       1. Re: NFS Datastore vmware esxi failover (Matt Weiss)
       2. Re: NFS Datastore vmware esxi failover (Richard Elling)
       3. Re: system hangs randomly (Richard Elling)

    ----------------------------------------------------------------------

    Message: 1
    Date: Fri, 08 Nov 2013 12:08:42 -0600
    From: Matt Weiss <mweiss at cimlbr.com>
    To: omnios-discuss at lists.omniti.com
    Subject: Re: [OmniOS-discuss] NFS Datastore vmware esxi failover
    Message-ID: <527D28AA.7000509 at cimlbr.com>
    Content-Type: text/plain; charset=ISO-8859-1; format=flowed

    On 11/8/2013 11:36 AM, Saso Kiselkov wrote:
    > On 11/8/13, 4:17 PM, Matt Weiss wrote:
    >> I am working on a failover script using OmniOS as a NFS server.
    >>
    >> According to VMware, if I mount and nfs datastore via its IP Address
    >> then I should be able to move the IP around and still mount it, however
    >> it is not working right.
    >>
    >> For example:
    >>
    >> On an ESXi instance (5.1U1) I mount the following NFS Datastore
    >> 172.16.50.100
    >> /tank/vmrep
    >> which amounts to a UUID of 6c0c1d0d-928ef591 in /vmfs/volumes
    >>
    >>
    >> omni-rep1: 172.16.50.1
    >> omni-rep2: 172.16.50.2
    >>
    >> I am using zrep to failover my zfs dataset.
    >> http://www.bolthole.com/solaris/zrep/zrep.documentation.html
    >>
    >> Essential, it puts primary into read-only, does a zfs send/receive, then
    >> sets the secondary to rw.
    >>
    >>
    >> To expose my dataset (tank/vmrep) I am using sharenfs property of zfs. I
    >> have created a virtual ip to use for this purpose.
    >>
    >> #setnfsip.sh
    >> ipadm create-addr -T static -a 172.16.50.100/24 vmxnet3s0/nfs
    >>
    >> #removenfsip.sh
    >> ipadm delete-addr vmxnet3s0/nfs
    >>
    >>
    >> So, when I want to failover, I just do the following:
    >>
    >> #!/bin/sh
    >> #zfs unshare tank/vmrep
    >> #sleep 5
    >> /scripts/removenfsip.sh
    >> sync
    >> sleep 5
    >> #zrep sync tank/vmrep
    >> #sleep 5
    >> #the following does the zfs snapshot/send/receive
    >> zrep failover tank/vmrep
    >> sleep 5
    >> #ssh 172.16.50.2 /usr/sbin/zfs share tank/vmrep
    >> #sleep 5
    >> ssh 172.16.50.2 /scripts/setnfsip.sh
    >>
    >>
    >> So, all goes well, omni-rep2 is now exporting tank/vmrep with NFS, it
    >> has the 172.16.50.100 ip address, it is the exact replica of omni-rep1.
    >>
    >> The problem is in ESXi the datastore goes inaccessable.  I can fail back
    >> and the datastore comes back online like fine.  I can mount the nfs
    >> datastore as a new one with the .100 ip on omni-rep2 so it is being
    >> exported properly.
    >>
    >> According to the last paragraph of this
    >>
    >> https://communities.netapp.com/community/netapp-blogs/getvirtical/blog/2011/09/28/nfs-datastore-uuids-how-they-work-and-what-changed-in-vsphere-5
    >>
    >>
    >> It should work, I have merely changed which host is broadcasting my
    >> datastore's IP address.
    >>
    >> I know a guy named saso? did some iScsi failover recently and noted it
    >> worked with NFS.  I am just wondering what I am missing here.
    > I haven't done NFS datastore failover from ESXi myself, but off the top
    > of my head I guess what's going haywire here is that you're setting the
    > dataset read-only before moving it over. Don't do that. Simply tear down
    > the IP address, migrate the dataset, set up a new NFS share on the
    > target machine and then reinstate the IP address at the target. ESXi
    > aggressively monitors the health of its datastores and if it gets to a
    > state it can't deal with (e.g. write a datastore that refuses to process
    > it), it will offline the whole datastore, awaiting administrator
    > intervention.
    >
    > Don't worry about the datastore being offline for a while, ESXi will
    > hold VM writes and the VMs themselves won't usually complain for up to
    > 1-2 minutes (defaults on Windows/Linux).
    >
    I have experimented with
    sharing / unsharing
    leaving shared all the time
    setting RO, RW
    changing IP before/after etc
    adding sleeps, syncs

    So far all have the same result.  I believe the issue now lies with the
    NFS state file.  I may be able to get away with migrating that file with
    my failover script, but we will see.

    Does anyone know how to change OmniOS NFS server parameters?  I would be
    wanting to put the state files into the zfs dataset somewhere to perhaps
    handle that.  Somehow I believe pacemaker takes care of this, so surely,
    somehow NFS can be made to be a true failover.

    ------------------------------

    Message: 2
    Date: Fri, 8 Nov 2013 13:54:50 -0800
    From: Richard Elling <richard.elling at richardelling.com>
    To: Matt Weiss <mweiss at cimlbr.com>
    Cc: omnios-discuss <omnios-discuss at lists.omniti.com>
    Subject: Re: [OmniOS-discuss] NFS Datastore vmware esxi failover
    Message-ID: <3BA1C366-1A6B-448B-B299-9264183B9997 at RichardElling.com>
    Content-Type: text/plain; charset="us-ascii"

    Comment below...

    On Nov 8, 2013, at 8:17 AM, Matt Weiss <mweiss at cimlbr.com> wrote:

    > I am working on a failover script using OmniOS as a NFS server.
    >
    > According to VMware, if I mount and nfs datastore via its IP Address then I should be able to move the IP around and still mount it, however it is not working right.
    >
    > For example:
    >
    > On an ESXi instance (5.1U1) I mount the following NFS Datastore
    > 172.16.50.100
    > /tank/vmrep
    > which amounts to a UUID of 6c0c1d0d-928ef591 in /vmfs/volumes
    >
    >
    > omni-rep1: 172.16.50.1
    > omni-rep2: 172.16.50.2
    >
    > I am using zrep to failover my zfs dataset.
    > http://www.bolthole.com/solaris/zrep/zrep.documentation.html
    >
    > Essential, it puts primary into read-only, does a zfs send/receive, then sets the secondary to rw.
    >
    >
    > To expose my dataset (tank/vmrep) I am using sharenfs property of zfs. I have created a virtual ip to use for this purpose.
    >
    > #setnfsip.sh
    > ipadm create-addr -T static -a 172.16.50.100/24 vmxnet3s0/nfs
    >
    > #removenfsip.sh
    > ipadm delete-addr vmxnet3s0/nfs
    >
    >
    > So, when I want to failover, I just do the following:
    >
    > #!/bin/sh
    > #zfs unshare tank/vmrep
    > #sleep 5
    > /scripts/removenfsip.sh
    > sync
    > sleep 5
    > #zrep sync tank/vmrep
    > #sleep 5
    > #the following does the zfs snapshot/send/receive
    > zrep failover tank/vmrep
    > sleep 5
    > #ssh 172.16.50.2 /usr/sbin/zfs share tank/vmrep
    > #sleep 5
    > ssh 172.16.50.2 /scripts/setnfsip.sh
    >
    >
    > So, all goes well, omni-rep2 is now exporting tank/vmrep with NFS, it has the 172.16.50.100 ip address, it is the exact replica of omni-rep1.

    It is a replica, but it isn't the same from an NFS perspective. The files may have the
    same contents, but the NFSv3 file handles are different because they are in two different
    file systems.  For this to work, the clients would have to remount, which blows your
    requirement for transparency.

    >
    > The problem is in ESXi the datastore goes inaccessable.  I can fail back and the datastore comes back online like fine.  I can mount the nfs datastore as a new one with the .100 ip on omni-rep2 so it is being exported properly.

    Yes, works as designed.

    >
    > According to the last paragraph of this
    >
    > https://communities.netapp.com/community/netapp-blogs/getvirtical/blog/2011/09/28/nfs-datastore-uuids-how-they-work-and-what-changed-in-vsphere-5
    >
    > It should work, I have merely changed which host is broadcasting my datastore's IP address.
    >
    > I know a guy named saso? did some iScsi failover recently and noted it worked with NFS.  I am just wondering what I am missing here.

    Saso's solution, and indeed most failover cluster solutions, use shared storage and not
    ZFS replication between the nodes. One good reason to do this is so you can transparently
    failover NFS service.

    NB, the replication method is used often for disaster recovery, because in DR there often is
    no transparent failover requirement.
     -- richard

    --

    Richard.Elling at RichardElling.com
    +1-760-896-4422

    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20131108/e9b7d40b/attachment-0001.html>

    ------------------------------

    Message: 3
    Date: Fri, 8 Nov 2013 14:02:26 -0800
    From: Richard Elling <richard.elling at richardelling.com>
    To: jimklimov at cos.ru
    Cc: omnios-discuss at lists.omniti.com
    Subject: Re: [OmniOS-discuss] system hangs randomly
    Message-ID: <DB663170-F9B6-472D-BC6A-9DB6D2A4F1C5 at RichardElling.com>
    Content-Type: text/plain; charset="us-ascii"

    On Nov 8, 2013, at 8:20 AM, Jim Klimov <jimklimov at cos.ru> wrote:

    > The logs specify that your IDE devices (I believe, these are the rpool
    > SSDs in legacy mode) return errors on reads and timeout on retries or
    > resets. This may mean a few things:
    >
    > 1) Imminent device death i.e. due to wear over lifetime, try to get
    > these replaced with new units (especially if their age or some actual
    > diagnostics results from "smartctl" or vendor tools also indicate the
    > possibility of such scenario)

    I vote for this one. The X-25E are well-known for behaving this way as a
    failure mode. The only recourse is to replace the disk.

    >
    > 2) Bad diagnostics, perhaps due to IDE protocol limitations - try to
    > switch the controller into SATA mode and use some illumos live media
    > (OI LiveCD/LiveUSB or OmniOS equivalents) to boot the server with the
    > rpool disks in SATA mode and run:

    This isn't the cause or solution for the disk's woes, but I recommend going
    to AHCI mode at your convenience. You might be able to replace the disk
    without an outage, but this step will require an outage.
     -- richard

    >
    > zpool import -N -R /a -f rpool
    > mount -F zfs rpool/ROOT/your_BE_name /a && \
    >  touch /a/reconfigure
    > zpool export rpool
    >
    > Depending on your OS setup, the BE mounting may require some other
    > command (like "zfs mount rpool/ROOT/your_BE_name").
    >
    > This routine mounts the pool, indicates to the BE that it should make
    > new device nodes (so it runs "devfsadm" early in the boot), and exports
    > the pool. In the process, the rpool ZFS labels begin referencing the new
    > hard-disk device node names which is what the rootfs procedure relies
    > on. In some more difficult cases it might help to also copy (rsync) the
    > /dev/ and /devices/ from the live environment into the on-disk BE so
    > that these device names saved into the pool labels would match those
    > discovered by the kernel upon boot.
    >
    > Do have backups; it might make sense to complete this experiment with
    > one of the mirror halves removed, so that if nothing works (even rolling
    > back to an IDE-only setup) you can destroy this half's content and boot
    > in IDE mode from the other half and re-attach the mirrored part to it.
    >
    > As a variant, it might make sense (if you'd also refresh the hardware)
    > to attach the new device(s) to the rpool as a 3/4-way mirror, and then
    > completing the switcheroo to SATA with only the new couple plugged in -
    > you'd be able to fall back on the old and tested set if all goes wrong
    > somehow.
    >
    > Good luck,
    > //Jim
    >
    >
    > On 2013-11-08 13:35, Hafiz Rafibeyli wrote:
    >> log on  monitor when system hangs was like this:(can send actuall taken screenshot to individual mail adresses)
    >>
    >> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
    >>         timeout: reset bus, target=0 lun=0
    >> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
    >>         timeout: early timeout, target=0 lun=0
    >> gda: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0/cmdk at 0,0 (Disk0):
    >>         Error for command 'read sector'   Error Level: Informational
    >> gda: [ID 107833 kern.notice]           Sense Key: aborted command
    >> gda: [ID 107833 kern.notice]           Vendor 'Gen-ATA ' error code: 0x3
    >> gda: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0/cmdk at 0,0 (Disk0):
    >>         Error for command 'read sector'   Error Level: Informational
    >> gda: [ID 107833 kern.notice]           Sense Key: aborted command
    >> gda: [ID 107833 kern.notice]           Vendor 'Gen-ATA ' error code: 0x3
    >> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
    >>         timeout: abort request, target=0 lun=0
    >> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
    >>         timeout: abort device, target=0 lun=0
    >> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
    >>         timeout: reset target, target=0 lun=0
    >> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
    >>         timeout: reset bus, target=0 lun=0
    >> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
    >>         timeout: early timeout, target=0 lun=0
    >> gda: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0/cmdk at 0,0 (Disk0):
    >>         Error for command 'read sector'   Error Level: Informational
    >> gda: [ID 107833 kern.notice]           Sense Key: aborted command
    >> gda: [ID 107833 kern.notice]           Vendor 'Gen-ATA ' error code: 0x3
    >> gda: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0/cmdk at 0,0 (Disk0):
    >>
    >>
    >> Hello,
    >>
    >> Omnios version:SunOS  5.11 omnios-b281e50
    >> Server:Supermicro X8DAH (24x storage chassis)
    >>
    >> we are using omnios as a production nfs server for Esxi hosts.
    >>
    >> everything was ok,but last 20 days system hangs 3 times.Nothing changed on hardware side.
    >>
    >> for  OS disks we are using two SSDSA2SH032G1GN(32 Gb Intel X25-E SSD)  in zfs mirror  attached onboard sata ports of motherboard.
    >>
    >> I captured monitor screenshot when system hangs,and sending as attachment.
    >>
    >>
    >> My pools info:
    >>
    >> pool: rpool
    >>  state: ONLINE
    >>   scan: resilvered 20.0G in 0h3m with 0 errors on Sun Oct 20 14:01:01 2013
    >> config:
    >>
    >> 	NAME        STATE     READ WRITE CKSUM
    >> 	rpool       ONLINE       0     0     0
    >> 	  mirror-0  ONLINE       0     0     0
    >> 	    c4d0s0  ONLINE       0     0     0
    >> 	    c3d1s0  ONLINE       0     0     0
    >>
    >> errors: No known data errors
    >>
    >>
    >>   pool: zpool1
    >>  state: ONLINE
    >> status: Some supported features are not enabled on the pool. The pool can
    >> 	still be used, but some features are unavailable.
    >> action: Enable all features using 'zpool upgrade'. Once this is done,
    >> 	the pool may no longer be accessible by software that does not support
    >> 	the features. See zpool-features(5) for details.
    >>   scan: scrub repaired 0 in 5h0m with 0 errors on Sat Oct 12 19:00:53 2013
    >> config:
    >>
    >> 	NAME                       STATE     READ WRITE CKSUM
    >> 	zpool1                     ONLINE       0     0     0
    >> 	  raidz1-0                 ONLINE       0     0     0
    >> 	    c1t5000C50041E9D9A7d0  ONLINE       0     0     0
    >> 	    c1t5000C50041F1A5EFd0  ONLINE       0     0     0
    >> 	    c1t5000C5004253FF87d0  ONLINE       0     0     0
    >> 	    c1t5000C50055A607E3d0  ONLINE       0     0     0
    >> 	    c1t5000C50055A628EFd0  ONLINE       0     0     0
    >> 	    c1t5000C50055A62F57d0  ONLINE       0     0     0
    >> 	logs
    >> 	  mirror-1                 ONLINE       0     0     0
    >> 	    c1t5001517959627219d0  ONLINE       0     0     0
    >> 	    c1t5001517BB2747BE7d0  ONLINE       0     0     0
    >> 	cache
    >> 	  c1t5001517803D007D8d0    ONLINE       0     0     0
    >> 	  c1t5001517BB2AFB592d0    ONLINE       0     0     0
    >> 	spares
    >> 	  c1t5000C5005600A6B3d0    AVAIL
    >> 	  c1t5000C5005600B43Bd0    AVAIL
    >>
    >> errors: No known data errors
    >>
    >>
    >>
    >>
    >>
    >> _______________________________________________
    >> OmniOS-discuss mailing list
    >> OmniOS-discuss at lists.omniti.com
    >> http://lists.omniti.com/mailman/listinfo/omnios-discuss
    >>
    >
    > _______________________________________________
    > OmniOS-discuss mailing list
    > OmniOS-discuss at lists.omniti.com
    > http://lists.omniti.com/mailman/listinfo/omnios-discuss

    --

    Richard.Elling at RichardElling.com
    +1-760-896-4422

    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20131108/7601fda4/attachment.html>

    ------------------------------

    Subject: Digest Footer

    _______________________________________________
    OmniOS-discuss mailing list
    OmniOS-discuss at lists.omniti.com
    http://lists.omniti.com/mailman/listinfo/omnios-discuss

    ------------------------------

    End of OmniOS-discuss Digest, Vol 20, Issue 7
    *********************************************

- Previous message: [OmniOS-discuss] system hangs randomly
- Next message: [OmniOS-discuss] From Stable to Bloody and vice-versa
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the OmniOS-discuss mailing list