From skeltonr at btconnect.com  Fri Sep  1 23:04:07 2017
From: skeltonr at btconnect.com (Richard Skelton)
Date: Sat, 02 Sep 2017 00:04:07 +0100
Subject: [OmniOS-discuss] ZFS cache device allocation a free seem strange on
 my OmniOS r151022 system.
Message-ID: <59A9E767.1030608@btconnect.com>

Hi,
The cache allocation a free seem strange on my OmniOS r151022 system.
The cache device is an Intel SSD DC P3700 Series 800GB, 1/2 Height PCIe
3.0, 20nm, MLC
root at omnios:/scratch# zpool list -v scratch
NAME                         SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP 
DEDUP  HEALTH  ALTROOT
scratch                     2.66T  1.94T   738G         -    17%    72% 
1.00x  ONLINE  -
  mirror                     544G   395G   149G         -    16%    72%
    c7t5000CCA018309809d0       -      -      -         -      -      -
    c18t5000CCA038718395d0      -      -      -         -      -      -
  mirror                     544G   396G   148G         -    15%    72%
    c14t5000CCA038710745d0      -      -      -         -      -      -
    c11t5000CCA038721F5Dd0      -      -      -         -      -      -
  mirror                     544G   396G   148G         -    18%    72%
    c12t5000CCA03871048Dd0      -      -      -         -      -      -
    c13t5000CCA0387102E1d0      -      -      -         -      -      -
  mirror                     544G   398G   146G         -    19%    73%
    c15t5000CCA03871058Dd0      -      -      -         -      -      -
    c16t5000CCA0387224EDd0      -      -      -         -      -      -
  mirror                     544G   396G   148G         -    17%    72%
    c17t5000CCA038717F65d0      -      -      -         -      -      -
    c10t5000CCA038717AC9d0      -      -      -         -      -      -
log                             -      -      -         -      -      -
  c6t0d0                    3.84G      0  3.84G         -     0%     0%
cache                           -      -      -         -      -      -
  c3t1d0                     745G  *1.07T  16.0E         -     0%   147%*
spare                           -      -      -         -      -      -
  c9t5000CCA038717B2Dd0         -      -      -         -      -      -
  c8t5000CCA0387104B1d0         -      -      -         -      -      -
root at omnios:/scratch# format </dev/null
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c2t4d0 <Samsung-SSD 850 PRO 256GB-EXM02B6Q-238.47GB>
          /pci at 0,0/pci8086,35c5 at 1f,2/disk at 4,0
       1. c2t5d0 <Samsung-SSD 850 PRO 256GB-EXM02B6Q-238.47GB>
          /pci at 0,0/pci8086,35c5 at 1f,2/disk at 5,0
       2. c3t1d0 <INTEL-SSDPEDMD800G4-8DV10171-745.21GB>
          /pci at 75,0/pci8086,6f0a at 3,2/pci8086,3702 at 0/blkdev at 1,0
       3. c6t0d0 <DDRDRIVE-X1-0078-3.87GB>
          /pci at 75,0/pci8086,6f06 at 2,2/pci19e3,81 at 0/sd at 0,0


Cheers
Richard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170902/778b9380/attachment.html>

From an3s.annema at gmail.com  Mon Sep  4 18:17:36 2017
From: an3s.annema at gmail.com (Andries Annema)
Date: Mon, 4 Sep 2017 20:17:36 +0200
Subject: [OmniOS-discuss] pkglint: Python 2.7 MemoryError
In-Reply-To: <52dddb24-3ebf-4de1-36b8-c88715c8b0b0@gmail.com>
References: <52dddb24-3ebf-4de1-36b8-c88715c8b0b0@gmail.com>
Message-ID: <2835e1ba-390e-31c0-c6ec-f0530b9053d4@gmail.com>

Finally found the time to solve this issue. And actually, it is a little 
embarrassing. Category "user error".

Anyway, for the record, here is the cause and the solution.

While I did give the VM 1, 2 and even 4 GB of RAM to play with, I tend 
to deploy my zones with a memory cap.
In this case, I had this zone set to only use 256MB which was clearly 
too little. But, when I increased RAM to the VM, I forgot to also 
increase the memory cap on the zone.... D'oh!
Increased it to 512MB and bam! No "MemoryError" anymore whatsoever.

Andries


On 2017-07-13 19:41, Andries Annema wrote:
> Hi guys,
>
> I've got a freshly installed r151022 VM here (under VMWare 
> Workstation) to fiddle around a bit. Very clean, no extra packages 
> installed yet, not even NFS- or SMB-shares. Only two non-global zones: 
> one to host a repository and one as build environment for some 
> packages I use myself.
> I have done all this on r151014 machines (VM/test as well as 
> physical/production) as well, but have never bumped into the error I'm 
> getting now.
>
> Not sure if it is package-specific or more of a general error - I 
> assume the latter - but for completeness sake, I'm trying to put 
> Znapzend 0.17.0 into a neat IPS packages.
> .configure, gmake, gmake install, as well as creating the SMF method 
> and manifest files, and the IPS package manifest go just like I got 
> before on r151014, but now on r151022 the "pkglint" process fails.
> This is what it throws at me:
>
> ==begin quote==
> builder at vm09ngz02build:/tank/build/1_scratch$ sudo pkglint -c 
> ./lint-cache -r http://<ip-of-local-ips-repo> znapzend.p5m.3.res
> Lint engine setup...
>
> Ignoring -r option, existing image found.
> Lint setup 3?????????????????????????????? 1485/1838Traceback (most 
> recent call last):
> ? File "/usr/bin/pkglint", line 317, in <module>
> ??? __ret = main_func()
> ? File "/usr/bin/pkglint", line 148, in main_func
> ??? release=opts.release)
> ? File "/usr/lib/python2.7/vendor-packages/pkg/lint/engine.py", line 
> 607, in setup
> ??? checker.startup(self)
> ? File 
> "/usr/lib/python2.7/vendor-packages/pkg/lint/pkglint_action.py", line 
> 174, in startup
> ??? seed_dict(manifest, "path", self.ref_paths)
> ? File 
> "/usr/lib/python2.7/vendor-packages/pkg/lint/pkglint_action.py", line 
> 126, in seed_dict
> ??? for action in mf_gen(atype):
> ? File 
> "/usr/lib/python2.7/vendor-packages/pkg/lint/pkglint_action.py", line 
> 123, in mf_gen
> ??? for a in mf.gen_actions():
> ? File "/usr/lib/python2.7/vendor-packages/pkg/manifest.py", line 
> 1898, in gen_actions
> ? File "/usr/lib/python2.7/vendor-packages/pkg/manifest.py", line 
> 1563, in __load
> ??? self.set_content(excludes=self.excludes, pathname=self.pathname)
> ? File "/usr/lib/python2.7/vendor-packages/pkg/manifest.py", line 
> 1052, in set_content
> ??? self.add_action(action, excludes)
> ? File "/usr/lib/python2.7/vendor-packages/pkg/manifest.py", line 
> 1094, in add_action
> ??? self.actions.append(action)
> MemoryError
> Error:
>
> This is an internal error in pkg(5) version 1493165709.? Please log a
> Service Request about this issue including the information above and this
> message.
> ==end quote==
>
> Any ideas? Am I doing something wrong or is there a bug in Python 2.7 
> or even pkg(5)?
> Assuming it simply had not enough memory to get the job done, I tried 
> increasing the VM's assigned memory from 1 to 2, even to 4GB. But that 
> didn't help.
>
> This proves I'm still a novice on compiling and maintaining 
> packages... *sigh*
> Anyway, any hints appreciated! Tnx.
>
> Regards,
> Andries
>
>
>


From johan.kragsterman at capvert.se  Tue Sep  5 07:29:40 2017
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Tue, 5 Sep 2017 09:29:40 +0200
Subject: [OmniOS-discuss] email notifications from SMF and FMA
Message-ID: <OF36592AB3.41A49AC8-ONC1258192.002925FC-C1258192.00292B60@inse.com>

An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170905/12fea0de/attachment.html>

From doug at will.to  Tue Sep  5 12:28:19 2017
From: doug at will.to (Doug Hughes)
Date: Tue, 5 Sep 2017 08:28:19 -0400
Subject: [OmniOS-discuss] email notifications from SMF and FMA
In-Reply-To: <OF36592AB3.41A49AC8-ONC1258192.002925FC-C1258192.00292B60@inse.com>
References: <OF36592AB3.41A49AC8-ONC1258192.002925FC-C1258192.00292B60@inse.com>
Message-ID: <59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost>

wouldn't it be easier to use your local MTA (postfix, sendmail) to do this? Then SMF just has to execute /bin/mail or equivalent and your MTA can act as an outbound relay agent.

Sent from my android device.

-----Original Message-----
From: Johan Kragsterman <johan.kragsterman at capvert.se>
To: omnios-discuss <omnios-discuss at lists.omniti.com>
Sent: Tue, 05 Sep 2017 3:44
Subject: [OmniOS-discuss] email notifications from SMF and FMA

Hi!


I need to send mails from SMF to a remote SMTP server, and want/need to include auth credentials.


I know how to use the SMF service to send the mail.


I know how to get the user/passwd into encode_base64, but how do I pass it on to the mail? As well as the port. I guess the port would look like this: root at mailserver.com:465, would it?


But the rest...?


I've seen Peter Tribble been blogging about these features, but not exactly how it would look like to pass on the credentials to your mail.


Best regards from/Med v?nliga h?lsningar fr?n

Johan Kragsterman


Capvert

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170905/0cd17676/attachment-0001.html>

From johan.kragsterman at capvert.se  Tue Sep  5 14:23:15 2017
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Tue, 5 Sep 2017 16:23:15 +0200
Subject: [OmniOS-discuss] Ang: Re: Ang: Re: email notifications from SMF and
	FMA
In-Reply-To: <05bdf145-bd95-e97c-fb3a-0d1f5ed7454b@will.to>
References: <05bdf145-bd95-e97c-fb3a-0d1f5ed7454b@will.to>,
	<59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost>
	<OF36592AB3.41A49AC8-ONC1258192.002925FC-C1258192.00292B60@inse.com>
	<OFBB20D28B.90E204BF-ONC1258192.00476B42-C1258192.004821B8@inse.com>
Message-ID: <OF43773ADD.834EAB74-ONC1258192.004DF7AD-C1258192.004F0875@inse.com>


Hi!

Of coarse, forgot to include list, haven't been using this in a while, therefore a bit rusty...




-----Doug Hughes <doug at will.to> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se>
Fr?n: Doug Hughes <doug at will.to>
Datum: 2017-09-05 15:44
?rende: Re: Ang: Re: [OmniOS-discuss] email notifications from SMF and FMA

Perhaps you meant to reply-all?

I don't know what SMF uses for sure, but I'd be surprised if it was not
the local mailer.

nullmailer seems like it would be a good choice, though it would require
some porting. postfix in relay mode would be very low overhead.





It is not really the overhead I am worrying about, it is an extra service that isn't needed, I want to remove every possible piece that can cause failure.

I'm sure someone on this list knows what SMF uses. And as well how to send mail from SMF to remote hosts as well, including auth.


BR Johan





On 9/5/2017 9:07 AM, Johan Kragsterman wrote:
> Hi!
>
> -----Doug Hughes <doug at will.to> skrev: -----
> Till: omnios-discuss <omnios-discuss at lists.omniti.com>, Johan Kragsterman <johan.kragsterman at capvert.se>
> Fr?n: Doug Hughes <doug at will.to>
> Datum: 2017-09-05 14:28
> ?rende: Re: [OmniOS-discuss] email notifications from SMF and FMA
>
> - wouldn't it be easier to use your local MTA (postfix, sendmail) to do this? Then SMF just has to execute /bin/mail or equivalent and your MTA can act as an -
> - outbound relay agent.
>
>
>
>
>
> Yes, of coarse, but this is a SAN array controller, and if it is something I want to avoid, it is to run unnecessary services on a machine like that. I want to keep it as clean as possible.
>
> By the way, do someone know what SMF is using for sending mail? Is it good ol' mailx, or...?
>
> If I knew where to look, I'm sure I could fix it, since I know how to send mail with mailx to remote hosts.
>
>
>
> BR Johan
>
>
>
>
>
>
>
> Sent from my android device.
>
> -----Original Message-----
> From: Johan Kragsterman <johan.kragsterman at capvert.se>
> To: omnios-discuss <omnios-discuss at lists.omniti.com>
> Sent: Tue, 05 Sep 2017 3:44
> Subject: [OmniOS-discuss] email notifications from SMF and FMA
>
> Hi!
>
>
> I need to send mails from SMF to a remote SMTP server, and want/need to include auth credentials.
>
> I know how to use the SMF service to send the mail.
>
> I know how to get the user/passwd into encode_base64, but how do I pass it on to the mail? As well as the port. I guess the port would look like this: root at mailserver.com:465, would it?
>
> But the rest...?
>
> I've seen Peter Tribble been blogging about these features, but not exactly how it would look like to pass on the credentials to your mail.
>
>
> Best regards from/Med v?nliga h?lsningar fr?n
>
> Johan Kragsterman
>
> Capvert
>
>
>







From rjahnel at ellipseinc.com  Tue Sep  5 15:03:33 2017
From: rjahnel at ellipseinc.com (Richard Jahnel)
Date: Tue, 5 Sep 2017 15:03:33 +0000
Subject: [OmniOS-discuss] ZFS cache device allocation a free seem
 strange on my OmniOS r151022 system.
In-Reply-To: <59A9E767.1030608@btconnect.com>
References: <59A9E767.1030608@btconnect.com>
Message-ID: <65DC5816D4BEE043885A89FD54E273FC781396AF@MAIL101.Ellipseinc.com>

Yes, this issue has been lurking around since they made the arcs compressible. I really wish they would revert that change. There have been at least 3 illumos issues filed for this and two fixes attempted. So far it still persist. For now I have my cache drives removed from my production pools.

From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Richard Skelton
Sent: Friday, September 1, 2017 6:04 PM
To: omniOS-discuss <omnios-discuss at lists.omniti.com>
Subject: [OmniOS-discuss] ZFS cache device allocation a free seem strange on my OmniOS r151022 system.

Hi,
The cache allocation a free seem strange on my OmniOS r151022 system.
The cache device is an Intel SSD DC P3700 Series 800GB, 1/2 Height PCIe 3.0, 20nm, MLC
root at omnios:/scratch# zpool list -v scratch
NAME                         SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
scratch                     2.66T  1.94T   738G         -    17%    72%  1.00x  ONLINE  -
  mirror                     544G   395G   149G         -    16%    72%
    c7t5000CCA018309809d0       -      -      -         -      -      -
    c18t5000CCA038718395d0      -      -      -         -      -      -
  mirror                     544G   396G   148G         -    15%    72%
    c14t5000CCA038710745d0      -      -      -         -      -      -
    c11t5000CCA038721F5Dd0      -      -      -         -      -      -
  mirror                     544G   396G   148G         -    18%    72%
    c12t5000CCA03871048Dd0      -      -      -         -      -      -
    c13t5000CCA0387102E1d0      -      -      -         -      -      -
  mirror                     544G   398G   146G         -    19%    73%
    c15t5000CCA03871058Dd0      -      -      -         -      -      -
    c16t5000CCA0387224EDd0      -      -      -         -      -      -
  mirror                     544G   396G   148G         -    17%    72%
    c17t5000CCA038717F65d0      -      -      -         -      -      -
    c10t5000CCA038717AC9d0      -      -      -         -      -      -
log                             -      -      -         -      -      -
  c6t0d0                    3.84G      0  3.84G         -     0%     0%
cache                           -      -      -         -      -      -
  c3t1d0                     745G  1.07T  16.0E         -     0%   147%
spare                           -      -      -         -      -      -
  c9t5000CCA038717B2Dd0         -      -      -         -      -      -
  c8t5000CCA0387104B1d0         -      -      -         -      -      -
root at omnios:/scratch# format </dev/null
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c2t4d0 <Samsung-SSD 850 PRO 256GB-EXM02B6Q-238.47GB>
          /pci at 0,0/pci8086,35c5 at 1f,2/disk at 4,0
       1. c2t5d0 <Samsung-SSD 850 PRO 256GB-EXM02B6Q-238.47GB>
          /pci at 0,0/pci8086,35c5 at 1f,2/disk at 5,0
       2. c3t1d0 <INTEL-SSDPEDMD800G4-8DV10171-745.21GB>
          /pci at 75,0/pci8086,6f0a at 3,2/pci8086,3702 at 0/blkdev at 1,0
       3. c6t0d0 <DDRDRIVE-X1-0078-3.87GB>
          /pci at 75,0/pci8086,6f06 at 2,2/pci19e3,81 at 0/sd at 0,0


Cheers
Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170905/96d99d56/attachment.html>

From matjaz85 at gmail.com  Tue Sep  5 17:21:10 2017
From: matjaz85 at gmail.com (=?utf-8?B?TWF0amHFviBN?=)
Date: Tue, 5 Sep 2017 19:21:10 +0200
Subject: [OmniOS-discuss] email notifications from SMF and FMA
In-Reply-To: <59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost>
References: <OF36592AB3.41A49AC8-ONC1258192.002925FC-C1258192.00292B60@inse.com>
	<59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost>
Message-ID: <B25BFBD2-0999-42BD-A67F-44A27C383DB4@gmail.com>

The easiest way to send an email from CLI or crontab I found is Heirloom mailx. You can find it here:
http://heirloom.sourceforge.net/mailx.html <http://heirloom.sourceforge.net/mailx.html>

You will need to make/make install yourself.


And then you can use info on how to set it up from here:
https://www.systutorials.com/1411/sending-email-from-mailx-command-in-linux-using-gmails-smtp/ <https://www.systutorials.com/1411/sending-email-from-mailx-command-in-linux-using-gmails-smtp/>

BR, Matjaz


> On 5. sep. 2017, at 14:28, Doug Hughes <doug at will.to> wrote:
> 
> wouldn't it be easier to use your local MTA (postfix, sendmail) to do this? Then SMF just has to execute /bin/mail or equivalent and your MTA can act as an outbound relay agent.
> 
> Sent from my android device.
> 
> -----Original Message-----
> From: Johan Kragsterman <johan.kragsterman at capvert.se>
> To: omnios-discuss <omnios-discuss at lists.omniti.com>
> Sent: Tue, 05 Sep 2017 3:44
> Subject: [OmniOS-discuss] email notifications from SMF and FMA
> 
> Hi!
> 
> 
> I need to send mails from SMF to a remote SMTP server, and want/need to include auth credentials.
> 
> I know how to use the SMF service to send the mail.
> 
> I know how to get the user/passwd into encode_base64, but how do I pass it on to the mail? As well as the port. I guess the port would look like this: root at mailserver.com <mailto:root at mailserver.com>:465, would it?
> 
> But the rest...?
> 
> I've seen Peter Tribble been blogging about these features, but not exactly how it would look like to pass on the credentials to your mail.
> 
> 
> Best regards from/Med v?nliga h?lsningar fr?n
> 
> Johan Kragsterman
> 
> Capvert
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170905/6d883be4/attachment-0001.html>

From johan.kragsterman at capvert.se  Tue Sep  5 17:30:02 2017
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Tue, 5 Sep 2017 19:30:02 +0200
Subject: [OmniOS-discuss] Ang: Re:  email notifications from SMF and FMA
In-Reply-To: <B25BFBD2-0999-42BD-A67F-44A27C383DB4@gmail.com>
References: <B25BFBD2-0999-42BD-A67F-44A27C383DB4@gmail.com>,
	<OF36592AB3.41A49AC8-ONC1258192.002925FC-C1258192.00292B60@inse.com>
	<59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost>
Message-ID: <OF463953E3.473FBDB4-ONC1258192.0060222F-C1258192.00602230@inse.com>

Hi!


Thanks, Matjaz, but I wanted to use the SMF service: svc:/system/fm/notify-params:default. I just wondered how I would configure the mail address(in SMF) to be able to send to remote mail host with auth.

BR Johan




Best regards from/Med v?nliga h?lsningar fr?n

Johan Kragsterman

Capvert


-----Matja&#382; M <matjaz85 at gmail.com> skrev: -----
Till: Doug Hughes <doug at will.to>
Fr?n: Matja&#382; M <matjaz85 at gmail.com>
Datum: 2017-09-05 19:21
Kopia: omnios-discuss <omnios-discuss at lists.omniti.com>, Johan Kragsterman <johan.kragsterman at capvert.se>
?rende: Re: [OmniOS-discuss] email notifications from SMF and FMA

The easiest way to send an email from CLI or crontab I found is Heirloom mailx. You can find it here:
http://heirloom.sourceforge.net/mailx.html

You will need to make/make install yourself.


And then you can use info on how to set it up from here:
https://www.systutorials.com/1411/sending-email-from-mailx-command-in-linux-using-gmails-smtp/

BR, Matjaz


On 5. sep. 2017, at 14:28, Doug Hughes <doug at will.to> wrote:

wouldn't it be easier to use your local MTA (postfix, sendmail) to do this? Then SMF just has to execute /bin/mail or equivalent and your MTA can act as an outbound relay agent.

Sent from my android device.

-----Original Message-----
From: Johan Kragsterman <johan.kragsterman at capvert.se>
To: omnios-discuss <omnios-discuss at lists.omniti.com>
Sent: Tue, 05 Sep 2017 3:44
Subject: [OmniOS-discuss] email notifications from SMF and FMA

Hi!


I need to send mails from SMF to a remote SMTP server, and want/need to include auth credentials.

I know how to use the SMF service to send the mail.

I know how to get the user/passwd into encode_base64, but how do I pass it on to the mail? As well as the port. I guess the port would look like this: root at mailserver.com:465, would it?

But the rest...?

I've seen Peter Tribble been blogging about these features, but not exactly how it would look like to pass on the credentials to your mail.


Best regards from/Med v?nliga h?lsningar fr?n

Johan Kragsterman

Capvert
_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss





From jimklimov at cos.ru  Tue Sep  5 17:39:28 2017
From: jimklimov at cos.ru (Jim Klimov)
Date: Tue, 05 Sep 2017 17:39:28 +0000
Subject: [OmniOS-discuss] email notifications from SMF and FMA
In-Reply-To: <59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost>
References: <OF36592AB3.41A49AC8-ONC1258192.002925FC-C1258192.00292B60@inse.com>
	<59300424-f56d-429e-9f43-a0e319d9802e.maildroid@localhost>
Message-ID: <7211FA9F-2A6A-4F10-A138-4DF774C5B5CF@cos.ru>

On September 5, 2017 2:28:19 PM GMT+02:00, Doug Hughes <doug at will.to> wrote:
>wouldn't it be easier to use your local MTA (postfix, sendmail) to do
>this? Then SMF just has to execute /bin/mail or equivalent and your MTA
>can act as an outbound relay agent.
>
>Sent from my android device.
>
>-----Original Message-----
>From: Johan Kragsterman <johan.kragsterman at capvert.se>
>To: omnios-discuss <omnios-discuss at lists.omniti.com>
>Sent: Tue, 05 Sep 2017 3:44
>Subject: [OmniOS-discuss] email notifications from SMF and FMA
>
>Hi!
>
>
>I need to send mails from SMF to a remote SMTP server, and want/need to
>include auth credentials.
>
>
>I know how to use the SMF service to send the mail.
>
>
>I know how to get the user/passwd into encode_base64, but how do I pass
>it on to the mail? As well as the port. I guess the port would look
>like this: root at mailserver.com:465, would it?
>
>
>But the rest...?
>
>
>I've seen Peter Tribble been blogging about these features, but not
>exactly how it would look like to pass on the credentials to your mail.
>
>
>Best regards from/Med v?nliga h?lsningar fr?n
>
>Johan Kragsterman
>
>
>Capvert

Not sure: you're most interested in such notifications when something has failed. This state can preclude running an external submission program on the local system, or having it queue-dequeue your message.

But generally (e.g. to forward user mails or logs/events summaries posted from app servers and other systems especially in highly volatile development setups where connectivity and remote services are not 100% granted) it is often convenient to have dumb programs post to localhost:25 and a real MTA/MSA repost that - often to a common org mail relay that is properly known in worldwide DNS and has secure setup to pass offsite recipients' antispams.

Jim
--
Typos courtesy of K-9 Mail on my Android

From stephan.budach at jvm.de  Thu Sep  7 08:33:54 2017
From: stephan.budach at jvm.de (Stephan Budach)
Date: Thu, 7 Sep 2017 10:33:54 +0200 (CEST)
Subject: [OmniOS-discuss] COMSTAR and blocksizes
In-Reply-To: <1413021884.656.1504772876331.JavaMail.stephan.budach@stephanbudach.local>
Message-ID: <1804012031.676.1504773331844.JavaMail.stephan.budach@stephanbudach.local>


Hi, 


I am having trouble getting an issue sorted out, where omniOS 151020 complaints about mismatched blocksizes on some COMSTAR iSCSI LUNS, like this: 



Sep 7 08:52:07 zfsha02gh79 104 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! 
Sep 7 08:52:07 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94): 
Sep 7 08:52:07 zfsha02gh79 79 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! 
Sep 7 08:52:16 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3132 (sd88): 
Sep 7 08:52:16 zfsha02gh79 20 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! 
Sep 7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd110): 
Sep 7 08:52:17 zfsha02gh79 1 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! 
Sep 7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94): 
Sep 7 08:52:17 zfsha02gh79 24 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low! 


These COMSTAR LUNs are configured to export a blocksize 8k like this: 



LU Name: 600144F0564D504F4F4C3037534C3132 
Operational Status: Online 
Provider Name : sbd 
Alias : nfsvmpool07Slot12 
View Entry Count : 1 
Data File : /dev/rdsk/c3t50015178F364A264d0p1 
Meta File : not set 
Size : 200042414080 
Block Size : 8192 
Management URL : not set 
Vendor ID : SUN 
Product ID : COMSTAR 
Serial Num : not set 
Write Protect : Disabled 
Writeback Cache : Enabled 
Access State : Active 


Now, the system seems to recognize the 8k, but for whatever reason, doesn't adjust the block size accordingly. It does that for the 4k LUNs, however, so I am unsure, on how to tackle this? Any tipp, anyone could share? 


Thanks, 
Stephan 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170907/39bdf414/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5546 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170907/39bdf414/attachment-0001.bin>

From johan.kragsterman at capvert.se  Thu Sep  7 10:08:45 2017
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Thu, 7 Sep 2017 12:08:45 +0200
Subject: [OmniOS-discuss] Ang:  COMSTAR and blocksizes
In-Reply-To: <1804012031.676.1504773331844.JavaMail.stephan.budach@stephanbudach.local>
References: <1804012031.676.1504773331844.JavaMail.stephan.budach@stephanbudach.local>
Message-ID: <OF9C7A176C.44C956B4-ONC1258194.0037BB9C-C1258194.0037BB9D@inse.com>


Hi!

-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev: -----
Till: omnios-discuss at lists.omniti.com
Fr?n: Stephan Budach 
S?nt av: "OmniOS-discuss" 
Datum: 2017-09-07 10:38
?rende: [OmniOS-discuss] COMSTAR and blocksizes

Hi,

I am having trouble getting an issue sorted out, where omniOS 151020 complaints about mismatched blocksizes on some COMSTAR iSCSI LUNS, like this:

Sep  7 08:52:07 zfsha02gh79     104 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low!
Sep  7 08:52:07 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94):
Sep  7 08:52:07 zfsha02gh79     79 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low!
Sep  7 08:52:16 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3132 (sd88):
Sep  7 08:52:16 zfsha02gh79     20 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low!
Sep  7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd110):
Sep  7 08:52:17 zfsha02gh79     1 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low!
Sep  7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94):
Sep  7 08:52:17 zfsha02gh79     24 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low!

These COMSTAR LUNs are configured to export a blocksize 8k like this:

LU Name: 600144F0564D504F4F4C3037534C3132
    Operational Status: Online
    Provider Name     : sbd
    Alias             : nfsvmpool07Slot12
    View Entry Count  : 1
    Data File         : /dev/rdsk/c3t50015178F364A264d0p1
    Meta File         : not set
    Size              : 200042414080
    Block Size        : 8192
    Management URL    : not set
    Vendor ID         : SUN
    Product ID        : COMSTAR
    Serial Num        : not set
    Write Protect     : Disabled
    Writeback Cache   : Enabled
    Access State      : Active

Now, the system seems to recognize the 8k, but for whatever reason, doesn't adjust the block size accordingly. It does that for the 4k LUNs, however, so I am unsure, on how to tackle this? Any tipp, anyone could share?

Thanks,
Stephan






What's in the bottom here?

Data File         : /dev/rdsk/c3t50015178F364A264d0p1

It dosesn't look like a zvol?

/Johan









_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss




From stephan.budach at jvm.de  Thu Sep  7 10:22:06 2017
From: stephan.budach at jvm.de (Stephan Budach)
Date: Thu, 7 Sep 2017 12:22:06 +0200 (CEST)
Subject: [OmniOS-discuss] Ang:  COMSTAR and blocksizes
In-Reply-To: <OF9C7A176C.44C956B4-ONC1258194.0037BB9C-C1258194.0037BB9D@inse.com>
References: <1804012031.676.1504773331844.JavaMail.stephan.budach@stephanbudach.local>
	<OF9C7A176C.44C956B4-ONC1258194.0037BB9C-C1258194.0037BB9D@inse.com>
Message-ID: <857477775.718.1504779825689.JavaMail.stephan.budach@stephanbudach.local>



----- Urspr?ngliche Mail -----
> Von: "Johan Kragsterman" <johan.kragsterman at capvert.se>
> An: "Stephan Budach" <stephan.budach at jvm.de>
> CC: omnios-discuss at lists.omniti.com
> Gesendet: Donnerstag, 7. September 2017 12:08:45
> Betreff: Ang: [OmniOS-discuss] COMSTAR and blocksizes
> 
> 
> Hi!
> 
> -----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com>
> skrev: -----
> Till: omnios-discuss at lists.omniti.com
> Fr?n: Stephan Budach
> S?nt av: "OmniOS-discuss"
> Datum: 2017-09-07 10:38
> ?rende: [OmniOS-discuss] COMSTAR and blocksizes
> 
> Hi,
> 
> I am having trouble getting an issue sorted out, where omniOS 151020
> complaints about mismatched blocksizes on some COMSTAR iSCSI LUNS,
> like this:
> 
> Sep  7 08:52:07 zfsha02gh79     104 I/O requests are not aligned with
> 8192 disk sector size in 10 seconds. They are handled through Read
> Modify Write but the performance is very low!
> Sep  7 08:52:07 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING:
> /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94):
> Sep  7 08:52:07 zfsha02gh79     79 I/O requests are not aligned with
> 8192 disk sector size in 10 seconds. They are handled through Read
> Modify Write but the performance is very low!
> Sep  7 08:52:16 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING:
> /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3132 (sd88):
> Sep  7 08:52:16 zfsha02gh79     20 I/O requests are not aligned with
> 8192 disk sector size in 10 seconds. They are handled through Read
> Modify Write but the performance is very low!
> Sep  7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING:
> /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd110):
> Sep  7 08:52:17 zfsha02gh79     1 I/O requests are not aligned with
> 8192 disk sector size in 10 seconds. They are handled through Read
> Modify Write but the performance is very low!
> Sep  7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING:
> /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94):
> Sep  7 08:52:17 zfsha02gh79     24 I/O requests are not aligned with
> 8192 disk sector size in 10 seconds. They are handled through Read
> Modify Write but the performance is very low!
> 
> These COMSTAR LUNs are configured to export a blocksize 8k like this:
> 
> LU Name: 600144F0564D504F4F4C3037534C3132
>     Operational Status: Online
>     Provider Name     : sbd
>     Alias             : nfsvmpool07Slot12
>     View Entry Count  : 1
>     Data File         : /dev/rdsk/c3t50015178F364A264d0p1
>     Meta File         : not set
>     Size              : 200042414080
>     Block Size        : 8192
>     Management URL    : not set
>     Vendor ID         : SUN
>     Product ID        : COMSTAR
>     Serial Num        : not set
>     Write Protect     : Disabled
>     Writeback Cache   : Enabled
>     Access State      : Active
> 
> Now, the system seems to recognize the 8k, but for whatever reason,
> doesn't adjust the block size accordingly. It does that for the 4k
> LUNs, however, so I am unsure, on how to tackle this? Any tipp,
> anyone could share?
> 
> Thanks,
> Stephan
> 
> 
> 
> 
> 
> 
> What's in the bottom here?
> 
> Data File         : /dev/rdsk/c3t50015178F364A264d0p1
> 
> It dosesn't look like a zvol?
> 
> /Johan

It isn't one - it's a raw partition, an Intel SSD in this case.

Cheers,
Stephan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5546 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170907/2c6229fc/attachment.bin>

From sylvain at yesik.it  Fri Sep  8 12:38:17 2017
From: sylvain at yesik.it (Sylvain Leroux)
Date: Fri, 8 Sep 2017 14:38:17 +0200
Subject: [OmniOS-discuss] Status of the OmniOS project
Message-ID: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it>

Hi everyone,

After Oracle has laid off the core Solaris developers, I'm writing an
article for https://itsfoss.com about the various
OpenSolaris-based/Illumos-based distributions.

I've already heard about OmniOS previously and wanted to mention it as
the de-facto Illumos server distribution.
However while doing some researches, I found that article:

https://www.theregister.co.uk/2017/04/25/oracle_free_solaris_project_stops/

Here is the headline:
"""
Development of OmniOS ? an Oracle-free open-source variant of Solaris ?
is being killed after five years of work.
"""

The author then reports several quotes from OmniTI chief executive
Robert Treat to support that assertion.
On the other hand, by what I saw on the mailing list the project _seems_
still to be active.


Could you give more informations about the current state of the project?


Thanks in advance for your time,
- Sylvain

-- -- Sylvain Leroux -- Yes, I Know IT ! -- sylvain at yesik.it --
https://yesik.it -- -- Follow me on Twitter! --
https://twitter.com/Yes_I_Know_IT


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170908/04109a69/attachment.bin>

From sylvain at yesik.it  Fri Sep  8 12:50:27 2017
From: sylvain at yesik.it (Sylvain Leroux)
Date: Fri, 8 Sep 2017 14:50:27 +0200
Subject: [OmniOS-discuss] Status of the OmniOS project
In-Reply-To: <B9B4644A-81DA-4E6A-8985-B1E989A9BA77@icloud.com>
References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it>
	<B9B4644A-81DA-4E6A-8985-B1E989A9BA77@icloud.com>
Message-ID: <b0b866d6-e662-5ad5-7ff4-1eb17ed4095a@yesik.it>

Thank you Robert for the link.

I wasn't aware at all of the "Community Edition". It is unfortunate it
does not yet appear in the first results when searching on Google for
"OmniOS". Hopefully this will change soon.

Cheers :)

On 09/08/2017 02:44 PM, DELROY A BLAKE wrote:
>
>
> http://www.omniosce.org/about.html
>
> The community is alive and well . 
>
>
>> On Sep 8, 2017, at 8:38 AM, Sylvain Leroux <sylvain at yesik.it
>> <mailto:sylvain at yesik.it>> wrote:
>>
>> Hi everyone,
>>
>> After Oracle has laid off the core Solaris developers, I'm writing an
>> article for https://itsfoss.com about the various
>> OpenSolaris-based/Illumos-based distributions.
>>
>> I've already heard about OmniOS previously and wanted to mention it as
>> the de-facto Illumos server distribution.
>> However while doing some researches, I found that article:
>>
>> https://www.theregister.co.uk/2017/04/25/oracle_free_solaris_project_stops/
>>
>> Here is the headline:
>> """
>> Development of OmniOS ? an Oracle-free open-source variant of Solaris ?
>> is being killed after five years of work.
>> """
>>
>> The author then reports several quotes from OmniTI chief executive
>> Robert Treat to support that assertion.
>> On the other hand, by what I saw on the mailing list the project _seems_
>> still to be active.
>>
>>
>> Could you give more informations about the current state of the project?
>>
>>
>> Thanks in advance for your time,
>> - Sylvain
>>
>> -- -- Sylvain Leroux -- Yes, I Know IT ! -- sylvain at yesik.it --
>> https://yesik.it -- -- Follow me on Twitter! --
>> https://twitter.com/Yes_I_Know_IT
>>
>>
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>

-- 
-- Sylvain Leroux -- Yes, I Know IT !
-- sylvain at yesik.it
-- https://yesik.it
--
-- Follow me on LinkedIn!
-- https://fr.linkedin.com/in/sylvain-leroux-knows-it

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170908/3e5718fa/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170908/3e5718fa/attachment-0001.bin>

From jesus at omniti.com  Fri Sep  8 12:53:45 2017
From: jesus at omniti.com (Theo Schlossnagle)
Date: Fri, 8 Sep 2017 08:53:45 -0400
Subject: [OmniOS-discuss] Status of the OmniOS project
In-Reply-To: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it>
References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it>
Message-ID: <CACLsApt=UB4Fh4y_8tJ_6MU3j4eWGb0OuQ6=6--2dyjsoJR3Ww@mail.gmail.com>

My two cents on this.

Sensational title and misleading.  If you read the archives you'll see that
OmniTI forced themselves out of corporate stewardship with the desire for
the OmniOS product to be community developed, managed and governed.  There
was never any intention (or outcome) that resembles killing.  One of the
advantages of this move is that it prevents OmniOS from meeting any of the
Oracle-induced outcomes brutally met by Solaris.

The community has taken over... and releases are even on schedule.

http://www.omniosce.org/

As far as developer activity, the OmniOS project has wider participation
and more activity now than it did prior to the transition.

If you're looking for an "It's dead Jim" article, you might not get what
you're looking for.

Best regards,

Theo

On Fri, Sep 8, 2017 at 8:38 AM, Sylvain Leroux <sylvain at yesik.it> wrote:

> Hi everyone,
>
> After Oracle has laid off the core Solaris developers, I'm writing an
> article for https://itsfoss.com about the various
> OpenSolaris-based/Illumos-based distributions.
>
> I've already heard about OmniOS previously and wanted to mention it as
> the de-facto Illumos server distribution.
> However while doing some researches, I found that article:
>
> https://www.theregister.co.uk/2017/04/25/oracle_free_
> solaris_project_stops/
>
> Here is the headline:
> """
> Development of OmniOS ? an Oracle-free open-source variant of Solaris ?
> is being killed after five years of work.
> """
>
> The author then reports several quotes from OmniTI chief executive
> Robert Treat to support that assertion.
> On the other hand, by what I saw on the mailing list the project _seems_
> still to be active.
>
>
> Could you give more informations about the current state of the project?
>
>
> Thanks in advance for your time,
> - Sylvain
>
> -- -- Sylvain Leroux -- Yes, I Know IT ! -- sylvain at yesik.it --
> https://yesik.it -- -- Follow me on Twitter! --
> https://twitter.com/Yes_I_Know_IT
>
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>


-- 

Theo Schlossnagle

http://omniti.com/is/theo-schlossnagle
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170908/30837e07/attachment.html>

From skiselkov.ml at gmail.com  Fri Sep  8 12:54:25 2017
From: skiselkov.ml at gmail.com (Saso Kiselkov)
Date: Fri, 8 Sep 2017 14:54:25 +0200
Subject: [OmniOS-discuss] Status of the OmniOS project
In-Reply-To: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it>
References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it>
Message-ID: <8decfd16-7b34-d227-d0aa-4ffc7dd3691d@gmail.com>

On 9/8/17 2:38 PM, Sylvain Leroux wrote:
> Hi everyone,
> 
> After Oracle has laid off the core Solaris developers, I'm writing an
> article for https://itsfoss.com about the various
> OpenSolaris-based/Illumos-based distributions.
> 
> I've already heard about OmniOS previously and wanted to mention it as
> the de-facto Illumos server distribution.
> However while doing some researches, I found that article:
> 
> https://www.theregister.co.uk/2017/04/25/oracle_free_solaris_project_stops/
> 
> Here is the headline:
> """
> Development of OmniOS ? an Oracle-free open-source variant of Solaris ?
> is being killed after five years of work.
> """
> 
> The author then reports several quotes from OmniTI chief executive
> Robert Treat to support that assertion.
> On the other hand, by what I saw on the mailing list the project _seems_
> still to be active.
> 
> 
> Could you give more informations about the current state of the project?
> 
> 
> Thanks in advance for your time,

While "OmniOS" is no longer being developed by OmniTI, its development
has been taken over by the community (at OmniTI's request, actually) and
can be found under http://www.omniosce.org with new releases still ongoing.

Cheers,
-- 
Saso

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170908/aecee446/attachment.bin>

From sylvain at yesik.it  Fri Sep  8 13:03:58 2017
From: sylvain at yesik.it (Sylvain Leroux)
Date: Fri, 8 Sep 2017 15:03:58 +0200
Subject: [OmniOS-discuss] Status of the OmniOS project
In-Reply-To: <CACLsApt=UB4Fh4y_8tJ_6MU3j4eWGb0OuQ6=6--2dyjsoJR3Ww@mail.gmail.com>
References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it>
	<CACLsApt=UB4Fh4y_8tJ_6MU3j4eWGb0OuQ6=6--2dyjsoJR3Ww@mail.gmail.com>
Message-ID: <275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it>

Some comments inline:


On 09/08/2017 02:53 PM, Theo Schlossnagle wrote:
> The community has taken over... and releases are even on schedule.
>
> http://www.omniosce.org/
>
> As far as developer activity, the OmniOS project has wider
> participation and more activity now than it did prior to the transition.
Thank you Theo: as I said above to Delroy A Blake, I wasn't aware of
OmniOS "Community Edition". And this project certainly deserves being
advertised. So if I may add my own little stone to the wall for that, I
will be glad to do it.


>
> If you're looking for an "It's dead Jim" article, you might not get
> what you're looking for.
As a matter of fact, I was looking to the exact opposite: I I used a lot
Solaris in the 90s and early 2000s. And I was strongly disappointed to
see what happened to that great OS years after years under the umbrella
of Oracle in the 2010s.

My goal was to say Solaris was still alive -- and there are actively
maintained projects and distributions that have taken up the succession.

Regards,
- Sylvain

-- 
-- Sylvain Leroux -- Yes, I Know IT !
-- sylvain at yesik.it
-- https://yesik.it
--
-- Follow me on LinkedIn!
-- https://fr.linkedin.com/in/sylvain-leroux-knows-it




-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170908/74a50c15/attachment.bin>

From chip at innovates.com  Fri Sep  8 13:43:15 2017
From: chip at innovates.com (Schweiss, Chip)
Date: Fri, 8 Sep 2017 08:43:15 -0500
Subject: [OmniOS-discuss] [zfs] SAS 9305-16e HBA support in Illumos
In-Reply-To: <CALeZrrR447M8EHK1bo_phjZE_MVSUuGFAGBF9y=Y2GhedJMxKg@mail.gmail.com>
References: <CALeZrrTUO+mQeH9+nAvmhDvYA6-GgKRxM4vpdPa6JVoGG8rBKw@mail.gmail.com>
	<4FB32C07-51A0-43C1-9E69-69C1876B13EA@elemental.org>
	<19DC39EC-F31F-4C5E-867A-A164E044565B@elemental.org>
	<78B3A268-125F-41ED-9898-E1DC42D195F2@joyent.com>
	<CALeZrrT+L7d7D=TO-WwkJ1a-xqMV7TmSk6KCzLi2-ivfHMao_Q@mail.gmail.com>
	<CALeZrrSNa7cve=N4peusXPNn-G5ubuJd_dgqbo01t_6kpWim4g@mail.gmail.com>
	<328231EA-C034-4337-BB37-F3FF2AA1C2D7@elemental.org>
	<CALeZrrR447M8EHK1bo_phjZE_MVSUuGFAGBF9y=Y2GhedJMxKg@mail.gmail.com>
Message-ID: <CALeZrrQ-7NuRzQ2E-p=nzETHptQTBgLP8b4UHCd0Zkj=Z1ep1w@mail.gmail.com>

Now that I'm back to working on this.   The only way I could get the
firmware updated was booting into the UEFI shell.   A bit of a pain but it
worked.

Unfortunately, it has not changed the behavior of the HBA.

Where do I go from here?    Any hope of getting this working on OmniOS?

-Chip

On Thu, Aug 31, 2017 at 9:53 AM, Schweiss, Chip <chip at innovates.com> wrote:

> This server will be serving NFS for vSphere.  It is running OmniOS CE,
> nothing VMware.
>
> I'm working on flashing firmware now and will report back any changes.
>
> -Chip
>
> On Thu, Aug 31, 2017 at 9:42 AM, Dale Ghent <daleg at elemental.org> wrote:
>
>>
>> > On Aug 31, 2017, at 9:29 AM, Schweiss, Chip <chip at innovates.com> wrote:
>> >
>> > I've added mpt_sas "pciex1000,c9" to /etc/driver_aliases and rebooted.
>> >
>> > Looks like it's partially working, but it's not fully functional.
>> Service are timing out:
>> >
>> > Here's what I see in /var/adm/messages:
>> >
>> >
>> > Aug 31 08:15:49 vsphere-zfs01 scsi: [ID 107833 kern.warning] WARNING:
>> /pci at 0,0/pci8086,1905 at 1,1/pci1000,3180 at 0 (mpt_sas0):
>> > Aug 31 08:15:49 vsphere-zfs01   MPT Firmware Fault, code: 2667
>> > Aug 31 08:15:49 vsphere-zfs01 scsi: [ID 107833 kern.warning] WARNING:
>> /pci at 0,0/pci8086,1905 at 1,1/pci1000,3180 at 0 (mpt_sas0):
>>
>> The driver is reporting that the MPT IOC (IO Controller) is reporting a
>> fault. It's just reading this condition off the controller chip itself, and
>> unfortunately there doesn't seem to be a handy reference published by
>> LSI/Avago regarding what 2667h actually means.
>>
>> However I note from your machine's hostname that this is perhaps a ESI
>> guest that is being given the HBA in passthrough mode? It would seem that
>> someone else has encountered a similar issue as yourself in this case, with
>> the same MPT fault code, but on Linux running Proxmox. According to this
>> forum thread, they ended up flashing the firmware on the card to something
>> newer and the problem went away:
>>
>> https://forum.proxmox.com/threads/pci-passthrough.16483/
>>
>> I would suggest Tim's approach and flashing your card up to the newest IT
>> (not IR) firmware.
>>
>> /dale
>>
>>
>> ------------------------------------------
>> illumos-zfs
>> Archives: https://illumos.topicbox.com/groups/zfs/discussions/T372d7dd
>> d75316296-Mb0dd6c92e5393440a8b0c8fb
>> Powered by Topicbox: https://topicbox.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170908/3da02b84/attachment-0001.html>

From illumos at cucumber.demon.co.uk  Fri Sep  8 13:52:24 2017
From: illumos at cucumber.demon.co.uk (Andrew Gabriel)
Date: Fri, 8 Sep 2017 14:52:24 +0100
Subject: [OmniOS-discuss] Status of the OmniOS project
In-Reply-To: <275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it>
References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it>
	<CACLsApt=UB4Fh4y_8tJ_6MU3j4eWGb0OuQ6=6--2dyjsoJR3Ww@mail.gmail.com>
	<275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it>
Message-ID: <4032b9e4-a9e1-7f20-553a-4fc7a700c8c2@cucumber.demon.co.uk>

On 08/09/2017 14:03, Sylvain Leroux wrote:
> My goal was to say Solaris was still alive -- and there are actively 
> maintained projects and distributions that have taken up the succession.

Do check out Bryan Cantrill's blog on this, if you haven't seen it yet.
http://dtrace.org/blogs/bmc/2017/09/04/the-sudden-death-and-eternal-life-of-solaris/

-- 
Andrew

From jimklimov at cos.ru  Fri Sep  8 14:04:11 2017
From: jimklimov at cos.ru (Jim Klimov)
Date: Fri, 08 Sep 2017 14:04:11 +0000
Subject: [OmniOS-discuss] Status of the OmniOS project
In-Reply-To: <275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it>
References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it>
	<CACLsApt=UB4Fh4y_8tJ_6MU3j4eWGb0OuQ6=6--2dyjsoJR3Ww@mail.gmail.com>
	<275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it>
Message-ID: <D001CE67-B140-4FB5-B917-729B7EA463C7@cos.ru>

On September 8, 2017 3:03:58 PM GMT+02:00, Sylvain Leroux <sylvain at yesik.it> wrote:
>Some comments inline:
>
>
>On 09/08/2017 02:53 PM, Theo Schlossnagle wrote:
>> The community has taken over... and releases are even on schedule.
>>
>> http://www.omniosce.org/
>>
>> As far as developer activity, the OmniOS project has wider
>> participation and more activity now than it did prior to the
>transition.
>Thank you Theo: as I said above to Delroy A Blake, I wasn't aware of
>OmniOS "Community Edition". And this project certainly deserves being
>advertised. So if I may add my own little stone to the wall for that, I
>will be glad to do it.
>
>
>>
>> If you're looking for an "It's dead Jim" article, you might not get
>> what you're looking for.
>As a matter of fact, I was looking to the exact opposite: I I used a
>lot
>Solaris in the 90s and early 2000s. And I was strongly disappointed to
>see what happened to that great OS years after years under the umbrella
>of Oracle in the 2010s.
>
>My goal was to say Solaris was still alive -- and there are actively
>maintained projects and distributions that have taken up the
>succession.
>
>Regards,
>- Sylvain

>My goal was to say Solaris was still alive -- and there are actively
>maintained projects and distributions that have taken up the
>succession.

To that effect, OpenIndiana Hipster is another major general-purpose illumos distribution that's being actively debeloped too, aimed at both desktop and server, with a lot of software and GUI desktop pre-packaged in its standard repository - in contrast with OmniOS that aims to be the minimal foundation of an OS.

Also there are many smaller projects, outlined on wiki.illumos.org as well as unknown cores of storage, networking and other appliances.

Hope this helps,
Jim
--
Typos courtesy of K-9 Mail on my Android

From david.ledger at ivdcs.co.uk  Fri Sep  8 14:21:30 2017
From: david.ledger at ivdcs.co.uk (David Ledger)
Date: Fri, 08 Sep 2017 15:21:30 +0100
Subject: [OmniOS-discuss] networking between zones
Message-ID: <A0F07B66-AEFD-4C33-936A-30A73EF9A65A@ivdcs.co.uk>


We have several zones running on an OmniOS server that use exclusive 
addressing but in the same subnet at the global zone. These work fine.

We now need to set up a couple of zones that have their own subnet, but 
talk to the outside world through the global zone. These will need to be 
network isolated from the existing zones and with access controlled, 
presumably by ipf/ipnat filtering done in the global zone. I?m having 
difficulty setting this up. It is readily admitted on the ?net that 
Solaris network config is different to anything else, and that it has 
moved on in stages from the old hosts, hostname etc. files that were so 
easy back in the 80?s.

Could someone please point me to some documentation that is relevant to 
OmniOS. We?re not up to r151022 even yet. I can?t tell from what?s 
out there whether I should be using ipfilter config within svcs, or 
ipf/ipnat type files or what. Some things say I need an etherstub, 
others say I don?t. The <zone>.cfg file gives the zone a vmic, but the 
zone can?t see it, but the zone can use it ???

David the confused.


David Ledger - computers since ?69, Unix since ?83, Web since ?95

From jimklimov at cos.ru  Fri Sep  8 14:18:07 2017
From: jimklimov at cos.ru (Jim Klimov)
Date: Fri, 08 Sep 2017 14:18:07 +0000
Subject: [OmniOS-discuss] Status of the OmniOS project
In-Reply-To: <D001CE67-B140-4FB5-B917-729B7EA463C7@cos.ru>
References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it>
	<CACLsApt=UB4Fh4y_8tJ_6MU3j4eWGb0OuQ6=6--2dyjsoJR3Ww@mail.gmail.com>
	<275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it>
	<D001CE67-B140-4FB5-B917-729B7EA463C7@cos.ru>
Message-ID: <21F7B203-B7DD-4390-BE89-4966A2B9F37F@cos.ru>

On September 8, 2017 4:04:11 PM GMT+02:00, Jim Klimov <jimklimov at cos.ru> wrote:
>On September 8, 2017 3:03:58 PM GMT+02:00, Sylvain Leroux
><sylvain at yesik.it> wrote:
>>Some comments inline:
>>
>>
>>On 09/08/2017 02:53 PM, Theo Schlossnagle wrote:
>>> The community has taken over... and releases are even on schedule.
>>>
>>> http://www.omniosce.org/
>>>
>>> As far as developer activity, the OmniOS project has wider
>>> participation and more activity now than it did prior to the
>>transition.
>>Thank you Theo: as I said above to Delroy A Blake, I wasn't aware of
>>OmniOS "Community Edition". And this project certainly deserves being
>>advertised. So if I may add my own little stone to the wall for that,
>I
>>will be glad to do it.
>>
>>
>>>
>>> If you're looking for an "It's dead Jim" article, you might not get
>>> what you're looking for.
>>As a matter of fact, I was looking to the exact opposite: I I used a
>>lot
>>Solaris in the 90s and early 2000s. And I was strongly disappointed to
>>see what happened to that great OS years after years under the
>umbrella
>>of Oracle in the 2010s.
>>
>>My goal was to say Solaris was still alive -- and there are actively
>>maintained projects and distributions that have taken up the
>>succession.
>>
>>Regards,
>>- Sylvain
>
>>My goal was to say Solaris was still alive -- and there are actively
>>maintained projects and distributions that have taken up the
>>succession.
>
>To that effect, OpenIndiana Hipster is another major general-purpose
>illumos distribution that's being actively debeloped too, aimed at both
>desktop and server, with a lot of software and GUI desktop pre-packaged
>in its standard repository - in contrast with OmniOS that aims to be
>the minimal foundation of an OS.
>
>Also there are many smaller projects, outlined on wiki.illumos.org as
>well as unknown cores of storage, networking and other appliances.
>
>Hope this helps,
>Jim
>--
>Typos courtesy of K-9 Mail on my Android

...And for clarification - I spoke of general purpose distros but ended on a note of the hearts of appliances.

In that area of purpose-optimized illumos products there are of course some well-known larger bi-lateral opensource commercial players, such as Joyent (now part of Samsung) aimed at hypervisors and related storage needs, Delphix for databases, Nexenta for storage...

Also, the OpenZFS community codebase is a stripped-down illumos source tree that grows with BSD, Linux and other contributions.

Jim
--
Typos courtesy of K-9 Mail on my Android

From jdg117 at elvis.arl.psu.edu  Fri Sep  8 14:23:07 2017
From: jdg117 at elvis.arl.psu.edu (John D Groenveld)
Date: Fri, 08 Sep 2017 10:23:07 -0400
Subject: [OmniOS-discuss] [zfs] SAS 9305-16e HBA support in Illumos
In-Reply-To: Your message of "Fri, 08 Sep 2017 08:43:15 CDT."
	<CALeZrrQ-7NuRzQ2E-p=nzETHptQTBgLP8b4UHCd0Zkj=Z1ep1w@mail.gmail.com> 
References: <CALeZrrTUO+mQeH9+nAvmhDvYA6-GgKRxM4vpdPa6JVoGG8rBKw@mail.gmail.com>
	<4FB32C07-51A0-43C1-9E69-69C1876B13EA@elemental.org>
	<19DC39EC-F31F-4C5E-867A-A164E044565B@elemental.org>
	<78B3A268-125F-41ED-9898-E1DC42D195F2@joyent.com>
	<CALeZrrT+L7d7D=TO-WwkJ1a-xqMV7TmSk6KCzLi2-ivfHMao_Q@mail.gmail.com>
	<CALeZrrSNa7cve=N4peusXPNn-G5ubuJd_dgqbo01t_6kpWim4g@mail.gmail.com>
	<328231EA-C034-4337-BB37-F3FF2AA1C2D7@elemental.org>
	<CALeZrrR447M8EHK1bo_phjZE_MVSUuGFAGBF9y=Y2GhedJMxKg@mail.gmail.com>
	<CALeZrrQ-7NuRzQ2E-p=nzETHptQTBgLP8b4UHCd0Zkj=Z1ep1w@mail.gmail.com>
Message-ID: <201709081423.v88EN7Xe013815@elvis.arl.psu.edu>

In message <CALeZrrQ-7NuRzQ2E-p=nzETHptQTBgLP8b4UHCd0Zkj=Z1ep1w at mail.gmail.com>
, "Schweiss, Chip" writes:
>Now that I'm back to working on this.   The only way I could get the
>firmware updated was booting into the UEFI shell.   A bit of a pain but it
>worked.

Did you go to P14 or P15?

>Unfortunately, it has not changed the behavior of the HBA.
>
>Where do I go from here?    Any hope of getting this working on OmniOS?

Can you reproduce with FreeBSD?
<URL:https://www.broadcom.com/products/storage/host-bus-adapters/sas-9305-16e#downloads>

John
groenveld at acm.org

From danmcd at kebe.com  Fri Sep  8 14:31:22 2017
From: danmcd at kebe.com (Dan McDonald)
Date: Fri, 8 Sep 2017 10:31:22 -0400
Subject: [OmniOS-discuss] networking between zones
In-Reply-To: <A0F07B66-AEFD-4C33-936A-30A73EF9A65A@ivdcs.co.uk>
References: <A0F07B66-AEFD-4C33-936A-30A73EF9A65A@ivdcs.co.uk>
Message-ID: <20170908143122.GA33110@everywhere.local>

On Fri, Sep 08, 2017 at 03:21:30PM +0100, David Ledger wrote:
> 

<SNIP!>

> We now need to set up a couple of zones that have their own subnet, but talk
> to the outside world through the global zone. These will need to be network
> isolated from the existing zones and with access controlled, presumably by
> ipf/ipnat filtering done in the global zone. I?m having difficulty setting
> this up. It is readily admitted on the ?net that Solaris network config is
> different to anything else, and that it has moved on in stages from the old
> hosts, hostname etc. files that were so easy back in the 80?s.

I think you wish to create an etherstub (in-machine "LAN" as it were).  From
global:

	dladm create-etherstub internal0

And once created, you create vnics attached to that etherstub:

	dladm create-vnic -l internal0 stubnet0
	dladm create-vnic -l internal0 stubnet1
	...

And then you assign the vnics to your "have their own subnet" zones like you
would any other nic.  You will also need your global, or even a dedicated
router zone, attach to both the etherstub and the external network (running
ipf or whatever else).

Does this help?

Dan

From david.ledger at ivdcs.co.uk  Fri Sep  8 14:40:44 2017
From: david.ledger at ivdcs.co.uk (David Ledger)
Date: Fri, 08 Sep 2017 15:40:44 +0100
Subject: [OmniOS-discuss] networking between zones
In-Reply-To: <20170908143122.GA33110@everywhere.local>
References: <A0F07B66-AEFD-4C33-936A-30A73EF9A65A@ivdcs.co.uk>
	<20170908143122.GA33110@everywhere.local>
Message-ID: <370F5B5F-83EB-429A-A747-F0FFD7CC4AE9@ivdcs.co.uk>

On 8 Sep 2017, at 15:31, Dan McDonald wrote:

> On Fri, Sep 08, 2017 at 03:21:30PM +0100, David Ledger wrote:
>>
>
> <SNIP!>
>
>> We now need to set up a couple of zones that have their own subnet, 
>> but talk
>> to the outside world through the global zone. These will need to be 
>> network
>> isolated from the existing zones and with access controlled, 
>> presumably by
>> ipf/ipnat filtering done in the global zone. I?m having difficulty 
>> setting
>> this up. It is readily admitted on the ?net that Solaris network 
>> config is
>> different to anything else, and that it has moved on in stages from 
>> the old
>> hosts, hostname etc. files that were so easy back in the 80?s.
>
> I think you wish to create an etherstub (in-machine "LAN" as it were). 
>  From
> global:
>
> 	dladm create-etherstub internal0
>
> And once created, you create vnics attached to that etherstub:
>
> 	dladm create-vnic -l internal0 stubnet0
> 	dladm create-vnic -l internal0 stubnet1
> 	...
>
> And then you assign the vnics to your "have their own subnet" zones 
> like you
> would any other nic.  You will also need your global, or even a 
> dedicated
> router zone, attach to both the etherstub and the external network 
> (running
> ipf or whatever else).
>
> Does this help?
>
> Dan

Knowing that certainly helps, thanks. Can I also ignore those online 
documents that say it?s all done by configurations within the svcs 
setup and that I need to disable one version of ipfilter and enable 
another?

David

From danmcd at kebe.com  Fri Sep  8 14:54:33 2017
From: danmcd at kebe.com (Dan McDonald)
Date: Fri, 8 Sep 2017 10:54:33 -0400
Subject: [OmniOS-discuss] networking between zones
In-Reply-To: <370F5B5F-83EB-429A-A747-F0FFD7CC4AE9@ivdcs.co.uk>
References: <A0F07B66-AEFD-4C33-936A-30A73EF9A65A@ivdcs.co.uk>
	<20170908143122.GA33110@everywhere.local>
	<370F5B5F-83EB-429A-A747-F0FFD7CC4AE9@ivdcs.co.uk>
Message-ID: <20170908145433.GC33110@everywhere.local>

On Fri, Sep 08, 2017 at 03:40:44PM +0100, David Ledger wrote:
> 
> Knowing that certainly helps, thanks. Can I also ignore those online
> documents that say it?s all done by configurations within the svcs setup and
> that I need to disable one version of ipfilter and enable another?

I use stock ipfilter (ipnat only, to be honest) in its own zone.  I didn't
use SMF magic to configure it, save "svcadm enable ipfilter" once had
/etc/ipf/ipnat.conf where I wanted it.

Dan

From jdg117 at elvis.arl.psu.edu  Fri Sep  8 14:57:08 2017
From: jdg117 at elvis.arl.psu.edu (John D Groenveld)
Date: Fri, 08 Sep 2017 10:57:08 -0400
Subject: [OmniOS-discuss] networking between zones
In-Reply-To: Your message of "Fri, 08 Sep 2017 15:40:44 BST."
	<370F5B5F-83EB-429A-A747-F0FFD7CC4AE9@ivdcs.co.uk> 
References: <A0F07B66-AEFD-4C33-936A-30A73EF9A65A@ivdcs.co.uk>
	<20170908143122.GA33110@everywhere.local>
	<370F5B5F-83EB-429A-A747-F0FFD7CC4AE9@ivdcs.co.uk> 
Message-ID: <201709081457.v88Ev8I3014024@elvis.arl.psu.edu>

In message <370F5B5F-83EB-429A-A747-F0FFD7CC4AE9 at ivdcs.co.uk>, "David Ledger" w
rites:
>Knowing that certainly helps, thanks. Can I also ignore those online 
>documents that say it???s all done by configurations within the svcs 

You assign the VNIC's to your zones via zonecfg(1M).

>setup and that I need to disable one version of ipfilter and enable 
>another?

PF has not yet been ported to illumos.
If you absolutely must move to PF, you'll need to run it under
a VBox or KVM VM.

I use native IPF for my Crossbow network in a box.

John
groenveld at acm.org

From philipcristiano at philipcristiano.com  Fri Sep  8 15:25:07 2017
From: philipcristiano at philipcristiano.com (Philip Cristiano)
Date: Fri, 08 Sep 2017 11:25:07 -0400
Subject: [OmniOS-discuss] networking between zones
In-Reply-To: <201709081457.v88Ev8I3014024@elvis.arl.psu.edu>
References: <A0F07B66-AEFD-4C33-936A-30A73EF9A65A@ivdcs.co.uk>
	<20170908143122.GA33110@everywhere.local>
	<370F5B5F-83EB-429A-A747-F0FFD7CC4AE9@ivdcs.co.uk>
	<201709081457.v88Ev8I3014024@elvis.arl.psu.edu>
Message-ID: <1504884307.139027.1099714504.7EB2D20B@webmail.messagingengine.com>

This helped me quite a bit when trying to do the same thing 

http://www.scalingbits.com/book/export/html/479

-- 
  Philip Cristiano
  philipcristiano at philipcristiano.com

On Fri, Sep 8, 2017, at 10:57 AM, John D Groenveld wrote:
> In message <370F5B5F-83EB-429A-A747-F0FFD7CC4AE9 at ivdcs.co.uk>, "David
> Ledger" w
> rites:
> >Knowing that certainly helps, thanks. Can I also ignore those online 
> >documents that say it?s all done by configurations within the svcs 
> 
> You assign the VNIC's to your zones via zonecfg(1M).
> 
> >setup and that I need to disable one version of ipfilter and enable 
> >another?
> 
> PF has not yet been ported to illumos.
> If you absolutely must move to PF, you'll need to run it under
> a VBox or KVM VM.
> 
> I use native IPF for my Crossbow network in a box.
> 
> John
> groenveld at acm.org
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

From chip at innovates.com  Fri Sep  8 15:38:03 2017
From: chip at innovates.com (Schweiss, Chip)
Date: Fri, 8 Sep 2017 10:38:03 -0500
Subject: [OmniOS-discuss] [zfs] SAS 9305-16e HBA support in Illumos
In-Reply-To: <f6d0efd4-6fd6-a93f-b113-f2ea627272b2@joyent.com>
References: <CALeZrrTUO+mQeH9+nAvmhDvYA6-GgKRxM4vpdPa6JVoGG8rBKw@mail.gmail.com>
	<4FB32C07-51A0-43C1-9E69-69C1876B13EA@elemental.org>
	<19DC39EC-F31F-4C5E-867A-A164E044565B@elemental.org>
	<78B3A268-125F-41ED-9898-E1DC42D195F2@joyent.com>
	<CALeZrrT+L7d7D=TO-WwkJ1a-xqMV7TmSk6KCzLi2-ivfHMao_Q@mail.gmail.com>
	<CALeZrrSNa7cve=N4peusXPNn-G5ubuJd_dgqbo01t_6kpWim4g@mail.gmail.com>
	<328231EA-C034-4337-BB37-F3FF2AA1C2D7@elemental.org>
	<CALeZrrR447M8EHK1bo_phjZE_MVSUuGFAGBF9y=Y2GhedJMxKg@mail.gmail.com>
	<CALeZrrQ-7NuRzQ2E-p=nzETHptQTBgLP8b4UHCd0Zkj=Z1ep1w@mail.gmail.com>
	<f6d0efd4-6fd6-a93f-b113-f2ea627272b2@joyent.com>
Message-ID: <CALeZrrSoL+Jm57BchOCogWnr7jwY7n=jWJ3s+gGFu069-piNkw@mail.gmail.com>

Robert,

That is awesome.   I'd definitely be interested in testing this.

I'll get my feet wet with building OmniOS CE with it.

Thanks!
-Chip

On Fri, Sep 8, 2017 at 10:07 AM, Robert Mustacchi <rm at joyent.com> wrote:

> On 9/8/17 6:43 , Schweiss, Chip wrote:
> > Now that I'm back to working on this.   The only way I could get the
> > firmware updated was booting into the UEFI shell.   A bit of a pain but
> it
> > worked.
> >
> > Unfortunately, it has not changed the behavior of the HBA.
> >
> > Where do I go from here?    Any hope of getting this working on OmniOS?
>
> Hi Chip,
>
> I'm just catching up on this. So, I do have some good news and bad news.
> First, the bad news. This is based on the SAS3224 chipset, which it
> appears the 16e is also describing itself as. Of note, this uses a
> slightly newer version of the MPI specification and the driver as it is
> written doesn't quite notice that it requires slightly different
> behavior and a simple PCI ID update isn't sufficient.
>
> The good news is that I just finished doing this work for the LSI
> 9305-24i and was going to send that up to illumos shortly. If you want,
> I can send those changes your way if you're comfortable building illumos
> and want to test that.
>
> Robert
>
> > On Thu, Aug 31, 2017 at 9:53 AM, Schweiss, Chip <chip at innovates.com>
> wrote:
> >
> >> This server will be serving NFS for vSphere.  It is running OmniOS CE,
> >> nothing VMware.
> >>
> >> I'm working on flashing firmware now and will report back any changes.
> >>
> >> -Chip
> >>
> >> On Thu, Aug 31, 2017 at 9:42 AM, Dale Ghent <daleg at elemental.org>
> wrote:
> >>
> >>>> On Aug 31, 2017, at 9:29 AM, Schweiss, Chip <chip at innovates.com>
> wrote:
> >>>>
> >>>> I've added mpt_sas "pciex1000,c9" to /etc/driver_aliases and rebooted.
> >>>>
> >>>> Looks like it's partially working, but it's not fully functional.
> >>> Service are timing out:
> >>>>
> >>>> Here's what I see in /var/adm/messages:
> >>>>
> >>>>
> >>>> Aug 31 08:15:49 vsphere-zfs01 scsi: [ID 107833 kern.warning] WARNING:
> >>> /pci at 0,0/pci8086,1905 at 1,1/pci1000,3180 at 0 (mpt_sas0):
> >>>> Aug 31 08:15:49 vsphere-zfs01   MPT Firmware Fault, code: 2667
> >>>> Aug 31 08:15:49 vsphere-zfs01 scsi: [ID 107833 kern.warning] WARNING:
> >>> /pci at 0,0/pci8086,1905 at 1,1/pci1000,3180 at 0 (mpt_sas0):
> >>>
> >>> The driver is reporting that the MPT IOC (IO Controller) is reporting a
> >>> fault. It's just reading this condition off the controller chip
> itself, and
> >>> unfortunately there doesn't seem to be a handy reference published by
> >>> LSI/Avago regarding what 2667h actually means.
> >>>
> >>> However I note from your machine's hostname that this is perhaps a ESI
> >>> guest that is being given the HBA in passthrough mode? It would seem
> that
> >>> someone else has encountered a similar issue as yourself in this case,
> with
> >>> the same MPT fault code, but on Linux running Proxmox. According to
> this
> >>> forum thread, they ended up flashing the firmware on the card to
> something
> >>> newer and the problem went away:
> >>>
> >>> https://forum.proxmox.com/threads/pci-passthrough.16483/
> >>>
> >>> I would suggest Tim's approach and flashing your card up to the newest
> IT
> >>> (not IR) firmware.
> >>>
> >>> /dale
> >>>
> >
> > ------------------------------------------
> > illumos-zfs
> > Archives: https://illumos.topicbox.com/groups/zfs/discussions/
> T372d7ddd75316296-M4bd824d5e1881e2772ee518a
> > Powered by Topicbox: https://topicbox.com
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170908/8f1de912/attachment.html>

From sylvain at yesik.it  Fri Sep  8 15:40:22 2017
From: sylvain at yesik.it (Sylvain Leroux)
Date: Fri, 8 Sep 2017 17:40:22 +0200
Subject: [OmniOS-discuss] Status of the OmniOS project
In-Reply-To: <21F7B203-B7DD-4390-BE89-4966A2B9F37F@cos.ru>
References: <70ed4931-b5db-3372-6e6e-a87ac3b9d27b@yesik.it>
	<CACLsApt=UB4Fh4y_8tJ_6MU3j4eWGb0OuQ6=6--2dyjsoJR3Ww@mail.gmail.com>
	<275e825c-008d-a3fa-5d5c-ebe5d5dfca3d@yesik.it>
	<D001CE67-B140-4FB5-B917-729B7EA463C7@cos.ru>
	<21F7B203-B7DD-4390-BE89-4966A2B9F37F@cos.ru>
Message-ID: <e8540ded-b2e9-fb3a-c9b1-8a28943faed1@yesik.it>



On 09/08/2017 04:18 PM, Jim Klimov wrote:
> On September 8, 2017 4:04:11 PM GMT+02:00, Jim Klimov <jimklimov at cos.ru> wrote:
>> To that effect, OpenIndiana Hipster is another major general-purpose
>> illumos distribution that's being actively debeloped too, aimed at both
>> desktop and server, with a lot of software and GUI desktop pre-packaged
>> in its standard repository - in contrast with OmniOS that aims to be
>> the minimal foundation of an OS.
>>
>> Also there are many smaller projects, outlined on wiki.illumos.org as
>> well as unknown cores of storage, networking and other appliances.
>>
>> Hope this helps,
>> Jim
>> --
>> Typos courtesy of K-9 Mail on my Android
> ...And for clarification - I spoke of general purpose distros but ended on a note of the hearts of appliances.
>
> In that area of purpose-optimized illumos products there are of course some well-known larger bi-lateral opensource commercial players, such as Joyent (now part of Samsung) aimed at hypervisors and related storage needs, Delphix for databases, Nexenta for storage...

Thank you Jim.

Indeed OpenIndiana is clearly the Illumos flagship distribution. I
couldn't have missed it. And in fact, I probably have some OpenIndiana 1
installation CD handing around somewhere!

Concerning the distributions mentioned on wiki.illumos.org:
Unfortunately, I can't provide an exhaustive coverage of all available
distributions. My editor want I focus on 4 or 5 of them. So I focused on
what I considered "enterprise grade" and/or "production ready"
solutions. Namely:
- OpenIndiana,
- SmartOS
- NexentaStor (now closed source)
- OmniOSce


I examined a couple of others I didn't know before like Tribblix 
(http://www.tribblix.org/) or Dyson OS
(https://www.osdyson.org/projects/dyson/wiki). But they seems much less
accomplished. Or not very active like Dilos (http://www.dilos.org/).


Concerning Delphix -- I didn't know that one. I will definitely take a
look at that...

Once again, thank you Jim
- Sylvain

-- 
-- Sylvain Leroux -- Yes, I Know IT !
-- sylvain at yesik.it
-- https://yesik.it
--
-- Follow me on Google+!
-- https://plus.google.com/+YesikIT


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170908/61161e1c/attachment.bin>

From omnios at citrus-it.net  Fri Sep  8 16:50:35 2017
From: omnios at citrus-it.net (Andy Fiddaman)
Date: Fri, 8 Sep 2017 16:50:35 +0000 (UTC)
Subject: [OmniOS-discuss] [zfs] SAS 9305-16e HBA support in Illumos
In-Reply-To: <CALeZrrSoL+Jm57BchOCogWnr7jwY7n=jWJ3s+gGFu069-piNkw@mail.gmail.com>
References: <CALeZrrTUO+mQeH9+nAvmhDvYA6-GgKRxM4vpdPa6JVoGG8rBKw@mail.gmail.com>
	<4FB32C07-51A0-43C1-9E69-69C1876B13EA@elemental.org>
	<19DC39EC-F31F-4C5E-867A-A164E044565B@elemental.org>
	<78B3A268-125F-41ED-9898-E1DC42D195F2@joyent.com>
	<CALeZrrT+L7d7D=TO-WwkJ1a-xqMV7TmSk6KCzLi2-ivfHMao_Q@mail.gmail.com>
	<CALeZrrSNa7cve=N4peusXPNn-G5ubuJd_dgqbo01t_6kpWim4g@mail.gmail.com>
	<328231EA-C034-4337-BB37-F3FF2AA1C2D7@elemental.org>
	<CALeZrrR447M8EHK1bo_phjZE_MVSUuGFAGBF9y=Y2GhedJMxKg@mail.gmail.com>
	<CALeZrrQ-7NuRzQ2E-p=nzETHptQTBgLP8b4UHCd0Zkj=Z1ep1w@mail.gmail.com>
	<f6d0efd4-6fd6-a93f-b113-f2ea627272b2@joyent.com>
	<CALeZrrSoL+Jm57BchOCogWnr7jwY7n=jWJ3s+gGFu069-piNkw@mail.gmail.com>
Message-ID: <nycvar.TFB.7.76.1709081647220.9693@erncre.pvgehf-vg.arg>


On Fri, 8 Sep 2017, Schweiss, Chip wrote:

; Robert,
;
; That is awesome.   I'd definitely be interested in testing this.
;
; I'll get my feet wet with building OmniOS CE with it.

Hi Chip,

If you don't already know about it you might want to take a look at
https://github.com/omniosorg/omni/ which makes building OmniOS on
OmniOS nice and easy, globally or in a zone.

You won't need to build the userland bits so the whole process should
take around an hour.

If you run into trouble, come and chat to us at
https://gitter.im/omniosorg/Lobby

Cheers,

Andy

-- 
Citrus IT Limited | +44 (0)333 0124 007 | enquiries at citrus-it.co.uk
Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ
Registered in England and Wales | Company number 4899123


From stephan.budach at jvm.de  Fri Sep  8 18:23:29 2017
From: stephan.budach at jvm.de (Stephan Budach)
Date: Fri, 8 Sep 2017 20:23:29 +0200 (CEST)
Subject: [OmniOS-discuss] Ang:  COMSTAR and blocksizes
In-Reply-To: <857477775.718.1504779825689.JavaMail.stephan.budach@stephanbudach.local>
References: <1804012031.676.1504773331844.JavaMail.stephan.budach@stephanbudach.local>
	<OF9C7A176C.44C956B4-ONC1258194.0037BB9C-C1258194.0037BB9D@inse.com>
	<857477775.718.1504779825689.JavaMail.stephan.budach@stephanbudach.local>
Message-ID: <471339516.653.1504895006099.JavaMail.stephan.budach@budy.stephanbudach.de>



----- Urspr?ngliche Mail -----
> Von: "Stephan Budach" <stephan.budach at jvm.de>
> An: "Johan Kragsterman" <johan.kragsterman at capvert.se>
> CC: omnios-discuss at lists.omniti.com
> Gesendet: Donnerstag, 7. September 2017 12:22:06
> Betreff: Re: [OmniOS-discuss] Ang:  COMSTAR and blocksizes
> 
> 
> 
> ----- Urspr?ngliche Mail -----
> > Von: "Johan Kragsterman" <johan.kragsterman at capvert.se>
> > An: "Stephan Budach" <stephan.budach at jvm.de>
> > CC: omnios-discuss at lists.omniti.com
> > Gesendet: Donnerstag, 7. September 2017 12:08:45
> > Betreff: Ang: [OmniOS-discuss] COMSTAR and blocksizes
> > 
> > 
> > Hi!
> > 
> > -----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com>
> > skrev: -----
> > Till: omnios-discuss at lists.omniti.com
> > Fr?n: Stephan Budach
> > S?nt av: "OmniOS-discuss"
> > Datum: 2017-09-07 10:38
> > ?rende: [OmniOS-discuss] COMSTAR and blocksizes
> > 
> > Hi,
> > 
> > I am having trouble getting an issue sorted out, where omniOS
> > 151020
> > complaints about mismatched blocksizes on some COMSTAR iSCSI LUNS,
> > like this:
> > 
> > Sep  7 08:52:07 zfsha02gh79     104 I/O requests are not aligned
> > with
> > 8192 disk sector size in 10 seconds. They are handled through Read
> > Modify Write but the performance is very low!
> > Sep  7 08:52:07 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING:
> > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94):
> > Sep  7 08:52:07 zfsha02gh79     79 I/O requests are not aligned
> > with
> > 8192 disk sector size in 10 seconds. They are handled through Read
> > Modify Write but the performance is very low!
> > Sep  7 08:52:16 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING:
> > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3132 (sd88):
> > Sep  7 08:52:16 zfsha02gh79     20 I/O requests are not aligned
> > with
> > 8192 disk sector size in 10 seconds. They are handled through Read
> > Modify Write but the performance is very low!
> > Sep  7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING:
> > /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd110):
> > Sep  7 08:52:17 zfsha02gh79     1 I/O requests are not aligned with
> > 8192 disk sector size in 10 seconds. They are handled through Read
> > Modify Write but the performance is very low!
> > Sep  7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning] WARNING:
> > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94):
> > Sep  7 08:52:17 zfsha02gh79     24 I/O requests are not aligned
> > with
> > 8192 disk sector size in 10 seconds. They are handled through Read
> > Modify Write but the performance is very low!
> > 
> > These COMSTAR LUNs are configured to export a blocksize 8k like
> > this:
> > 
> > LU Name: 600144F0564D504F4F4C3037534C3132
> >     Operational Status: Online
> >     Provider Name     : sbd
> >     Alias             : nfsvmpool07Slot12
> >     View Entry Count  : 1
> >     Data File         : /dev/rdsk/c3t50015178F364A264d0p1
> >     Meta File         : not set
> >     Size              : 200042414080
> >     Block Size        : 8192
> >     Management URL    : not set
> >     Vendor ID         : SUN
> >     Product ID        : COMSTAR
> >     Serial Num        : not set
> >     Write Protect     : Disabled
> >     Writeback Cache   : Enabled
> >     Access State      : Active
> > 
> > Now, the system seems to recognize the 8k, but for whatever reason,
> > doesn't adjust the block size accordingly. It does that for the 4k
> > LUNs, however, so I am unsure, on how to tackle this? Any tipp,
> > anyone could share?
> > 
> > Thanks,
> > Stephan
> > 
> > 
> > 
> > 
> > 
> > 
> > What's in the bottom here?
> > 
> > Data File         : /dev/rdsk/c3t50015178F364A264d0p1
> > 
> > It dosesn't look like a zvol?
> > 
> > /Johan
> 
> It isn't one - it's a raw partition, an Intel SSD in this case.
> 
> Cheers,
> Stephan
> 

Hi all,

I just exported this zpool from my omniOS 020 and imported it on a recent oi box and the issue with the 8k writes doesn't seem to be present on oi. Any thoughts on this?

Thanks,
Stephan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5546 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170908/c69d8c85/attachment.bin>

From stephan.budach at jvm.de  Mon Sep 11 07:24:04 2017
From: stephan.budach at jvm.de (Stephan Budach)
Date: Mon, 11 Sep 2017 09:24:04 +0200 (CEST)
Subject: [OmniOS-discuss] Ang:  COMSTAR and blocksizes
In-Reply-To: <471339516.653.1504895006099.JavaMail.stephan.budach@budy.stephanbudach.de>
References: <1804012031.676.1504773331844.JavaMail.stephan.budach@stephanbudach.local>
	<OF9C7A176C.44C956B4-ONC1258194.0037BB9C-C1258194.0037BB9D@inse.com>
	<857477775.718.1504779825689.JavaMail.stephan.budach@stephanbudach.local>
	<471339516.653.1504895006099.JavaMail.stephan.budach@budy.stephanbudach.de>
Message-ID: <1384013344.979.1505114641278.JavaMail.stephan.budach@stephan.budach.wlan.jvm.de>

> > ----- Urspr?ngliche Mail -----
> > > Von: "Johan Kragsterman" <johan.kragsterman at capvert.se>
> > > An: "Stephan Budach" <stephan.budach at jvm.de>
> > > CC: omnios-discuss at lists.omniti.com
> > > Gesendet: Donnerstag, 7. September 2017 12:08:45
> > > Betreff: Ang: [OmniOS-discuss] COMSTAR and blocksizes
> > > 
> > > 
> > > Hi!
> > > 
> > > -----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com>
> > > skrev: -----
> > > Till: omnios-discuss at lists.omniti.com
> > > Fr?n: Stephan Budach
> > > S?nt av: "OmniOS-discuss"
> > > Datum: 2017-09-07 10:38
> > > ?rende: [OmniOS-discuss] COMSTAR and blocksizes
> > > 
> > > Hi,
> > > 
> > > I am having trouble getting an issue sorted out, where omniOS
> > > 151020
> > > complaints about mismatched blocksizes on some COMSTAR iSCSI
> > > LUNS,
> > > like this:
> > > 
> > > Sep  7 08:52:07 zfsha02gh79     104 I/O requests are not aligned
> > > with
> > > 8192 disk sector size in 10 seconds. They are handled through
> > > Read
> > > Modify Write but the performance is very low!
> > > Sep  7 08:52:07 zfsha02gh79 scsi: [ID 107833 kern.warning]
> > > WARNING:
> > > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94):
> > > Sep  7 08:52:07 zfsha02gh79     79 I/O requests are not aligned
> > > with
> > > 8192 disk sector size in 10 seconds. They are handled through
> > > Read
> > > Modify Write but the performance is very low!
> > > Sep  7 08:52:16 zfsha02gh79 scsi: [ID 107833 kern.warning]
> > > WARNING:
> > > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3132 (sd88):
> > > Sep  7 08:52:16 zfsha02gh79     20 I/O requests are not aligned
> > > with
> > > 8192 disk sector size in 10 seconds. They are handled through
> > > Read
> > > Modify Write but the performance is very low!
> > > Sep  7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning]
> > > WARNING:
> > > /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd110):
> > > Sep  7 08:52:17 zfsha02gh79     1 I/O requests are not aligned
> > > with
> > > 8192 disk sector size in 10 seconds. They are handled through
> > > Read
> > > Modify Write but the performance is very low!
> > > Sep  7 08:52:17 zfsha02gh79 scsi: [ID 107833 kern.warning]
> > > WARNING:
> > > /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd94):
> > > Sep  7 08:52:17 zfsha02gh79     24 I/O requests are not aligned
> > > with
> > > 8192 disk sector size in 10 seconds. They are handled through
> > > Read
> > > Modify Write but the performance is very low!
> > > 
> > > These COMSTAR LUNs are configured to export a blocksize 8k like
> > > this:
> > > 
> > > LU Name: 600144F0564D504F4F4C3037534C3132
> > >     Operational Status: Online
> > >     Provider Name     : sbd
> > >     Alias             : nfsvmpool07Slot12
> > >     View Entry Count  : 1
> > >     Data File         : /dev/rdsk/c3t50015178F364A264d0p1
> > >     Meta File         : not set
> > >     Size              : 200042414080
> > >     Block Size        : 8192
> > >     Management URL    : not set
> > >     Vendor ID         : SUN
> > >     Product ID        : COMSTAR
> > >     Serial Num        : not set
> > >     Write Protect     : Disabled
> > >     Writeback Cache   : Enabled
> > >     Access State      : Active
> > > 
> > > Now, the system seems to recognize the 8k, but for whatever
> > > reason,
> > > doesn't adjust the block size accordingly. It does that for the
> > > 4k
> > > LUNs, however, so I am unsure, on how to tackle this? Any tipp,
> > > anyone could share?
> > > 
> > > Thanks,
> > > Stephan
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > What's in the bottom here?
> > > 
> > > Data File         : /dev/rdsk/c3t50015178F364A264d0p1
> > > 
> > > It dosesn't look like a zvol?
> > > 
> > > /Johan
> > 
> > It isn't one - it's a raw partition, an Intel SSD in this case.
> > 
> > Cheers,
> > Stephan
> > 
> 
> Hi all,
> 
> I just exported this zpool from my omniOS 020 and imported it on a
> recent oi box and the issue with the 8k writes doesn't seem to be
> present on oi. Any thoughts on this?
> 
> Thanks,
> Stephan

I have the exact same behaviour on a freshly installed and updated omniosce 151022o:

root at vsm02:~# cat /etc/release
  OmniOS v11 r151022o
  Copyright 2017 OmniTI Computer Consulting, Inc. All rights reserved.
  Copyright 2017 OmniOS Community Edition (OmniOSce) Association.
  All rights reserved. Use is subject to license terms.
root at vsm02:~# tail -f /var/adm/messages
Sep 11 11:21:42 vsm02 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd29):
Sep 11 11:21:42 vsm02   18 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low!
Sep 11 11:21:47 vsm02 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3033 (sd13):
Sep 11 11:21:47 vsm02   19 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low!
Sep 11 11:21:48 vsm02 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3132 (sd7):
Sep 11 11:21:48 vsm02   18 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low!
Sep 11 11:21:50 vsm02 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3037534c3131 (sd6):
Sep 11 11:21:50 vsm02   9 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low!
Sep 11 11:21:52 vsm02 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g600144f0564d504f4f4c3038534c3033 (sd29):
Sep 11 11:21:52 vsm02   9 I/O requests are not aligned with 8192 disk sector size in 10 seconds. They are handled through Read Modify Write but the performance is very low!

Is this even a COMSTAR/iSCSI Initiator thing, or is this something else?

/stephan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5546 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170911/1c584a61/attachment-0001.bin>

From sylvain at yesik.it  Mon Sep 11 16:27:12 2017
From: sylvain at yesik.it (Sylvain Leroux)
Date: Mon, 11 Sep 2017 18:27:12 +0200
Subject: [OmniOS-discuss] Supported architectures
Message-ID: <20183570-795b-9da4-2a5d-3ce288203e68@yesik.it>

I was asked about the supported architecture for OmniOSce.

As far as I can tell, this was not mentioned on the
http://www.omniosce.org website. So, I assume x86_64.

But, I *think* OmniOS did support x86_32 too. Is this the case for
OmniOSce? What about the SPARC architecture?

- Sylvain



-- 

-- Sylvain Leroux -- Yes, I Know IT !
-- sylvain at yesik.it
-- https://yesik.it
--
-- Follow me on Google+!
-- https://plus.google.com/+YesikIT


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170911/9fd16bf0/attachment.bin>

From danmcd at kebe.com  Mon Sep 11 16:48:28 2017
From: danmcd at kebe.com (Dan McDonald)
Date: Mon, 11 Sep 2017 12:48:28 -0400
Subject: [OmniOS-discuss] Supported architectures
In-Reply-To: <20183570-795b-9da4-2a5d-3ce288203e68@yesik.it>
References: <20183570-795b-9da4-2a5d-3ce288203e68@yesik.it>
Message-ID: <20170911164828.GA35134@everywhere.local>

On Mon, Sep 11, 2017 at 06:27:12PM +0200, Sylvain Leroux wrote:
> I was asked about the supported architecture for OmniOSce.
> 
> As far as I can tell, this was not mentioned on the
> http://www.omniosce.org website. So, I assume x86_64.
> 
> But, I *think* OmniOS did support x86_32 too. Is this the case for
> OmniOSce? What about the SPARC architecture?

Pre-ce, OmniOS could boot on 32-bit x86, but OmniTI would not provide support
for 32-bit boots.  SPARC was always unsupported and unbuilt.

I assume the new community management will not change any of that.  Your
32-bit x86 apps, BTW, will continue to run on an amd64/x64/x86_64 boot just
fine.

Dan

From jboren at drakecooper.com  Wed Sep 13 01:27:41 2017
From: jboren at drakecooper.com (Joseph Boren)
Date: Tue, 12 Sep 2017 19:27:41 -0600
Subject: [OmniOS-discuss] weird issue with active BE on new Omnios CE install
Message-ID: <CAGoaUhN5f-+W5riiagC=z+xKHbpK5o4DhGt2AM0okqEUkeqo7A@mail.gmail.com>

Greetings,

Had a strange thing happen with a new server build today.  Installing
r151022q on a new Supermicro X11SSH-CTF motherboard, downloaded the flash
drive image, burned to a 64Gb USB flash drive, installed just fine,
rebooted, all is well.  Installed napp-it, which creates a new BE and sets
it active.  Rebooted and let it try to boot into the new BE and got this:

Loading now....
can't find 'now'
can't find 'now'

Loading /boot/defualts/loader.conf
Loading now.....
can't find 'now'
can't find 'now'

Type ? for a list of commands *blah blah blah*
ok

Rebooted and selected the BE from before the install and it booted fine.
So I created a new BE with beadmin, set it active, and rebooted and got the
same error.  So I set the old (working) BE active and rebooted again, and
now I'm getting that same error on that BE too.  Rebooted yet again,
selected the new, manually created BE and it booted just fine.  Set it
active and rebooted again - now that BE is broken too.  Rebooted for the
millionth time and selected the other (old - not active) BE and lo and
behold it boots just fine.

So it appears that whatever BE is set active will not boot.  I believe
that's actual irony right there.  Any non-active BE will boot just fine if
selected in the loader screen.

Anyone seen this problem before?  Any Ideas what may be going on?  It's
probably my fault somehow....

Thanks,

Joseph Boren

IT Specialist

+c: 208.891.2128

+ 416 S 8th St., Boise, ID 83702

+ drakecooper.com <http://www.drakecooper.com/>
*Read our minds *Blog <https://drakecooper.com/blog/>
Facebook <https://www.facebook.com/DrakeCooper>, Twitter
<https://twitter.com/drakecooper>, Linkedin
<https://www.linkedin.com/company/drake-cooper>, Instagram
<https://www.instagram.com/drakecooperadvertising/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170912/d5b0f304/attachment.html>

From jdg117 at elvis.arl.psu.edu  Wed Sep 13 14:08:31 2017
From: jdg117 at elvis.arl.psu.edu (John D Groenveld)
Date: Wed, 13 Sep 2017 10:08:31 -0400
Subject: [OmniOS-discuss] weird issue with active BE on new Omnios CE
	install
In-Reply-To: Your message of "Tue, 12 Sep 2017 19:27:41 MDT."
	<CAGoaUhN5f-+W5riiagC=z+xKHbpK5o4DhGt2AM0okqEUkeqo7A@mail.gmail.com> 
References: <CAGoaUhN5f-+W5riiagC=z+xKHbpK5o4DhGt2AM0okqEUkeqo7A@mail.gmail.com>
Message-ID: <201709131408.v8DE8VDa029615@elvis.arl.psu.edu>

In message <CAGoaUhN5f-+W5riiagC=z+xKHbpK5o4DhGt2AM0okqEUkeqo7A at mail.gmail.com>
, Joseph Boren writes:
>Loading now....
>can't find 'now'
>can't find 'now'

Has the stench of reboot(1M) now 
       boot_arguments

           An optional boot_arguments specifies arguments to the uadmin(2)
           function that are passed to the boot program and kernel upon
           restart. The form and list of arguments is described in the
           boot(1M) and kernel(1M) man pages.. If the arguments are specified,
           whitespace between them is replaced by single spaces unless the
           whitespace is quoted for the shell. If the boot_arguments begin
           with a hyphen, they must be preceded by the -- delimiter (two
           hyphens) to denote the end of the reboot argument list.

I find shutdown(1M) and init(1M) are safer when moving between OSs.

John
groenveld at acm.org

From omnios at citrus-it.net  Wed Sep 13 18:29:53 2017
From: omnios at citrus-it.net (Andy Fiddaman)
Date: Wed, 13 Sep 2017 18:29:53 +0000 (UTC)
Subject: [OmniOS-discuss] weird issue with active BE on new Omnios CE
 install
In-Reply-To: <CAGoaUhN5f-+W5riiagC=z+xKHbpK5o4DhGt2AM0okqEUkeqo7A@mail.gmail.com>
References: <CAGoaUhN5f-+W5riiagC=z+xKHbpK5o4DhGt2AM0okqEUkeqo7A@mail.gmail.com>
Message-ID: <nycvar.TFB.7.76.1709131827500.19150@erncre.pvgehf-vg.arg>


On Tue, 12 Sep 2017, Joseph Boren wrote:

; Greetings,
;
; Had a strange thing happen with a new server build today.  Installing
; r151022q on a new Supermicro X11SSH-CTF motherboard, downloaded the flash
; drive image, burned to a 64Gb USB flash drive, installed just fine,
; rebooted, all is well.  Installed napp-it, which creates a new BE and sets
; it active.  Rebooted and let it try to boot into the new BE and got this:
;
; Loading now....
; can't find 'now'
; can't find 'now'

How are you rebooting?
If you are using 'reboot now' then it will result in this behaviour.

Use just 'reboot' or 'init 6' - it isn't Linux : )

Andy

-- 
Citrus IT Limited | +44 (0)333 0124 007 | enquiries at citrus-it.co.uk
Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ
Registered in England and Wales | Company number 4899123


From jboren at drakecooper.com  Wed Sep 13 19:07:00 2017
From: jboren at drakecooper.com (Joseph Boren)
Date: Wed, 13 Sep 2017 13:07:00 -0600
Subject: [OmniOS-discuss] weird issue with active BE on new Omnios CE
	install
In-Reply-To: <nycvar.TFB.7.76.1709131827500.19150@erncre.pvgehf-vg.arg>
References: <CAGoaUhN5f-+W5riiagC=z+xKHbpK5o4DhGt2AM0okqEUkeqo7A@mail.gmail.com>
	<nycvar.TFB.7.76.1709131827500.19150@erncre.pvgehf-vg.arg>
Message-ID: <CAGoaUhNCc10hhQDgrXCBK3kJE8LhEUStNO=0u=ZSH0vspsAoUA@mail.gmail.com>

Arghhh, are you freaking kidding me?!?   At least I was right about it
being something I was doing.  :)

Thank you Andy, I appreciate the reminder.  Embarrassing, but I guess it's
not wasted time if I learned something.  Or re-learned it in this case...
Thanks again!

Cheers!


Joseph Boren

IT Specialist

+c: 208.891.2128

+ 416 S 8th St., Boise, ID 83702

+ drakecooper.com <http://www.drakecooper.com/>
*Read our minds *Blog <https://drakecooper.com/blog/>
Facebook <https://www.facebook.com/DrakeCooper>, Twitter
<https://twitter.com/drakecooper>, Linkedin
<https://www.linkedin.com/company/drake-cooper>, Instagram
<https://www.instagram.com/drakecooperadvertising/>



On Wed, Sep 13, 2017 at 12:29 PM, Andy Fiddaman <omnios at citrus-it.net>
wrote:

>
> On Tue, 12 Sep 2017, Joseph Boren wrote:
>
> ; Greetings,
> ;
> ; Had a strange thing happen with a new server build today.  Installing
> ; r151022q on a new Supermicro X11SSH-CTF motherboard, downloaded the flash
> ; drive image, burned to a 64Gb USB flash drive, installed just fine,
> ; rebooted, all is well.  Installed napp-it, which creates a new BE and
> sets
> ; it active.  Rebooted and let it try to boot into the new BE and got this:
> ;
> ; Loading now....
> ; can't find 'now'
> ; can't find 'now'
>
> How are you rebooting?
> If you are using 'reboot now' then it will result in this behaviour.
>
> Use just 'reboot' or 'init 6' - it isn't Linux : )
>
> Andy
>
> --
> Citrus IT Limited | +44 (0)333 0124 007 | enquiries at citrus-it.co.uk
> Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ
> Registered in England and Wales | Company number 4899123
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170913/f379959c/attachment-0001.html>

From dirk.willems at exitas.be  Thu Sep 14 12:26:13 2017
From: dirk.willems at exitas.be (Dirk Willems)
Date: Thu, 14 Sep 2017 14:26:13 +0200
Subject: [OmniOS-discuss] questions
Message-ID: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be>

Hello,


I'm trying to understand something let me explain.


Oracle always told to me that if you create a etherstub switch it has 
infiniband speed 40GB/s.

But I have a customer running on Solaris (Yeah I know but let me 
explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan 
(I know told him to to use etherstub).

The copy witch is performed for a Oracle database with sql command, the 
DBA witch have 5 streams say it's waiting on the disk, the disk are 50 - 
60 % busy the speed is 30 mb/s.


So I did some test just to see and understand if it's the database or 
the system, but with doing my tests I get very confused ???


On another Solaris at my work copy over etherstub switch => copy speed 
is 185MB/s expected much more of infiniband speed ???


root at test1:/export/home/Admin# scp test10G 
Admin at 192.168.1.2:/export/home/Admin/
Password:
test10G              100% 
|****************************************************************| 10240 
MB    00:59


root at test2:~# dlstat -i 2

  LINK    IPKTS   RBYTES    OPKTS   OBYTES
            net1   25.76K 185.14M 10.08K    2.62M
            net1   27.04K  187.16M   11.23K    3.22M
            net1   26.97K  186.37M   11.24K    3.23M
            net1   26.63K  187.67M   10.82K    2.99M
            net1   27.94K  186.65M   12.17K    3.75M
            net1   27.45K  187.46M   11.70K    3.47M
            net1   26.01K  181.95M   10.63K    2.99M
            net1   27.95K  188.19M   12.14K    3.69M
            net1   27.91K  188.36M   12.08K    3.64M

The disks are all separate luns with all separated pools => disk are 20 
- 30% busy


On my OmniOSce at my lab over etherstub


root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/
Password:
test10G 76% 7853MB 116.4MB/s


=> copy is 116.4 MB/s => expected much more from infiniband speed is 
just the same as Lan ???


Is not that my disk can not follow 17% busy there sleeping ...

    extended device statistics
     r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
     0,0  248,4    0,0    2,1  0,0  1,3    0,0    5,3   0 102 c1
     0,0   37,5    0,0    0,7  0,0  0,2    0,0    4,7   0  17 c1t0d0 => 
rpool
     0,0   38,5    0,0    0,7  0,0  0,2    0,0    4,9   0  17 c1t1d0 => 
rpool
     0,0   40,5    0,0    0,1  0,0  0,2    0,0    5,6   0  17 c1t2d0 => 
data pool
     0,0   43,5    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t3d0 => 
data pool
     0,0   44,5    0,0    0,2  0,0  0,2    0,0    5,5   0  18 c1t4d0 => 
data pool
     0,0   44,0    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t5d0 => 
data pool
     0,0   76,0    0,0    1,5  7,4  0,4   97,2    4,9  14  18 rpool
     0,0  172,4    0,0    0,6  2,0  0,9   11,4    5,5  12  20 DATA



root at NGINX:/root# dlstat show-link NGINX1 -i 2

  LINK  TYPE      ID  INDEX     PKTS    BYTES
          NGINX1    rx   bcast     --        0        0
          NGINX1    rx      sw     --        0        0
          NGINX1    tx   bcast     --        0        0
          NGINX1    tx      sw     --    9.26K  692.00K
          NGINX1    rx   local     --   26.00K 216.32M
          NGINX1    rx   bcast     --        0        0
          NGINX1    rx      sw     --        0        0
          NGINX1    tx   bcast     --        0        0
          NGINX1    tx      sw     --    7.01K  531.38K
          NGINX1    rx   local     --   30.65K 253.73M
          NGINX1    rx   bcast     --        0        0
          NGINX1    rx      sw     --        0        0
          NGINX1    tx   bcast     --        0        0
          NGINX1    tx      sw     --    8.95K  669.32K
          NGINX1    rx   local     --   29.10K 241.15M


On the other NGZ I receive 250MB/s ????


- So my question is how comes that the speed is equal to Lan 100MB/s on 
OmniOSce but i receive 250MB/s ?

- Why is etherstub so slow if infiniband speed is 40GB/s ???


I'm very confused right now ...


And want to know for sure how to understand and see this in the right 
way, because this customer will be the first customer from my who gonna 
switch complety over to OmniOSce on production and because this customer 
is one or the biggest company's in Belgium I really don't want to mess 
up !!!


So any help and clarification will be highly appreciate !!!


Thank you very much.


Kind Regards,


Dirk


-- 
	Dirk Willems
System Engineer


+32 (0)3 443 12 38
Dirk.Willems at exitas.be <mailto:Dirk.Willems at exitas.be>

Quality. Passion. Personality

www.exitas.be <http://www.exitas.be/> | Veldkant 31 | 2550 Kontich

Illumos OmniOS Installation and Configuration Implementation Specialist.
Oracle Solaris 11 Installation and Configuration Certified 
Implementation Specialist. 	

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/95561ee3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SmallPhoenixLogotypeRGB.png
Type: image/png
Size: 4648 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/95561ee3/attachment.png>

From lorban at bitronix.be  Thu Sep 14 13:27:36 2017
From: lorban at bitronix.be (Ludovic Orban)
Date: Thu, 14 Sep 2017 15:27:36 +0200
Subject: [OmniOS-discuss] questions
In-Reply-To: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be>
References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be>
Message-ID: <CAN0RhAMhGOkLJkmeHxpkAu97KNF4g-bez5zRULWRKcoJzr6nYg@mail.gmail.com>

Here's a quick thought: if you are copying your files with scp, you might
be CPU bound because of all the crypto work. You should try to copy with a
much more CPU-lightweight tool. I personally usually use netcat for this
kind of stuff.

Just my 2 cents.

--
Ludovic

On Thu, Sep 14, 2017 at 2:26 PM, Dirk Willems <dirk.willems at exitas.be>
wrote:

> Hello,
>
>
> I'm trying to understand something let me explain.
>
>
> Oracle always told to me that if you create a etherstub switch it has
> infiniband speed 40GB/s.
>
> But I have a customer running on Solaris (Yeah I know but let me explain)
> who is copy from 1 NGZ to another NGZ on the same GZ over Lan (I know told
> him to to use etherstub).
>
> The copy witch is performed for a Oracle database with sql command, the
> DBA witch have 5 streams say it's waiting on the disk, the disk are 50 - 60
> % busy the speed is 30 mb/s.
>
>
> So I did some test just to see and understand if it's the database or the
> system, but with doing my tests I get very confused ???
>
>
> On another Solaris at my work copy over etherstub switch => copy speed is
> 185MB/s expected much more of infiniband speed ???
>
>
> root at test1:/export/home/Admin# scp test10G Admin at 192.168.1.2:/export/
> home/Admin/
> Password:
> test10G              100% |*****************************
> ***********************************| 10240 MB    00:59
>
>
> root at test2:~# dlstat -i 2
>
>  LINK    IPKTS   RBYTES    OPKTS   OBYTES
>            net1   25.76K  185.14M   10.08K    2.62M
>            net1   27.04K  187.16M   11.23K    3.22M
>            net1   26.97K  186.37M   11.24K    3.23M
>            net1   26.63K  187.67M   10.82K    2.99M
>            net1   27.94K  186.65M   12.17K    3.75M
>            net1   27.45K  187.46M   11.70K    3.47M
>            net1   26.01K  181.95M   10.63K    2.99M
>            net1   27.95K  188.19M   12.14K    3.69M
>            net1   27.91K  188.36M   12.08K    3.64M
>
> The disks are all separate luns with all separated pools => disk are 20 -
> 30% busy
>
>
> On my OmniOSce at my lab over etherstub
>
>
> root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/
> Password:
> test10G
> 76% 7853MB 116.4MB/s
>
>
> => copy is 116.4 MB/s => expected much more from infiniband speed is just
> the same as Lan ???
>
>
> Is not that my disk can not follow 17% busy there sleeping ...
>
>    extended device statistics
>     r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
>     0,0  248,4    0,0    2,1  0,0  1,3    0,0    5,3   0 102 c1
>     0,0   37,5    0,0    0,7  0,0  0,2    0,0    4,7   0  17 c1t0d0 =>
> rpool
>     0,0   38,5    0,0    0,7  0,0  0,2    0,0    4,9   0  17 c1t1d0 =>
> rpool
>     0,0   40,5    0,0    0,1  0,0  0,2    0,0    5,6   0  17 c1t2d0 =>
> data pool
>     0,0   43,5    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t3d0 =>
> data pool
>     0,0   44,5    0,0    0,2  0,0  0,2    0,0    5,5   0  18 c1t4d0 =>
> data pool
>     0,0   44,0    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t5d0 =>
> data pool
>     0,0   76,0    0,0    1,5  7,4  0,4   97,2    4,9  14  18 rpool
>     0,0  172,4    0,0    0,6  2,0  0,9   11,4    5,5  12  20 DATA
>
>
>
> root at NGINX:/root# dlstat show-link NGINX1 -i 2
>
>  LINK  TYPE      ID  INDEX     PKTS    BYTES
>          NGINX1    rx   bcast     --        0        0
>          NGINX1    rx      sw     --        0        0
>          NGINX1    tx   bcast     --        0        0
>          NGINX1    tx      sw     --    9.26K  692.00K
>          NGINX1    rx   local     --   26.00K  216.32M
>          NGINX1    rx   bcast     --        0        0
>          NGINX1    rx      sw     --        0        0
>          NGINX1    tx   bcast     --        0        0
>          NGINX1    tx      sw     --    7.01K  531.38K
>          NGINX1    rx   local     --   30.65K  253.73M
>          NGINX1    rx   bcast     --        0        0
>          NGINX1    rx      sw     --        0        0
>          NGINX1    tx   bcast     --        0        0
>          NGINX1    tx      sw     --    8.95K  669.32K
>          NGINX1    rx   local     --   29.10K  241.15M
>
>
> On the other NGZ I receive 250MB/s ????
>
>
> - So my question is how comes that the speed is equal to Lan 100MB/s on
> OmniOSce but i receive 250MB/s ?
>
> - Why is etherstub so slow if infiniband speed is 40GB/s ???
>
>
> I'm very confused right now ...
>
>
> And want to know for sure how to understand and see this in the right way,
> because this customer will be the first customer from my who gonna switch
> complety over to OmniOSce on production and because this customer is one or
> the biggest company's in Belgium I really don't want to mess up !!!
>
>
> So any help and clarification will be highly appreciate !!!
>
>
> Thank you very much.
>
>
> Kind Regards,
>
>
> Dirk
>
>
> --
> Dirk Willems
> System Engineer
>
>
> +32 (0)3 443 12 38 <03%20443%2012%2038>
> Dirk.Willems at exitas.be
>
> Quality. Passion. Personality
>
> www.exitas.be | Veldkant 31 | 2550 Kontich
>
> Illumos OmniOS Installation and Configuration Implementation Specialist.
> Oracle Solaris 11 Installation and Configuration Certified Implementation
> Specialist.
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/db4b0b37/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SmallPhoenixLogotypeRGB.png
Type: image/png
Size: 4648 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/db4b0b37/attachment-0001.png>

From jimklimov at cos.ru  Thu Sep 14 14:10:09 2017
From: jimklimov at cos.ru (Jim Klimov)
Date: Thu, 14 Sep 2017 14:10:09 +0000
Subject: [OmniOS-discuss] questions
In-Reply-To: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be>
References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be>
Message-ID: <7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru>

On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems <dirk.willems at exitas.be> wrote:
>Hello,
>
>
>I'm trying to understand something let me explain.
>
>
>Oracle always told to me that if you create a etherstub switch it has 
>infiniband speed 40GB/s.
>
>But I have a customer running on Solaris (Yeah I know but let me 
>explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan 
>(I know told him to to use etherstub).
>
>The copy witch is performed for a Oracle database with sql command, the
>
>DBA witch have 5 streams say it's waiting on the disk, the disk are 50
>- 
>60 % busy the speed is 30 mb/s.
>
>
>So I did some test just to see and understand if it's the database or 
>the system, but with doing my tests I get very confused ???
>
>
>On another Solaris at my work copy over etherstub switch => copy speed 
>is 185MB/s expected much more of infiniband speed ???
>
>
>root at test1:/export/home/Admin# scp test10G 
>Admin at 192.168.1.2:/export/home/Admin/
>Password:
>test10G              100% 
>|****************************************************************|
>10240 
>MB    00:59
>
>
>root at test2:~# dlstat -i 2
>
>  LINK    IPKTS   RBYTES    OPKTS   OBYTES
>            net1   25.76K 185.14M 10.08K    2.62M
>            net1   27.04K  187.16M   11.23K    3.22M
>            net1   26.97K  186.37M   11.24K    3.23M
>            net1   26.63K  187.67M   10.82K    2.99M
>            net1   27.94K  186.65M   12.17K    3.75M
>            net1   27.45K  187.46M   11.70K    3.47M
>            net1   26.01K  181.95M   10.63K    2.99M
>            net1   27.95K  188.19M   12.14K    3.69M
>            net1   27.91K  188.36M   12.08K    3.64M
>
>The disks are all separate luns with all separated pools => disk are 20
>
>- 30% busy
>
>
>On my OmniOSce at my lab over etherstub
>
>
>root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/
>Password:
>test10G 76% 7853MB 116.4MB/s
>
>
>=> copy is 116.4 MB/s => expected much more from infiniband speed is 
>just the same as Lan ???
>
>
>Is not that my disk can not follow 17% busy there sleeping ...
>
>    extended device statistics
>     r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
>     0,0  248,4    0,0    2,1  0,0  1,3    0,0    5,3   0 102 c1
>    0,0   37,5    0,0    0,7  0,0  0,2    0,0    4,7   0  17 c1t0d0 => 
>rpool
>    0,0   38,5    0,0    0,7  0,0  0,2    0,0    4,9   0  17 c1t1d0 => 
>rpool
>    0,0   40,5    0,0    0,1  0,0  0,2    0,0    5,6   0  17 c1t2d0 => 
>data pool
>    0,0   43,5    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t3d0 => 
>data pool
>    0,0   44,5    0,0    0,2  0,0  0,2    0,0    5,5   0  18 c1t4d0 => 
>data pool
>    0,0   44,0    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t5d0 => 
>data pool
>     0,0   76,0    0,0    1,5  7,4  0,4   97,2    4,9  14  18 rpool
>     0,0  172,4    0,0    0,6  2,0  0,9   11,4    5,5  12  20 DATA
>
>
>
>root at NGINX:/root# dlstat show-link NGINX1 -i 2
>
>  LINK  TYPE      ID  INDEX     PKTS    BYTES
>          NGINX1    rx   bcast     --        0        0
>          NGINX1    rx      sw     --        0        0
>          NGINX1    tx   bcast     --        0        0
>          NGINX1    tx      sw     --    9.26K  692.00K
>          NGINX1    rx   local     --   26.00K 216.32M
>          NGINX1    rx   bcast     --        0        0
>          NGINX1    rx      sw     --        0        0
>          NGINX1    tx   bcast     --        0        0
>          NGINX1    tx      sw     --    7.01K  531.38K
>          NGINX1    rx   local     --   30.65K 253.73M
>          NGINX1    rx   bcast     --        0        0
>          NGINX1    rx      sw     --        0        0
>          NGINX1    tx   bcast     --        0        0
>          NGINX1    tx      sw     --    8.95K  669.32K
>          NGINX1    rx   local     --   29.10K 241.15M
>
>
>On the other NGZ I receive 250MB/s ????
>
>
>- So my question is how comes that the speed is equal to Lan 100MB/s on
>
>OmniOSce but i receive 250MB/s ?
>
>- Why is etherstub so slow if infiniband speed is 40GB/s ???
>
>
>I'm very confused right now ...
>
>
>And want to know for sure how to understand and see this in the right 
>way, because this customer will be the first customer from my who gonna
>
>switch complety over to OmniOSce on production and because this
>customer 
>is one or the biggest company's in Belgium I really don't want to mess 
>up !!!
>
>
>So any help and clarification will be highly appreciate !!!
>
>
>Thank you very much.
>
>
>Kind Regards,
>
>
>Dirk

I am not sure where the infiniband claim comes from, but copying data disk to disk, you involve the slow layers like disk, skewed by faster layers like cache of already-read data and delayed writes :)

If you have a wide pipe that you may fill, it doesn't mean you do have the means to fill it with a few disks.

To estimate the speeds, try pure UDP streams from process to process (no disk), large-packet floodping, etc.

I believe etherstub is not constrained artificially, and defaults to jumbo frames. Going to LAN and back can in fact use external hardware (IIRC there may be a system option to disable that, not sure) and so is constrained by that.

Jim
--
Typos courtesy of K-9 Mail on my Android

From ikaufman at eng.ucsd.edu  Thu Sep 14 15:07:06 2017
From: ikaufman at eng.ucsd.edu (Ian Kaufman)
Date: Thu, 14 Sep 2017 08:07:06 -0700
Subject: [OmniOS-discuss] questions
In-Reply-To: <7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru>
References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be>
	<7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru>
Message-ID: <CAPJtH1h9dfpi3OmiH7A+uu2D0am_4XhOiRwZBOvQC=Y_=FRTvQ@mail.gmail.com>

Some other things you need to take into account:

QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 difference.
That is also a theoretical maximum throughput, there is some overhead. In
reality, you will never see 40Gbps.

My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with DDR
(20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops the
theoretical max rates to 18Gbps and 36Gbps respectively.

If you are getting 185MB/s, you are seeing 1.48Gbps.

Keep your B's and b's straight. Did you play with your frame size at all?

Ian

On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov <jimklimov at cos.ru> wrote:

> On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems <
> dirk.willems at exitas.be> wrote:
> >Hello,
> >
> >
> >I'm trying to understand something let me explain.
> >
> >
> >Oracle always told to me that if you create a etherstub switch it has
> >infiniband speed 40GB/s.
> >
> >But I have a customer running on Solaris (Yeah I know but let me
> >explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan
> >(I know told him to to use etherstub).
> >
> >The copy witch is performed for a Oracle database with sql command, the
> >
> >DBA witch have 5 streams say it's waiting on the disk, the disk are 50
> >-
> >60 % busy the speed is 30 mb/s.
> >
> >
> >So I did some test just to see and understand if it's the database or
> >the system, but with doing my tests I get very confused ???
> >
> >
> >On another Solaris at my work copy over etherstub switch => copy speed
> >is 185MB/s expected much more of infiniband speed ???
> >
> >
> >root at test1:/export/home/Admin# scp test10G
> >Admin at 192.168.1.2:/export/home/Admin/
> >Password:
> >test10G              100%
> >|****************************************************************|
> >10240
> >MB    00:59
> >
> >
> >root at test2:~# dlstat -i 2
> >
> >  LINK    IPKTS   RBYTES    OPKTS   OBYTES
> >            net1   25.76K 185.14M 10.08K    2.62M
> >            net1   27.04K  187.16M   11.23K    3.22M
> >            net1   26.97K  186.37M   11.24K    3.23M
> >            net1   26.63K  187.67M   10.82K    2.99M
> >            net1   27.94K  186.65M   12.17K    3.75M
> >            net1   27.45K  187.46M   11.70K    3.47M
> >            net1   26.01K  181.95M   10.63K    2.99M
> >            net1   27.95K  188.19M   12.14K    3.69M
> >            net1   27.91K  188.36M   12.08K    3.64M
> >
> >The disks are all separate luns with all separated pools => disk are 20
> >
> >- 30% busy
> >
> >
> >On my OmniOSce at my lab over etherstub
> >
> >
> >root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/
> >Password:
> >test10G 76% 7853MB 116.4MB/s
> >
> >
> >=> copy is 116.4 MB/s => expected much more from infiniband speed is
> >just the same as Lan ???
> >
> >
> >Is not that my disk can not follow 17% busy there sleeping ...
> >
> >    extended device statistics
> >     r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
> >     0,0  248,4    0,0    2,1  0,0  1,3    0,0    5,3   0 102 c1
> >    0,0   37,5    0,0    0,7  0,0  0,2    0,0    4,7   0  17 c1t0d0 =>
> >rpool
> >    0,0   38,5    0,0    0,7  0,0  0,2    0,0    4,9   0  17 c1t1d0 =>
> >rpool
> >    0,0   40,5    0,0    0,1  0,0  0,2    0,0    5,6   0  17 c1t2d0 =>
> >data pool
> >    0,0   43,5    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t3d0 =>
> >data pool
> >    0,0   44,5    0,0    0,2  0,0  0,2    0,0    5,5   0  18 c1t4d0 =>
> >data pool
> >    0,0   44,0    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t5d0 =>
> >data pool
> >     0,0   76,0    0,0    1,5  7,4  0,4   97,2    4,9  14  18 rpool
> >     0,0  172,4    0,0    0,6  2,0  0,9   11,4    5,5  12  20 DATA
> >
> >
> >
> >root at NGINX:/root# dlstat show-link NGINX1 -i 2
> >
> >  LINK  TYPE      ID  INDEX     PKTS    BYTES
> >          NGINX1    rx   bcast     --        0        0
> >          NGINX1    rx      sw     --        0        0
> >          NGINX1    tx   bcast     --        0        0
> >          NGINX1    tx      sw     --    9.26K  692.00K
> >          NGINX1    rx   local     --   26.00K 216.32M
> >          NGINX1    rx   bcast     --        0        0
> >          NGINX1    rx      sw     --        0        0
> >          NGINX1    tx   bcast     --        0        0
> >          NGINX1    tx      sw     --    7.01K  531.38K
> >          NGINX1    rx   local     --   30.65K 253.73M
> >          NGINX1    rx   bcast     --        0        0
> >          NGINX1    rx      sw     --        0        0
> >          NGINX1    tx   bcast     --        0        0
> >          NGINX1    tx      sw     --    8.95K  669.32K
> >          NGINX1    rx   local     --   29.10K 241.15M
> >
> >
> >On the other NGZ I receive 250MB/s ????
> >
> >
> >- So my question is how comes that the speed is equal to Lan 100MB/s on
> >
> >OmniOSce but i receive 250MB/s ?
> >
> >- Why is etherstub so slow if infiniband speed is 40GB/s ???
> >
> >
> >I'm very confused right now ...
> >
> >
> >And want to know for sure how to understand and see this in the right
> >way, because this customer will be the first customer from my who gonna
> >
> >switch complety over to OmniOSce on production and because this
> >customer
> >is one or the biggest company's in Belgium I really don't want to mess
> >up !!!
> >
> >
> >So any help and clarification will be highly appreciate !!!
> >
> >
> >Thank you very much.
> >
> >
> >Kind Regards,
> >
> >
> >Dirk
>
> I am not sure where the infiniband claim comes from, but copying data disk
> to disk, you involve the slow layers like disk, skewed by faster layers
> like cache of already-read data and delayed writes :)
>
> If you have a wide pipe that you may fill, it doesn't mean you do have the
> means to fill it with a few disks.
>
> To estimate the speeds, try pure UDP streams from process to process (no
> disk), large-packet floodping, etc.
>
> I believe etherstub is not constrained artificially, and defaults to jumbo
> frames. Going to LAN and back can in fact use external hardware (IIRC there
> may be a system option to disable that, not sure) and so is constrained by
> that.
>
> Jim
> --
> Typos courtesy of K-9 Mail on my Android
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>



-- 
Ian Kaufman
Research Systems Administrator
UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/0b30fd66/attachment.html>

From dirk.willems at exitas.be  Thu Sep 14 16:13:30 2017
From: dirk.willems at exitas.be (Dirk Willems)
Date: Thu, 14 Sep 2017 18:13:30 +0200
Subject: [OmniOS-discuss] questions
In-Reply-To: <CAPJtH1h9dfpi3OmiH7A+uu2D0am_4XhOiRwZBOvQC=Y_=FRTvQ@mail.gmail.com>
References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be>
	<7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru>
	<CAPJtH1h9dfpi3OmiH7A+uu2D0am_4XhOiRwZBOvQC=Y_=FRTvQ@mail.gmail.com>
Message-ID: <1402925b-54f1-607c-8f04-f34be61946cc@exitas.be>

Thank you all,  the water is already clearing up :)


So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s Gbps why 
they not take a standaard and set everything in GB/s or MB/s ?

A lot of people make a lot of mistakes between them, me too ...

If it is 40 Gbps a factor of 8 then we theoretical have max 5 GB/s 
throughput.

Little difference 40 or 5 :)

So Ian you have the full blow with 36Gbps very cool looks more like it  :)

Did I play with the frame size, not really sure what you mean by that 
sorry but I think its default on 9000

Backend_Switch0 etherstub 9000 up


Do understand that if we use UDP streams from process to process it will 
be much quicker over the etherstub gonna need more test to do.

We used for a customer Mbuffer with zfs send over Lan that is also very 
quick sometimes I also use it at my home very good prog.

But still do not understand how it is that I copy from 1 NGZ with 
100MB/s, I receive on the other NGZ 250MB/s  very strange ?


the command dlstat difference between OmniOSce and Solaris ?

RBYTES => receiving

OBYTES => sending

root at test2:~# dlstat -i 2
 >
 >  LINK    IPKTS   RBYTES    OPKTS   OBYTES
 >            net1   25.76K 185.14M 10.08K    2.62M
 >            net1   27.04K  187.16M   11.23K    3.22M



BYTES => receiving and sending ?

But then still if the copy is not running I have 0 so doesn't explain 
why I see 216 MB where come the rest of the 116 MB from is it compression ?

root at NGINX:/root# dlstat show-link NGINX1 -i 2
 >
 >  LINK  TYPE      ID  INDEX     PKTS    BYTES
 >          NGINX1    rx   bcast     --        0        0
 >          NGINX1    rx      sw     --        0        0
 >          NGINX1    tx   bcast     --        0        0
 >          NGINX1    tx      sw     --    9.26K  692.00K
 >          NGINX1    rx   local     --   26.00K 216.32M


Thank you all for your feedback much appreciations !


Kind Regards,


Dirk



On 14-09-17 17:07, Ian Kaufman wrote:
> Some other things you need to take into account:
>
> QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 
> difference. That is also a theoretical maximum throughput, there is 
> some overhead. In reality, you will never see 40Gbps.
>
> My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with DDR 
> (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops the 
> theoretical max rates to 18Gbps and 36Gbps respectively.
>
> If you are getting 185MB/s, you are seeing 1.48Gbps.
>
> Keep your B's and b's straight. Did you play with your frame size at all?
>
> Ian
>
> On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov <jimklimov at cos.ru 
> <mailto:jimklimov at cos.ru>> wrote:
>
>     On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems
>     <dirk.willems at exitas.be <mailto:dirk.willems at exitas.be>> wrote:
>     >Hello,
>     >
>     >
>     >I'm trying to understand something let me explain.
>     >
>     >
>     >Oracle always told to me that if you create a etherstub switch it has
>     >infiniband speed 40GB/s.
>     >
>     >But I have a customer running on Solaris (Yeah I know but let me
>     >explain) who is copy from 1 NGZ to another NGZ on the same GZ
>     over Lan
>     >(I know told him to to use etherstub).
>     >
>     >The copy witch is performed for a Oracle database with sql
>     command, the
>     >
>     >DBA witch have 5 streams say it's waiting on the disk, the disk
>     are 50
>     >-
>     >60 % busy the speed is 30 mb/s.
>     >
>     >
>     >So I did some test just to see and understand if it's the database or
>     >the system, but with doing my tests I get very confused ???
>     >
>     >
>     >On another Solaris at my work copy over etherstub switch => copy
>     speed
>     >is 185MB/s expected much more of infiniband speed ???
>     >
>     >
>     >root at test1:/export/home/Admin# scp test10G
>     >Admin at 192.168.1.2:/export/home/Admin/
>     >Password:
>     >test10G              100%
>     >|****************************************************************|
>     >10240
>     >MB    00:59
>     >
>     >
>     >root at test2:~# dlstat -i 2
>     >
>     >  LINK    IPKTS   RBYTES    OPKTS   OBYTES
>     >            net1   25.76K 185.14M 10.08K    2.62M
>     >            net1   27.04K  187.16M   11.23K    3.22M
>     >            net1   26.97K  186.37M   11.24K    3.23M
>     >            net1   26.63K  187.67M   10.82K    2.99M
>     >            net1   27.94K  186.65M   12.17K    3.75M
>     >            net1   27.45K  187.46M   11.70K    3.47M
>     >            net1   26.01K  181.95M   10.63K    2.99M
>     >            net1   27.95K  188.19M   12.14K    3.69M
>     >            net1   27.91K  188.36M   12.08K    3.64M
>     >
>     >The disks are all separate luns with all separated pools => disk
>     are 20
>     >
>     >- 30% busy
>     >
>     >
>     >On my OmniOSce at my lab over etherstub
>     >
>     >
>     >root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/
>     >Password:
>     >test10G 76% 7853MB 116.4MB/s
>     >
>     >
>     >=> copy is 116.4 MB/s => expected much more from infiniband speed is
>     >just the same as Lan ???
>     >
>     >
>     >Is not that my disk can not follow 17% busy there sleeping ...
>     >
>     >    extended device statistics
>     >     r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
>     >     0,0  248,4    0,0    2,1  0,0  1,3    0,0 5,3   0 102 c1
>     >    0,0   37,5    0,0    0,7  0,0  0,2    0,0 4,7   0  17 c1t0d0 =>
>     >rpool
>     >    0,0   38,5    0,0    0,7  0,0  0,2    0,0 4,9   0  17 c1t1d0 =>
>     >rpool
>     >    0,0   40,5    0,0    0,1  0,0  0,2    0,0 5,6   0  17 c1t2d0 =>
>     >data pool
>     >    0,0   43,5    0,0    0,2  0,0  0,2    0,0 5,4   0  17 c1t3d0 =>
>     >data pool
>     >    0,0   44,5    0,0    0,2  0,0  0,2    0,0 5,5   0  18 c1t4d0 =>
>     >data pool
>     >    0,0   44,0    0,0    0,2  0,0  0,2    0,0 5,4   0  17 c1t5d0 =>
>     >data pool
>     >     0,0   76,0    0,0    1,5  7,4  0,4   97,2 4,9  14  18 rpool
>     >     0,0  172,4    0,0    0,6  2,0  0,9   11,4 5,5  12  20 DATA
>     >
>     >
>     >
>     >root at NGINX:/root# dlstat show-link NGINX1 -i 2
>     >
>     >  LINK  TYPE      ID  INDEX     PKTS    BYTES
>     >          NGINX1    rx   bcast     --        0   0
>     >          NGINX1    rx      sw     --        0   0
>     >          NGINX1    tx   bcast     --        0   0
>     >          NGINX1    tx      sw     --    9.26K 692.00K
>     >          NGINX1    rx   local     --   26.00K 216.32M
>     >          NGINX1    rx   bcast     --        0   0
>     >          NGINX1    rx      sw     --        0   0
>     >          NGINX1    tx   bcast     --        0   0
>     >          NGINX1    tx      sw     --    7.01K 531.38K
>     >          NGINX1    rx   local     --   30.65K 253.73M
>     >          NGINX1    rx   bcast     --        0   0
>     >          NGINX1    rx      sw     --        0   0
>     >          NGINX1    tx   bcast     --        0   0
>     >          NGINX1    tx      sw     --    8.95K 669.32K
>     >          NGINX1    rx   local     --   29.10K 241.15M
>     >
>     >
>     >On the other NGZ I receive 250MB/s ????
>     >
>     >
>     >- So my question is how comes that the speed is equal to Lan
>     100MB/s on
>     >
>     >OmniOSce but i receive 250MB/s ?
>     >
>     >- Why is etherstub so slow if infiniband speed is 40GB/s ???
>     >
>     >
>     >I'm very confused right now ...
>     >
>     >
>     >And want to know for sure how to understand and see this in the right
>     >way, because this customer will be the first customer from my who
>     gonna
>     >
>     >switch complety over to OmniOSce on production and because this
>     >customer
>     >is one or the biggest company's in Belgium I really don't want to
>     mess
>     >up !!!
>     >
>     >
>     >So any help and clarification will be highly appreciate !!!
>     >
>     >
>     >Thank you very much.
>     >
>     >
>     >Kind Regards,
>     >
>     >
>     >Dirk
>
>     I am not sure where the infiniband claim comes from, but copying
>     data disk to disk, you involve the slow layers like disk, skewed
>     by faster layers like cache of already-read data and delayed writes :)
>
>     If you have a wide pipe that you may fill, it doesn't mean you do
>     have the means to fill it with a few disks.
>
>     To estimate the speeds, try pure UDP streams from process to
>     process (no disk), large-packet floodping, etc.
>
>     I believe etherstub is not constrained artificially, and defaults
>     to jumbo frames. Going to LAN and back can in fact use external
>     hardware (IIRC there may be a system option to disable that, not
>     sure) and so is constrained by that.
>
>     Jim
>     --
>     Typos courtesy of K-9 Mail on my Android
>     _______________________________________________
>     OmniOS-discuss mailing list
>     OmniOS-discuss at lists.omniti.com
>     <mailto:OmniOS-discuss at lists.omniti.com>
>     http://lists.omniti.com/mailman/listinfo/omnios-discuss
>     <http://lists.omniti.com/mailman/listinfo/omnios-discuss>
>
>
>
>
> -- 
> Ian Kaufman
> Research Systems Administrator
> UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

-- 
	Dirk Willems
System Engineer


+32 (0)3 443 12 38
Dirk.Willems at exitas.be <mailto:Dirk.Willems at exitas.be>

Quality. Passion. Personality

www.exitas.be <http://www.exitas.be/> | Veldkant 31 | 2550 Kontich

Illumos OmniOS Installation and Configuration Implementation Specialist.
Oracle Solaris 11 Installation and Configuration Certified 
Implementation Specialist. 	

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/c02266f0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SmallPhoenixLogotypeRGB.png
Type: image/png
Size: 4648 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/c02266f0/attachment-0001.png>

From ikaufman at eng.ucsd.edu  Thu Sep 14 16:26:27 2017
From: ikaufman at eng.ucsd.edu (Ian Kaufman)
Date: Thu, 14 Sep 2017 09:26:27 -0700
Subject: [OmniOS-discuss] questions
In-Reply-To: <1402925b-54f1-607c-8f04-f34be61946cc@exitas.be>
References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be>
	<7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru>
	<CAPJtH1h9dfpi3OmiH7A+uu2D0am_4XhOiRwZBOvQC=Y_=FRTvQ@mail.gmail.com>
	<1402925b-54f1-607c-8f04-f34be61946cc@exitas.be>
Message-ID: <CAPJtH1iYYZiJjptT3G_LWpR63P480YwObAdO+ZVyMSc=8n+L7g@mail.gmail.com>

Networking has always used *bps - that's been the standard for many years.
Megabits, Gigabits ...

Disk tools have always measured in bytes since that is how the capacity is
defined.

How did you create your etherstub? I know you can set a maxbw (maximum
bandiwdth), but I don't know what the default behavior is.

Ian



On Thu, Sep 14, 2017 at 9:13 AM, Dirk Willems <dirk.willems at exitas.be>
wrote:

> Thank you all,  the water is already clearing up :)
>
>
> So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s Gbps why they
> not take a standaard and set everything in GB/s or MB/s ?
>
> A lot of people make a lot of mistakes between them, me too ...
>
> If it is 40 Gbps a factor of 8 then we theoretical have max 5 GB/s
> throughput.
>
> Little difference 40 or 5 :)
>
> So Ian you have the full blow with 36Gbps very cool looks more like it  :)
>
> Did I play with the frame size, not really sure what you mean by that
> sorry but I think its default on 9000
>
> Backend_Switch0 etherstub 9000 up
>
>
> Do understand that if we use UDP streams from process to process it will
> be much quicker over the etherstub gonna need more test to do.
>
> We used for a customer Mbuffer with zfs send over Lan that is also very
> quick sometimes I also use it at my home very good prog.
>
> But still do not understand how it is that I copy from 1 NGZ with 100MB/s,
> I receive on the other NGZ 250MB/s  very strange ?
>
>
> the command dlstat difference between OmniOSce and Solaris ?
>
> RBYTES => receiving
>
> OBYTES => sending
>
> root at test2:~# dlstat -i 2
> >
> >  LINK    IPKTS   RBYTES    OPKTS   OBYTES
> >            net1   25.76K 185.14M 10.08K    2.62M
> >            net1   27.04K  187.16M   11.23K    3.22M
>
>
>
> BYTES => receiving and sending ?
>
> But then still if the copy is not running I have 0 so doesn't explain why
> I see 216 MB where come the rest of the 116 MB from is it compression ?
>
> root at NGINX:/root# dlstat show-link NGINX1 -i 2
> >
> >  LINK  TYPE      ID  INDEX     PKTS    BYTES
> >          NGINX1    rx   bcast     --        0        0
> >          NGINX1    rx      sw     --        0        0
> >          NGINX1    tx   bcast     --        0        0
> >          NGINX1    tx      sw     --    9.26K  692.00K
> >          NGINX1    rx   local     --   26.00K 216.32M
>
>
> Thank you all for your feedback much appreciations !
>
>
> Kind Regards,
>
>
> Dirk
>
>
>
> On 14-09-17 17:07, Ian Kaufman wrote:
>
> Some other things you need to take into account:
>
> QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 difference.
> That is also a theoretical maximum throughput, there is some overhead. In
> reality, you will never see 40Gbps.
>
> My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with DDR
> (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops the
> theoretical max rates to 18Gbps and 36Gbps respectively.
>
> If you are getting 185MB/s, you are seeing 1.48Gbps.
>
> Keep your B's and b's straight. Did you play with your frame size at all?
>
> Ian
>
> On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov <jimklimov at cos.ru> wrote:
>
>> On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems <
>> dirk.willems at exitas.be> wrote:
>> >Hello,
>> >
>> >
>> >I'm trying to understand something let me explain.
>> >
>> >
>> >Oracle always told to me that if you create a etherstub switch it has
>> >infiniband speed 40GB/s.
>> >
>> >But I have a customer running on Solaris (Yeah I know but let me
>> >explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan
>> >(I know told him to to use etherstub).
>> >
>> >The copy witch is performed for a Oracle database with sql command, the
>> >
>> >DBA witch have 5 streams say it's waiting on the disk, the disk are 50
>> >-
>> >60 % busy the speed is 30 mb/s.
>> >
>> >
>> >So I did some test just to see and understand if it's the database or
>> >the system, but with doing my tests I get very confused ???
>> >
>> >
>> >On another Solaris at my work copy over etherstub switch => copy speed
>> >is 185MB/s expected much more of infiniband speed ???
>> >
>> >
>> >root at test1:/export/home/Admin# scp test10G
>> >Admin at 192.168.1.2:/export/home/Admin/
>> >Password:
>> >test10G              100%
>> >|****************************************************************|
>> >10240
>> >MB    00:59
>> >
>> >
>> >root at test2:~# dlstat -i 2
>> >
>> >  LINK    IPKTS   RBYTES    OPKTS   OBYTES
>> >            net1   25.76K 185.14M 10.08K    2.62M
>> >            net1   27.04K  187.16M   11.23K    3.22M
>> >            net1   26.97K  186.37M   11.24K    3.23M
>> >            net1   26.63K  187.67M   10.82K    2.99M
>> >            net1   27.94K  186.65M   12.17K    3.75M
>> >            net1   27.45K  187.46M   11.70K    3.47M
>> >            net1   26.01K  181.95M   10.63K    2.99M
>> >            net1   27.95K  188.19M   12.14K    3.69M
>> >            net1   27.91K  188.36M   12.08K    3.64M
>> >
>> >The disks are all separate luns with all separated pools => disk are 20
>> >
>> >- 30% busy
>> >
>> >
>> >On my OmniOSce at my lab over etherstub
>> >
>> >
>> >root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/
>> >Password:
>> >test10G 76% 7853MB 116.4MB/s
>> >
>> >
>> >=> copy is 116.4 MB/s => expected much more from infiniband speed is
>> >just the same as Lan ???
>> >
>> >
>> >Is not that my disk can not follow 17% busy there sleeping ...
>> >
>> >    extended device statistics
>> >     r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
>> >     0,0  248,4    0,0    2,1  0,0  1,3    0,0    5,3   0 102 c1
>> >    0,0   37,5    0,0    0,7  0,0  0,2    0,0    4,7   0  17 c1t0d0 =>
>> >rpool
>> >    0,0   38,5    0,0    0,7  0,0  0,2    0,0    4,9   0  17 c1t1d0 =>
>> >rpool
>> >    0,0   40,5    0,0    0,1  0,0  0,2    0,0    5,6   0  17 c1t2d0 =>
>> >data pool
>> >    0,0   43,5    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t3d0 =>
>> >data pool
>> >    0,0   44,5    0,0    0,2  0,0  0,2    0,0    5,5   0  18 c1t4d0 =>
>> >data pool
>> >    0,0   44,0    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t5d0 =>
>> >data pool
>> >     0,0   76,0    0,0    1,5  7,4  0,4   97,2    4,9  14  18 rpool
>> >     0,0  172,4    0,0    0,6  2,0  0,9   11,4    5,5  12  20 DATA
>> >
>> >
>> >
>> >root at NGINX:/root# dlstat show-link NGINX1 -i 2
>> >
>> >  LINK  TYPE      ID  INDEX     PKTS    BYTES
>> >          NGINX1    rx   bcast     --        0        0
>> >          NGINX1    rx      sw     --        0        0
>> >          NGINX1    tx   bcast     --        0        0
>> >          NGINX1    tx      sw     --    9.26K  692.00K
>> >          NGINX1    rx   local     --   26.00K 216.32M
>> >          NGINX1    rx   bcast     --        0        0
>> >          NGINX1    rx      sw     --        0        0
>> >          NGINX1    tx   bcast     --        0        0
>> >          NGINX1    tx      sw     --    7.01K  531.38K
>> >          NGINX1    rx   local     --   30.65K 253.73M
>> >          NGINX1    rx   bcast     --        0        0
>> >          NGINX1    rx      sw     --        0        0
>> >          NGINX1    tx   bcast     --        0        0
>> >          NGINX1    tx      sw     --    8.95K  669.32K
>> >          NGINX1    rx   local     --   29.10K 241.15M
>> >
>> >
>> >On the other NGZ I receive 250MB/s ????
>> >
>> >
>> >- So my question is how comes that the speed is equal to Lan 100MB/s on
>> >
>> >OmniOSce but i receive 250MB/s ?
>> >
>> >- Why is etherstub so slow if infiniband speed is 40GB/s ???
>> >
>> >
>> >I'm very confused right now ...
>> >
>> >
>> >And want to know for sure how to understand and see this in the right
>> >way, because this customer will be the first customer from my who gonna
>> >
>> >switch complety over to OmniOSce on production and because this
>> >customer
>> >is one or the biggest company's in Belgium I really don't want to mess
>> >up !!!
>> >
>> >
>> >So any help and clarification will be highly appreciate !!!
>> >
>> >
>> >Thank you very much.
>> >
>> >
>> >Kind Regards,
>> >
>> >
>> >Dirk
>>
>> I am not sure where the infiniband claim comes from, but copying data
>> disk to disk, you involve the slow layers like disk, skewed by faster
>> layers like cache of already-read data and delayed writes :)
>>
>> If you have a wide pipe that you may fill, it doesn't mean you do have
>> the means to fill it with a few disks.
>>
>> To estimate the speeds, try pure UDP streams from process to process (no
>> disk), large-packet floodping, etc.
>>
>> I believe etherstub is not constrained artificially, and defaults to
>> jumbo frames. Going to LAN and back can in fact use external hardware (IIRC
>> there may be a system option to disable that, not sure) and so is
>> constrained by that.
>>
>> Jim
>> --
>> Typos courtesy of K-9 Mail on my Android
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>
>
>
>
> --
> Ian Kaufman
> Research Systems Administrator
> UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
>
>
> _______________________________________________
> OmniOS-discuss mailing listOmniOS-discuss at lists.omniti.comhttp://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
> --
> Dirk Willems
> System Engineer
>
>
> +32 (0)3 443 12 38 <+32%203%20443%2012%2038>
> Dirk.Willems at exitas.be
>
> Quality. Passion. Personality
>
> www.exitas.be | Veldkant 31 | 2550 Kontich
>
> Illumos OmniOS Installation and Configuration Implementation Specialist.
> Oracle Solaris 11 Installation and Configuration Certified Implementation
> Specialist.
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>


-- 
Ian Kaufman
Research Systems Administrator
UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/679f2627/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SmallPhoenixLogotypeRGB.png
Type: image/png
Size: 4648 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/679f2627/attachment-0001.png>

From dirk.willems at exitas.be  Thu Sep 14 16:41:23 2017
From: dirk.willems at exitas.be (Dirk Willems)
Date: Thu, 14 Sep 2017 18:41:23 +0200
Subject: [OmniOS-discuss] questions
In-Reply-To: <CAPJtH1iYYZiJjptT3G_LWpR63P480YwObAdO+ZVyMSc=8n+L7g@mail.gmail.com>
References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be>
	<7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru>
	<CAPJtH1h9dfpi3OmiH7A+uu2D0am_4XhOiRwZBOvQC=Y_=FRTvQ@mail.gmail.com>
	<1402925b-54f1-607c-8f04-f34be61946cc@exitas.be>
	<CAPJtH1iYYZiJjptT3G_LWpR63P480YwObAdO+ZVyMSc=8n+L7g@mail.gmail.com>
Message-ID: <ebafc7b7-8fd3-d5f1-330b-dfe8454a7d52@exitas.be>

just execute => dladm create-etherstub Backend_Switch0


and having => Backend_Switch0 etherstub 9000 up


On 14-09-17 18:26, Ian Kaufman wrote:
> Networking has always used *bps - that's been the standard for many 
> years. Megabits, Gigabits ...
>
> Disk tools have always measured in bytes since that is how the 
> capacity is defined.
>
> How did you create your etherstub? I know you can set a maxbw (maximum 
> bandiwdth), but I don't know what the default behavior is.
>
> Ian
>
>
>
> On Thu, Sep 14, 2017 at 9:13 AM, Dirk Willems <dirk.willems at exitas.be 
> <mailto:dirk.willems at exitas.be>> wrote:
>
>     Thank you all,  the water is already clearing up :)
>
>
>     So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s Gbps
>     why they not take a standaard and set everything in GB/s or MB/s ?
>
>     A lot of people make a lot of mistakes between them, me too ...
>
>     If it is 40 Gbps a factor of 8 then we theoretical have max 5 GB/s
>     throughput.
>
>     Little difference 40 or 5 :)
>
>     So Ian you have the full blow with 36Gbps very cool looks more
>     like it  :)
>
>     Did I play with the frame size, not really sure what you mean by
>     that sorry but I think its default on 9000
>
>     Backend_Switch0 etherstub 9000 up
>
>
>     Do understand that if we use UDP streams from process to process
>     it will be much quicker over the etherstub gonna need more test to do.
>
>     We used for a customer Mbuffer with zfs send over Lan that is also
>     very quick sometimes I also use it at my home very good prog.
>
>     But still do not understand how it is that I copy from 1 NGZ with
>     100MB/s, I receive on the other NGZ 250MB/s very strange ?
>
>
>     the command dlstat difference between OmniOSce and Solaris ?
>
>     RBYTES => receiving
>
>     OBYTES => sending
>
>     root at test2:~# dlstat -i 2
>     >
>     >  LINK    IPKTS   RBYTES    OPKTS   OBYTES
>     >            net1   25.76K 185.14M 10.08K    2.62M
>     >            net1   27.04K  187.16M   11.23K 3.22M
>
>
>
>     BYTES => receiving and sending ?
>
>     But then still if the copy is not running I have 0 so doesn't
>     explain why I see 216 MB where come the rest of the 116 MB from is
>     it compression ?
>
>     root at NGINX:/root# dlstat show-link NGINX1 -i 2
>     >
>     >  LINK  TYPE      ID  INDEX     PKTS    BYTES
>     >          NGINX1    rx   bcast     --        0     0
>     >          NGINX1    rx      sw     --        0     0
>     >          NGINX1    tx   bcast     --        0     0
>     >          NGINX1    tx      sw     --    9.26K 692.00K
>     >          NGINX1    rx   local     --   26.00K 216.32M
>
>
>     Thank you all for your feedback much appreciations !
>
>
>     Kind Regards,
>
>
>     Dirk
>
>
>
>     On 14-09-17 17:07, Ian Kaufman wrote:
>>     Some other things you need to take into account:
>>
>>     QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8
>>     difference. That is also a theoretical maximum throughput, there
>>     is some overhead. In reality, you will never see 40Gbps.
>>
>>     My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with
>>     DDR (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops
>>     the theoretical max rates to 18Gbps and 36Gbps respectively.
>>
>>     If you are getting 185MB/s, you are seeing 1.48Gbps.
>>
>>     Keep your B's and b's straight. Did you play with your frame size
>>     at all?
>>
>>     Ian
>>
>>     On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov <jimklimov at cos.ru
>>     <mailto:jimklimov at cos.ru>> wrote:
>>
>>         On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems
>>         <dirk.willems at exitas.be <mailto:dirk.willems at exitas.be>> wrote:
>>         >Hello,
>>         >
>>         >
>>         >I'm trying to understand something let me explain.
>>         >
>>         >
>>         >Oracle always told to me that if you create a etherstub
>>         switch it has
>>         >infiniband speed 40GB/s.
>>         >
>>         >But I have a customer running on Solaris (Yeah I know but let me
>>         >explain) who is copy from 1 NGZ to another NGZ on the same
>>         GZ over Lan
>>         >(I know told him to to use etherstub).
>>         >
>>         >The copy witch is performed for a Oracle database with sql
>>         command, the
>>         >
>>         >DBA witch have 5 streams say it's waiting on the disk, the
>>         disk are 50
>>         >-
>>         >60 % busy the speed is 30 mb/s.
>>         >
>>         >
>>         >So I did some test just to see and understand if it's the
>>         database or
>>         >the system, but with doing my tests I get very confused ???
>>         >
>>         >
>>         >On another Solaris at my work copy over etherstub switch =>
>>         copy speed
>>         >is 185MB/s expected much more of infiniband speed ???
>>         >
>>         >
>>         >root at test1:/export/home/Admin# scp test10G
>>         >Admin at 192.168.1.2:/export/
>>         <mailto:Admin at 192.168.1.2:/export/>home/Admin/
>>         >Password:
>>         >test10G              100%
>>         >|****************************************************************|
>>         >10240
>>         >MB    00:59
>>         >
>>         >
>>         >root at test2:~# dlstat -i 2
>>         >
>>         >  LINK    IPKTS   RBYTES    OPKTS  OBYTES
>>         >            net1   25.76K 185.14M 10.08K    2.62M
>>         >            net1   27.04K  187.16M  11.23K    3.22M
>>         >            net1   26.97K  186.37M  11.24K    3.23M
>>         >            net1   26.63K  187.67M  10.82K    2.99M
>>         >            net1   27.94K  186.65M  12.17K    3.75M
>>         >            net1   27.45K  187.46M  11.70K    3.47M
>>         >            net1   26.01K  181.95M  10.63K    2.99M
>>         >            net1   27.95K  188.19M  12.14K    3.69M
>>         >            net1   27.91K  188.36M  12.08K    3.64M
>>         >
>>         >The disks are all separate luns with all separated pools =>
>>         disk are 20
>>         >
>>         >- 30% busy
>>         >
>>         >
>>         >On my OmniOSce at my lab over etherstub
>>         >
>>         >
>>         >root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/
>>         <mailto:witte at 192.168.20.3:/export/>home/witte/
>>         >Password:
>>         >test10G 76% 7853MB 116.4MB/s
>>         >
>>         >
>>         >=> copy is 116.4 MB/s => expected much more from infiniband
>>         speed is
>>         >just the same as Lan ???
>>         >
>>         >
>>         >Is not that my disk can not follow 17% busy there sleeping ...
>>         >
>>         >    extended device statistics
>>         >     r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w 
>>         %b device
>>         >     0,0  248,4    0,0    2,1  0,0 1,3    0,0    5,3   0 102 c1
>>         >    0,0   37,5    0,0    0,7  0,0 0,2    0,0    4,7   0  17
>>         c1t0d0 =>
>>         >rpool
>>         >    0,0   38,5    0,0    0,7  0,0 0,2    0,0    4,9   0  17
>>         c1t1d0 =>
>>         >rpool
>>         >    0,0   40,5    0,0    0,1  0,0 0,2    0,0    5,6   0  17
>>         c1t2d0 =>
>>         >data pool
>>         >    0,0   43,5    0,0    0,2  0,0 0,2    0,0    5,4   0  17
>>         c1t3d0 =>
>>         >data pool
>>         >    0,0   44,5    0,0    0,2  0,0 0,2    0,0    5,5   0  18
>>         c1t4d0 =>
>>         >data pool
>>         >    0,0   44,0    0,0    0,2  0,0 0,2    0,0    5,4   0  17
>>         c1t5d0 =>
>>         >data pool
>>         >     0,0   76,0    0,0    1,5  7,4 0,4   97,2    4,9  14  18
>>         rpool
>>         >     0,0  172,4    0,0    0,6  2,0 0,9   11,4    5,5  12  20
>>         DATA
>>         >
>>         >
>>         >
>>         >root at NGINX:/root# dlstat show-link NGINX1 -i 2
>>         >
>>         >  LINK  TYPE      ID  INDEX     PKTS   BYTES
>>         >          NGINX1    rx   bcast     --       0        0
>>         >          NGINX1    rx      sw     --       0        0
>>         >          NGINX1    tx   bcast     --       0        0
>>         >          NGINX1    tx      sw     --   9.26K  692.00K
>>         >          NGINX1    rx   local     --  26.00K 216.32M
>>         >          NGINX1    rx   bcast     --       0        0
>>         >          NGINX1    rx      sw     --       0        0
>>         >          NGINX1    tx   bcast     --       0        0
>>         >          NGINX1    tx      sw     --   7.01K  531.38K
>>         >          NGINX1    rx   local     --  30.65K 253.73M
>>         >          NGINX1    rx   bcast     --       0        0
>>         >          NGINX1    rx      sw     --       0        0
>>         >          NGINX1    tx   bcast     --       0        0
>>         >          NGINX1    tx      sw     --   8.95K  669.32K
>>         >          NGINX1    rx   local     --  29.10K 241.15M
>>         >
>>         >
>>         >On the other NGZ I receive 250MB/s ????
>>         >
>>         >
>>         >- So my question is how comes that the speed is equal to Lan
>>         100MB/s on
>>         >
>>         >OmniOSce but i receive 250MB/s ?
>>         >
>>         >- Why is etherstub so slow if infiniband speed is 40GB/s ???
>>         >
>>         >
>>         >I'm very confused right now ...
>>         >
>>         >
>>         >And want to know for sure how to understand and see this in
>>         the right
>>         >way, because this customer will be the first customer from
>>         my who gonna
>>         >
>>         >switch complety over to OmniOSce on production and because this
>>         >customer
>>         >is one or the biggest company's in Belgium I really don't
>>         want to mess
>>         >up !!!
>>         >
>>         >
>>         >So any help and clarification will be highly appreciate !!!
>>         >
>>         >
>>         >Thank you very much.
>>         >
>>         >
>>         >Kind Regards,
>>         >
>>         >
>>         >Dirk
>>
>>         I am not sure where the infiniband claim comes from, but
>>         copying data disk to disk, you involve the slow layers like
>>         disk, skewed by faster layers like cache of already-read data
>>         and delayed writes :)
>>
>>         If you have a wide pipe that you may fill, it doesn't mean
>>         you do have the means to fill it with a few disks.
>>
>>         To estimate the speeds, try pure UDP streams from process to
>>         process (no disk), large-packet floodping, etc.
>>
>>         I believe etherstub is not constrained artificially, and
>>         defaults to jumbo frames. Going to LAN and back can in fact
>>         use external hardware (IIRC there may be a system option to
>>         disable that, not sure) and so is constrained by that.
>>
>>         Jim
>>         --
>>         Typos courtesy of K-9 Mail on my Android
>>         _______________________________________________
>>         OmniOS-discuss mailing list
>>         OmniOS-discuss at lists.omniti.com
>>         <mailto:OmniOS-discuss at lists.omniti.com>
>>         http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>         <http://lists.omniti.com/mailman/listinfo/omnios-discuss>
>>
>>
>>
>>
>>     -- 
>>     Ian Kaufman
>>     Research Systems Administrator
>>     UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
>>
>>
>>     _______________________________________________
>>     OmniOS-discuss mailing list
>>     OmniOS-discuss at lists.omniti.com
>>     <mailto:OmniOS-discuss at lists.omniti.com>
>>     http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>     <http://lists.omniti.com/mailman/listinfo/omnios-discuss>
>
>     -- 
>     	Dirk Willems
>     System Engineer
>
>
>     +32 (0)3 443 12 38 <tel:+32%203%20443%2012%2038>
>     Dirk.Willems at exitas.be <mailto:Dirk.Willems at exitas.be>
>
>     Quality. Passion. Personality
>
>     www.exitas.be <http://www.exitas.be/> | Veldkant 31 | 2550 Kontich
>
>     Illumos OmniOS Installation and Configuration Implementation
>     Specialist.
>     Oracle Solaris 11 Installation and Configuration Certified
>     Implementation Specialist. 	
>
>
>     _______________________________________________
>     OmniOS-discuss mailing list
>     OmniOS-discuss at lists.omniti.com
>     <mailto:OmniOS-discuss at lists.omniti.com>
>     http://lists.omniti.com/mailman/listinfo/omnios-discuss
>     <http://lists.omniti.com/mailman/listinfo/omnios-discuss>
>
>
>
>
> -- 
> Ian Kaufman
> Research Systems Administrator
> UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu

-- 
	Dirk Willems
System Engineer


+32 (0)3 443 12 38
Dirk.Willems at exitas.be <mailto:Dirk.Willems at exitas.be>

Quality. Passion. Personality

www.exitas.be <http://www.exitas.be/> | Veldkant 31 | 2550 Kontich

Illumos OmniOS Installation and Configuration Implementation Specialist.
Oracle Solaris 11 Installation and Configuration Certified 
Implementation Specialist. 	

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/7e704b40/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SmallPhoenixLogotypeRGB.png
Type: image/png
Size: 4648 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/7e704b40/attachment-0001.png>

From ikaufman at eng.ucsd.edu  Thu Sep 14 16:46:40 2017
From: ikaufman at eng.ucsd.edu (Ian Kaufman)
Date: Thu, 14 Sep 2017 09:46:40 -0700
Subject: [OmniOS-discuss] questions
In-Reply-To: <ebafc7b7-8fd3-d5f1-330b-dfe8454a7d52@exitas.be>
References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be>
	<7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru>
	<CAPJtH1h9dfpi3OmiH7A+uu2D0am_4XhOiRwZBOvQC=Y_=FRTvQ@mail.gmail.com>
	<1402925b-54f1-607c-8f04-f34be61946cc@exitas.be>
	<CAPJtH1iYYZiJjptT3G_LWpR63P480YwObAdO+ZVyMSc=8n+L7g@mail.gmail.com>
	<ebafc7b7-8fd3-d5f1-330b-dfe8454a7d52@exitas.be>
Message-ID: <CAPJtH1hnHFF3TNtA7tWp9Wy50ym+bmHFiQvLTrRQCBZcBOTWsw@mail.gmail.com>

What about the attached vnics?

Can you do:

dladm show-linkprop vnic# for the vnics connected to the etherstub? There
may be a maxbw setting ...

On Thu, Sep 14, 2017 at 9:41 AM, Dirk Willems <dirk.willems at exitas.be>
wrote:

> just execute => dladm create-etherstub Backend_Switch0
>
>
> and having => Backend_Switch0 etherstub 9000 up
>
> On 14-09-17 18:26, Ian Kaufman wrote:
>
> Networking has always used *bps - that's been the standard for many years.
> Megabits, Gigabits ...
>
> Disk tools have always measured in bytes since that is how the capacity is
> defined.
>
> How did you create your etherstub? I know you can set a maxbw (maximum
> bandiwdth), but I don't know what the default behavior is.
>
> Ian
>
>
>
> On Thu, Sep 14, 2017 at 9:13 AM, Dirk Willems <dirk.willems at exitas.be>
> wrote:
>
>> Thank you all,  the water is already clearing up :)
>>
>>
>> So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s Gbps why they
>> not take a standaard and set everything in GB/s or MB/s ?
>>
>> A lot of people make a lot of mistakes between them, me too ...
>>
>> If it is 40 Gbps a factor of 8 then we theoretical have max 5 GB/s
>> throughput.
>>
>> Little difference 40 or 5 :)
>>
>> So Ian you have the full blow with 36Gbps very cool looks more like it  :)
>>
>> Did I play with the frame size, not really sure what you mean by that
>> sorry but I think its default on 9000
>>
>> Backend_Switch0 etherstub 9000 up
>>
>>
>> Do understand that if we use UDP streams from process to process it will
>> be much quicker over the etherstub gonna need more test to do.
>>
>> We used for a customer Mbuffer with zfs send over Lan that is also very
>> quick sometimes I also use it at my home very good prog.
>>
>> But still do not understand how it is that I copy from 1 NGZ with
>> 100MB/s, I receive on the other NGZ 250MB/s  very strange ?
>>
>>
>> the command dlstat difference between OmniOSce and Solaris ?
>>
>> RBYTES => receiving
>>
>> OBYTES => sending
>>
>> root at test2:~# dlstat -i 2
>> >
>> >  LINK    IPKTS   RBYTES    OPKTS   OBYTES
>> >            net1   25.76K 185.14M 10.08K    2.62M
>> >            net1   27.04K  187.16M   11.23K    3.22M
>>
>>
>>
>> BYTES => receiving and sending ?
>>
>> But then still if the copy is not running I have 0 so doesn't explain why
>> I see 216 MB where come the rest of the 116 MB from is it compression ?
>>
>> root at NGINX:/root# dlstat show-link NGINX1 -i 2
>> >
>> >  LINK  TYPE      ID  INDEX     PKTS    BYTES
>> >          NGINX1    rx   bcast     --        0        0
>> >          NGINX1    rx      sw     --        0        0
>> >          NGINX1    tx   bcast     --        0        0
>> >          NGINX1    tx      sw     --    9.26K  692.00K
>> >          NGINX1    rx   local     --   26.00K 216.32M
>>
>>
>> Thank you all for your feedback much appreciations !
>>
>>
>> Kind Regards,
>>
>>
>> Dirk
>>
>>
>>
>> On 14-09-17 17:07, Ian Kaufman wrote:
>>
>> Some other things you need to take into account:
>>
>> QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8 difference.
>> That is also a theoretical maximum throughput, there is some overhead. In
>> reality, you will never see 40Gbps.
>>
>> My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with DDR
>> (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops the
>> theoretical max rates to 18Gbps and 36Gbps respectively.
>>
>> If you are getting 185MB/s, you are seeing 1.48Gbps.
>>
>> Keep your B's and b's straight. Did you play with your frame size at all?
>>
>> Ian
>>
>> On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov <jimklimov at cos.ru> wrote:
>>
>>> On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems <
>>> dirk.willems at exitas.be> wrote:
>>> >Hello,
>>> >
>>> >
>>> >I'm trying to understand something let me explain.
>>> >
>>> >
>>> >Oracle always told to me that if you create a etherstub switch it has
>>> >infiniband speed 40GB/s.
>>> >
>>> >But I have a customer running on Solaris (Yeah I know but let me
>>> >explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan
>>> >(I know told him to to use etherstub).
>>> >
>>> >The copy witch is performed for a Oracle database with sql command, the
>>> >
>>> >DBA witch have 5 streams say it's waiting on the disk, the disk are 50
>>> >-
>>> >60 % busy the speed is 30 mb/s.
>>> >
>>> >
>>> >So I did some test just to see and understand if it's the database or
>>> >the system, but with doing my tests I get very confused ???
>>> >
>>> >
>>> >On another Solaris at my work copy over etherstub switch => copy speed
>>> >is 185MB/s expected much more of infiniband speed ???
>>> >
>>> >
>>> >root at test1:/export/home/Admin# scp test10G
>>> >Admin at 192.168.1.2:/export/home/Admin/
>>> >Password:
>>> >test10G              100%
>>> >|****************************************************************|
>>> >10240
>>> >MB    00:59
>>> >
>>> >
>>> >root at test2:~# dlstat -i 2
>>> >
>>> >  LINK    IPKTS   RBYTES    OPKTS   OBYTES
>>> >            net1   25.76K 185.14M 10.08K    2.62M
>>> >            net1   27.04K  187.16M   11.23K    3.22M
>>> >            net1   26.97K  186.37M   11.24K    3.23M
>>> >            net1   26.63K  187.67M   10.82K    2.99M
>>> >            net1   27.94K  186.65M   12.17K    3.75M
>>> >            net1   27.45K  187.46M   11.70K    3.47M
>>> >            net1   26.01K  181.95M   10.63K    2.99M
>>> >            net1   27.95K  188.19M   12.14K    3.69M
>>> >            net1   27.91K  188.36M   12.08K    3.64M
>>> >
>>> >The disks are all separate luns with all separated pools => disk are 20
>>> >
>>> >- 30% busy
>>> >
>>> >
>>> >On my OmniOSce at my lab over etherstub
>>> >
>>> >
>>> >root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/
>>> >Password:
>>> >test10G 76% 7853MB 116.4MB/s
>>> >
>>> >
>>> >=> copy is 116.4 MB/s => expected much more from infiniband speed is
>>> >just the same as Lan ???
>>> >
>>> >
>>> >Is not that my disk can not follow 17% busy there sleeping ...
>>> >
>>> >    extended device statistics
>>> >     r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
>>> >     0,0  248,4    0,0    2,1  0,0  1,3    0,0    5,3   0 102 c1
>>> >    0,0   37,5    0,0    0,7  0,0  0,2    0,0    4,7   0  17 c1t0d0 =>
>>> >rpool
>>> >    0,0   38,5    0,0    0,7  0,0  0,2    0,0    4,9   0  17 c1t1d0 =>
>>> >rpool
>>> >    0,0   40,5    0,0    0,1  0,0  0,2    0,0    5,6   0  17 c1t2d0 =>
>>> >data pool
>>> >    0,0   43,5    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t3d0 =>
>>> >data pool
>>> >    0,0   44,5    0,0    0,2  0,0  0,2    0,0    5,5   0  18 c1t4d0 =>
>>> >data pool
>>> >    0,0   44,0    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t5d0 =>
>>> >data pool
>>> >     0,0   76,0    0,0    1,5  7,4  0,4   97,2    4,9  14  18 rpool
>>> >     0,0  172,4    0,0    0,6  2,0  0,9   11,4    5,5  12  20 DATA
>>> >
>>> >
>>> >
>>> >root at NGINX:/root# dlstat show-link NGINX1 -i 2
>>> >
>>> >  LINK  TYPE      ID  INDEX     PKTS    BYTES
>>> >          NGINX1    rx   bcast     --        0        0
>>> >          NGINX1    rx      sw     --        0        0
>>> >          NGINX1    tx   bcast     --        0        0
>>> >          NGINX1    tx      sw     --    9.26K  692.00K
>>> >          NGINX1    rx   local     --   26.00K 216.32M
>>> >          NGINX1    rx   bcast     --        0        0
>>> >          NGINX1    rx      sw     --        0        0
>>> >          NGINX1    tx   bcast     --        0        0
>>> >          NGINX1    tx      sw     --    7.01K  531.38K
>>> >          NGINX1    rx   local     --   30.65K 253.73M
>>> >          NGINX1    rx   bcast     --        0        0
>>> >          NGINX1    rx      sw     --        0        0
>>> >          NGINX1    tx   bcast     --        0        0
>>> >          NGINX1    tx      sw     --    8.95K  669.32K
>>> >          NGINX1    rx   local     --   29.10K 241.15M
>>> >
>>> >
>>> >On the other NGZ I receive 250MB/s ????
>>> >
>>> >
>>> >- So my question is how comes that the speed is equal to Lan 100MB/s on
>>> >
>>> >OmniOSce but i receive 250MB/s ?
>>> >
>>> >- Why is etherstub so slow if infiniband speed is 40GB/s ???
>>> >
>>> >
>>> >I'm very confused right now ...
>>> >
>>> >
>>> >And want to know for sure how to understand and see this in the right
>>> >way, because this customer will be the first customer from my who gonna
>>> >
>>> >switch complety over to OmniOSce on production and because this
>>> >customer
>>> >is one or the biggest company's in Belgium I really don't want to mess
>>> >up !!!
>>> >
>>> >
>>> >So any help and clarification will be highly appreciate !!!
>>> >
>>> >
>>> >Thank you very much.
>>> >
>>> >
>>> >Kind Regards,
>>> >
>>> >
>>> >Dirk
>>>
>>> I am not sure where the infiniband claim comes from, but copying data
>>> disk to disk, you involve the slow layers like disk, skewed by faster
>>> layers like cache of already-read data and delayed writes :)
>>>
>>> If you have a wide pipe that you may fill, it doesn't mean you do have
>>> the means to fill it with a few disks.
>>>
>>> To estimate the speeds, try pure UDP streams from process to process (no
>>> disk), large-packet floodping, etc.
>>>
>>> I believe etherstub is not constrained artificially, and defaults to
>>> jumbo frames. Going to LAN and back can in fact use external hardware (IIRC
>>> there may be a system option to disable that, not sure) and so is
>>> constrained by that.
>>>
>>> Jim
>>> --
>>> Typos courtesy of K-9 Mail on my Android
>>> _______________________________________________
>>> OmniOS-discuss mailing list
>>> OmniOS-discuss at lists.omniti.com
>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>>
>>
>>
>>
>> --
>> Ian Kaufman
>> Research Systems Administrator
>> UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
>>
>>
>> _______________________________________________
>> OmniOS-discuss mailing listOmniOS-discuss at lists.omniti.comhttp://lists.omniti.com/mailman/listinfo/omnios-discuss
>>
>>
>> --
>> Dirk Willems
>> System Engineer
>>
>>
>> +32 (0)3 443 12 38 <+32%203%20443%2012%2038>
>> Dirk.Willems at exitas.be
>>
>> Quality. Passion. Personality
>>
>> www.exitas.be | Veldkant 31
>> <https://maps.google.com/?q=Veldkant+31&entry=gmail&source=g> | 2550
>> Kontich
>>
>> Illumos OmniOS Installation and Configuration Implementation Specialist.
>> Oracle Solaris 11 Installation and Configuration Certified Implementation
>> Specialist.
>>
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>
>>
>
>
> --
> Ian Kaufman
> Research Systems Administrator
> UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
>
>
> --
> Dirk Willems
> System Engineer
>
>
> +32 (0)3 443 12 38 <+32%203%20443%2012%2038>
> Dirk.Willems at exitas.be
>
> Quality. Passion. Personality
>
> www.exitas.be | Veldkant 31
> <https://maps.google.com/?q=Veldkant+31&entry=gmail&source=g> | 2550
> Kontich
>
> Illumos OmniOS Installation and Configuration Implementation Specialist.
> Oracle Solaris 11 Installation and Configuration Certified Implementation
> Specialist.
>



-- 
Ian Kaufman
Research Systems Administrator
UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/8be18509/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SmallPhoenixLogotypeRGB.png
Type: image/png
Size: 4648 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/8be18509/attachment-0001.png>

From dirk.willems at exitas.be  Thu Sep 14 16:55:51 2017
From: dirk.willems at exitas.be (Dirk Willems)
Date: Thu, 14 Sep 2017 18:55:51 +0200
Subject: [OmniOS-discuss] questions
In-Reply-To: <CAPJtH1hnHFF3TNtA7tWp9Wy50ym+bmHFiQvLTrRQCBZcBOTWsw@mail.gmail.com>
References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be>
	<7CCF4F53-0A02-4FA1-893D-CF4F6F4BB905@cos.ru>
	<CAPJtH1h9dfpi3OmiH7A+uu2D0am_4XhOiRwZBOvQC=Y_=FRTvQ@mail.gmail.com>
	<1402925b-54f1-607c-8f04-f34be61946cc@exitas.be>
	<CAPJtH1iYYZiJjptT3G_LWpR63P480YwObAdO+ZVyMSc=8n+L7g@mail.gmail.com>
	<ebafc7b7-8fd3-d5f1-330b-dfe8454a7d52@exitas.be>
	<CAPJtH1hnHFF3TNtA7tWp9Wy50ym+bmHFiQvLTrRQCBZcBOTWsw@mail.gmail.com>
Message-ID: <c1ab4251-f71f-44c1-f21f-d942a7c27fdd@exitas.be>

Actuality have it test with this etherstub => NGINX_Switch0

But all the etherstub switches are created excellently the same

But idd on the etherstub the mtu on Default is standing on 1500 but the 
value is on 9000 so 9000 is in use ?


root at OmniOS:/root# dladm show-linkprop NGINX_Switch0
LINK         PROPERTY        PERM VALUE          DEFAULT POSSIBLE
NGINX_Switch0 autopush       rw   --             --             --
NGINX_Switch0 zone           rw   --             --             --
NGINX_Switch0 state          r-   up             up up,down
NGINX_Switch0 mtu            rw   9000           1500 576-9000
NGINX_Switch0 maxbw          rw   --             --             --
NGINX_Switch0 cpus           rw   --             --             --
NGINX_Switch0 cpus-effective r-   --             --             --
NGINX_Switch0 pool           rw   --             --             --
NGINX_Switch0 pool-effective r-   --             --             --
NGINX_Switch0 priority       rw   high           high low,medium,high
NGINX_Switch0 forward        rw   1              1 1,0
NGINX_Switch0 default_tag    rw   1              1              --
NGINX_Switch0 learn_limit    rw   1000           1000           --
NGINX_Switch0 learn_decay    rw   200            200            --
NGINX_Switch0 stp            rw   1              1 1,0
NGINX_Switch0 stp_priority   rw   128            128            --
NGINX_Switch0 stp_cost       rw   auto           auto           --
NGINX_Switch0 stp_edge       rw   1              1 1,0
NGINX_Switch0 stp_p2p        rw   auto           auto true,false,auto
NGINX_Switch0 stp_mcheck     rw   0              0 1,0
NGINX_Switch0 protection     rw   --             -- mac-nospoof,
restricted,
ip-nospoof,
dhcp-nospoof
NGINX_Switch0 allowed-ips    rw   --             --             --
NGINX_Switch0 allowed-dhcp-cids rw --            --             --
NGINX_Switch0 rxrings        rw   --             --             --
NGINX_Switch0 rxrings-effective r- --            --             --
NGINX_Switch0 txrings        rw   --             --             --
NGINX_Switch0 txrings-effective r- --            --             --
NGINX_Switch0 txrings-available r- 0             --             --
NGINX_Switch0 rxrings-available r- 0             --             --
NGINX_Switch0 rxhwclnt-available r- 0            --             --
NGINX_Switch0 txhwclnt-available r- 0            --             --
root at OmniOS:/root# dladm show-linkprop NGINX1
LINK         PROPERTY        PERM VALUE          DEFAULT POSSIBLE
NGINX1       autopush        rw   --             --             --
NGINX1       zone            rw   NGINX          --             --
NGINX1       state           r-   up             up up,down
NGINX1       mtu             rw   9000           9000 576-9000
NGINX1       secondary-macs  rw   --             --             --
NGINX1       maxbw           rw   --             --             --
NGINX1       cpus            rw   --             --             --
NGINX1       cpus-effective  r-   0,23           --             --
NGINX1       pool            rw   --             --             --
NGINX1       pool-effective  r-   --             --             --
NGINX1       priority        rw   high           high low,medium,high
NGINX1       tagmode         rw   vlanonly       vlanonly normal,vlanonly
NGINX1       protection      rw   ip-nospoof     -- mac-nospoof,
restricted,
ip-nospoof,
dhcp-nospoof
NGINX1       promisc-filtered rw  on             on off,on
NGINX1       allowed-ips     rw   192.168.20.3/32 --            --
NGINX1       allowed-dhcp-cids rw --             --             --
NGINX1       rxrings         rw   --             --             --
NGINX1       rxrings-effective r- --             --             --
NGINX1       txrings         rw   --             --             --
NGINX1       txrings-effective r- --             --             --
NGINX1       txrings-available r- 0              --             --
NGINX1       rxrings-available r- 0              --             --
NGINX1       rxhwclnt-available r- 0             --             --
NGINX1       txhwclnt-available r- 0             --             --
root at OmniOS:/root# dladm show-linkprop GNUHealth0
LINK         PROPERTY        PERM VALUE          DEFAULT POSSIBLE
GNUHealth0   autopush        rw   --             --             --
GNUHealth0   zone            rw   GNUHealth      --             --
GNUHealth0   state           r-   up             up up,down
GNUHealth0   mtu             rw   9000           9000 576-9000
GNUHealth0   secondary-macs  rw   --             --             --
GNUHealth0   maxbw           rw   --             --             --
GNUHealth0   cpus            rw   --             --             --
GNUHealth0   cpus-effective  r-   21-22          --             --
GNUHealth0   pool            rw   --             --             --
GNUHealth0   pool-effective  r-   --             --             --
GNUHealth0   priority        rw   high           high low,medium,high
GNUHealth0   tagmode         rw   vlanonly       vlanonly normal,vlanonly
GNUHealth0   protection      rw   ip-nospoof     -- mac-nospoof,
restricted,
ip-nospoof,
dhcp-nospoof
GNUHealth0   promisc-filtered rw  on             on off,on
GNUHealth0   allowed-ips     rw   192.168.20.5/32 --            --
GNUHealth0   allowed-dhcp-cids rw --             --             --
GNUHealth0   rxrings         rw   --             --             --
GNUHealth0   rxrings-effective r- --             --             --
GNUHealth0   txrings         rw   --             --             --
GNUHealth0   txrings-effective r- --             --             --
GNUHealth0   txrings-available r- 0              --             --
GNUHealth0   rxrings-available r- 0              --             --
GNUHealth0   rxhwclnt-available r- 0             --             --
GNUHealth0   txhwclnt-available r- 0             --             --
root at OmniOS:/root#


On 14-09-17 18:46, Ian Kaufman wrote:
> What about the attached vnics?
>
> Can you do:
>
> dladm show-linkprop vnic# for the vnics connected to the etherstub? 
> There may be a maxbw setting ...
>
> On Thu, Sep 14, 2017 at 9:41 AM, Dirk Willems <dirk.willems at exitas.be 
> <mailto:dirk.willems at exitas.be>> wrote:
>
>     just execute => dladm create-etherstub Backend_Switch0
>
>
>     and having => Backend_Switch0 etherstub 9000 up
>
>
>     On 14-09-17 18:26, Ian Kaufman wrote:
>>     Networking has always used *bps - that's been the standard for
>>     many years. Megabits, Gigabits ...
>>
>>     Disk tools have always measured in bytes since that is how the
>>     capacity is defined.
>>
>>     How did you create your etherstub? I know you can set a maxbw
>>     (maximum bandiwdth), but I don't know what the default behavior is.
>>
>>     Ian
>>
>>
>>
>>     On Thu, Sep 14, 2017 at 9:13 AM, Dirk Willems
>>     <dirk.willems at exitas.be <mailto:dirk.willems at exitas.be>> wrote:
>>
>>         Thank you all,  the water is already clearing up :)
>>
>>
>>         So infiniband is 40 Gbps an not 40GB/s, very confusing GB/s
>>         Gbps why they not take a standaard and set everything in GB/s
>>         or MB/s ?
>>
>>         A lot of people make a lot of mistakes between them, me too ...
>>
>>         If it is 40 Gbps a factor of 8 then we theoretical have max 5
>>         GB/s throughput.
>>
>>         Little difference 40 or 5 :)
>>
>>         So Ian you have the full blow with 36Gbps very cool looks
>>         more like it  :)
>>
>>         Did I play with the frame size, not really sure what you mean
>>         by that sorry but I think its default on 9000
>>
>>         Backend_Switch0 etherstub 9000 up
>>
>>
>>         Do understand that if we use UDP streams from process to
>>         process it will be much quicker over the etherstub gonna need
>>         more test to do.
>>
>>         We used for a customer Mbuffer with zfs send over Lan that is
>>         also very quick sometimes I also use it at my home very good
>>         prog.
>>
>>         But still do not understand how it is that I copy from 1 NGZ
>>         with 100MB/s, I receive on the other NGZ 250MB/s  very strange ?
>>
>>
>>         the command dlstat difference between OmniOSce and Solaris ?
>>
>>         RBYTES => receiving
>>
>>         OBYTES => sending
>>
>>         root at test2:~# dlstat -i 2
>>         >
>>         >  LINK    IPKTS   RBYTES    OPKTS  OBYTES
>>         >            net1   25.76K 185.14M 10.08K    2.62M
>>         >            net1   27.04K  187.16M  11.23K    3.22M
>>
>>
>>
>>         BYTES => receiving and sending ?
>>
>>         But then still if the copy is not running I have 0 so doesn't
>>         explain why I see 216 MB where come the rest of the 116 MB
>>         from is it compression ?
>>
>>         root at NGINX:/root# dlstat show-link NGINX1 -i 2
>>         >
>>         >  LINK  TYPE      ID  INDEX  PKTS    BYTES
>>         >          NGINX1    rx   bcast  --        0        0
>>         >          NGINX1    rx      sw  --        0        0
>>         >          NGINX1    tx   bcast  --        0        0
>>         >          NGINX1    tx      sw  --    9.26K  692.00K
>>         >          NGINX1    rx   local  --   26.00K 216.32M
>>
>>
>>         Thank you all for your feedback much appreciations !
>>
>>
>>         Kind Regards,
>>
>>
>>         Dirk
>>
>>
>>
>>         On 14-09-17 17:07, Ian Kaufman wrote:
>>>         Some other things you need to take into account:
>>>
>>>         QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8
>>>         difference. That is also a theoretical maximum throughput,
>>>         there is some overhead. In reality, you will never see 40Gbps.
>>>
>>>         My system tested out at 6Gbps - 8Gbps using NFS over IPoIB,
>>>         with DDR (20Gbps) nodes and a QDR (40Gbps) storage server.
>>>         IPoIB drops the theoretical max rates to 18Gbps and 36Gbps
>>>         respectively.
>>>
>>>         If you are getting 185MB/s, you are seeing 1.48Gbps.
>>>
>>>         Keep your B's and b's straight. Did you play with your frame
>>>         size at all?
>>>
>>>         Ian
>>>
>>>         On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov
>>>         <jimklimov at cos.ru <mailto:jimklimov at cos.ru>> wrote:
>>>
>>>             On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems
>>>             <dirk.willems at exitas.be <mailto:dirk.willems at exitas.be>>
>>>             wrote:
>>>             >Hello,
>>>             >
>>>             >
>>>             >I'm trying to understand something let me explain.
>>>             >
>>>             >
>>>             >Oracle always told to me that if you create a etherstub
>>>             switch it has
>>>             >infiniband speed 40GB/s.
>>>             >
>>>             >But I have a customer running on Solaris (Yeah I know
>>>             but let me
>>>             >explain) who is copy from 1 NGZ to another NGZ on the
>>>             same GZ over Lan
>>>             >(I know told him to to use etherstub).
>>>             >
>>>             >The copy witch is performed for a Oracle database with
>>>             sql command, the
>>>             >
>>>             >DBA witch have 5 streams say it's waiting on the disk,
>>>             the disk are 50
>>>             >-
>>>             >60 % busy the speed is 30 mb/s.
>>>             >
>>>             >
>>>             >So I did some test just to see and understand if it's
>>>             the database or
>>>             >the system, but with doing my tests I get very confused ???
>>>             >
>>>             >
>>>             >On another Solaris at my work copy over etherstub
>>>             switch => copy speed
>>>             >is 185MB/s expected much more of infiniband speed ???
>>>             >
>>>             >
>>>             >root at test1:/export/home/Admin# scp test10G
>>>             >Admin at 192.168.1.2:/export/
>>>             <mailto:Admin at 192.168.1.2:/export/>home/Admin/
>>>             >Password:
>>>             >test10G 100%
>>>             >|****************************************************************|
>>>             >10240
>>>             >MB    00:59
>>>             >
>>>             >
>>>             >root at test2:~# dlstat -i 2
>>>             >
>>>             >  LINK    IPKTS  RBYTES    OPKTS   OBYTES
>>>             >            net1  25.76K 185.14M 10.08K 2.62M
>>>             >            net1  27.04K  187.16M   11.23K 3.22M
>>>             >            net1  26.97K  186.37M   11.24K 3.23M
>>>             >            net1  26.63K  187.67M   10.82K 2.99M
>>>             >            net1  27.94K  186.65M   12.17K 3.75M
>>>             >            net1  27.45K  187.46M   11.70K 3.47M
>>>             >            net1  26.01K  181.95M   10.63K 2.99M
>>>             >            net1  27.95K  188.19M   12.14K 3.69M
>>>             >            net1  27.91K  188.36M   12.08K 3.64M
>>>             >
>>>             >The disks are all separate luns with all separated
>>>             pools => disk are 20
>>>             >
>>>             >- 30% busy
>>>             >
>>>             >
>>>             >On my OmniOSce at my lab over etherstub
>>>             >
>>>             >
>>>             >root at GNUHealth:~# scp test10G
>>>             witte at 192.168.20.3:/export/
>>>             <mailto:witte at 192.168.20.3:/export/>home/witte/
>>>             >Password:
>>>             >test10G 76% 7853MB 116.4MB/s
>>>             >
>>>             >
>>>             >=> copy is 116.4 MB/s => expected much more from
>>>             infiniband speed is
>>>             >just the same as Lan ???
>>>             >
>>>             >
>>>             >Is not that my disk can not follow 17% busy there
>>>             sleeping ...
>>>             >
>>>             >    extended device statistics
>>>             >     r/s    w/s   Mr/s  Mw/s wait actv wsvc_t asvc_t 
>>>             %w  %b device
>>>             >     0,0  248,4    0,0   2,1  0,0  1,3    0,0 5,3   0
>>>             102 c1
>>>             >    0,0   37,5    0,0 0,7  0,0  0,2    0,0    4,7  0 
>>>             17 c1t0d0 =>
>>>             >rpool
>>>             >    0,0   38,5    0,0 0,7  0,0  0,2    0,0    4,9  0 
>>>             17 c1t1d0 =>
>>>             >rpool
>>>             >    0,0   40,5    0,0 0,1  0,0  0,2    0,0    5,6  0 
>>>             17 c1t2d0 =>
>>>             >data pool
>>>             >    0,0   43,5    0,0 0,2  0,0  0,2    0,0    5,4  0 
>>>             17 c1t3d0 =>
>>>             >data pool
>>>             >    0,0   44,5    0,0 0,2  0,0  0,2    0,0    5,5  0 
>>>             18 c1t4d0 =>
>>>             >data pool
>>>             >    0,0   44,0    0,0 0,2  0,0  0,2    0,0    5,4  0 
>>>             17 c1t5d0 =>
>>>             >data pool
>>>             >     0,0   76,0    0,0   1,5  7,4  0,4   97,2 4,9  14 
>>>             18 rpool
>>>             >     0,0  172,4    0,0   0,6  2,0  0,9   11,4 5,5  12 
>>>             20 DATA
>>>             >
>>>             >
>>>             >
>>>             >root at NGINX:/root# dlstat show-link NGINX1 -i 2
>>>             >
>>>             >  LINK  TYPE      ID INDEX     PKTS    BYTES
>>>             >          NGINX1    rx  bcast     --        0   0
>>>             >          NGINX1    rx     sw     --        0   0
>>>             >          NGINX1    tx  bcast     --        0   0
>>>             >          NGINX1    tx     sw     --    9.26K 692.00K
>>>             >          NGINX1    rx  local     --   26.00K 216.32M
>>>             >          NGINX1    rx  bcast     --        0   0
>>>             >          NGINX1    rx     sw     --        0   0
>>>             >          NGINX1    tx  bcast     --        0   0
>>>             >          NGINX1    tx     sw     --    7.01K 531.38K
>>>             >          NGINX1    rx  local     --   30.65K 253.73M
>>>             >          NGINX1    rx  bcast     --        0   0
>>>             >          NGINX1    rx     sw     --        0   0
>>>             >          NGINX1    tx  bcast     --        0   0
>>>             >          NGINX1    tx     sw     --    8.95K 669.32K
>>>             >          NGINX1    rx  local     --   29.10K 241.15M
>>>             >
>>>             >
>>>             >On the other NGZ I receive 250MB/s ????
>>>             >
>>>             >
>>>             >- So my question is how comes that the speed is equal
>>>             to Lan 100MB/s on
>>>             >
>>>             >OmniOSce but i receive 250MB/s ?
>>>             >
>>>             >- Why is etherstub so slow if infiniband speed is
>>>             40GB/s ???
>>>             >
>>>             >
>>>             >I'm very confused right now ...
>>>             >
>>>             >
>>>             >And want to know for sure how to understand and see
>>>             this in the right
>>>             >way, because this customer will be the first customer
>>>             from my who gonna
>>>             >
>>>             >switch complety over to OmniOSce on production and
>>>             because this
>>>             >customer
>>>             >is one or the biggest company's in Belgium I really
>>>             don't want to mess
>>>             >up !!!
>>>             >
>>>             >
>>>             >So any help and clarification will be highly appreciate !!!
>>>             >
>>>             >
>>>             >Thank you very much.
>>>             >
>>>             >
>>>             >Kind Regards,
>>>             >
>>>             >
>>>             >Dirk
>>>
>>>             I am not sure where the infiniband claim comes from, but
>>>             copying data disk to disk, you involve the slow layers
>>>             like disk, skewed by faster layers like cache of
>>>             already-read data and delayed writes :)
>>>
>>>             If you have a wide pipe that you may fill, it doesn't
>>>             mean you do have the means to fill it with a few disks.
>>>
>>>             To estimate the speeds, try pure UDP streams from
>>>             process to process (no disk), large-packet floodping, etc.
>>>
>>>             I believe etherstub is not constrained artificially, and
>>>             defaults to jumbo frames. Going to LAN and back can in
>>>             fact use external hardware (IIRC there may be a system
>>>             option to disable that, not sure) and so is constrained
>>>             by that.
>>>
>>>             Jim
>>>             --
>>>             Typos courtesy of K-9 Mail on my Android
>>>             _______________________________________________
>>>             OmniOS-discuss mailing list
>>>             OmniOS-discuss at lists.omniti.com
>>>             <mailto:OmniOS-discuss at lists.omniti.com>
>>>             http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>>             <http://lists.omniti.com/mailman/listinfo/omnios-discuss>
>>>
>>>
>>>
>>>
>>>         -- 
>>>         Ian Kaufman
>>>         Research Systems Administrator
>>>         UC San Diego, Jacobs School of Engineering ikaufman AT ucsd
>>>         DOT edu
>>>
>>>
>>>         _______________________________________________
>>>         OmniOS-discuss mailing list
>>>         OmniOS-discuss at lists.omniti.com
>>>         <mailto:OmniOS-discuss at lists.omniti.com>
>>>         http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>>         <http://lists.omniti.com/mailman/listinfo/omnios-discuss>
>>
>>         -- 
>>         	Dirk Willems
>>         System Engineer
>>
>>
>>         +32 (0)3 443 12 38 <tel:+32%203%20443%2012%2038>
>>         Dirk.Willems at exitas.be <mailto:Dirk.Willems at exitas.be>
>>
>>         Quality. Passion. Personality
>>
>>         www.exitas.be <http://www.exitas.be/> | Veldkant 31
>>         <https://maps.google.com/?q=Veldkant+31&entry=gmail&source=g>
>>         | 2550 Kontich
>>
>>         Illumos OmniOS Installation and Configuration Implementation
>>         Specialist.
>>         Oracle Solaris 11 Installation and Configuration Certified
>>         Implementation Specialist. 	
>>
>>
>>         _______________________________________________
>>         OmniOS-discuss mailing list
>>         OmniOS-discuss at lists.omniti.com
>>         <mailto:OmniOS-discuss at lists.omniti.com>
>>         http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>         <http://lists.omniti.com/mailman/listinfo/omnios-discuss>
>>
>>
>>
>>
>>     -- 
>>     Ian Kaufman
>>     Research Systems Administrator
>>     UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
>
>     -- 
>     	Dirk Willems
>     System Engineer
>
>
>     +32 (0)3 443 12 38 <tel:+32%203%20443%2012%2038>
>     Dirk.Willems at exitas.be <mailto:Dirk.Willems at exitas.be>
>
>     Quality. Passion. Personality
>
>     www.exitas.be <http://www.exitas.be/> | Veldkant 31
>     <https://maps.google.com/?q=Veldkant+31&entry=gmail&source=g> |
>     2550 Kontich
>
>     Illumos OmniOS Installation and Configuration Implementation
>     Specialist.
>     Oracle Solaris 11 Installation and Configuration Certified
>     Implementation Specialist. 	
>
>
>
>
> -- 
> Ian Kaufman
> Research Systems Administrator
> UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu

-- 
	Dirk Willems
System Engineer


+32 (0)3 443 12 38
Dirk.Willems at exitas.be <mailto:Dirk.Willems at exitas.be>

Quality. Passion. Personality

www.exitas.be <http://www.exitas.be/> | Veldkant 31 | 2550 Kontich

Illumos OmniOS Installation and Configuration Implementation Specialist.
Oracle Solaris 11 Installation and Configuration Certified 
Implementation Specialist. 	

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/d3d30255/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SmallPhoenixLogotypeRGB.png
Type: image/png
Size: 4648 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170914/d3d30255/attachment-0001.png>

From danmcd at kebe.com  Fri Sep 15 11:30:00 2017
From: danmcd at kebe.com (Dan McDonald)
Date: Fri, 15 Sep 2017 07:30:00 -0400
Subject: [OmniOS-discuss] Fwd: Invitation to the SmartOS/OmniOS/OI/illumos
 user meeting at Erlangen, Germany on October 12
References: <7713BD48-D528-486E-A34A-344E7C5F7371@peterkelm.com>
Message-ID: <944515EE-242E-4C4D-9029-B9A7C59FF2C5@kebe.com>

Peter asked me to forward this to the list.

Dan

=====================

Von: Peter Kelm <peter.kelm at peterkelm.com <mailto:peter.kelm at peterkelm.com>>
Betreff: Invitation to the SmartOS/OmniOS/OI/illumos user meeting at Erlangen, Germany on October 12
Datum: 15. September 2017 um 11:15:28 MESZ
An: openindiana-discuss-owner at openindiana.org <mailto:openindiana-discuss-owner at openindiana.org>, smartos-discuss at lists.smartos.org <mailto:smartos-discuss at lists.smartos.org>, omnios-discuss at lists.omniti.com <mailto:omnios-discuss at lists.omniti.com>


All,

We?d like to invite you to the SmartOS/OmniOS/OI/illumos user meeting at Erlangen on Oct 12, 2017.

Location:
  KelmMoyles office (at the IGZ Erlangen)
  Am Weichselgarten 7
  91058 Erlangen
  Germany

Date/Time:
  Oct 12, 2017, starting at 6:00pm

There is no firm agenda yet but please let us know what topics you?d be interested in to make this meeting as productive and useful as it can be. Your contributions are highly appreciated!

I?d expect that some of the questions below might come up:
- How do you use SmartOS/OmniOS/OI/illumos today or how do you intend to use it ?tomorrow??
- What are the factors that drive your interest? Where do you see strengths and weaknesses (vs. competing solutions)?
- What are the ?best practices? that you?d like to share? (E.g. how do you handle maintenance and updates for your deployment?)

Participation is of course free but we?d like to ask you to register via eMail to: illumos-meetup at kelmmoyles.com <mailto:illumos-meetup at kelmmoyles.com>

FYI: The timing coincides with the IT-SA fair at Nuremberg, so you might want to consider spending the day there before heading over to us: https://www.it-sa.de <https://www.it-sa.de/>

We are very open to additional thoughts and suggestions, so please let us know.

Looking forward to seeing you in October!

Peter

P.S. We haven?t setup a ?meetup" group yet. Any volunteers?

Dipl.-Ing. Peter Kelm
KelmMoyles - Tailored Engineering Marketing
Am Weichselgarten 7 
91058 Erlangen
T +49 (9132) 9060595 
F +49 (9132) 9060596
E peter.kelm at kelmmoyles.com <mailto:peter.kelm at kelmmoyles.com>





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170915/955c7a6c/attachment.html>

From rjahnel at ellipseinc.com  Fri Sep 15 14:10:06 2017
From: rjahnel at ellipseinc.com (Richard Jahnel)
Date: Fri, 15 Sep 2017 14:10:06 +0000
Subject: [OmniOS-discuss] questions
In-Reply-To: <CAN0RhAMhGOkLJkmeHxpkAu97KNF4g-bez5zRULWRKcoJzr6nYg@mail.gmail.com>
References: <4e79b129-cce8-6538-37bc-0f8f1a124ac8@exitas.be>
	<CAN0RhAMhGOkLJkmeHxpkAu97KNF4g-bez5zRULWRKcoJzr6nYg@mail.gmail.com>
Message-ID: <65DC5816D4BEE043885A89FD54E273FC781396F9@MAIL101.Ellipseinc.com>

/agreed for internal network transfers, netcat is the way to go. I have used it in the past for ZFS sends between internal machines with SSD backed volumes. All the crypto work from ssh/scp and other secure copy methods are the bottleneck in these cases.

Obviously if you transfers leave the secure network you?ll want to suck it up and take the CPU hit. ;)

From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Ludovic Orban
Sent: Thursday, September 14, 2017 8:28 AM
To: Dirk Willems <dirk.willems at exitas.be>
Cc: omnios-discuss <omnios-discuss at lists.omniti.com>
Subject: Re: [OmniOS-discuss] questions

Here's a quick thought: if you are copying your files with scp, you might be CPU bound because of all the crypto work. You should try to copy with a much more CPU-lightweight tool. I personally usually use netcat for this kind of stuff.

Just my 2 cents.

--
Ludovic

On Thu, Sep 14, 2017 at 2:26 PM, Dirk Willems <dirk.willems at exitas.be<mailto:dirk.willems at exitas.be>> wrote:

Hello,



I'm trying to understand something let me explain.



Oracle always told to me that if you create a etherstub switch it has infiniband speed 40GB/s.

But I have a customer running on Solaris (Yeah I know but let me explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan (I know told him to to use etherstub).

The copy witch is performed for a Oracle database with sql command, the DBA witch have 5 streams say it's waiting on the disk, the disk are 50 - 60 % busy the speed is 30 mb/s.



So I did some test just to see and understand if it's the database or the system, but with doing my tests I get very confused ???



On another Solaris at my work copy over etherstub switch => copy speed is 185MB/s expected much more of infiniband speed ???



root at test1:/export/home/Admin# scp test10G Admin at 192.168.1.2:/export/home/Admin/<mailto:Admin at 192.168.1.2:/export/home/Admin/>
Password:
test10G              100% |****************************************************************| 10240 MB    00:59



root at test2:~# dlstat -i 2

 LINK    IPKTS   RBYTES    OPKTS   OBYTES
           net1   25.76K  185.14M   10.08K    2.62M
           net1   27.04K  187.16M   11.23K    3.22M
           net1   26.97K  186.37M   11.24K    3.23M
           net1   26.63K  187.67M   10.82K    2.99M
           net1   27.94K  186.65M   12.17K    3.75M
           net1   27.45K  187.46M   11.70K    3.47M
           net1   26.01K  181.95M   10.63K    2.99M
           net1   27.95K  188.19M   12.14K    3.69M
           net1   27.91K  188.36M   12.08K    3.64M

The disks are all separate luns with all separated pools => disk are 20 - 30% busy



On my OmniOSce at my lab over etherstub



root at GNUHealth:~# scp test10G witte at 192.168.20.3:/export/home/witte/<mailto:witte at 192.168.20.3:/export/home/witte/>
Password:
test10G                                                                 76% 7853MB 116.4MB/s



=> copy is 116.4 MB/s => expected much more from infiniband speed is just the same as Lan ???



Is not that my disk can not follow 17% busy there sleeping ...

   extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0,0  248,4    0,0    2,1  0,0  1,3    0,0    5,3   0 102 c1
    0,0   37,5    0,0    0,7  0,0  0,2    0,0    4,7   0  17 c1t0d0 => rpool
    0,0   38,5    0,0    0,7  0,0  0,2    0,0    4,9   0  17 c1t1d0 => rpool
    0,0   40,5    0,0    0,1  0,0  0,2    0,0    5,6   0  17 c1t2d0 => data pool
    0,0   43,5    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t3d0 => data pool
    0,0   44,5    0,0    0,2  0,0  0,2    0,0    5,5   0  18 c1t4d0 => data pool
    0,0   44,0    0,0    0,2  0,0  0,2    0,0    5,4   0  17 c1t5d0 => data pool
    0,0   76,0    0,0    1,5  7,4  0,4   97,2    4,9  14  18 rpool
    0,0  172,4    0,0    0,6  2,0  0,9   11,4    5,5  12  20 DATA





root at NGINX:/root# dlstat show-link NGINX1 -i 2

 LINK  TYPE      ID  INDEX     PKTS    BYTES
         NGINX1    rx   bcast     --        0        0
         NGINX1    rx      sw     --        0        0
         NGINX1    tx   bcast     --        0        0
         NGINX1    tx      sw     --    9.26K  692.00K
         NGINX1    rx   local     --   26.00K  216.32M
         NGINX1    rx   bcast     --        0        0
         NGINX1    rx      sw     --        0        0
         NGINX1    tx   bcast     --        0        0
         NGINX1    tx      sw     --    7.01K  531.38K
         NGINX1    rx   local     --   30.65K  253.73M
         NGINX1    rx   bcast     --        0        0
         NGINX1    rx      sw     --        0        0
         NGINX1    tx   bcast     --        0        0
         NGINX1    tx      sw     --    8.95K  669.32K
         NGINX1    rx   local     --   29.10K  241.15M



On the other NGZ I receive 250MB/s ????



- So my question is how comes that the speed is equal to Lan 100MB/s on OmniOSce but i receive 250MB/s ?

- Why is etherstub so slow if infiniband speed is 40GB/s ???



I'm very confused right now ...



And want to know for sure how to understand and see this in the right way, because this customer will be the first customer from my who gonna switch complety over to OmniOSce on production and because this customer is one or the biggest company's in Belgium I really don't want to mess up !!!



So any help and clarification will be highly appreciate !!!



Thank you very much.



Kind Regards,



Dirk


--
[http://signatures.sidekick.be/exitas/images/placeholder-exitas.jpg]

Dirk Willems
System Engineer


+32 (0)3 443 12 38<tel:03%20443%2012%2038>
Dirk.Willems at exitas.be<mailto:Dirk.Willems at exitas.be>

Quality. Passion. Personality


www.exitas.be<http://www.exitas.be/> | Veldkant 31 | 2550 Kontich


Illumos OmniOS Installation and Configuration Implementation Specialist.
Oracle Solaris 11 Installation and Configuration Certified Implementation Specialist.

[cid:image001.png at 01D32D3B.E8AE3E10]


_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170915/c235f810/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4648 bytes
Desc: image001.png
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170915/c235f810/attachment-0001.png>

From jim.oltman at gmail.com  Fri Sep 15 18:10:19 2017
From: jim.oltman at gmail.com (Jim Oltman)
Date: Fri, 15 Sep 2017 12:10:19 -0600
Subject: [OmniOS-discuss] Upgrade from 151022 to CE bootadm error
In-Reply-To: <982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com>
References: <CA+Rs3OeZHiXno1_CSe7XPXsBC6fynQ-0=SvYyYwwbEgfE-_u5w@mail.gmail.com>
	<alpine.GSO.2.20.1708270841570.4883@freddy.simplesystems.org>
	<C15D0069-84E7-4579-BE4D-93B6CA5AF9CF@cos.ru>
	<CA+Rs3OcW05fKceYrnjK6p8P16QbHF4nVp2ztiZgseeJi2fr4mA@mail.gmail.com>
	<982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com>
Message-ID: <CA+Rs3Oea6Fh8gwu0SMV1H1w3FsW3Y5mdE1ifUhdKkGaY=pbPGg@mail.gmail.com>

On Mon, Aug 28, 2017 at 10:00 PM, Dan McDonald <danmcd at kebe.com> wrote:

> Three swaps are three mountpoints using tmpfs.  Please share "zfs list"
> and "beadm list" again, please?
>
> Dan
>
> Sent from my iPhone (typos, autocorrect, and all)
>
>
Sorry for the super late reply.  Been busy at home and work.  Here's Dan's
requested output:

joltman at OmniOS-NAS:/export/home/joltman$ zfs list
NAME                                           USED  AVAIL  REFER
MOUNTPOINT
rpool                                         24.1G  37.4M    31K  /rpool
rpool/ROOT                                    7.90G  37.4M    23K  legacy
rpool/ROOT/20170826_01_Pre_OmniOS-CE           101K  37.4M  2.67G  /
rpool/ROOT/20170826_Pre_OmniOS-CE             2.16M  37.4M  2.56G  /
rpool/ROOT/20170826_Pre_OmniOS-CE-1           7.82G  37.4M  2.55G  /
rpool/ROOT/OmniOS-NAS_20170324_1226            125K  37.4M  2.62G  /
rpool/ROOT/omnios                             58.9M  37.4M  2.46G  /
rpool/ROOT/omnios-1                           1.08M  37.4M  2.52G  /
rpool/ROOT/omnios-2                           3.22M  37.4M  2.56G  /
rpool/ROOT/omnios-2-backup-1                   166K  37.4M  2.64G  /
rpool/ROOT/omnios-2-backup-2                   163K  37.4M  2.53G  /
rpool/ROOT/omnios-2-backup-3                   164K  37.4M  2.54G  /
rpool/ROOT/omnios-backup-1                     220K  37.4M  1.72G  /
rpool/ROOT/omnios-backup-2                     226K  37.4M  1.75G  /
rpool/ROOT/omniosvar                            23K  37.4M    23K  legacy
rpool/ROOT/pre_download_17.01free_1490380649     1K  37.4M  2.61G  /
rpool/ROOT/pre_napp-it-16.07f                  220K  37.4M  1.73G  /
rpool/ROOT/pre_napp-it-17.01free               166K  37.4M  2.54G  /
rpool/ROOT/r151020_upgrade_151022             1.89M  37.4M  2.56G  /
rpool/ROOT/r151020_upgrade_151022-backup-1    12.9M  37.4M  2.55G  /
rpool/ROOT/r151020_upgrade_151022-backup-2     150K  37.4M  2.55G  /
rpool/dump                                    12.0G  37.4M  12.0G  -
rpool/export                                  26.9M  37.4M    23K  /export
rpool/export/home                             26.8M  37.4M  26.8M
/export/home
rpool/swap                                    4.13G  3.89G   276M  -
tank                                          2.34T  1.17T    36K  /tank
tank/backups                                  77.4G  1.17T  77.4G
/tank/backups
...
tank/joltman                                  93.7G  1.17T  60.9G
/tank/joltman
...
tank/mythtv                                    339G  1.17T   339G
/tank/mythtv
tank/pictures                                 34.4G  1.17T  34.3G
/tank/pictures
tank/public                                   3.62G  1.17T  3.60G
/tank/public
tank/seccams                                    20K  1.17T    20K
/tank/seccams
tank/software                                 73.9G  1.17T  73.9G
/tank/software
tank/videos                                   52.5K  1.17T  52.5K
/tank/videos
tank/voltman                                  55.3G  1.17T  55.2G
/tank/voltman
tank_00                                       14.0T  41.3G   272K  /tank_00
tank_00/videos                                14.0T  41.3G  14.0T
/tank_00/videos
tank_01                                       13.7T   355G   272K  /tank_01
tank_01/videos                                13.7T   355G  13.7T
/tank_01/videos
joltman at OmniOS-NAS:/export/home/joltman$

joltman at OmniOS-NAS:/export/home/joltman$ beadm list
BE                                Active Mountpoint Space Policy Created
omnios                            -      -          58.9M static 2016-11-04
15:18
omniosvar                         -      -          23.0K static 2016-11-04
15:18
omnios-backup-1                   -      -          220K  static 2016-11-11
15:38
pre_napp-it-16.07f                -      -          220K  static 2016-11-11
15:40
omnios-backup-2                   -      -          226K  static 2016-11-11
15:40
omnios-1                          -      -          1.08M static 2016-11-21
15:28
OmniOS-NAS_20170324_1226          -      -          125K  static 2017-03-24
12:27
omnios-2                          -      -          3.22M static 2017-03-24
12:27
pre_download_17.01free_1490380649 -      -          1.00K static 2017-03-24
12:37
omnios-2-backup-1                 -      -          166K  static 2017-03-28
17:06
omnios-2-backup-2                 -      -          163K  static 2017-04-16
18:33
omnios-2-backup-3                 -      -          164K  static 2017-05-13
22:45
r151020_upgrade_151022            -      -          1.89M static 2017-05-13
22:57
pre_napp-it-17.01free             -      -          166K  static 2017-05-18
18:05
r151020_upgrade_151022-backup-1   N      /          12.9M static 2017-08-26
12:29
r151020_upgrade_151022-backup-2   -      -          150K  static 2017-08-26
12:30
20170826_Pre_OmniOS-CE            -      -          2.16M static 2017-08-26
12:37
20170826_01_Pre_OmniOS-CE         -      -          101K  static 2017-08-26
15:02
20170826_Pre_OmniOS-CE-1          R      -          9.05G static 2017-08-26
15:02
joltman at OmniOS-NAS:/export/home/joltman$
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170915/45dbfe9f/attachment.html>

From hasslerd at gmx.li  Fri Sep 15 18:21:41 2017
From: hasslerd at gmx.li (Dominik Hassler)
Date: Fri, 15 Sep 2017 20:21:41 +0200
Subject: [OmniOS-discuss] Upgrade from 151022 to CE bootadm error
In-Reply-To: <CA+Rs3Oea6Fh8gwu0SMV1H1w3FsW3Y5mdE1ifUhdKkGaY=pbPGg@mail.gmail.com>
References: <CA+Rs3OeZHiXno1_CSe7XPXsBC6fynQ-0=SvYyYwwbEgfE-_u5w@mail.gmail.com>
	<alpine.GSO.2.20.1708270841570.4883@freddy.simplesystems.org>
	<C15D0069-84E7-4579-BE4D-93B6CA5AF9CF@cos.ru>
	<CA+Rs3OcW05fKceYrnjK6p8P16QbHF4nVp2ztiZgseeJi2fr4mA@mail.gmail.com>
	<982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com>
	<CA+Rs3Oea6Fh8gwu0SMV1H1w3FsW3Y5mdE1ifUhdKkGaY=pbPGg@mail.gmail.com>
Message-ID: <397055fb-5c4d-f9ff-4969-4aa2c443fab2@gmx.li>

It looks like your rpool is running out of free space.

On 09/15/2017 08:10 PM, Jim Oltman wrote:
> On Mon, Aug 28, 2017 at 10:00 PM, Dan McDonald <danmcd at kebe.com 
> <mailto:danmcd at kebe.com>> wrote:
> 
>     Three swaps are three mountpoints using tmpfs.  Please share "zfs
>     list" and "beadm list" again, please?
> 
>     Dan
> 
>     Sent from my iPhone (typos, autocorrect, and all)
> 
> 
> Sorry for the super late reply.  Been busy at home and work.  Here's 
> Dan's requested output:
> 
> joltman at OmniOS-NAS:/export/home/joltman$ zfs list
> NAME                                           USED  AVAIL  REFER  
> MOUNTPOINT
> rpool                                         24.1G  37.4M    31K  /rpool
> rpool/ROOT                                    7.90G  37.4M    23K  legacy
> rpool/ROOT/20170826_01_Pre_OmniOS-CE           101K  37.4M  2.67G  /
> rpool/ROOT/20170826_Pre_OmniOS-CE             2.16M  37.4M  2.56G  /
> rpool/ROOT/20170826_Pre_OmniOS-CE-1           7.82G  37.4M  2.55G  /
> rpool/ROOT/OmniOS-NAS_20170324_1226            125K  37.4M  2.62G  /
> rpool/ROOT/omnios                             58.9M  37.4M  2.46G  /
> rpool/ROOT/omnios-1                           1.08M  37.4M  2.52G  /
> rpool/ROOT/omnios-2                           3.22M  37.4M  2.56G  /
> rpool/ROOT/omnios-2-backup-1                   166K  37.4M  2.64G  /
> rpool/ROOT/omnios-2-backup-2                   163K  37.4M  2.53G  /
> rpool/ROOT/omnios-2-backup-3                   164K  37.4M  2.54G  /
> rpool/ROOT/omnios-backup-1                     220K  37.4M  1.72G  /
> rpool/ROOT/omnios-backup-2                     226K  37.4M  1.75G  /
> rpool/ROOT/omniosvar                            23K  37.4M    23K  legacy
> rpool/ROOT/pre_download_17.01free_1490380649     1K  37.4M  2.61G  /
> rpool/ROOT/pre_napp-it-16.07f                  220K  37.4M  1.73G  /
> rpool/ROOT/pre_napp-it-17.01free               166K  37.4M  2.54G  /
> rpool/ROOT/r151020_upgrade_151022             1.89M  37.4M  2.56G  /
> rpool/ROOT/r151020_upgrade_151022-backup-1    12.9M  37.4M  2.55G  /
> rpool/ROOT/r151020_upgrade_151022-backup-2     150K  37.4M  2.55G  /
> rpool/dump                                    12.0G  37.4M  12.0G  -
> rpool/export                                  26.9M  37.4M    23K  /export
> rpool/export/home                             26.8M  37.4M  26.8M  
> /export/home
> rpool/swap                                    4.13G  3.89G   276M  -
> tank                                          2.34T  1.17T    36K  /tank
> tank/backups                                  77.4G  1.17T  77.4G  
> /tank/backups
> ...
> tank/joltman                                  93.7G  1.17T  60.9G  
> /tank/joltman
> ...
> tank/mythtv                                    339G  1.17T   339G  
> /tank/mythtv
> tank/pictures                                 34.4G  1.17T  34.3G  
> /tank/pictures
> tank/public                                   3.62G  1.17T  3.60G  
> /tank/public
> tank/seccams                                    20K  1.17T    20K  
> /tank/seccams
> tank/software                                 73.9G  1.17T  73.9G  
> /tank/software
> tank/videos                                   52.5K  1.17T  52.5K  
> /tank/videos
> tank/voltman                                  55.3G  1.17T  55.2G  
> /tank/voltman
> tank_00                                       14.0T  41.3G   272K  /tank_00
> tank_00/videos                                14.0T  41.3G  14.0T  
> /tank_00/videos
> tank_01                                       13.7T   355G   272K  /tank_01
> tank_01/videos                                13.7T   355G  13.7T  
> /tank_01/videos
> joltman at OmniOS-NAS:/export/home/joltman$
> 
> joltman at OmniOS-NAS:/export/home/joltman$ beadm list
> BE                                Active Mountpoint Space Policy Created
> omnios                            -      -          58.9M static 
> 2016-11-04 15:18
> omniosvar                         -      -          23.0K static 
> 2016-11-04 15:18
> omnios-backup-1                   -      -          220K  static 
> 2016-11-11 15:38
> pre_napp-it-16.07f                -      -          220K  static 
> 2016-11-11 15:40
> omnios-backup-2                   -      -          226K  static 
> 2016-11-11 15:40
> omnios-1                          -      -          1.08M static 
> 2016-11-21 15:28
> OmniOS-NAS_20170324_1226          -      -          125K  static 
> 2017-03-24 12:27
> omnios-2                          -      -          3.22M static 
> 2017-03-24 12:27
> pre_download_17.01free_1490380649 -      -          1.00K static 
> 2017-03-24 12:37
> omnios-2-backup-1                 -      -          166K  static 
> 2017-03-28 17:06
> omnios-2-backup-2                 -      -          163K  static 
> 2017-04-16 18:33
> omnios-2-backup-3                 -      -          164K  static 
> 2017-05-13 22:45
> r151020_upgrade_151022            -      -          1.89M static 
> 2017-05-13 22:57
> pre_napp-it-17.01free             -      -          166K  static 
> 2017-05-18 18:05
> r151020_upgrade_151022-backup-1   N      /          12.9M static 
> 2017-08-26 12:29
> r151020_upgrade_151022-backup-2   -      -          150K  static 
> 2017-08-26 12:30
> 20170826_Pre_OmniOS-CE            -      -          2.16M static 
> 2017-08-26 12:37
> 20170826_01_Pre_OmniOS-CE         -      -          101K  static 
> 2017-08-26 15:02
> 20170826_Pre_OmniOS-CE-1          R      -          9.05G static 
> 2017-08-26 15:02
> joltman at OmniOS-NAS:/export/home/joltman$
> 
> 
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> 

From jim.oltman at gmail.com  Fri Sep 15 18:46:35 2017
From: jim.oltman at gmail.com (Jim Oltman)
Date: Fri, 15 Sep 2017 12:46:35 -0600
Subject: [OmniOS-discuss] Upgrade from 151022 to CE bootadm error
In-Reply-To: <397055fb-5c4d-f9ff-4969-4aa2c443fab2@gmx.li>
References: <CA+Rs3OeZHiXno1_CSe7XPXsBC6fynQ-0=SvYyYwwbEgfE-_u5w@mail.gmail.com>
	<alpine.GSO.2.20.1708270841570.4883@freddy.simplesystems.org>
	<C15D0069-84E7-4579-BE4D-93B6CA5AF9CF@cos.ru>
	<CA+Rs3OcW05fKceYrnjK6p8P16QbHF4nVp2ztiZgseeJi2fr4mA@mail.gmail.com>
	<982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com>
	<CA+Rs3Oea6Fh8gwu0SMV1H1w3FsW3Y5mdE1ifUhdKkGaY=pbPGg@mail.gmail.com>
	<397055fb-5c4d-f9ff-4969-4aa2c443fab2@gmx.li>
Message-ID: <CA+Rs3OfemVnm0onX8VC9p_NHP5vMAYCLNMc8vmYgNt415Yg7Pg@mail.gmail.com>

Dominik,

Yes I saw that too, but I'm not sure why 12GB is being taken up by
rpool/dump.  And there's really not that much used by BEs.  Is there a
reason that dumps has to be so large or should I allocate more space to
rpool?  Thanks!

Jim

On Fri, Sep 15, 2017 at 12:21 PM, Dominik Hassler <hasslerd at gmx.li> wrote:

> It looks like your rpool is running out of free space.
>
> On 09/15/2017 08:10 PM, Jim Oltman wrote:
>
>> On Mon, Aug 28, 2017 at 10:00 PM, Dan McDonald <danmcd at kebe.com <mailto:
>> danmcd at kebe.com>> wrote:
>>
>>     Three swaps are three mountpoints using tmpfs.  Please share "zfs
>>     list" and "beadm list" again, please?
>>
>>     Dan
>>
>>     Sent from my iPhone (typos, autocorrect, and all)
>>
>>
>> Sorry for the super late reply.  Been busy at home and work.  Here's
>> Dan's requested output:
>>
>> joltman at OmniOS-NAS:/export/home/joltman$ zfs list
>> NAME                                           USED  AVAIL  REFER
>> MOUNTPOINT
>> rpool                                         24.1G  37.4M    31K  /rpool
>> rpool/ROOT                                    7.90G  37.4M    23K  legacy
>> rpool/ROOT/20170826_01_Pre_OmniOS-CE           101K  37.4M  2.67G  /
>> rpool/ROOT/20170826_Pre_OmniOS-CE             2.16M  37.4M  2.56G  /
>> rpool/ROOT/20170826_Pre_OmniOS-CE-1           7.82G  37.4M  2.55G  /
>> rpool/ROOT/OmniOS-NAS_20170324_1226            125K  37.4M  2.62G  /
>> rpool/ROOT/omnios                             58.9M  37.4M  2.46G  /
>> rpool/ROOT/omnios-1                           1.08M  37.4M  2.52G  /
>> rpool/ROOT/omnios-2                           3.22M  37.4M  2.56G  /
>> rpool/ROOT/omnios-2-backup-1                   166K  37.4M  2.64G  /
>> rpool/ROOT/omnios-2-backup-2                   163K  37.4M  2.53G  /
>> rpool/ROOT/omnios-2-backup-3                   164K  37.4M  2.54G  /
>> rpool/ROOT/omnios-backup-1                     220K  37.4M  1.72G  /
>> rpool/ROOT/omnios-backup-2                     226K  37.4M  1.75G  /
>> rpool/ROOT/omniosvar                            23K  37.4M    23K  legacy
>> rpool/ROOT/pre_download_17.01free_1490380649     1K  37.4M  2.61G  /
>> rpool/ROOT/pre_napp-it-16.07f                  220K  37.4M  1.73G  /
>> rpool/ROOT/pre_napp-it-17.01free               166K  37.4M  2.54G  /
>> rpool/ROOT/r151020_upgrade_151022             1.89M  37.4M  2.56G  /
>> rpool/ROOT/r151020_upgrade_151022-backup-1    12.9M  37.4M  2.55G  /
>> rpool/ROOT/r151020_upgrade_151022-backup-2     150K  37.4M  2.55G  /
>> rpool/dump                                    12.0G  37.4M  12.0G  -
>> rpool/export                                  26.9M  37.4M    23K  /export
>> rpool/export/home                             26.8M  37.4M  26.8M
>> /export/home
>> rpool/swap                                    4.13G  3.89G   276M  -
>> tank                                          2.34T  1.17T    36K  /tank
>> tank/backups                                  77.4G  1.17T  77.4G
>> /tank/backups
>> ...
>> tank/joltman                                  93.7G  1.17T  60.9G
>> /tank/joltman
>> ...
>> tank/mythtv                                    339G  1.17T   339G
>> /tank/mythtv
>> tank/pictures                                 34.4G  1.17T  34.3G
>> /tank/pictures
>> tank/public                                   3.62G  1.17T  3.60G
>> /tank/public
>> tank/seccams                                    20K  1.17T    20K
>> /tank/seccams
>> tank/software                                 73.9G  1.17T  73.9G
>> /tank/software
>> tank/videos                                   52.5K  1.17T  52.5K
>> /tank/videos
>> tank/voltman                                  55.3G  1.17T  55.2G
>> /tank/voltman
>> tank_00                                       14.0T  41.3G   272K
>> /tank_00
>> tank_00/videos                                14.0T  41.3G  14.0T
>> /tank_00/videos
>> tank_01                                       13.7T   355G   272K
>> /tank_01
>> tank_01/videos                                13.7T   355G  13.7T
>> /tank_01/videos
>> joltman at OmniOS-NAS:/export/home/joltman$
>>
>> joltman at OmniOS-NAS:/export/home/joltman$ beadm list
>> BE                                Active Mountpoint Space Policy Created
>> omnios                            -      -          58.9M static
>> 2016-11-04 15:18
>> omniosvar                         -      -          23.0K static
>> 2016-11-04 15:18
>> omnios-backup-1                   -      -          220K  static
>> 2016-11-11 15:38
>> pre_napp-it-16.07f                -      -          220K  static
>> 2016-11-11 15:40
>> omnios-backup-2                   -      -          226K  static
>> 2016-11-11 15:40
>> omnios-1                          -      -          1.08M static
>> 2016-11-21 15:28
>> OmniOS-NAS_20170324_1226          -      -          125K  static
>> 2017-03-24 12:27
>> omnios-2                          -      -          3.22M static
>> 2017-03-24 12:27
>> pre_download_17.01free_1490380649 -      -          1.00K static
>> 2017-03-24 12:37
>> omnios-2-backup-1                 -      -          166K  static
>> 2017-03-28 17:06
>> omnios-2-backup-2                 -      -          163K  static
>> 2017-04-16 18:33
>> omnios-2-backup-3                 -      -          164K  static
>> 2017-05-13 22:45
>> r151020_upgrade_151022            -      -          1.89M static
>> 2017-05-13 22:57
>> pre_napp-it-17.01free             -      -          166K  static
>> 2017-05-18 18:05
>> r151020_upgrade_151022-backup-1   N      /          12.9M static
>> 2017-08-26 12:29
>> r151020_upgrade_151022-backup-2   -      -          150K  static
>> 2017-08-26 12:30
>> 20170826_Pre_OmniOS-CE            -      -          2.16M static
>> 2017-08-26 12:37
>> 20170826_01_Pre_OmniOS-CE         -      -          101K  static
>> 2017-08-26 15:02
>> 20170826_Pre_OmniOS-CE-1          R      -          9.05G static
>> 2017-08-26 15:02
>> joltman at OmniOS-NAS:/export/home/joltman$
>>
>>
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170915/0e72a451/attachment-0001.html>

From cks at cs.toronto.edu  Fri Sep 15 20:13:43 2017
From: cks at cs.toronto.edu (Chris Siebenmann)
Date: Fri, 15 Sep 2017 16:13:43 -0400
Subject: [OmniOS-discuss] Upgrade from 151022 to CE bootadm error
In-Reply-To: jim.oltman's message of Fri, 15 Sep 2017 12:46:35 -0600.
	<CA+Rs3OfemVnm0onX8VC9p_NHP5vMAYCLNMc8vmYgNt415Yg7Pg@mail.gmail.com>
Message-ID: <20170915201343.DD0D45A0170@apps1.cs.toronto.edu>

> Yes I saw that too, but I'm not sure why 12GB is being taken up by
> rpool/dump.  And there's really not that much used by BEs.  Is there a
> reason that dumps has to be so large or should I allocate more space
> to rpool?  Thanks!

 If I remember correctly, by default rpool/dump is sized during install
based on the machine's physical memory. On machines with any decent
amount of memory this can wind up using up an annoying amount of space,
especially if you're installing on to not-that-huge system SSDs.  My
further understanding is that during crash dumps the kernel will use
whatever free space is available in the rpool (in addition to whatever
is reserved for rpool/dump), so that if you have the free space at the
time you can capture even relatively large crash dumps. All that the
rpool/dump reservation does is either guarantee or increase the chances
that you can capture such large dumps.

 Our habitual practice is to resize rpool/dump down significantly on
our OmniOS machines. I believe we reserve 4 GB by default, and even
that might be considered generous. If you're short of space I wouldn't
hesitate to either delete it entirely or shrink it drastically.

(There are environments where reliably obtaining crash dumps if your
kernel does go down is sufficiently important that it's worth paying for
the extra disk space needed. My personal suspicion is that most people
are not deploying OmniOS in such an environment.)

	- cks

From johan.kragsterman at capvert.se  Sat Sep 16 04:38:17 2017
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Sat, 16 Sep 2017 06:38:17 +0200
Subject: [OmniOS-discuss] Ang: Re: Upgrade from 151022 to CE bootadm error
In-Reply-To: <CA+Rs3OfemVnm0onX8VC9p_NHP5vMAYCLNMc8vmYgNt415Yg7Pg@mail.gmail.com>
References: <CA+Rs3OfemVnm0onX8VC9p_NHP5vMAYCLNMc8vmYgNt415Yg7Pg@mail.gmail.com>,
	<CA+Rs3OeZHiXno1_CSe7XPXsBC6fynQ-0=SvYyYwwbEgfE-_u5w@mail.gmail.com>	<alpine.GSO.2.20.1708270841570.4883@freddy.simplesystems.org>
	<C15D0069-84E7-4579-BE4D-93B6CA5AF9CF@cos.ru>	<CA+Rs3OcW05fKceYrnjK6p8P16QbHF4nVp2ztiZgseeJi2fr4mA@mail.gmail.com>
	<982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com>	<CA+Rs3Oea6Fh8gwu0SMV1H1w3FsW3Y5mdE1ifUhdKkGaY=pbPGg@mail.gmail.com>
	<397055fb-5c4d-f9ff-4969-4aa2c443fab2@gmx.li>
Message-ID: <OF3F6D88D5.DAE86DB9-ONC125819D.0017CB6E-C125819D.00197A86@inse.com>

Hi!


-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev: -----
Till: Dominik Hassler <hasslerd at gmx.li>
Fr?n: Jim Oltman 
S?nt av: "OmniOS-discuss" 
Datum: 2017-09-15 20:48
Kopia: omnios-discuss <omnios-discuss at lists.omniti.com>
?rende: Re: [OmniOS-discuss] Upgrade from 151022 to CE bootadm error

Dominik,

Yes I saw that too, but I'm not sure why 12GB is being taken up by rpool/dump.  And there's really not that much used by BEs.  Is there a reason that dumps has to be so large or should I allocate more space to rpool?  Thanks!

Jim





Just move your rpool/dump to tank/dump instead. If you're concerned of the space of tank, use a sparse volume. You can move rpool/swap as well. Then you regain 16 GB of rpool space.


Rgrds Johan








joltman at OmniOS-NAS:/export/home/joltman$ zfs list
NAME                                           USED  AVAIL  REFER  MOUNTPOINT
rpool                                         24.1G  37.4M    31K  /rpool
rpool/ROOT                                    7.90G  37.4M    23K  legacy
rpool/ROOT/20170826_01_Pre_OmniOS-CE           101K  37.4M  2.67G  /
rpool/ROOT/20170826_Pre_OmniOS-CE             2.16M  37.4M  2.56G  /
rpool/ROOT/20170826_Pre_OmniOS-CE-1           7.82G  37.4M  2.55G  /
rpool/ROOT/OmniOS-NAS_20170324_1226            125K  37.4M  2.62G  /
rpool/ROOT/omnios                             58.9M  37.4M  2.46G  /
rpool/ROOT/omnios-1                           1.08M  37.4M  2.52G  /
rpool/ROOT/omnios-2                           3.22M  37.4M  2.56G  /
rpool/ROOT/omnios-2-backup-1                   166K  37.4M  2.64G  /
rpool/ROOT/omnios-2-backup-2                   163K  37.4M  2.53G  /
rpool/ROOT/omnios-2-backup-3                   164K  37.4M  2.54G  /
rpool/ROOT/omnios-backup-1                     220K  37.4M  1.72G  /
rpool/ROOT/omnios-backup-2                     226K  37.4M  1.75G  /
rpool/ROOT/omniosvar                            23K  37.4M    23K  legacy
rpool/ROOT/pre_download_17.01free_1490380649     1K  37.4M  2.61G  /
rpool/ROOT/pre_napp-it-16.07f                  220K  37.4M  1.73G  /
rpool/ROOT/pre_napp-it-17.01free               166K  37.4M  2.54G  /
rpool/ROOT/r151020_upgrade_151022             1.89M  37.4M  2.56G  /
rpool/ROOT/r151020_upgrade_151022-backup-1    12.9M  37.4M  2.55G  /
rpool/ROOT/r151020_upgrade_151022-backup-2     150K  37.4M  2.55G  /
rpool/dump                                    12.0G  37.4M  12.0G  -
rpool/export                                  26.9M  37.4M    23K  /export
rpool/export/home                             26.8M  37.4M  26.8M  /export/home
rpool/swap                                    4.13G  3.89G   276M  -
tank                                          2.34T  1.17T    36K  /tank
tank/backups                                  77.4G  1.17T  77.4G  /tank/backups
...
tank/joltman                                  93.7G  1.17T  60.9G  /tank/joltman
...
tank/mythtv                                    339G  1.17T   339G  /tank/mythtv
tank/pictures                                 34.4G  1.17T  34.3G  /tank/pictures
tank/public                                   3.62G  1.17T  3.60G  /tank/public
tank/seccams                                    20K  1.17T    20K  /tank/seccams
tank/software                                 73.9G  1.17T  73.9G  /tank/software
tank/videos                                   52.5K  1.17T  52.5K  /tank/videos
tank/voltman                                  55.3G  1.17T  55.2G  /tank/voltman
tank_00                                       14.0T  41.3G   272K  /tank_00
tank_00/videos                                14.0T  41.3G  14.0T  /tank_00/videos
tank_01                                       13.7T   355G   272K  /tank_01
tank_01/videos                                13.7T   355G  13.7T  /tank_01/videos
joltman at OmniOS-NAS:/export/home/joltman$

joltman at OmniOS-NAS:/export/home/joltman$ beadm list
BE                                Active Mountpoint Space Policy Created
omnios                            -      -          58.9M static 2016-11-04 15:18
omniosvar                         -      -          23.0K static 2016-11-04 15:18
omnios-backup-1                   -      -          220K  static 2016-11-11 15:38
pre_napp-it-16.07f                -      -          220K  static 2016-11-11 15:40
omnios-backup-2                   -      -          226K  static 2016-11-11 15:40
omnios-1                          -      -          1.08M static 2016-11-21 15:28
OmniOS-NAS_20170324_1226          -      -          125K  static 2017-03-24 12:27
omnios-2                          -      -          3.22M static 2017-03-24 12:27
pre_download_17.01free_1490380649 -      -          1.00K static 2017-03-24 12:37
omnios-2-backup-1                 -      -          166K  static 2017-03-28 17:06
omnios-2-backup-2                 -      -          163K  static 2017-04-16 18:33
omnios-2-backup-3                 -      -          164K  static 2017-05-13 22:45
r151020_upgrade_151022            -      -          1.89M static 2017-05-13 22:57
pre_napp-it-17.01free             -      -          166K  static 2017-05-18 18:05
r151020_upgrade_151022-backup-1   N      /          12.9M static 2017-08-26 12:29
r151020_upgrade_151022-backup-2   -      -          150K  static 2017-08-26 12:30
20170826_Pre_OmniOS-CE            -      -          2.16M static 2017-08-26 12:37
20170826_01_Pre_OmniOS-CE         -      -          101K  static 2017-08-26 15:02
20170826_Pre_OmniOS-CE-1          R      -          9.05G static 2017-08-26 15:02
joltman at OmniOS-NAS:/export/home/joltman$


_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss


_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss




From jimklimov at cos.ru  Sat Sep 16 06:15:00 2017
From: jimklimov at cos.ru (Jim Klimov)
Date: Sat, 16 Sep 2017 06:15:00 +0000
Subject: [OmniOS-discuss] Ang: Re: Upgrade from 151022 to CE bootadm
	error
In-Reply-To: <OF3F6D88D5.DAE86DB9-ONC125819D.0017CB6E-C125819D.00197A86@inse.com>
References: <CA+Rs3OfemVnm0onX8VC9p_NHP5vMAYCLNMc8vmYgNt415Yg7Pg@mail.gmail.com>
	<CA+Rs3OeZHiXno1_CSe7XPXsBC6fynQ-0=SvYyYwwbEgfE-_u5w@mail.gmail.com>
	<alpine.GSO.2.20.1708270841570.4883@freddy.simplesystems.org>
	<C15D0069-84E7-4579-BE4D-93B6CA5AF9CF@cos.ru>
	<CA+Rs3OcW05fKceYrnjK6p8P16QbHF4nVp2ztiZgseeJi2fr4mA@mail.gmail.com>
	<982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com>
	<CA+Rs3Oea6Fh8gwu0SMV1H1w3FsW3Y5mdE1ifUhdKkGaY=pbPGg@mail.gmail.com>
	<397055fb-5c4d-f9ff-4969-4aa2c443fab2@gmx.li>
	<OF3F6D88D5.DAE86DB9-ONC125819D.0017CB6E-C125819D.00197A86@inse.com>
Message-ID: <80BA71CC-D1F9-4053-9BD2-2F54A860F07F@cos.ru>

On September 16, 2017 6:38:17 AM GMT+02:00, Johan Kragsterman <johan.kragsterman at capvert.se> wrote:
>Hi!
>
>
>-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev:
>-----
>Till: Dominik Hassler <hasslerd at gmx.li>
>Fr?n: Jim Oltman 
>S?nt av: "OmniOS-discuss" 
>Datum: 2017-09-15 20:48
>Kopia: omnios-discuss <omnios-discuss at lists.omniti.com>
>?rende: Re: [OmniOS-discuss] Upgrade from 151022 to CE bootadm error
>
>Dominik,
>
>Yes I saw that too, but I'm not sure why 12GB is being taken up by
>rpool/dump.  And there's really not that much used by BEs.  Is there a
>reason that dumps has to be so large or should I allocate more space to
>rpool?  Thanks!
>
>Jim
>
>
>
>
>
>Just move your rpool/dump to tank/dump instead. If you're concerned of
>the space of tank, use a sparse volume. You can move rpool/swap as
>well. Then you regain 16 GB of rpool space.
>
>
>Rgrds Johan
>
>
>
>
>
>
>
>
>joltman at OmniOS-NAS:/export/home/joltman$ zfs list
>NAME                                           USED  AVAIL  REFER 
>MOUNTPOINT
>rpool                                         24.1G  37.4M    31K 
>/rpool
>rpool/ROOT                                    7.90G  37.4M    23K 
>legacy
>rpool/ROOT/20170826_01_Pre_OmniOS-CE           101K  37.4M  2.67G  /
>rpool/ROOT/20170826_Pre_OmniOS-CE             2.16M  37.4M  2.56G  /
>rpool/ROOT/20170826_Pre_OmniOS-CE-1           7.82G  37.4M  2.55G  /
>rpool/ROOT/OmniOS-NAS_20170324_1226            125K  37.4M  2.62G  /
>rpool/ROOT/omnios                             58.9M  37.4M  2.46G  /
>rpool/ROOT/omnios-1                           1.08M  37.4M  2.52G  /
>rpool/ROOT/omnios-2                           3.22M  37.4M  2.56G  /
>rpool/ROOT/omnios-2-backup-1                   166K  37.4M  2.64G  /
>rpool/ROOT/omnios-2-backup-2                   163K  37.4M  2.53G  /
>rpool/ROOT/omnios-2-backup-3                   164K  37.4M  2.54G  /
>rpool/ROOT/omnios-backup-1                     220K  37.4M  1.72G  /
>rpool/ROOT/omnios-backup-2                     226K  37.4M  1.75G  /
>rpool/ROOT/omniosvar                            23K  37.4M    23K 
>legacy
>rpool/ROOT/pre_download_17.01free_1490380649     1K  37.4M  2.61G  /
>rpool/ROOT/pre_napp-it-16.07f                  220K  37.4M  1.73G  /
>rpool/ROOT/pre_napp-it-17.01free               166K  37.4M  2.54G  /
>rpool/ROOT/r151020_upgrade_151022             1.89M  37.4M  2.56G  /
>rpool/ROOT/r151020_upgrade_151022-backup-1    12.9M  37.4M  2.55G  /
>rpool/ROOT/r151020_upgrade_151022-backup-2     150K  37.4M  2.55G  /
>rpool/dump                                    12.0G  37.4M  12.0G  -
>rpool/export                                  26.9M  37.4M    23K 
>/export
>rpool/export/home                             26.8M  37.4M  26.8M 
>/export/home
>rpool/swap                                    4.13G  3.89G   276M  -
>tank                                          2.34T  1.17T    36K 
>/tank
>tank/backups                                  77.4G  1.17T  77.4G 
>/tank/backups
>...
>tank/joltman                                  93.7G  1.17T  60.9G 
>/tank/joltman
>...
>tank/mythtv                                    339G  1.17T   339G 
>/tank/mythtv
>tank/pictures                                 34.4G  1.17T  34.3G 
>/tank/pictures
>tank/public                                   3.62G  1.17T  3.60G 
>/tank/public
>tank/seccams                                    20K  1.17T    20K 
>/tank/seccams
>tank/software                                 73.9G  1.17T  73.9G 
>/tank/software
>tank/videos                                   52.5K  1.17T  52.5K 
>/tank/videos
>tank/voltman                                  55.3G  1.17T  55.2G 
>/tank/voltman
>tank_00                                       14.0T  41.3G   272K 
>/tank_00
>tank_00/videos                                14.0T  41.3G  14.0T 
>/tank_00/videos
>tank_01                                       13.7T   355G   272K 
>/tank_01
>tank_01/videos                                13.7T   355G  13.7T 
>/tank_01/videos
>joltman at OmniOS-NAS:/export/home/joltman$
>
>joltman at OmniOS-NAS:/export/home/joltman$ beadm list
>BE                                Active Mountpoint Space Policy
>Created
>omnios                            -      -          58.9M static
>2016-11-04 15:18
>omniosvar                         -      -          23.0K static
>2016-11-04 15:18
>omnios-backup-1                   -      -          220K  static
>2016-11-11 15:38
>pre_napp-it-16.07f                -      -          220K  static
>2016-11-11 15:40
>omnios-backup-2                   -      -          226K  static
>2016-11-11 15:40
>omnios-1                          -      -          1.08M static
>2016-11-21 15:28
>OmniOS-NAS_20170324_1226          -      -          125K  static
>2017-03-24 12:27
>omnios-2                          -      -          3.22M static
>2017-03-24 12:27
>pre_download_17.01free_1490380649 -      -          1.00K static
>2017-03-24 12:37
>omnios-2-backup-1                 -      -          166K  static
>2017-03-28 17:06
>omnios-2-backup-2                 -      -          163K  static
>2017-04-16 18:33
>omnios-2-backup-3                 -      -          164K  static
>2017-05-13 22:45
>r151020_upgrade_151022            -      -          1.89M static
>2017-05-13 22:57
>pre_napp-it-17.01free             -      -          166K  static
>2017-05-18 18:05
>r151020_upgrade_151022-backup-1   N      /          12.9M static
>2017-08-26 12:29
>r151020_upgrade_151022-backup-2   -      -          150K  static
>2017-08-26 12:30
>20170826_Pre_OmniOS-CE            -      -          2.16M static
>2017-08-26 12:37
>20170826_01_Pre_OmniOS-CE         -      -          101K  static
>2017-08-26 15:02
>20170826_Pre_OmniOS-CE-1          R      -          9.05G static
>2017-08-26 15:02
>joltman at OmniOS-NAS:/export/home/joltman$
>
>
>_______________________________________________
>OmniOS-discuss mailing list
>OmniOS-discuss at lists.omniti.com
>http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
>_______________________________________________
>OmniOS-discuss mailing list
>OmniOS-discuss at lists.omniti.com
>http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
>
>_______________________________________________
>OmniOS-discuss mailing list
>OmniOS-discuss at lists.omniti.com
>http://lists.omniti.com/mailman/listinfo/omnios-discuss

But note there's dumpadm and /etc/vfstab to take care of to reconfigure the system's dumping and swapping locations.

Jim
--
Typos courtesy of K-9 Mail on my Android

From vab at bb-c.de  Sat Sep 16 09:32:35 2017
From: vab at bb-c.de (Volker A. Brandt)
Date: Sat, 16 Sep 2017 11:32:35 +0200
Subject: [OmniOS-discuss] Ang: Re: Upgrade from 151022 to CE bootadm error
In-Reply-To: <OF3F6D88D5.DAE86DB9-ONC125819D.0017CB6E-C125819D.00197A86@inse.com>
References: <CA+Rs3OfemVnm0onX8VC9p_NHP5vMAYCLNMc8vmYgNt415Yg7Pg@mail.gmail.com>
	<CA+Rs3OeZHiXno1_CSe7XPXsBC6fynQ-0=SvYyYwwbEgfE-_u5w@mail.gmail.com>
	<alpine.GSO.2.20.1708270841570.4883@freddy.simplesystems.org>
	<C15D0069-84E7-4579-BE4D-93B6CA5AF9CF@cos.ru>
	<CA+Rs3OcW05fKceYrnjK6p8P16QbHF4nVp2ztiZgseeJi2fr4mA@mail.gmail.com>
	<982706AB-D6E6-4D9F-951B-CF47B755FACC@kebe.com>
	<CA+Rs3Oea6Fh8gwu0SMV1H1w3FsW3Y5mdE1ifUhdKkGaY=pbPGg@mail.gmail.com>
	<397055fb-5c4d-f9ff-4969-4aa2c443fab2@gmx.li>
	<OF3F6D88D5.DAE86DB9-ONC125819D.0017CB6E-C125819D.00197A86@inse.com>
Message-ID: <22972.61363.317755.917107@shelob.bb-c.de>

> Just move your rpool/dump to tank/dump instead. If you're concerned of the
> space of tank, use a sparse volume. You can move rpool/swap as well. Then
> you regain 16 GB of rpool space.

I would not do that.  If the system crashes because it cannot mount the
"tank" pool, then you will not have a crash dump. :-)

Note that "dumpadm -e" will tell you how much space it thinks it needs.


Regards -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt               Consulting and Support for Oracle Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim, GERMANY            Email: vab at bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgr??e: 46
Gesch?ftsf?hrer: Rainer J.H. Brandt und Volker A. Brandt

"When logic and proportion have fallen sloppy dead"

From dirk.willems at exitas.be  Sat Sep 16 14:59:35 2017
From: dirk.willems at exitas.be (Dirk Willems)
Date: Sat, 16 Sep 2017 16:59:35 +0200
Subject: [OmniOS-discuss] upgrade Debian in LXZones
Message-ID: <f9df9f2e-47ac-7af5-089e-2b3c565cda30@exitas.be>

Hello,


Tried to upgrade my Debian LXZone but getting following ...

So installed a new clean Debian LXZone and tried with that one and 
having exactly the same error

Also don't see my interface in the new LXZones but is working


root at debian-8:~# ifconfig -a
OnlyOffice_vnic: error fetching interface information: No such device or 
address
root at debian-8:~# ping 192.168.20.3
PING 192.168.20.3 (192.168.20.3) 56(84) bytes of data.
64 bytes from 192.168.20.3: icmp_seq=1 ttl=255 time=0.300 ms
64 bytes from 192.168.20.3: icmp_seq=2 ttl=255 time=0.111 ms
64 bytes from 192.168.20.3: icmp_seq=3 ttl=255 time=0.131 ms


Any ideas ?


Setting up udev (215-17+deb8u7) ...
addgroup: The group `input' already exists as a system group. Exiting.
udev does not support containers, not started.
insserv: Service udev has to be enabled to start service udev-finish
insserv: exiting now!
update-rc.d: error: insserv rejected the script header
dpkg: error processing package udev (--configure):
  subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
  udev
E: Sub-process /usr/bin/dpkg returned an error code (1)
root at debian-8:~#


Thanks,


Dirk

-- 
	Dirk Willems
System Engineer


+32 (0)3 443 12 38
Dirk.Willems at exitas.be <mailto:Dirk.Willems at exitas.be>

Quality. Passion. Personality

www.exitas.be <http://www.exitas.be/> | Veldkant 31 | 2550 Kontich

Illumos OmniOS Installation and Configuration Implementation Specialist.
Oracle Solaris 11 Installation and Configuration Certified 
Implementation Specialist. 	

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170916/4dcfd428/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SmallPhoenixLogotypeRGB.png
Type: image/png
Size: 4648 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170916/4dcfd428/attachment-0001.png>

From johan.kragsterman at capvert.se  Sun Sep 17 05:58:32 2017
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Sun, 17 Sep 2017 07:58:32 +0200
Subject: [OmniOS-discuss] Ang: Re: [developer] ZFS cache device allocation
 and free seem strange on my OmniOS r151022 system.
In-Reply-To: <nycvar.TFB.7.76.1709162228370.3442@erncre.pvgehf-vg.arg>
References: <nycvar.TFB.7.76.1709162228370.3442@erncre.pvgehf-vg.arg>,
	<59AD7975.7010000@btconnect.com>
	<CAJjvXiFyBj731caHLZd4qfXdvpq7Lvy7sS8PHACxuE6r87puAg@mail.gmail.com>
Message-ID: <OFD9F9145E.11FFAC12-ONC125819E.001F9E0F-C125819E.0020D32B@inse.com>

Hi!



-----Andy Fiddaman <omnios at citrus-it.net> skrev: -----
Till: illumos-developer <developer at lists.illumos.org>
Fr?n: Andy Fiddaman <omnios at citrus-it.net>
Datum: 2017-09-17 00:32
Kopia: Andriy Gapon <avg at freebsd.org>
?rende: Re: [developer] ZFS cache device allocation and free seem strange on my OmniOS r151022 system.

On Thu, 14 Sep 2017, Matthew Ahrens wrote:

; The following commit should address the symptom you're describing; is it in
; your version of omnios?
;
; commit 6de76ce2a90f54fecb0dba46dca08c99cef7aa08
; Author: Andriy Gapon <avg at FreeBSD.org>
; Date:   Mon Feb 27 14:48:16 2017 -0800
;
;     7867 ARC space accounting leak
;     Reviewed by: Matthew Ahrens <mahrens at delphix.com>
;     Reviewed by: Dan Kimmel <dan.kimmel at delphix.com>
;     Approved by: Dan McDonald <danmcd at omniti.com>

That commit was merged to illumos-omnios on the 7th of March, so is definitely
in OmniOS r151022.

Andy





So apparently, that commit doesn't solve the problem...? It's a bit worrying...I got the exact same problem. Haven't hit me with any consequences yet, though, but that server doesn't have a lot of load.

I remember when this first showed up, someone had substantial problems, and we were all recommended to remove any cache devices until the problem was solved. And now it seems the problem isn't solved...?
So what is the recommendation now...? Remove it again?

Is someone looking into this? Is it reopened as an issue again? Or is it marked as solved and closed?



Rgrds Johan





















------------------------------------------
illumos-developer
Archives: https://illumos.topicbox.com/groups/developer/discussions/T737ffd52f97590b7-M831caefa636559cd9b878164
Powered by Topicbox: https://topicbox.com






From danmcd at kebe.com  Mon Sep 18 21:34:42 2017
From: danmcd at kebe.com (Dan McDonald)
Date: Mon, 18 Sep 2017 17:34:42 -0400
Subject: [OmniOS-discuss] Apache update on omniti-ms?
Message-ID: <20170918213442.GA46127@everywhere.local>

Some of us still use omniti-ms for things.  I know it's not a supported
publisher, but I'm curious about this Apache bug:

https://blog.fuzzing-project.org/60-Optionsbleed-HTTP-OPTIONS-method-can-leak-Apaches-server-memory.html

And whether or not the apache24 package will get a bump to fix this?

Thanks,
Dan

From vab at bb-c.de  Tue Sep 19 05:47:50 2017
From: vab at bb-c.de (Volker A. Brandt)
Date: Tue, 19 Sep 2017 07:47:50 +0200
Subject: [OmniOS-discuss] Apache update on omniti-ms?
In-Reply-To: <20170918213442.GA46127@everywhere.local>
References: <20170918213442.GA46127@everywhere.local>
Message-ID: <22976.44934.28297.757373@shelob.bb-c.de>

Dan McDonald writes:
> Some of us still use omniti-ms for things.  I know it's not a supported
> publisher, but I'm curious about this Apache bug:
> 
> https://blog.fuzzing-project.org/60-Optionsbleed-HTTP-OPTIONS-method-can-leak-Apaches-server-memory.html
> 
> And whether or not the apache24 package will get a bump to fix this?

There already is an "extra" repository for OmniOS CE.  While I don't
like yet another path ("/opt/ooce"), this is the right place for it.
Now "someone" must do the work. :-)

Where would I find patches etc. that were used to build the 2.2 in
omniti-ms?


Regards -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt               Consulting and Support for Oracle Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim, GERMANY            Email: vab at bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgr??e: 46
Gesch?ftsf?hrer: Rainer J.H. Brandt und Volker A. Brandt

"When logic and proportion have fallen sloppy dead"

From tobi at oetiker.ch  Thu Sep 21 21:43:54 2017
From: tobi at oetiker.ch (Tobias Oetiker)
Date: Thu, 21 Sep 2017 23:43:54 +0200 (CEST)
Subject: [OmniOS-discuss] ANNOUNCEMENT OmniOS 151022s (Security Update)
Message-ID: <780059164.637092.1506030234028.JavaMail.zimbra@oetiker.ch>

Release Notes for OmniOSce 151022s
----------------------------------

Early weekly release for w/c 25th of September 2017, uname -a shows omnios-r151022-eb9d5cb557 

Since this release contains security fixes we urge people to upgrade as soon as possible.

** This update requires a reboot. **

* Security updates for in-kernel CIFS client & server 

 ?* 8662 SMB server ioctls should be appropriately sized 
 ?* 8663 SMB client assumes serialized ioctls 

* Perl fixes: 

??* CVE-2017-12837
??* CVE-2017-12883

* Other changes 

??* 8651 loader: fix problem where No rootfs module provided, aborting could appear on some systems. 
??* IPsec observability improvements. 


Instructions for updating from OmniTI OmniOS r151022 are available on our web site 
http://www.omniosce.org/setup/switch

Full Release Notes
https://github.com/omniosorg/omnios-build/blob/r151022/doc/ReleaseNotes.md

Due to the fix to the loader, new release media will be built for this release and will be available tomorrow.

cheers
tobi

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch tobi at oetiker.ch +41 62 775 9902

From stephan.budach at jvm.de  Fri Sep 22 18:18:30 2017
From: stephan.budach at jvm.de (Stephan Budach)
Date: Fri, 22 Sep 2017 20:18:30 +0200 (CEST)
Subject: [OmniOS-discuss] Can't destroy ZFS
In-Reply-To: <404501801.4.1506104037315.JavaMail.stephan.budach@budy.stephanbudach.de>
Message-ID: <1771700854.19.1506104309829.JavaMail.stephan.budach@budy.stephanbudach.de>

Hi, 


after having received a zvol from my old S11 box vis zfs send/recv and being unable to re-import the LUN via stmfadm, I decided to remove that zvol/ZFS and start over. However, I cannot remove that particular ZFS and trying to do so yields this strange error: 


root at omnios:~# zfs destroy vmpool/iSCSI-Targets/EyeTV 
cannot destroy 'vmpool/iSCSI-Targets/EyeTV': dataset already exists 


The zpool currently looks like this? (by the way, no snaps whatsoever on it): 


root at omnios:~# zfs list -r vmpool 
NAME USED AVAIL REFER MOUNTPOINT 
vmpool 348G 3.17T 23K /vmpool 
vmpool/esxi 39.4G 1.96T 39.4G /vmpool/esxi 
vmpool/iSCSI-Targets 238G 3.17T 23K /tank/iSCSI-Targets 
vmpool/iSCSI-Targets/EyeTV 238G 3.17T 238G - 
vmpool/nfsCloudData 70.3G 186G 70.3G /vmpool/nfsCloudData 
vmpool/nfsZimbraData 41K 100G 41K /vmpool/nfsZimbraData 




I have scrubbed the web a bit for this particular error, but all the reports seem to relate to either snapshots or clones. 


Any idea is greatly appeciated. 


Cheers, 
stephan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170922/fe193ab1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5546 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170922/fe193ab1/attachment-0001.bin>

From peter.tribble at gmail.com  Fri Sep 22 18:35:21 2017
From: peter.tribble at gmail.com (Peter Tribble)
Date: Fri, 22 Sep 2017 19:35:21 +0100
Subject: [OmniOS-discuss] Can't destroy ZFS
In-Reply-To: <1771700854.19.1506104309829.JavaMail.stephan.budach@budy.stephanbudach.de>
References: <404501801.4.1506104037315.JavaMail.stephan.budach@budy.stephanbudach.de>
	<1771700854.19.1506104309829.JavaMail.stephan.budach@budy.stephanbudach.de>
Message-ID: <CAEgYsbH=0QB0eXBLb7Dt-qXECnCkZyCLD6hoyZLcEkFZT3K+7w@mail.gmail.com>

On Fri, Sep 22, 2017 at 7:18 PM, Stephan Budach <stephan.budach at jvm.de>
wrote:

> Hi,
>
> after having received a zvol from my old S11 box vis zfs send/recv and
> being unable to re-import the LUN via stmfadm, I decided to remove that
> zvol/ZFS and start over. However, I cannot remove that particular ZFS and
> trying to do so yields this strange error:
>
> root at omnios:~# zfs destroy vmpool/iSCSI-Targets/EyeTV
> cannot destroy 'vmpool/iSCSI-Targets/EyeTV': dataset already exists
>

Does anything in this help:

https://serverfault.com/questions/66414/cannot-destroy-zfs-snapshot-dataset-already-exists


> The zpool currently looks like this? (by the way, no snaps whatsoever on
> it):
>
> root at omnios:~# zfs list -r vmpool
> NAME                         USED  AVAIL  REFER  MOUNTPOINT
> vmpool                       348G  3.17T    23K  /vmpool
> vmpool/esxi                 39.4G  1.96T  39.4G  /vmpool/esxi
> vmpool/iSCSI-Targets         238G  3.17T    23K  /tank/iSCSI-Targets
> vmpool/iSCSI-Targets/EyeTV   238G  3.17T   238G  -
> vmpool/nfsCloudData         70.3G   186G  70.3G  /vmpool/nfsCloudData
> vmpool/nfsZimbraData          41K   100G    41K  /vmpool/nfsZimbraData
>
>
> I have scrubbed the web a bit for this particular error, but all the
> reports seem to relate to either snapshots or clones.
>
> Any idea is greatly appeciated.
>
> Cheers,
> stephan
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>


-- 
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170922/e150f732/attachment.html>

From stephan.budach at jvm.de  Fri Sep 22 18:37:59 2017
From: stephan.budach at jvm.de (Stephan Budach)
Date: Fri, 22 Sep 2017 20:37:59 +0200 (CEST)
Subject: [OmniOS-discuss] [SOLVED] Re:  Can't destroy ZFS
In-Reply-To: <1771700854.19.1506104309829.JavaMail.stephan.budach@budy.stephanbudach.de>
References: <1771700854.19.1506104309829.JavaMail.stephan.budach@budy.stephanbudach.de>
Message-ID: <764590023.49.1506105474726.JavaMail.stephan.budach@budy.stephanbudach.de>

Shoot? ;) Right after hitting the send button, I issued this: 

root at omnios:~# zdb -d vmpool 
Dataset mos [META], ID 0, cr_txg 4, 20.1M, 309 objects 
Dataset vmpool/nfsZimbraData [ZPL], ID 82, cr_txg 29169, 41.0K, 16 objects 
Dataset vmpool/esxi [ZPL], ID 49, cr_txg 20, 40.7G, 119 objects 
Dataset vmpool/iSCSI-Targets/EyeTV/iSCSI-Targets [ZPL], ID 268, cr_txg 296266, 23.0K, 7 objects 
Dataset vmpool/iSCSI-Targets/EyeTV [ZVOL], ID 262, cr_txg 296222, 238G, 2 objects 
Dataset vmpool/iSCSI-Targets [ZPL], ID 256, cr_txg 296164, 23.0K, 9 objects 
Dataset vmpool/nfsCloudData [ZPL], ID 94, cr_txg 30110, 70.3G, 280046 objects 
Dataset vmpool [ZPL], ID 21, cr_txg 1, 23.0K, 10 objects 
Verified large_blocks feature refcount of 0 is correct 
Verified sha512 feature refcount of 0 is correct 
Verified skein feature refcount of 0 is correct 
Verified edonr feature refcount of 0 is correct 

After destroying this "invisible" Dataset vmpool/iSCSI-Targets/EyeTV/iSCSI-Targets dataset, I was able to delete it's parent datasets. 

Sorry for the noise. 

Cheers, 
stephan 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170922/85293b3c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5546 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170922/85293b3c/attachment.bin>

From jbs at lean-on.com  Sat Sep 23 09:49:27 2017
From: jbs at lean-on.com (=?utf-8?B?SmFrb2IgQi4gU8O4cmVuc2Vu?=)
Date: Sat, 23 Sep 2017 09:49:27 +0000
Subject: [OmniOS-discuss] Can't destroy ZFS
In-Reply-To: <CAEgYsbH=0QB0eXBLb7Dt-qXECnCkZyCLD6hoyZLcEkFZT3K+7w@mail.gmail.com>
References: <404501801.4.1506104037315.JavaMail.stephan.budach@budy.stephanbudach.de><1771700854.19.1506104309829.JavaMail.stephan.budach@budy.stephanbudach.de>
	<CAEgYsbH=0QB0eXBLb7Dt-qXECnCkZyCLD6hoyZLcEkFZT3K+7w@mail.gmail.com>
Message-ID: <C1DD4A8AB690D34C8D8F169329A7D1270140453A35@LONEP-EXCH01.lean-on.local>

Hi there ? we work all the time with temporary  datasets and sometimes it all locks up.
Not to be cruel?. But !
Create a OI usb stick
Use Gparted in the  OI desktop ? delete it all.
More cruel?
Boot a W7 stick ? use the diskmanager  - it will leave the drives in a unformatted state.
If the disks are write protected state Use diskpart ? use the command attributes
diskpart attributes remove read only <https://www.google.dk/search?dcr=0&q=diskpart+attributes+read+only+clear&spell=1&sa=X&ved=0ahUKEwjagoGlg7vWAhWlApoKHcFAC-MQvwUIIigA>

then you are good to go ? and the disk are reset

all he best
Jakob


Fra: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] P? vegne af Peter Tribble
Sendt: 22. september 2017 20:35
Til: Stephan Budach
Cc: omnios-discuss
Emne: Re: [OmniOS-discuss] Can't destroy ZFS



On Fri, Sep 22, 2017 at 7:18 PM, Stephan Budach <stephan.budach at jvm.de<mailto:stephan.budach at jvm.de>> wrote:
Hi,

after having received a zvol from my old S11 box vis zfs send/recv and being unable to re-import the LUN via stmfadm, I decided to remove that zvol/ZFS and start over. However, I cannot remove that particular ZFS and trying to do so yields this strange error:

root at omnios:~# zfs destroy vmpool/iSCSI-Targets/EyeTV
cannot destroy 'vmpool/iSCSI-Targets/EyeTV': dataset already exists

Does anything in this help:

https://serverfault.com/questions/66414/cannot-destroy-zfs-snapshot-dataset-already-exists

The zpool currently looks like this? (by the way, no snaps whatsoever on it):

root at omnios:~# zfs list -r vmpool
NAME                         USED  AVAIL  REFER  MOUNTPOINT
vmpool                       348G  3.17T    23K  /vmpool
vmpool/esxi                 39.4G  1.96T  39.4G  /vmpool/esxi
vmpool/iSCSI-Targets         238G  3.17T    23K  /tank/iSCSI-Targets
vmpool/iSCSI-Targets/EyeTV   238G  3.17T   238G  -
vmpool/nfsCloudData         70.3G   186G  70.3G  /vmpool/nfsCloudData
vmpool/nfsZimbraData          41K   100G    41K  /vmpool/nfsZimbraData


I have scrubbed the web a bit for this particular error, but all the reports seem to relate to either snapshots or clones.

Any idea is greatly appeciated.

Cheers,
stephan

_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss



--
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170923/303e0e4f/attachment-0001.html>

From apenner.it at gmail.com  Mon Sep 25 08:22:28 2017
From: apenner.it at gmail.com (Artem Penner)
Date: Mon, 25 Sep 2017 08:22:28 +0000
Subject: [OmniOS-discuss] scsi_vhci. Command Timeout on path
Message-ID: <CAE9csxPqtD0E59EPJxpmUO61u_=7pTJwAk2oD_8e4Xsz5LneQg@mail.gmail.com>

Hi, all.

I ran into the following problem. One of the zfs data disk is become
unavailable and solaris marked it as bad for a too long time.

As I understand, disk must be mark offline, if disk timeout grater than *10
seconds  (scsi_timeout) * scsi_retry == 30 seconds*, but in log I see that
I/O was degraded more than 5 minutes!

How can I fix It?


Why settings in sd.conf do not apply to all disks?

"", "retries-timeout:1,retries-busy:1,retries-reset:1,retries-victim:2",

*Actual sd_io_time and retries-timeout *


root at znstor5-n1:~# echo "sd_io_time::print" | mdb -k

0xa


# Per disk timeout and retry count

root at znstor4-n1:~# echo "::walk sd_state | ::grep '.!=0' | ::sd_state" |
mdb -k | egrep "^un|un_retry_count|un_cmd_timeout"

un: ffffa1c0afbe9d00

    un_retry_count = 5

    un_cmd_timeout = 0xa

un: ffffa1c0afbea980

    un_retry_count = 5

    un_cmd_timeout = 0xa

un: ffffa1c0afbea340

    un_retry_count = 5

    un_cmd_timeout = 0xa

un: ffffa1c13d99a980

    un_retry_count = 3

    un_cmd_timeout = 0xa

un: ffffa1c13d2ca680

    un_retry_count = 3

    un_cmd_timeout = 0xa

un: ffffa1c1443a4cc0

    un_retry_count = 3

    un_cmd_timeout = 0xa

........

........


-----


*MY CONFIGURATION*
*scsi_vhci.conf *

# File is managed by Ansible



name="scsi_vhci" class="root";



load-balance="round-robin";



auto-failback="enable";

ddi-forceload =

"misc/scsi_vhci/scsi_vhci_f_asym_sun",

"misc/scsi_vhci/scsi_vhci_f_asym_lsi",

"misc/scsi_vhci/scsi_vhci_f_asym_emc",

"misc/scsi_vhci/scsi_vhci_f_sym_emc",

"misc/scsi_vhci/scsi_vhci_f_sym_hds",

"misc/scsi_vhci/scsi_vhci_f_sym",

"misc/scsi_vhci/scsi_vhci_f_tpgs";



scsi-vhci-failover-override =

"HGST    HUSMH8020BSS204",      "f_sym",

"HGST    HUC101890CS4204",      "f_sym",

"TOSHIBA PX04SHB020",           "f_sym",

"TOSHIBA PX05SHB080",           "f_sym";


*sd.conf*

# File is managed by Ansible

name="sd" class="scsi" target=0 lun=0;

name="sd" class="scsi" target=1 lun=0;

name="sd" class="scsi" target=2 lun=0;

name="sd" class="scsi" target=3 lun=0;

name="sd" class="scsi" target=4 lun=0;

name="sd" class="scsi" target=5 lun=0;

name="sd" class="scsi" target=6 lun=0;

name="sd" class="scsi" target=7 lun=0;

name="sd" class="scsi" target=8 lun=0;

name="sd" class="scsi" target=9 lun=0;

name="sd" class="scsi" target=10 lun=0;

name="sd" class="scsi" target=11 lun=0;

name="sd" class="scsi" target=12 lun=0;

name="sd" class="scsi" target=13 lun=0;

name="sd" class="scsi" target=14 lun=0;

name="sd" class="scsi" target=15 lun=0;



name="sd" class="scsi-self-identifying";

sd-config-list=

"", "retries-timeout:1,retries-busy:1,retries-reset:1,retries-victim:2",

"HGST    HUC101890CS4204", "physical-block-size:4096",

"HGST    HUSMH8020BSS204",
"throttle-max:32,disksort:false,power-condition:false,physical-block-size:4096",

"TOSHIBA PX05SMB080",
"throttle-max:32,disksort:false,power-condition:false,physical-block-size:4096",

"TOSHIBA PX04SHB020",
"throttle-max:32,disksort:false,power-condition:false,physical-block-size:4096";

sd_retry_on_reservation_conflict=0;


*Pool configuration*

config:



        NAME                         STATE     READ WRITE CKSUM

        dpool2                       DEGRADED     0     0     0

          raidz2-0                   DEGRADED     0     0     0

            c0t5000CCA02C417C30d0    ONLINE       0     0     0

            c0t5000CCA02C3F3430d0    ONLINE       0     0     0

            c0t5000CCA02C400064d0    ONLINE       0     0     0

            c0t5000CCA02C417B10d0    ONLINE       0     0     0

            spare-4                  DEGRADED     0     0     0

              c0t5000CCA02C3F5A78d0  UNAVAIL    141 4.59K     0

              c0t5000CCA02C4015E4d0  ONLINE       0     0     0

            c0t5000CCA02C417AA0d0    ONLINE       0     0     0

            c0t5000CCA02C3E3C6Cd0    ONLINE       0     0     0

            c0t5000CCA02C4170BCd0    ONLINE       0     0     0

            c0t5000CCA02C41448Cd0    ONLINE       0     0     0

            c0t5000CCA02C404B34d0    ONLINE       0     0     0

          raidz2-1                   ONLINE       0     0     0

            c0t5000CCA02C3F0148d0    ONLINE       0     0     0

            c0t5000CCA02C417D64d0    ONLINE       0     0     0

            c0t5000CCA02C417E80d0    ONLINE       0     0     0

            c0t5000CCA02C3F6A58d0    ONLINE       0     0     0

            c0t5000CCA02C4113D8d0    ONLINE       0     0     0

            c0t5000CCA02C411114d0    ONLINE       0     0     0

            c0t5000CCA02C414D0Cd0    ONLINE       0     0     0

            c0t5000CCA02C416FD0d0    ONLINE       0     0     0

            c0t5000CCA02C3E7634d0    ONLINE       0     0     0

            c0t5000CCA02C4143B8d0    ONLINE       0     0     0

          raidz2-2                   ONLINE       0     0     0

            c0t5000CCA02C416FA8d0    ONLINE       0     0     0

            c0t5000CCA02C414CB0d0    ONLINE       0     0     0

            c0t5000CCA02C414514d0    ONLINE       0     0     0

            c0t5000CCA02C414260d0    ONLINE       0     0     0

            c0t5000CCA02C3F6BE0d0    ONLINE       0     0     0

            c0t5000CCA02C417EA4d0    ONLINE       0     0     0

            c0t5000CCA02C414368d0    ONLINE       0     0     0

            c0t5000CCA02C414014d0    ONLINE       0     0     0

            c0t5000CCA02C416FACd0    ONLINE       0     0     0

            c0t5000CCA02C409FB8d0    ONLINE       0     0     0

          raidz2-3                   ONLINE       0     0     0

            c0t5000CCA02C41712Cd0    ONLINE       0     0     0

            c0t5000CCA02C41436Cd0    ONLINE       0     0     0

            c0t5000CCA02C414520d0    ONLINE       0     0     0

            c0t5000CCA02C417AECd0    ONLINE       0     0     0

            c0t5000CCA02C3F28FCd0    ONLINE       0     0     0

            c0t5000CCA02C40ED00d0    ONLINE       0     0     0

            c0t5000CCA02C416F94d0    ONLINE       0     0     0

            c0t5000CCA02C403310d0    ONLINE       0     0     0

            c0t5000CCA02C417AE0d0    ONLINE       0     0     0

            c0t5000CCA02C4078C8d0    ONLINE       0     0     0

          raidz2-6                   ONLINE       0     0     0

            c0t5000CCA02C2EADE8d0    ONLINE       0     0     0

            c0t5000CCA02C2E1FD4d0    ONLINE       0     0     0

            c0t5000CCA02C2E2044d0    ONLINE       0     0     0

            c0t5000CCA02C2E6988d0    ONLINE       0     0     0

            c0t5000CCA02C2E0E18d0    ONLINE       0     0     0

            c0t5000CCA02C2C4C28d0    ONLINE       0     0     0

            c0t5000CCA02C2CED00d0    ONLINE       0     0     0

            c0t5000CCA02C2BF270d0    ONLINE       0     0     0

            c0t5000CCA02C2C2954d0    ONLINE       0     0     0

            c0t5000CCA02C2E1F90d0    ONLINE       0     0     0

          raidz2-7                   ONLINE       0     0     0

            c0t5000CCA02C2E9428d0    ONLINE       0     0     0

            c0t5000CCA02C2E5D1Cd0    ONLINE       0     0     0

            c0t5000CCA02C2E8038d0    ONLINE       0     0     0

            c0t5000CCA02C2C3C54d0    ONLINE       0     0     0

            c0t5000CCA02C2E5F00d0    ONLINE       0     0     0

            c0t5000CCA02C2E7838d0    ONLINE       0     0     0

            c0t5000CCA02C2E38B8d0    ONLINE       0     0     0

            c0t5000CCA02C2E8538d0    ONLINE       0     0     0

            c0t5000CCA02C2E82D4d0    ONLINE       0     0     0

            c0t5000CCA02C2E9978d0    ONLINE       0     0     0

          raidz2-8                   ONLINE       0     0     0

            c0t5000CCA02C2E1FB8d0    ONLINE       0     0     0

            c0t5000CCA02C2E1F10d0    ONLINE       0     0     0

            c0t5000CCA02C2E1EF8d0    ONLINE       0     0     0

            c0t5000CCA02C2E30A8d0    ONLINE       0     0     0

            c0t5000CCA02C2E1FC0d0    ONLINE       0     0     0

            c0t5000CCA02C2DD6C4d0    ONLINE       0     0     0

            c0t5000CCA02C2E204Cd0    ONLINE       0     0     0

            c0t5000CCA02C2E2054d0    ONLINE       0     0     0

            c0t5000CCA03103B274d0    ONLINE       0     0     0

            c0t5000CCA03103B2C8d0    ONLINE       0     0     0

          raidz2-9                   ONLINE       0     0     0

            c0t5000CCA03103B34Cd0    ONLINE       0     0     0

            c0t5000CCA03104AE18d0    ONLINE       0     0     0

            c0t5000CCA03104AE54d0    ONLINE       0     0     0

            c0t5000CCA03103B2D4d0    ONLINE       0     0     0

            c0t5000CCA03103B26Cd0    ONLINE       0     0     0

            c0t5000CCA03103B3ACd0    ONLINE       0     0     0

            c0t5000CCA02C2E8344d0    ONLINE       0     0     0

            c0t5000CCA02C2E8018d0    ONLINE       0     0     0

            c0t5000CCA02C2E1EE4d0    ONLINE       0     0     0

            c0t5000CCA081016134d0    ONLINE       0     0     0

        logs

          mirror-4                   ONLINE       0     0     0

            c0t500003979C88FF55d0    ONLINE       0     0     0

            c0t58CE38E06C897DD5d0    ONLINE       0     0     0

          mirror-5                   ONLINE       0     0     0

            c0t500003973C88B6F9d0    ONLINE       0     0     0

            c0t500003973C88B709d0    ONLINE       0     0     0

          mirror-10                  ONLINE       0     0     0

            c0t58CE38E06C899C49d0    ONLINE       0     0     0

            c0t500003979C88FF91d0    ONLINE       0     0     0

          mirror-11                  ONLINE       0     0     0

            c0t58CE38E06C899C09d0    ONLINE       0     0     0

            c0t500003979C88FF5Dd0    ONLINE       0     0     0

        cache

          c0t5000CCA04B129374d0      ONLINE       0     0     0

          c0t5000CCA04B12A1E4d0      ONLINE       0     0     0

          c0t5000CCA04E27F1E8d0      ONLINE       0     0     0

          c0t5000CCA04B12952Cd0      ONLINE       0     0     0

        spares

          c0t5000CCA02C4015E4d0      INUSE

          c0t5000CCA02C416F7Cd0      AVAIL

          c0t5000CCA02C2E7850d0      AVAIL

          c0t5000CCA02C2E71F4d0      AVAIL



*dmesg*

Sep 24 18:03:59 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 18:03:59 znstor4-n1      /scsi_vhci/disk at g500003979c88ff55 (sd224):
Command Timeout on path lsc5/disk at w500003979c88ff57,0

Sep 24 20:18:48 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:18:48 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:19:09 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:19:09 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:19:29 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:19:29 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:19:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:19:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:19:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:19:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:19:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:20:10 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:20:10 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:20:30 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:20:30 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:20:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:20:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:20:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:20:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:21:21 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:21:21 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:21:51 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:21:51 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:22:21 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:22:21 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:22:41 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:22:41 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:23:02 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:23:02 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:23:22 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:23:22 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:23:43 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:23:43 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:24:13 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:24:13 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:24:34 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:24:34 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:25:04 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:25:04 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:25:04 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:25:04 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:25:04 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:25:24 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:25:24 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:25:44 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:25:44 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:26:25 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:26:25 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:26:45 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:26:45 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:27:05 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:27:05 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:27:25 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:27:25 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:27:46 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:27:46 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:28:06 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:28:06 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:28:37 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:28:37 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:28:57 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:28:57 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:29:18 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:29:18 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:29:18 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:29:18 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:29:48 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:29:48 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:30:28 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:30:28 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:30:58 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:30:58 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0

Sep 24 20:48:30 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 20:48:30 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc5/disk at w5000cca02c3f5a7a,0

Sep 24 20:51:44 znstor4-n1 scsi: [ID 107833 kern.warning] WARNING:
/scsi_vhci/disk at g5000cca02c3f5a78 (sd218):

Sep 24 20:51:44 znstor4-n1      drive offline

Sep 24 21:44:01 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 21:44:01 znstor4-n1      /scsi_vhci/disk at g500003973c88b709 (sd229):
Command Timeout on path lsc4/disk at w500003973c88b70b,0

Sep 24 21:48:31 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 24 21:48:31 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
Command Timeout on path lsc5/disk at w5000cca02c3f5a7a,0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170925/4c961d79/attachment-0001.html>

From omnios at citrus-it.net  Mon Sep 25 23:31:46 2017
From: omnios at citrus-it.net (Andy Fiddaman)
Date: Mon, 25 Sep 2017 23:31:46 +0000 (UTC)
Subject: [OmniOS-discuss] OmniOS bloody update - counting down to r151024
Message-ID: <nycvar.TFB.7.76.1709252325300.12982@erncre.pvgehf-vg.arg>


It's now around two weeks until we plan to freeze OmniOS bloody as we
prepare for a r151024 release in November - as per the release schedule
that we announced back in August ( https://www.omniosce.org/schedule ).

In preparation for this, we have just updated our bloody IPS repository with
all of the latest bits and would appreciate your help in putting them through
their paces.

If you have some spare cycles please grab the ISO and set up a bloody system
for testing, or do a test upgrade of an r151022 system to a bloody
boot-environment. The open VM tools package has been brought right up-to-date
so it would also be good to get some feedback about running this version under
different versions of VMware.

Downloads can be found at https://downloads.omniosce.org/media/bloody/

More details and draft release notes for r151024 are at
https://www.omniosce.org/article/bloody-20170925

Thanks in advance from the OmniOSce team. Please join us at
https://gitter.im/omniosorg/Lobby to discuss this update.

Regards,

Andy

-- 
Citrus IT Limited | +44 (0)333 0124 007 | enquiries at citrus-it.co.uk
Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ
Registered in England and Wales | Company number 4899123


From apenner.it at gmail.com  Tue Sep 26 12:17:25 2017
From: apenner.it at gmail.com (Artem Penner)
Date: Tue, 26 Sep 2017 12:17:25 +0000
Subject: [OmniOS-discuss] scsi_vhci. Command Timeout on path
In-Reply-To: <CAE9csxPqtD0E59EPJxpmUO61u_=7pTJwAk2oD_8e4Xsz5LneQg@mail.gmail.com>
References: <CAE9csxPqtD0E59EPJxpmUO61u_=7pTJwAk2oD_8e4Xsz5LneQg@mail.gmail.com>
Message-ID: <CAE9csxOw8WNu5xYc85cn8cbZhGPU-ajJR7ziL+XG-_zD7t_2Vg@mail.gmail.com>

I reproduced issue.

I have a pool with problem disks and I/o on this pool is extremely slow.

here is *zpool iostat output:*

root at znstor5-n2:~# zpool iostat dpool7 1

           capacity     operations    bandwidth

pool    alloc   free   read  write   read  write

------  -----  -----  -----  -----  -----  -----

dpool7  6.24T   124T    485    359  5.55M  3.23M

dpool7  6.24T   124T      0      1      0  7.97K

dpool7  6.24T   124T     83     44  1002K   360K

dpool7  6.24T   124T     47     28   572K   246K

dpool7  6.24T   124T    286  1.04K  3.35M  9.75M

dpool7  6.24T   124T    783    532  9.18M  3.98M

dpool7  6.24T   124T      5      2  71.5K  23.8K

^C



iostat report 100% busy on two disks:

*iostat -xnmMpz 1*

                    extended device statistics

    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device

    0.0   12.0    0.0    0.0  0.0  0.0    0.0    0.2   0   0
c3t50000397B831FBA6d0

    0.0    6.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0
c3t50000397B831FBA6d0s0

    0.0    6.0    0.0    0.0  0.0  0.0    0.0    0.3   0   0
c3t50000397B831FBA6d0s1

    0.0   12.0    0.0    0.0  0.0  0.0    0.0    0.2   0   0
c1t50000397B83011BEd0

    0.0    6.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0
c1t50000397B83011BEd0s0

    0.0    6.0    0.0    0.0  0.0  0.0    0.0    0.3   0   0
c1t50000397B83011BEd0s1

    0.0   12.0    0.0    0.0  0.0  0.0    0.0    2.5   0   3
c2t50000397B8307346d0

    0.0    6.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0
c2t50000397B8307346d0s0

    0.0    6.0    0.0    0.0  0.0  0.0    0.0    5.1   0   3
c2t50000397B8307346d0s1

    0.0    0.0    0.0    0.0  7.0  0.9    0.0    0.0 100  91
c0t5000CCA02C51CFB0d0

    0.0    0.0    0.0    0.0  7.0  0.9    0.0    0.0 100  91
c0t5000CCA02C51CFB0d0s0

    0.0    0.0    0.0    0.0  9.0  0.9    0.0    0.0 100  89
c0t5000CCA02C51AB38d0

    0.0    0.0    0.0    0.0  9.0  0.9    0.0    0.0 100  89
c0t5000CCA02C51AB38d0s0





in /var/adm/messages I see timeouts from disks:

Sep 26 15:01:58 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 26 15:01:58 znstor5-n2      /scsi_vhci/disk at g5000cca02c51eb00 (sd50):
Command Timeout on path lsc4/disk at w5000cca02c51eb02,0

Sep 26 15:02:09 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 26 15:02:09 znstor5-n2      /scsi_vhci/disk at g5000cca02c51ea68 (sd52):
Command Timeout on path lsc4/disk at w5000cca02c51ea6a,0

Sep 26 15:02:13 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 26 15:02:13 znstor5-n2      /scsi_vhci/disk at g5000cca02c51ab38 (sd108):
Command Timeout on path lsc3/disk at w5000cca02c51ab39,0

Sep 26 15:02:14 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 26 15:02:14 znstor5-n2      /scsi_vhci/disk at g5000cca02c51ea54 (sd103):
Command Timeout on path lsc2/disk at w5000cca02c51ea55,0

Sep 26 15:02:27 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 26 15:02:27 znstor5-n2      /scsi_vhci/disk at g5000cca02c51ab38 (sd108):
Command Timeout on path lsc3/disk at w5000cca02c51ab39,0

Sep 26 15:02:38 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 26 15:02:38 znstor5-n2      /scsi_vhci/disk at g5000cca02c51ab38 (sd108):
Command Timeout on path lsc3/disk at w5000cca02c51ab39,0

Sep 26 15:02:49 znstor5-n2 scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Sep 26 15:02:49 znstor5-n2      /scsi_vhci/disk at g5000cca02c51ab38 (sd108):
Command Timeout on path lsc3/disk at w5000cca02c51ab39,0





My SCSI settings is follow:

*echo "*sd_state::walk softstate | ::print -d struct sd_lun un_sd
un_f_disksort_disabled un_f_suppress_cache_flush un_throttle
un_min_throttle un_cmd_timeout un_retry_count un_victim_retry_count" | sudo
mdb -k*

un_sd = 0xffffa1c143aac6f8

un_f_disksort_disabled = 1

un_f_suppress_cache_flush = 1

un_throttle = 0t256

un_min_throttle = 8

un_cmd_timeout = 5

un_retry_count = 1

un_victim_retry_count = 2



*fmdump -eV *report following errors:

Sep 26 2017 15:04:05.142507316 ereport.io.scsi.cmd.disk.tran

nvlist version: 0

        class = ereport.io.scsi.cmd.disk.tran

        ena = 0x582bda1c3c809c01

        detector = (embedded nvlist)

        nvlist version: 0

                version = 0x0

                scheme = dev

                cna_dev = 0x59ca325100000064

                device-path = /pci at 0,0/pci8086,6f08 at 3/pci1137,14c at 0/iport at f
/disk at w5000cca02c51eb76,0

        (end detector)



        devid = id1,sd at n5000cca02c51eb74

        driver-assessment = fail

        op-code = 0x2a

        cdb = 0x2a 0x0 0x2 0x9f 0xc5 0x28 0x0 0x0 0xe 0x0

        pkt-reason = 0x6

        pkt-state = 0x17

        pkt-stats = 0x50

        __ttl = 0x1

        __tod = 0x59ca4235 0x87e7d34



Sep 26 2017 15:04:07.392324757 ereport.io.scsi.cmd.disk.tran

nvlist version: 0

        class = ereport.io.scsi.cmd.disk.tran

        ena = 0x58219b9ffda04c1d

        detector = (embedded nvlist)

        nvlist version: 0

                version = 0x0

                scheme = dev

                cna_dev = 0x59ca32510000010f

                device-path = /pci at 0,0/pci8086,6f08 at 3/pci1137,14c at 0/iport at f0
/disk at w5000cca02c51190a,0

        (end detector)



        devid = id1,sd at n5000cca02c511908

        driver-assessment = fail

        op-code = 0x28

        cdb = 0x28 0x0 0x3 0xaa 0x10 0xec 0x0 0x0 0x1 0x0

        pkt-reason = 0x4

        pkt-state = 0x17

        pkt-stats = 0x10

        __ttl = 0x1

        __tod = 0x59ca4237 0x17626695





??, 25 ????. 2017 ?. ? 11:22, Artem Penner <apenner.it at gmail.com>:

> Hi, all.
>
> I ran into the following problem. One of the zfs data disk is become
> unavailable and solaris marked it as bad for a too long time.
>
> As I understand, disk must be mark offline, if disk timeout grater than *10
> seconds  (scsi_timeout) * scsi_retry == 30 seconds*, but in log I see
> that I/O was degraded more than 5 minutes!
>
> How can I fix It?
>
>
> Why settings in sd.conf do not apply to all disks?
>
> "", "retries-timeout:1,retries-busy:1,retries-reset:1,retries-victim:2",
>
> *Actual sd_io_time and retries-timeout *
>
>
> root at znstor5-n1:~# echo "sd_io_time::print" | mdb -k
>
> 0xa
>
>
> # Per disk timeout and retry count
>
> root at znstor4-n1:~# echo "::walk sd_state | ::grep '.!=0' | ::sd_state" |
> mdb -k | egrep "^un|un_retry_count|un_cmd_timeout"
>
> un: ffffa1c0afbe9d00
>
>     un_retry_count = 5
>
>     un_cmd_timeout = 0xa
>
> un: ffffa1c0afbea980
>
>     un_retry_count = 5
>
>     un_cmd_timeout = 0xa
>
> un: ffffa1c0afbea340
>
>     un_retry_count = 5
>
>     un_cmd_timeout = 0xa
>
> un: ffffa1c13d99a980
>
>     un_retry_count = 3
>
>     un_cmd_timeout = 0xa
>
> un: ffffa1c13d2ca680
>
>     un_retry_count = 3
>
>     un_cmd_timeout = 0xa
>
> un: ffffa1c1443a4cc0
>
>     un_retry_count = 3
>
>     un_cmd_timeout = 0xa
>
> ........
>
> ........
>
>
> -----
>
>
> *MY CONFIGURATION*
> *scsi_vhci.conf *
>
> # File is managed by Ansible
>
>
>
> name="scsi_vhci" class="root";
>
>
>
> load-balance="round-robin";
>
>
>
> auto-failback="enable";
>
> ddi-forceload =
>
> "misc/scsi_vhci/scsi_vhci_f_asym_sun",
>
> "misc/scsi_vhci/scsi_vhci_f_asym_lsi",
>
> "misc/scsi_vhci/scsi_vhci_f_asym_emc",
>
> "misc/scsi_vhci/scsi_vhci_f_sym_emc",
>
> "misc/scsi_vhci/scsi_vhci_f_sym_hds",
>
> "misc/scsi_vhci/scsi_vhci_f_sym",
>
> "misc/scsi_vhci/scsi_vhci_f_tpgs";
>
>
>
> scsi-vhci-failover-override =
>
> "HGST    HUSMH8020BSS204",      "f_sym",
>
> "HGST    HUC101890CS4204",      "f_sym",
>
> "TOSHIBA PX04SHB020",           "f_sym",
>
> "TOSHIBA PX05SHB080",           "f_sym";
>
>
> *sd.conf*
>
> # File is managed by Ansible
>
> name="sd" class="scsi" target=0 lun=0;
>
> name="sd" class="scsi" target=1 lun=0;
>
> name="sd" class="scsi" target=2 lun=0;
>
> name="sd" class="scsi" target=3 lun=0;
>
> name="sd" class="scsi" target=4 lun=0;
>
> name="sd" class="scsi" target=5 lun=0;
>
> name="sd" class="scsi" target=6 lun=0;
>
> name="sd" class="scsi" target=7 lun=0;
>
> name="sd" class="scsi" target=8 lun=0;
>
> name="sd" class="scsi" target=9 lun=0;
>
> name="sd" class="scsi" target=10 lun=0;
>
> name="sd" class="scsi" target=11 lun=0;
>
> name="sd" class="scsi" target=12 lun=0;
>
> name="sd" class="scsi" target=13 lun=0;
>
> name="sd" class="scsi" target=14 lun=0;
>
> name="sd" class="scsi" target=15 lun=0;
>
>
>
> name="sd" class="scsi-self-identifying";
>
> sd-config-list=
>
> "", "retries-timeout:1,retries-busy:1,retries-reset:1,retries-victim:2",
>
> "HGST    HUC101890CS4204", "physical-block-size:4096",
>
> "HGST    HUSMH8020BSS204",
> "throttle-max:32,disksort:false,power-condition:false,physical-block-size:4096",
>
> "TOSHIBA PX05SMB080",
> "throttle-max:32,disksort:false,power-condition:false,physical-block-size:4096",
>
> "TOSHIBA PX04SHB020",
> "throttle-max:32,disksort:false,power-condition:false,physical-block-size:4096";
>
> sd_retry_on_reservation_conflict=0;
>
>
> *Pool configuration*
>
> config:
>
>
>
>         NAME                         STATE     READ WRITE CKSUM
>
>         dpool2                       DEGRADED     0     0     0
>
>           raidz2-0                   DEGRADED     0     0     0
>
>             c0t5000CCA02C417C30d0    ONLINE       0     0     0
>
>             c0t5000CCA02C3F3430d0    ONLINE       0     0     0
>
>             c0t5000CCA02C400064d0    ONLINE       0     0     0
>
>             c0t5000CCA02C417B10d0    ONLINE       0     0     0
>
>             spare-4                  DEGRADED     0     0     0
>
>               c0t5000CCA02C3F5A78d0  UNAVAIL    141 4.59K     0
>
>               c0t5000CCA02C4015E4d0  ONLINE       0     0     0
>
>             c0t5000CCA02C417AA0d0    ONLINE       0     0     0
>
>             c0t5000CCA02C3E3C6Cd0    ONLINE       0     0     0
>
>             c0t5000CCA02C4170BCd0    ONLINE       0     0     0
>
>             c0t5000CCA02C41448Cd0    ONLINE       0     0     0
>
>             c0t5000CCA02C404B34d0    ONLINE       0     0     0
>
>           raidz2-1                   ONLINE       0     0     0
>
>             c0t5000CCA02C3F0148d0    ONLINE       0     0     0
>
>             c0t5000CCA02C417D64d0    ONLINE       0     0     0
>
>             c0t5000CCA02C417E80d0    ONLINE       0     0     0
>
>             c0t5000CCA02C3F6A58d0    ONLINE       0     0     0
>
>             c0t5000CCA02C4113D8d0    ONLINE       0     0     0
>
>             c0t5000CCA02C411114d0    ONLINE       0     0     0
>
>             c0t5000CCA02C414D0Cd0    ONLINE       0     0     0
>
>             c0t5000CCA02C416FD0d0    ONLINE       0     0     0
>
>             c0t5000CCA02C3E7634d0    ONLINE       0     0     0
>
>             c0t5000CCA02C4143B8d0    ONLINE       0     0     0
>
>           raidz2-2                   ONLINE       0     0     0
>
>             c0t5000CCA02C416FA8d0    ONLINE       0     0     0
>
>             c0t5000CCA02C414CB0d0    ONLINE       0     0     0
>
>             c0t5000CCA02C414514d0    ONLINE       0     0     0
>
>             c0t5000CCA02C414260d0    ONLINE       0     0     0
>
>             c0t5000CCA02C3F6BE0d0    ONLINE       0     0     0
>
>             c0t5000CCA02C417EA4d0    ONLINE       0     0     0
>
>             c0t5000CCA02C414368d0    ONLINE       0     0     0
>
>             c0t5000CCA02C414014d0    ONLINE       0     0     0
>
>             c0t5000CCA02C416FACd0    ONLINE       0     0     0
>
>             c0t5000CCA02C409FB8d0    ONLINE       0     0     0
>
>           raidz2-3                   ONLINE       0     0     0
>
>             c0t5000CCA02C41712Cd0    ONLINE       0     0     0
>
>             c0t5000CCA02C41436Cd0    ONLINE       0     0     0
>
>             c0t5000CCA02C414520d0    ONLINE       0     0     0
>
>             c0t5000CCA02C417AECd0    ONLINE       0     0     0
>
>             c0t5000CCA02C3F28FCd0    ONLINE       0     0     0
>
>             c0t5000CCA02C40ED00d0    ONLINE       0     0     0
>
>             c0t5000CCA02C416F94d0    ONLINE       0     0     0
>
>             c0t5000CCA02C403310d0    ONLINE       0     0     0
>
>             c0t5000CCA02C417AE0d0    ONLINE       0     0     0
>
>             c0t5000CCA02C4078C8d0    ONLINE       0     0     0
>
>           raidz2-6                   ONLINE       0     0     0
>
>             c0t5000CCA02C2EADE8d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E1FD4d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E2044d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E6988d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E0E18d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2C4C28d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2CED00d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2BF270d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2C2954d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E1F90d0    ONLINE       0     0     0
>
>           raidz2-7                   ONLINE       0     0     0
>
>             c0t5000CCA02C2E9428d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E5D1Cd0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E8038d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2C3C54d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E5F00d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E7838d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E38B8d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E8538d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E82D4d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E9978d0    ONLINE       0     0     0
>
>           raidz2-8                   ONLINE       0     0     0
>
>             c0t5000CCA02C2E1FB8d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E1F10d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E1EF8d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E30A8d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E1FC0d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2DD6C4d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E204Cd0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E2054d0    ONLINE       0     0     0
>
>             c0t5000CCA03103B274d0    ONLINE       0     0     0
>
>             c0t5000CCA03103B2C8d0    ONLINE       0     0     0
>
>           raidz2-9                   ONLINE       0     0     0
>
>             c0t5000CCA03103B34Cd0    ONLINE       0     0     0
>
>             c0t5000CCA03104AE18d0    ONLINE       0     0     0
>
>             c0t5000CCA03104AE54d0    ONLINE       0     0     0
>
>             c0t5000CCA03103B2D4d0    ONLINE       0     0     0
>
>             c0t5000CCA03103B26Cd0    ONLINE       0     0     0
>
>             c0t5000CCA03103B3ACd0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E8344d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E8018d0    ONLINE       0     0     0
>
>             c0t5000CCA02C2E1EE4d0    ONLINE       0     0     0
>
>             c0t5000CCA081016134d0    ONLINE       0     0     0
>
>         logs
>
>           mirror-4                   ONLINE       0     0     0
>
>             c0t500003979C88FF55d0    ONLINE       0     0     0
>
>             c0t58CE38E06C897DD5d0    ONLINE       0     0     0
>
>           mirror-5                   ONLINE       0     0     0
>
>             c0t500003973C88B6F9d0    ONLINE       0     0     0
>
>             c0t500003973C88B709d0    ONLINE       0     0     0
>
>           mirror-10                  ONLINE       0     0     0
>
>             c0t58CE38E06C899C49d0    ONLINE       0     0     0
>
>             c0t500003979C88FF91d0    ONLINE       0     0     0
>
>           mirror-11                  ONLINE       0     0     0
>
>             c0t58CE38E06C899C09d0    ONLINE       0     0     0
>
>             c0t500003979C88FF5Dd0    ONLINE       0     0     0
>
>         cache
>
>           c0t5000CCA04B129374d0      ONLINE       0     0     0
>
>           c0t5000CCA04B12A1E4d0      ONLINE       0     0     0
>
>           c0t5000CCA04E27F1E8d0      ONLINE       0     0     0
>
>           c0t5000CCA04B12952Cd0      ONLINE       0     0     0
>
>         spares
>
>           c0t5000CCA02C4015E4d0      INUSE
>
>           c0t5000CCA02C416F7Cd0      AVAIL
>
>           c0t5000CCA02C2E7850d0      AVAIL
>
>           c0t5000CCA02C2E71F4d0      AVAIL
>
>
>
> *dmesg*
>
> Sep 24 18:03:59 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 18:03:59 znstor4-n1      /scsi_vhci/disk at g500003979c88ff55
> (sd224): Command Timeout on path lsc5/disk at w500003979c88ff57,0
>
> Sep 24 20:18:48 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:18:48 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:19:09 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:19:09 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:19:29 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:19:29 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:19:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:19:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:19:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:19:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:19:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:19:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:20:10 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:20:10 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:20:30 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:20:30 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:20:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:20:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:20:50 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:20:50 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:21:21 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:21:21 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:21:51 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:21:51 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:22:21 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:22:21 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:22:41 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:22:41 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:23:02 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:23:02 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:23:22 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:23:22 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:23:43 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:23:43 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:24:13 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:24:13 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:24:34 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:24:34 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:25:04 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:25:04 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:25:04 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:25:04 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:25:04 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:25:04 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:25:24 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:25:24 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:25:44 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:25:44 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:26:25 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:26:25 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:26:45 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:26:45 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:27:05 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:27:05 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:27:25 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:27:25 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:27:46 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:27:46 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:28:06 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:28:06 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:28:37 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:28:37 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:28:57 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:28:57 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:29:18 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:29:18 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:29:18 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:29:18 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:29:48 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:29:48 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:30:28 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:30:28 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:30:58 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:30:58 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc3/disk at w5000cca02c3f5a79,0
>
> Sep 24 20:48:30 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 20:48:30 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc5/disk at w5000cca02c3f5a7a,0
>
> Sep 24 20:51:44 znstor4-n1 scsi: [ID 107833 kern.warning] WARNING:
> /scsi_vhci/disk at g5000cca02c3f5a78 (sd218):
>
> Sep 24 20:51:44 znstor4-n1      drive offline
>
> Sep 24 21:44:01 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 21:44:01 znstor4-n1      /scsi_vhci/disk at g500003973c88b709
> (sd229): Command Timeout on path lsc4/disk at w500003973c88b70b,0
>
> Sep 24 21:48:31 znstor4-n1 scsi: [ID 243001 kern.warning] WARNING:
> /scsi_vhci (scsi_vhci0):
>
> Sep 24 21:48:31 znstor4-n1      /scsi_vhci/disk at g5000cca02c3f5a78
> (sd218): Command Timeout on path lsc5/disk at w5000cca02c3f5a7a,0
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170926/ab6b536d/attachment-0001.html>

From icoomnios at gmail.com  Wed Sep 27 07:57:28 2017
From: icoomnios at gmail.com (anthony omnios)
Date: Wed, 27 Sep 2017 09:57:28 +0200
Subject: [OmniOS-discuss] write amplification zvol
Message-ID: <CAPwVdD2RSS_oKrdbzMfFWZ8=-D1O4E3=Bi1GoUPyN2Bk4Ch8VQ@mail.gmail.com>

Hi,

i have a problem, i used many ISCSI zvol (for each vm), network traffic is
2MB/s between kvm host and filer but i write on disks many more than that.
I used a pool with separated mirror zil (intel s3710) and 8 ssd samsung
850 evo 1To

 zpool status
  pool: filervm2
 state: ONLINE
  scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20 15:45:48 2017
config:

        NAME                       STATE     READ WRITE CKSUM
        filervm2                   ONLINE       0     0     0
          mirror-0                 ONLINE       0     0     0
            c7t5002538D41657AAFd0  ONLINE       0     0     0
            c7t5002538D41F85C0Dd0  ONLINE       0     0     0
          mirror-2                 ONLINE       0     0     0
            c7t5002538D41CC7105d0  ONLINE       0     0     0
            c7t5002538D41CC7127d0  ONLINE       0     0     0
          mirror-3                 ONLINE       0     0     0
            c7t5002538D41CD7F7Ed0  ONLINE       0     0     0
            c7t5002538D41CD83FDd0  ONLINE       0     0     0
          mirror-4                 ONLINE       0     0     0
            c7t5002538D41CD7F7Ad0  ONLINE       0     0     0
            c7t5002538D41CD7F7Dd0  ONLINE       0     0     0
        logs
          mirror-1                 ONLINE       0     0     0
            c4t2d0                 ONLINE       0     0     0
            c4t4d0                 ONLINE       0     0     0

i used correct ashift of 13 for samsung 850 evo
zdb|grep ashift :

ashift: 13
ashift: 13
ashift: 13
ashift: 13
ashift: 13

But i write a lot on ssd every 5 seconds (many more than the network
traffic of 2 MB/s)

iostat -xn -d 1 :

 r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   11.0 3067.5  288.3 153457.4  6.8  0.5    2.2    0.2   5  14 filervm2
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 rpool
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c4t0d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c4t1d0
    0.0  552.6    0.0 17284.0  0.0  0.1    0.0    0.2   0   8 c4t2d0
    0.0  552.6    0.0 17284.0  0.0  0.1    0.0    0.2   0   8 c4t4d0
    1.0  233.3   48.1 10051.6  0.0  0.0    0.0    0.1   0   3
c7t5002538D41657AAFd0
    5.0  250.3  144.2 13207.3  0.0  0.0    0.0    0.1   0   3
c7t5002538D41CC7127d0
    2.0  254.3   24.0 13207.3  0.0  0.0    0.0    0.1   0   4
c7t5002538D41CC7105d0
    3.0  235.3   72.1 10051.6  0.0  0.0    0.0    0.1   0   3
c7t5002538D41F85C0Dd0
    0.0  228.3    0.0 16178.7  0.0  0.0    0.0    0.2   0   4
c7t5002538D41CD83FDd0
    0.0  225.3    0.0 16210.7  0.0  0.0    0.0    0.2   0   4
c7t5002538D41CD7F7Ed0
    0.0  282.3    0.0 19991.1  0.0  0.0    0.0    0.2   0   5
c7t5002538D41CD7F7Dd0
    0.0  280.3    0.0 19871.0  0.0  0.0    0.0    0.2   0   5
c7t5002538D41CD7F7Ad0

I used zvol of 64k, i try with 8k and problem is the same.

zfs get all filervm2/hdd-110022a :

NAME                  PROPERTY              VALUE                  SOURCE
filervm2/hdd-110022a  type                  volume                 -
filervm2/hdd-110022a  creation              Tue May 16 10:24 2017  -
filervm2/hdd-110022a  used                  5.26G                  -
filervm2/hdd-110022a  available             2.90T                  -
filervm2/hdd-110022a  referenced            5.24G                  -
filervm2/hdd-110022a  compressratio         3.99x                  -
filervm2/hdd-110022a  reservation           none                   default
filervm2/hdd-110022a  volsize               25G                    local
filervm2/hdd-110022a  volblocksize          64K                    -
filervm2/hdd-110022a  checksum              on                     default
filervm2/hdd-110022a  compression           lz4                    local
filervm2/hdd-110022a  readonly              off                    default
filervm2/hdd-110022a  copies                1                      default
filervm2/hdd-110022a  refreservation        none                   default
filervm2/hdd-110022a  primarycache          all                    default
filervm2/hdd-110022a  secondarycache        all                    default
filervm2/hdd-110022a  usedbysnapshots       15.4M                  -
filervm2/hdd-110022a  usedbydataset         5.24G                  -
filervm2/hdd-110022a  usedbychildren        0                      -
filervm2/hdd-110022a  usedbyrefreservation  0                      -
filervm2/hdd-110022a  logbias               latency                default
filervm2/hdd-110022a  dedup                 off                    default
filervm2/hdd-110022a  mlslabel              none                   default
filervm2/hdd-110022a  sync                  standard               local
filervm2/hdd-110022a  refcompressratio      3.99x                  -
filervm2/hdd-110022a  written               216K                   -
filervm2/hdd-110022a  logicalused           20.9G                  -
filervm2/hdd-110022a  logicalreferenced     20.9G                  -
filervm2/hdd-110022a  snapshot_limit        none                   default
filervm2/hdd-110022a  snapshot_count        none                   default
filervm2/hdd-110022a  redundant_metadata    all                    default

Sorry for my bad english.

What can be the problem ? thanks

Best regards,

Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170927/f86c83fe/attachment.html>

From apenner.it at gmail.com  Wed Sep 27 10:56:13 2017
From: apenner.it at gmail.com (Artem Penner)
Date: Wed, 27 Sep 2017 10:56:13 +0000
Subject: [OmniOS-discuss] write amplification zvol
In-Reply-To: <CAPwVdD2RSS_oKrdbzMfFWZ8=-D1O4E3=Bi1GoUPyN2Bk4Ch8VQ@mail.gmail.com>
References: <CAPwVdD2RSS_oKrdbzMfFWZ8=-D1O4E3=Bi1GoUPyN2Bk4Ch8VQ@mail.gmail.com>
Message-ID: <CAE9csxPxgZHE3Tid+P5SsaHXs3Xk0R6ypSGhnbMpvdkxT50pOA@mail.gmail.com>

Use https://github.com/richardelling/tools/blob/master/iscsisvrtop to
observe iscsi I/O


??, 27 ????. 2017 ?. ? 11:06, anthony omnios <icoomnios at gmail.com>:

> Hi,
>
> i have a problem, i used many ISCSI zvol (for each vm), network traffic is
> 2MB/s between kvm host and filer but i write on disks many more than that.
> I used a pool with separated mirror zil (intel s3710) and 8 ssd samsung
> 850 evo 1To
>
>  zpool status
>   pool: filervm2
>  state: ONLINE
>   scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20 15:45:48 2017
> config:
>
>         NAME                       STATE     READ WRITE CKSUM
>         filervm2                   ONLINE       0     0     0
>           mirror-0                 ONLINE       0     0     0
>             c7t5002538D41657AAFd0  ONLINE       0     0     0
>             c7t5002538D41F85C0Dd0  ONLINE       0     0     0
>           mirror-2                 ONLINE       0     0     0
>             c7t5002538D41CC7105d0  ONLINE       0     0     0
>             c7t5002538D41CC7127d0  ONLINE       0     0     0
>           mirror-3                 ONLINE       0     0     0
>             c7t5002538D41CD7F7Ed0  ONLINE       0     0     0
>             c7t5002538D41CD83FDd0  ONLINE       0     0     0
>           mirror-4                 ONLINE       0     0     0
>             c7t5002538D41CD7F7Ad0  ONLINE       0     0     0
>             c7t5002538D41CD7F7Dd0  ONLINE       0     0     0
>         logs
>           mirror-1                 ONLINE       0     0     0
>             c4t2d0                 ONLINE       0     0     0
>             c4t4d0                 ONLINE       0     0     0
>
> i used correct ashift of 13 for samsung 850 evo
> zdb|grep ashift :
>
> ashift: 13
> ashift: 13
> ashift: 13
> ashift: 13
> ashift: 13
>
> But i write a lot on ssd every 5 seconds (many more than the network
> traffic of 2 MB/s)
>
> iostat -xn -d 1 :
>
>  r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
>    11.0 3067.5  288.3 153457.4  6.8  0.5    2.2    0.2   5  14 filervm2
>     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 rpool
>     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c4t0d0
>     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c4t1d0
>     0.0  552.6    0.0 17284.0  0.0  0.1    0.0    0.2   0   8 c4t2d0
>     0.0  552.6    0.0 17284.0  0.0  0.1    0.0    0.2   0   8 c4t4d0
>     1.0  233.3   48.1 10051.6  0.0  0.0    0.0    0.1   0   3
> c7t5002538D41657AAFd0
>     5.0  250.3  144.2 13207.3  0.0  0.0    0.0    0.1   0   3
> c7t5002538D41CC7127d0
>     2.0  254.3   24.0 13207.3  0.0  0.0    0.0    0.1   0   4
> c7t5002538D41CC7105d0
>     3.0  235.3   72.1 10051.6  0.0  0.0    0.0    0.1   0   3
> c7t5002538D41F85C0Dd0
>     0.0  228.3    0.0 16178.7  0.0  0.0    0.0    0.2   0   4
> c7t5002538D41CD83FDd0
>     0.0  225.3    0.0 16210.7  0.0  0.0    0.0    0.2   0   4
> c7t5002538D41CD7F7Ed0
>     0.0  282.3    0.0 19991.1  0.0  0.0    0.0    0.2   0   5
> c7t5002538D41CD7F7Dd0
>     0.0  280.3    0.0 19871.0  0.0  0.0    0.0    0.2   0   5
> c7t5002538D41CD7F7Ad0
>
> I used zvol of 64k, i try with 8k and problem is the same.
>
> zfs get all filervm2/hdd-110022a :
>
> NAME                  PROPERTY              VALUE                  SOURCE
> filervm2/hdd-110022a  type                  volume                 -
> filervm2/hdd-110022a  creation              Tue May 16 10:24 2017  -
> filervm2/hdd-110022a  used                  5.26G                  -
> filervm2/hdd-110022a  available             2.90T                  -
> filervm2/hdd-110022a  referenced            5.24G                  -
> filervm2/hdd-110022a  compressratio         3.99x                  -
> filervm2/hdd-110022a  reservation           none                   default
> filervm2/hdd-110022a  volsize               25G                    local
> filervm2/hdd-110022a  volblocksize          64K                    -
> filervm2/hdd-110022a  checksum              on                     default
> filervm2/hdd-110022a  compression           lz4                    local
> filervm2/hdd-110022a  readonly              off                    default
> filervm2/hdd-110022a  copies                1                      default
> filervm2/hdd-110022a  refreservation        none                   default
> filervm2/hdd-110022a  primarycache          all                    default
> filervm2/hdd-110022a  secondarycache        all                    default
> filervm2/hdd-110022a  usedbysnapshots       15.4M                  -
> filervm2/hdd-110022a  usedbydataset         5.24G                  -
> filervm2/hdd-110022a  usedbychildren        0                      -
> filervm2/hdd-110022a  usedbyrefreservation  0                      -
> filervm2/hdd-110022a  logbias               latency                default
> filervm2/hdd-110022a  dedup                 off                    default
> filervm2/hdd-110022a  mlslabel              none                   default
> filervm2/hdd-110022a  sync                  standard               local
> filervm2/hdd-110022a  refcompressratio      3.99x                  -
> filervm2/hdd-110022a  written               216K                   -
> filervm2/hdd-110022a  logicalused           20.9G                  -
> filervm2/hdd-110022a  logicalreferenced     20.9G                  -
> filervm2/hdd-110022a  snapshot_limit        none                   default
> filervm2/hdd-110022a  snapshot_count        none                   default
> filervm2/hdd-110022a  redundant_metadata    all                    default
>
> Sorry for my bad english.
>
> What can be the problem ? thanks
>
> Best regards,
>
> Anthony
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170927/6e503820/attachment-0001.html>

From icoomnios at gmail.com  Wed Sep 27 15:12:25 2017
From: icoomnios at gmail.com (anthony omnios)
Date: Wed, 27 Sep 2017 17:12:25 +0200
Subject: [OmniOS-discuss] write amplification zvol
In-Reply-To: <CAE9csxPxgZHE3Tid+P5SsaHXs3Xk0R6ypSGhnbMpvdkxT50pOA@mail.gmail.com>
References: <CAPwVdD2RSS_oKrdbzMfFWZ8=-D1O4E3=Bi1GoUPyN2Bk4Ch8VQ@mail.gmail.com>
	<CAE9csxPxgZHE3Tid+P5SsaHXs3Xk0R6ypSGhnbMpvdkxT50pOA@mail.gmail.com>
Message-ID: <CAPwVdD1F9mTm3aR0-AC49iGVUPrDdbWXvAeT_UgKqAzyWud62w@mail.gmail.com>

 thanks, this is the result of the test:

./iscsisvrtop 1 30 >> /tmp/iscsisvrtop.txt
more /tmp/iscsisvrtop.txt :

Tracing... Please wait.
2017 Sep 27 17:01:48 load: 0.22 read_KB: 345 write_KB: 56
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247           4       0       0       0       0       0
0       0       0       0       0       0
1.1.193.250         105      91       1       0     345      56
3      56       4     756       0     100
all                 109      91       1       0     345      56
3      56       4     756       0       0
2017 Sep 27 17:01:49 load: 0.22 read_KB: 163 write_KB: 41
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          32      26       2       0     117      41
4      20       6     417       0     100
1.1.193.250          42      34       0       0      46       0
1       0       7       0       0       0
all                  74      60       2       0     163      41
2      20       7     417       0       0
2017 Sep 27 17:01:50 load: 0.22 read_KB: 499 write_KB: 232
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          45      40       3       0     210     196
5      65       5     763       0     100
1.1.193.250          77      65       2       0     288      36
4      18       4     439       0     100
all                 122     105       5       0     499     232
4      46       4     634       0       0
2017 Sep 27 17:01:51 load: 0.22 read_KB: 314 write_KB: 84
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247           3       1       0       0       0       0
0       0       2       0       0       0
1.1.193.250         100      88       4       0     313      84
3      21       4     396       0     100
all                 103      89       4       0     314      84
3      21       4     396       0       0
2017 Sep 27 17:01:52 load: 0.22 read_KB: 184 write_KB: 104
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          23      17       1       0      59      88
3      88       5     871       0     100
1.1.193.250          50      44       1       0     125      16
2      16       8     445       0     100
all                  73      61       2       0     184     104
3      52       7     658       0       0
2017 Sep 27 17:01:53 load: 0.22 read_KB: 250 write_KB: 1920
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247           7       6       0       0      12       0
2       0       6       0       0       0
1.1.193.250          71      44      16       0     263    1920       5
120       6    2531       0     100
all                  78      50      16       0     276    1920       5
120       6    2531       0       0
2017 Sep 27 17:01:54 load: 0.22 read_KB: 93 write_KB: 0
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247           7       0       0       0       0       0
0       0       0       0       0       0
1.1.193.250          38      28       0       0      70       0
2       0       6       0       0       0
all                  45      28       0       0      70       0
2       0       6       0       0       0
2017 Sep 27 17:01:55 load: 0.22 read_KB: 467 write_KB: 156
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          23      21       0       0      23       0
1       0       6       0       0       0
1.1.193.250         115     106       4       0     441     156
4      39       5     538       0     100
all                 138     127       4       0     464     156
3      39       5     538       0       0
2017 Sep 27 17:01:56 load: 0.22 read_KB: 485 write_KB: 152
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          16      13       0       0      22       0
1       0       2       0       0       0
1.1.193.250         133     119       4       0     462     152
3      38       4     427       0     100
all                 149     132       4       0     485     152
3      38       3     427       0       0
2017 Sep 27 17:01:57 load: 0.22 read_KB: 804 write_KB: 248
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          36      33       1       0     137     104       4
104       6    1064       0     100
1.1.193.250         133     131       2       0     667     144
5      72       5     885       0     100
all                 169     164       3       0     804     248
4      82       5     945       0       0
2017 Sep 27 17:01:58 load: 0.22 read_KB: 631 write_KB: 36
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          91      87       0       0     373       0
4       0       2       0       0       0
1.1.193.250          93      75       2       0     257      36
3      18       4     252       0     100
all                 184     162       2       0     631      36
3      18       3     252       0       0
2017 Sep 27 17:01:59 load: 0.21 read_KB: 1472 write_KB: 764
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.250          76      68       6       0     281     636       4
106       4     803       0     100
1.1.193.247         265     262       2       0    1191     128
4      64       3     482       0     100
all                 341     330       8       0    1472     764
4      95       3     723       0       0
2017 Sep 27 17:02:00 load: 0.21 read_KB: 3559 write_KB: 376
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          83      82       0       0     270       0
3       0       1       0       0       0
1.1.193.250         541     524       8       0    3289     376
6      47       6     359       0     100
all                 624     606       8       0    3559     376
5      47       5     359       0       0
2017 Sep 27 17:02:01 load: 0.21 read_KB: 2079 write_KB: 232
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247         120     118       0       0     612       0
5       0       2       0       0       0
1.1.193.250         418     416       2       0    1476     232       3
116       4     765       0     100
all                 538     534       2       0    2088     232       3
116       4     765       0       0
2017 Sep 27 17:02:02 load: 0.21 read_KB: 2123 write_KB: 168
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          84      80       3       0     317     168
3      56       4     292       0     100
1.1.193.250         367     366       0       0    1812       0
4       0       6       0       0       0
all                 451     446       3       0    2129     168
4      56       6     292       0       0
2017 Sep 27 17:02:03 load: 0.21 read_KB: 307 write_KB: 484
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          14      14       0       0      19       0
1       0       5       0       0       0
1.1.193.250          90      85       5       0     273     484
3      96       1     302       0     100
all                 104      99       5       0     292     484
2      96       2     302       0       0
2017 Sep 27 17:02:04 load: 0.22 read_KB: 298 write_KB: 0
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          10       4       0       0       2       0
0       0       9       0       0       0
1.1.193.250          85      70       0       0     296       0
4       0       5       0       0       0
all                  95      74       0       0     298       0
4       0       5       0       0       0
2017 Sep 27 17:02:05 load: 0.22 read_KB: 296 write_KB: 420
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247           1       0       0       0       0       0
0       0       0       0       0       0
1.1.193.250          86      76       5       0     306     420
4      84       6     739       0     100
all                  87      76       5       0     306     420
4      84       6     739       0       0
2017 Sep 27 17:02:06 load: 0.22 read_KB: 1149 write_KB: 379
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          66      56       3       0     310      75
5      25       4     538       0     100
1.1.193.250         182     166       5       0     828     304
4      60       4     581       0     100
all                 248     222       8       0    1138     379
5      47       4     565       0       0
2017 Sep 27 17:02:07 load: 0.23 read_KB: 615 write_KB: 164
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          77      75       2       0     374      28
4      14       4     399       0     100
1.1.193.250          89      82       2       0     241     136
2      68       3     266       0     100
all                 166     157       4       0     615     164
3      41       3     333       0       0
2017 Sep 27 17:02:08 load: 0.23 read_KB: 1505 write_KB: 9712
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247           9       6       0       0       7       0
1       0       5       0       0       0
1.1.193.250         302     166     124       0    1978   14288      11
115       3    2037       0     100
all                 311     172     124       0    1985   14288      11
115      11    2037       0       0
2017 Sep 27 17:02:09 load: 0.23 read_KB: 4267 write_KB: 61980
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          12       7       0       0      15       0
2       0       4       0       0       0
1.1.193.250         644     156     484       0    3772   57404      24
118       1    1728       0     100
all                 656     163     484       0    3787   57404      23
118       2    1728       0       0
2017 Sep 27 17:02:10 load: 0.24 read_KB: 610 write_KB: 48
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          13      10       1       0      95      16
9      16       4     374       0     100
1.1.193.250         116     107       1       0     514      32
4      32       6     495       0     100
all                 129     117       2       0     610      48
5      24       5     435       0       0
2017 Sep 27 17:02:11 load: 0.24 read_KB: 684 write_KB: 68
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          26      20       1       0      59      32
2      32       5     545       0     100
1.1.193.250         169     158       2       0     624      36
3      18       4     451       0     100
all                 195     178       3       0     684      68
3      22       4     482       0       0
2017 Sep 27 17:02:12 load: 0.24 read_KB: 154 write_KB: 176
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          14      12       1       0      46      96
3      96       5     854       0     100
1.1.193.250          43      35       1       0     492      80
14      80      28     947       0     100
all                  57      47       2       0     538     176
11      88      22     900       0       0
2017 Sep 27 17:02:13 load: 0.25 read_KB: 1134 write_KB: 12
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.250          36      24       1       0     191      12
7      12      16     469       0     100
1.1.193.247         122     117       0       0     558       0
4       0       5       0       0       0
all                 158     141       1       0     750      12
5      12       7     469       0       0
2017 Sep 27 17:02:14 load: 0.25 read_KB: 6357 write_KB: 90908
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247           5       2       0       0       1       0
0       0       9       0       0       0
1.1.193.250        1003     233     762       0    6357   90908      27
119      14    4844       0     100
all                1008     235     762       0    6358   90908      27
119      14    4844       0       0
2017 Sep 27 17:02:15 load: 0.25 read_KB: 243 write_KB: 0
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          23      18       0       0      37       0
2       0       2       0       0       0
1.1.193.250          70      58       0       0     207       0
3       0       4       0       0       0
all                  93      76       0       0     244       0
3       0       3       0       0       0
2017 Sep 27 17:02:16 load: 0.25 read_KB: 382 write_KB: 16
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247          42      38       0       0     191       0
5       0       5       0       0       0
1.1.193.250          59      50       1       0     189      16
3      16       8     427       0     100
all                 101      88       1       0     381      16
4      16       7     427       0       0
2017 Sep 27 17:02:17 load: 0.25 read_KB: 23 write_KB: 0
client              ops   reads  writes    nops   rd_bw   wr_bw  ard_sz
awr_sz    rd_t    wr_t   nop_t  align%
1.1.193.247           6       3       0       0       1       0
0       0       7       0       0       0
1.1.193.250          21      13       0       0      21       0
1       0       5       0       0       0
all                  27      16       0       0      23       0
1       0       6       0       0       0

How can i have 2MB/s network traffic (verified on omnios filer and also in
kvm host) and write on disk many more than that ?



2017-09-27 12:56 GMT+02:00 Artem Penner <apenner.it at gmail.com>:

> Use https://github.com/richardelling/tools/blob/master/iscsisvrtop to
> observe iscsi I/O
>
>
> ??, 27 ????. 2017 ?. ? 11:06, anthony omnios <icoomnios at gmail.com>:
>
>> Hi,
>>
>> i have a problem, i used many ISCSI zvol (for each vm), network traffic
>> is 2MB/s between kvm host and filer but i write on disks many more than
>> that. I used a pool with separated mirror zil (intel s3710) and 8 ssd
>> samsung  850 evo 1To
>>
>>  zpool status
>>   pool: filervm2
>>  state: ONLINE
>>   scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20 15:45:48 2017
>> config:
>>
>>         NAME                       STATE     READ WRITE CKSUM
>>         filervm2                   ONLINE       0     0     0
>>           mirror-0                 ONLINE       0     0     0
>>             c7t5002538D41657AAFd0  ONLINE       0     0     0
>>             c7t5002538D41F85C0Dd0  ONLINE       0     0     0
>>           mirror-2                 ONLINE       0     0     0
>>             c7t5002538D41CC7105d0  ONLINE       0     0     0
>>             c7t5002538D41CC7127d0  ONLINE       0     0     0
>>           mirror-3                 ONLINE       0     0     0
>>             c7t5002538D41CD7F7Ed0  ONLINE       0     0     0
>>             c7t5002538D41CD83FDd0  ONLINE       0     0     0
>>           mirror-4                 ONLINE       0     0     0
>>             c7t5002538D41CD7F7Ad0  ONLINE       0     0     0
>>             c7t5002538D41CD7F7Dd0  ONLINE       0     0     0
>>         logs
>>           mirror-1                 ONLINE       0     0     0
>>             c4t2d0                 ONLINE       0     0     0
>>             c4t4d0                 ONLINE       0     0     0
>>
>> i used correct ashift of 13 for samsung 850 evo
>> zdb|grep ashift :
>>
>> ashift: 13
>> ashift: 13
>> ashift: 13
>> ashift: 13
>> ashift: 13
>>
>> But i write a lot on ssd every 5 seconds (many more than the network
>> traffic of 2 MB/s)
>>
>> iostat -xn -d 1 :
>>
>>  r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
>>    11.0 3067.5  288.3 153457.4  6.8  0.5    2.2    0.2   5  14 filervm2
>>     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 rpool
>>     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c4t0d0
>>     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c4t1d0
>>     0.0  552.6    0.0 17284.0  0.0  0.1    0.0    0.2   0   8 c4t2d0
>>     0.0  552.6    0.0 17284.0  0.0  0.1    0.0    0.2   0   8 c4t4d0
>>     1.0  233.3   48.1 10051.6  0.0  0.0    0.0    0.1   0   3
>> c7t5002538D41657AAFd0
>>     5.0  250.3  144.2 13207.3  0.0  0.0    0.0    0.1   0   3
>> c7t5002538D41CC7127d0
>>     2.0  254.3   24.0 13207.3  0.0  0.0    0.0    0.1   0   4
>> c7t5002538D41CC7105d0
>>     3.0  235.3   72.1 10051.6  0.0  0.0    0.0    0.1   0   3
>> c7t5002538D41F85C0Dd0
>>     0.0  228.3    0.0 16178.7  0.0  0.0    0.0    0.2   0   4
>> c7t5002538D41CD83FDd0
>>     0.0  225.3    0.0 16210.7  0.0  0.0    0.0    0.2   0   4
>> c7t5002538D41CD7F7Ed0
>>     0.0  282.3    0.0 19991.1  0.0  0.0    0.0    0.2   0   5
>> c7t5002538D41CD7F7Dd0
>>     0.0  280.3    0.0 19871.0  0.0  0.0    0.0    0.2   0   5
>> c7t5002538D41CD7F7Ad0
>>
>> I used zvol of 64k, i try with 8k and problem is the same.
>>
>> zfs get all filervm2/hdd-110022a :
>>
>> NAME                  PROPERTY              VALUE                  SOURCE
>> filervm2/hdd-110022a  type                  volume                 -
>> filervm2/hdd-110022a  creation              Tue May 16 10:24 2017  -
>> filervm2/hdd-110022a  used                  5.26G                  -
>> filervm2/hdd-110022a  available             2.90T                  -
>> filervm2/hdd-110022a  referenced            5.24G                  -
>> filervm2/hdd-110022a  compressratio         3.99x                  -
>> filervm2/hdd-110022a  reservation           none                   default
>> filervm2/hdd-110022a  volsize               25G                    local
>> filervm2/hdd-110022a  volblocksize          64K                    -
>> filervm2/hdd-110022a  checksum              on                     default
>> filervm2/hdd-110022a  compression           lz4                    local
>> filervm2/hdd-110022a  readonly              off                    default
>> filervm2/hdd-110022a  copies                1                      default
>> filervm2/hdd-110022a  refreservation        none                   default
>> filervm2/hdd-110022a  primarycache          all                    default
>> filervm2/hdd-110022a  secondarycache        all                    default
>> filervm2/hdd-110022a  usedbysnapshots       15.4M                  -
>> filervm2/hdd-110022a  usedbydataset         5.24G                  -
>> filervm2/hdd-110022a  usedbychildren        0                      -
>> filervm2/hdd-110022a  usedbyrefreservation  0                      -
>> filervm2/hdd-110022a  logbias               latency                default
>> filervm2/hdd-110022a  dedup                 off                    default
>> filervm2/hdd-110022a  mlslabel              none                   default
>> filervm2/hdd-110022a  sync                  standard               local
>> filervm2/hdd-110022a  refcompressratio      3.99x                  -
>> filervm2/hdd-110022a  written               216K                   -
>> filervm2/hdd-110022a  logicalused           20.9G                  -
>> filervm2/hdd-110022a  logicalreferenced     20.9G                  -
>> filervm2/hdd-110022a  snapshot_limit        none                   default
>> filervm2/hdd-110022a  snapshot_count        none                   default
>> filervm2/hdd-110022a  redundant_metadata    all                    default
>>
>> Sorry for my bad english.
>>
>> What can be the problem ? thanks
>>
>> Best regards,
>>
>> Anthony
>>
>>
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170927/b4c0e55f/attachment-0001.html>

From sergey57 at gmail.com  Wed Sep 27 19:31:05 2017
From: sergey57 at gmail.com (sergey ivanov)
Date: Wed, 27 Sep 2017 15:31:05 -0400
Subject: [OmniOS-discuss] OmniOS based redundant NFS
Message-ID: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>

Hi,
as end-of-life of r151014 approaches, we are planning upgrade for our
NFS servers.
I'm thinking about 2 servers providing ISCSI targets, and 2 another
OmniOS servers using these ISCSI block devices in mirrored ZPOOL
setup. IP address for NFS service can be a floating IP between those 2
servers.
I have the following questions:
1. Are there any advantages to have separate ISCSI target servers and
NFS servers or I should better combine one ISCSI target and NFS
server on each of 2 hosts?
2. I do not want snapshots, checksums, and other ZFS features for
block devices at the level where they are exported as ISCSI targets, -
I would prefer these features at the level where these block devices
are combined into mirror Zpools. Maybe it's better to have these ISCSI
target servers running some not so advanced OS and have 2 Linux boxes?
3. But if I have SSD for intent log and for cache, - maybe they can
improve performance for ZVOLs used as block devices for ISCSI targets?

Does anybody have experience setting up such redundant NFS servers?
-- 
Regards,
Sergey Ivanov

From stephan.budach at jvm.de  Wed Sep 27 20:34:14 2017
From: stephan.budach at jvm.de (Stephan Budach)
Date: Wed, 27 Sep 2017 22:34:14 +0200 (CEST)
Subject: [OmniOS-discuss] OmniOS based redundant NFS
In-Reply-To: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>
References: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>
Message-ID: <634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de>

Hi Sergey,

----- Urspr?ngliche Mail -----
> Von: "sergey ivanov" <sergey57 at gmail.com>
> An: "omnios-discuss" <omnios-discuss at lists.omniti.com>
> Gesendet: Mittwoch, 27. September 2017 21:31:05
> Betreff: [OmniOS-discuss] OmniOS based redundant NFS
> 
> Hi,
> as end-of-life of r151014 approaches, we are planning upgrade for our
> NFS servers.
> I'm thinking about 2 servers providing ISCSI targets, and 2 another
> OmniOS servers using these ISCSI block devices in mirrored ZPOOL
> setup. IP address for NFS service can be a floating IP between those
> 2
> servers.

This is quite the same setup, as we have it. I am running two omniOS hosts as iSCSI targets and RSF-1 on two other omniOS hosts as NFS heads, where the NFS VIPs are failling over between. This setup has been very stable for quite some time and failovers have been occurring on a couple of occasions.

> I have the following questions:
> 1. Are there any advantages to have separate ISCSI target servers and
> NFS servers or I should better combine one ISCSI target and NFS
> server on each of 2 hosts?

The reason to use two x two seperate servers is, that the mirrored zpool's vdevs look the same on each NFS head.
This makes for a very straight forward setup, but on the other hand brings some interesting decisions to the table, when it comes to the design of the iSCSI targets.

We all know, that ZFS works best when presented with raw devices and the closest to that would be iSCSI to raw devices/partitions. However, this will leave you to decide how you want to arrange your targets. If you chose to have fewer targets with more LUNs, you will face some interesting challenges when it comes to device failures, which will have you to offline that whole target on your NFS heads, leaving you running with a degraded zpool on your NFS head. I just went through that and I can tell you that you will need to prepare for such a case, and the more drives you are using the more challenging it becomes.


> 2. I do not want snapshots, checksums, and other ZFS features for
> block devices at the level where they are exported as ISCSI targets,
> -
> I would prefer these features at the level where these block devices
> are combined into mirror Zpools. Maybe it's better to have these
> ISCSI
> target servers running some not so advanced OS and have 2 Linux
> boxes?

Me neither, but I chose omniOS as my iSCSI targets nevertheless. I do make heavy use of ZFS' features like snapshots and clones on my NFS heads and I am very comfortable with that. No bells and whistles on my target nodes.

> 3. But if I have SSD for intent log and for cache, - maybe they can
> improve performance for ZVOLs used as block devices for ISCSI
> targets?
> 

It will depend on your workload. I do also have some S3700s for ZIL on some of my iSCSI-based ZPOOls.

> Does anybody have experience setting up such redundant NFS servers?

Well? yes, and afaik, there are also some other people around this list, who are using similar setups, short of the iSCS target nodes, providing non-ZFS based LUNs, that seems to be more exotic? ;)

> --
> Regards,
> Sergey Ivanov

Cheers,
Stephan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5546 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170927/0e209f0a/attachment.bin>

From sergey57 at gmail.com  Wed Sep 27 21:15:49 2017
From: sergey57 at gmail.com (sergey ivanov)
Date: Wed, 27 Sep 2017 17:15:49 -0400
Subject: [OmniOS-discuss] OmniOS based redundant NFS
In-Reply-To: <634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de>
References: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>
	<634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de>
Message-ID: <CAAv=BDG0msF00TbmBfZ+E4Vg-t74JkrFmPsEQociK7QM9bSkqQ@mail.gmail.com>

Thanks, Stephan!

Please, explain "The reason to use two x two separate servers is, that
the mirrored zpool's vdevs look the same on each NFS head".

I understand that, if I want to have the same zpool based on iscsi
devices, I should not mix local disks with iscsi target disks.

But I think I can have 2 computers, each exporting a set of local
disks as iscsi targets. And to have iscsi initiators on the same
computers importing these targets to build zpools.

Also, looking at sbdadm, I think I can 'create lu /dev/rdsk/c0t0d3s2'.

Ok, I think I would better try it and report how it goes.
-- 
  Regards,
  Sergey.

-- 

Regards,
Sergey Ivanov


On Wed, Sep 27, 2017 at 4:34 PM, Stephan Budach <stephan.budach at jvm.de> wrote:
> Hi Sergey,
>
> ----- Urspr?ngliche Mail -----
>> Von: "sergey ivanov" <sergey57 at gmail.com>
>> An: "omnios-discuss" <omnios-discuss at lists.omniti.com>
>> Gesendet: Mittwoch, 27. September 2017 21:31:05
>> Betreff: [OmniOS-discuss] OmniOS based redundant NFS
>>
>> Hi,
>> as end-of-life of r151014 approaches, we are planning upgrade for our
>> NFS servers.
>> I'm thinking about 2 servers providing ISCSI targets, and 2 another
>> OmniOS servers using these ISCSI block devices in mirrored ZPOOL
>> setup. IP address for NFS service can be a floating IP between those
>> 2
>> servers.
>
> This is quite the same setup, as we have it. I am running two omniOS hosts as iSCSI targets and RSF-1 on two other omniOS hosts as NFS heads, where the NFS VIPs are failling over between. This setup has been very stable for quite some time and failovers have been occurring on a couple of occasions.
>
>> I have the following questions:
>> 1. Are there any advantages to have separate ISCSI target servers and
>> NFS servers or I should better combine one ISCSI target and NFS
>> server on each of 2 hosts?
>
> The reason to use two x two seperate servers is, that the mirrored zpool's vdevs look the same on each NFS head.
> This makes for a very straight forward setup, but on the other hand brings some interesting decisions to the table, when it comes to the design of the iSCSI targets.
>
> We all know, that ZFS works best when presented with raw devices and the closest to that would be iSCSI to raw devices/partitions. However, this will leave you to decide how you want to arrange your targets. If you chose to have fewer targets with more LUNs, you will face some interesting challenges when it comes to device failures, which will have you to offline that whole target on your NFS heads, leaving you running with a degraded zpool on your NFS head. I just went through that and I can tell you that you will need to prepare for such a case, and the more drives you are using the more challenging it becomes.
>
>
>> 2. I do not want snapshots, checksums, and other ZFS features for
>> block devices at the level where they are exported as ISCSI targets,
>> -
>> I would prefer these features at the level where these block devices
>> are combined into mirror Zpools. Maybe it's better to have these
>> ISCSI
>> target servers running some not so advanced OS and have 2 Linux
>> boxes?
>
> Me neither, but I chose omniOS as my iSCSI targets nevertheless. I do make heavy use of ZFS' features like snapshots and clones on my NFS heads and I am very comfortable with that. No bells and whistles on my target nodes.
>
>> 3. But if I have SSD for intent log and for cache, - maybe they can
>> improve performance for ZVOLs used as block devices for ISCSI
>> targets?
>>
>
> It will depend on your workload. I do also have some S3700s for ZIL on some of my iSCSI-based ZPOOls.
>
>> Does anybody have experience setting up such redundant NFS servers?
>
> Well? yes, and afaik, there are also some other people around this list, who are using similar setups, short of the iSCS target nodes, providing non-ZFS based LUNs, that seems to be more exotic? ;)
>
>> --
>> Regards,
>> Sergey Ivanov
>
> Cheers,
> Stephan

From richard.elling at richardelling.com  Wed Sep 27 23:29:52 2017
From: richard.elling at richardelling.com (Richard Elling)
Date: Wed, 27 Sep 2017 16:29:52 -0700
Subject: [OmniOS-discuss] write amplification zvol
In-Reply-To: <CAPwVdD2RSS_oKrdbzMfFWZ8=-D1O4E3=Bi1GoUPyN2Bk4Ch8VQ@mail.gmail.com>
References: <CAPwVdD2RSS_oKrdbzMfFWZ8=-D1O4E3=Bi1GoUPyN2Bk4Ch8VQ@mail.gmail.com>
Message-ID: <F3A29A1B-48A4-454B-BD8F-07F0B73CB5BE@richardelling.com>

Comment below...

> On Sep 27, 2017, at 12:57 AM, anthony omnios <icoomnios at gmail.com> wrote:
> 
> Hi,
> 
> i have a problem, i used many ISCSI zvol (for each vm), network traffic is 2MB/s between kvm host and filer but i write on disks many more than that. I used a pool with separated mirror zil (intel s3710) and 8 ssd samsung  850 evo 1To
> 
>  zpool status                        
>   pool: filervm2
>  state: ONLINE
>   scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20 15:45:48 2017
> config:
> 
>         NAME                       STATE     READ WRITE CKSUM
>         filervm2                   ONLINE       0     0     0
>           mirror-0                 ONLINE       0     0     0
>             c7t5002538D41657AAFd0  ONLINE       0     0     0
>             c7t5002538D41F85C0Dd0  ONLINE       0     0     0
>           mirror-2                 ONLINE       0     0     0
>             c7t5002538D41CC7105d0  ONLINE       0     0     0
>             c7t5002538D41CC7127d0  ONLINE       0     0     0
>           mirror-3                 ONLINE       0     0     0
>             c7t5002538D41CD7F7Ed0  ONLINE       0     0     0
>             c7t5002538D41CD83FDd0  ONLINE       0     0     0
>           mirror-4                 ONLINE       0     0     0
>             c7t5002538D41CD7F7Ad0  ONLINE       0     0     0
>             c7t5002538D41CD7F7Dd0  ONLINE       0     0     0
>         logs
>           mirror-1                 ONLINE       0     0     0
>             c4t2d0                 ONLINE       0     0     0
>             c4t4d0                 ONLINE       0     0     0
> 
> i used correct ashift of 13 for samsung 850 evo
> zdb|grep ashift :
> 
> ashift: 13
> ashift: 13
> ashift: 13
> ashift: 13
> ashift: 13
> 
> But i write a lot on ssd every 5 seconds (many more than the network traffic of 2 MB/s)
> 
> iostat -xn -d 1 : 
> 
>  r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
>    11.0 3067.5  288.3 153457.4  6.8  0.5    2.2    0.2   5  14 filervm2

filervm2 is seeing 3067 writes per second. This is the interface to the upper layers.
These writes are small.

>     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 rpool
>     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c4t0d0
>     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c4t1d0
>     0.0  552.6    0.0 17284.0  0.0  0.1    0.0    0.2   0   8 c4t2d0
>     0.0  552.6    0.0 17284.0  0.0  0.1    0.0    0.2   0   8 c4t4d0

The log devices are seeing 552 writes per second and since sync=standard that 
means that the upper layers are requesting syncs.

>     1.0  233.3   48.1 10051.6  0.0  0.0    0.0    0.1   0   3 c7t5002538D41657AAFd0
>     5.0  250.3  144.2 13207.3  0.0  0.0    0.0    0.1   0   3 c7t5002538D41CC7127d0
>     2.0  254.3   24.0 13207.3  0.0  0.0    0.0    0.1   0   4 c7t5002538D41CC7105d0
>     3.0  235.3   72.1 10051.6  0.0  0.0    0.0    0.1   0   3 c7t5002538D41F85C0Dd0
>     0.0  228.3    0.0 16178.7  0.0  0.0    0.0    0.2   0   4 c7t5002538D41CD83FDd0
>     0.0  225.3    0.0 16210.7  0.0  0.0    0.0    0.2   0   4 c7t5002538D41CD7F7Ed0
>     0.0  282.3    0.0 19991.1  0.0  0.0    0.0    0.2   0   5 c7t5002538D41CD7F7Dd0
>     0.0  280.3    0.0 19871.0  0.0  0.0    0.0    0.2   0   5 c7t5002538D41CD7F7Ad0

The pool disks see 1989 writes per second total or 994 writes per second logically.

It seems to me that reducing 3067 requested writes to 994 logical writes is the opposite
of amplification. What do you expect?
 -- richard

> 
> I used zvol of 64k, i try with 8k and problem is the same.
> 
> zfs get all filervm2/hdd-110022a :
> 
> NAME                  PROPERTY              VALUE                  SOURCE
> filervm2/hdd-110022a  type                  volume                 -
> filervm2/hdd-110022a  creation              Tue May 16 10:24 2017  -
> filervm2/hdd-110022a  used                  5.26G                  -
> filervm2/hdd-110022a  available             2.90T                  -
> filervm2/hdd-110022a  referenced            5.24G                  -
> filervm2/hdd-110022a  compressratio         3.99x                  -
> filervm2/hdd-110022a  reservation           none                   default
> filervm2/hdd-110022a  volsize               25G                    local
> filervm2/hdd-110022a  volblocksize          64K                    -
> filervm2/hdd-110022a  checksum              on                     default
> filervm2/hdd-110022a  compression           lz4                    local
> filervm2/hdd-110022a  readonly              off                    default
> filervm2/hdd-110022a  copies                1                      default
> filervm2/hdd-110022a  refreservation        none                   default
> filervm2/hdd-110022a  primarycache          all                    default
> filervm2/hdd-110022a  secondarycache        all                    default
> filervm2/hdd-110022a  usedbysnapshots       15.4M                  -
> filervm2/hdd-110022a  usedbydataset         5.24G                  -
> filervm2/hdd-110022a  usedbychildren        0                      -
> filervm2/hdd-110022a  usedbyrefreservation  0                      -
> filervm2/hdd-110022a  logbias               latency                default
> filervm2/hdd-110022a  dedup                 off                    default
> filervm2/hdd-110022a  mlslabel              none                   default
> filervm2/hdd-110022a  sync                  standard               local
> filervm2/hdd-110022a  refcompressratio      3.99x                  -
> filervm2/hdd-110022a  written               216K                   -
> filervm2/hdd-110022a  logicalused           20.9G                  -
> filervm2/hdd-110022a  logicalreferenced     20.9G                  -
> filervm2/hdd-110022a  snapshot_limit        none                   default
> filervm2/hdd-110022a  snapshot_count        none                   default
> filervm2/hdd-110022a  redundant_metadata    all                    default
> 
> Sorry for my bad english.
> 
> What can be the problem ? thanks
> 
> Best regards,
> 
> Anthony
> 
> 
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss


From stephan.budach at jvm.de  Thu Sep 28 04:49:25 2017
From: stephan.budach at jvm.de (Stephan Budach)
Date: Thu, 28 Sep 2017 06:49:25 +0200 (CEST)
Subject: [OmniOS-discuss] OmniOS based redundant NFS
In-Reply-To: <CAAv=BDG0msF00TbmBfZ+E4Vg-t74JkrFmPsEQociK7QM9bSkqQ@mail.gmail.com>
References: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>
	<634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de>
	<CAAv=BDG0msF00TbmBfZ+E4Vg-t74JkrFmPsEQociK7QM9bSkqQ@mail.gmail.com>
Message-ID: <968909667.804.1506574164745.JavaMail.stephan.budach@budy.stephanbudach.de>

Hi Sergey,

----- Urspr?ngliche Mail -----
> Von: "sergey ivanov" <sergey57 at gmail.com>
> An: "Stephan Budach" <stephan.budach at jvm.de>
> CC: "omnios-discuss" <omnios-discuss at lists.omniti.com>
> Gesendet: Mittwoch, 27. September 2017 23:15:49
> Betreff: Re: [OmniOS-discuss] OmniOS based redundant NFS
> 
> Thanks, Stephan!
> 
> Please, explain "The reason to use two x two separate servers is,
> that
> the mirrored zpool's vdevs look the same on each NFS head".
> 
> I understand that, if I want to have the same zpool based on iscsi
> devices, I should not mix local disks with iscsi target disks.
> 
> But I think I can have 2 computers, each exporting a set of local
> disks as iscsi targets. And to have iscsi initiators on the same
> computers importing these targets to build zpools.
> 
> Also, looking at sbdadm, I think I can 'create lu
> /dev/rdsk/c0t0d3s2'.
> 
> Ok, I think I would better try it and report how it goes.

Actually, things can become quite complex, I'd like to reduce the "mental" involvement to the absolute minimum, mainly because we often faced a situation where something would suddenly break, which had been running for a long time without problems. This is when peeple start? well maybe not panicking, but having to recap what the current setup was like and what they had to do to tackle this.

So, uniformity is a great deal of help on such systems - at least for us. Technically, there is no issue with mixing local and remote iSCST targets on the same node, which serves as an iSCSI target and a NFS head.

Also, if one of the nodes really goes down, you will be loosing your failover NFS head as well, maybe not a big deal and depending on your requirements okay. I do have such a setup as well, although only for an archive ZPOOL, where I can tolerate this reduced redundancy for the benefit of a more lightweight setup.

Cheers,
Stephan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5546 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170928/2f23c01b/attachment.bin>

From apenner.it at gmail.com  Thu Sep 28 07:25:15 2017
From: apenner.it at gmail.com (Artem Penner)
Date: Thu, 28 Sep 2017 07:25:15 +0000
Subject: [OmniOS-discuss] OmniOS based redundant NFS
In-Reply-To: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>
References: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>
Message-ID: <CAE9csxNg9rPk6_7heX2Dje=3y_LKA3mZYwDceXoZACn8YPLXvg@mail.gmail.com>

Hi, Sergey.
My be following information will be useful for you.
We use solaris as HA NAS in following configuration:

DATA_DISKS: HUC101818CS4204 / HUH728080AL5204
SLOG: PX04SHB020 / PX05SMB040 / HUSMH8020BSS204  (PX04SHB020 has best
performance
https://dhelios.blogspot.ru/2016/11/ssd-hgst-husmh8020bss204_11.html)
L2ARC: HUSMH8080BSS204 / PX05SMB080

Servers:
2 x Cisco UCS 240M4 ( 2 x E5-2697A, 768gb ram, 3 local hard disk for OS, 2
x UCSC-SAS9300-8E, 2 x Intel X520 Dual Port 10Gb SFP+ Adapter)

JBODS:
216BE2C-R741JBOD - for SFF disks
SC847E2C-R1K28JBOD - for LFF disks

We have two HA variant:
1) Solaris 11.3 + Solaris Cluster 4.3 (for VMware)
2) OmniOS + RSF-1  (as Cinder for Openstack)

If you need some additional info, I'll be glad to share any information
that I have.

??, 27 ????. 2017 ?. ? 22:39, sergey ivanov <sergey57 at gmail.com>:

> Hi,
> as end-of-life of r151014 approaches, we are planning upgrade for our
> NFS servers.
> I'm thinking about 2 servers providing ISCSI targets, and 2 another
> OmniOS servers using these ISCSI block devices in mirrored ZPOOL
> setup. IP address for NFS service can be a floating IP between those 2
> servers.
> I have the following questions:
> 1. Are there any advantages to have separate ISCSI target servers and
> NFS servers or I should better combine one ISCSI target and NFS
> server on each of 2 hosts?
> 2. I do not want snapshots, checksums, and other ZFS features for
> block devices at the level where they are exported as ISCSI targets, -
> I would prefer these features at the level where these block devices
> are combined into mirror Zpools. Maybe it's better to have these ISCSI
> target servers running some not so advanced OS and have 2 Linux boxes?
> 3. But if I have SSD for intent log and for cache, - maybe they can
> improve performance for ZVOLs used as block devices for ISCSI targets?
>
> Does anybody have experience setting up such redundant NFS servers?
> --
> Regards,
> Sergey Ivanov
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170928/e231051b/attachment.html>

From icoomnios at gmail.com  Thu Sep 28 07:56:42 2017
From: icoomnios at gmail.com (anthony omnios)
Date: Thu, 28 Sep 2017 09:56:42 +0200
Subject: [OmniOS-discuss] write amplification zvol
In-Reply-To: <F3A29A1B-48A4-454B-BD8F-07F0B73CB5BE@richardelling.com>
References: <CAPwVdD2RSS_oKrdbzMfFWZ8=-D1O4E3=Bi1GoUPyN2Bk4Ch8VQ@mail.gmail.com>
	<F3A29A1B-48A4-454B-BD8F-07F0B73CB5BE@richardelling.com>
Message-ID: <CAPwVdD2MZ6XSvLYAc=4C9J3vdEgQqCUw0pUOyykxd92Ry-xfuA@mail.gmail.com>

Thanks Richard for your help.

My problem is that i have a network ISCSI traffic of 2 MB/s, each 5 seconds
i need to write on disks 10 MB of network traffic but on pool filervm2 I am
writing much more that, approximatively 60 MB each 5 seconds. Each ssd of
filervm2 is writting 15 MB every 5 second. When i check with smartmootools
every ssd is writing approximatively 250 GB of data each day.

How can i reduce amont of data writting on each ssd ? i have try to reduce
block size of zvol but it change nothing.

Anthony





2017-09-28 1:29 GMT+02:00 Richard Elling <richard.elling at richardelling.com>:

> Comment below...
>
> > On Sep 27, 2017, at 12:57 AM, anthony omnios <icoomnios at gmail.com>
> wrote:
> >
> > Hi,
> >
> > i have a problem, i used many ISCSI zvol (for each vm), network traffic
> is 2MB/s between kvm host and filer but i write on disks many more than
> that. I used a pool with separated mirror zil (intel s3710) and 8 ssd
> samsung  850 evo 1To
> >
> >  zpool status
> >   pool: filervm2
> >  state: ONLINE
> >   scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20 15:45:48
> 2017
> > config:
> >
> >         NAME                       STATE     READ WRITE CKSUM
> >         filervm2                   ONLINE       0     0     0
> >           mirror-0                 ONLINE       0     0     0
> >             c7t5002538D41657AAFd0  ONLINE       0     0     0
> >             c7t5002538D41F85C0Dd0  ONLINE       0     0     0
> >           mirror-2                 ONLINE       0     0     0
> >             c7t5002538D41CC7105d0  ONLINE       0     0     0
> >             c7t5002538D41CC7127d0  ONLINE       0     0     0
> >           mirror-3                 ONLINE       0     0     0
> >             c7t5002538D41CD7F7Ed0  ONLINE       0     0     0
> >             c7t5002538D41CD83FDd0  ONLINE       0     0     0
> >           mirror-4                 ONLINE       0     0     0
> >             c7t5002538D41CD7F7Ad0  ONLINE       0     0     0
> >             c7t5002538D41CD7F7Dd0  ONLINE       0     0     0
> >         logs
> >           mirror-1                 ONLINE       0     0     0
> >             c4t2d0                 ONLINE       0     0     0
> >             c4t4d0                 ONLINE       0     0     0
> >
> > i used correct ashift of 13 for samsung 850 evo
> > zdb|grep ashift :
> >
> > ashift: 13
> > ashift: 13
> > ashift: 13
> > ashift: 13
> > ashift: 13
> >
> > But i write a lot on ssd every 5 seconds (many more than the network
> traffic of 2 MB/s)
> >
> > iostat -xn -d 1 :
> >
> >  r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
> >    11.0 3067.5  288.3 153457.4  6.8  0.5    2.2    0.2   5  14 filervm2
>
> filervm2 is seeing 3067 writes per second. This is the interface to the
> upper layers.
> These writes are small.
>
> >     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 rpool
> >     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c4t0d0
> >     0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c4t1d0
> >     0.0  552.6    0.0 17284.0  0.0  0.1    0.0    0.2   0   8 c4t2d0
> >     0.0  552.6    0.0 17284.0  0.0  0.1    0.0    0.2   0   8 c4t4d0
>
> The log devices are seeing 552 writes per second and since sync=standard
> that
> means that the upper layers are requesting syncs.
>
> >     1.0  233.3   48.1 10051.6  0.0  0.0    0.0    0.1   0   3
> c7t5002538D41657AAFd0
> >     5.0  250.3  144.2 13207.3  0.0  0.0    0.0    0.1   0   3
> c7t5002538D41CC7127d0
> >     2.0  254.3   24.0 13207.3  0.0  0.0    0.0    0.1   0   4
> c7t5002538D41CC7105d0
> >     3.0  235.3   72.1 10051.6  0.0  0.0    0.0    0.1   0   3
> c7t5002538D41F85C0Dd0
> >     0.0  228.3    0.0 16178.7  0.0  0.0    0.0    0.2   0   4
> c7t5002538D41CD83FDd0
> >     0.0  225.3    0.0 16210.7  0.0  0.0    0.0    0.2   0   4
> c7t5002538D41CD7F7Ed0
> >     0.0  282.3    0.0 19991.1  0.0  0.0    0.0    0.2   0   5
> c7t5002538D41CD7F7Dd0
> >     0.0  280.3    0.0 19871.0  0.0  0.0    0.0    0.2   0   5
> c7t5002538D41CD7F7Ad0
>
> The pool disks see 1989 writes per second total or 994 writes per second
> logically.
>
> It seems to me that reducing 3067 requested writes to 994 logical writes
> is the opposite
> of amplification. What do you expect?
>  -- richard
>
> >
> > I used zvol of 64k, i try with 8k and problem is the same.
> >
> > zfs get all filervm2/hdd-110022a :
> >
> > NAME                  PROPERTY              VALUE                  SOURCE
> > filervm2/hdd-110022a  type                  volume                 -
> > filervm2/hdd-110022a  creation              Tue May 16 10:24 2017  -
> > filervm2/hdd-110022a  used                  5.26G                  -
> > filervm2/hdd-110022a  available             2.90T                  -
> > filervm2/hdd-110022a  referenced            5.24G                  -
> > filervm2/hdd-110022a  compressratio         3.99x                  -
> > filervm2/hdd-110022a  reservation           none
>  default
> > filervm2/hdd-110022a  volsize               25G                    local
> > filervm2/hdd-110022a  volblocksize          64K                    -
> > filervm2/hdd-110022a  checksum              on
>  default
> > filervm2/hdd-110022a  compression           lz4                    local
> > filervm2/hdd-110022a  readonly              off
> default
> > filervm2/hdd-110022a  copies                1
> default
> > filervm2/hdd-110022a  refreservation        none
>  default
> > filervm2/hdd-110022a  primarycache          all
> default
> > filervm2/hdd-110022a  secondarycache        all
> default
> > filervm2/hdd-110022a  usedbysnapshots       15.4M                  -
> > filervm2/hdd-110022a  usedbydataset         5.24G                  -
> > filervm2/hdd-110022a  usedbychildren        0                      -
> > filervm2/hdd-110022a  usedbyrefreservation  0                      -
> > filervm2/hdd-110022a  logbias               latency
> default
> > filervm2/hdd-110022a  dedup                 off
> default
> > filervm2/hdd-110022a  mlslabel              none
>  default
> > filervm2/hdd-110022a  sync                  standard               local
> > filervm2/hdd-110022a  refcompressratio      3.99x                  -
> > filervm2/hdd-110022a  written               216K                   -
> > filervm2/hdd-110022a  logicalused           20.9G                  -
> > filervm2/hdd-110022a  logicalreferenced     20.9G                  -
> > filervm2/hdd-110022a  snapshot_limit        none
>  default
> > filervm2/hdd-110022a  snapshot_count        none
>  default
> > filervm2/hdd-110022a  redundant_metadata    all
> default
> >
> > Sorry for my bad english.
> >
> > What can be the problem ? thanks
> >
> > Best regards,
> >
> > Anthony
> >
> >
> > _______________________________________________
> > OmniOS-discuss mailing list
> > OmniOS-discuss at lists.omniti.com
> > http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170928/532b8ecd/attachment-0001.html>

From stephan.budach at jvm.de  Thu Sep 28 08:33:17 2017
From: stephan.budach at jvm.de (Stephan Budach)
Date: Thu, 28 Sep 2017 10:33:17 +0200 (CEST)
Subject: [OmniOS-discuss] write amplification zvol
In-Reply-To: <CAPwVdD2MZ6XSvLYAc=4C9J3vdEgQqCUw0pUOyykxd92Ry-xfuA@mail.gmail.com>
References: <CAPwVdD2RSS_oKrdbzMfFWZ8=-D1O4E3=Bi1GoUPyN2Bk4Ch8VQ@mail.gmail.com>
	<F3A29A1B-48A4-454B-BD8F-07F0B73CB5BE@richardelling.com>
	<CAPwVdD2MZ6XSvLYAc=4C9J3vdEgQqCUw0pUOyykxd92Ry-xfuA@mail.gmail.com>
Message-ID: <1009260033.996.1506587596423.JavaMail.stephan.budach@stephanbudach.local>

----- Urspr?ngliche Mail ----- 

> Von: "anthony omnios" <icoomnios at gmail.com>
> An: "Richard Elling" <richard.elling at richardelling.com>
> CC: omnios-discuss at lists.omniti.com
> Gesendet: Donnerstag, 28. September 2017 09:56:42
> Betreff: Re: [OmniOS-discuss] write amplification zvol

> Thanks Richard for your help.

> My problem is that i have a network ISCSI traffic of 2 MB/s, each 5
> seconds i need to write on disks 10 MB of network traffic but on
> pool filervm2 I am writing much more that, approximatively 60 MB
> each 5 seconds. Each ssd of filervm2 is writting 15 MB every 5
> second. When i check with smartmootools every ssd is writing
> approximatively 250 GB of data each day.

> How can i reduce amont of data writting on each ssd ? i have try to
> reduce block size of zvol but it change nothing.

> Anthony

> 2017-09-28 1:29 GMT+02:00 Richard Elling <
> richard.elling at richardelling.com > :

> > Comment below...
> 

> > > On Sep 27, 2017, at 12:57 AM, anthony omnios <
> > > icoomnios at gmail.com
> > > > wrote:
> 
> > >
> 
> > > Hi,
> 
> > >
> 
> > > i have a problem, i used many ISCSI zvol (for each vm), network
> > > traffic is 2MB/s between kvm host and filer but i write on disks
> > > many more than that. I used a pool with separated mirror zil
> > > (intel s3710) and 8 ssd samsung 850 evo 1To
> 
> > >
> 
> > > zpool status
> 
> > > pool: filervm2
> 
> > > state: ONLINE
> 
> > > scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20
> > > 15:45:48
> > > 2017
> 
> > > config:
> 
> > >
> 
> > > NAME STATE READ WRITE CKSUM
> 
> > > filervm2 ONLINE 0 0 0
> 
> > > mirror-0 ONLINE 0 0 0
> 
> > > c7t5002538D41657AAFd0 ONLINE 0 0 0
> 
> > > c7t5002538D41F85C0Dd0 ONLINE 0 0 0
> 
> > > mirror-2 ONLINE 0 0 0
> 
> > > c7t5002538D41CC7105d0 ONLINE 0 0 0
> 
> > > c7t5002538D41CC7127d0 ONLINE 0 0 0
> 
> > > mirror-3 ONLINE 0 0 0
> 
> > > c7t5002538D41CD7F7Ed0 ONLINE 0 0 0
> 
> > > c7t5002538D41CD83FDd0 ONLINE 0 0 0
> 
> > > mirror-4 ONLINE 0 0 0
> 
> > > c7t5002538D41CD7F7Ad0 ONLINE 0 0 0
> 
> > > c7t5002538D41CD7F7Dd0 ONLINE 0 0 0
> 
> > > logs
> 
> > > mirror-1 ONLINE 0 0 0
> 
> > > c4t2d0 ONLINE 0 0 0
> 
> > > c4t4d0 ONLINE 0 0 0
> 
> > >
> 
> > > i used correct ashift of 13 for samsung 850 evo
> 
> > > zdb|grep ashift :
> 
> > >
> 
> > > ashift: 13
> 
> > > ashift: 13
> 
> > > ashift: 13
> 
> > > ashift: 13
> 
> > > ashift: 13
> 
> > >
> 
> > > But i write a lot on ssd every 5 seconds (many more than the
> > > network traffic of 2 MB/s)
> 
> > >
> 
> > > iostat -xn -d 1 :
> 
> > >
> 
> > > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
> 
> > > 11.0 3067.5 288.3 153457.4 6.8 0.5 2.2 0.2 5 14 filervm2
> 

> > filervm2 is seeing 3067 writes per second. This is the interface to
> > the upper layers.
> 
> > These writes are small.
> 

> > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 rpool
> 
> > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0
> 
> > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0
> 
> > > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t2d0
> 
> > > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t4d0
> 

> > The log devices are seeing 552 writes per second and since
> > sync=standard that
> 
> > means that the upper layers are requesting syncs.
> 

> > > 1.0 233.3 48.1 10051.6 0.0 0.0 0.0 0.1 0 3 c7t5002538D41657AAFd0
> 
> > > 5.0 250.3 144.2 13207.3 0.0 0.0 0.0 0.1 0 3 c7t5002538D41CC7127d0
> 
> > > 2.0 254.3 24.0 13207.3 0.0 0.0 0.0 0.1 0 4 c7t5002538D41CC7105d0
> 
> > > 3.0 235.3 72.1 10051.6 0.0 0.0 0.0 0.1 0 3 c7t5002538D41F85C0Dd0
> 
> > > 0.0 228.3 0.0 16178.7 0.0 0.0 0.0 0.2 0 4 c7t5002538D41CD83FDd0
> 
> > > 0.0 225.3 0.0 16210.7 0.0 0.0 0.0 0.2 0 4 c7t5002538D41CD7F7Ed0
> 
> > > 0.0 282.3 0.0 19991.1 0.0 0.0 0.0 0.2 0 5 c7t5002538D41CD7F7Dd0
> 
> > > 0.0 280.3 0.0 19871.0 0.0 0.0 0.0 0.2 0 5 c7t5002538D41CD7F7Ad0
> 

> > The pool disks see 1989 writes per second total or 994 writes per
> > second logically.
> 

> > It seems to me that reducing 3067 requested writes to 994 logical
> > writes is the opposite
> 
> > of amplification. What do you expect?
> 
> > -- richard
> 

> > >
> 
> > > I used zvol of 64k, i try with 8k and problem is the same.
> 
> > >
> 
> > > zfs get all filervm2/hdd-110022a :
> 
> > >
> 
> > > NAME PROPERTY VALUE SOURCE
> 
> > > filervm2/hdd-110022a type volume -
> 
> > > filervm2/hdd-110022a creation Tue May 16 10:24 2017 -
> 
> > > filervm2/hdd-110022a used 5.26G -
> 
> > > filervm2/hdd-110022a available 2.90T -
> 
> > > filervm2/hdd-110022a referenced 5.24G -
> 
> > > filervm2/hdd-110022a compressratio 3.99x -
> 
> > > filervm2/hdd-110022a reservation none default
> 
> > > filervm2/hdd-110022a volsize 25G local
> 
> > > filervm2/hdd-110022a volblocksize 64K -
> 
> > > filervm2/hdd-110022a checksum on default
> 
> > > filervm2/hdd-110022a compression lz4 local
> 
> > > filervm2/hdd-110022a readonly off default
> 
> > > filervm2/hdd-110022a copies 1 default
> 
> > > filervm2/hdd-110022a refreservation none default
> 
> > > filervm2/hdd-110022a primarycache all default
> 
> > > filervm2/hdd-110022a secondarycache all default
> 
> > > filervm2/hdd-110022a usedbysnapshots 15.4M -
> 
> > > filervm2/hdd-110022a usedbydataset 5.24G -
> 
> > > filervm2/hdd-110022a usedbychildren 0 -
> 
> > > filervm2/hdd-110022a usedbyrefreservation 0 -
> 
> > > filervm2/hdd-110022a logbias latency default
> 
> > > filervm2/hdd-110022a dedup off default
> 
> > > filervm2/hdd-110022a mlslabel none default
> 
> > > filervm2/hdd-110022a sync standard local
> 
> > > filervm2/hdd-110022a refcompressratio 3.99x -
> 
> > > filervm2/hdd-110022a written 216K -
> 
> > > filervm2/hdd-110022a logicalused 20.9G -
> 
> > > filervm2/hdd-110022a logicalreferenced 20.9G -
> 
> > > filervm2/hdd-110022a snapshot_limit none default
> 
> > > filervm2/hdd-110022a snapshot_count none default
> 
> > > filervm2/hdd-110022a redundant_metadata all default
> 
> > >
> 
> > > Sorry for my bad english.
> 
> > >
> 
> > > What can be the problem ? thanks
> 
> > >
> 
> > > Best regards,
> 
> > >
> 
> > > Anthony
> 

How did you setup your LUNs? Especially, what is the block size for those LUNs. Could it be, that you went with the default of 512b blocks, where the drives do have 4k or even 8k blocks?

Cheers,
Stephan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5546 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170928/02552c16/attachment.bin>

From icoomnios at gmail.com  Thu Sep 28 09:48:18 2017
From: icoomnios at gmail.com (anthony omnios)
Date: Thu, 28 Sep 2017 11:48:18 +0200
Subject: [OmniOS-discuss] write amplification zvol
In-Reply-To: <1009260033.996.1506587596423.JavaMail.stephan.budach@stephanbudach.local>
References: <CAPwVdD2RSS_oKrdbzMfFWZ8=-D1O4E3=Bi1GoUPyN2Bk4Ch8VQ@mail.gmail.com>
	<F3A29A1B-48A4-454B-BD8F-07F0B73CB5BE@richardelling.com>
	<CAPwVdD2MZ6XSvLYAc=4C9J3vdEgQqCUw0pUOyykxd92Ry-xfuA@mail.gmail.com>
	<1009260033.996.1506587596423.JavaMail.stephan.budach@stephanbudach.local>
Message-ID: <CAPwVdD3NJ=w7XqzgM62VvGBiNrB71_vQBxBKCsrsG8GMM+2LRg@mail.gmail.com>

Thanks for you help Stephan.

i have tried differents LUN with default of 512b and 4096:

LU Name: 600144F04D4F0600000059A588910001
    Operational Status: Online
    Provider Name     : sbd
    Alias             : /dev/zvol/rdsk/filervm2/hdd-110002b
    View Entry Count  : 1
    Data File         : /dev/zvol/rdsk/filervm2/hdd-110002b
    Meta File         : not set
    Size              : 26843545600
    Block Size        : 4096
    Management URL    : not set
    Vendor ID         : SUN
    Product ID        : COMSTAR
    Serial Num        : not set
    Write Protect     : Disabled
    Writeback Cache   : Disabled
    Access State      : Active

Problem is the same.

Cheers,

Anthony

2017-09-28 10:33 GMT+02:00 Stephan Budach <stephan.budach at jvm.de>:

> ----- Urspr?ngliche Mail -----
>
> > Von: "anthony omnios" <icoomnios at gmail.com>
> > An: "Richard Elling" <richard.elling at richardelling.com>
> > CC: omnios-discuss at lists.omniti.com
> > Gesendet: Donnerstag, 28. September 2017 09:56:42
> > Betreff: Re: [OmniOS-discuss] write amplification zvol
>
> > Thanks Richard for your help.
>
> > My problem is that i have a network ISCSI traffic of 2 MB/s, each 5
> > seconds i need to write on disks 10 MB of network traffic but on
> > pool filervm2 I am writing much more that, approximatively 60 MB
> > each 5 seconds. Each ssd of filervm2 is writting 15 MB every 5
> > second. When i check with smartmootools every ssd is writing
> > approximatively 250 GB of data each day.
>
> > How can i reduce amont of data writting on each ssd ? i have try to
> > reduce block size of zvol but it change nothing.
>
> > Anthony
>
> > 2017-09-28 1:29 GMT+02:00 Richard Elling <
> > richard.elling at richardelling.com > :
>
> > > Comment below...
> >
>
> > > > On Sep 27, 2017, at 12:57 AM, anthony omnios <
> > > > icoomnios at gmail.com
> > > > > wrote:
> >
> > > >
> >
> > > > Hi,
> >
> > > >
> >
> > > > i have a problem, i used many ISCSI zvol (for each vm), network
> > > > traffic is 2MB/s between kvm host and filer but i write on disks
> > > > many more than that. I used a pool with separated mirror zil
> > > > (intel s3710) and 8 ssd samsung 850 evo 1To
> >
> > > >
> >
> > > > zpool status
> >
> > > > pool: filervm2
> >
> > > > state: ONLINE
> >
> > > > scan: resilvered 406G in 0h22m with 0 errors on Wed Sep 20
> > > > 15:45:48
> > > > 2017
> >
> > > > config:
> >
> > > >
> >
> > > > NAME STATE READ WRITE CKSUM
> >
> > > > filervm2 ONLINE 0 0 0
> >
> > > > mirror-0 ONLINE 0 0 0
> >
> > > > c7t5002538D41657AAFd0 ONLINE 0 0 0
> >
> > > > c7t5002538D41F85C0Dd0 ONLINE 0 0 0
> >
> > > > mirror-2 ONLINE 0 0 0
> >
> > > > c7t5002538D41CC7105d0 ONLINE 0 0 0
> >
> > > > c7t5002538D41CC7127d0 ONLINE 0 0 0
> >
> > > > mirror-3 ONLINE 0 0 0
> >
> > > > c7t5002538D41CD7F7Ed0 ONLINE 0 0 0
> >
> > > > c7t5002538D41CD83FDd0 ONLINE 0 0 0
> >
> > > > mirror-4 ONLINE 0 0 0
> >
> > > > c7t5002538D41CD7F7Ad0 ONLINE 0 0 0
> >
> > > > c7t5002538D41CD7F7Dd0 ONLINE 0 0 0
> >
> > > > logs
> >
> > > > mirror-1 ONLINE 0 0 0
> >
> > > > c4t2d0 ONLINE 0 0 0
> >
> > > > c4t4d0 ONLINE 0 0 0
> >
> > > >
> >
> > > > i used correct ashift of 13 for samsung 850 evo
> >
> > > > zdb|grep ashift :
> >
> > > >
> >
> > > > ashift: 13
> >
> > > > ashift: 13
> >
> > > > ashift: 13
> >
> > > > ashift: 13
> >
> > > > ashift: 13
> >
> > > >
> >
> > > > But i write a lot on ssd every 5 seconds (many more than the
> > > > network traffic of 2 MB/s)
> >
> > > >
> >
> > > > iostat -xn -d 1 :
> >
> > > >
> >
> > > > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
> >
> > > > 11.0 3067.5 288.3 153457.4 6.8 0.5 2.2 0.2 5 14 filervm2
> >
>
> > > filervm2 is seeing 3067 writes per second. This is the interface to
> > > the upper layers.
> >
> > > These writes are small.
> >
>
> > > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 rpool
> >
> > > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t0d0
> >
> > > > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c4t1d0
> >
> > > > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t2d0
> >
> > > > 0.0 552.6 0.0 17284.0 0.0 0.1 0.0 0.2 0 8 c4t4d0
> >
>
> > > The log devices are seeing 552 writes per second and since
> > > sync=standard that
> >
> > > means that the upper layers are requesting syncs.
> >
>
> > > > 1.0 233.3 48.1 10051.6 0.0 0.0 0.0 0.1 0 3 c7t5002538D41657AAFd0
> >
> > > > 5.0 250.3 144.2 13207.3 0.0 0.0 0.0 0.1 0 3 c7t5002538D41CC7127d0
> >
> > > > 2.0 254.3 24.0 13207.3 0.0 0.0 0.0 0.1 0 4 c7t5002538D41CC7105d0
> >
> > > > 3.0 235.3 72.1 10051.6 0.0 0.0 0.0 0.1 0 3 c7t5002538D41F85C0Dd0
> >
> > > > 0.0 228.3 0.0 16178.7 0.0 0.0 0.0 0.2 0 4 c7t5002538D41CD83FDd0
> >
> > > > 0.0 225.3 0.0 16210.7 0.0 0.0 0.0 0.2 0 4 c7t5002538D41CD7F7Ed0
> >
> > > > 0.0 282.3 0.0 19991.1 0.0 0.0 0.0 0.2 0 5 c7t5002538D41CD7F7Dd0
> >
> > > > 0.0 280.3 0.0 19871.0 0.0 0.0 0.0 0.2 0 5 c7t5002538D41CD7F7Ad0
> >
>
> > > The pool disks see 1989 writes per second total or 994 writes per
> > > second logically.
> >
>
> > > It seems to me that reducing 3067 requested writes to 994 logical
> > > writes is the opposite
> >
> > > of amplification. What do you expect?
> >
> > > -- richard
> >
>
> > > >
> >
> > > > I used zvol of 64k, i try with 8k and problem is the same.
> >
> > > >
> >
> > > > zfs get all filervm2/hdd-110022a :
> >
> > > >
> >
> > > > NAME PROPERTY VALUE SOURCE
> >
> > > > filervm2/hdd-110022a type volume -
> >
> > > > filervm2/hdd-110022a creation Tue May 16 10:24 2017 -
> >
> > > > filervm2/hdd-110022a used 5.26G -
> >
> > > > filervm2/hdd-110022a available 2.90T -
> >
> > > > filervm2/hdd-110022a referenced 5.24G -
> >
> > > > filervm2/hdd-110022a compressratio 3.99x -
> >
> > > > filervm2/hdd-110022a reservation none default
> >
> > > > filervm2/hdd-110022a volsize 25G local
> >
> > > > filervm2/hdd-110022a volblocksize 64K -
> >
> > > > filervm2/hdd-110022a checksum on default
> >
> > > > filervm2/hdd-110022a compression lz4 local
> >
> > > > filervm2/hdd-110022a readonly off default
> >
> > > > filervm2/hdd-110022a copies 1 default
> >
> > > > filervm2/hdd-110022a refreservation none default
> >
> > > > filervm2/hdd-110022a primarycache all default
> >
> > > > filervm2/hdd-110022a secondarycache all default
> >
> > > > filervm2/hdd-110022a usedbysnapshots 15.4M -
> >
> > > > filervm2/hdd-110022a usedbydataset 5.24G -
> >
> > > > filervm2/hdd-110022a usedbychildren 0 -
> >
> > > > filervm2/hdd-110022a usedbyrefreservation 0 -
> >
> > > > filervm2/hdd-110022a logbias latency default
> >
> > > > filervm2/hdd-110022a dedup off default
> >
> > > > filervm2/hdd-110022a mlslabel none default
> >
> > > > filervm2/hdd-110022a sync standard local
> >
> > > > filervm2/hdd-110022a refcompressratio 3.99x -
> >
> > > > filervm2/hdd-110022a written 216K -
> >
> > > > filervm2/hdd-110022a logicalused 20.9G -
> >
> > > > filervm2/hdd-110022a logicalreferenced 20.9G -
> >
> > > > filervm2/hdd-110022a snapshot_limit none default
> >
> > > > filervm2/hdd-110022a snapshot_count none default
> >
> > > > filervm2/hdd-110022a redundant_metadata all default
> >
> > > >
> >
> > > > Sorry for my bad english.
> >
> > > >
> >
> > > > What can be the problem ? thanks
> >
> > > >
> >
> > > > Best regards,
> >
> > > >
> >
> > > > Anthony
> >
>
> How did you setup your LUNs? Especially, what is the block size for those
> LUNs. Could it be, that you went with the default of 512b blocks, where the
> drives do have 4k or even 8k blocks?
>
> Cheers,
> Stephan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170928/e05a9e18/attachment-0001.html>

From sergey57 at gmail.com  Fri Sep 29 20:14:02 2017
From: sergey57 at gmail.com (sergey ivanov)
Date: Fri, 29 Sep 2017 16:14:02 -0400
Subject: [OmniOS-discuss] OmniOS based redundant NFS
In-Reply-To: <CAE9csxNg9rPk6_7heX2Dje=3y_LKA3mZYwDceXoZACn8YPLXvg@mail.gmail.com>
References: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>
	<CAE9csxNg9rPk6_7heX2Dje=3y_LKA3mZYwDceXoZACn8YPLXvg@mail.gmail.com>
Message-ID: <CAAv=BDGeWjSpc5NdPbcj=PQsdVwLYX_820xa03_rNWAoM_kmaw@mail.gmail.com>

Thanks, Artem, it is a very good list of suitable hardware!
I have a question about ZIL.
I read that SSD for it should be about 8G. I did not see more than one
gigabyte allocated for ZIL on our servers. Can you comment it?
-- 
  Sergey.

Regards,
Sergey Ivanov


On Thu, Sep 28, 2017 at 3:25 AM, Artem Penner <apenner.it at gmail.com> wrote:
> Hi, Sergey.
> My be following information will be useful for you.
> We use solaris as HA NAS in following configuration:
>
> DATA_DISKS: HUC101818CS4204 / HUH728080AL5204
> SLOG: PX04SHB020 / PX05SMB040 / HUSMH8020BSS204  (PX04SHB020 has best
> performance
> https://dhelios.blogspot.ru/2016/11/ssd-hgst-husmh8020bss204_11.html)
> L2ARC: HUSMH8080BSS204 / PX05SMB080
>
> Servers:
> 2 x Cisco UCS 240M4 ( 2 x E5-2697A, 768gb ram, 3 local hard disk for OS, 2 x
> UCSC-SAS9300-8E, 2 x Intel X520 Dual Port 10Gb SFP+ Adapter)
>
> JBODS:
> 216BE2C-R741JBOD - for SFF disks
> SC847E2C-R1K28JBOD - for LFF disks
>
> We have two HA variant:
> 1) Solaris 11.3 + Solaris Cluster 4.3 (for VMware)
> 2) OmniOS + RSF-1  (as Cinder for Openstack)
>
> If you need some additional info, I'll be glad to share any information that
> I have.
>
> ??, 27 ????. 2017 ?. ? 22:39, sergey ivanov <sergey57 at gmail.com>:
>>
>> Hi,
>> as end-of-life of r151014 approaches, we are planning upgrade for our
>> NFS servers.
>> I'm thinking about 2 servers providing ISCSI targets, and 2 another
>> OmniOS servers using these ISCSI block devices in mirrored ZPOOL
>> setup. IP address for NFS service can be a floating IP between those 2
>> servers.
>> I have the following questions:
>> 1. Are there any advantages to have separate ISCSI target servers and
>> NFS servers or I should better combine one ISCSI target and NFS
>> server on each of 2 hosts?
>> 2. I do not want snapshots, checksums, and other ZFS features for
>> block devices at the level where they are exported as ISCSI targets, -
>> I would prefer these features at the level where these block devices
>> are combined into mirror Zpools. Maybe it's better to have these ISCSI
>> target servers running some not so advanced OS and have 2 Linux boxes?
>> 3. But if I have SSD for intent log and for cache, - maybe they can
>> improve performance for ZVOLs used as block devices for ISCSI targets?
>>
>> Does anybody have experience setting up such redundant NFS servers?
>> --
>> Regards,
>> Sergey Ivanov
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss

From apenner.it at gmail.com  Fri Sep 29 20:30:09 2017
From: apenner.it at gmail.com (Artem Penner)
Date: Fri, 29 Sep 2017 20:30:09 +0000
Subject: [OmniOS-discuss] OmniOS based redundant NFS
In-Reply-To: <CAAv=BDGeWjSpc5NdPbcj=PQsdVwLYX_820xa03_rNWAoM_kmaw@mail.gmail.com>
References: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>
	<CAE9csxNg9rPk6_7heX2Dje=3y_LKA3mZYwDceXoZACn8YPLXvg@mail.gmail.com>
	<CAAv=BDGeWjSpc5NdPbcj=PQsdVwLYX_820xa03_rNWAoM_kmaw@mail.gmail.com>
Message-ID: <CAE9csxOyw-yh69oq3DT2HBGKYi+XHUQeFHu_NrBb7N0F0ukSFw@mail.gmail.com>

As you see, we use Toshiba PX04SHB020 as SLOG devices. It's a 200gb in
size, but we reduce size of the disk to extend their life (increase DWPD)
we do it as follows:
root at znstor1-n1:~# sg_format /dev/rdsk/c0t500003979C88FF79d0
    TOSHIBA   PX04SHB020        0106   peripheral_type: disk [0x0]
      << supports protection information>>
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=390721968 [0x1749f1b0]
  Block size=512 [0x200]
Read Capacity (10) results:
   Number of logical blocks=390721968
   Logical block size=512 bytes


root at znstor1-n1:~# sg_format --size 4096 --format
/dev/rdsk/c0t500003979C88FF79d0
    TOSHIBA   PX04SHB020        0106   peripheral_type: disk [0x0]
      << supports protection information>>
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=390721968 [0x1749f1b0]
  Block size=512 [0x200]

A FORMAT will commence in 10 seconds
    ALL data on /dev/rdsk/c0t500003979C88FF79d0 will be DESTROYED
        Press control-C to abort
A FORMAT will commence in 5 seconds
    ALL data on /dev/rdsk/c0t500003979C88FF79d0 will be DESTROYED
        Press control-C to abort

Format has started
FORMAT Complete


root at znstor1-n1:~# sg_format --resize --count=0x1749f1b
/dev/rdsk/c0t500003979C88FF79d0
    TOSHIBA   PX04SHB020        0106   peripheral_type: disk [0x0]
      << supports protection information>>
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=48840246 [0x2e93e36]
  Block size=4096 [0x1000]
Resize operation seems to have been successful


root at znstor1-n1:~# update_drv -f sd
Every jbod has following layout (24 disk total)
2 disk for mirror slog
1 disk for l2arc
1 disk for spare
2 x raidz2 (8D + 2P)

I didn't observe used space on every SLOG mirror group more than 3gb.




??, 29 ????. 2017 ?. ? 23:14, sergey ivanov <sergey57 at gmail.com>:

> Thanks, Artem, it is a very good list of suitable hardware!
> I have a question about ZIL.
> I read that SSD for it should be about 8G. I did not see more than one
> gigabyte allocated for ZIL on our servers. Can you comment it?
> --
>   Sergey.
>
> Regards,
> Sergey Ivanov
>
>
> On Thu, Sep 28, 2017 at 3:25 AM, Artem Penner <apenner.it at gmail.com>
> wrote:
> > Hi, Sergey.
> > My be following information will be useful for you.
> > We use solaris as HA NAS in following configuration:
> >
> > DATA_DISKS: HUC101818CS4204 / HUH728080AL5204
> > SLOG: PX04SHB020 / PX05SMB040 / HUSMH8020BSS204  (PX04SHB020 has best
> > performance
> > https://dhelios.blogspot.ru/2016/11/ssd-hgst-husmh8020bss204_11.html)
> > L2ARC: HUSMH8080BSS204 / PX05SMB080
> >
> > Servers:
> > 2 x Cisco UCS 240M4 ( 2 x E5-2697A, 768gb ram, 3 local hard disk for OS,
> 2 x
> > UCSC-SAS9300-8E, 2 x Intel X520 Dual Port 10Gb SFP+ Adapter)
> >
> > JBODS:
> > 216BE2C-R741JBOD - for SFF disks
> > SC847E2C-R1K28JBOD - for LFF disks
> >
> > We have two HA variant:
> > 1) Solaris 11.3 + Solaris Cluster 4.3 (for VMware)
> > 2) OmniOS + RSF-1  (as Cinder for Openstack)
> >
> > If you need some additional info, I'll be glad to share any information
> that
> > I have.
> >
> > ??, 27 ????. 2017 ?. ? 22:39, sergey ivanov <sergey57 at gmail.com>:
> >>
> >> Hi,
> >> as end-of-life of r151014 approaches, we are planning upgrade for our
> >> NFS servers.
> >> I'm thinking about 2 servers providing ISCSI targets, and 2 another
> >> OmniOS servers using these ISCSI block devices in mirrored ZPOOL
> >> setup. IP address for NFS service can be a floating IP between those 2
> >> servers.
> >> I have the following questions:
> >> 1. Are there any advantages to have separate ISCSI target servers and
> >> NFS servers or I should better combine one ISCSI target and NFS
> >> server on each of 2 hosts?
> >> 2. I do not want snapshots, checksums, and other ZFS features for
> >> block devices at the level where they are exported as ISCSI targets, -
> >> I would prefer these features at the level where these block devices
> >> are combined into mirror Zpools. Maybe it's better to have these ISCSI
> >> target servers running some not so advanced OS and have 2 Linux boxes?
> >> 3. But if I have SSD for intent log and for cache, - maybe they can
> >> improve performance for ZVOLs used as block devices for ISCSI targets?
> >>
> >> Does anybody have experience setting up such redundant NFS servers?
> >> --
> >> Regards,
> >> Sergey Ivanov
> >> _______________________________________________
> >> OmniOS-discuss mailing list
> >> OmniOS-discuss at lists.omniti.com
> >> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170929/06135955/attachment-0001.html>

From sergey57 at gmail.com  Fri Sep 29 20:30:31 2017
From: sergey57 at gmail.com (sergey ivanov)
Date: Fri, 29 Sep 2017 16:30:31 -0400
Subject: [OmniOS-discuss] OmniOS based redundant NFS
In-Reply-To: <968909667.804.1506574164745.JavaMail.stephan.budach@budy.stephanbudach.de>
References: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>
	<634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de>
	<CAAv=BDG0msF00TbmBfZ+E4Vg-t74JkrFmPsEQociK7QM9bSkqQ@mail.gmail.com>
	<968909667.804.1506574164745.JavaMail.stephan.budach@budy.stephanbudach.de>
Message-ID: <CAAv=BDFkUqPpTryscEDaekrUgo8q19xmVA=fL5tQH=JjGWG8GQ@mail.gmail.com>

Thanks, Stephan.
I did a simple test with creating lu over physical disks for use as
ISSI targets, and it worked well. I am going to directly connect 2
servers and export their disks as separate I SCSI targets. Or maybe
different LUNs in a target. And then on the active server start
initiator to get these targets, and combine them into a pool of 2-way
mirrors so that it stays degraded but working if one of the servers
dies.
So, manual failover for this configuration will be the following. If
the server to be disabled is still active, stop NFS, export zpool on
it, stop iscsiadm, release shared IP. On the other server: import
zpool and start NFS, activate shared IP.
I read once there are some tricks which make clients do not recognize
NFS server is changed underneath all mounts, but I never tried it.
-- 
  Regards,
  Sergey Ivanov.

Regards,
Sergey Ivanov


On Thu, Sep 28, 2017 at 12:49 AM, Stephan Budach <stephan.budach at jvm.de> wrote:
> Hi Sergey,
>
> ----- Urspr?ngliche Mail -----
>> Von: "sergey ivanov" <sergey57 at gmail.com>
>> An: "Stephan Budach" <stephan.budach at jvm.de>
>> CC: "omnios-discuss" <omnios-discuss at lists.omniti.com>
>> Gesendet: Mittwoch, 27. September 2017 23:15:49
>> Betreff: Re: [OmniOS-discuss] OmniOS based redundant NFS
>>
>> Thanks, Stephan!
>>
>> Please, explain "The reason to use two x two separate servers is,
>> that
>> the mirrored zpool's vdevs look the same on each NFS head".
>>
>> I understand that, if I want to have the same zpool based on iscsi
>> devices, I should not mix local disks with iscsi target disks.
>>
>> But I think I can have 2 computers, each exporting a set of local
>> disks as iscsi targets. And to have iscsi initiators on the same
>> computers importing these targets to build zpools.
>>
>> Also, looking at sbdadm, I think I can 'create lu
>> /dev/rdsk/c0t0d3s2'.
>>
>> Ok, I think I would better try it and report how it goes.
>
> Actually, things can become quite complex, I'd like to reduce the "mental" involvement to the absolute minimum, mainly because we often faced a situation where something would suddenly break, which had been running for a long time without problems. This is when peeple start? well maybe not panicking, but having to recap what the current setup was like and what they had to do to tackle this.
>
> So, uniformity is a great deal of help on such systems - at least for us. Technically, there is no issue with mixing local and remote iSCST targets on the same node, which serves as an iSCSI target and a NFS head.
>
> Also, if one of the nodes really goes down, you will be loosing your failover NFS head as well, maybe not a big deal and depending on your requirements okay. I do have such a setup as well, although only for an archive ZPOOL, where I can tolerate this reduced redundancy for the benefit of a more lightweight setup.
>
> Cheers,
> Stephan

From jdg117 at elvis.arl.psu.edu  Fri Sep 29 20:37:16 2017
From: jdg117 at elvis.arl.psu.edu (John D Groenveld)
Date: Fri, 29 Sep 2017 16:37:16 -0400
Subject: [OmniOS-discuss] Apache Serf
Message-ID: <201709292037.v8TKbGMx019519@elvis.arl.psu.edu>

Anyone gotten serf to build on OmniOS?

$ env PATH=/opt/subversion/apache24/bin:/usr/bin:/usr/sbin:/usr/ccs/bin:/usr/gnu/bin:/usr/sfw/bin /usr/bin/python2.7 /tmp/scons.py APR=/opt/subversion/apache24 APU=/opt/subversion/apache24 OPENSSL=/usr PREFIX=/opt/subversion/serf CC="gcc" CFLAGS="-fPIC -D__EXTENSIONS__" LINKFLAGS="-G -shared -m64 -R/opt/subversion/apache24/lib" check
== Testing test/testcases/chunked-empty.response ==
ERROR: test case test/testcases/chunked-empty.response failed
scons: *** [check] Error 1
scons: building terminated because of errors.

$ pstack core
core 'core' of 11900:   test/serf_response
 fffffd7fff3ee939 main () + 219
 0000000000000001 ???????? ()
pstack: warning: librtld_db failed to initialize; symbols from shared libraries will not be available

John
groenveld at acm.org

From bfriesen at simple.dallas.tx.us  Fri Sep 29 22:55:40 2017
From: bfriesen at simple.dallas.tx.us (Bob Friesenhahn)
Date: Fri, 29 Sep 2017 17:55:40 -0500 (CDT)
Subject: [OmniOS-discuss] OmniOS based redundant NFS
In-Reply-To: <CAAv=BDGeWjSpc5NdPbcj=PQsdVwLYX_820xa03_rNWAoM_kmaw@mail.gmail.com>
References: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>
	<CAE9csxNg9rPk6_7heX2Dje=3y_LKA3mZYwDceXoZACn8YPLXvg@mail.gmail.com>
	<CAAv=BDGeWjSpc5NdPbcj=PQsdVwLYX_820xa03_rNWAoM_kmaw@mail.gmail.com>
Message-ID: <alpine.GSO.2.20.1709291745410.17039@scrappy.simplesystems.org>

On Fri, 29 Sep 2017, sergey ivanov wrote:

> Thanks, Artem, it is a very good list of suitable hardware!
> I have a question about ZIL.
> I read that SSD for it should be about 8G. I did not see more than one
> gigabyte allocated for ZIL on our servers. Can you comment it?

It only needs to be large enough to contain the maximum amount of 
synchronous write data to be committed in one transaction group. 
This means that it is dependent on the I/O performance of the primary 
storage devices in the pool, the amount of main memory, and the 
maximum time allowed without writing a transaction group, all of which 
are factors when deciding when a TXG should be written.  The amount of 
data to be stored also depends on how large the synchronous writes are 
since large writes (e.g. 128k) by-pass the dedicated ZIL and go 
directly to main store.

If there was one TXG per second, then the 8GB SSD would provide for 
almost 8GB of synchronous writes per second (assuming that writes are 
small), which is quite a lot.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

From stephan.budach at jvm.de  Sat Sep 30 06:45:58 2017
From: stephan.budach at jvm.de (Stephan Budach)
Date: Sat, 30 Sep 2017 08:45:58 +0200 (CEST)
Subject: [OmniOS-discuss] OmniOS based redundant NFS
In-Reply-To: <CAAv=BDFkUqPpTryscEDaekrUgo8q19xmVA=fL5tQH=JjGWG8GQ@mail.gmail.com>
References: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>
	<634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de>
	<CAAv=BDG0msF00TbmBfZ+E4Vg-t74JkrFmPsEQociK7QM9bSkqQ@mail.gmail.com>
	<968909667.804.1506574164745.JavaMail.stephan.budach@budy.stephanbudach.de>
	<CAAv=BDFkUqPpTryscEDaekrUgo8q19xmVA=fL5tQH=JjGWG8GQ@mail.gmail.com>
Message-ID: <150975111.1847.1506753957992.JavaMail.stephan.budach@budy.stephanbudach.de>

Hi Sergey,

----- Urspr?ngliche Mail -----
> Von: "sergey ivanov" <sergey57 at gmail.com>
> An: "Stephan Budach" <stephan.budach at jvm.de>
> CC: "omnios-discuss" <omnios-discuss at lists.omniti.com>
> Gesendet: Freitag, 29. September 2017 22:30:31
> Betreff: Re: [OmniOS-discuss] OmniOS based redundant NFS
> 
> Thanks, Stephan.
> I did a simple test with creating lu over physical disks for use as
> ISSI targets, and it worked well. I am going to directly connect 2
> servers and export their disks as separate I SCSI targets. Or maybe
> different LUNs in a target. And then on the active server start
> initiator to get these targets, and combine them into a pool of 2-way
> mirrors so that it stays degraded but working if one of the servers
> dies.

Well, different targets mean, that you will be able to service single disks on one node, without having to degrade the whole zpool, but only the affected vdevs. On the other hand, there is more complexity since, you will have of course quite a big number of iSCSI targets to login to. This may be ok, if the number doesn't get too hight, but going with hundreds of disks, I chose to use fewer targets with more LUNs.

One thing to keep in mind is, that stmfad allows you to create the guuid to your liking. That is that you can freely choose the last 20 bytes to be anything you want. I used that to ascii-code the node name and slot into the guid, such as that it displays on my NFS heads, when running format. This helps a lot in mapping the LUNs to drives.

> So, manual failover for this configuration will be the following. If
> the server to be disabled is still active, stop NFS, export zpool on
> it, stop iscsiadm, release shared IP. On the other server: import
> zpool and start NFS, activate shared IP.

I am using the sharenfs properties of ZFS, but you will likely have to run zpool export -f <zpool> if you want to fail over the service, since the zpool is still busy. Also, you'd better set zpool failmode to panic instead of wait, such as that an issue triggers a reboot, rather than keeping you NFS head waiting.

> I read once there are some tricks which make clients do not recognize
> NFS server is changed underneath all mounts, but I never tried it.

The only issue I came across was, when I deliberatley failed over the NFS service forth and back within the a too short period, which causes the NFSd on the former primary node to re-use the tcp packets numbers, insisting on reusing it's old NFS connections to the clients. I solved that by resetting the NFSd each time a service starts on any NFS head. The currently connected NFS clients are not affected by that and this solved this particular issue for me.

Cheers,
Stephan

> --
>   Regards,
>   Sergey Ivanov.
> 
> Regards,
> Sergey Ivanov
> 
> 
> On Thu, Sep 28, 2017 at 12:49 AM, Stephan Budach
> <stephan.budach at jvm.de> wrote:
> > Hi Sergey,
> >
> > ----- Urspr?ngliche Mail -----
> >> Von: "sergey ivanov" <sergey57 at gmail.com>
> >> An: "Stephan Budach" <stephan.budach at jvm.de>
> >> CC: "omnios-discuss" <omnios-discuss at lists.omniti.com>
> >> Gesendet: Mittwoch, 27. September 2017 23:15:49
> >> Betreff: Re: [OmniOS-discuss] OmniOS based redundant NFS
> >>
> >> Thanks, Stephan!
> >>
> >> Please, explain "The reason to use two x two separate servers is,
> >> that
> >> the mirrored zpool's vdevs look the same on each NFS head".
> >>
> >> I understand that, if I want to have the same zpool based on iscsi
> >> devices, I should not mix local disks with iscsi target disks.
> >>
> >> But I think I can have 2 computers, each exporting a set of local
> >> disks as iscsi targets. And to have iscsi initiators on the same
> >> computers importing these targets to build zpools.
> >>
> >> Also, looking at sbdadm, I think I can 'create lu
> >> /dev/rdsk/c0t0d3s2'.
> >>
> >> Ok, I think I would better try it and report how it goes.
> >
> > Actually, things can become quite complex, I'd like to reduce the
> > "mental" involvement to the absolute minimum, mainly because we
> > often faced a situation where something would suddenly break,
> > which had been running for a long time without problems. This is
> > when peeple start? well maybe not panicking, but having to recap
> > what the current setup was like and what they had to do to tackle
> > this.
> >
> > So, uniformity is a great deal of help on such systems - at least
> > for us. Technically, there is no issue with mixing local and
> > remote iSCST targets on the same node, which serves as an iSCSI
> > target and a NFS head.
> >
> > Also, if one of the nodes really goes down, you will be loosing
> > your failover NFS head as well, maybe not a big deal and depending
> > on your requirements okay. I do have such a setup as well,
> > although only for an archive ZPOOL, where I can tolerate this
> > reduced redundancy for the benefit of a more lightweight setup.
> >
> > Cheers,
> > Stephan
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5546 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170930/517e5f07/attachment.bin>

From alka at hfg-gmuend.de  Sat Sep 30 08:17:11 2017
From: alka at hfg-gmuend.de (=?UTF-8?Q?G=c3=bcnther_Alka?=)
Date: Sat, 30 Sep 2017 10:17:11 +0200
Subject: [OmniOS-discuss] OmniOS based redundant NFS
In-Reply-To: <150975111.1847.1506753957992.JavaMail.stephan.budach@budy.stephanbudach.de>
References: <CAAv=BDHQZ+wwPUPQu5BYfJLSJxKwuhHRt8N6==0BSa1O9L3JMQ@mail.gmail.com>
	<634237972.756.1506544449540.JavaMail.stephan.budach@budy.stephanbudach.de>
	<CAAv=BDG0msF00TbmBfZ+E4Vg-t74JkrFmPsEQociK7QM9bSkqQ@mail.gmail.com>
	<968909667.804.1506574164745.JavaMail.stephan.budach@budy.stephanbudach.de>
	<CAAv=BDFkUqPpTryscEDaekrUgo8q19xmVA=fL5tQH=JjGWG8GQ@mail.gmail.com>
	<150975111.1847.1506753957992.JavaMail.stephan.budach@budy.stephanbudach.de>
Message-ID: <e41d9ab0-f1b8-c592-aa1a-055a853ea4da@hfg-gmuend.de>

I have added a similar solution with full service and storage failover 
of two identical storage servers last year into my napp-it web ui as an 
experimental solution. Main difference is that I do nor use single disks 
as targets but the whole ZFS pools on two servers to mirror them over 
the network and share the resulting network mirror. This will allow a 
head to function with full redundancy if the other server fails. As you 
only need one LUN per server this reduces complexity.? I also switch 
target ip and NFS/SMB service ip for an instant service failover, see my 
solution under http://www.napp-it.org/doc/downloads/z-raid.pdf

Gea
@napp-it.org



Am 30.09.2017 um 08:45 schrieb Stephan Budach:
> Hi Sergey,
>
> ----- Urspr?ngliche Mail -----
>> Von: "sergey ivanov" <sergey57 at gmail.com>
>> An: "Stephan Budach" <stephan.budach at jvm.de>
>> CC: "omnios-discuss" <omnios-discuss at lists.omniti.com>
>> Gesendet: Freitag, 29. September 2017 22:30:31
>> Betreff: Re: [OmniOS-discuss] OmniOS based redundant NFS
>>
>> Thanks, Stephan.
>> I did a simple test with creating lu over physical disks for use as
>> ISSI targets, and it worked well. I am going to directly connect 2
>> servers and export their disks as separate I SCSI targets. Or maybe
>> different LUNs in a target. And then on the active server start
>> initiator to get these targets, and combine them into a pool of 2-way
>> mirrors so that it stays degraded but working if one of the servers
>> dies.
> Well, different targets mean, that you will be able to service single disks on one node, without having to degrade the whole zpool, but only the affected vdevs. On the other hand, there is more complexity since, you will have of course quite a big number of iSCSI targets to login to. This may be ok, if the number doesn't get too hight, but going with hundreds of disks, I chose to use fewer targets with more LUNs.
>
> One thing to keep in mind is, that stmfad allows you to create the guuid to your liking. That is that you can freely choose the last 20 bytes to be anything you want. I used that to ascii-code the node name and slot into the guid, such as that it displays on my NFS heads, when running format. This helps a lot in mapping the LUNs to drives.
>
>> So, manual failover for this configuration will be the following. If
>> the server to be disabled is still active, stop NFS, export zpool on
>> it, stop iscsiadm, release shared IP. On the other server: import
>> zpool and start NFS, activate shared IP.
> I am using the sharenfs properties of ZFS, but you will likely have to run zpool export -f <zpool> if you want to fail over the service, since the zpool is still busy. Also, you'd better set zpool failmode to panic instead of wait, such as that an issue triggers a reboot, rather than keeping you NFS head waiting.
>
>> I read once there are some tricks which make clients do not recognize
>> NFS server is changed underneath all mounts, but I never tried it.
> The only issue I came across was, when I deliberatley failed over the NFS service forth and back within the a too short period, which causes the NFSd on the former primary node to re-use the tcp packets numbers, insisting on reusing it's old NFS connections to the clients. I solved that by resetting the NFSd each time a service starts on any NFS head. The currently connected NFS clients are not affected by that and this solved this particular issue for me.
>
> Cheers,
> Stephan
>
>> --
>>    Regards,
>>    Sergey Ivanov.
>>
>> Regards,
>> Sergey Ivanov
>>
>>
>> On Thu, Sep 28, 2017 at 12:49 AM, Stephan Budach
>> <stephan.budach at jvm.de> wrote:
>>> Hi Sergey,
>>>
>>> ----- Urspr?ngliche Mail -----
>>>> Von: "sergey ivanov" <sergey57 at gmail.com>
>>>> An: "Stephan Budach" <stephan.budach at jvm.de>
>>>> CC: "omnios-discuss" <omnios-discuss at lists.omniti.com>
>>>> Gesendet: Mittwoch, 27. September 2017 23:15:49
>>>> Betreff: Re: [OmniOS-discuss] OmniOS based redundant NFS
>>>>
>>>> Thanks, Stephan!
>>>>
>>>> Please, explain "The reason to use two x two separate servers is,
>>>> that
>>>> the mirrored zpool's vdevs look the same on each NFS head".
>>>>
>>>> I understand that, if I want to have the same zpool based on iscsi
>>>> devices, I should not mix local disks with iscsi target disks.
>>>>
>>>> But I think I can have 2 computers, each exporting a set of local
>>>> disks as iscsi targets. And to have iscsi initiators on the same
>>>> computers importing these targets to build zpools.
>>>>
>>>> Also, looking at sbdadm, I think I can 'create lu
>>>> /dev/rdsk/c0t0d3s2'.
>>>>
>>>> Ok, I think I would better try it and report how it goes.
>>> Actually, things can become quite complex, I'd like to reduce the
>>> "mental" involvement to the absolute minimum, mainly because we
>>> often faced a situation where something would suddenly break,
>>> which had been running for a long time without problems. This is
>>> when peeple start? well maybe not panicking, but having to recap
>>> what the current setup was like and what they had to do to tackle
>>> this.
>>>
>>> So, uniformity is a great deal of help on such systems - at least
>>> for us. Technically, there is no issue with mixing local and
>>> remote iSCST targets on the same node, which serves as an iSCSI
>>> target and a NFS head.
>>>
>>> Also, if one of the nodes really goes down, you will be loosing
>>> your failover NFS head as well, maybe not a big deal and depending
>>> on your requirements okay. I do have such a setup as well,
>>> although only for an archive ZPOOL, where I can tolerate this
>>> reduced redundancy for the benefit of a more lightweight setup.
>>>
>>> Cheers,
>>> Stephan
>>
>>
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20170930/a93cfa7a/attachment-0001.html>

