From admdwrf at gmail.com  Sat Nov  1 00:14:02 2014
From: admdwrf at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBBbGl4?=)
Date: Sat, 1 Nov 2014 01:14:02 +0100
Subject: [OmniOS-discuss] OmniOS doc
Message-ID: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>

Hi,

Since few years, i'm a big OmniOS fan.

I tried to evangelisch it around me.

I had a website to share about it, but my ex-business partner delete it and
backups too :-(

Few days ago, i was at linux hack day at Rennes in France. I talked about
OmniOS.

Some admin sys were very interesting about it. I restarted a blog at
blog.fredalix.com

But, in my country, some peoples know SmartOS, and when they go to OmniOS
website, they don't like it.

I am sad to know sysadmin no want try OmniOS, because the actual wiki is
not for newbies.

Yesterday, i was at Linux and Docker french event, and i talked about
OmniOS and zones.

But returns were  "OmniOS wiki is good for Solaris admins !"

OmniTI's team make a super job, but the OmniOS is not attractive for
newbies.

it's sad :-(

what can we'll do about that ?

I am sure with Omnios's community we can publish an OmniOS Survival guide.

Yep ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141101/14a9c00f/attachment.html>

From ajacoutot at bsdfrog.org  Sat Nov  1 00:44:33 2014
From: ajacoutot at bsdfrog.org (Antoine Jacoutot)
Date: Sat, 1 Nov 2014 01:44:33 +0100
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
Message-ID: <20141101004433.GA12410@janus.obspm.bsdfrog.org>

>    Few days ago, i was at linux hack day at Rennes in France. I talked
>    about OmniOS.
> 
>    Some admin sys were very interesting about it. I restarted a blog at
>    [1]blog.fredalix.com
> 
>    But, in my country, some peoples know SmartOS, and when they go to
>    OmniOS website, they don't like it.

Yeah but some others *do* like it.

>    I am sad to know sysadmin no want try OmniOS, because the actual wiki
>    is not for newbies.
> 
>    Yesterday, i was at Linux and Docker french event, and i talked about
>    OmniOS and zones.
> 
>    But returns were ? "OmniOS wiki is good for Solaris admins !"
> 
>    OmniTI's team make a super job, but the OmniOS is not attractive for
>    newbies.
> 
>    it's sad :-(
> 
>    what can we'll do about that ?
> 
>    I am sure with Omnios's community we can publish an OmniOS Survival
>    guide.
> 
>    Yep ?

Sure, go ahead... but do we really want to attract "newbies"?
I don't know, just asking.

-- 
Antoine

From rt at steait.net  Sat Nov  1 01:07:33 2014
From: rt at steait.net (Rune Tipsmark)
Date: Sat, 1 Nov 2014 01:07:33 +0000
Subject: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%
In-Reply-To: <aaa3c0225c574520b578c83a459fdc2d@EX1301.steait.net>
References: <4153d29f2b3a42349a5b701edfb06e92@EX1301.steait.net>
	<CAO8hXRCk-wJPfTa3krtGU57c2yNpL9G3c0m6LXi+B1iZNtd5+w@mail.gmail.com>
	<8FD2525B-70E3-4227-B9D5-8D4399D31335@richardelling.com>
	<aaa3c0225c574520b578c83a459fdc2d@EX1301.steait.net>
Message-ID: <c1bbdd298b814dbea6744f9cc87a8017@EX1301.steait.net>

So actually started storage vmotions on 3 host, 6 concurrent and am getting about 1GB/sec
Guess I need more hosts to really push this, the disk are not more than 20-25% busy, so in theory I could push a bit more.

I think this is resolved for now.... cpu sitting at 30-40% usage while moving 1GB/sec

Iostat -xn 1
pool04       396G  39.5T      9  15.9K   325K  1.01G
pool04       396G  39.5T      7  17.0K   270K  1.03G
pool04       396G  39.5T     12  17.4K   558K  1.10G
pool04       396G  39.5T     10  16.9K   442K  1.03G
pool04       397G  39.5T      6  16.9K   332K  1021M
pool04       397G  39.5T      1  16.3K  74.9K  1.01G
pool04       397G  39.5T      8  17.0K   433K  1.05G
pool04       397G  39.5T     20  17.1K   716K  1023M
pool04       397G  39.5T     11  18.3K   425K  1.14G
pool04       398G  39.5T      0  18.3K  65.9K  1.11G
pool04       398G  39.5T     16  17.9K   551K  1.06G
pool04       398G  39.5T      0  16.8K   105K  1.03G
pool04       398G  39.5T      1  18.2K   124K  1.11G
pool04       398G  39.5T      0  17.1K  45.9K  1.05G
pool04       399G  39.5T      6  17.3K   454K  1.08G
pool04       399G  39.5T      0  17.9K      0  1.06G
pool04       399G  39.5T      2  16.9K   116K  1.04G
pool04       399G  39.5T      2  18.8K   130K  1.09G
pool04       399G  39.5T      0  17.6K      0  1.03G
pool04       400G  39.5T      3  17.5K   155K  1.04G
pool04       400G  39.5T      0  17.6K  31.5K  1.03G

-----Original Message-----
From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Rune Tipsmark
Sent: Friday, October 31, 2014 12:38 PM
To: Richard Elling; Eric Sproul
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%

Ok, makes sense.
What other kind of  indicators can I look at?

I get decent results from DD but still feels a bit slow...

Compression lz4 should not slow it down right? Cpu is not doing much when copying data over, maybe 15% busy or so... 

Sync=always, block size 1M
204800000000 bytes (205 GB) copied, 296.379 s, 691 MB/s
real    4m56.382s
user    0m0.461s
sys     3m12.662s

Sync=disabled, block size 1M
204800000000 bytes (205 GB) copied, 117.774 s, 1.7 GB/s
real    1m57.777s
user    0m0.237s
sys     1m57.466s

... while doing this I was looking at my FIO cards, I think the reason is that the SLC's need more power to deliver higher performance, they are supposed to deliver 1.5GB/sec but only delivers around 350MB/sec each....

Now looking for aux power cables and will retest...

Br,
Rune

-----Original Message-----
From: Richard Elling [mailto:richard.elling at richardelling.com] 
Sent: Friday, October 31, 2014 9:03 AM
To: Eric Sproul
Cc: Rune Tipsmark; omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%


On Oct 31, 2014, at 7:14 AM, Eric Sproul <eric.sproul at circonus.com> wrote:

> On Fri, Oct 31, 2014 at 2:33 AM, Rune Tipsmark <rt at steait.net> wrote:
> 
>> Why is this pool showing near 100% busy when the underlying disks are 
>> doing nothing at all....
> 
> Simply put, it's just how the accounting works in iostat.  It treats 
> the pool like any other device, so if there is even one outstanding 
> request to the pool, it counts towards the busy%.  Keith W. from 
> Joyent explained this recently on the illumos-zfs list:
> http://www.listbox.com/member/archive/182191/2014/10/sort/time_rev/pag
> e/3/entry/18:93/20141017161955:F3E11AB2-563A-11E4-8EDC-D0C677981E2F/
> 
> The TL;DR is: if your pool has more than one disk in it, the pool-wide 
> busy% is useless.

FWIW, we use %busy as an indicator that we can ignore a device/subsystem when looking for performance problems. We don't use it as an indicator of problems. In other words, if the device isn't > 10% busy, forgetabouddit. If it is more busy, look in more detail at the meaningful performance indicators.
 -- richard

_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

From admdwrf at gmail.com  Sat Nov  1 01:21:17 2014
From: admdwrf at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBBbGl4?=)
Date: Sat, 1 Nov 2014 02:21:17 +0100
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <20141101004433.GA12410@janus.obspm.bsdfrog.org>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
	<20141101004433.GA12410@janus.obspm.bsdfrog.org>
Message-ID: <CAN4tE3_X+dBtmFnRToc2akpL1g+cOgjw004FKmfEv-zFUs3wvw@mail.gmail.com>

> Yeah but some others *do* like it.
yes, like me, but i use Solaris since many years...

>Sure, go ahead... but do we really want to attract "newbies"?
>I don't know, just asking.
Yes of course !
Do you want an IT world with just GNU/TuxTux ????
OmniOS vs Linux/Docker is amazing !
With OmniOS, we can have a real cpu and memory capping !
OpenSolaris is dead, and i think we must show OmniOS is an amazing
alternative.
We can't stay in our corner. We should show OmniOS Zones and KVM power :)
And look how the network stack is manage in a Docker ... beurk !!!
Few months ago, one of my client switch from Gnu/Linux to OmniOS and he
tell me thank you every day.
We should share and display what we can do with OmniOS.




2014-11-01 1:44 GMT+01:00 Antoine Jacoutot <ajacoutot at bsdfrog.org>:

> >    Few days ago, i was at linux hack day at Rennes in France. I talked
> >    about OmniOS.
> >
> >    Some admin sys were very interesting about it. I restarted a blog at
> >    [1]blog.fredalix.com
> >
> >    But, in my country, some peoples know SmartOS, and when they go to
> >    OmniOS website, they don't like it.
>
> Yeah but some others *do* like it.
>
> >    I am sad to know sysadmin no want try OmniOS, because the actual wiki
> >    is not for newbies.
> >
> >    Yesterday, i was at Linux and Docker french event, and i talked about
> >    OmniOS and zones.
> >
> >    But returns were ? "OmniOS wiki is good for Solaris admins !"
> >
> >    OmniTI's team make a super job, but the OmniOS is not attractive for
> >    newbies.
> >
> >    it's sad :-(
> >
> >    what can we'll do about that ?
> >
> >    I am sure with Omnios's community we can publish an OmniOS Survival
> >    guide.
> >
> >    Yep ?
>
> Sure, go ahead... but do we really want to attract "newbies"?
> I don't know, just asking.
>
> --
> Antoine
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141101/4c1b7ef8/attachment.html>

From jesus at omniti.com  Sat Nov  1 06:17:44 2014
From: jesus at omniti.com (Theo Schlossnagle)
Date: Sat, 1 Nov 2014 10:17:44 +0400
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <20141101004433.GA12410@janus.obspm.bsdfrog.org>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
	<20141101004433.GA12410@janus.obspm.bsdfrog.org>
Message-ID: <CACLsApuod9WnBT5FS6t7tN6eRtx1jbwUbkRJS9jZKxJv8jfcog@mail.gmail.com>

On Sat, Nov 1, 2014 at 3:44 AM, Antoine Jacoutot <ajacoutot at bsdfrog.org>
wrote:
>
> >    I am sure with Omnios's community we can publish an OmniOS Survival
> >    guide.
> >
> >    Yep ?
>
> Sure, go ahead... but do we really want to attract "newbies"?
> I don't know, just asking.
>

I think an OmniOS Survival guide would be great.

If you have to put something like "newbies" in quotes and talk about them
as different than us, you are viewing this community in an unhealthy way.
OmniOS is designed to take Illumos and all of its features and package it
in a way that makes a minimal and useful general operating system targeted
at servers and appliances.  Anyone with whom that vision aligns is most
welcome in the community and we should encourage efforts to make them feel
as an accepted part of the whole.


-- 

Theo Schlossnagle
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141101/d2747489/attachment.html>

From danmcd at omniti.com  Sat Nov  1 21:56:52 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Sat, 1 Nov 2014 17:56:52 -0400
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <CAN4tE3_X+dBtmFnRToc2akpL1g+cOgjw004FKmfEv-zFUs3wvw@mail.gmail.com>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
	<20141101004433.GA12410@janus.obspm.bsdfrog.org>
	<CAN4tE3_X+dBtmFnRToc2akpL1g+cOgjw004FKmfEv-zFUs3wvw@mail.gmail.com>
Message-ID: <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>


> On Oct 31, 2014, at 9:21 PM, Fr?d?ric Alix <admdwrf at gmail.com> wrote:
> 
> >Sure, go ahead... but do we really want to attract "newbies"?
> >I don't know, just asking.
> Yes of course !

Theo already jumped in on this point, but both he and Fr?d?ric are absolutely right.  Of course!  We don't grow without newbies.  We should treat them with kindness, respect,  a willingness to help, and a willingness to lead by example (esp. w.r.t. not doing things stupidly).

> We should share and display what we can do with OmniOS.

OmniOS provides the traditionally-managed general-purpose OS distribution of Illumos. As someone here told me upon my joining, "We're the promise OpenSolaris was supposed to be."

BTW, don't take this thread as an opportunity to spew on any other OS or distro, especially other illumos ones.  I won't have it.

I do believe, and someone here mentioned this to me offline, that we might need to promote OmniOS a bit more.  I'm not sure how best to do that, modulo people like the folks here telling their managers, friends, and customers about how OmniOS, quite simply, Helps You Solve Problems.

If we need more docs because it will attract people to OmniOS and help it grow, I'm all for it.  Specific examples of what people think will help are appreciated.

Dan


From mir at miras.org  Sat Nov  1 22:44:14 2014
From: mir at miras.org (Michael Rasmussen)
Date: Sat, 1 Nov 2014 23:44:14 +0100
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
	<20141101004433.GA12410@janus.obspm.bsdfrog.org>
	<CAN4tE3_X+dBtmFnRToc2akpL1g+cOgjw004FKmfEv-zFUs3wvw@mail.gmail.com>
	<519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>
Message-ID: <20141101234414.43f42589@sleipner.datanom.net>

On Sat, 1 Nov 2014 17:56:52 -0400
Dan McDonald <danmcd at omniti.com> wrote:

> 
> If we need more docs because it will attract people to OmniOS and help it grow, I'm all for it.  Specific examples of what people think will help are appreciated.
> 
If we want to be more friendly to newcomers some kind of basic stuff is
needed. I can remember my own first journey into Omnios which demanded
endless searches and readings from Oracle (Solaris), OpenIndiana,
Nexenta, SmartOS, Google, and Youtube etc. And of course reading Omniti
and this list. A lot of stuff from Oracle is good but cannot always be
used directly since Omnios/Illumos is somewhere in between Solaris 10
and Solaris 11. What would have helped me a great deal would have been
able to get all this stuff from one source, and a source with examples
directly able to be used.

First I think a list of hardware known to work and not known to work
would be great. The list should be divided into a section for each
stable and LTS version. Something like:
1) CPU
2) Chipset
3) GPU
4) Controller. Onboard, HBA and RAID
5) Nics
6) Motherboards
An something more which I am sure I have forgotten;-)

I my opinion the best thing would be a web based searchable database.
It should be open to everyone registered and everyone should be
encourage to add there own findings.

Second, I really mis some good tutorials about various subjects. This
could be implemented in some kind of wiki. It should be open to
everyone registered and everyone should be encourage to add input they
think would be of interest to others. There is nothing better than to
stand on the shoulders of other wise guys.

Third, a section of typical problems users can find them selfs in with
guides to either solve the problem or if to solution is found in the
section yet and guide to extract all needed information to be send to
the list when asking for help. Providing a list of required information
to the ones able to help you as part of the first request on the list
is more likely to receive a swift answer with, hopefully, the correct
solution. 

As I am very pleased with Omnios I would love to see its popularity rise
so I will participate to the best of my knowledge and spare time.

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
It would be illogical to assume that all conditions remain stable.
		-- Spock, "The Enterprise Incident", stardate 5027.3
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141101/93e006bf/attachment.bin>

From lkateley at kateley.com  Sat Nov  1 22:46:03 2014
From: lkateley at kateley.com (Linda Kateley)
Date: Sat, 01 Nov 2014 17:46:03 -0500
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <CAN4tE3_X+dBtmFnRToc2akpL1g+cOgjw004FKmfEv-zFUs3wvw@mail.gmail.com>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>	<20141101004433.GA12410@janus.obspm.bsdfrog.org>
	<CAN4tE3_X+dBtmFnRToc2akpL1g+cOgjw004FKmfEv-zFUs3wvw@mail.gmail.com>
Message-ID: <545562AB.30006@kateley.com>

I actually really enjoy doing little how-to videos. I put up a set a 
couple weeks ago on zfs

http://kateleyco.com/?page_id=783

I have taught solaris for most of my career.. so I plan on creating some 
more.. intro to dtrace.. services.. fma

please feel free to link to them or republish or consume. I did them on 
omni.

linda


On 10/31/14, 8:21 PM, Fr?d?ric Alix wrote:
> > Yeah but some others *do* like it.
> yes, like me, but i use Solaris since many years...
>
> >Sure, go ahead... but do we really want to attract "newbies"?
> >I don't know, just asking.
> Yes of course !
> Do you want an IT world with just GNU/TuxTux ????
> OmniOS vs Linux/Docker is amazing !
> With OmniOS, we can have a real cpu and memory capping !
> OpenSolaris is dead, and i think we must show OmniOS is an amazing 
> alternative.
> We can't stay in our corner. We should show OmniOS Zones and KVM power :)
> And look how the network stack is manage in a Docker ... beurk !!!
> Few months ago, one of my client switch from Gnu/Linux to OmniOS and 
> he tell me thank you every day.
> We should share and display what we can do with OmniOS.
>
>
>
>
> 2014-11-01 1:44 GMT+01:00 Antoine Jacoutot <ajacoutot at bsdfrog.org 
> <mailto:ajacoutot at bsdfrog.org>>:
>
>     >    Few days ago, i was at linux hack day at Rennes in France. I talked
>     >    about OmniOS.
>     >
>     >    Some admin sys were very interesting about it. I restarted a
>     blog at
>     >    [1]blog.fredalix.com <http://blog.fredalix.com>
>     >
>     >    But, in my country, some peoples know SmartOS, and when they
>     go to
>     >    OmniOS website, they don't like it.
>
>     Yeah but some others *do* like it.
>
>     >    I am sad to know sysadmin no want try OmniOS, because the
>     actual wiki
>     >    is not for newbies.
>     >
>     >    Yesterday, i was at Linux and Docker french event, and i
>     talked about
>     >    OmniOS and zones.
>     >
>     >    But returns were ? "OmniOS wiki is good for Solaris admins !"
>     >
>     >    OmniTI's team make a super job, but the OmniOS is not
>     attractive for
>     >    newbies.
>     >
>     >    it's sad :-(
>     >
>     >    what can we'll do about that ?
>     >
>     >    I am sure with Omnios's community we can publish an OmniOS
>     Survival
>     >    guide.
>     >
>     >    Yep ?
>
>     Sure, go ahead... but do we really want to attract "newbies"?
>     I don't know, just asking.
>
>     --
>     Antoine
>
>
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141101/9393a0e3/attachment.html>

From cks at cs.toronto.edu  Sat Nov  1 22:48:19 2014
From: cks at cs.toronto.edu (Chris Siebenmann)
Date: Sat, 01 Nov 2014 18:48:19 -0400
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: Your message of Sat, 01 Nov 2014 17:56:52 -0400.
	<519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>
Message-ID: <20141101224819.2365A5A0974@testapps.cs.toronto.edu>

> I do believe, and someone here mentioned this to me offline, that we
> might need to promote OmniOS a bit more. I'm not sure how best to
> do that, modulo people like the folks here telling their managers,
> friends, and customers about how OmniOS, quite simply, Helps You Solve
> Problems.

 One problem with doing such promotion today is that for quite a lot
of people, OmniOS does not solve your problems but is instead just
a building block for doing so. This is distinctly different from,
say, FreeBSD or a random Linux distributions, which *will* solve many
people's problems out of the box. Here I'm talking about problems like
'I need a mail server' or 'I need an IMAP server' or 'I need a web
server' or 'I need a DB server'.

(I am on the mailing list for Exim, and I can assure you that there
are many people who show up there wanting an out of the box 'solve
my problems' experience with Exim on their system, even for unusual
problems. These people have not even seriously read the documentation
for Exim.)

 I understand and accept that OmniOS is never going to explicitly
package such things. What I think would help here is a curated list of
additional package sources with explicit 'where to get <X>' for popular
<X>s. This would give people who want their problems solved an easily
found and explicitly documented two-step process for doing so, in
contrast to the current approach where you get to do a bunch of searching
(and experimentation and guessing about which is the right package source
to use and so on).

	- cks

From rt at steait.net  Sun Nov  2 00:45:44 2014
From: rt at steait.net (Rune Tipsmark)
Date: Sun, 2 Nov 2014 00:45:44 +0000
Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
Message-ID: <79c2871d23ae447eaebcc7e8c2a4626d@EX1301.steait.net>

Hi  all,

Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.

If not, is there a way I can aggregate say 8 or 16  IPoIB partitions and push throughput to a more reasonable speed...

Br,
Rune
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141102/1cebd98d/attachment.html>

From turbo124 at gmail.com  Sun Nov  2 01:01:04 2014
From: turbo124 at gmail.com (David Bomba)
Date: Sun, 2 Nov 2014 12:01:04 +1100
Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
In-Reply-To: <79c2871d23ae447eaebcc7e8c2a4626d@EX1301.steait.net>
References: <79c2871d23ae447eaebcc7e8c2a4626d@EX1301.steait.net>
Message-ID: <CA588692-BCAB-40B6-BCDC-5E3DA53AB14C@gmail.com>

I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential. 


> On 2 Nov 2014, at 11:45 am, Rune Tipsmark <rt at steait.net <mailto:rt at steait.net>> wrote:
> 
> Hi  all,
>  
> Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.
>  
> If not, is there a way I can aggregate say 8 or 16  IPoIB partitions and push throughput to a more reasonable speed?
>  
> Br,
> Rune
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com <mailto:OmniOS-discuss at lists.omniti.com>
> http://lists.omniti.com/mailman/listinfo/omnios-discuss <http://lists.omniti.com/mailman/listinfo/omnios-discuss>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141102/5e0ceb56/attachment-0001.html>

From rt at steait.net  Sun Nov  2 06:14:21 2014
From: rt at steait.net (Rune Tipsmark)
Date: Sun, 2 Nov 2014 06:14:21 +0000
Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
In-Reply-To: <CA588692-BCAB-40B6-BCDC-5E3DA53AB14C@gmail.com>
References: <79c2871d23ae447eaebcc7e8c2a4626d@EX1301.steait.net>
	<CA588692-BCAB-40B6-BCDC-5E3DA53AB14C@gmail.com>
Message-ID: <8bb61255a8fb49f28ff5ef9a251316df@EX1301.steait.net>

Sounds sensible, how do I do that?
I tried creating a view for a thin lu with my other zfs box, but how do I detect it?

I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html

Br,
Rune

From: David Bomba [mailto:turbo124 at gmail.com]
Sent: Saturday, November 01, 2014 6:01 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential.


On 2 Nov 2014, at 11:45 am, Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> wrote:

Hi  all,

Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.

If not, is there a way I can aggregate say 8 or 16  IPoIB partitions and push throughput to a more reasonable speed?

Br,
Rune
_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141102/f1cdc85b/attachment.html>

From turbo124 at gmail.com  Sun Nov  2 06:16:13 2014
From: turbo124 at gmail.com (David Bomba)
Date: Sun, 2 Nov 2014 17:16:13 +1100
Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
In-Reply-To: <8bb61255a8fb49f28ff5ef9a251316df@EX1301.steait.net>
References: <79c2871d23ae447eaebcc7e8c2a4626d@EX1301.steait.net>
	<CA588692-BCAB-40B6-BCDC-5E3DA53AB14C@gmail.com>
	<8bb61255a8fb49f28ff5ef9a251316df@EX1301.steait.net>
Message-ID: <70929479-3ED6-4E12-98F6-7EF9E8ECFD1D@gmail.com>

The oracle docs have a step by step for setting up target and initiator. 

http://docs.oracle.com/cd/E23824_01/html/821-1459/fnnop.html <http://docs.oracle.com/cd/E23824_01/html/821-1459/fnnop.html>


> On 2 Nov 2014, at 5:14 pm, Rune Tipsmark <rt at steait.net> wrote:
> 
> Sounds sensible, how do I do that?
> I tried creating a view for a thin lu with my other zfs box, but how do I detect it?
>  
> I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html <http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html>
>  
> Br,
> Rune
>  
> From: David Bomba [mailto:turbo124 at gmail.com] 
> Sent: Saturday, November 01, 2014 6:01 PM
> To: Rune Tipsmark
> Cc: omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
>  
> I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential. 
>  
>  
> On 2 Nov 2014, at 11:45 am, Rune Tipsmark <rt at steait.net <mailto:rt at steait.net>> wrote:
>  
> Hi  all,
>  
> Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.
>  
> If not, is there a way I can aggregate say 8 or 16  IPoIB partitions and push throughput to a more reasonable speed?
>  
> Br,
> Rune
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com <mailto:OmniOS-discuss at lists.omniti.com>
> http://lists.omniti.com/mailman/listinfo/omnios-discuss <http://lists.omniti.com/mailman/listinfo/omnios-discuss>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141102/f8109ed7/attachment.html>

From johan.kragsterman at capvert.se  Sun Nov  2 08:26:03 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Sun, 2 Nov 2014 09:26:03 +0100
Subject: [OmniOS-discuss] Ang: Re:  OmniOS doc
In-Reply-To: <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>
References: <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>,
	<CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>	<20141101004433.GA12410@janus.obspm.bsdfrog.org>
	<CAN4tE3_X+dBtmFnRToc2akpL1g+cOgjw004FKmfEv-zFUs3wvw@mail.gmail.com>
Message-ID: <OF715C2DE1.88602674-ONC1257D84.002E54B5-C1257D84.002E54B8@inse.com>

The answer to this discussion thread would in my opinion be:


How do we design, manage and maintain the omnios public documentation?

I believe that is a project to take seriously...


Johan


-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev: -----
Till: omnios-discuss <omnios-discuss at lists.omniti.com>
Fr?n: Dan McDonald 
S?nt av: "OmniOS-discuss" 
Datum: 2014-11-01 22:59
?rende: Re: [OmniOS-discuss] OmniOS doc

> On Oct 31, 2014, at 9:21 PM, Fr?d?ric Alix <admdwrf at gmail.com> wrote:
> 
> >Sure, go ahead... but do we really want to attract "newbies"?
> >I don't know, just asking.
> Yes of course !

Theo already jumped in on this point, but both he and Fr?d?ric are absolutely right. ?Of course! ?We don't grow without newbies. ?We should treat them with kindness, respect, ?a willingness to help, and a willingness to lead by example (esp. w.r.t. not doing things stupidly).

> We should share and display what we can do with OmniOS.

OmniOS provides the traditionally-managed general-purpose OS distribution of Illumos. As someone here told me upon my joining, "We're the promise OpenSolaris was supposed to be."

BTW, don't take this thread as an opportunity to spew on any other OS or distro, especially other illumos ones. ?I won't have it.

I do believe, and someone here mentioned this to me offline, that we might need to promote OmniOS a bit more. ?I'm not sure how best to do that, modulo people like the folks here telling their managers, friends, and customers about how OmniOS, quite simply, Helps You Solve Problems.

If we need more docs because it will attract people to OmniOS and help it grow, I'm all for it. ?Specific examples of what people think will help are appreciated.

Dan

_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss



From johan.kragsterman at capvert.se  Sun Nov  2 10:02:13 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Sun, 2 Nov 2014 11:02:13 +0100
Subject: [OmniOS-discuss] Ang:  Ang: Re:  OmniOS doc
In-Reply-To: <OF715C2DE1.88602674-ONC1257D84.002E54B5-C1257D84.002E54B8@inse.com>
References: <OF715C2DE1.88602674-ONC1257D84.002E54B5-C1257D84.002E54B8@inse.com>,
	<519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>,
	<CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
	<20141101004433.GA12410@janus.obspm.bsdfrog.org>	<CAN4tE3_X+dBtmFnRToc2akpL1g+cOgjw004FKmfEv-zFUs3wvw@mail.gmail.com>
Message-ID: <OF3AFAD5DC.39057D05-ONC1257D84.0036B86F-C1257D84.003722B2@inse.com>


-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev: -----
Till: Dan McDonald <danmcd at omniti.com>
Fr?n: Johan Kragsterman 
S?nt av: "OmniOS-discuss" 
Datum: 2014-11-02 09:27
Kopia: omnios-discuss <omnios-discuss at lists.omniti.com>
?rende: [OmniOS-discuss] Ang: Re: OmniOS doc

The answer to this discussion thread would in my opinion be:


How do we design, manage and maintain the omnios public documentation?

I believe that is a project to take seriously...


Johan





To reply and add to my own opinon, I'd like to continue with:

I'd believe it is OmniTI that has the possibility to start a project around a subject like this. The discussion about it should of coarse be here, on omnios-discuss, but I believe OmniTI should tell us which infrastructure solutions that would be possible to provide for it.

?If OmniTI can provide appropriate infrastructure, I believe the community can provide content.

The mantainance of the solution?will of coarse be of great importance, and will of coarse also need man hours, and how this should be done, is a matter of this community to discuss...

Johan








-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev: -----
Till: omnios-discuss <omnios-discuss at lists.omniti.com>
Fr?n: Dan McDonald 
S?nt av: "OmniOS-discuss" 
Datum: 2014-11-01 22:59
?rende: Re: [OmniOS-discuss] OmniOS doc

> On Oct 31, 2014, at 9:21 PM, Fr?d?ric Alix <admdwrf at gmail.com> wrote:
> 
> >Sure, go ahead... but do we really want to attract "newbies"?
> >I don't know, just asking.
> Yes of course !

Theo already jumped in on this point, but both he and Fr?d?ric are absolutely right. ?Of course! ?We don't grow without newbies. ?We should treat them with kindness, respect, ?a willingness to help, and a willingness to lead by example (esp. w.r.t. not doing things stupidly).

> We should share and display what we can do with OmniOS.

OmniOS provides the traditionally-managed general-purpose OS distribution of Illumos. As someone here told me upon my joining, "We're the promise OpenSolaris was supposed to be."

BTW, don't take this thread as an opportunity to spew on any other OS or distro, especially other illumos ones. ?I won't have it.

I do believe, and someone here mentioned this to me offline, that we might need to promote OmniOS a bit more. ?I'm not sure how best to do that, modulo people like the folks here telling their managers, friends, and customers about how OmniOS, quite simply, Helps You Solve Problems.

If we need more docs because it will attract people to OmniOS and help it grow, I'm all for it. ?Specific examples of what people think will help are appreciated.

Dan

_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss


_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss




From tobi at oetiker.ch  Sun Nov  2 12:15:20 2014
From: tobi at oetiker.ch (Tobias Oetiker)
Date: Sun, 2 Nov 2014 13:15:20 +0100 (CET)
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
	<20141101004433.GA12410@janus.obspm.bsdfrog.org>
	<CAN4tE3_X+dBtmFnRToc2akpL1g+cOgjw004FKmfEv-zFUs3wvw@mail.gmail.com>
	<519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>
Message-ID: <alpine.DEB.2.10.1411021313330.5357@engelberg.oetiker.ch>

Yesterday Dan McDonald wrote:

> If we need more docs because it will attract people to OmniOS and
> help it grow, I'm all for it.  Specific examples of what people
> think will help are appreciated.

what I notice, when searching for illumos/omnios information I
often end up on the oracle website ...

an easy way to remedy this in part would be to publish the illumos
manual pages in a friendly way on the web, so that google searches
have somewhere to go.

cheers
tobi

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch tobi at oetiker.ch +41 62 775 9902


From tim at multitalents.net  Sun Nov  2 16:21:32 2014
From: tim at multitalents.net (Tim Rice)
Date: Sun, 2 Nov 2014 08:21:32 -0800 (PST)
Subject: [OmniOS-discuss] Compiling GNUCOBOL : pre-requisites present?
In-Reply-To: <20141027164015.GA17098@hp-aio>
References: <85f940caa925131c4e59fe2b99c0ae89@kathe.in>
	<6F3F930C-13BC-4459-B212-7A32EAA757F4@zinascii.com>
	<20141027164015.GA17098@hp-aio>
Message-ID: <alpine.UW2.2.11.1411020819290.22694@server01.int.multitalents.net>

On Mon, 27 Oct 2014, Mayuresh Kathe wrote:

[snip]
> > http://omnios.omniti.com/wiki.php/Packaging
> 
> I did what you'd suggested, got BDB installed, but somehow the configure
> script is unable to find "gmp.h" at "/usr/include/gmp/".
> Is there anything I need to do to make the system know that gmp.h exists
> at "/usr/include/gmp/"?

Try
CPPFLAGS="/usr/include/gmp" ./configure .......

> 
> Thanks,
> 
> ~Mayuresh
> 

-- 
Tim Rice				Multitalents
tim at multitalents.net



From johan.kragsterman at capvert.se  Sun Nov  2 17:56:12 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Sun, 2 Nov 2014 18:56:12 +0100
Subject: [OmniOS-discuss] Ang: Re: zfs send via SRP or other RDMA enabled
	protocol
Message-ID: <OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>


-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev: -----
Till: David Bomba <turbo124 at gmail.com>
Fr?n: Rune Tipsmark 
S?nt av: "OmniOS-discuss" 
Datum: 2014-11-02 07:55
Kopia: "omnios-discuss at lists.omniti.com" <omnios-discuss at lists.omniti.com>
?rende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

Sounds sensible, how do I do that?

I tried creating a view for a thin lu with my other zfs box, but how do I detect it?





First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application.

Did you create SRP target service using COMSTAR?

# svcadm enable -r ibsrp/target

What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v"

Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link?

If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA.


Rgrds Johan




?

I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html

?

Br,

Rune

?

From: David Bomba [mailto:turbo124 at gmail.com] 
Sent: Saturday, November 01, 2014 6:01 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

?

I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential.?

?

?

On 2 Nov 2014, at 11:45 am, Rune Tipsmark <rt at steait.net> wrote:

?

Hi ?all,

?

Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.

?

If not, is there a way I can aggregate say 8 or 16? IPoIB partitions and push throughput to a more reasonable speed&#8230;

?

Br,

Rune













_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

?

_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss



From rt at steait.net  Sun Nov  2 18:10:41 2014
From: rt at steait.net (Rune Tipsmark)
Date: Sun, 2 Nov 2014 18:10:41 +0000
Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
In-Reply-To: <OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>
References: <OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>
Message-ID: <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>

Hi Johan

Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine..

What I can't  figure out is how to share a LUN with the other zfs box... see pasted info below...

ZFS00: (the box I want to receive my zfs snapshot)
The below are all ESX servers, cannot see the other ZFS box

root at zfs00:/pool03# stmfadm list-target -v
Target: eui.0002C90300095E7C
    Operational Status: Online
    Provider Name     : srpt
    Alias             : -
    Protocol          : SRP
    Sessions          : 8
        Initiator: eui.0002C903000F397C
            Alias: 8102c90300095e7e:0002c903000f397c
            Logged in since: Sun Nov  2 02:09:41 2014
        Initiator: eui.0002C903000F397B
            Alias: 8102c90300095e7d:0002c903000f397b
            Logged in since: Sun Nov  2 02:09:40 2014
        Initiator: eui.0002C903000D3D04
            Alias: 8102c90300095e7e:0002c903000d3d04
            Logged in since: Sat Nov  1 21:14:47 2014
        Initiator: eui.0002C90300104F47
            Alias: 8102c90300095e7d:0002c90300104f47
            Logged in since: Sat Nov  1 21:12:54 2014
        Initiator: eui.0002C903000D3D03
            Alias: 8102c90300095e7d:0002c903000d3d03
            Logged in since: Sat Nov  1 21:12:32 2014
        Initiator: eui.0002C90300104F48
            Alias: 8102c90300095e7e:0002c90300104f48
            Logged in since: Sat Nov  1 21:10:45 2014
        Initiator: eui.0002C903000A48FA
            Alias: 8102c90300095e7e:0002c903000a48fa
            Logged in since: Sat Nov  1 21:10:40 2014
        Initiator: eui.0002C903000D3CA0
            Alias: 8102c90300095e7e:0002c903000d3ca0
            Logged in since: Sat Nov  1 21:10:39 2014
Target: iqn.2010-09.org.napp-it:1394106801
    Operational Status: Online
    Provider Name     : iscsit
    Alias             : 03.06.2014
    Protocol          : iSCSI
    Sessions          : 0

root at zfs00:/pool03# stmfadm list-lu -v
LU Name: 600144F007780B7F00005455EDD50002
    Operational Status: Online
    Provider Name     : sbd
    Alias             : /pool03/LU11
    View Entry Count  : 1
    Data File         : /pool03/LU11
    Meta File         : not set
    Size              : 2199023255552
    Block Size        : 512
    Management URL    : not set
    Vendor ID         : SUN
    Product ID        : COMSTAR
    Serial Num        : not set
    Write Protect     : Disabled
    Writeback Cache   : Disabled
    Access State      : Active

root at zfs00:/pool03# stmfadm list-hg -v
Host Group: ESX
Host Group: Windows
Host Group: ESX-iSER
Host Group: OmniOS
        Member: eui.0002C903000923E6 <<--- The other ZFS box
        Member: iqn.2010-09.org.napp-it:1402013225
        Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d

root at zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002
View Entry: 0
    Host group   : OmniOS
    Target group : All
    LUN          : 0

ZFS10: (the sending box where I want to see the LUN from ZFS00)
No disk show up from ZFS00...

root at zfs10:/root# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c4t5000C50055FC9533d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50055fc9533
       1. c4t5000C50055FE6A63d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50055fe6a63
       2. c4t5000C500625B7EA7d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b7ea7
       3. c4t5000C500625B86E3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b86e3
       4. c4t5000C500625B886Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b886f
       5. c4t5000C500625B8137d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b8137
       6. c4t5000C500625B8427d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b8427
       7. c4t5000C500625BB773d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bb773
       8. c4t5000C500625BC2C3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bc2c3
       9. c4t5000C500625BD3EBd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bd3eb
      10. c4t5000C50057085A6Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057085a6b
      11. c4t5000C50057086B67d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057086b67
      12. c4t5000C50062878C0Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062878c0b
      13. c4t5000C50062878C43d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062878c43
      14. c4t5000C500570858EFd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500570858ef
      15. c4t5000C500570870D3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500570870d3
      16. c4t5000C5005708351Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c5005708351b
      17. c4t5000C5005708296Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c5005708296f
      18. c4t5000C50057089753d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057089753
      19. c4t5000C50057086307d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057086307
      20. c4t5000C50062879687d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879687
      21. c4t5000C50062879707d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879707
      22. c4t5000C50062879723d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879723
      23. c4t5000C50062879787d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879787
      24. c7t0d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 0,0
      25. c7t1d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 1,0
      26. c7t4d0 <ATA-InnoLite SATADOM-19-14.91GB>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 4,0
      27. c7t5d0 <ATA-InnoLite SATADOM-19-14.91GB>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 5,0
      28. c10d0 <Unknown-Unknown-0001-149.00GB>
          /pci at 0,0/pci8086,3c08 at 3/pci103c,178b at 0
      29. c11d0 <Unknown-Unknown-0001-149.00GB>
          /pci at 0,0/pci8086,3c0a at 3,2/pci103c,178b at 0
      30. c12d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
      31. c13d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
      32. c14d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
      33. c15d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
Specify disk (enter its number):



-----Original Message-----
From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se] 
Sent: Sunday, November 02, 2014 9:56 AM
To: Rune Tipsmark
Cc: David Bomba; omnios-discuss at lists.omniti.com
Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev: -----
Till: David Bomba <turbo124 at gmail.com>
Fr?n: Rune Tipsmark 
S?nt av: "OmniOS-discuss" 
Datum: 2014-11-02 07:55
Kopia: "omnios-discuss at lists.omniti.com" <omnios-discuss at lists.omniti.com>
?rende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

Sounds sensible, how do I do that?

I tried creating a view for a thin lu with my other zfs box, but how do I detect it?





First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application.

Did you create SRP target service using COMSTAR?

# svcadm enable -r ibsrp/target

What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v"

Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link?

If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA.


Rgrds Johan




?

I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html

?

Br,

Rune

?

From: David Bomba [mailto:turbo124 at gmail.com] 
Sent: Saturday, November 01, 2014 6:01 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

?

I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential.?

?

?

On 2 Nov 2014, at 11:45 am, Rune Tipsmark <rt at steait.net> wrote:

?

Hi ?all,

?

Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.

?

If not, is there a way I can aggregate say 8 or 16? IPoIB partitions and push throughput to a more reasonable speed&#8230;

?

Br,

Rune













_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

?

_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss



From johan.kragsterman at capvert.se  Sun Nov  2 18:33:19 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Sun, 2 Nov 2014 19:33:19 +0100
Subject: [OmniOS-discuss] Ang: RE: Re: zfs send via SRP or other RDMA
 enabled protocol
In-Reply-To: <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>
References: <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>,
	<OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>
Message-ID: <OF69908773.2D8DA960-ONC1257D84.0065651F-C1257D84.0065ED8B@inse.com>

An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141102/afa27111/attachment-0001.html>

From rt at steait.net  Sun Nov  2 19:05:10 2014
From: rt at steait.net (Rune Tipsmark)
Date: Sun, 2 Nov 2014 19:05:10 +0000
Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
In-Reply-To: <OF69908773.2D8DA960-ONC1257D84.0065651F-C1257D84.0065ED8B@inse.com>
References: <898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>,
	<OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>
	<OF69908773.2D8DA960-ONC1257D84.0065651F-C1257D84.0065ED8B@inse.com>
Message-ID: <64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>

I know, but how do I initiate a session from ZFS10?

Br,
Rune

From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 10:33 AM
To: Rune Tipsmark
Cc: David Bomba; omnios-discuss at lists.omniti.com
Subject: Ang: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol



-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-02 19:11
Kopia: David Bomba <turbo124 at gmail.com<mailto:turbo124 at gmail.com>>, "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
Hi Johan

Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine..

What I can't  figure out is how to share a LUN with the other zfs box... see pasted info below...

ZFS00: (the box I want to receive my zfs snapshot)
The below are all ESX servers, cannot see the other ZFS box

root at zfs00:/pool03# stmfadm list-target -v
Target: eui.0002C90300095E7C
    Operational Status: Online
    Provider Name     : srpt
    Alias             : -
    Protocol          : SRP
    Sessions          : 8
        Initiator: eui.0002C903000F397C
            Alias: 8102c90300095e7e:0002c903000f397c
            Logged in since: Sun Nov  2 02:09:41 2014
        Initiator: eui.0002C903000F397B
            Alias: 8102c90300095e7d:0002c903000f397b
            Logged in since: Sun Nov  2 02:09:40 2014
        Initiator: eui.0002C903000D3D04
            Alias: 8102c90300095e7e:0002c903000d3d04
            Logged in since: Sat Nov  1 21:14:47 2014
        Initiator: eui.0002C90300104F47
            Alias: 8102c90300095e7d:0002c90300104f47
            Logged in since: Sat Nov  1 21:12:54 2014
        Initiator: eui.0002C903000D3D03
            Alias: 8102c90300095e7d:0002c903000d3d03
            Logged in since: Sat Nov  1 21:12:32 2014
        Initiator: eui.0002C90300104F48
            Alias: 8102c90300095e7e:0002c90300104f48
            Logged in since: Sat Nov  1 21:10:45 2014
        Initiator: eui.0002C903000A48FA
            Alias: 8102c90300095e7e:0002c903000a48fa
            Logged in since: Sat Nov  1 21:10:40 2014
        Initiator: eui.0002C903000D3CA0
            Alias: 8102c90300095e7e:0002c903000d3ca0
            Logged in since: Sat Nov  1 21:10:39 2014
Target: iqn.2010-09.org.napp-it:1394106801
    Operational Status: Online
    Provider Name     : iscsit
    Alias             : 03.06.2014
    Protocol          : iSCSI
    Sessions          : 0

root at zfs00:/pool03# stmfadm list-lu -v
LU Name: 600144F007780B7F00005455EDD50002
    Operational Status: Online
    Provider Name     : sbd
    Alias             : /pool03/LU11
    View Entry Count  : 1
    Data File         : /pool03/LU11
    Meta File         : not set
    Size              : 2199023255552
    Block Size        : 512
    Management URL    : not set
    Vendor ID         : SUN
    Product ID        : COMSTAR
    Serial Num        : not set
    Write Protect     : Disabled
    Writeback Cache   : Disabled
    Access State      : Active

root at zfs00:/pool03# stmfadm list-hg -v
Host Group: ESX
Host Group: Windows
Host Group: ESX-iSER
Host Group: OmniOS
        Member: eui.0002C903000923E6 <<--- The other ZFS box
        Member: iqn.2010-09.org.napp-it:1402013225
        Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d

root at zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002
View Entry: 0
    Host group   : OmniOS
    Target group : All
    LUN          : 0

ZFS10: (the sending box where I want to see the LUN from ZFS00)
No disk show up from ZFS00...



 Hi!


You got eui.0002C903000923E6 in host group OmniOS, but you don't have a session from that eui to the target.

Rgrds Johan












root at zfs10:/root# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c4t5000C50055FC9533d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50055fc9533
       1. c4t5000C50055FE6A63d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50055fe6a63
       2. c4t5000C500625B7EA7d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b7ea7
       3. c4t5000C500625B86E3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b86e3
       4. c4t5000C500625B886Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b886f
       5. c4t5000C500625B8137d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b8137
       6. c4t5000C500625B8427d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b8427
       7. c4t5000C500625BB773d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bb773
       8. c4t5000C500625BC2C3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bc2c3
       9. c4t5000C500625BD3EBd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bd3eb
      10. c4t5000C50057085A6Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057085a6b
      11. c4t5000C50057086B67d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057086b67
      12. c4t5000C50062878C0Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062878c0b
      13. c4t5000C50062878C43d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062878c43
      14. c4t5000C500570858EFd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500570858ef
      15. c4t5000C500570870D3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500570870d3
      16. c4t5000C5005708351Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c5005708351b
      17. c4t5000C5005708296Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c5005708296f
      18. c4t5000C50057089753d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057089753
      19. c4t5000C50057086307d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057086307
      20. c4t5000C50062879687d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879687
      21. c4t5000C50062879707d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879707
      22. c4t5000C50062879723d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879723
      23. c4t5000C50062879787d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879787
      24. c7t0d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 0,0
      25. c7t1d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 1,0
      26. c7t4d0 <ATA-InnoLite SATADOM-19-14.91GB>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 4,0
      27. c7t5d0 <ATA-InnoLite SATADOM-19-14.91GB>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 5,0
      28. c10d0 <Unknown-Unknown-0001-149.00GB>
          /pci at 0,0/pci8086,3c08 at 3/pci103c,178b at 0
      29. c11d0 <Unknown-Unknown-0001-149.00GB>
          /pci at 0,0/pci8086,3c0a at 3,2/pci103c,178b at 0
      30. c12d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
      31. c13d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
      32. c14d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
      33. c15d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
Specify disk (enter its number):



-----Original Message-----
From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 9:56 AM
To: Rune Tipsmark
Cc: David Bomba; omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com<mailto:omnios-discuss-bounces at lists.omniti.com>> skrev: -----
Till: David Bomba <turbo124 at gmail.com<mailto:turbo124 at gmail.com>>
Fr?n: Rune Tipsmark
S?nt av: "OmniOS-discuss"
Datum: 2014-11-02 07:55
Kopia: "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

Sounds sensible, how do I do that?

I tried creating a view for a thin lu with my other zfs box, but how do I detect it?





First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application.

Did you create SRP target service using COMSTAR?

# svcadm enable -r ibsrp/target

What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v"

Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link?

If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA.


Rgrds Johan






I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html



Br,

Rune



From: David Bomba [mailto:turbo124 at gmail.com]
Sent: Saturday, November 01, 2014 6:01 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol



I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential.





On 2 Nov 2014, at 11:45 am, Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> wrote:



Hi  all,



Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.



If not, is there a way I can aggregate say 8 or 16  IPoIB partitions and push throughput to a more reasonable speed...



Br,

Rune













_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss



_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141102/85d83305/attachment-0001.html>

From geoffn at gnaa.net  Sun Nov  2 23:34:47 2014
From: geoffn at gnaa.net (Geoff Nordli)
Date: Sun, 02 Nov 2014 15:34:47 -0800
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>	<20141101004433.GA12410@janus.obspm.bsdfrog.org>	<CAN4tE3_X+dBtmFnRToc2akpL1g+cOgjw004FKmfEv-zFUs3wvw@mail.gmail.com>
	<519BB74C-9802-4287-BA0B-D3297E38935A@omniti.com>
Message-ID: <5456BF97.30603@gnaa.net>

On 14-11-01 02:56 PM, Dan McDonald wrote:
>> On Oct 31, 2014, at 9:21 PM, Fr?d?ric Alix <admdwrf at gmail.com> wrote:
>>
>>> Sure, go ahead... but do we really want to attract "newbies"?
>>> I don't know, just asking.
>> Yes of course !
> Theo already jumped in on this point, but both he and Fr?d?ric are absolutely right.  Of course!  We don't grow without newbies.  We should treat them with kindness, respect,  a willingness to help, and a willingness to lead by example (esp. w.r.t. not doing things stupidly).
>
>> We should share and display what we can do with OmniOS.
> OmniOS provides the traditionally-managed general-purpose OS distribution of Illumos. As someone here told me upon my joining, "We're the promise OpenSolaris was supposed to be."
>
> BTW, don't take this thread as an opportunity to spew on any other OS or distro, especially other illumos ones.  I won't have it.
>
> I do believe, and someone here mentioned this to me offline, that we might need to promote OmniOS a bit more.  I'm not sure how best to do that, modulo people like the folks here telling their managers, friends, and customers about how OmniOS, quite simply, Helps You Solve Problems.
>
> If we need more docs because it will attract people to OmniOS and help it grow, I'm all for it.  Specific examples of what people think will help are appreciated.
>
>

How Tos are a good documentation method.  One of the issues is knowing 
if the method you are following from some blog/document is actually the 
right method for OmniOS or some other solaris derivative.

Geoff



From richard.elling at richardelling.com  Mon Nov  3 02:07:00 2014
From: richard.elling at richardelling.com (Richard Elling)
Date: Sun, 2 Nov 2014 18:07:00 -0800
Subject: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%
In-Reply-To: <c1bbdd298b814dbea6744f9cc87a8017@EX1301.steait.net>
References: <4153d29f2b3a42349a5b701edfb06e92@EX1301.steait.net>
	<CAO8hXRCk-wJPfTa3krtGU57c2yNpL9G3c0m6LXi+B1iZNtd5+w@mail.gmail.com>
	<8FD2525B-70E3-4227-B9D5-8D4399D31335@richardelling.com>
	<aaa3c0225c574520b578c83a459fdc2d@EX1301.steait.net>
	<c1bbdd298b814dbea6744f9cc87a8017@EX1301.steait.net>
Message-ID: <72A25F36-8BA9-4071-B009-AA90BDCA18B2@richardelling.com>


On Oct 31, 2014, at 6:07 PM, Rune Tipsmark <rt at steait.net> wrote:

> So actually started storage vmotions on 3 host, 6 concurrent and am getting about 1GB/sec
> Guess I need more hosts to really push this, the disk are not more than 20-25% busy, so in theory I could push a bit more.
> 
> I think this is resolved for now.... cpu sitting at 30-40% usage while moving 1GB/sec

Yes, that seems about right.
 -- richard

> 
> Iostat -xn 1
> pool04       396G  39.5T      9  15.9K   325K  1.01G
> pool04       396G  39.5T      7  17.0K   270K  1.03G
> pool04       396G  39.5T     12  17.4K   558K  1.10G
> pool04       396G  39.5T     10  16.9K   442K  1.03G
> pool04       397G  39.5T      6  16.9K   332K  1021M
> pool04       397G  39.5T      1  16.3K  74.9K  1.01G
> pool04       397G  39.5T      8  17.0K   433K  1.05G
> pool04       397G  39.5T     20  17.1K   716K  1023M
> pool04       397G  39.5T     11  18.3K   425K  1.14G
> pool04       398G  39.5T      0  18.3K  65.9K  1.11G
> pool04       398G  39.5T     16  17.9K   551K  1.06G
> pool04       398G  39.5T      0  16.8K   105K  1.03G
> pool04       398G  39.5T      1  18.2K   124K  1.11G
> pool04       398G  39.5T      0  17.1K  45.9K  1.05G
> pool04       399G  39.5T      6  17.3K   454K  1.08G
> pool04       399G  39.5T      0  17.9K      0  1.06G
> pool04       399G  39.5T      2  16.9K   116K  1.04G
> pool04       399G  39.5T      2  18.8K   130K  1.09G
> pool04       399G  39.5T      0  17.6K      0  1.03G
> pool04       400G  39.5T      3  17.5K   155K  1.04G
> pool04       400G  39.5T      0  17.6K  31.5K  1.03G
> 
> -----Original Message-----
> From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Rune Tipsmark
> Sent: Friday, October 31, 2014 12:38 PM
> To: Richard Elling; Eric Sproul
> Cc: omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%
> 
> Ok, makes sense.
> What other kind of  indicators can I look at?
> 
> I get decent results from DD but still feels a bit slow...
> 
> Compression lz4 should not slow it down right? Cpu is not doing much when copying data over, maybe 15% busy or so... 
> 
> Sync=always, block size 1M
> 204800000000 bytes (205 GB) copied, 296.379 s, 691 MB/s
> real    4m56.382s
> user    0m0.461s
> sys     3m12.662s
> 
> Sync=disabled, block size 1M
> 204800000000 bytes (205 GB) copied, 117.774 s, 1.7 GB/s
> real    1m57.777s
> user    0m0.237s
> sys     1m57.466s
> 
> ... while doing this I was looking at my FIO cards, I think the reason is that the SLC's need more power to deliver higher performance, they are supposed to deliver 1.5GB/sec but only delivers around 350MB/sec each....
> 
> Now looking for aux power cables and will retest...
> 
> Br,
> Rune
> 
> -----Original Message-----
> From: Richard Elling [mailto:richard.elling at richardelling.com] 
> Sent: Friday, October 31, 2014 9:03 AM
> To: Eric Sproul
> Cc: Rune Tipsmark; omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%
> 
> 
> On Oct 31, 2014, at 7:14 AM, Eric Sproul <eric.sproul at circonus.com> wrote:
> 
>> On Fri, Oct 31, 2014 at 2:33 AM, Rune Tipsmark <rt at steait.net> wrote:
>> 
>>> Why is this pool showing near 100% busy when the underlying disks are 
>>> doing nothing at all....
>> 
>> Simply put, it's just how the accounting works in iostat.  It treats 
>> the pool like any other device, so if there is even one outstanding 
>> request to the pool, it counts towards the busy%.  Keith W. from 
>> Joyent explained this recently on the illumos-zfs list:
>> http://www.listbox.com/member/archive/182191/2014/10/sort/time_rev/pag
>> e/3/entry/18:93/20141017161955:F3E11AB2-563A-11E4-8EDC-D0C677981E2F/
>> 
>> The TL;DR is: if your pool has more than one disk in it, the pool-wide 
>> busy% is useless.
> 
> FWIW, we use %busy as an indicator that we can ignore a device/subsystem when looking for performance problems. We don't use it as an indicator of problems. In other words, if the device isn't > 10% busy, forgetabouddit. If it is more busy, look in more detail at the meaningful performance indicators.
> -- richard
> 
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss


From rt at steait.net  Mon Nov  3 02:24:29 2014
From: rt at steait.net (Rune Tipsmark)
Date: Mon, 3 Nov 2014 02:24:29 +0000
Subject: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%
In-Reply-To: <72A25F36-8BA9-4071-B009-AA90BDCA18B2@richardelling.com>
References: <4153d29f2b3a42349a5b701edfb06e92@EX1301.steait.net>
	<CAO8hXRCk-wJPfTa3krtGU57c2yNpL9G3c0m6LXi+B1iZNtd5+w@mail.gmail.com>
	<8FD2525B-70E3-4227-B9D5-8D4399D31335@richardelling.com>
	<aaa3c0225c574520b578c83a459fdc2d@EX1301.steait.net>
	<c1bbdd298b814dbea6744f9cc87a8017@EX1301.steait.net>
	<72A25F36-8BA9-4071-B009-AA90BDCA18B2@richardelling.com>
Message-ID: <17b60b6ed2424559a706c5649f548c1e@EX1301.steait.net>

Looking a bit more at these numbers, am I seeing twice the actual rate due to mirroring?
How does compression affect the numbers?

Say I have 1 vdev mirrored and I see the pool writing 100 MB/sec, its what? 50 MB each disk? But from the client side only 50 MB/sec total? What if I compress it at the same time at say 1.50 ratio, will the pool show 100 MB/sec and the client write 75 MB/sec actual?

Br,
Rune

-----Original Message-----
From: Richard Elling [mailto:richard.elling at richardelling.com] 
Sent: Sunday, November 02, 2014 6:07 PM
To: Rune Tipsmark
Cc: Eric Sproul; omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%


On Oct 31, 2014, at 6:07 PM, Rune Tipsmark <rt at steait.net> wrote:

> So actually started storage vmotions on 3 host, 6 concurrent and am 
> getting about 1GB/sec Guess I need more hosts to really push this, the disk are not more than 20-25% busy, so in theory I could push a bit more.
> 
> I think this is resolved for now.... cpu sitting at 30-40% usage while 
> moving 1GB/sec

Yes, that seems about right.
 -- richard

> 
> Iostat -xn 1
> pool04       396G  39.5T      9  15.9K   325K  1.01G
> pool04       396G  39.5T      7  17.0K   270K  1.03G
> pool04       396G  39.5T     12  17.4K   558K  1.10G
> pool04       396G  39.5T     10  16.9K   442K  1.03G
> pool04       397G  39.5T      6  16.9K   332K  1021M
> pool04       397G  39.5T      1  16.3K  74.9K  1.01G
> pool04       397G  39.5T      8  17.0K   433K  1.05G
> pool04       397G  39.5T     20  17.1K   716K  1023M
> pool04       397G  39.5T     11  18.3K   425K  1.14G
> pool04       398G  39.5T      0  18.3K  65.9K  1.11G
> pool04       398G  39.5T     16  17.9K   551K  1.06G
> pool04       398G  39.5T      0  16.8K   105K  1.03G
> pool04       398G  39.5T      1  18.2K   124K  1.11G
> pool04       398G  39.5T      0  17.1K  45.9K  1.05G
> pool04       399G  39.5T      6  17.3K   454K  1.08G
> pool04       399G  39.5T      0  17.9K      0  1.06G
> pool04       399G  39.5T      2  16.9K   116K  1.04G
> pool04       399G  39.5T      2  18.8K   130K  1.09G
> pool04       399G  39.5T      0  17.6K      0  1.03G
> pool04       400G  39.5T      3  17.5K   155K  1.04G
> pool04       400G  39.5T      0  17.6K  31.5K  1.03G
> 
> -----Original Message-----
> From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] 
> On Behalf Of Rune Tipsmark
> Sent: Friday, October 31, 2014 12:38 PM
> To: Richard Elling; Eric Sproul
> Cc: omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%
> 
> Ok, makes sense.
> What other kind of  indicators can I look at?
> 
> I get decent results from DD but still feels a bit slow...
> 
> Compression lz4 should not slow it down right? Cpu is not doing much when copying data over, maybe 15% busy or so... 
> 
> Sync=always, block size 1M
> 204800000000 bytes (205 GB) copied, 296.379 s, 691 MB/s
> real    4m56.382s
> user    0m0.461s
> sys     3m12.662s
> 
> Sync=disabled, block size 1M
> 204800000000 bytes (205 GB) copied, 117.774 s, 1.7 GB/s
> real    1m57.777s
> user    0m0.237s
> sys     1m57.466s
> 
> ... while doing this I was looking at my FIO cards, I think the reason is that the SLC's need more power to deliver higher performance, they are supposed to deliver 1.5GB/sec but only delivers around 350MB/sec each....
> 
> Now looking for aux power cables and will retest...
> 
> Br,
> Rune
> 
> -----Original Message-----
> From: Richard Elling [mailto:richard.elling at richardelling.com]
> Sent: Friday, October 31, 2014 9:03 AM
> To: Eric Sproul
> Cc: Rune Tipsmark; omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%
> 
> 
> On Oct 31, 2014, at 7:14 AM, Eric Sproul <eric.sproul at circonus.com> wrote:
> 
>> On Fri, Oct 31, 2014 at 2:33 AM, Rune Tipsmark <rt at steait.net> wrote:
>> 
>>> Why is this pool showing near 100% busy when the underlying disks 
>>> are doing nothing at all....
>> 
>> Simply put, it's just how the accounting works in iostat.  It treats 
>> the pool like any other device, so if there is even one outstanding 
>> request to the pool, it counts towards the busy%.  Keith W. from 
>> Joyent explained this recently on the illumos-zfs list:
>> http://www.listbox.com/member/archive/182191/2014/10/sort/time_rev/pa
>> g 
>> e/3/entry/18:93/20141017161955:F3E11AB2-563A-11E4-8EDC-D0C677981E2F/
>> 
>> The TL;DR is: if your pool has more than one disk in it, the 
>> pool-wide busy% is useless.
> 
> FWIW, we use %busy as an indicator that we can ignore a device/subsystem when looking for performance problems. We don't use it as an indicator of problems. In other words, if the device isn't > 10% busy, forgetabouddit. If it is more busy, look in more detail at the meaningful performance indicators.
> -- richard
> 
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss


From johan.kragsterman at capvert.se  Mon Nov  3 06:49:08 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Mon, 3 Nov 2014 07:49:08 +0100
Subject: [OmniOS-discuss] Ang: RE: RE: RE: Re: zfs send via SRP or other
 RDMA enabled protocol
In-Reply-To: <e5b49b9c7bc64e6a8fd9c88a76fd029f@EX1301.steait.net>
References: <e5b49b9c7bc64e6a8fd9c88a76fd029f@EX1301.steait.net>,
	<64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>,
	<898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, 
	<OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>
	<OF69908773.2D8DA960-ONC1257D84.0065651F-C1257D84.0065ED8B@inse.com>
	<OF88628ACC.EFA25FDC-ONC1257D84.006DAC9E-C1257D84.006DACA1@inse.com>
Message-ID: <OF3AF7BA9E.CCF8DE80-ONC1257D85.0025401A-C1257D85.0025754B@inse.com>

An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/e59cd7e5/attachment-0001.html>

From rt at steait.net  Mon Nov  3 07:09:06 2014
From: rt at steait.net (Rune Tipsmark)
Date: Mon, 3 Nov 2014 07:09:06 +0000
Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
In-Reply-To: <OF3AF7BA9E.CCF8DE80-ONC1257D85.0025401A-C1257D85.0025754B@inse.com>
References: <e5b49b9c7bc64e6a8fd9c88a76fd029f@EX1301.steait.net>,
	<64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>,
	<898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>,
	<OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>
	<OF69908773.2D8DA960-ONC1257D84.0065651F-C1257D84.0065ED8B@inse.com>
	<OF88628ACC.EFA25FDC-ONC1257D84.006DAC9E-C1257D84.006DACA1@inse.com>
	<OF3AF7BA9E.CCF8DE80-ONC1257D85.0025401A-C1257D85.0025754B@inse.com>
Message-ID: <164a9de15eb64f88a4785762899729a8@EX1301.steait.net>

connectX2 and drivers are loaded, both OmniOS servers have LUNs I can access from both ESX and Windows... just the conection between them that I cant figure out.
Br,
Rune

From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 10:49 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com
Subject: Ang: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


Hej!

I can see I forgot the list in previous mail...

Anyway, are you using connectx adapters?

Have you checked if the driver is loaded in the host system?


Best regards from/Med v?nliga h?lsningar fr?n

Johan Kragsterman

Capvert


-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-03 00:07
?rende: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
It doesn't show up in stmfadm list-target.... only my other servers such as ESX and Windows show up...

Subnet Manager is OpenSM running on the ESX hosts, has been working OK ever since I installed then, prior to that I had a couple of Cisco(Topspin) DDR Switches with inbuilt SM.

Any other idea? All things I configured so far on both ZFS boxes,windows and esx has worked as expected... just this darn thing that's not doing what its supposed to...

It should be enough to just add the HCA from ZFS10 onto the host-group on ZFS00 for ZFS10 to see a LUN on ZFS00 granted that there is a view configured on ZFS00 as well... or am I missing a step?

Br
Rune

-----Original Message-----
From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 11:58 AM
To: Rune Tipsmark
Subject: Ang: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-02 20:05
Kopia: David Bomba <turbo124 at gmail.com<mailto:turbo124 at gmail.com>>, "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

I know, but how do I initiate a session from ZFS10?



If a session doesn't show up in "stmfadm list-targte -v", then you got something wrong in the IB fabric, if the view is right. Do you use a switch? Where do you have your IB Storage Manager?








Br,

Rune



From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 10:33 AM
To: Rune Tipsmark
Cc: David Bomba; omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Ang: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol






-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----

Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-02 19:11
Kopia: David Bomba <turbo124 at gmail.com<mailto:turbo124 at gmail.com>>, "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

Hi Johan

Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine..

What I can't  figure out is how to share a LUN with the other zfs box... see pasted info below...

ZFS00: (the box I want to receive my zfs snapshot) The below are all ESX servers, cannot see the other ZFS box

root at zfs00:/pool03# stmfadm list-target -v
Target: eui.0002C90300095E7C
    Operational Status: Online
    Provider Name     : srpt
    Alias             : -
    Protocol          : SRP
    Sessions          : 8
        Initiator: eui.0002C903000F397C
            Alias: 8102c90300095e7e:0002c903000f397c
            Logged in since: Sun Nov  2 02:09:41 2014
        Initiator: eui.0002C903000F397B
            Alias: 8102c90300095e7d:0002c903000f397b
            Logged in since: Sun Nov  2 02:09:40 2014
        Initiator: eui.0002C903000D3D04
            Alias: 8102c90300095e7e:0002c903000d3d04
            Logged in since: Sat Nov  1 21:14:47 2014
        Initiator: eui.0002C90300104F47
            Alias: 8102c90300095e7d:0002c90300104f47
            Logged in since: Sat Nov  1 21:12:54 2014
        Initiator: eui.0002C903000D3D03
            Alias: 8102c90300095e7d:0002c903000d3d03
            Logged in since: Sat Nov  1 21:12:32 2014
        Initiator: eui.0002C90300104F48
            Alias: 8102c90300095e7e:0002c90300104f48
            Logged in since: Sat Nov  1 21:10:45 2014
        Initiator: eui.0002C903000A48FA
            Alias: 8102c90300095e7e:0002c903000a48fa
            Logged in since: Sat Nov  1 21:10:40 2014
        Initiator: eui.0002C903000D3CA0
            Alias: 8102c90300095e7e:0002c903000d3ca0
            Logged in since: Sat Nov  1 21:10:39 2014
Target: iqn.2010-09.org.napp-it:1394106801
    Operational Status: Online
    Provider Name     : iscsit
    Alias             : 03.06.2014
    Protocol          : iSCSI
    Sessions          : 0

root at zfs00:/pool03# stmfadm list-lu -v
LU Name: 600144F007780B7F00005455EDD50002
    Operational Status: Online
    Provider Name     : sbd
    Alias             : /pool03/LU11
    View Entry Count  : 1
    Data File         : /pool03/LU11
    Meta File         : not set
    Size              : 2199023255552
    Block Size        : 512
    Management URL    : not set
    Vendor ID         : SUN
    Product ID        : COMSTAR
    Serial Num        : not set
    Write Protect     : Disabled
    Writeback Cache   : Disabled
    Access State      : Active

root at zfs00:/pool03# stmfadm list-hg -v
Host Group: ESX
Host Group: Windows
Host Group: ESX-iSER
Host Group: OmniOS
        Member: eui.0002C903000923E6 <<--- The other ZFS box
        Member: iqn.2010-09.org.napp-it:1402013225
        Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d

root at zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002 View Entry: 0
    Host group   : OmniOS
    Target group : All
    LUN          : 0

ZFS10: (the sending box where I want to see the LUN from ZFS00) No disk show up from ZFS00...







 Hi!





You got eui.0002C903000923E6 in host group OmniOS, but you don't have a session from that eui to the target.



Rgrds Johan























root at zfs10:/root# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c4t5000C50055FC9533d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50055fc9533
       1. c4t5000C50055FE6A63d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50055fe6a63
       2. c4t5000C500625B7EA7d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b7ea7
       3. c4t5000C500625B86E3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b86e3
       4. c4t5000C500625B886Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b886f
       5. c4t5000C500625B8137d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b8137
       6. c4t5000C500625B8427d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b8427
       7. c4t5000C500625BB773d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bb773
       8. c4t5000C500625BC2C3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bc2c3
       9. c4t5000C500625BD3EBd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bd3eb
      10. c4t5000C50057085A6Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057085a6b
      11. c4t5000C50057086B67d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057086b67
      12. c4t5000C50062878C0Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062878c0b
      13. c4t5000C50062878C43d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062878c43
      14. c4t5000C500570858EFd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500570858ef
      15. c4t5000C500570870D3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500570870d3
      16. c4t5000C5005708351Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c5005708351b
      17. c4t5000C5005708296Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c5005708296f
      18. c4t5000C50057089753d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057089753
      19. c4t5000C50057086307d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057086307
      20. c4t5000C50062879687d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879687
      21. c4t5000C50062879707d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879707
      22. c4t5000C50062879723d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879723
      23. c4t5000C50062879787d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879787
      24. c7t0d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 0,0
      25. c7t1d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 1,0
      26. c7t4d0 <ATA-InnoLite SATADOM-19-14.91GB>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 4,0
      27. c7t5d0 <ATA-InnoLite SATADOM-19-14.91GB>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 5,0
      28. c10d0 <Unknown-Unknown-0001-149.00GB>
          /pci at 0,0/pci8086,3c08 at 3/pci103c,178b at 0
      29. c11d0 <Unknown-Unknown-0001-149.00GB>
          /pci at 0,0/pci8086,3c0a at 3,2/pci103c,178b at 0
      30. c12d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
      31. c13d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
      32. c14d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
      33. c15d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
Specify disk (enter its number):



-----Original Message-----
From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 9:56 AM
To: Rune Tipsmark
Cc: David Bomba; omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com<mailto:omnios-discuss-bounces at lists.omniti.com>> skrev: -----
Till: David Bomba <turbo124 at gmail.com<mailto:turbo124 at gmail.com>>
Fr?n: Rune Tipsmark
S?nt av: "OmniOS-discuss"
Datum: 2014-11-02 07:55
Kopia: "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

Sounds sensible, how do I do that?

I tried creating a view for a thin lu with my other zfs box, but how do I detect it?





First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application.

Did you create SRP target service using COMSTAR?

# svcadm enable -r ibsrp/target

What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v"

Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link?

If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA.


Rgrds Johan






I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html



Br,

Rune



From: David Bomba [mailto:turbo124 at gmail.com]
Sent: Saturday, November 01, 2014 6:01 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol



I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential.





On 2 Nov 2014, at 11:45 am, Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> wrote:



Hi  all,



Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.



If not, is there a way I can aggregate say 8 or 16  IPoIB partitions and push throughput to a more reasonable speed...



Br,

Rune













_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss



_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/0919d1b1/attachment-0001.html>

From johan.kragsterman at capvert.se  Mon Nov  3 07:24:03 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Mon, 3 Nov 2014 08:24:03 +0100
Subject: [OmniOS-discuss] Ang: RE: RE: RE: RE: Re: zfs send via SRP or other
 RDMA enabled protocol
In-Reply-To: <164a9de15eb64f88a4785762899729a8@EX1301.steait.net>
References: <164a9de15eb64f88a4785762899729a8@EX1301.steait.net>,
	<e5b49b9c7bc64e6a8fd9c88a76fd029f@EX1301.steait.net>,
	<64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>, 
	<898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>,
	<OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>
	<OF69908773.2D8DA960-ONC1257D84.0065651F-C1257D84.0065ED8B@inse.com>
	<OF88628ACC.EFA25FDC-ONC1257D84.006DAC9E-C1257D84.006DACA1@inse.com>
	<OF3AF7BA9E.CCF8DE80-ONC1257D85.0025401A-C1257D85.0025754B@inse.com>
Message-ID: <OF5DC75C48.22F22E1D-ONC1257D85.0027EAF6-C1257D85.0028A7AC@inse.com>

An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/ded6af3a/attachment.html>

From rt at steait.net  Mon Nov  3 07:37:37 2014
From: rt at steait.net (Rune Tipsmark)
Date: Mon, 3 Nov 2014 07:37:37 +0000
Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
In-Reply-To: <OF5DC75C48.22F22E1D-ONC1257D85.0027EAF6-C1257D85.0028A7AC@inse.com>
References: <164a9de15eb64f88a4785762899729a8@EX1301.steait.net>,
	<e5b49b9c7bc64e6a8fd9c88a76fd029f@EX1301.steait.net>,
	<64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>,
	<898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>,
	<OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>
	<OF69908773.2D8DA960-ONC1257D84.0065651F-C1257D84.0065ED8B@inse.com>
	<OF88628ACC.EFA25FDC-ONC1257D84.006DAC9E-C1257D84.006DACA1@inse.com>
	<OF3AF7BA9E.CCF8DE80-ONC1257D85.0025401A-C1257D85.0025754B@inse.com>
	<OF5DC75C48.22F22E1D-ONC1257D85.0027EAF6-C1257D85.0028A7AC@inse.com>
Message-ID: <aea894721c69434b933ce5bc76cdf904@EX1301.steait.net>

Tjenare,

Maybe its not possible, would explain why I cannot see the shared lun on the other system.
I don't have any spare PCI-E slots on either, all filled with IODrives etc.

Br,
Rune

From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 11:24 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com
Subject: Ang: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


Hej!


 Hmm, how about the target/initiiator configuration of the HCA's?

 When I think about it, I have never done this that you're trying: To use SRP both as target and initiator on the same system. I don't even know if it is possible. Since an HCA is entirely(with all ports) a target, you need to set one HCA as a target, and one HCA as an initiator. I've done it on fibre channel, but never on IB/SRP.


Best regards from/Med v?nliga h?lsningar fr?n

Johan Kragsterman

Capvert


-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-03 08:09
Kopia: "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
connectX2 and drivers are loaded, both OmniOS servers have LUNs I can access from both ESX and Windows... just the conection between them that I cant figure out.
Br,
Rune

From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 10:49 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Ang: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


Hej!
I can see I forgot the list in previous mail...

Anyway, are you using connectx adapters?

Have you checked if the driver is loaded in the host system?


Best regards from/Med v?nliga h?lsningar fr?n

Johan Kragsterman

Capvert


-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-03 00:07
?rende: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
It doesn't show up in stmfadm list-target.... only my other servers such as ESX and Windows show up...

Subnet Manager is OpenSM running on the ESX hosts, has been working OK ever since I installed then, prior to that I had a couple of Cisco(Topspin) DDR Switches with inbuilt SM.

Any other idea? All things I configured so far on both ZFS boxes,windows and esx has worked as expected... just this darn thing that's not doing what its supposed to...

It should be enough to just add the HCA from ZFS10 onto the host-group on ZFS00 for ZFS10 to see a LUN on ZFS00 granted that there is a view configured on ZFS00 as well... or am I missing a step?

Br
Rune

-----Original Message-----
From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 11:58 AM
To: Rune Tipsmark
Subject: Ang: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-02 20:05
Kopia: David Bomba <turbo124 at gmail.com<mailto:turbo124 at gmail.com>>, "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

I know, but how do I initiate a session from ZFS10?



If a session doesn't show up in "stmfadm list-targte -v", then you got something wrong in the IB fabric, if the view is right. Do you use a switch? Where do you have your IB Storage Manager?








Br,

Rune



From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 10:33 AM
To: Rune Tipsmark
Cc: David Bomba; omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Ang: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol






-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----

Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-02 19:11
Kopia: David Bomba <turbo124 at gmail.com<mailto:turbo124 at gmail.com>>, "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

Hi Johan

Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine..

What I can't  figure out is how to share a LUN with the other zfs box... see pasted info below...

ZFS00: (the box I want to receive my zfs snapshot) The below are all ESX servers, cannot see the other ZFS box

root at zfs00:/pool03# stmfadm list-target -v
Target: eui.0002C90300095E7C
    Operational Status: Online
    Provider Name     : srpt
    Alias             : -
    Protocol          : SRP
    Sessions          : 8
        Initiator: eui.0002C903000F397C
            Alias: 8102c90300095e7e:0002c903000f397c
            Logged in since: Sun Nov  2 02:09:41 2014
        Initiator: eui.0002C903000F397B
            Alias: 8102c90300095e7d:0002c903000f397b
            Logged in since: Sun Nov  2 02:09:40 2014
        Initiator: eui.0002C903000D3D04
            Alias: 8102c90300095e7e:0002c903000d3d04
            Logged in since: Sat Nov  1 21:14:47 2014
        Initiator: eui.0002C90300104F47
            Alias: 8102c90300095e7d:0002c90300104f47
            Logged in since: Sat Nov  1 21:12:54 2014
        Initiator: eui.0002C903000D3D03
            Alias: 8102c90300095e7d:0002c903000d3d03
            Logged in since: Sat Nov  1 21:12:32 2014
        Initiator: eui.0002C90300104F48
            Alias: 8102c90300095e7e:0002c90300104f48
            Logged in since: Sat Nov  1 21:10:45 2014
        Initiator: eui.0002C903000A48FA
            Alias: 8102c90300095e7e:0002c903000a48fa
            Logged in since: Sat Nov  1 21:10:40 2014
        Initiator: eui.0002C903000D3CA0
            Alias: 8102c90300095e7e:0002c903000d3ca0
            Logged in since: Sat Nov  1 21:10:39 2014
Target: iqn.2010-09.org.napp-it:1394106801
    Operational Status: Online
    Provider Name     : iscsit
    Alias             : 03.06.2014
    Protocol          : iSCSI
    Sessions          : 0

root at zfs00:/pool03# stmfadm list-lu -v
LU Name: 600144F007780B7F00005455EDD50002
    Operational Status: Online
    Provider Name     : sbd
    Alias             : /pool03/LU11
    View Entry Count  : 1
    Data File         : /pool03/LU11
    Meta File         : not set
    Size              : 2199023255552
    Block Size        : 512
    Management URL    : not set
    Vendor ID         : SUN
    Product ID        : COMSTAR
    Serial Num        : not set
    Write Protect     : Disabled
    Writeback Cache   : Disabled
    Access State      : Active

root at zfs00:/pool03# stmfadm list-hg -v
Host Group: ESX
Host Group: Windows
Host Group: ESX-iSER
Host Group: OmniOS
        Member: eui.0002C903000923E6 <<--- The other ZFS box
        Member: iqn.2010-09.org.napp-it:1402013225
        Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d

root at zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002 View Entry: 0
    Host group   : OmniOS
    Target group : All
    LUN          : 0

ZFS10: (the sending box where I want to see the LUN from ZFS00) No disk show up from ZFS00...







 Hi!





You got eui.0002C903000923E6 in host group OmniOS, but you don't have a session from that eui to the target.



Rgrds Johan























root at zfs10:/root# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c4t5000C50055FC9533d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50055fc9533
       1. c4t5000C50055FE6A63d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50055fe6a63
       2. c4t5000C500625B7EA7d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b7ea7
       3. c4t5000C500625B86E3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b86e3
       4. c4t5000C500625B886Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b886f
       5. c4t5000C500625B8137d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b8137
       6. c4t5000C500625B8427d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b8427
       7. c4t5000C500625BB773d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bb773
       8. c4t5000C500625BC2C3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bc2c3
       9. c4t5000C500625BD3EBd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bd3eb
      10. c4t5000C50057085A6Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057085a6b
      11. c4t5000C50057086B67d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057086b67
      12. c4t5000C50062878C0Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062878c0b
      13. c4t5000C50062878C43d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062878c43
      14. c4t5000C500570858EFd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500570858ef
      15. c4t5000C500570870D3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500570870d3
      16. c4t5000C5005708351Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c5005708351b
      17. c4t5000C5005708296Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c5005708296f
      18. c4t5000C50057089753d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057089753
      19. c4t5000C50057086307d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057086307
      20. c4t5000C50062879687d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879687
      21. c4t5000C50062879707d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879707
      22. c4t5000C50062879723d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879723
      23. c4t5000C50062879787d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879787
      24. c7t0d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 0,0
      25. c7t1d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 1,0
      26. c7t4d0 <ATA-InnoLite SATADOM-19-14.91GB>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 4,0
      27. c7t5d0 <ATA-InnoLite SATADOM-19-14.91GB>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 5,0
      28. c10d0 <Unknown-Unknown-0001-149.00GB>
          /pci at 0,0/pci8086,3c08 at 3/pci103c,178b at 0
      29. c11d0 <Unknown-Unknown-0001-149.00GB>
          /pci at 0,0/pci8086,3c0a at 3,2/pci103c,178b at 0
      30. c12d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
      31. c13d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
      32. c14d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
      33. c15d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
Specify disk (enter its number):



-----Original Message-----
From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 9:56 AM
To: Rune Tipsmark
Cc: David Bomba; omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com<mailto:omnios-discuss-bounces at lists.omniti.com>> skrev: -----
Till: David Bomba <turbo124 at gmail.com<mailto:turbo124 at gmail.com>>
Fr?n: Rune Tipsmark
S?nt av: "OmniOS-discuss"
Datum: 2014-11-02 07:55
Kopia: "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

Sounds sensible, how do I do that?

I tried creating a view for a thin lu with my other zfs box, but how do I detect it?





First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application.

Did you create SRP target service using COMSTAR?

# svcadm enable -r ibsrp/target

What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v"

Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link?

If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA.


Rgrds Johan






I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html



Br,

Rune



From: David Bomba [mailto:turbo124 at gmail.com]
Sent: Saturday, November 01, 2014 6:01 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol



I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential.





On 2 Nov 2014, at 11:45 am, Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> wrote:



Hi  all,



Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.



If not, is there a way I can aggregate say 8 or 16  IPoIB partitions and push throughput to a more reasonable speed...



Br,

Rune













_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss



_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/6226ffa5/attachment-0001.html>

From johan.kragsterman at capvert.se  Mon Nov  3 07:45:39 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Mon, 3 Nov 2014 08:45:39 +0100
Subject: [OmniOS-discuss] Ang: RE: RE: RE: RE: RE: Re: zfs send via SRP or
 other RDMA enabled protocol
In-Reply-To: <aea894721c69434b933ce5bc76cdf904@EX1301.steait.net>
References: <aea894721c69434b933ce5bc76cdf904@EX1301.steait.net>,
	<164a9de15eb64f88a4785762899729a8@EX1301.steait.net>,
	<e5b49b9c7bc64e6a8fd9c88a76fd029f@EX1301.steait.net>, 
	<64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>,
	<898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, 
	<OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>
	<OF69908773.2D8DA960-ONC1257D84.0065651F-C1257D84.0065ED8B@inse.com>
	<OF88628ACC.EFA25FDC-ONC1257D84.006DAC9E-C1257D84.006DACA1@inse.com>
	<OF3AF7BA9E.CCF8DE80-ONC1257D85.0025401A-C1257D85.0025754B@inse.com>
	<OF5DC75C48.22F22E1D-ONC1257D85.0027EAF6-C1257D85.0028A7AC@inse.com>
Message-ID: <OF30230E79.3D0D8BEE-ONC1257D85.002AA1E4-C1257D85.002AA1E7@inse.com>

An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/3b921af0/attachment-0001.html>

From johan.kragsterman at capvert.se  Mon Nov  3 08:40:50 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Mon, 3 Nov 2014 09:40:50 +0100
Subject: [OmniOS-discuss] Ang: RE: RE: RE: RE: RE: Re: zfs send via SRP or
 other RDMA enabled protocol
In-Reply-To: <aea894721c69434b933ce5bc76cdf904@EX1301.steait.net>
References: <aea894721c69434b933ce5bc76cdf904@EX1301.steait.net>,
	<164a9de15eb64f88a4785762899729a8@EX1301.steait.net>,
	<e5b49b9c7bc64e6a8fd9c88a76fd029f@EX1301.steait.net>, 
	<64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>,
	<898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>, 
	<OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>
	<OF69908773.2D8DA960-ONC1257D84.0065651F-C1257D84.0065ED8B@inse.com>
	<OF88628ACC.EFA25FDC-ONC1257D84.006DAC9E-C1257D84.006DACA1@inse.com>
	<OF3AF7BA9E.CCF8DE80-ONC1257D85.0025401A-C1257D85.0025754B@inse.com>
	<OF5DC75C48.22F22E1D-ONC1257D85.0027EAF6-C1257D85.0028A7AC@inse.com>
Message-ID: <OFAA49335F.E1FBE157-ONC1257D85.002FAF0B-C1257D85.002FAF0E@inse.com>

An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/ec0f74d6/attachment-0001.html>

From rt at steait.net  Mon Nov  3 08:50:56 2014
From: rt at steait.net (Rune Tipsmark)
Date: Mon, 3 Nov 2014 08:50:56 +0000
Subject: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
In-Reply-To: <OFAA49335F.E1FBE157-ONC1257D85.002FAF0B-C1257D85.002FAF0E@inse.com>
References: <aea894721c69434b933ce5bc76cdf904@EX1301.steait.net>,
	<164a9de15eb64f88a4785762899729a8@EX1301.steait.net>,
	<e5b49b9c7bc64e6a8fd9c88a76fd029f@EX1301.steait.net>,
	<64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>,
	<898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>,
	<OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>
	<OF69908773.2D8DA960-ONC1257D84.0065651F-C1257D84.0065ED8B@inse.com>
	<OF88628ACC.EFA25FDC-ONC1257D84.006DAC9E-C1257D84.006DACA1@inse.com>
	<OF3AF7BA9E.CCF8DE80-ONC1257D85.0025401A-C1257D85.0025754B@inse.com>
	<OF5DC75C48.22F22E1D-ONC1257D85.0027EAF6-C1257D85.0028A7AC@inse.com>
	<OFAA49335F.E1FBE157-ONC1257D85.002FAF0B-C1257D85.002FAF0E@inse.com>
Message-ID: <ed8af74acfbd4ebea07151721ab265d9@EX1301.steait.net>

That was a secondary thought, maybe worth testing one day.
Primarily I was looking at a way of speeding up zfs send-recv.

Guess it's a no go on a single HCA...

From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Monday, November 03, 2014 12:41 AM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com
Subject: Ang: RE: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


Hej!


If you want to mirror a local volume with a remote LU, I don't think it is a good idea. Asking for trubble...

I understand you want to get an HA solution. But I believe HA solutions needs other strategies than that.


Best regards from/Med v?nliga h?lsningar fr?n

Johan Kragsterman

Capvert


-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-03 08:37
Kopia: "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: RE: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
Tjenare,

Maybe its not possible, would explain why I cannot see the shared lun on the other system.
I don't have any spare PCI-E slots on either, all filled with IODrives etc.

Br,
Rune

From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 11:24 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Ang: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


Hej!


 Hmm, how about the target/initiiator configuration of the HCA's?

 When I think about it, I have never done this that you're trying: To use SRP both as target and initiator on the same system. I don't even know if it is possible. Since an HCA is entirely(with all ports) a target, you need to set one HCA as a target, and one HCA as an initiator. I've done it on fibre channel, but never on IB/SRP.


Best regards from/Med v?nliga h?lsningar fr?n

Johan Kragsterman

Capvert


-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-03 08:09
Kopia: "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
connectX2 and drivers are loaded, both OmniOS servers have LUNs I can access from both ESX and Windows... just the conection between them that I cant figure out.
Br,
Rune

From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 10:49 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Ang: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


Hej!
I can see I forgot the list in previous mail...

Anyway, are you using connectx adapters?

Have you checked if the driver is loaded in the host system?


Best regards from/Med v?nliga h?lsningar fr?n

Johan Kragsterman

Capvert


-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-03 00:07
?rende: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol
It doesn't show up in stmfadm list-target.... only my other servers such as ESX and Windows show up...

Subnet Manager is OpenSM running on the ESX hosts, has been working OK ever since I installed then, prior to that I had a couple of Cisco(Topspin) DDR Switches with inbuilt SM.

Any other idea? All things I configured so far on both ZFS boxes,windows and esx has worked as expected... just this darn thing that's not doing what its supposed to...

It should be enough to just add the HCA from ZFS10 onto the host-group on ZFS00 for ZFS10 to see a LUN on ZFS00 granted that there is a view configured on ZFS00 as well... or am I missing a step?

Br
Rune

-----Original Message-----
From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 11:58 AM
To: Rune Tipsmark
Subject: Ang: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-02 20:05
Kopia: David Bomba <turbo124 at gmail.com<mailto:turbo124 at gmail.com>>, "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

I know, but how do I initiate a session from ZFS10?



If a session doesn't show up in "stmfadm list-targte -v", then you got something wrong in the IB fabric, if the view is right. Do you use a switch? Where do you have your IB Storage Manager?








Br,

Rune



From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 10:33 AM
To: Rune Tipsmark
Cc: David Bomba; omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Ang: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol






-----Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> skrev: -----

Till: Johan Kragsterman <johan.kragsterman at capvert.se<mailto:johan.kragsterman at capvert.se>>
Fr?n: Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>>
Datum: 2014-11-02 19:11
Kopia: David Bomba <turbo124 at gmail.com<mailto:turbo124 at gmail.com>>, "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

Hi Johan

Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine..

What I can't  figure out is how to share a LUN with the other zfs box... see pasted info below...

ZFS00: (the box I want to receive my zfs snapshot) The below are all ESX servers, cannot see the other ZFS box

root at zfs00:/pool03# stmfadm list-target -v
Target: eui.0002C90300095E7C
    Operational Status: Online
    Provider Name     : srpt
    Alias             : -
    Protocol          : SRP
    Sessions          : 8
        Initiator: eui.0002C903000F397C
            Alias: 8102c90300095e7e:0002c903000f397c
            Logged in since: Sun Nov  2 02:09:41 2014
        Initiator: eui.0002C903000F397B
            Alias: 8102c90300095e7d:0002c903000f397b
            Logged in since: Sun Nov  2 02:09:40 2014
        Initiator: eui.0002C903000D3D04
            Alias: 8102c90300095e7e:0002c903000d3d04
            Logged in since: Sat Nov  1 21:14:47 2014
        Initiator: eui.0002C90300104F47
            Alias: 8102c90300095e7d:0002c90300104f47
            Logged in since: Sat Nov  1 21:12:54 2014
        Initiator: eui.0002C903000D3D03
            Alias: 8102c90300095e7d:0002c903000d3d03
            Logged in since: Sat Nov  1 21:12:32 2014
        Initiator: eui.0002C90300104F48
            Alias: 8102c90300095e7e:0002c90300104f48
            Logged in since: Sat Nov  1 21:10:45 2014
        Initiator: eui.0002C903000A48FA
            Alias: 8102c90300095e7e:0002c903000a48fa
            Logged in since: Sat Nov  1 21:10:40 2014
        Initiator: eui.0002C903000D3CA0
            Alias: 8102c90300095e7e:0002c903000d3ca0
            Logged in since: Sat Nov  1 21:10:39 2014
Target: iqn.2010-09.org.napp-it:1394106801
    Operational Status: Online
    Provider Name     : iscsit
    Alias             : 03.06.2014
    Protocol          : iSCSI
    Sessions          : 0

root at zfs00:/pool03# stmfadm list-lu -v
LU Name: 600144F007780B7F00005455EDD50002
    Operational Status: Online
    Provider Name     : sbd
    Alias             : /pool03/LU11
    View Entry Count  : 1
    Data File         : /pool03/LU11
    Meta File         : not set
    Size              : 2199023255552
    Block Size        : 512
    Management URL    : not set
    Vendor ID         : SUN
    Product ID        : COMSTAR
    Serial Num        : not set
    Write Protect     : Disabled
    Writeback Cache   : Disabled
    Access State      : Active

root at zfs00:/pool03# stmfadm list-hg -v
Host Group: ESX
Host Group: Windows
Host Group: ESX-iSER
Host Group: OmniOS
        Member: eui.0002C903000923E6 <<--- The other ZFS box
        Member: iqn.2010-09.org.napp-it:1402013225
        Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d

root at zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002 View Entry: 0
    Host group   : OmniOS
    Target group : All
    LUN          : 0

ZFS10: (the sending box where I want to see the LUN from ZFS00) No disk show up from ZFS00...







 Hi!





You got eui.0002C903000923E6 in host group OmniOS, but you don't have a session from that eui to the target.



Rgrds Johan























root at zfs10:/root# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c4t5000C50055FC9533d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50055fc9533
       1. c4t5000C50055FE6A63d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50055fe6a63
       2. c4t5000C500625B7EA7d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b7ea7
       3. c4t5000C500625B86E3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b86e3
       4. c4t5000C500625B886Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b886f
       5. c4t5000C500625B8137d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b8137
       6. c4t5000C500625B8427d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625b8427
       7. c4t5000C500625BB773d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bb773
       8. c4t5000C500625BC2C3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bc2c3
       9. c4t5000C500625BD3EBd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500625bd3eb
      10. c4t5000C50057085A6Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057085a6b
      11. c4t5000C50057086B67d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057086b67
      12. c4t5000C50062878C0Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062878c0b
      13. c4t5000C50062878C43d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062878c43
      14. c4t5000C500570858EFd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500570858ef
      15. c4t5000C500570870D3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c500570870d3
      16. c4t5000C5005708351Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c5005708351b
      17. c4t5000C5005708296Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c5005708296f
      18. c4t5000C50057089753d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057089753
      19. c4t5000C50057086307d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50057086307
      20. c4t5000C50062879687d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879687
      21. c4t5000C50062879707d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879707
      22. c4t5000C50062879723d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879723
      23. c4t5000C50062879787d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
          /scsi_vhci/disk at g5000c50062879787
      24. c7t0d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 0,0
      25. c7t1d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 1,0
      26. c7t4d0 <ATA-InnoLite SATADOM-19-14.91GB>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 4,0
      27. c7t5d0 <ATA-InnoLite SATADOM-19-14.91GB>
          /pci at 0,0/pci15d9,704 at 1f,2/disk at 5,0
      28. c10d0 <Unknown-Unknown-0001-149.00GB>
          /pci at 0,0/pci8086,3c08 at 3/pci103c,178b at 0
      29. c11d0 <Unknown-Unknown-0001-149.00GB>
          /pci at 0,0/pci8086,3c0a at 3,2/pci103c,178b at 0
      30. c12d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
      31. c13d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c04 at 2/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
      32. c14d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
      33. c15d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c08 at 3/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
Specify disk (enter its number):



-----Original Message-----
From: Johan Kragsterman [mailto:johan.kragsterman at capvert.se]
Sent: Sunday, November 02, 2014 9:56 AM
To: Rune Tipsmark
Cc: David Bomba; omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol


-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com<mailto:omnios-discuss-bounces at lists.omniti.com>> skrev: -----
Till: David Bomba <turbo124 at gmail.com<mailto:turbo124 at gmail.com>>
Fr?n: Rune Tipsmark
S?nt av: "OmniOS-discuss"
Datum: 2014-11-02 07:55
Kopia: "omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>" <omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>>
?rende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

Sounds sensible, how do I do that?

I tried creating a view for a thin lu with my other zfs box, but how do I detect it?





First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application.

Did you create SRP target service using COMSTAR?

# svcadm enable -r ibsrp/target

What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v"

Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link?

If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA.


Rgrds Johan






I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html



Br,

Rune



From: David Bomba [mailto:turbo124 at gmail.com]
Sent: Saturday, November 01, 2014 6:01 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol



I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential.





On 2 Nov 2014, at 11:45 am, Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> wrote:



Hi  all,



Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.



If not, is there a way I can aggregate say 8 or 16  IPoIB partitions and push throughput to a more reasonable speed...



Br,

Rune













_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss



_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/3712c46a/attachment-0001.html>

From johan.kragsterman at capvert.se  Mon Nov  3 09:08:35 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Mon, 3 Nov 2014 10:08:35 +0100
Subject: [OmniOS-discuss] Ang: RE: RE: RE: RE: RE: RE: Re: zfs send via SRP
 or other RDMA enabled protocol
In-Reply-To: <ed8af74acfbd4ebea07151721ab265d9@EX1301.steait.net>
References: <ed8af74acfbd4ebea07151721ab265d9@EX1301.steait.net>,
	<aea894721c69434b933ce5bc76cdf904@EX1301.steait.net>,
	<164a9de15eb64f88a4785762899729a8@EX1301.steait.net>, 
	<e5b49b9c7bc64e6a8fd9c88a76fd029f@EX1301.steait.net>,
	<64be1d574d7546e79d2515c737b62f0f@EX1301.steait.net>, 
	<898828e1fc7142379e23d126b5c4dbea@EX1301.steait.net>,
	<OF1FF16B00.D87BF953-ONC1257D84.00299CB1-C1257D84.006287C5@inse.com>
	<OF69908773.2D8DA960-ONC1257D84.0065651F-C1257D84.0065ED8B@inse.com>
	<OF88628ACC.EFA25FDC-ONC1257D84.006DAC9E-C1257D84.006DACA1@inse.com>
	<OF3AF7BA9E.CCF8DE80-ONC1257D85.0025401A-C1257D85.0025754B@inse.com>
	<OF5DC75C48.22F22E1D-ONC1257D85.0027EAF6-C1257D85.0028A7AC@inse.com>
	<OFAA49335F.E1FBE157-ONC1257D85.002FAF0B-C1257D85.002FAF0E@inse.com>
Message-ID: <OF6397DF87.ED98C8B9-ONC1257D85.003239AD-C1257D85.003239B0@inse.com>

An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/e5b8b751/attachment-0001.html>

From sk at kram.io  Mon Nov  3 10:21:15 2014
From: sk at kram.io (Steffen Kram)
Date: Mon, 3 Nov 2014 11:21:15 +0100
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <CACLsApuod9WnBT5FS6t7tN6eRtx1jbwUbkRJS9jZKxJv8jfcog@mail.gmail.com>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
	<20141101004433.GA12410@janus.obspm.bsdfrog.org>
	<CACLsApuod9WnBT5FS6t7tN6eRtx1jbwUbkRJS9jZKxJv8jfcog@mail.gmail.com>
Message-ID: <38CD52DB-4FFA-465B-8BF6-F57395718473@kram.io>

> I think an OmniOS Survival guide would be great.

Theo, I think that would be a really good idea. In my opinion some pages of the current wiki admin guide are already a very good first start (e.g. networking and zones). What?s missing in my point of view are ?Migration Guides? from other operating systems, especially linux and freebsd. Those migration guides should cover all things that look ?strange? for non-solaris folks e.g. partitioning, they should elaborate how to best use the ?specialities? e.g. how to use smf, fmd ..., what are the benefits of crossbow, what?s that automounter thing? Why is my home not in /home?

Those are the questions I often get from people not familiar with Illumos/Solaris. I already startet a couple of notes that would cover some of those questions. Helping newbies or linux-sysadmins to get started with omnios. So I would really like to contribute if there would be a real community site.

> If you have to put something like "newbies" in quotes and talk about them as different than us, you are viewing this community in an unhealthy way.  OmniOS is designed to take Illumos and all of its features and package it in a way that makes a minimal and useful general operating system targeted at servers and appliances.  Anyone with whom that vision aligns is most welcome in the community and we should encourage efforts to make them feel as an accepted part of the whole.

I think the OmniOS community might be small but I would never call it unhealthy and I must confess I like the familiar atmosphere.

One thing why I prefer OmniOS about any other os/distro out there is its focus on minimalism. This is something that totally contradicts with easy accessibility for newbies. However, we as a community could do a better job to pave the way for the more experienced linux/bsd sysadm who wants to dig into OmniOS. This might be to provide a comprehensive guide how to run typical software stacks. This should include how to build from the source, how to package those things and how to run it (smf). 

Although I provide a public OmniOS package server myself. I would never recommend to use omnios just with 3rd party repositories. I think the provided build scripts are a wealthy source of information to build own packages or to do a quick proof of concept but they should for sure not be viewed as the ?Missing OmniOS Distribution Packages?. Those packages might be highly customized for someone else very own specific needs.

What I?m getting at, it?s for sure that it would be nice to be more attractive for a broader user base, but all efforts should be in the spirit of the existing OmniOS. Hey, that?s why the existing user base is here in the first place. So let?s educate people on how to use OmniOS to unleash it?s full potential and don?t try to just provide them with easy to use pre-packaged solutions! 

Let me close with a pointer to the scala community. They had a similar problem for a long time, and since they come up with http://docs.scala-lang.org the documentation is constantly improving. I really like this approach of a comprehensive website, containing in-depth guides, easy how-tos for primers and handy cheat-cheats for the daily use. I?m ready to contribute to a great community doc and willing to invest time to build it if others also jump in!

Cheers, Steffen

?
Dr. Steffen Kram                   Ulm University
steffen.kram at uni-ulm.de            Institute for Applied
www.mathematik.uni-ulm.de/iai      Information Processing

Helmholtzstra?e 18                 Phone: 0731 50-23575
89081 Ulm, Germany                 Twitter: @stefritz

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2629 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/c45e2626/attachment.bin>

From richard at netbsd.org  Mon Nov  3 12:07:12 2014
From: richard at netbsd.org (Richard PALO)
Date: Mon, 03 Nov 2014 13:07:12 +0100
Subject: [OmniOS-discuss] SSL routines:SSL3_GET_RECORD:wrong version number
Message-ID: <m37r5a$tjl$1@ger.gmane.org>

Curious, haven't noticed this before:
>
> richard at omnis:/home/richard/src/solaris-userland~gate$ hg pull -uv
> abandon : erreur : _ssl.c:491: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

Some other mercurial sites seem okay, so I wonder if this is primarily 
on Or*cle's side or omnios (bloody)?

found here: https://hg.java.net/hg/solaris-userland~gate
this one too: https://hg.java.net/hg/solaris-desktop~spec-files


From richard at netbsd.org  Mon Nov  3 12:22:30 2014
From: richard at netbsd.org (Richard PALO)
Date: Mon, 03 Nov 2014 13:22:30 +0100
Subject: [OmniOS-discuss] SSL routines:SSL3_GET_RECORD:wrong version
	number
In-Reply-To: <m37r5a$tjl$1@ger.gmane.org>
References: <m37r5a$tjl$1@ger.gmane.org>
Message-ID: <m37s20$aom$1@ger.gmane.org>

Le 03/11/14 13:07, Richard PALO a ?crit :
> Curious, haven't noticed this before:
>>
>> richard at omnis:/home/richard/src/solaris-userland~gate$ hg pull -uv
>> abandon : erreur : _ssl.c:491: error:1408F10B:SSL
>> routines:SSL3_GET_RECORD:wrong version number
>
> Some other mercurial sites seem okay, so I wonder if this is primarily
> on Or*cle's side or omnios (bloody)?
>
> found here: https://hg.java.net/hg/solaris-userland~gate
> this one too: https://hg.java.net/hg/solaris-desktop~spec-files
>

Found the solution here:
http://stackoverflow.com/questions/21477683/mercurial-https-clone-abort-error-wrong-version-number

need to add now the following to ~/.hgrc:
 > tls = false

sorry for the noise.


From richard at netbsd.org  Mon Nov  3 13:57:37 2014
From: richard at netbsd.org (Richard PALO)
Date: Mon, 03 Nov 2014 14:57:37 +0100
Subject: [OmniOS-discuss] SSL routines:SSL3_GET_RECORD:wrong version
	number
In-Reply-To: <m37s20$aom$1@ger.gmane.org>
References: <m37r5a$tjl$1@ger.gmane.org> <m37s20$aom$1@ger.gmane.org>
Message-ID: <m381ka$a9i$1@ger.gmane.org>

Le 03/11/14 13:22, Richard PALO a ?crit :
> Le 03/11/14 13:07, Richard PALO a ?crit :
>> Curious, haven't noticed this before:
>>>
>>> richard at omnis:/home/richard/src/solaris-userland~gate$ hg pull -uv
>>> abandon : erreur : _ssl.c:491: error:1408F10B:SSL
>>> routines:SSL3_GET_RECORD:wrong version number
>>
>> Some other mercurial sites seem okay, so I wonder if this is primarily
>> on Or*cle's side or omnios (bloody)?
>>
>> found here: https://hg.java.net/hg/solaris-userland~gate
>> this one too: https://hg.java.net/hg/solaris-desktop~spec-files
>>
>
> Found the solution here:
> http://stackoverflow.com/questions/21477683/mercurial-https-clone-abort-error-wrong-version-number
>
>
> need to add now the following to ~/.hgrc:
>  > tls = false
>
> sorry for the noise.
>
actually, testing with pkgsrc, upgrading to 3.2 seems to work fine too, 
so that is probably a preferable solution.


From admdwrf at gmail.com  Mon Nov  3 17:04:56 2014
From: admdwrf at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBBbGl4?=)
Date: Mon, 3 Nov 2014 18:04:56 +0100
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <38CD52DB-4FFA-465B-8BF6-F57395718473@kram.io>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
	<20141101004433.GA12410@janus.obspm.bsdfrog.org>
	<CACLsApuod9WnBT5FS6t7tN6eRtx1jbwUbkRJS9jZKxJv8jfcog@mail.gmail.com>
	<38CD52DB-4FFA-465B-8BF6-F57395718473@kram.io>
Message-ID: <CAN4tE38GZ+Nt15TVgAgrQKgqSB_wuLHjyvDg5sJoOsFpkDArjg@mail.gmail.com>

Yep, a survival guide can be a good solution.
It should show to Linux/FreeBSD admin how to use ZFS, Zones,
Etherstub/Vnic, FSS, Capping memory, tuning the system.
In what format can we do this if you are agree ?
A page in the actual wiki ? a dedicated website ? cms ? pdf file ?

See you, Fred

2014-11-03 11:21 GMT+01:00 Steffen Kram <sk at kram.io>:

> > I think an OmniOS Survival guide would be great.
>
> Theo, I think that would be a really good idea. In my opinion some pages
> of the current wiki admin guide are already a very good first start (e.g.
> networking and zones). What?s missing in my point of view are ?Migration
> Guides? from other operating systems, especially linux and freebsd. Those
> migration guides should cover all things that look ?strange? for
> non-solaris folks e.g. partitioning, they should elaborate how to best use
> the ?specialities? e.g. how to use smf, fmd ..., what are the benefits of
> crossbow, what?s that automounter thing? Why is my home not in /home?
>
> Those are the questions I often get from people not familiar with
> Illumos/Solaris. I already startet a couple of notes that would cover some
> of those questions. Helping newbies or linux-sysadmins to get started with
> omnios. So I would really like to contribute if there would be a real
> community site.
>
> > If you have to put something like "newbies" in quotes and talk about
> them as different than us, you are viewing this community in an unhealthy
> way.  OmniOS is designed to take Illumos and all of its features and
> package it in a way that makes a minimal and useful general operating
> system targeted at servers and appliances.  Anyone with whom that vision
> aligns is most welcome in the community and we should encourage efforts to
> make them feel as an accepted part of the whole.
>
> I think the OmniOS community might be small but I would never call it
> unhealthy and I must confess I like the familiar atmosphere.
>
> One thing why I prefer OmniOS about any other os/distro out there is its
> focus on minimalism. This is something that totally contradicts with easy
> accessibility for newbies. However, we as a community could do a better job
> to pave the way for the more experienced linux/bsd sysadm who wants to dig
> into OmniOS. This might be to provide a comprehensive guide how to run
> typical software stacks. This should include how to build from the source,
> how to package those things and how to run it (smf).
>
> Although I provide a public OmniOS package server myself. I would never
> recommend to use omnios just with 3rd party repositories. I think the
> provided build scripts are a wealthy source of information to build own
> packages or to do a quick proof of concept but they should for sure not be
> viewed as the ?Missing OmniOS Distribution Packages?. Those packages might
> be highly customized for someone else very own specific needs.
>
> What I?m getting at, it?s for sure that it would be nice to be more
> attractive for a broader user base, but all efforts should be in the spirit
> of the existing OmniOS. Hey, that?s why the existing user base is here in
> the first place. So let?s educate people on how to use OmniOS to unleash
> it?s full potential and don?t try to just provide them with easy to use
> pre-packaged solutions!
>
> Let me close with a pointer to the scala community. They had a similar
> problem for a long time, and since they come up with
> http://docs.scala-lang.org the documentation is constantly improving. I
> really like this approach of a comprehensive website, containing in-depth
> guides, easy how-tos for primers and handy cheat-cheats for the daily use.
> I?m ready to contribute to a great community doc and willing to invest time
> to build it if others also jump in!
>
> Cheers, Steffen
>
> ?
> Dr. Steffen Kram                   Ulm University
> steffen.kram at uni-ulm.de            Institute for Applied
> www.mathematik.uni-ulm.de/iai      Information Processing
>
> Helmholtzstra?e 18                 Phone: 0731 50-23575
> 89081 Ulm, Germany                 Twitter: @stefritz
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/0feb45b2/attachment-0001.html>

From estibi at me.com  Mon Nov  3 18:37:58 2014
From: estibi at me.com (Piotr Jasiukajtis)
Date: Mon, 03 Nov 2014 19:37:58 +0100
Subject: [OmniOS-discuss] Dell R730xd
In-Reply-To: <DUB124-W225C0A27016AF53374FE89E0970@phx.gbl>
References: <DUB124-W3369D713B391C619CBA6D6E0970@phx.gbl>
	<CA13CC18-0789-4D19-9824-791116539AAF@omniti.com>
	<8D3E3DD7-EA48-4693-93DE-374C3E67BEFB@ohsu.edu>
	<CALeZrrTeLNRn1ABtWPfNP4NCe4DQwyPvHTvtUhN4Ojxoby177g@mail.gmail.com>
	<3F65C05E-30CA-4916-8304-578B0F927F0D@omniti.com>
	<DUB124-W225C0A27016AF53374FE89E0970@phx.gbl>
Message-ID: <47E2A81E-A722-4394-9706-601256F76315@me.com>

Hi Randy,

I would be interested in your results regarding R730xd, please post them once you have the machine.

?
Piotr Jasiukajtis



> On 20 Oct 2014, at 23:44, Randy S <sim.ple at live.nl> wrote:
> 
> Thanks for the input guys!
> I'll certainly be sure to have it assembled with intel nics and LSI hba's.
> If there is any interrest I can post results of preliminary tests here.
> It will be some weeks though before I receive the stuff (if all goes well).
> 
> Regards,
> 
> Randy
> 
> > From: danmcd at omniti.com
> > Date: Mon, 20 Oct 2014 16:27:47 -0400
> > To: chip at innovates.com
> > CC: omnios-discuss at lists.omniti.com
> > Subject: Re: [OmniOS-discuss] Dell R730xd
> > 
> > 
> > > On Oct 20, 2014, at 4:26 PM, Schweiss, Chip <chip at innovates.com> wrote:
> > > 
> > > Recently we had a quote from Dell w/ Nexenta. They specify Intel NICs and LSI HBAs when building with Nexenta.
> > 
> > Any HW setup that works with Nexenta should also work with OmniOS.
> > 
> > Dan
> > 
> > _______________________________________________
> > OmniOS-discuss mailing list
> > OmniOS-discuss at lists.omniti.com
> > http://lists.omniti.com/mailman/listinfo/omnios-discuss
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss


From richard.elling at richardelling.com  Mon Nov  3 19:30:19 2014
From: richard.elling at richardelling.com (Richard Elling)
Date: Mon, 3 Nov 2014 11:30:19 -0800
Subject: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%
In-Reply-To: <17b60b6ed2424559a706c5649f548c1e@EX1301.steait.net>
References: <4153d29f2b3a42349a5b701edfb06e92@EX1301.steait.net>
	<CAO8hXRCk-wJPfTa3krtGU57c2yNpL9G3c0m6LXi+B1iZNtd5+w@mail.gmail.com>
	<8FD2525B-70E3-4227-B9D5-8D4399D31335@richardelling.com>
	<aaa3c0225c574520b578c83a459fdc2d@EX1301.steait.net>
	<c1bbdd298b814dbea6744f9cc87a8017@EX1301.steait.net>
	<72A25F36-8BA9-4071-B009-AA90BDCA18B2@richardelling.com>
	<17b60b6ed2424559a706c5649f548c1e@EX1301.steait.net>
Message-ID: <0399C870-25B0-40BF-9C4F-16E50BEAF443@richardelling.com>


On Nov 2, 2014, at 6:24 PM, Rune Tipsmark <rt at steait.net> wrote:

> Looking a bit more at these numbers, am I seeing twice the actual rate due to mirroring?

You need to know what you are measuring. iostat and zpool iostat measure the I/O from
the pool to the disk. This is often very different than an application to the file system (pool).
In practice, it is often so different that it is impractical to correlate the two.

> How does compression affect the numbers?
> 
> Say I have 1 vdev mirrored and I see the pool writing 100 MB/sec, its what? 50 MB each disk? But from the client side only 50 MB/sec total? What if I compress it at the same time at say 1.50 ratio, will the pool show 100 MB/sec and the client write 75 MB/sec actual?

Measure both and compare. Usually we measure application side bandwidth using a tool
such as fsstat.
 -- richard

> 
> Br,
> Rune
> 
> -----Original Message-----
> From: Richard Elling [mailto:richard.elling at richardelling.com] 
> Sent: Sunday, November 02, 2014 6:07 PM
> To: Rune Tipsmark
> Cc: Eric Sproul; omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%
> 
> 
> On Oct 31, 2014, at 6:07 PM, Rune Tipsmark <rt at steait.net> wrote:
> 
>> So actually started storage vmotions on 3 host, 6 concurrent and am 
>> getting about 1GB/sec Guess I need more hosts to really push this, the disk are not more than 20-25% busy, so in theory I could push a bit more.
>> 
>> I think this is resolved for now.... cpu sitting at 30-40% usage while 
>> moving 1GB/sec
> 
> Yes, that seems about right.
> -- richard
> 
>> 
>> Iostat -xn 1
>> pool04       396G  39.5T      9  15.9K   325K  1.01G
>> pool04       396G  39.5T      7  17.0K   270K  1.03G
>> pool04       396G  39.5T     12  17.4K   558K  1.10G
>> pool04       396G  39.5T     10  16.9K   442K  1.03G
>> pool04       397G  39.5T      6  16.9K   332K  1021M
>> pool04       397G  39.5T      1  16.3K  74.9K  1.01G
>> pool04       397G  39.5T      8  17.0K   433K  1.05G
>> pool04       397G  39.5T     20  17.1K   716K  1023M
>> pool04       397G  39.5T     11  18.3K   425K  1.14G
>> pool04       398G  39.5T      0  18.3K  65.9K  1.11G
>> pool04       398G  39.5T     16  17.9K   551K  1.06G
>> pool04       398G  39.5T      0  16.8K   105K  1.03G
>> pool04       398G  39.5T      1  18.2K   124K  1.11G
>> pool04       398G  39.5T      0  17.1K  45.9K  1.05G
>> pool04       399G  39.5T      6  17.3K   454K  1.08G
>> pool04       399G  39.5T      0  17.9K      0  1.06G
>> pool04       399G  39.5T      2  16.9K   116K  1.04G
>> pool04       399G  39.5T      2  18.8K   130K  1.09G
>> pool04       399G  39.5T      0  17.6K      0  1.03G
>> pool04       400G  39.5T      3  17.5K   155K  1.04G
>> pool04       400G  39.5T      0  17.6K  31.5K  1.03G
>> 
>> -----Original Message-----
>> From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] 
>> On Behalf Of Rune Tipsmark
>> Sent: Friday, October 31, 2014 12:38 PM
>> To: Richard Elling; Eric Sproul
>> Cc: omnios-discuss at lists.omniti.com
>> Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%
>> 
>> Ok, makes sense.
>> What other kind of  indicators can I look at?
>> 
>> I get decent results from DD but still feels a bit slow...
>> 
>> Compression lz4 should not slow it down right? Cpu is not doing much when copying data over, maybe 15% busy or so... 
>> 
>> Sync=always, block size 1M
>> 204800000000 bytes (205 GB) copied, 296.379 s, 691 MB/s
>> real    4m56.382s
>> user    0m0.461s
>> sys     3m12.662s
>> 
>> Sync=disabled, block size 1M
>> 204800000000 bytes (205 GB) copied, 117.774 s, 1.7 GB/s
>> real    1m57.777s
>> user    0m0.237s
>> sys     1m57.466s
>> 
>> ... while doing this I was looking at my FIO cards, I think the reason is that the SLC's need more power to deliver higher performance, they are supposed to deliver 1.5GB/sec but only delivers around 350MB/sec each....
>> 
>> Now looking for aux power cables and will retest...
>> 
>> Br,
>> Rune
>> 
>> -----Original Message-----
>> From: Richard Elling [mailto:richard.elling at richardelling.com]
>> Sent: Friday, October 31, 2014 9:03 AM
>> To: Eric Sproul
>> Cc: Rune Tipsmark; omnios-discuss at lists.omniti.com
>> Subject: Re: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%
>> 
>> 
>> On Oct 31, 2014, at 7:14 AM, Eric Sproul <eric.sproul at circonus.com> wrote:
>> 
>>> On Fri, Oct 31, 2014 at 2:33 AM, Rune Tipsmark <rt at steait.net> wrote:
>>> 
>>>> Why is this pool showing near 100% busy when the underlying disks 
>>>> are doing nothing at all....
>>> 
>>> Simply put, it's just how the accounting works in iostat.  It treats 
>>> the pool like any other device, so if there is even one outstanding 
>>> request to the pool, it counts towards the busy%.  Keith W. from 
>>> Joyent explained this recently on the illumos-zfs list:
>>> http://www.listbox.com/member/archive/182191/2014/10/sort/time_rev/pa
>>> g 
>>> e/3/entry/18:93/20141017161955:F3E11AB2-563A-11E4-8EDC-D0C677981E2F/
>>> 
>>> The TL;DR is: if your pool has more than one disk in it, the 
>>> pool-wide busy% is useless.
>> 
>> FWIW, we use %busy as an indicator that we can ignore a device/subsystem when looking for performance problems. We don't use it as an indicator of problems. In other words, if the device isn't > 10% busy, forgetabouddit. If it is more busy, look in more detail at the meaningful performance indicators.
>> -- richard
>> 
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> 


From richard.elling at richardelling.com  Mon Nov  3 19:31:49 2014
From: richard.elling at richardelling.com (Richard Elling)
Date: Mon, 3 Nov 2014 11:31:49 -0800
Subject: [OmniOS-discuss] zfs pool 100% busy, disks less than 10%
In-Reply-To: <0399C870-25B0-40BF-9C4F-16E50BEAF443@richardelling.com>
References: <4153d29f2b3a42349a5b701edfb06e92@EX1301.steait.net>
	<CAO8hXRCk-wJPfTa3krtGU57c2yNpL9G3c0m6LXi+B1iZNtd5+w@mail.gmail.com>
	<8FD2525B-70E3-4227-B9D5-8D4399D31335@richardelling.com>
	<aaa3c0225c574520b578c83a459fdc2d@EX1301.steait.net>
	<c1bbdd298b814dbea6744f9cc87a8017@EX1301.steait.net>
	<72A25F36-8BA9-4071-B009-AA90BDCA18B2@richardelling.com>
	<17b60b6ed2424559a706c5649f548c1e@EX1301.steait.net>
	<0399C870-25B0-40BF-9C4F-16E50BEAF443@richardelling.com>
Message-ID: <BD6AAA8A-FEDF-4DEF-9840-CB721EA2A905@richardelling.com>

[responding to myself, considering the audience]

On Nov 3, 2014, at 11:30 AM, Richard Elling <richard.elling at richardelling.com> wrote:

> 
> On Nov 2, 2014, at 6:24 PM, Rune Tipsmark <rt at steait.net> wrote:
> 
>> Looking a bit more at these numbers, am I seeing twice the actual rate due to mirroring?
> 
> You need to know what you are measuring. iostat and zpool iostat measure the I/O from
> the pool to the disk. This is often very different than an application to the file system (pool).
> In practice, it is often so different that it is impractical to correlate the two.
> 
>> How does compression affect the numbers?
>> 
>> Say I have 1 vdev mirrored and I see the pool writing 100 MB/sec, its what? 50 MB each disk? But from the client side only 50 MB/sec total? What if I compress it at the same time at say 1.50 ratio, will the pool show 100 MB/sec and the client write 75 MB/sec actual?
> 
> Measure both and compare. Usually we measure application side bandwidth using a tool
> such as fsstat.

actually, we measure it using some sort of data collector that stores data for long-term
analysis and includes handy graphing features (cue Circonus)
 -- richard



From doug at will.to  Mon Nov  3 20:33:34 2014
From: doug at will.to (Doug Hughes)
Date: Mon, 3 Nov 2014 15:33:34 -0500
Subject: [OmniOS-discuss] strange console message
Message-ID: <CAOpmc6wzSA9D1bnaHtwqbsxw62fX2T=LbuPYJFhSKXo28GxCdg@mail.gmail.com>

I have a strange console message on a relatively newly installed machine:

root at x4275-3-15-39:/root# uname -a
SunOS x4275-3-15-39 5.11 omnios-f808c81 i86pc i386 i86pc
root at x4275-3-15-39:/root# more /etc/release
  OmniOS v11 r151012
  Copyright 2014 OmniTI Computer Consulting, Inc. All rights reserved.
  Use is subject to license terms.
root at x4275-3-15-39:/root#

They go back a ways in time:
       ###!!! [Child][DispatchInterruptMessage] Error: Unknown message: not
processed


     ###!!! [Child][DispatchInterruptMessage] Error: Unknown message: not
processed


   ###!!! [Child][DispatchInterruptMessage] Error: Unknown message: not
processed


 ###!!! [Child][DispatchInterruptMessage] Error: Unknown message: not
processed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/99edecb5/attachment.html>

From danmcd at omniti.com  Mon Nov  3 20:47:10 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Mon, 3 Nov 2014 15:47:10 -0500
Subject: [OmniOS-discuss] strange console message
In-Reply-To: <CAOpmc6wzSA9D1bnaHtwqbsxw62fX2T=LbuPYJFhSKXo28GxCdg@mail.gmail.com>
References: <CAOpmc6wzSA9D1bnaHtwqbsxw62fX2T=LbuPYJFhSKXo28GxCdg@mail.gmail.com>
Message-ID: <9AF6B5B7-0E65-4165-A0CD-8D57C3D5BB36@omniti.com>


> On Nov 3, 2014, at 3:33 PM, Doug Hughes <doug at will.to> wrote:
> 
> DispatchInterruptMessage


Are you running firefox on this box?  I don't see this string in the illumos source at all.  Googling it pointed me at bugzilla entries for what appears to be gecko (the firefox rendering engine).

Dan


From admdwrf at gmail.com  Mon Nov  3 21:10:40 2014
From: admdwrf at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBBbGl4?=)
Date: Mon, 3 Nov 2014 22:10:40 +0100
Subject: [OmniOS-discuss] Pb to compile vioif module
Message-ID: <CAN4tE38EQ2H6HWCgOoHAQwACbJAanwb-oD6gq6XSaSK0pVKi6Q@mail.gmail.com>

Since few days, i try to compile the virtio net driver.
I downloaded the last vioblk from nexenta git repo, no problem
But, when i forward one step on vioif, i eat a new wall after :p
Someone already compile it and can help me a bit please ;-) ?
Actually, i have this error message:

Current working directory
/code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif
fralix at build0:/code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif$
ksh /code/-omnios-151013/illumos-omnios/bldenv.sh -d
/code/-omnios-151013/illumos-omnios/illumos.sh 'make'


ATLOG=/code/-omnios-151013/illumos-omnios/log
HISTFILESIZE=100000
HISTSIZE=200
HOME=/home/fralix
HZ=100
LANG=C
LC_ALL=C
LC_COLLATE=C
LC_CTYPE=C
LC_MESSAGES=C
LC_MONETARY=C
LC_NUMERIC=C
LC_TIME=C
LOGFILE=/code/-omnios-151013/illumos-omnios/log/nightly.log
LOGNAME=fralix
MACH=i386
MAIL=/var/mail/fralix
MAILTO=fralix
MULTI_PROTO=no
NIGHTLY_OPTIONS=-nDCmpr
OLDPWD=/code/-omnios-151013/illumos-omnios/usr/src/uts/intel
ON_CLOSED_BINS=/code/-omnios-151013/illumos-omnios/closed
ON_CRYPTO_BINS=/code/-omnios-151013/illumos-omnios/on-crypto.i386.tar.bz2
PAGER='/usr/bin/less -ins'
PARENT_WS=''
PATH=/usr/gnu/bin:/usr/bin:/usr/sbin:/sbin:/usr/ucb:/usr/ccs/bin
PWD=/code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif
REF_PROTO_LIST=/usr/src/proto_list_i386
ROOT=/code/-omnios-151013/illumos-omnios/proto/root_i386
SHELL=/bin/bash
SHLVL=2
:ld: fatal: file obj64/vioif.o: open failed: No such file or directory
make: Fatal error: Command failed for target `obj64/vioif'
Current working directory
/code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif
make: Fatal error: Command failed for target `def.obj64'
Current working directory
/code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif

SRC=/code/-omnios-151013/illumos-omnios/usr/src

STAFFER=fralix
TERM=screen
_=CLASSPATH

Build type   is  DEBUG
RELEASE      is
VERSION      is
RELEASE_DATE is nightly

The top-level 'setup' target is available to build headers and tools.

Using /bin/bash as shell.
/usr/ccs/bin/ld -r -dy -N misc/virtio -N misc/mac -o debug64/vioif
debug64/vioif.o
ld: fatal: file debug64/vioif.o: open failed: No such file or directory
*** Error code 1
make: Fatal error: Command failed for target `debug64/vioif'
Current working directory
/code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif
*** Error code 1
The following command caused the error:
BUILD_TYPE=DBG64 VERSION='SunOS Development' make  def.targ
make: Fatal error: Command failed for target `def.debug64'
Current working directory
/code/-omnios-151013/illumos-omnios/usr/src/uts/intel/vioif


Fred
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/38e10f1b/attachment.html>

From youzhong at gmail.com  Mon Nov  3 21:14:55 2014
From: youzhong at gmail.com (Youzhong Yang)
Date: Mon, 3 Nov 2014 16:14:55 -0500
Subject: [OmniOS-discuss] How to release reserved TCP port binding in BOUND
	state
Message-ID: <CADpNCvaqgU5ZCxpW8onc8oJRoreMxH4Y_F=GZBt8dRV_zt1SEQ@mail.gmail.com>

Hello,

We are having a very strange issue on one of servers. The issue is that
fcntl locking over NFS returns 'no locks available' immediately.

dtrace shows that bindresvport() returns error code 125 (EADDRINUSE):

# dtrace -n 'fbt:rpcmod:bindresvport:return /arg1 != 0/ {stack();
printf("ret = %d", arg1);}'
  9  52692              bindresvport:return
              rpcmod`connmgr_get+0x560
              rpcmod`connmgr_wrapget+0x63
              rpcmod`clnt_cots_kcallit+0x198
              rpcmod`rpcbind_getaddr+0x245
              klmmod`update_host_rpcbinding+0x4f
              klmmod`nlm_host_get_rpc+0x6d
              klmmod`nlm_do_lock+0x10d
              klmmod`nlm4_lock_4_svc+0x2a
              klmmod`nlm_dispatch+0xe6
              klmmod`nlm_prog_4+0x34
              rpcmod`svc_getreq+0x1c1
              rpcmod`svc_run+0x146
              rpcmod`svc_do_run+0x8e
              nfs`nfssys+0xf1
              unix`_sys_sysenter_post_swapgs+0x149
ret = 125

netstat shows that 501 reserved ports are in BOUND state:

# netstat -an | grep BOUND
      *.935                *.*                0      0 1049740      0 BOUND
      *.801                *.*                0      0 1049740      0 BOUND
      *.798                *.*                0      0 1049740      0 BOUND
      *.561                *.*                0      0 1049740      0 BOUND
      *.613                *.*                0      0 1049740      0 BOUND
      ....
# netstat -an | grep BOUND | wc -l
     501

Has anyone seen this similar issue? is it possible to unbind those reserved
ports? Rebooting the server is our last resort.

Any advice would be very much appreciated.

Thanks,

-Youzhong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/f332d99b/attachment.html>

From doug at will.to  Mon Nov  3 21:55:26 2014
From: doug at will.to (Doug Hughes)
Date: Mon, 3 Nov 2014 16:55:26 -0500
Subject: [OmniOS-discuss] strange console message
In-Reply-To: <9AF6B5B7-0E65-4165-A0CD-8D57C3D5BB36@omniti.com>
References: <CAOpmc6wzSA9D1bnaHtwqbsxw62fX2T=LbuPYJFhSKXo28GxCdg@mail.gmail.com>
	<9AF6B5B7-0E65-4165-A0CD-8D57C3D5BB36@omniti.com>
Message-ID: <CAOpmc6y0Ar1th3pjD63eq4Q6egpW1RkGqik5d5sewO_o0q94hA@mail.gmail.com>

Oops, forgot to reply-all.

No windowing or windowing tools of any kind. Fresh boot, naked OS.

On Mon, Nov 3, 2014 at 3:47 PM, Dan McDonald <danmcd at omniti.com> wrote:

>
> > On Nov 3, 2014, at 3:33 PM, Doug Hughes <doug at will.to> wrote:
> >
> > DispatchInterruptMessage
>
>
> Are you running firefox on this box?  I don't see this string in the
> illumos source at all.  Googling it pointed me at bugzilla entries for what
> appears to be gecko (the firefox rendering engine).
>
> Dan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141103/076269c9/attachment.html>

From contact at novhost.eu  Tue Nov  4 02:35:59 2014
From: contact at novhost.eu (NovHost)
Date: Tue, 4 Nov 2014 03:35:59 +0100
Subject: [OmniOS-discuss] OmniOS KVM guest with virtio net support
Message-ID: <CAN4tE39jsQRJ-h_EkAf2CDHCefUr_x5YJhRkZoAo8C8o7qu4VA@mail.gmail.com>

I try to run OmniOS under Runabove cloud.
To this, i build this OmniOS image for KVM.
It's a qcow2 img. I run it with virtio blk and virtio net support on Fedora
and Ubuntu with virt-manager, without problem.

Actually, i have a problem to run it on Runabove, but it should run in
another KVM VPS providers.

You can download it at:
https://drive.google.com/file/d/0B3424f6EdZMmbFY2VkN0dnZ6TzQ/view?usp=sharing

I'll send my notes soon how i put virtio net driver.

fred.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141104/18fdbff9/attachment.html>

From nicolas.digregorio at gmail.com  Tue Nov  4 07:20:28 2014
From: nicolas.digregorio at gmail.com (Nicolas Di Gregorio)
Date: Tue, 4 Nov 2014 08:20:28 +0100
Subject: [OmniOS-discuss] Webdav recommandations
Message-ID: <CAO2ysPZYa1MTAGOyh3ypM41JtHDMiqpe7-SDkxyHWUAXd3PSWA@mail.gmail.com>

Hello,

What are you recommand as webdav client? I would like to mount a webdav
share at boot time is possible?

Thanks in advance

Kind Regards
digrouz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141104/7c6b2566/attachment.html>

From sk at kram.io  Tue Nov  4 09:23:06 2014
From: sk at kram.io (Steffen Kram)
Date: Tue, 4 Nov 2014 10:23:06 +0100
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <CAN4tE38GZ+Nt15TVgAgrQKgqSB_wuLHjyvDg5sJoOsFpkDArjg@mail.gmail.com>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
	<20141101004433.GA12410@janus.obspm.bsdfrog.org>
	<CACLsApuod9WnBT5FS6t7tN6eRtx1jbwUbkRJS9jZKxJv8jfcog@mail.gmail.com>
	<38CD52DB-4FFA-465B-8BF6-F57395718473@kram.io>
	<CAN4tE38GZ+Nt15TVgAgrQKgqSB_wuLHjyvDg5sJoOsFpkDArjg@mail.gmail.com>
Message-ID: <F8627EB9-5DFE-406C-B8C3-1A95497636DB@kram.io>

> In what format can we do this if you are agree ?
> A page in the actual wiki ? a dedicated website ? cms ? pdf file ?

What about putting it on github and just build a nice community doc with jekyll based github pages? That would allow easy contribution and gives us maximum flexibility.

Cheers, Steffen


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2629 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141104/68929ce3/attachment-0001.bin>

From sk at kram.io  Tue Nov  4 12:51:24 2014
From: sk at kram.io (Steffen Kram)
Date: Tue, 4 Nov 2014 13:51:24 +0100
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <F8627EB9-5DFE-406C-B8C3-1A95497636DB@kram.io>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
	<20141101004433.GA12410@janus.obspm.bsdfrog.org>
	<CACLsApuod9WnBT5FS6t7tN6eRtx1jbwUbkRJS9jZKxJv8jfcog@mail.gmail.com>
	<38CD52DB-4FFA-465B-8BF6-F57395718473@kram.io>
	<CAN4tE38GZ+Nt15TVgAgrQKgqSB_wuLHjyvDg5sJoOsFpkDArjg@mail.gmail.com>
	<F8627EB9-5DFE-406C-B8C3-1A95497636DB@kram.io>
Message-ID: <4469AEB7-B3EA-41D5-BAA3-6C3C1AB0BAC4@kram.io>


>> In what format can we do this if you are agree ?
>> A page in the actual wiki ? a dedicated website ? cms ? pdf file ?
> 
> What about putting it on github and just build a nice community doc with jekyll based github pages? That would allow easy contribution and gives us maximum flexibility.

@omniti: Would it be appreciated if I?m jumping in and just setup such a basic community doc page? I?m not a great artist but I think I?m able to produce something useable.

Cheers, Steffen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2629 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141104/b1711d10/attachment.bin>

From lotheac at iki.fi  Tue Nov  4 13:43:21 2014
From: lotheac at iki.fi (Lauri Tirkkonen)
Date: Tue, 4 Nov 2014 15:43:21 +0200
Subject: [OmniOS-discuss] pkgsend generate bug with spaces in file names
In-Reply-To: <5458D62B.9040006@scluk.com>
References: <20140318155845.GC21841@gutsman.lotheac.fi>
	<5458D62B.9040006@scluk.com>
Message-ID: <20141104134321.GA12810@gutsman.lotheac.fi>

On Tue, Nov 04 2014 13:35:39 +0000, Al Slater wrote:
> I have run into the same problem while packaging cmake.  Is there a
> solution?

I have an open pull request for this, but from what I understand OmniOS'
pkg isn't currently in a state where they can merge it. The OmniTIers
can probably elaborate on that, but if you want to apply the patch
yourself, it's at https://github.com/postwait/pkg5/pull/4

-- 
Lauri Tirkkonen | +358 50 5341376 | lotheac @ IRCnet

From admdwrf at gmail.com  Tue Nov  4 14:09:11 2014
From: admdwrf at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBBbGl4?=)
Date: Tue, 4 Nov 2014 15:09:11 +0100
Subject: [OmniOS-discuss] OmniOS KVM guest with virtio net support
In-Reply-To: <CAN4tE39jsQRJ-h_EkAf2CDHCefUr_x5YJhRkZoAo8C8o7qu4VA@mail.gmail.com>
References: <CAN4tE39jsQRJ-h_EkAf2CDHCefUr_x5YJhRkZoAo8C8o7qu4VA@mail.gmail.com>
Message-ID: <CAN4tE39uY_JjtujBUf3pmb=dUDJ6tUXi9FWe344RHURuN5TetA@mail.gmail.com>

Hum, very strange,
Some KVM host providers use this param to add disk img

-drive file=img.qcow2,if=virtio,id=drive-virtiodisk0,cache=none

If i use it for OmniOS, the boot process block at the load of vioblk module

With NexentaOS, it load without problem.

To start OmniOS correctly, i must use this:

-drive file=img.qcow2,if=none,id=drive-virtio-disk0,format=qcow2
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1

Since few hours, i am trying to understand why it's ok with nexenta
and not with OmniOS.

I used the illumos-nexenta source to build virtio, vioblk and blkdev
and nothing change.

Did you already had this situation ?


fred.


2014-11-04 3:35 GMT+01:00 NovHost <contact at novhost.eu>:

> I try to run OmniOS under Runabove cloud.
> To this, i build this OmniOS image for KVM.
> It's a qcow2 img. I run it with virtio blk and virtio net support on
> Fedora and Ubuntu with virt-manager, without problem.
>
> Actually, i have a problem to run it on Runabove, but it should run in
> another KVM VPS providers.
>
> You can download it at:
>
> https://drive.google.com/file/d/0B3424f6EdZMmbFY2VkN0dnZ6TzQ/view?usp=sharing
>
> I'll send my notes soon how i put virtio net driver.
>
> fred.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141104/a737482b/attachment.html>

From al.slater at scluk.com  Tue Nov  4 14:09:16 2014
From: al.slater at scluk.com (Al Slater)
Date: Tue, 04 Nov 2014 14:09:16 +0000
Subject: [OmniOS-discuss] pkgsend generate bug with spaces in file names
In-Reply-To: <20141104134321.GA12810@gutsman.lotheac.fi>
References: <20140318155845.GC21841@gutsman.lotheac.fi>
	<5458D62B.9040006@scluk.com>
	<20141104134321.GA12810@gutsman.lotheac.fi>
Message-ID: <5458DE0C.6040302@scluk.com>

On 04/11/14 13:43, Lauri Tirkkonen wrote:
> On Tue, Nov 04 2014 13:35:39 +0000, Al Slater wrote:
>> I have run into the same problem while packaging cmake.  Is there a
>> solution?
> 
> I have an open pull request for this, but from what I understand OmniOS'
> pkg isn't currently in a state where they can merge it. The OmniTIers
> can probably elaborate on that, but if you want to apply the patch
> yourself, it's at https://github.com/postwait/pkg5/pull/4

Thanks for that.


-- 
Al Slater

Technical Director
SCL

Phone : +44 (0)1273 666607
Fax   : +44 (0)1273 666601
email : al.slater at scluk.com

Stanton Consultancy Ltd

Park Gate, 161 Preston Road, Brighton, East Sussex, BN1 6AU

Registered in England Company number: 1957652 VAT number: GB 760 2433 55


From youzhong at gmail.com  Tue Nov  4 18:32:34 2014
From: youzhong at gmail.com (Youzhong Yang)
Date: Tue, 4 Nov 2014 13:32:34 -0500
Subject: [OmniOS-discuss] [developer] How to release reserved TCP port
	binding in BOUND state
In-Reply-To: <20141103215021.GG11456@telcontar.in.telka.sk>
References: <CADpNCvaqgU5ZCxpW8onc8oJRoreMxH4Y_F=GZBt8dRV_zt1SEQ@mail.gmail.com>
	<20141103215021.GG11456@telcontar.in.telka.sk>
Message-ID: <CADpNCvaqy9_Hami=zCqps53N=_U5RQW4VEp_MGvXtMayLyqkrw@mail.gmail.com>

Hi Marcel,

I really appreciate your input. We will try to find out what is wrong in
connmgr_get() which caused port leaking.

In the meantime, we want to change clnt_cots_do_bindresvport from 1 to 0
through mdb so that new connection will start using non-reserved port,

http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/rpc/clnt_cots.c#508

Will dynamically changing this variable have any side effect?

Thanks,
-Youzhong

On Mon, Nov 3, 2014 at 4:50 PM, Marcel Telka <marcel at telka.sk> wrote:

> Hi Youzhong,
>
> On Mon, Nov 03, 2014 at 04:14:55PM -0500, Youzhong Yang via
> illumos-developer wrote:
> > Hello,
> >
> > We are having a very strange issue on one of servers. The issue is that
> > fcntl locking over NFS returns 'no locks available' immediately.
> >
> > dtrace shows that bindresvport() returns error code 125 (EADDRINUSE):
> >
> > # dtrace -n 'fbt:rpcmod:bindresvport:return /arg1 != 0/ {stack();
> > printf("ret = %d", arg1);}'
> >   9  52692              bindresvport:return
> >               rpcmod`connmgr_get+0x560
> >               rpcmod`connmgr_wrapget+0x63
> >               rpcmod`clnt_cots_kcallit+0x198
> >               rpcmod`rpcbind_getaddr+0x245
> >               klmmod`update_host_rpcbinding+0x4f
> >               klmmod`nlm_host_get_rpc+0x6d
> >               klmmod`nlm_do_lock+0x10d
> >               klmmod`nlm4_lock_4_svc+0x2a
> >               klmmod`nlm_dispatch+0xe6
> >               klmmod`nlm_prog_4+0x34
> >               rpcmod`svc_getreq+0x1c1
> >               rpcmod`svc_run+0x146
> >               rpcmod`svc_do_run+0x8e
> >               nfs`nfssys+0xf1
> >               unix`_sys_sysenter_post_swapgs+0x149
> > ret = 125
> >
> > netstat shows that 501 reserved ports are in BOUND state:
> >
> > # netstat -an | grep BOUND
> >       *.935                *.*                0      0 1049740      0
> BOUND
> >       *.801                *.*                0      0 1049740      0
> BOUND
> >       *.798                *.*                0      0 1049740      0
> BOUND
> >       *.561                *.*                0      0 1049740      0
> BOUND
> >       *.613                *.*                0      0 1049740      0
> BOUND
> >       ....
> > # netstat -an | grep BOUND | wc -l
> >      501
> >
> > Has anyone seen this similar issue? is it possible to unbind those
> reserved
> > ports? Rebooting the server is our last resort.
> >
> > Any advice would be very much appreciated.
>
> I faced similar issue in connmgr_get().  It is filed as #1616 and the
> problem
> is that the dead connection is not properly closed (there seems to be
> missing
> connmgr_cancelconn() call somewhere), so the client could properly
> reconnect.
> Unfortunately, I had no time to finish the analysis of this bug.
>
>
> HTH
>
> --
> +-------------------------------------------+
> | Marcel Telka   e-mail:   marcel at telka.sk  |
> |                homepage: http://telka.sk/ |
> |                jabber:   marcel at jabber.sk |
> +-------------------------------------------+
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141104/24240421/attachment-0001.html>

From basil.crow at delphix.com  Tue Nov  4 19:07:48 2014
From: basil.crow at delphix.com (Basil Crow)
Date: Tue, 4 Nov 2014 11:07:48 -0800
Subject: [OmniOS-discuss] pkgsend generate bug with spaces in file names
In-Reply-To: <5458DE0C.6040302@scluk.com>
References: <20140318155845.GC21841@gutsman.lotheac.fi>
	<5458D62B.9040006@scluk.com>
	<20141104134321.GA12810@gutsman.lotheac.fi>
	<5458DE0C.6040302@scluk.com>
Message-ID: <CALszTP=y57hoypfeX0d6E+ObgzMC=2gOYdzUBwbpeP9NKZNe=A@mail.gmail.com>

On Tue, Nov 04 2014 13:35:39 +0000, Al Slater wrote:
> I have run into the same problem while packaging cmake.  Is there a
> solution?

We ran into three separate issues at Delphix using IPS with filenames
that involve spaces.

1.) --- Applying transforms
pkgmogrify: File /tmp/build_jenkins/masking.p5m.int line 912:
Malformed action at position: 72:
    file opt/masking/apache-tomcat-7.0.42/webapps/masking/image/Copy
of bg.gif group=bin mode=0644 owner=root
path="opt/masking/apache-tomcat-7.0.42/webapps/masking/image/Copy of
bg.gif"

2.) --- Applying transforms
--- Resolving dependencies
pkgdepend: Could not parse manifest
/tmp/build_delphix/masking.p5m.int.2 because of the following line:
file opt/masking/Application/Data Integration
32-bit.app/Contents/Info.plist path="opt/masking/Application/Data
Integration 32-bit.app/Contents/Info.plist" owner=root group=bin
mode=0755

3.) $ sudo pkg install masking
Creating Plan -
pkg install: The requested operation cannot be completed due to
invalid package metadata.  Details follow:
Malformed action at position: 56:
    dir path=opt/masking/Application/Data Integration
32-bit.app/Contents/Resources

Each problem above required a different patches:

1.) https://hg.openindiana.org/upstream/oracle/pkg-gate/rev/707707aba792
(generic.py)

2.) https://hg.openindiana.org/upstream/oracle/pkg-gate/rev/fe4116bd7737
(pkgfmt.py)

3.) https://hg.openindiana.org/upstream/oracle/pkg-gate/rev/a543cfbce08f
(manifest.py)

With these patches applied, we are able to use the full IPS workflow
with files that have spaces.

On Tue, Nov 4, 2014 at 6:09 AM, Al Slater <al.slater at scluk.com> wrote:
> On 04/11/14 13:43, Lauri Tirkkonen wrote:
>> On Tue, Nov 04 2014 13:35:39 +0000, Al Slater wrote:
>>> I have run into the same problem while packaging cmake.  Is there a
>>> solution?
>>
>> I have an open pull request for this, but from what I understand OmniOS'
>> pkg isn't currently in a state where they can merge it. The OmniTIers
>> can probably elaborate on that, but if you want to apply the patch
>> yourself, it's at https://github.com/postwait/pkg5/pull/4
>
> Thanks for that.
>
>
> --
> Al Slater
>
> Technical Director
> SCL
>
> Phone : +44 (0)1273 666607
> Fax   : +44 (0)1273 666601
> email : al.slater at scluk.com
>
> Stanton Consultancy Ltd
>
> Park Gate, 161 Preston Road, Brighton, East Sussex, BN1 6AU
>
> Registered in England Company number: 1957652 VAT number: GB 760 2433 55
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

From marcel at telka.sk  Mon Nov  3 21:50:21 2014
From: marcel at telka.sk (Marcel Telka)
Date: Mon, 3 Nov 2014 22:50:21 +0100
Subject: [OmniOS-discuss] [developer] How to release reserved TCP port
 binding in BOUND state
In-Reply-To: <CADpNCvaqgU5ZCxpW8onc8oJRoreMxH4Y_F=GZBt8dRV_zt1SEQ@mail.gmail.com>
References: <CADpNCvaqgU5ZCxpW8onc8oJRoreMxH4Y_F=GZBt8dRV_zt1SEQ@mail.gmail.com>
Message-ID: <20141103215021.GG11456@telcontar.in.telka.sk>

Hi Youzhong,

On Mon, Nov 03, 2014 at 04:14:55PM -0500, Youzhong Yang via illumos-developer wrote:
> Hello,
> 
> We are having a very strange issue on one of servers. The issue is that
> fcntl locking over NFS returns 'no locks available' immediately.
> 
> dtrace shows that bindresvport() returns error code 125 (EADDRINUSE):
> 
> # dtrace -n 'fbt:rpcmod:bindresvport:return /arg1 != 0/ {stack();
> printf("ret = %d", arg1);}'
>   9  52692              bindresvport:return
>               rpcmod`connmgr_get+0x560
>               rpcmod`connmgr_wrapget+0x63
>               rpcmod`clnt_cots_kcallit+0x198
>               rpcmod`rpcbind_getaddr+0x245
>               klmmod`update_host_rpcbinding+0x4f
>               klmmod`nlm_host_get_rpc+0x6d
>               klmmod`nlm_do_lock+0x10d
>               klmmod`nlm4_lock_4_svc+0x2a
>               klmmod`nlm_dispatch+0xe6
>               klmmod`nlm_prog_4+0x34
>               rpcmod`svc_getreq+0x1c1
>               rpcmod`svc_run+0x146
>               rpcmod`svc_do_run+0x8e
>               nfs`nfssys+0xf1
>               unix`_sys_sysenter_post_swapgs+0x149
> ret = 125
> 
> netstat shows that 501 reserved ports are in BOUND state:
> 
> # netstat -an | grep BOUND
>       *.935                *.*                0      0 1049740      0 BOUND
>       *.801                *.*                0      0 1049740      0 BOUND
>       *.798                *.*                0      0 1049740      0 BOUND
>       *.561                *.*                0      0 1049740      0 BOUND
>       *.613                *.*                0      0 1049740      0 BOUND
>       ....
> # netstat -an | grep BOUND | wc -l
>      501
> 
> Has anyone seen this similar issue? is it possible to unbind those reserved
> ports? Rebooting the server is our last resort.
> 
> Any advice would be very much appreciated.

I faced similar issue in connmgr_get().  It is filed as #1616 and the problem
is that the dead connection is not properly closed (there seems to be missing
connmgr_cancelconn() call somewhere), so the client could properly reconnect.
Unfortunately, I had no time to finish the analysis of this bug.


HTH

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel at telka.sk  |
|                homepage: http://telka.sk/ |
|                jabber:   marcel at jabber.sk |
+-------------------------------------------+

From marcel at telka.sk  Tue Nov  4 20:02:55 2014
From: marcel at telka.sk (Marcel Telka)
Date: Tue, 4 Nov 2014 21:02:55 +0100
Subject: [OmniOS-discuss] [developer] How to release reserved TCP port
 binding in BOUND state
In-Reply-To: <CADpNCvaqy9_Hami=zCqps53N=_U5RQW4VEp_MGvXtMayLyqkrw@mail.gmail.com>
References: <CADpNCvaqgU5ZCxpW8onc8oJRoreMxH4Y_F=GZBt8dRV_zt1SEQ@mail.gmail.com>
	<20141103215021.GG11456@telcontar.in.telka.sk>
	<CADpNCvaqy9_Hami=zCqps53N=_U5RQW4VEp_MGvXtMayLyqkrw@mail.gmail.com>
Message-ID: <20141104200255.GH11456@telcontar.in.telka.sk>

On Tue, Nov 04, 2014 at 01:32:34PM -0500, Youzhong Yang via illumos-developer wrote:
> Hi Marcel,
> 
> I really appreciate your input. We will try to find out what is wrong in
> connmgr_get() which caused port leaking.
> 
> In the meantime, we want to change clnt_cots_do_bindresvport from 1 to 0
> through mdb so that new connection will start using non-reserved port,
> 
> http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/rpc/clnt_cots.c#508
> 
> Will dynamically changing this variable have any side effect?

I do not think it will cause any troubles when changed dynamically.
IOW, it might help you :-).

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   marcel at telka.sk  |
|                homepage: http://telka.sk/ |
|                jabber:   marcel at jabber.sk |
+-------------------------------------------+

From johan.kragsterman at capvert.se  Wed Nov  5 20:09:10 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Wed, 5 Nov 2014 21:09:10 +0100
Subject: [OmniOS-discuss] KVM and SMF
Message-ID: <OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>


Hi!


I'm on my way to try to get my KVM VM's under SMF control. Have been looking at various exampel xml-files, and there is something I'm not clear about, that I'd like to ask those who knows how to handle this.

The service name in this example looks like this: <service name='kvm/pfsense'

But there is no directory for "kvm" under /var/svc/manifest/ or /lib/svc/manifest, so where does it go? is svc creating a directory when I import the file?


I'd used the example on the omnios wiki, just changed the service name. It looks like this in my version:


<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
	<service name='kvm/pfsense' type='service' version='0'>
		<create_default_instance enabled='true'/>
		<single_instance/>
		<dependency name='network'
		grouping='require_all'
		restart_on='none'
		type='service'>
			<service_fmri value='svc:/milestone/network:default' />
		</dependency>
		<dependency name='filesystem'
		grouping='require_all'
		restart_on='none'
		type='service'>
			<service_fmri value='svc:/system/filesystem/local:default' />
		</dependency>
		<exec_method name='start'
		type='method'
		exec='/usr/bin/vmpfsense.sh'
		timeout_seconds='60'/>
		<exec_method name='stop'
		type='method'
		exec='/usr/bin/vmpfsense.sh'
		timeout_seconds='60'/>
		<exec_method name='stop'
		type='method'
		exec=':kill'
		timeout_seconds='60'/>
		<stability value='Unstable'/>
		<template>
			<common_name>
				<loctext xml:lang='C'>KVM-pfsense</loctext>
			<common_name>
		<template>
	</service>
</service_bundle>




What do you think, would that do?




Best regards from/Med v?nliga h?lsningar fr?n

Johan Kragsterman

Capvert


From chip at innovates.com  Wed Nov  5 20:31:08 2014
From: chip at innovates.com (Schweiss, Chip)
Date: Wed, 5 Nov 2014 14:31:08 -0600
Subject: [OmniOS-discuss] crash dump speed painfully slow
Message-ID: <CALeZrrRRjKBjp5WJ4CWhS1-CESc9A__+wYe3SS7kSwtkrATPHg@mail.gmail.com>

I had a system crash this morning when I/O on a pool was hung with
failmode=panic.

That was 7 1/2 hours ago and it's at 64%.    I'm going to let it complete
so the failure can be analyzed more, but I really need to find a way to
significantly speed up the crash dumps.

This system has 256GB ram and a dedicated dump pool of two 320GB mirrored
SATA disks.

I'm aware of the multi-threaded dump issue so I have these to lines in
/etc/system

set dump_plat_mincpu=0
set dump_bzip2_level=1

It's not clear where the bottleneck is.  I don't think it's the disk.  The
disk activity lights are a light flicker, by no means being hit hard.
Unless it's a serial latency issue.

What would be my best approach to get good dumps from a system like this in
minutes, not hours?   Assuming that is possible.

Thanks!
-Chip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141105/9d3da963/attachment.html>

From danmcd at omniti.com  Wed Nov  5 20:36:14 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Wed, 5 Nov 2014 15:36:14 -0500
Subject: [OmniOS-discuss] [discuss] crash dump speed painfully slow
In-Reply-To: <CALeZrrRRjKBjp5WJ4CWhS1-CESc9A__+wYe3SS7kSwtkrATPHg@mail.gmail.com>
References: <CALeZrrRRjKBjp5WJ4CWhS1-CESc9A__+wYe3SS7kSwtkrATPHg@mail.gmail.com>
Message-ID: <6279409A-0B87-4DF3-88E1-12FB3B0A0CD0@omniti.com>


> On Nov 5, 2014, at 3:31 PM, Schweiss, Chip via illumos-discuss <discuss at lists.illumos.org> wrote:
> 
> I had a system crash this morning when I/O on a pool was hung with failmode=panic.

What version of OmniOS are you running?

> That was 7 1/2 hours ago and it's at 64%.    I'm going to let it complete so the failure can be analyzed more, but I really need to find a way to significantly speed up the crash dumps.
> 
> This system has 256GB ram and a dedicated dump pool of two 320GB mirrored SATA disks.
> 
> I'm aware of the multi-threaded dump issue so I have these to lines in /etc/system 
> 
> set dump_plat_mincpu=0

That should be the default now.  I can't remember which rev of OmniOS it showed up in, but certainly it's in 010 or later.

> It's not clear where the bottleneck is.  I don't think it's the disk.  The disk activity lights are a light flicker, by no means being hit hard.  Unless it's a serial latency issue.
> 
> What would be my best approach to get good dumps from a system like this in minutes, not hours?   Assuming that is possible.

You weren't dumping to the same pool or the same controller where you had the I/O hang, were you?

Dan


From danmcd at omniti.com  Wed Nov  5 20:37:38 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Wed, 5 Nov 2014 15:37:38 -0500
Subject: [OmniOS-discuss] [discuss] crash dump speed painfully slow
In-Reply-To: <CALeZrrRRjKBjp5WJ4CWhS1-CESc9A__+wYe3SS7kSwtkrATPHg@mail.gmail.com>
References: <CALeZrrRRjKBjp5WJ4CWhS1-CESc9A__+wYe3SS7kSwtkrATPHg@mail.gmail.com>
Message-ID: <C651BF25-D59C-469E-ADF1-78A7631CA6CF@omniti.com>

> 
> set dump_plat_mincpu=0
> set dump_bzip2_level=1

Both of these are the default in any supported-now OmniOS release (006, 010, 012, and bloody).

Dan


From tobi at oetiker.ch  Wed Nov  5 20:41:56 2014
From: tobi at oetiker.ch (Tobias Oetiker)
Date: Wed, 5 Nov 2014 21:41:56 +0100 (CET)
Subject: [OmniOS-discuss] KVM and SMF
In-Reply-To: <OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
References: <OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
Message-ID: <alpine.DEB.2.10.1411052141140.5357@engelberg.oetiker.ch>

Hi Johan,

habe a look at

https://github.com/hadfl/kvmadm

cheers
tobi

Today Johan Kragsterman wrote:

>
> Hi!
>
>
> I'm on my way to try to get my KVM VM's under SMF control. Have been looking at various exampel xml-files, and there is something I'm not clear about, that I'd like to ask those who knows how to handle this.
>
> The service name in this example looks like this: <service name='kvm/pfsense'
>
> But there is no directory for "kvm" under /var/svc/manifest/ or /lib/svc/manifest, so where does it go? is svc creating a directory when I import the file?
>
>
> I'd used the example on the omnios wiki, just changed the service name. It looks like this in my version:
>
>
> <?xml version='1.0'?>
> <!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
> <service_bundle type='manifest' name='export'>
> 	<service name='kvm/pfsense' type='service' version='0'>
> 		<create_default_instance enabled='true'/>
> 		<single_instance/>
> 		<dependency name='network'
> 		grouping='require_all'
> 		restart_on='none'
> 		type='service'>
> 			<service_fmri value='svc:/milestone/network:default' />
> 		</dependency>
> 		<dependency name='filesystem'
> 		grouping='require_all'
> 		restart_on='none'
> 		type='service'>
> 			<service_fmri value='svc:/system/filesystem/local:default' />
> 		</dependency>
> 		<exec_method name='start'
> 		type='method'
> 		exec='/usr/bin/vmpfsense.sh'
> 		timeout_seconds='60'/>
> 		<exec_method name='stop'
> 		type='method'
> 		exec='/usr/bin/vmpfsense.sh'
> 		timeout_seconds='60'/>
> 		<exec_method name='stop'
> 		type='method'
> 		exec=':kill'
> 		timeout_seconds='60'/>
> 		<stability value='Unstable'/>
> 		<template>
> 			<common_name>
> 				<loctext xml:lang='C'>KVM-pfsense</loctext>
> 			<common_name>
> 		<template>
> 	</service>
> </service_bundle>
>
>
>
>
> What do you think, would that do?
>
>
>
>
> Best regards from/Med v?nliga h?lsningar fr?n
>
> Johan Kragsterman
>
> Capvert
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch tobi at oetiker.ch +41 62 775 9902

From chip at innovates.com  Wed Nov  5 20:42:56 2014
From: chip at innovates.com (Schweiss, Chip)
Date: Wed, 5 Nov 2014 14:42:56 -0600
Subject: [OmniOS-discuss] [discuss] crash dump speed painfully slow
In-Reply-To: <6279409A-0B87-4DF3-88E1-12FB3B0A0CD0@omniti.com>
References: <CALeZrrRRjKBjp5WJ4CWhS1-CESc9A__+wYe3SS7kSwtkrATPHg@mail.gmail.com>
	<6279409A-0B87-4DF3-88E1-12FB3B0A0CD0@omniti.com>
Message-ID: <CALeZrrQxo4ryj2_PB6bvgOvmemjqLR1BQCocrOAgfSZS45PxJQ@mail.gmail.com>

On Wed, Nov 5, 2014 at 2:36 PM, Dan McDonald <danmcd at omniti.com> wrote:

>
> > On Nov 5, 2014, at 3:31 PM, Schweiss, Chip via illumos-discuss <
> discuss at lists.illumos.org> wrote:
> >
> > I had a system crash this morning when I/O on a pool was hung with
> failmode=panic.
>
> What version of OmniOS are you running?
>

r1510010


>
> > That was 7 1/2 hours ago and it's at 64%.    I'm going to let it
> complete so the failure can be analyzed more, but I really need to find a
> way to significantly speed up the crash dumps.
> >
> > This system has 256GB ram and a dedicated dump pool of two 320GB
> mirrored SATA disks.
> >
> > I'm aware of the multi-threaded dump issue so I have these to lines in
> /etc/system
> >
> > set dump_plat_mincpu=0
>
> That should be the default now.  I can't remember which rev of OmniOS it
> showed up in, but certainly it's in 010 or later.
>
> > It's not clear where the bottleneck is.  I don't think it's the disk.
> The disk activity lights are a light flicker, by no means being hit hard.
> Unless it's a serial latency issue.
> >
> > What would be my best approach to get good dumps from a system like this
> in minutes, not hours?   Assuming that is possible.
>
> You weren't dumping to the same pool or the same controller where you had
> the I/O hang, were you?
>

No, I setup a dedicated dump pool to a pair of mirrored SATA disks
connected to the motherboard AHCI controller.

-Chip

>
> Dan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141105/92dd6b17/attachment.html>

From johan.kragsterman at capvert.se  Wed Nov  5 20:54:26 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Wed, 5 Nov 2014 21:54:26 +0100
Subject: [OmniOS-discuss] Ang: Re: Ang: Re:  KVM and SMF
In-Reply-To: <90cff68c42e6d90bb590279e49f449ce@blackdot.be>
References: <90cff68c42e6d90bb590279e49f449ce@blackdot.be>,
	<9eb8e5a35d515d2c87e47e031100bfdb@blackdot.be>,
	<OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
	<OF99AFB31A.F9AB8572-ONC1257D87.0070282C-C1257D87.0070282D@inse.com>
Message-ID: <OF58AAE2A2.EB136B95-ONC1257D87.0072D8CB-C1257D87.0072D8CD@inse.com>

Okay, thanks, Jorge!

I'll try it and see what happens...



-----Jorge Schrauwen <sjorge+ml at blackdot.be> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se>
Fr?n: Jorge Schrauwen <sjorge+ml at blackdot.be>
Datum: 2014-11-05 21:27
?rende: Re: Ang: Re: [OmniOS-discuss] KVM and SMF

I don't know. But from experience svccfg will do what is needed to 
import it.

But I know for a fact service names <> paths. base on I have a few 
services that do not have a corresponding directory.

Regards

Jorge



On 2014-11-05 21:25, Johan Kragsterman wrote:
> -----Jorge Schrauwen <jorge at blackdot.be> skrev: -----
> Till: Johan Kragsterman <johan.kragsterman at capvert.se>
> Fr?n: Jorge Schrauwen <jorge at blackdot.be>
> Datum: 2014-11-05 21:11
> ?rende: Re: [OmniOS-discuss] KVM and SMF
> 
> You can import it using 'svccfg import mysmf.xml'.
> 
> ---
> ~ sjorge
> 
> 
> Hey, Jorge, I thought you were going to answer me...!
> 
> 
> Yeah, I know how I import it, but what I asked about was the naming:
> 
> ?If I call it: service name='kvm/pfsense', where does it go, when I
> import it, since there is no "kvm" directory under /var/svc/manifest/
> or /lib/svc/manifest? Is smf creating a directory for it?
> 
> 
> 
> 
> 
> On 2014-11-05 21:09, Johan Kragsterman wrote:
>> Hi!
>> 
>> 
>> I'm on my way to try to get my KVM VM's under SMF control. Have been
>> looking at various exampel xml-files, and there is something I'm not
>> clear about, that I'd like to ask those who knows how to handle this.
>> 
>> The service name in this example looks like this: <service
>> name='kvm/pfsense'
>> 
>> But there is no directory for "kvm" under /var/svc/manifest/ or
>> /lib/svc/manifest, so where does it go? is svc creating a directory
>> when I import the file?
>> 
>> 
>> I'd used the example on the omnios wiki, just changed the service
>> name. It looks like this in my version:
>> 
>> 
>> <?xml version='1.0'?>
>> <!DOCTYPE service_bundle SYSTEM
>> '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
>> <service_bundle type='manifest' name='export'>
>> <service name='kvm/pfsense' type='service' version='0'>
>> <create_default_instance enabled='true'/>
>> <single_instance/>
>> <dependency name='network'
>> grouping='require_all'
>> restart_on='none'
>> type='service'>
>> <service_fmri value='svc:/milestone/network:default' />
>> </dependency>
>> <dependency name='filesystem'
>> grouping='require_all'
>> restart_on='none'
>> type='service'>
>> <service_fmri value='svc:/system/filesystem/local:default' />
>> </dependency>
>> <exec_method name='start'
>> type='method'
>> exec='/usr/bin/vmpfsense.sh'
>> timeout_seconds='60'/>
>> <exec_method name='stop'
>> type='method'
>> exec='/usr/bin/vmpfsense.sh'
>> timeout_seconds='60'/>
>> <exec_method name='stop'
>> type='method'
>> exec=':kill'
>> timeout_seconds='60'/>
>> <stability value='Unstable'/>
>> <template>
>> <common_name>
>> <loctext xml:lang='C'>KVM-pfsense</loctext>
>> <common_name>
>> <template>
>> </service>
>> </service_bundle>
>> 
>> 
>> 
>> 
>> What do you think, would that do?
>> 
>> 
>> 
>> 
>> Best regards from/Med v?nliga h?lsningar fr?n
>> 
>> Johan Kragsterman
>> 
>> Capvert
>> 
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss




From richard.elling at richardelling.com  Wed Nov  5 21:16:29 2014
From: richard.elling at richardelling.com (Richard Elling)
Date: Wed, 5 Nov 2014 13:16:29 -0800
Subject: [OmniOS-discuss] [discuss] crash dump speed painfully slow
In-Reply-To: <CALeZrrQxo4ryj2_PB6bvgOvmemjqLR1BQCocrOAgfSZS45PxJQ@mail.gmail.com>
References: <CALeZrrRRjKBjp5WJ4CWhS1-CESc9A__+wYe3SS7kSwtkrATPHg@mail.gmail.com>
	<6279409A-0B87-4DF3-88E1-12FB3B0A0CD0@omniti.com>
	<CALeZrrQxo4ryj2_PB6bvgOvmemjqLR1BQCocrOAgfSZS45PxJQ@mail.gmail.com>
Message-ID: <296A8AD7-D6C3-4DE7-B52C-D56985C9C63F@richardelling.com>


On Nov 5, 2014, at 12:42 PM, Schweiss, Chip via illumos-discuss <discuss at lists.illumos.org> wrote:

> 
> On Wed, Nov 5, 2014 at 2:36 PM, Dan McDonald <danmcd at omniti.com> wrote:
> 
> > On Nov 5, 2014, at 3:31 PM, Schweiss, Chip via illumos-discuss <discuss at lists.illumos.org> wrote:
> >
> > I had a system crash this morning when I/O on a pool was hung with failmode=panic.
> 
> What version of OmniOS are you running?
> 
> r1510010
>  
> 
> > That was 7 1/2 hours ago and it's at 64%.    I'm going to let it complete so the failure can be analyzed more, but I really need to find a way to significantly speed up the crash dumps.
> >
> > This system has 256GB ram and a dedicated dump pool of two 320GB mirrored SATA disks.
> >
> > I'm aware of the multi-threaded dump issue so I have these to lines in /etc/system
> >
> > set dump_plat_mincpu=0
> 
> That should be the default now.  I can't remember which rev of OmniOS it showed up in, but certainly it's in 010 or later.
> 
> > It's not clear where the bottleneck is.  I don't think it's the disk.  The disk activity lights are a light flicker, by no means being hit hard.  Unless it's a serial latency issue.
> >
> > What would be my best approach to get good dumps from a system like this in minutes, not hours?   Assuming that is possible.
> 
> You weren't dumping to the same pool or the same controller where you had the I/O hang, were you?
> 
> No, I setup a dedicated dump pool to a pair of mirrored SATA disks connected to the motherboard AHCI controller. 

Try dumping to a raw slice instead of ZFS.
 -- richard

> 
> -Chip
> 
> Dan
> 
> 
> illumos-discuss | Archives  | Modify Your Subscription	 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141105/4fa05b63/attachment.html>

From danmcd at omniti.com  Thu Nov  6 01:05:08 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Wed, 5 Nov 2014 20:05:08 -0500
Subject: [OmniOS-discuss] ZFS Logical Unit stuck in the offlining state
	with COMSTAR
In-Reply-To: <4E00F844-8507-42E2-8765-AA791B6CCF2B@nasdoma.net>
References: <CABZ-ChSN+espiQjo+5-Y4Squ5ginE11pUt3+i9ipWGxf_e__Ag@mail.gmail.com>
	<1675854F-C90E-4153-85B0-328348C78CC7@omniti.com>
	<CABZ-ChSyNG5wHwDWg3qJTp_70oi6TENSFdf9z4zQ4v4u1uiP4Q@mail.gmail.com>
	<BE343F65-DC98-47AE-A083-D487CCF71F6A@omniti.com>
	<4E00F844-8507-42E2-8765-AA791B6CCF2B@nasdoma.net>
Message-ID: <7097273E-BF5A-4E26-A756-1B6CB215824F@omniti.com>


> On Nov 5, 2014, at 8:01 PM, Sergei Hlupnov <sh at nasdoma.net> wrote:
> 
> Difference between 006 and 013 is that 006 has these while 013 does not.
> 
> dtrace -l |grep dsl_pool_tempreserve_space
>  9094        sdt               zfs        dsl_pool_tempreserve_space set-error
> 44798        fbt               zfs        dsl_pool_tempreserve_space entry
> 44799        fbt               zfs        dsl_pool_tempreserve_space return
> 
> I am referring to commit 3464 because of comments in https://github.com/kdavyd/dtrace/blob/master/txg_monitor.v3.d - this script works in 006 and fails in 013. Not that it presents any issue, just this is the thing that hit me after upgrade to 013.

    4631 zvol_get_stats triggering too many reads

That function was added as part of the above bugfix.

Dan


From mir at miras.org  Thu Nov  6 02:36:49 2014
From: mir at miras.org (Michael Rasmussen)
Date: Thu, 6 Nov 2014 03:36:49 +0100
Subject: [OmniOS-discuss] infiniband
Message-ID: <20141106033649.00255813@sleipner.datanom.net>

Hi all,

I have a dual port HCA where I need both ports on the same subnet. When
configuring on Linux in this situation you are instructed to configure
the interfaces to ignore arp using this setting:
net.ipv4.conf.ib0.arp_ignore=1
net.ipv4.conf.ib1.arp_ignore=1

What is the equivalent on omnios?

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
Pick another fortune cookie.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141106/a405a8ec/attachment-0001.bin>

From danmcd at omniti.com  Thu Nov  6 07:18:19 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Thu, 6 Nov 2014 02:18:19 -0500
Subject: [OmniOS-discuss] OmniOS Bloody November 6 update
Message-ID: <16E34123-92F9-433E-BEDF-79733D76941A@omniti.com>

Back on schedule.  This update mostly is on the illumos side.  In the interim, I updated wget to 1.16 due to a security issue, but that's most of what's changed on the userland side.

For this update, only the relevant packages are being updated, which will cut down on your update times.  New with this release (r151013-20141106):

- omnios-build master branch, revision 42045e1.

- illumos-omnios master branch, revision ed6ad0c.

- The Sun DHCP server no longer exists.  We've had ISC DHCP available since r151012 and it's preceding bloody, so please start using that if you haven't already.

- Various man page fixes.

- ZFS fixes
	- Offlined vdevs now stay offline after a reboot.

- The EOF of the htable and gettable commands.

- Removal of ntfsprogs and parted.

- A NASTY floating-point rounding bug caught and diagnosed by the OmniOS community's Richard PALO has now been fixed.  Thank you Richard!  And remember, YOUR feedback helps us fix stuff, sometimes really nasty stuff.

- A new test in usr/src/test to verify the aforementioned rounding bug.

- Libproc can now understand Linux core files.  This is useful for post-mortem ptools analysis, or mdb.

- Updates to the emlxs(7D) driver to support 16G Fiber Channel.

- Wide-character fixes, also by another OmniOS community member (Lauri "lotheac" Tirkkonen).

- xdr UID & GID cleanups.

There is also new install media, as well as a new kayak .zfs.bz2, in place.

Thanks!
Dan


From johan.kragsterman at capvert.se  Thu Nov  6 10:42:07 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Thu, 6 Nov 2014 11:42:07 +0100
Subject: [OmniOS-discuss] Ang: Re: Ang: Re:  KVM and SMF
In-Reply-To: <90cff68c42e6d90bb590279e49f449ce@blackdot.be>
References: <90cff68c42e6d90bb590279e49f449ce@blackdot.be>,
	<9eb8e5a35d515d2c87e47e031100bfdb@blackdot.be>,
	<OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
	<OF99AFB31A.F9AB8572-ONC1257D87.0070282C-C1257D87.0070282D@inse.com>
Message-ID: <OF70B250A4.F725F1DC-ONC1257D88.003AC9E6-C1257D88.003AC9EA@inse.com>

Hi!

Trying to get the SMF service to work for my KVM VM's here....

I've succeeded to import the xml-file without any errors, but the service goes into maintenance mode after the startd have tried a couple of times.
Looks like this:


root at omni:/mainpool/nfs/Backup/KVM# svcs -p pfsense
STATE          STIME    FMRI
offline*       12:10:31 svc:/kvm/pfsense:default
               12:10:31      711 vmpfsense.sh
               12:10:31      715 qemu-system-x86


root at omni:/mainpool/nfs/Backup/KVM# svcs -l pfsense
fmri         svc:/kvm/pfsense:default
name         KVM-pfsense
enabled      true
state        offline
next_state   online
state_time    6 november 2014 12:10:31 CET
logfile      /var/svc/log/kvm-pfsense:default.log
restarter    svc:/system/svc/restarter:default
contract_id  90 
dependency   require_all/none svc:/milestone/network:default (online)
dependency   require_all/none svc:/system/filesystem/local:default (online)


root at omni:/mainpool/nfs/Backup/KVM# svcs -l pfsense
fmri         svc:/kvm/pfsense:default
name         KVM-pfsense
enabled      true
state        maintenance
next_state   none
state_time    6 november 2014 12:12:31 CET
logfile      /var/svc/log/kvm-pfsense:default.log
restarter    svc:/system/svc/restarter:default
contract_id  
dependency   require_all/none svc:/milestone/network:default (online)
dependency   require_all/none svc:/system/filesystem/local:default (online)
root at omni:/mainpool/nfs/Backup/KVM#


And like this:



root at omni:/mainpool/nfs/Backup/KVM# svcs -vx
svc:/kvm/pfsense:default (KVM-pfsense)
 State: maintenance since  6 november 2014 12:12:31 CET
Reason: Start method died on Killed (9).
   See: http://illumos.org/msg/SMF-8000-KS
   See: /var/svc/log/kvm-pfsense:default.log
Impact: This service is not running.
root at omni:/mainpool/nfs/Backup/KVM# 


I got the log as well, but it is unneccesary long. The important thing is here:



root at omni:/mainpool/nfs/Backup/KVM# tail $(svcs -L pfsense)
  2: 00000000000f0000 - 0000000000100000 = 2
  3: 0000000000100000 - 000000007fffd000 = 1
  4: 000000007fffd000 - 0000000080000000 = 2
  5: 00000000feffc000 - 00000000ff000000 = 2
  6: 00000000fffc0000 - 0000000100000000 = 2
enter handle_19:
  NULL
Booting from Hard Disk...
Booting from 0000:7c00
[ Nov  6 12:12:31 Method or service exit timed out.  Killing contract 90. ]
root at omni:/mainpool/nfs/Backup/KVM# 



This is the only thing in the log that shows any errors: That the "Method or service exit timed out.  Killing contract 90"




So I tried to increase the the "start timeout from 60 to 120 sec, but that made no difference"

I checked the reference to the illumos: http://illumos.org/msg/SMF-8000-KS , but that didn't give me any clue.


So it actually starts the service, but then something times out...what is it that times out...?







My SMF xml file looks like this:



<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
	<service version='0' type='service' name='kvm/pfsense'>
		<create_default_instance enabled='true' />
		<single_instance />
		<dependency restart_on='none' grouping='require_all' type='service' name='network'>
			<service_fmri value='svc:/milestone/network:default' />
		</dependency>
		<dependency restart_on='none' grouping='require_all' type='service' name='filesystem'>
			<service_fmri value='svc:/system/filesystem/local:default' />
		</dependency>
		<exec_method type='method' exec='/usr/bin/vmpfsense.sh' name='start' timeout_seconds='120' />
		<exec_method type='method' exec=':kill' name='stop' timeout_seconds='60' />
		<stability value='Unstable' />
		<template>
			<common_name>
				<loctext xml:lang='C'>KVM-pfsense</loctext>
			</common_name>
		</template>
	</service>
</service_bundle>



I'd be happy if anyone with some more knowledge than I have could give me some clues...!


Regards Johan



-----Jorge Schrauwen <sjorge+ml at blackdot.be> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se>
Fr?n: Jorge Schrauwen <sjorge+ml at blackdot.be>
Datum: 2014-11-05 21:27
?rende: Re: Ang: Re: [OmniOS-discuss] KVM and SMF

I don't know. But from experience svccfg will do what is needed to 
import it.

But I know for a fact service names <> paths. base on I have a few 
services that do not have a corresponding directory.

Regards

Jorge



On 2014-11-05 21:25, Johan Kragsterman wrote:
> -----Jorge Schrauwen <jorge at blackdot.be> skrev: -----
> Till: Johan Kragsterman <johan.kragsterman at capvert.se>
> Fr?n: Jorge Schrauwen <jorge at blackdot.be>
> Datum: 2014-11-05 21:11
> ?rende: Re: [OmniOS-discuss] KVM and SMF
> 
> You can import it using 'svccfg import mysmf.xml'.
> 
> ---
> ~ sjorge
> 
> 
> Hey, Jorge, I thought you were going to answer me...!
> 
> 
> Yeah, I know how I import it, but what I asked about was the naming:
> 
> ?If I call it: service name='kvm/pfsense', where does it go, when I
> import it, since there is no "kvm" directory under /var/svc/manifest/
> or /lib/svc/manifest? Is smf creating a directory for it?
> 
> 
> 
> 
> 
> On 2014-11-05 21:09, Johan Kragsterman wrote:
>> Hi!
>> 
>> 
>> I'm on my way to try to get my KVM VM's under SMF control. Have been
>> looking at various exampel xml-files, and there is something I'm not
>> clear about, that I'd like to ask those who knows how to handle this.
>> 
>> The service name in this example looks like this: <service
>> name='kvm/pfsense'
>> 
>> But there is no directory for "kvm" under /var/svc/manifest/ or
>> /lib/svc/manifest, so where does it go? is svc creating a directory
>> when I import the file?
>> 
>> 
>> I'd used the example on the omnios wiki, just changed the service
>> name. It looks like this in my version:
>> 
>> 
>> <?xml version='1.0'?>
>> <!DOCTYPE service_bundle SYSTEM
>> '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
>> <service_bundle type='manifest' name='export'>
>> <service name='kvm/pfsense' type='service' version='0'>
>> <create_default_instance enabled='true'/>
>> <single_instance/>
>> <dependency name='network'
>> grouping='require_all'
>> restart_on='none'
>> type='service'>
>> <service_fmri value='svc:/milestone/network:default' />
>> </dependency>
>> <dependency name='filesystem'
>> grouping='require_all'
>> restart_on='none'
>> type='service'>
>> <service_fmri value='svc:/system/filesystem/local:default' />
>> </dependency>
>> <exec_method name='start'
>> type='method'
>> exec='/usr/bin/vmpfsense.sh'
>> timeout_seconds='60'/>
>> <exec_method name='stop'
>> type='method'
>> exec='/usr/bin/vmpfsense.sh'
>> timeout_seconds='60'/>
>> <exec_method name='stop'
>> type='method'
>> exec=':kill'
>> timeout_seconds='60'/>
>> <stability value='Unstable'/>
>> <template>
>> <common_name>
>> <loctext xml:lang='C'>KVM-pfsense</loctext>
>> <common_name>
>> <template>
>> </service>
>> </service_bundle>
>> 
>> 
>> 
>> 
>> What do you think, would that do?
>> 
>> 
>> 
>> 
>> Best regards from/Med v?nliga h?lsningar fr?n
>> 
>> Johan Kragsterman
>> 
>> Capvert
>> 
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss




From sjorge+ml at blackdot.be  Thu Nov  6 11:35:35 2014
From: sjorge+ml at blackdot.be (Jorge Schrauwen)
Date: Thu, 06 Nov 2014 12:35:35 +0100
Subject: [OmniOS-discuss] Ang: Re: Ang: Re:  KVM and SMF
In-Reply-To: <OF70B250A4.F725F1DC-ONC1257D88.003AC9E6-C1257D88.003AC9EA@inse.com>
References: <90cff68c42e6d90bb590279e49f449ce@blackdot.be>, 
	<9eb8e5a35d515d2c87e47e031100bfdb@blackdot.be>,
	<OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
	<OF99AFB31A.F9AB8572-ONC1257D87.0070282C-C1257D87.0070282D@inse.com>
	<OF70B250A4.F725F1DC-ONC1257D88.003AC9E6-C1257D88.003AC9EA@inse.com>
Message-ID: <6f8dcc106245a0be45eec007f134d34c@blackdot.be>

Hi,

It could be you are using a wrong type of duration for your service.


     <property_group name='startd' type='framework'>
       <propval name='duration' type='astring' value='transient'/>
       <propval name='ignore_error' type='astring' value='core,signal'/>
     </property_group>

I use transient for my script that start qemu. By default if the process 
does not daemonize it is killed after... I think 90 seconds by default.

Regards

Jorge


On 2014-11-06 11:42, Johan Kragsterman wrote:
> Hi!
> 
> Trying to get the SMF service to work for my KVM VM's here....
> 
> I've succeeded to import the xml-file without any errors, but the
> service goes into maintenance mode after the startd have tried a
> couple of times.
> Looks like this:
> 
> 
> root at omni:/mainpool/nfs/Backup/KVM# svcs -p pfsense
> STATE          STIME    FMRI
> offline*       12:10:31 svc:/kvm/pfsense:default
>                12:10:31      711 vmpfsense.sh
>                12:10:31      715 qemu-system-x86
> 
> 
> root at omni:/mainpool/nfs/Backup/KVM# svcs -l pfsense
> fmri         svc:/kvm/pfsense:default
> name         KVM-pfsense
> enabled      true
> state        offline
> next_state   online
> state_time    6 november 2014 12:10:31 CET
> logfile      /var/svc/log/kvm-pfsense:default.log
> restarter    svc:/system/svc/restarter:default
> contract_id  90
> dependency   require_all/none svc:/milestone/network:default (online)
> dependency   require_all/none svc:/system/filesystem/local:default 
> (online)
> 
> 
> root at omni:/mainpool/nfs/Backup/KVM# svcs -l pfsense
> fmri         svc:/kvm/pfsense:default
> name         KVM-pfsense
> enabled      true
> state        maintenance
> next_state   none
> state_time    6 november 2014 12:12:31 CET
> logfile      /var/svc/log/kvm-pfsense:default.log
> restarter    svc:/system/svc/restarter:default
> contract_id
> dependency   require_all/none svc:/milestone/network:default (online)
> dependency   require_all/none svc:/system/filesystem/local:default 
> (online)
> root at omni:/mainpool/nfs/Backup/KVM#
> 
> 
> And like this:
> 
> 
> 
> root at omni:/mainpool/nfs/Backup/KVM# svcs -vx
> svc:/kvm/pfsense:default (KVM-pfsense)
>  State: maintenance since  6 november 2014 12:12:31 CET
> Reason: Start method died on Killed (9).
>    See: http://illumos.org/msg/SMF-8000-KS
>    See: /var/svc/log/kvm-pfsense:default.log
> Impact: This service is not running.
> root at omni:/mainpool/nfs/Backup/KVM#
> 
> 
> I got the log as well, but it is unneccesary long. The important thing 
> is here:
> 
> 
> 
> root at omni:/mainpool/nfs/Backup/KVM# tail $(svcs -L pfsense)
>   2: 00000000000f0000 - 0000000000100000 = 2
>   3: 0000000000100000 - 000000007fffd000 = 1
>   4: 000000007fffd000 - 0000000080000000 = 2
>   5: 00000000feffc000 - 00000000ff000000 = 2
>   6: 00000000fffc0000 - 0000000100000000 = 2
> enter handle_19:
>   NULL
> Booting from Hard Disk...
> Booting from 0000:7c00
> [ Nov  6 12:12:31 Method or service exit timed out.  Killing contract 
> 90. ]
> root at omni:/mainpool/nfs/Backup/KVM#
> 
> 
> 
> This is the only thing in the log that shows any errors: That the
> "Method or service exit timed out.  Killing contract 90"
> 
> 
> 
> 
> So I tried to increase the the "start timeout from 60 to 120 sec, but
> that made no difference"
> 
> I checked the reference to the illumos:
> http://illumos.org/msg/SMF-8000-KS , but that didn't give me any clue.
> 
> 
> So it actually starts the service, but then something times out...what
> is it that times out...?
> 
> 
> 
> 
> 
> 
> 
> My SMF xml file looks like this:
> 
> 
> 
> <?xml version='1.0'?>
> <!DOCTYPE service_bundle SYSTEM 
> '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
> <service_bundle type='manifest' name='export'>
> 	<service version='0' type='service' name='kvm/pfsense'>
> 		<create_default_instance enabled='true' />
> 		<single_instance />
> 		<dependency restart_on='none' grouping='require_all' type='service'
> name='network'>
> 			<service_fmri value='svc:/milestone/network:default' />
> 		</dependency>
> 		<dependency restart_on='none' grouping='require_all' type='service'
> name='filesystem'>
> 			<service_fmri value='svc:/system/filesystem/local:default' />
> 		</dependency>
> 		<exec_method type='method' exec='/usr/bin/vmpfsense.sh' name='start'
> timeout_seconds='120' />
> 		<exec_method type='method' exec=':kill' name='stop' 
> timeout_seconds='60' />
> 		<stability value='Unstable' />
> 		<template>
> 			<common_name>
> 				<loctext xml:lang='C'>KVM-pfsense</loctext>
> 			</common_name>
> 		</template>
> 	</service>
> </service_bundle>
> 
> 
> 
> I'd be happy if anyone with some more knowledge than I have could give
> me some clues...!
> 
> 
> Regards Johan
> 
> 
> 
> -----Jorge Schrauwen <sjorge+ml at blackdot.be> skrev: -----
> Till: Johan Kragsterman <johan.kragsterman at capvert.se>
> Fr?n: Jorge Schrauwen <sjorge+ml at blackdot.be>
> Datum: 2014-11-05 21:27
> ?rende: Re: Ang: Re: [OmniOS-discuss] KVM and SMF
> 
> I don't know. But from experience svccfg will do what is needed to
> import it.
> 
> But I know for a fact service names <> paths. base on I have a few
> services that do not have a corresponding directory.
> 
> Regards
> 
> Jorge
> 
> 
> 
> On 2014-11-05 21:25, Johan Kragsterman wrote:
>> -----Jorge Schrauwen <jorge at blackdot.be> skrev: -----
>> Till: Johan Kragsterman <johan.kragsterman at capvert.se>
>> Fr?n: Jorge Schrauwen <jorge at blackdot.be>
>> Datum: 2014-11-05 21:11
>> ?rende: Re: [OmniOS-discuss] KVM and SMF
>> 
>> You can import it using 'svccfg import mysmf.xml'.
>> 
>> ---
>> ~ sjorge
>> 
>> 
>> Hey, Jorge, I thought you were going to answer me...!
>> 
>> 
>> Yeah, I know how I import it, but what I asked about was the naming:
>> 
>> ?If I call it: service name='kvm/pfsense', where does it go, when I
>> import it, since there is no "kvm" directory under /var/svc/manifest/
>> or /lib/svc/manifest? Is smf creating a directory for it?
>> 
>> 
>> 
>> 
>> 
>> On 2014-11-05 21:09, Johan Kragsterman wrote:
>>> Hi!
>>> 
>>> 
>>> I'm on my way to try to get my KVM VM's under SMF control. Have been
>>> looking at various exampel xml-files, and there is something I'm not
>>> clear about, that I'd like to ask those who knows how to handle this.
>>> 
>>> The service name in this example looks like this: <service
>>> name='kvm/pfsense'
>>> 
>>> But there is no directory for "kvm" under /var/svc/manifest/ or
>>> /lib/svc/manifest, so where does it go? is svc creating a directory
>>> when I import the file?
>>> 
>>> 
>>> I'd used the example on the omnios wiki, just changed the service
>>> name. It looks like this in my version:
>>> 
>>> 
>>> <?xml version='1.0'?>
>>> <!DOCTYPE service_bundle SYSTEM
>>> '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
>>> <service_bundle type='manifest' name='export'>
>>> <service name='kvm/pfsense' type='service' version='0'>
>>> <create_default_instance enabled='true'/>
>>> <single_instance/>
>>> <dependency name='network'
>>> grouping='require_all'
>>> restart_on='none'
>>> type='service'>
>>> <service_fmri value='svc:/milestone/network:default' />
>>> </dependency>
>>> <dependency name='filesystem'
>>> grouping='require_all'
>>> restart_on='none'
>>> type='service'>
>>> <service_fmri value='svc:/system/filesystem/local:default' />
>>> </dependency>
>>> <exec_method name='start'
>>> type='method'
>>> exec='/usr/bin/vmpfsense.sh'
>>> timeout_seconds='60'/>
>>> <exec_method name='stop'
>>> type='method'
>>> exec='/usr/bin/vmpfsense.sh'
>>> timeout_seconds='60'/>
>>> <exec_method name='stop'
>>> type='method'
>>> exec=':kill'
>>> timeout_seconds='60'/>
>>> <stability value='Unstable'/>
>>> <template>
>>> <common_name>
>>> <loctext xml:lang='C'>KVM-pfsense</loctext>
>>> <common_name>
>>> <template>
>>> </service>
>>> </service_bundle>
>>> 
>>> 
>>> 
>>> 
>>> What do you think, would that do?
>>> 
>>> 
>>> 
>>> 
>>> Best regards from/Med v?nliga h?lsningar fr?n
>>> 
>>> Johan Kragsterman
>>> 
>>> Capvert
>>> 
>>> _______________________________________________
>>> OmniOS-discuss mailing list
>>> OmniOS-discuss at lists.omniti.com
>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss

From lotheac at iki.fi  Thu Nov  6 11:49:10 2014
From: lotheac at iki.fi (Lauri Tirkkonen)
Date: Thu, 6 Nov 2014 13:49:10 +0200
Subject: [OmniOS-discuss] Ang: Re: Ang: Re:  KVM and SMF
In-Reply-To: <6f8dcc106245a0be45eec007f134d34c@blackdot.be>
References: <90cff68c42e6d90bb590279e49f449ce@blackdot.be>
	<9eb8e5a35d515d2c87e47e031100bfdb@blackdot.be>
	<OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
	<OF99AFB31A.F9AB8572-ONC1257D87.0070282C-C1257D87.0070282D@inse.com>
	<OF70B250A4.F725F1DC-ONC1257D88.003AC9E6-C1257D88.003AC9EA@inse.com>
	<6f8dcc106245a0be45eec007f134d34c@blackdot.be>
Message-ID: <20141106114910.GA8704@gutsman.lotheac.fi>

On Thu, Nov 06 2014 12:35:35 +0100, Jorge Schrauwen wrote:
> I use transient for my script that start qemu. By default if the process
> does not daemonize it is killed after... I think 90 seconds by default.

svc.startd(1M) manual mentions:

> If a contract or transient service does not return from  its
> start  method before its defined timeout elapses, svc.startd
> sends a SIGKILL to the method, and returns  the  service  to
> the offline state.

and

> Defining a service as transient means that  svc.startd  does
> not  track  processes  for that service. Thus, the potential
> faults described for contract model services  are  not  con-
> sidered failures for transient services.

Because of these I would use 'contract'. If your start method does not
daemonize, you can just add a '&' at the end of start/exec.

-- 
Lauri Tirkkonen | +358 50 5341376 | lotheac @ IRCnet

From johan.kragsterman at capvert.se  Thu Nov  6 12:03:36 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Thu, 6 Nov 2014 13:03:36 +0100
Subject: [OmniOS-discuss] Ang: Re: Ang: Re: Ang: Re:  KVM and SMF
In-Reply-To: <6f8dcc106245a0be45eec007f134d34c@blackdot.be>
References: <6f8dcc106245a0be45eec007f134d34c@blackdot.be>,
	<90cff68c42e6d90bb590279e49f449ce@blackdot.be>,
	<9eb8e5a35d515d2c87e47e031100bfdb@blackdot.be>,
	<OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
	<OF99AFB31A.F9AB8572-ONC1257D87.0070282C-C1257D87.0070282D@inse.com>
	<OF70B250A4.F725F1DC-ONC1257D88.003AC9E6-C1257D88.003AC9EA@inse.com>
Message-ID: <OF31F495BE.3D21D6BA-ONC1257D88.004014A9-C1257D88.00423F6A@inse.com>

Thanks again, Jorge!

I tried that, but with no luck....

Like this:


root at omni:/mainpool/nfs/Backup/KVM# svcs -l pfsense
fmri         svc:/kvm/pfsense:default
name         KVM-pfsense
enabled      true
state        maintenance
next_state   none
state_time    6 november 2014 13:51:28 CET
logfile      /var/svc/log/kvm-pfsense:default.log
restarter    svc:/system/svc/restarter:default
dependency   require_all/none svc:/milestone/network:default (online)
dependency   require_all/none svc:/system/filesystem/local:default (online)
root at omni:/mainpool/nfs/Backup/KVM# svcs -vx
svc:/kvm/pfsense:default (KVM-pfsense)
 State: maintenance since  6 november 2014 13:51:28 CET
Reason: Start method failed repeatedly, last died on Killed (9).
   See: http://illumos.org/msg/SMF-8000-KS
   See: /var/svc/log/kvm-pfsense:default.log
Impact: This service is not running.
root at omni:/mainpool/nfs/Backup/KVM# 

Regards Johan


-----Jorge Schrauwen <sjorge+ml at blackdot.be> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se>
Fr?n: Jorge Schrauwen <sjorge+ml at blackdot.be>
Datum: 2014-11-06 12:36
Kopia: omnios-discuss at lists.omniti.com
?rende: Re: Ang: Re: Ang: Re: [OmniOS-discuss] KVM and SMF

Hi,

It could be you are using a wrong type of duration for your service.


?? ? <property_group name='startd' type='framework'>
?? ? ? <propval name='duration' type='astring' value='transient'/>
?? ? ? <propval name='ignore_error' type='astring' value='core,signal'/>
?? ? </property_group>

I use transient for my script that start qemu. By default if the process 
does not daemonize it is killed after... I think 90 seconds by default.

Regards

Jorge


On 2014-11-06 11:42, Johan Kragsterman wrote:
> Hi!
> 
> Trying to get the SMF service to work for my KVM VM's here....
> 
> I've succeeded to import the xml-file without any errors, but the
> service goes into maintenance mode after the startd have tried a
> couple of times.
> Looks like this:
> 
> 
> root at omni:/mainpool/nfs/Backup/KVM# svcs -p pfsense
> STATE ? ? ? ? ?STIME ? ?FMRI
> offline* ? ? ? 12:10:31 svc:/kvm/pfsense:default
> ? ? ? ? ? ? ? ?12:10:31 ? ? ?711 vmpfsense.sh
> ? ? ? ? ? ? ? ?12:10:31 ? ? ?715 qemu-system-x86
> 
> 
> root at omni:/mainpool/nfs/Backup/KVM# svcs -l pfsense
> fmri ? ? ? ? svc:/kvm/pfsense:default
> name ? ? ? ? KVM-pfsense
> enabled ? ? ?true
> state ? ? ? ?offline
> next_state ? online
> state_time ? ?6 november 2014 12:10:31 CET
> logfile ? ? ?/var/svc/log/kvm-pfsense:default.log
> restarter ? ?svc:/system/svc/restarter:default
> contract_id ?90
> dependency ? require_all/none svc:/milestone/network:default (online)
> dependency ? require_all/none svc:/system/filesystem/local:default 
> (online)
> 
> 
> root at omni:/mainpool/nfs/Backup/KVM# svcs -l pfsense
> fmri ? ? ? ? svc:/kvm/pfsense:default
> name ? ? ? ? KVM-pfsense
> enabled ? ? ?true
> state ? ? ? ?maintenance
> next_state ? none
> state_time ? ?6 november 2014 12:12:31 CET
> logfile ? ? ?/var/svc/log/kvm-pfsense:default.log
> restarter ? ?svc:/system/svc/restarter:default
> contract_id
> dependency ? require_all/none svc:/milestone/network:default (online)
> dependency ? require_all/none svc:/system/filesystem/local:default 
> (online)
> root at omni:/mainpool/nfs/Backup/KVM#
> 
> 
> And like this:
> 
> 
> 
> root at omni:/mainpool/nfs/Backup/KVM# svcs -vx
> svc:/kvm/pfsense:default (KVM-pfsense)
> ?State: maintenance since ?6 november 2014 12:12:31 CET
> Reason: Start method died on Killed (9).
> ? ?See: http://illumos.org/msg/SMF-8000-KS
> ? ?See: /var/svc/log/kvm-pfsense:default.log
> Impact: This service is not running.
> root at omni:/mainpool/nfs/Backup/KVM#
> 
> 
> I got the log as well, but it is unneccesary long. The important thing 
> is here:
> 
> 
> 
> root at omni:/mainpool/nfs/Backup/KVM# tail $(svcs -L pfsense)
> ? 2: 00000000000f0000 - 0000000000100000 = 2
> ? 3: 0000000000100000 - 000000007fffd000 = 1
> ? 4: 000000007fffd000 - 0000000080000000 = 2
> ? 5: 00000000feffc000 - 00000000ff000000 = 2
> ? 6: 00000000fffc0000 - 0000000100000000 = 2
> enter handle_19:
> ? NULL
> Booting from Hard Disk...
> Booting from 0000:7c00
> [ Nov ?6 12:12:31 Method or service exit timed out. ?Killing contract 
> 90. ]
> root at omni:/mainpool/nfs/Backup/KVM#
> 
> 
> 
> This is the only thing in the log that shows any errors: That the
> "Method or service exit timed out. ?Killing contract 90"
> 
> 
> 
> 
> So I tried to increase the the "start timeout from 60 to 120 sec, but
> that made no difference"
> 
> I checked the reference to the illumos:
> http://illumos.org/msg/SMF-8000-KS?, but that didn't give me any clue.
> 
> 
> So it actually starts the service, but then something times out...what
> is it that times out...?
> 
> 
> 
> 
> 
> 
> 
> My SMF xml file looks like this:
> 
> 
> 
> <?xml version='1.0'?>
> <!DOCTYPE service_bundle SYSTEM 
> '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
> <service_bundle type='manifest' name='export'>
> <service version='0' type='service' name='kvm/pfsense'>
> <create_default_instance enabled='true' />
> <single_instance />
> <dependency restart_on='none' grouping='require_all' type='service'
> name='network'>
> <service_fmri value='svc:/milestone/network:default' />
> </dependency>
> <dependency restart_on='none' grouping='require_all' type='service'
> name='filesystem'>
> <service_fmri value='svc:/system/filesystem/local:default' />
> </dependency>
> <exec_method type='method' exec='/usr/bin/vmpfsense.sh' name='start'
> timeout_seconds='120' />
> <exec_method type='method' exec=':kill' name='stop' 
> timeout_seconds='60' />
> <stability value='Unstable' />
> <template>
> <common_name>
> <loctext xml:lang='C'>KVM-pfsense</loctext>
> </common_name>
> </template>
> </service>
> </service_bundle>
> 
> 
> 
> I'd be happy if anyone with some more knowledge than I have could give
> me some clues...!
> 
> 
> Regards Johan
> 
> 
> 
> -----Jorge Schrauwen <sjorge+ml at blackdot.be> skrev: -----
> Till: Johan Kragsterman <johan.kragsterman at capvert.se>
> Fr?n: Jorge Schrauwen <sjorge+ml at blackdot.be>
> Datum: 2014-11-05 21:27
> ?rende: Re: Ang: Re: [OmniOS-discuss] KVM and SMF
> 
> I don't know. But from experience svccfg will do what is needed to
> import it.
> 
> But I know for a fact service names <> paths. base on I have a few
> services that do not have a corresponding directory.
> 
> Regards
> 
> Jorge
> 
> 
> 
> On 2014-11-05 21:25, Johan Kragsterman wrote:
>> -----Jorge Schrauwen <jorge at blackdot.be> skrev: -----
>> Till: Johan Kragsterman <johan.kragsterman at capvert.se>
>> Fr?n: Jorge Schrauwen <jorge at blackdot.be>
>> Datum: 2014-11-05 21:11
>> ?rende: Re: [OmniOS-discuss] KVM and SMF
>> 
>> You can import it using 'svccfg import mysmf.xml'.
>> 
>> ---
>> ~ sjorge
>> 
>> 
>> Hey, Jorge, I thought you were going to answer me...!
>> 
>> 
>> Yeah, I know how I import it, but what I asked about was the naming:
>> 
>> ?If I call it: service name='kvm/pfsense', where does it go, when I
>> import it, since there is no "kvm" directory under /var/svc/manifest/
>> or /lib/svc/manifest? Is smf creating a directory for it?
>> 
>> 
>> 
>> 
>> 
>> On 2014-11-05 21:09, Johan Kragsterman wrote:
>>> Hi!
>>> 
>>> 
>>> I'm on my way to try to get my KVM VM's under SMF control. Have been
>>> looking at various exampel xml-files, and there is something I'm not
>>> clear about, that I'd like to ask those who knows how to handle this.
>>> 
>>> The service name in this example looks like this: <service
>>> name='kvm/pfsense'
>>> 
>>> But there is no directory for "kvm" under /var/svc/manifest/ or
>>> /lib/svc/manifest, so where does it go? is svc creating a directory
>>> when I import the file?
>>> 
>>> 
>>> I'd used the example on the omnios wiki, just changed the service
>>> name. It looks like this in my version:
>>> 
>>> 
>>> <?xml version='1.0'?>
>>> <!DOCTYPE service_bundle SYSTEM
>>> '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
>>> <service_bundle type='manifest' name='export'>
>>> <service name='kvm/pfsense' type='service' version='0'>
>>> <create_default_instance enabled='true'/>
>>> <single_instance/>
>>> <dependency name='network'
>>> grouping='require_all'
>>> restart_on='none'
>>> type='service'>
>>> <service_fmri value='svc:/milestone/network:default' />
>>> </dependency>
>>> <dependency name='filesystem'
>>> grouping='require_all'
>>> restart_on='none'
>>> type='service'>
>>> <service_fmri value='svc:/system/filesystem/local:default' />
>>> </dependency>
>>> <exec_method name='start'
>>> type='method'
>>> exec='/usr/bin/vmpfsense.sh'
>>> timeout_seconds='60'/>
>>> <exec_method name='stop'
>>> type='method'
>>> exec='/usr/bin/vmpfsense.sh'
>>> timeout_seconds='60'/>
>>> <exec_method name='stop'
>>> type='method'
>>> exec=':kill'
>>> timeout_seconds='60'/>
>>> <stability value='Unstable'/>
>>> <template>
>>> <common_name>
>>> <loctext xml:lang='C'>KVM-pfsense</loctext>
>>> <common_name>
>>> <template>
>>> </service>
>>> </service_bundle>
>>> 
>>> 
>>> 
>>> 
>>> What do you think, would that do?
>>> 
>>> 
>>> 
>>> 
>>> Best regards from/Med v?nliga h?lsningar fr?n
>>> 
>>> Johan Kragsterman
>>> 
>>> Capvert
>>> 
>>> _______________________________________________
>>> OmniOS-discuss mailing list
>>> OmniOS-discuss at lists.omniti.com
>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss




From johan.kragsterman at capvert.se  Thu Nov  6 12:13:19 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Thu, 6 Nov 2014 13:13:19 +0100
Subject: [OmniOS-discuss] Ang: Re:  Ang: Re: Ang: Re:  KVM and SMF
In-Reply-To: <20141106114910.GA8704@gutsman.lotheac.fi>
References: <20141106114910.GA8704@gutsman.lotheac.fi>,
	<90cff68c42e6d90bb590279e49f449ce@blackdot.be>
	<9eb8e5a35d515d2c87e47e031100bfdb@blackdot.be>
	<OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
	<OF99AFB31A.F9AB8572-ONC1257D87.0070282C-C1257D87.0070282D@inse.com>
	<OF70B250A4.F725F1DC-ONC1257D88.003AC9E6-C1257D88.003AC9EA@inse.com>
	<6f8dcc106245a0be45eec007f134d34c@blackdot.be>
Message-ID: <OF7F864096.A0793304-ONC1257D88.0043236E-C1257D88.00432371@inse.com>

Sorry, forgot the list...



Hej, Lauri!


Thanks for your input!

Do you mean that I should just change the "transient" to "contract"?

And the "&" at the end of start/exec like this:

<exec_method type='method' exec='/usr/bin/vmpfsense.sh' name='start' timeout_seconds='120' & />

How about the spaces there in the end...?


Regards Johan





-----Lauri Tirkkonen <lotheac at iki.fi> skrev: -----
Till: Jorge Schrauwen <sjorge+ml at blackdot.be>
Fr?n: Lauri Tirkkonen <lotheac at iki.fi>
Datum: 2014-11-06 12:49
Kopia: Johan Kragsterman <johan.kragsterman at capvert.se>, omnios-discuss at lists.omniti.com
?rende: Re: [OmniOS-discuss] Ang: Re: Ang: Re: KVM and SMF

On Thu, Nov 06 2014 12:35:35 +0100, Jorge Schrauwen wrote:
> I use transient for my script that start qemu. By default if the process
> does not daemonize it is killed after... I think 90 seconds by default.

svc.startd(1M) manual mentions:

> If a contract or transient service does not return from ?its
> start ?method before its defined timeout elapses, svc.startd
> sends a SIGKILL to the method, and returns ?the ?service ?to
> the offline state.

and

> Defining a service as transient means that ?svc.startd ?does
> not ?track ?processes ?for that service. Thus, the potential
> faults described for contract model services ?are ?not ?con-
> sidered failures for transient services.

Because of these I would use 'contract'. If your start method does not
daemonize, you can just add a '&' at the end of start/exec.

-- 
Lauri Tirkkonen | +358 50 5341376 | lotheac @ IRCnet




From lotheac at iki.fi  Thu Nov  6 12:34:11 2014
From: lotheac at iki.fi (Lauri Tirkkonen)
Date: Thu, 6 Nov 2014 14:34:11 +0200
Subject: [OmniOS-discuss] Ang: Re:  Ang: Re: Ang: Re:  KVM and SMF
In-Reply-To: <OF7F864096.A0793304-ONC1257D88.0043236E-C1257D88.00432371@inse.com>
References: <20141106114910.GA8704@gutsman.lotheac.fi>
	<90cff68c42e6d90bb590279e49f449ce@blackdot.be>
	<9eb8e5a35d515d2c87e47e031100bfdb@blackdot.be>
	<OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
	<OF99AFB31A.F9AB8572-ONC1257D87.0070282C-C1257D87.0070282D@inse.com>
	<OF70B250A4.F725F1DC-ONC1257D88.003AC9E6-C1257D88.003AC9EA@inse.com>
	<6f8dcc106245a0be45eec007f134d34c@blackdot.be>
	<OF7F864096.A0793304-ONC1257D88.0043236E-C1257D88.00432371@inse.com>
Message-ID: <20141106123411.GA2030@gutsman.lotheac.fi>

On Thu, Nov 06 2014 13:13:19 +0100, Johan Kragsterman wrote:
> Do you mean that I should just change the "transient" to "contract"?

Yes, or you can leave the definition out, since 'contract' is the
default.

> And the "&" at the end of start/exec like this:
> 
> <exec_method type='method' exec='/usr/bin/vmpfsense.sh' name='start'
> timeout_seconds='120' & />

No. <exec_method exec="/usr/bin/vmpfsense.sh &" ... /> or so in XML. You
can also set the properties via svccfg without having to much around
with XML.

-- 
Lauri Tirkkonen | +358 50 5341376 | lotheac @ IRCnet

From johan.kragsterman at capvert.se  Thu Nov  6 13:27:20 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Thu, 6 Nov 2014 14:27:20 +0100
Subject: [OmniOS-discuss] Ang: Re: Ang: Re: Ang: Re: Ang: Re: KVM and SMF
In-Reply-To: <20141106123411.GA2030@gutsman.lotheac.fi>
References: <20141106123411.GA2030@gutsman.lotheac.fi>,
	<20141106114910.GA8704@gutsman.lotheac.fi>
	<90cff68c42e6d90bb590279e49f449ce@blackdot.be>
	<9eb8e5a35d515d2c87e47e031100bfdb@blackdot.be>
	<OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
	<OF99AFB31A.F9AB8572-ONC1257D87.0070282C-C1257D87.0070282D@inse.com>
	<OF70B250A4.F725F1DC-ONC1257D88.003AC9E6-C1257D88.003AC9EA@inse.com>
	<6f8dcc106245a0be45eec007f134d34c@blackdot.be>
	<OF7F864096.A0793304-ONC1257D88.0043236E-C1257D88.00432371@inse.com>
Message-ID: <OF4B35D7F2.7C79B3A2-ONC1257D88.0049C700-C1257D88.0049E9DA@inse.com>

Hi!

-----Lauri Tirkkonen <lotheac at iki.fi> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se>
Fr?n: Lauri Tirkkonen <lotheac at iki.fi>
Datum: 2014-11-06 13:34
Kopia: Jorge Schrauwen <sjorge+ml at blackdot.be>, omnios-discuss at lists.omniti.com
?rende: Re: Ang: Re: [OmniOS-discuss] Ang: Re: Ang: Re: KVM and SMF

On Thu, Nov 06 2014 13:13:19 +0100, Johan Kragsterman wrote:
> Do you mean that I should just change the "transient" to "contract"?

Yes, or you can leave the definition out, since 'contract' is the
default.

> And the "&" at the end of start/exec like this:
> 
> <exec_method type='method' exec='/usr/bin/vmpfsense.sh' name='start'
> timeout_seconds='120' & />

No. <exec_method exec="/usr/bin/vmpfsense.sh &" ... /> or so in XML. You
can also set the properties via svccfg without having to much around
with XML.




That is not parsable...

root at omni:/mainpool/nfs/Backup/KVM# svccfg import pfkvm.xml
pfkvm.xml:13: parser error : xmlParseEntityRef: no name
                <exec_method type='method' exec='/usr/bin/vmpfsense.sh &' name='start' timeout
                                                                        ^
svccfg: couldn't parse document
root at omni:/mainpool/nfs/Backup/KVM#


Regards Johan


-- 
Lauri Tirkkonen | +358 50 5341376 | lotheac @ IRCnet




From lotheac at iki.fi  Thu Nov  6 13:30:50 2014
From: lotheac at iki.fi (Lauri Tirkkonen)
Date: Thu, 6 Nov 2014 15:30:50 +0200
Subject: [OmniOS-discuss] Ang: Re: Ang: Re: Ang: Re: Ang: Re: KVM and SMF
In-Reply-To: <OF4B35D7F2.7C79B3A2-ONC1257D88.0049C700-C1257D88.0049E9DA@inse.com>
References: <20141106123411.GA2030@gutsman.lotheac.fi>
	<20141106114910.GA8704@gutsman.lotheac.fi>
	<90cff68c42e6d90bb590279e49f449ce@blackdot.be>
	<9eb8e5a35d515d2c87e47e031100bfdb@blackdot.be>
	<OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
	<OF99AFB31A.F9AB8572-ONC1257D87.0070282C-C1257D87.0070282D@inse.com>
	<OF70B250A4.F725F1DC-ONC1257D88.003AC9E6-C1257D88.003AC9EA@inse.com>
	<6f8dcc106245a0be45eec007f134d34c@blackdot.be>
	<OF7F864096.A0793304-ONC1257D88.0043236E-C1257D88.00432371@inse.com>
	<OF4B35D7F2.7C79B3A2-ONC1257D88.0049C700-C1257D88.0049E9DA@inse.com>
Message-ID: <20141106133050.GB2030@gutsman.lotheac.fi>

On Thu, Nov 06 2014 14:27:20 +0100, Johan Kragsterman wrote:
> That is not parsable...
> 
> root at omni:/mainpool/nfs/Backup/KVM# svccfg import pfkvm.xml
> pfkvm.xml:13: parser error : xmlParseEntityRef: no name
>                 <exec_method type='method' exec='/usr/bin/vmpfsense.sh &' name='start' timeout
>                                                                         ^
> svccfg: couldn't parse document
> root at omni:/mainpool/nfs/Backup/KVM#

Then I guess you need to escape it ("&amp;"). It's easier to set with
svccfg anyways.

-- 
Lauri Tirkkonen | +358 50 5341376 | lotheac @ IRCnet

From johan.kragsterman at capvert.se  Thu Nov  6 14:21:34 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Thu, 6 Nov 2014 15:21:34 +0100
Subject: [OmniOS-discuss] Ang: Re: Ang: Re: Ang: Re: Ang: Re: Ang: Re: KVM
	and SMF
In-Reply-To: <20141106133050.GB2030@gutsman.lotheac.fi>
References: <20141106133050.GB2030@gutsman.lotheac.fi>,
	<20141106123411.GA2030@gutsman.lotheac.fi>
	<20141106114910.GA8704@gutsman.lotheac.fi>
	<90cff68c42e6d90bb590279e49f449ce@blackdot.be>
	<9eb8e5a35d515d2c87e47e031100bfdb@blackdot.be>
	<OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
	<OF99AFB31A.F9AB8572-ONC1257D87.0070282C-C1257D87.0070282D@inse.com>
	<OF70B250A4.F725F1DC-ONC1257D88.003AC9E6-C1257D88.003AC9EA@inse.com>
	<6f8dcc106245a0be45eec007f134d34c@blackdot.be>
	<OF7F864096.A0793304-ONC1257D88.0043236E-C1257D88.00432371@inse.com>
	<OF4B35D7F2.7C79B3A2-ONC1257D88.0049C700-C1257D88.0049E9DA@inse.com>
Message-ID: <OFB992D471.2043BE0C-ONC1257D88.004D53D3-C1257D88.004EE12C@inse.com>

Hi!

-----Lauri Tirkkonen <lotheac at iki.fi> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se>
Fr?n: Lauri Tirkkonen <lotheac at iki.fi>
Datum: 2014-11-06 14:31
Kopia: Jorge Schrauwen <sjorge+ml at blackdot.be>, omnios-discuss at lists.omniti.com
?rende: Re: Ang: Re: Ang: Re: [OmniOS-discuss] Ang: Re: Ang: Re: KVM and SMF

On Thu, Nov 06 2014 14:27:20 +0100, Johan Kragsterman wrote:
> That is not parsable...
> 
> root at omni:/mainpool/nfs/Backup/KVM# svccfg import pfkvm.xml
> pfkvm.xml:13: parser error : xmlParseEntityRef: no name
> ? ? ? ? ? ? ? ? <exec_method type='method' exec='/usr/bin/vmpfsense.sh &' name='start' timeout
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^
> svccfg: couldn't parse document
> root at omni:/mainpool/nfs/Backup/KVM#

Then I guess you need to escape it ("&amp;"). It's easier to set with
svccfg anyways.






That did the trick! Thanks! I bet it is many others that have been struggling with this as well...


And as well:

I've been looking at svccfg(1M) but can't say I can figure out what to do to get the service to demonize...

Would be nice to learn...


Regards Johan




-- 
Lauri Tirkkonen | +358 50 5341376 | lotheac @ IRCnet




From johan.kragsterman at capvert.se  Thu Nov  6 15:07:10 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Thu, 6 Nov 2014 16:07:10 +0100
Subject: [OmniOS-discuss] Ang: Ang: Re: Ang: Re: Ang: Re: Ang: Re: Ang: Re:
 KVM	and SMF
In-Reply-To: <OFB992D471.2043BE0C-ONC1257D88.004D53D3-C1257D88.004EE12C@inse.com>
References: <OFB992D471.2043BE0C-ONC1257D88.004D53D3-C1257D88.004EE12C@inse.com>,
	<20141106133050.GB2030@gutsman.lotheac.fi>,
	<20141106123411.GA2030@gutsman.lotheac.fi>
	<20141106114910.GA8704@gutsman.lotheac.fi>	<90cff68c42e6d90bb590279e49f449ce@blackdot.be>
	<9eb8e5a35d515d2c87e47e031100bfdb@blackdot.be>	<OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>
	<OF99AFB31A.F9AB8572-ONC1257D87.0070282C-C1257D87.0070282D@inse.com>	<OF70B250A4.F725F1DC-ONC1257D88.003AC9E6-C1257D88.003AC9EA@inse.com>
	<6f8dcc106245a0be45eec007f134d34c@blackdot.be>	<OF7F864096.A0793304-ONC1257D88.0043236E-C1257D88.00432371@inse.com>
	<OF4B35D7F2.7C79B3A2-ONC1257D88.0049C700-C1257D88.0049E9DA@inse.com>
Message-ID: <OF63768A27.D6713329-ONC1257D88.005222DC-C1257D88.00530E10@inse.com>

Here I am Again as a follow up...



-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev: -----
Till: Lauri Tirkkonen <lotheac at iki.fi>
Fr?n: Johan Kragsterman 
S?nt av: "OmniOS-discuss" 
Datum: 2014-11-06 15:22
Kopia: omnios-discuss at lists.omniti.com
?rende: [OmniOS-discuss] Ang: Re: Ang: Re: Ang: Re: Ang: Re: Ang: Re: KVM	and SMF

Hi!

-----Lauri Tirkkonen <lotheac at iki.fi> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se>
Fr?n: Lauri Tirkkonen <lotheac at iki.fi>
Datum: 2014-11-06 14:31
Kopia: Jorge Schrauwen <sjorge+ml at blackdot.be>, omnios-discuss at lists.omniti.com
?rende: Re: Ang: Re: Ang: Re: [OmniOS-discuss] Ang: Re: Ang: Re: KVM and SMF

On Thu, Nov 06 2014 14:27:20 +0100, Johan Kragsterman wrote:
> That is not parsable...
> 
> root at omni:/mainpool/nfs/Backup/KVM# svccfg import pfkvm.xml
> pfkvm.xml:13: parser error : xmlParseEntityRef: no name
> ? ? ? ? ? ? ? ? <exec_method type='method' exec='/usr/bin/vmpfsense.sh &' name='start' timeout
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ^
> svccfg: couldn't parse document
> root at omni:/mainpool/nfs/Backup/KVM#

Then I guess you need to escape it ("&amp;"). It's easier to set with
svccfg anyways.






That did the trick! Thanks! I bet it is many others that have been struggling with this as well...


And as well:

I've been looking at svccfg(1M) but can't say I can figure out what to do to get the service to demonize...

Would be nice to learn...


Regards Johan






My kvm/pfsense is of coarse a firewall and router, and I wanted it to start before other VM's, s? the other VM's could get a route directly when they start.

So I set a dependency for my kvm/edu for kvm/pfsense, but that wasn't as easy as I thought it would. It failed with this msg:


root at omni:/mainpool/nfs/Backup/KVM# svccfg import edukvm.xml
svccfg: Service svc:/kvm/edu has property group with invalid name "kvm/pfsense" or type "dependency".
svccfg: Import of edukvm.xml failed. ?Progress:
svccfg: ? Service "kvm/edu": not reached.
svccfg: ? ? Instance "default": not reached.
svccfg: Import of edukvm.xml failed.


For me this seems to be something like the kvm/pfsense service is not properly placed. Like other smf services who has their directories in:
/lib/svc/manifest and in?/var/svc/manifest, like "application" ?"device" ?"milestone" ?"network" ?"platform" ?"site" ?"system".

Could that be the case? Someone knows something about this, and what could be done? Like creating a directory for kvm? Or putting the service under an already existing directory?

What do OmniTI do for managing this?


Regards Johan




-- 
Lauri Tirkkonen | +358 50 5341376 | lotheac @ IRCnet



_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss




From hasslerd at gmx.li  Thu Nov  6 17:47:54 2014
From: hasslerd at gmx.li (Dominik Hassler)
Date: Thu, 06 Nov 2014 18:47:54 +0100
Subject: [OmniOS-discuss] Ang: Ang: Re: Ang: Re: Ang: Re: Ang: Re: Ang:
 Re: KVM	and SMF
In-Reply-To: <OF63768A27.D6713329-ONC1257D88.005222DC-C1257D88.00530E10@inse.com>
References: <OFB992D471.2043BE0C-ONC1257D88.004D53D3-C1257D88.004EE12C@inse.com>,
	<20141106133050.GB2030@gutsman.lotheac.fi>,
	<20141106123411.GA2030@gutsman.lotheac.fi>	<20141106114910.GA8704@gutsman.lotheac.fi>	<90cff68c42e6d90bb590279e49f449ce@blackdot.be>	<9eb8e5a35d515d2c87e47e031100bfdb@blackdot.be>	<OFB164221C.B05C04C5-ONC1257D87.006AC74D-C1257D87.006EB428@inse.com>	<OF99AFB31A.F9AB8572-ONC1257D87.0070282C-C1257D87.0070282D@inse.com>	<OF70B250A4.F725F1DC-ONC1257D88.003AC9E6-C1257D88.003AC9EA@inse.com>	<6f8dcc106245a0be45eec007f134d34c@blackdot.be>	<OF7F864096.A0793304-ONC1257D88.0043236E-C1257D88.00432371@inse.com>	<OF4B35D7F2.7C79B3A2-ONC1257D88.0049C700-C1257D88.0049E9DA@inse.com>
	<OF63768A27.D6713329-ONC1257D88.005222DC-C1257D88.00530E10@inse.com>
Message-ID: <545BB44A.7080703@gmx.li>

try to name the dependency 'pfsense' instead of 'kvm/pfsense'. i don't
think that '/' is an allowed character for property names...

it would also help if you attach the manifest you are trying to import.

as lauri mentioned before: you can set all the properties directly with
svccfg, no need to fiddle around with the manifests.

On 11/06/2014 04:07 PM, Johan Kragsterman wrote:
> Here I am Again as a follow up...
> 
> 
> 
> -----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev: -----
> Till: Lauri Tirkkonen <lotheac at iki.fi>
> Fr?n: Johan Kragsterman 
> S?nt av: "OmniOS-discuss" 
> Datum: 2014-11-06 15:22
> Kopia: omnios-discuss at lists.omniti.com
> ?rende: [OmniOS-discuss] Ang: Re: Ang: Re: Ang: Re: Ang: Re: Ang: Re: KVM	and SMF
> 
> Hi!
> 
> -----Lauri Tirkkonen <lotheac at iki.fi> skrev: -----
> Till: Johan Kragsterman <johan.kragsterman at capvert.se>
> Fr?n: Lauri Tirkkonen <lotheac at iki.fi>
> Datum: 2014-11-06 14:31
> Kopia: Jorge Schrauwen <sjorge+ml at blackdot.be>, omnios-discuss at lists.omniti.com
> ?rende: Re: Ang: Re: Ang: Re: [OmniOS-discuss] Ang: Re: Ang: Re: KVM and SMF
> 
> On Thu, Nov 06 2014 14:27:20 +0100, Johan Kragsterman wrote:
>> That is not parsable...
>>
>> root at omni:/mainpool/nfs/Backup/KVM# svccfg import pfkvm.xml
>> pfkvm.xml:13: parser error : xmlParseEntityRef: no name
>>                 <exec_method type='method' exec='/usr/bin/vmpfsense.sh &' name='start' timeout
>>                                                                         ^
>> svccfg: couldn't parse document
>> root at omni:/mainpool/nfs/Backup/KVM#
> 
> Then I guess you need to escape it ("&amp;"). It's easier to set with
> svccfg anyways.
> 
> 
> 
> 
> 
> 
> That did the trick! Thanks! I bet it is many others that have been struggling with this as well...
> 
> 
> And as well:
> 
> I've been looking at svccfg(1M) but can't say I can figure out what to do to get the service to demonize...
> 
> Would be nice to learn...
> 
> 
> Regards Johan
> 
> 
> 
> 
> 
> 
> My kvm/pfsense is of coarse a firewall and router, and I wanted it to start before other VM's, s? the other VM's could get a route directly when they start.
> 
> So I set a dependency for my kvm/edu for kvm/pfsense, but that wasn't as easy as I thought it would. It failed with this msg:
> 
> 
> root at omni:/mainpool/nfs/Backup/KVM# svccfg import edukvm.xml
> svccfg: Service svc:/kvm/edu has property group with invalid name "kvm/pfsense" or type "dependency".
> svccfg: Import of edukvm.xml failed.  Progress:
> svccfg:   Service "kvm/edu": not reached.
> svccfg:     Instance "default": not reached.
> svccfg: Import of edukvm.xml failed.
> 
> 
> For me this seems to be something like the kvm/pfsense service is not properly placed. Like other smf services who has their directories in:
> /lib/svc/manifest and in /var/svc/manifest, like "application"  "device"  "milestone"  "network"  "platform"  "site"  "system".
> 
> Could that be the case? Someone knows something about this, and what could be done? Like creating a directory for kvm? Or putting the service under an already existing directory?
> 
> What do OmniTI do for managing this?
> 
> 
> Regards Johan
> 
> 
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0xF9ECC6A5.asc
Type: application/pgp-keys
Size: 2686 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141106/4789ac9d/attachment-0001.bin>

From admdwrf at gmail.com  Thu Nov  6 20:54:39 2014
From: admdwrf at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBBbGl4?=)
Date: Thu, 6 Nov 2014 21:54:39 +0100
Subject: [OmniOS-discuss] OmniOS doc
In-Reply-To: <4469AEB7-B3EA-41D5-BAA3-6C3C1AB0BAC4@kram.io>
References: <CAN4tE39rPpUJd1M35Ld+_Ryo72nxg0QhEOi6kcyfTBVWEVqiDA@mail.gmail.com>
	<20141101004433.GA12410@janus.obspm.bsdfrog.org>
	<CACLsApuod9WnBT5FS6t7tN6eRtx1jbwUbkRJS9jZKxJv8jfcog@mail.gmail.com>
	<38CD52DB-4FFA-465B-8BF6-F57395718473@kram.io>
	<CAN4tE38GZ+Nt15TVgAgrQKgqSB_wuLHjyvDg5sJoOsFpkDArjg@mail.gmail.com>
	<F8627EB9-5DFE-406C-B8C3-1A95497636DB@kram.io>
	<4469AEB7-B3EA-41D5-BAA3-6C3C1AB0BAC4@kram.io>
Message-ID: <CAN4tE3_mnqq93o1ugiBZD2uysN6jgp_=bpEg9UWtPw=5Bt3HGQ@mail.gmail.com>

Well, when we start ? :)
Steffen, if you are always ok, install jekyll in a sandbox, and we start to
write the plan.
Do you have already one  or do you want i'll send you one ?
fred.

2014-11-04 13:51 GMT+01:00 Steffen Kram <sk at kram.io>:

>
> >> In what format can we do this if you are agree ?
> >> A page in the actual wiki ? a dedicated website ? cms ? pdf file ?
> >
> > What about putting it on github and just build a nice community doc with
> jekyll based github pages? That would allow easy contribution and gives us
> maximum flexibility.
>
> @omniti: Would it be appreciated if I?m jumping in and just setup such a
> basic community doc page? I?m not a great artist but I think I?m able to
> produce something useable.
>
> Cheers, Steffen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141106/2262e028/attachment.html>

From admdwrf at gmail.com  Thu Nov  6 21:50:16 2014
From: admdwrf at gmail.com (=?UTF-8?B?RnLDqWTDqXJpYyBBbGl4?=)
Date: Thu, 6 Nov 2014 22:50:16 +0100
Subject: [OmniOS-discuss] OmniOS project donate
Message-ID: <CAN4tE3_aFNE2h0CKa+eqzSSh4ooYnyXMjCY8sXUmCnJYCALj8Q@mail.gmail.com>

Actually, when we are on the OmniOS home website project,
we can donate it with bitcoin.
Is it possible to do this with paypal ?

Regards, fred
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141106/ad72bf48/attachment.html>

From mir at miras.org  Fri Nov  7 22:49:51 2014
From: mir at miras.org (Michael Rasmussen)
Date: Fri, 7 Nov 2014 23:49:51 +0100
Subject: [OmniOS-discuss] infiniband
Message-ID: <20141107234951.26ef4aa0@sleipner.datanom.net>

Hi all,

My endeavor into infiniband has proven to be a relatively easy task,
much easier than expected, however two problems is still unresolved
which is the reason to address this list.

1) Since I have dual port HCA's I first tried to configure both ports
on the same subnet looking for a way to instruct omnios to ignore arp
request (in linux the tunable is arp_ignore=1). I have not be able to
find this tunable in omnios so I have therefore created the ports on
different subnets. This problem is minor though.

2) Performance is much slower than on Linux (Debian Stable). iperf
between two linux boxes shows 7.29 gbps which is max performance.
Between omnios an linux I am not able to pass 2.77 gbps which is
disappointing. Is there some tunable I can fiddle with to increase
performance?

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
Aliquid melius quam pessimum optimum non est.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141107/a5606528/attachment.bin>

From gate03 at landcroft.co.uk  Sat Nov  8 02:45:24 2014
From: gate03 at landcroft.co.uk (Michael Mounteney)
Date: Sat, 8 Nov 2014 12:45:24 +1000
Subject: [OmniOS-discuss] best guide to tftp ?
In-Reply-To: <CAHBEJzWx06iwAiVK3zKtG+XVmb3N-1krXe3CgoYwhcAG9XEMLw@mail.gmail.com>
References: <20141025065313.4b67dcbc@punda-mlia>
	<CAHBEJzWx06iwAiVK3zKtG+XVmb3N-1krXe3CgoYwhcAG9XEMLw@mail.gmail.com>
Message-ID: <20141108124524.6f42889a@emerior>

On Sat, 25 Oct 2014 09:51:13 +0200
Natxo Asenjo <natxo.asenjo at gmail.com> wrote:

> So you should just install the tftp package in the omnios publisher
> and then activate the tftp service by editing inetd.conf and adding
> the line:
> 
> tftp    dgram   udp6    wait    root    /usr/sbin/in.tftpd
> in.tftpd -s /tftpboot
> 
> and then running the "inetconv" command. This will create an SMF
> service, network/tftp/udp6.

I did this and:

root at diener:~# svcs -xv tftp/udp6
svc:/network/tftp/udp6:default (tftp)
 State: online since Sat Nov  8 00:58:54 2014
Impact: None.

which looks alright but:

root at diener:~# ps -ef|grep ftp
    root 19727 19582   0 02:26:18 pts/4       0:00 grep ftp

and tftp put/get requests just time out.  Here's the general manifest;
any ideas?

root at world:/support/Solaris# cat zone/diener/tftp-udp6.xml | grep -v '^$'
<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<!--
    Service manifest for the tftp service.
    Generated by inetconv(1M) from inetd.conf(4).
-->
<service_bundle type='manifest' name='inetconv:tftp'>
<service
        name='network/tftp/udp6'
        type='service'
        version='1'>
        <create_default_instance enabled='true'/>
        <restarter>
                <service_fmri value='svc:/network/inetd:default' />
        </restarter>
        <!--
            Set a timeout of 0 to signify to inetd that we don't want to
            timeout this service, since the forked process is the one that
            does the service's work. This is the case for most/all legacy
            inetd services; for services written to take advantage of SMF
            capabilities, the start method should fork off a process to
            handle the request and return a success code.
        -->
        <exec_method
                type='method'
                name='inetd_start'
                exec='/usr/sbin/in.tftpd -s /tftpboot'
                timeout_seconds='0'>
                <method_context>
                        <method_credential user='root' group='root' />
                </method_context>
        </exec_method>
        <!--
            Use inetd's built-in kill support to disable services.
        -->
        <exec_method
                type='method'
                name='inetd_disable'
                exec=':kill'
                timeout_seconds='0'>
        </exec_method>
        <!--
            Use inetd's built-in process kill support to offline wait type
            services.
        -->
        <exec_method
                type='method'
                name='inetd_offline'
                exec=':kill_process'
                timeout_seconds='0'>
        </exec_method>
        <!--
            This property group is used to record information about
            how this manifest was created.  It is an implementation
            detail which should not be modified or deleted.
        -->
        <property_group name='inetconv' type='framework'>
                <propval name='converted' type='boolean' value='true' />
                <propval name='version' type='integer' value='1' />
                <propval name='source_line' type='astring' value='tftp dgram udp6 wait root /usr/sbin/in.tftpd in.tftpd  -s      /tftpboot'
                />
        </property_group>
        <property_group name='inetd' type='framework'>
                <propval name='name' type='astring' value='tftp' />
                <propval name='endpoint_type' type='astring' value='dgram' />
                <propval name='proto' type='astring' value='udp6' />
                <propval name='wait' type='boolean' value='true' />
                <propval name='isrpc' type='boolean' value='false' />
        </property_group>
        <stability value='External' />
        <template>
                <common_name>
                        <loctext xml:lang='C'>
tftp
                        </loctext>
                </common_name>
        </template>
</service>
</service_bundle>

From mir at miras.org  Sat Nov  8 12:51:04 2014
From: mir at miras.org (Michael Rasmussen)
Date: Sat, 8 Nov 2014 13:51:04 +0100
Subject: [OmniOS-discuss] omnios and mpip
Message-ID: <20141108135104.17afacf5@sleipner.datanom.net>

Hi all,

How does omnios support mpip?

I have a dual port infiniband HCA where I would like to have both ports
responding to a single ip using mpip (to compensate for not having a
switch). I have followed oracle guides but it does not seem to work
since I am only able to ping the first port which is added to the group.

Any advice?

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
What is a magician but a practising theorist?
		-- Obi-Wan Kenobi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141108/23fcd84c/attachment.bin>

From gate03 at landcroft.co.uk  Sat Nov  8 20:30:15 2014
From: gate03 at landcroft.co.uk (Michael Mounteney)
Date: Sun, 9 Nov 2014 06:30:15 +1000
Subject: [OmniOS-discuss] best guide to tftp ?
In-Reply-To: <20141108124524.6f42889a@emerior>
References: <20141025065313.4b67dcbc@punda-mlia>
	<CAHBEJzWx06iwAiVK3zKtG+XVmb3N-1krXe3CgoYwhcAG9XEMLw@mail.gmail.com>
	<20141108124524.6f42889a@emerior>
Message-ID: <20141109063015.71b9614b@emerior>

[Later] just running

# /usr/sbin/in.tftpd -p -d -s /tftpboot

doesn't work either.  A client-side

$ tftp cortex -c get text

times out.

???

From mir at miras.org  Sun Nov  9 23:06:26 2014
From: mir at miras.org (Michael Rasmussen)
Date: Mon, 10 Nov 2014 00:06:26 +0100
Subject: [OmniOS-discuss] infiniband
In-Reply-To: <20141107234951.26ef4aa0@sleipner.datanom.net>
References: <20141107234951.26ef4aa0@sleipner.datanom.net>
Message-ID: <20141110000626.1d1b7424@sleipner.datanom.net>

On Fri, 7 Nov 2014 23:49:51 +0100
Michael Rasmussen <mir at miras.org> wrote:

> 
> 2) Performance is much slower than on Linux (Debian Stable). iperf
> between two linux boxes shows 7.29 gbps which is max performance.
> Between omnios an linux I am not able to pass 2.77 gbps which is
> disappointing. Is there some tunable I can fiddle with to increase
> performance?
> 
I might have found something here (same HCA on both Omnios and Linux):
From Omnios
08:00.0 InfiniBand: Mellanox Technologies MT25408 [ConnectX VPI - IB
SDR / 10GigE] (rev a0) Subsystem: Mellanox Technologies Device 0003
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF-
FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to
IRQ 11 Region 0: Memory at fe600000 (64-bit, non-prefetchable)
        Region 2: Memory at fc000000 (64-bit, prefetchable)
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable-
DSel=0 DScale=0 PME- Capabilities: [48] Vital Product Data
                Not readable
        Capabilities: [9c] MSI-X: Enable+ Count=256 Masked-
                Vector table: BAR=0 offset=0007c000
                PBA: BAR=0 offset=0007d000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<64ns, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x8, ASPM L0s,
Exit Latency L0s unlimited, L1 unlimited ClockPM- Surprise- LLActRep-
BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD,
TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us
to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed:
2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating
Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis:
-6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-,
EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-


From Linux:
01:00.0 InfiniBand: Mellanox Technologies MT25408 [ConnectX VPI - IB
SDR / 10GigE] (rev a0) Subsystem: Mellanox Technologies Device 0003
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF-
FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to
IRQ 28 Region 0: Memory at fe800000 (64-bit, non-prefetchable) [size=1M]
        Region 2: Memory at fb800000 (64-bit, prefetchable) [size=8M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable-
DSel=0 DScale=0 PME- Capabilities: [48] Vital Product Data
                Product Name: Eagle IB DDR
                Read-only fields:
                        [PN] Part number: HCA-00001            
                        [EC] Engineering changes: A5
                        [SN] Serial number: ML4209003684            
                        [V0] Vendor specific: HCA 500Ex-D     
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                Read/write fields:
                        [V1] Vendor specific: N/A    
                        [YA] Asset tag: N/A                             
                        [RW] Read-write area: 107 byte(s) free
                End
        Capabilities: [9c] MSI-X: Enable+ Count=256 Masked-
                Vector table: BAR=0 offset=0007c000
                PBA: BAR=0 offset=0007d000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<64ns, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x8, ASPM L0s,
Latency L0 unlimited, L1 unlimited ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain-
CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD,
TimeoutDis+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance-
SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal
Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance
De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-,
EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1]
Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next
Function: 1 ARICtl: MFVC- ACS-, Function Group: 0
        Kernel driver in use: mlx4_core

As can be seen above on both Omnios and Linux the LnkCap is reported to
2.5GT/s, width x8. But the LnkSta is different:
Linux: Speed 2.5GT/s, Width x8
Omnios: Speed 2.5GT/s, Width x2

This explains the difference in speed since 2.5GT/s x2 is roughly 3.5
gbps over IBoIP.

Can anybody explain why Omnios only uses 2 lains when PCIe slot and HCA
is capable of 8 lains?

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
Single tasking: Just Say No.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141110/b2cd6f6b/attachment-0001.bin>

From mir at miras.org  Mon Nov 10 01:20:41 2014
From: mir at miras.org (Michael Rasmussen)
Date: Mon, 10 Nov 2014 02:20:41 +0100
Subject: [OmniOS-discuss] infiniband
In-Reply-To: <20141110000626.1d1b7424@sleipner.datanom.net>
References: <20141107234951.26ef4aa0@sleipner.datanom.net>
	<20141110000626.1d1b7424@sleipner.datanom.net>
Message-ID: <20141110022041.427e1505@sleipner.datanom.net>

On Mon, 10 Nov 2014 00:06:26 +0100
Michael Rasmussen <mir at miras.org> wrote:

> 
> This explains the difference in speed since 2.5GT/s x2 is roughly 3.5
> gbps over IBoIP.
> 
> Can anybody explain why Omnios only uses 2 lains when PCIe slot and HCA
> is capable of 8 lains?
> 
Found the "bug". All required was to change the slot;-)

01:00.0 InfiniBand: Mellanox Technologies MT25408 [ConnectX VPI - IB
SDR / 10GigE] (rev a0) Subsystem: Mellanox Technologies Device 0003
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF-
FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to
IRQ 11 Region 0: Memory at fea00000 (64-bit, non-prefetchable)
        Region 2: Memory at fc000000 (64-bit, prefetchable)
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable-
DSel=0 DScale=0 PME- Capabilities: [48] Vital Product Data
                Not readable
        Capabilities: [9c] MSI-X: Enable+ Count=256 Masked-
                Vector table: BAR=0 offset=0007c000
                PBA: BAR=0 offset=0007d000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<64ns, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x8, ASPM L0s,
Exit Latency L0s unlimited, L1 unlimited ClockPM- Surprise- LLActRep-
BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD,
TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us
to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed:
2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating
Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis:
-6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-,
EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-

And some network performance tuning:
* network performance tuning
set ndd:tcp_wscale_always=1
set ndd:tcp_tstamp_if_wscale=1
set ndd:tcp_max_buf=16777216
set ndd:tcp_cwnd_max=8388608
set ndd:tcp_xmit_hiwat=1048576
set ndd:tcp_recv_hiwat=1048576

/var/web-gui/data/tools/iperf/iperf -s -B 10.0.2.10
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 10.0.2.10
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  4] local 10.0.2.10 port 5001 connected with 10.0.2.1 port 35745
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  27.4 GBytes  7.84 Gbits/sec


-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
ROMEO:		Courage, man; the hurt cannot be much.
MERCUTIO:	No, 'tis not so deep as a well, nor so wide
			as a church-door; but 'tis enough, 'twill serve.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141110/01cc8ae1/attachment.bin>

From rt at steait.net  Mon Nov 10 04:38:23 2014
From: rt at steait.net (Rune Tipsmark)
Date: Mon, 10 Nov 2014 04:38:23 +0000
Subject: [OmniOS-discuss] infiniband
In-Reply-To: <20141110022041.427e1505@sleipner.datanom.net>
References: <20141107234951.26ef4aa0@sleipner.datanom.net>
	<20141110000626.1d1b7424@sleipner.datanom.net>
	<20141110022041.427e1505@sleipner.datanom.net>
Message-ID: <95f5a26aa2fd438eb6dc0719349e91c3@EX1301.steait.net>

What network throughput were you looking at before the tweaking?

Br,
Rune

-----Original Message-----
From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Michael Rasmussen
Sent: Sunday, November 09, 2014 5:21 PM
To: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] infiniband

On Mon, 10 Nov 2014 00:06:26 +0100
Michael Rasmussen <mir at miras.org> wrote:

> 
> This explains the difference in speed since 2.5GT/s x2 is roughly 3.5 
> gbps over IBoIP.
> 
> Can anybody explain why Omnios only uses 2 lains when PCIe slot and 
> HCA is capable of 8 lains?
> 
Found the "bug". All required was to change the slot;-)

01:00.0 InfiniBand: Mellanox Technologies MT25408 [ConnectX VPI - IB SDR / 10GigE] (rev a0) Subsystem: Mellanox Technologies Device 0003
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr+ Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF-
FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 11 Region 0: Memory at fea00000 (64-bit, non-prefetchable)
        Region 2: Memory at fc000000 (64-bit, prefetchable)
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable-
DSel=0 DScale=0 PME- Capabilities: [48] Vital Product Data
                Not readable
        Capabilities: [9c] MSI-X: Enable+ Count=256 Masked-
                Vector table: BAR=0 offset=0007c000
                PBA: BAR=0 offset=0007d000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited ClockPM- Surprise- LLActRep-
BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD,
TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us
to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed:
2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis:
-6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-,
EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-

And some network performance tuning:
* network performance tuning
set ndd:tcp_wscale_always=1
set ndd:tcp_tstamp_if_wscale=1
set ndd:tcp_max_buf=16777216
set ndd:tcp_cwnd_max=8388608
set ndd:tcp_xmit_hiwat=1048576
set ndd:tcp_recv_hiwat=1048576

/var/web-gui/data/tools/iperf/iperf -s -B 10.0.2.10
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 10.0.2.10
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  4] local 10.0.2.10 port 5001 connected with 10.0.2.1 port 35745
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  27.4 GBytes  7.84 Gbits/sec


--
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
ROMEO:		Courage, man; the hurt cannot be much.
MERCUTIO:	No, 'tis not so deep as a well, nor so wide
			as a church-door; but 'tis enough, 'twill serve.

From mir at miras.org  Mon Nov 10 07:02:06 2014
From: mir at miras.org (Michael Rasmussen)
Date: Mon, 10 Nov 2014 08:02:06 +0100
Subject: [OmniOS-discuss] infiniband
In-Reply-To: <95f5a26aa2fd438eb6dc0719349e91c3@EX1301.steait.net>
References: <20141107234951.26ef4aa0@sleipner.datanom.net>
	<20141110000626.1d1b7424@sleipner.datanom.net>
	<20141110022041.427e1505@sleipner.datanom.net>
	<95f5a26aa2fd438eb6dc0719349e91c3@EX1301.steait.net>
Message-ID: <20141110080206.50e6f07a@sleipner.datanom.net>

On Mon, 10 Nov 2014 04:38:23 +0000
Rune Tipsmark <rt at steait.net> wrote:

> What network throughput were you looking at before the tweaking?
> 
Raised my performance from 5.2 gbps to 7.9 gbps (50% performance
increase)

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
Who needs friends when you can sit alone in your room and drink?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141110/a2893788/attachment.bin>

From rt at steait.net  Mon Nov 10 18:23:01 2014
From: rt at steait.net (Rune Tipsmark)
Date: Mon, 10 Nov 2014 18:23:01 +0000
Subject: [OmniOS-discuss] No space left on device - upgrade failed
Message-ID: <3e2d8cde01ce4513b878bad20107cec8@EX1301.steait.net>

Hi all,

Hoping someone can help here.

root at zfs00:~# /usr/bin/pkg update --be-name=omnios-r151012 entire at 11,5.11-0.151012
Creating Plan |pkg: An error was encountered while attempting to store information about the
current operation in client history.
pkg: [Errno 28] No space left on device: '/var/pkg/history/20141110T181542Z-01.xml'

root at zfs00:~# zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
pool03  10.9T  5.03G  10.9T         -     0%  1.00x  ONLINE  -
rpool   14.9G  10.5G  4.35G         -    70%  1.00x  ONLINE  -
root at zfs00:~#

seems like there is enough space?

How to fix? And how to keep  a tidy rpool at all times?

Br,
Rune

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141110/fc482fb7/attachment.html>

From jimklimov at cos.ru  Mon Nov 10 20:53:24 2014
From: jimklimov at cos.ru (Jim Klimov)
Date: Mon, 10 Nov 2014 21:53:24 +0100
Subject: [OmniOS-discuss] No space left on device - upgrade failed
In-Reply-To: <3e2d8cde01ce4513b878bad20107cec8@EX1301.steait.net>
References: <3e2d8cde01ce4513b878bad20107cec8@EX1301.steait.net>
Message-ID: <484095DF-26BB-4270-B9B4-561396357767@cos.ru>

10 ?????? 2014??. 19:23:01 CET, Rune Tipsmark <rt at steait.net> ?????:
>Hi all,
>
>Hoping someone can help here.
>
>root at zfs00:~# /usr/bin/pkg update --be-name=omnios-r151012
>entire at 11,5.11-0.151012
>Creating Plan |pkg: An error was encountered while attempting to store
>information about the
>current operation in client history.
>pkg: [Errno 28] No space left on device:
>'/var/pkg/history/20141110T181542Z-01.xml'
>
>root at zfs00:~# zpool list
>NAME     SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
>pool03  10.9T  5.03G  10.9T         -     0%  1.00x  ONLINE  -
>rpool   14.9G  10.5G  4.35G         -    70%  1.00x  ONLINE  -
>root at zfs00:~#
>
>seems like there is enough space?
>
>How to fix? And how to keep  a tidy rpool at all times?
>
>Br,
>Rune
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>OmniOS-discuss mailing list
>OmniOS-discuss at lists.omniti.com
>http://lists.omniti.com/mailman/listinfo/omnios-discuss

What is the dataset breakout (zfs list)?
Maybe you have reservations like swap and dump (volumes in general) - their unused space is not available for other datasets and not allocated on backend storage either (what zpool list reflects).

HTH Jim
--
Typos courtesy of K-9 Mail on my Samsung Android

From rt at steait.net  Mon Nov 10 23:32:14 2014
From: rt at steait.net (Rune Tipsmark)
Date: Mon, 10 Nov 2014 23:32:14 +0000
Subject: [OmniOS-discuss] No space left on device - upgrade failed
In-Reply-To: <484095DF-26BB-4270-B9B4-561396357767@cos.ru>
References: <3e2d8cde01ce4513b878bad20107cec8@EX1301.steait.net>
	<484095DF-26BB-4270-B9B4-561396357767@cos.ru>
Message-ID: <9ba4c7286e9b476da2c538aec0e6a285@EX1301.steait.net>

>What is the dataset breakout (zfs list)?
>Maybe you have reservations like swap and dump (volumes in general) - their unused space is not available for other datasets and not allocated on >backend storage either (what zpool list reflects).

root at zfs00:~# zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
pool03                     189G  10.7T  38.5K  /pool03
pool03/ISO                  34K  10.5T    34K  /pool03/ISO
pool03/vol01                16K  10.5T    16K  -
rpool                     13.7G   992M  38.5K  /rpool
rpool/ROOT                3.55G   992M    31K  legacy
rpool/ROOT/napp-it-0.9e1  3.55G   992M  3.25G  /
rpool/ROOT/omniosvar        31K   992M    31K  legacy
rpool/dump                5.99G   992M  5.99G  -
rpool/export                63K   992M    32K  /export
rpool/export/home           31K   992M    31K  /export/home
rpool/swap                4.13G  5.09G  4.44M  -

I managed to delete some snapshots and got back on track, now 922 MB free...  how much spare is OmniOS supposed to take up? Rpool of 13.7G where only 922M free... looks like its filling up worse than Windows! :)

Br,
Rune

From mir at miras.org  Mon Nov 10 23:47:15 2014
From: mir at miras.org (Michael Rasmussen)
Date: Tue, 11 Nov 2014 00:47:15 +0100
Subject: [OmniOS-discuss] No space left on device - upgrade failed
In-Reply-To: <9ba4c7286e9b476da2c538aec0e6a285@EX1301.steait.net>
References: <3e2d8cde01ce4513b878bad20107cec8@EX1301.steait.net>
	<484095DF-26BB-4270-B9B4-561396357767@cos.ru>
	<9ba4c7286e9b476da2c538aec0e6a285@EX1301.steait.net>
Message-ID: <20141111004715.20beeb14@sleipner.datanom.net>

On Mon, 10 Nov 2014 23:32:14 +0000
Rune Tipsmark <rt at steait.net> wrote:

> 
> root at zfs00:~# zfs list
> NAME                       USED  AVAIL  REFER  MOUNTPOINT
> pool03                     189G  10.7T  38.5K  /pool03
> pool03/ISO                  34K  10.5T    34K  /pool03/ISO
> pool03/vol01                16K  10.5T    16K  -
> rpool                     13.7G   992M  38.5K  /rpool
> rpool/ROOT                3.55G   992M    31K  legacy
> rpool/ROOT/napp-it-0.9e1  3.55G   992M  3.25G  /
> rpool/ROOT/omniosvar        31K   992M    31K  legacy
> rpool/dump                5.99G   992M  5.99G  -
> rpool/export                63K   992M    32K  /export
> rpool/export/home           31K   992M    31K  /export/home
> rpool/swap                4.13G  5.09G  4.44M  -
> 
> I managed to delete some snapshots and got back on track, now 922 MB free...  how much spare is OmniOS supposed to take up? Rpool of 13.7G where only 922M free... looks like its filling up worse than Windows! :)
> 
What does rpool/dump contain? It is roughly reserving half of the pool.

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
No one may kill a man.  Not for any purpose.  It cannot be condoned.
		-- Kirk, "Spock's Brain", stardate 5431.6
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141111/9de93958/attachment.bin>

From rt at steait.net  Tue Nov 11 01:09:29 2014
From: rt at steait.net (Rune Tipsmark)
Date: Tue, 11 Nov 2014 01:09:29 +0000
Subject: [OmniOS-discuss] No space left on device - upgrade failed
In-Reply-To: <20141111004715.20beeb14@sleipner.datanom.net>
References: <3e2d8cde01ce4513b878bad20107cec8@EX1301.steait.net>
	<484095DF-26BB-4270-B9B4-561396357767@cos.ru>
	<9ba4c7286e9b476da2c538aec0e6a285@EX1301.steait.net>
	<20141111004715.20beeb14@sleipner.datanom.net>
Message-ID: <c4fa69829ac447a098a8efcb2d06d87b@EX1301.steait.net>

No idea what dump contains, but I ran zfs set volsize=4G rpool/dump and its now 4G.
I also found a crash dump in /var/crash/unknown and that freed up another couple of gigs.

Can I reduce swap somehow?

root at zfs00:/root# zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
pool03                      189G  10.5T  38.5K  /pool03
pool03/ISO                   27K  10.4T    27K  /pool03/ISO
pool03/vol01                 16K  10.4T    16K  -
rpool                      11.5G  2.88G  31.5K  /rpool
rpool/ROOT                 3.40G  2.88G    31K  legacy
rpool/ROOT/omnios-r151012  3.40G  2.88G  2.37G  /
rpool/dump                 4.00G  2.88G  4.00G  -
rpool/export                 63K  2.88G    32K  /export
rpool/export/home            31K  2.88G    31K  /export/home
rpool/swap                 4.13G  7.00G  4.44M  -

-----Original Message-----
From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Michael Rasmussen
Sent: Monday, November 10, 2014 3:47 PM
To: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] No space left on device - upgrade failed

On Mon, 10 Nov 2014 23:32:14 +0000
Rune Tipsmark <rt at steait.net> wrote:

> 
> root at zfs00:~# zfs list
> NAME                       USED  AVAIL  REFER  MOUNTPOINT
> pool03                     189G  10.7T  38.5K  /pool03
> pool03/ISO                  34K  10.5T    34K  /pool03/ISO
> pool03/vol01                16K  10.5T    16K  -
> rpool                     13.7G   992M  38.5K  /rpool
> rpool/ROOT                3.55G   992M    31K  legacy
> rpool/ROOT/napp-it-0.9e1  3.55G   992M  3.25G  /
> rpool/ROOT/omniosvar        31K   992M    31K  legacy
> rpool/dump                5.99G   992M  5.99G  -
> rpool/export                63K   992M    32K  /export
> rpool/export/home           31K   992M    31K  /export/home
> rpool/swap                4.13G  5.09G  4.44M  -
> 
> I managed to delete some snapshots and got back on track, now 922 MB free...  how much spare is OmniOS supposed to take up? Rpool of 13.7G where only 922M free... looks like its filling up worse than Windows! :)
> 
What does rpool/dump contain? It is roughly reserving half of the pool.

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
No one may kill a man.  Not for any purpose.  It cannot be condoned.
		-- Kirk, "Spock's Brain", stardate 5431.6

From jdg117 at elvis.arl.psu.edu  Tue Nov 11 01:24:23 2014
From: jdg117 at elvis.arl.psu.edu (John D Groenveld)
Date: Mon, 10 Nov 2014 20:24:23 -0500
Subject: [OmniOS-discuss] No space left on device - upgrade failed
In-Reply-To: Your message of "Tue, 11 Nov 2014 01:09:29 GMT."
	<c4fa69829ac447a098a8efcb2d06d87b@EX1301.steait.net> 
References: <3e2d8cde01ce4513b878bad20107cec8@EX1301.steait.net>
	<484095DF-26BB-4270-B9B4-561396357767@cos.ru>
	<9ba4c7286e9b476da2c538aec0e6a285@EX1301.steait.net>
	<20141111004715.20beeb14@sleipner.datanom.net>
	<c4fa69829ac447a098a8efcb2d06d87b@EX1301.steait.net> 
Message-ID: <201411110124.sAB1ONcD002262@elvis.arl.psu.edu>

In message <c4fa69829ac447a098a8efcb2d06d87b at EX1301.steait.net>, Rune Tipsmark 
writes:
>Can I reduce swap somehow?

# zfs create -V 2G rpool/swap2 &&
swap -a /dev/zvol/dsk/rpool/swap2 &&
swap -d /dev/zvol/dsk/rpool/swap &&
zfs destroy rpool/swap

John
groenveld at acm.org


From johan.kragsterman at capvert.se  Tue Nov 11 05:13:34 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Tue, 11 Nov 2014 06:13:34 +0100
Subject: [OmniOS-discuss] Ang: Re: No space left on device - upgrade failed
In-Reply-To: <c4fa69829ac447a098a8efcb2d06d87b@EX1301.steait.net>
References: <c4fa69829ac447a098a8efcb2d06d87b@EX1301.steait.net>,
	<3e2d8cde01ce4513b878bad20107cec8@EX1301.steait.net>	<484095DF-26BB-4270-B9B4-561396357767@cos.ru>
	<9ba4c7286e9b476da2c538aec0e6a285@EX1301.steait.net>	<20141111004715.20beeb14@sleipner.datanom.net>
Message-ID: <OF49EEFAFB.B81A46A2-ONC1257D8D.001CB385-C1257D8D.001CB588@inse.com>

An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141111/34e2aaa8/attachment.html>

From rt at steait.net  Wed Nov 12 05:36:18 2014
From: rt at steait.net (Rune Tipsmark)
Date: Wed, 12 Nov 2014 05:36:18 +0000
Subject: [OmniOS-discuss] ZFS pool allocation remains after removing all
 files
In-Reply-To: <f6a587525c204d09a6ca1f7be14c80e9@EX1301.steait.net>
References: <3BE0DEED8863E5429BAE4CAEDF6245650368AFFA2BFC@AIRA-SRV.aira.local>
	<a50e1a82a2634c89a276d506382859b4@EX1301.steait.net>
	<056B925E-9202-46DA-8AB2-22D741122C96@richardelling.com>
	<8BE5066F-6143-4907-9C7F-C2134AB7986C@omniti.com>
	<7facc48a42cb4a51b9fd096c3c39e9ca@EX1301.steait.net>
	<0B94C1FB-8659-4024-9815-383809F660F0@omniti.com>
	<1471cfc65e4d453a98d873dfc6608a6e@EX1301.steait.net>
	<3599C645-8A66-49E0-94A4-A110CB5092D5@omniti.com>
	<3AD8817E-34CB-4CE3-A916-8A29382F85D3@omniti.com>
	<e2f91e6e00d34c8cae1a55436855f215@EX1301.steait.net>
	<EA2BFF89-478F-4CCB-ABC3-8363048691D7@omniti.com>
	<52148cc65f0e41c2a538e5e66241970e@EX1301.steait.net>
	<127df2661d744d70b22377bd854de648@EX1301.steait.net>
	<19800DA1-528D-4442-8326-63BB97B2B700@richardelling.com>
	<f6a587525c204d09a6ca1f7be14c80e9@EX1301.steait.net>
Message-ID: <d061ad0ec5764430b0cd59e69548996a@EX1301.steait.net>

So just to revive an older thread....

Thin Provisioned LU's have only one VAAI option supported:

naa.600144f007780b7f00005462ef090002
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: unsupported

But if I create a  thin provisioned Volume two VAAI options are supported:

naa.600144f007780b7f00005462eb3d0001
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: supported

-
How can this be? 

Anyone know if there are any news on the two remaining unsupported options?

Br,
Rune

-----Original Message-----
From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Rune Tipsmark
Sent: Friday, October 10, 2014 1:58 PM
To: Richard Elling
Cc: omnios-discuss
Subject: Re: [OmniOS-discuss] ZFS pool allocation remains after removing all files

Same acceleration on iSCSI

naa.600144f0908abf5d0000539106e40001
   VAAI Plugin Name:
   ATS Status: unsupported
   Clone Status: unsupported
   Zero Status: supported
   Delete Status: unsupported

Rune

-----Original Message-----
From: Richard Elling [mailto:richard.elling at richardelling.com] 
Sent: Friday, October 10, 2014 10:01 AM
To: Rune Tipsmark
Cc: Dan McDonald; omnios-discuss
Subject: Re: [OmniOS-discuss] ZFS pool allocation remains after removing all files


On Oct 9, 2014, at 4:58 PM, Rune Tipsmark <rt at steait.net> wrote:

> Just updated to latest version r151012
> 
> Still same... I checked for vdev settings, is there another place I can check?

It won't be a ZFS feature. On the initiator, use something like sg3_utils thusly:

[root at congo ~]# sg_opcodes /dev/rdsk/c0t5000C50030117C3Bd0
  SEAGATE   ST800FM0043       0005
  Peripheral device type: disk

Opcode  Service    CDB    Name
(hex)   action(h)  size       
-----------------------------------------------
 00                  6    Test Unit Ready
 01                  6    Rezero Unit
 03                  6    Request Sense
 04                  6    Format Unit
 07                  6    Reassign Blocks
 08                  6    Read(6)
 0a                  6    Write(6)
 0b                  6    Seek(6)
 12                  6    Inquiry
 15                  6    Mode select(6)
 16                  6    Reserve(6)
 17                  6    Release(6)
 1a                  6    Mode sense(6)
 1b                  6    Start stop unit
 1c                  6    Receive diagnostic results
 1d                  6    Send diagnostic
 25                 10    Read capacity(10)
 28                 10    Read(10)
 2a                 10    Write(10)
 2b                 10    Seek(10)
 2e                 10    Write and verify(10)
 2f                 10    Verify(10)
 35                 10    Synchronize cache(10)
 37                 10    Read defect data(10)
 3b        0        10    Write buffer, combined header and data [or multiple modes]
 3b        2        10    Write buffer, data
 3b        4        10    Write buffer, download microcode and activate
 3b        5        10    Write buffer, download microcode, save, and activate
 3b        6        10    Write buffer, download microcode with offsets and activate
 3b        7        10    Write buffer, download microcode with offsets, save, and activate
 3b        a        10    Write buffer, write data to echo buffer
 3b        d        10    Write buffer, download microcode with offsets, select activation events, save and defer activate
 3b        e        10    Write buffer, download microcode with offsets, save and defer activate
 3b        f        10    Write buffer, activate deferred microcode
 3b       1a        10    Write buffer, enable expander comms protocol and echo buffer
 3b       1c        10    Write buffer, download application client error history
 3c        0        10    Read buffer, combined header and data [or multiple modes]
 3c        2        10    Read buffer, data
 3c        3        10    Read buffer, descriptor
 3c        a        10    Read buffer, read data from echo buffer
 3c        b        10    Read buffer, echo buffer descriptor
 3c       1c        10    Read buffer, error history
 3e                 10    Read long(10)
 3f                 10    Write long(10)
 41                 10    Write same(10)
 42                 10    Unmap
 48        2        10    Sanitize, block erase
 48       1f        10    Sanitize, exit failure mode
 4c                 10    Log select
 4d                 10    Log sense
 55                 10    Mode select(10)
 56                 10    Reserve(10)
 57                 10    Release(10)
 5a                 10    Mode sense(10)
 5e        0        10    Persistent reserve in, read keys
 5e        1        10    Persistent reserve in, read reservation
 5e        2        10    Persistent reserve in, report capabilities
 5e        3        10    Persistent reserve in, read full status
 5f        0        10    Persistent reserve out, register
 5f        1        10    Persistent reserve out, reserve
 5f        2        10    Persistent reserve out, release
 5f        3        10    Persistent reserve out, clear
 5f        4        10    Persistent reserve out, preempt
 5f        5        10    Persistent reserve out, preempt and abort
 5f        6        10    Persistent reserve out, register and ignore existing key
 5f        7        10    Persistent reserve out, register and move
 7f        9        32    Read(32)
 7f        a        32    Verify(32)
 7f        b        32    Write(32)
 7f        c        32    Write an verify(32)
 7f        d        32    Write same(32)
 88                 16    Read(16)
 8a                 16    Write(16)
 8e                 16    Write and verify(16)
 8f                 16    Verify(16)
 91                 16    Synchronize cache(16)
 93                 16    Write same(16)
 9e       10        16    Read capacity(16)
 9e       11        16    Read long(16)
 9f       11        16    Write long(16)
 a0                 12    Report luns
 a3        5        12    Report identifying information
 a3        c        12    Report supported operation codes
 a3        d        12    Report supported task management functions
 a4        6        12    Set identifying information
 a4        f        12    Set timestamp
 b7                 12    Read defect data(12)
 e0                 10    Vendor specific [0xe0]
 e1                 10    Vendor specific [0xe1]
 e2                 10    Vendor specific [0xe2]
 e6                 10    Vendor specific [0xe6]
 f7                 10    Vendor specific [0xf7]

I'd try it for iSCSI, but since I no longer use iSCSI (yea AoE!) I can't test :-P
 -- richard

> 
> root at zfs10:/root# echo "::zfs_params" | mdb -k | grep vdev 
> zfs_vdev_max_active = 0x3e8 zfs_vdev_sync_read_min_active = 0xa 
> zfs_vdev_sync_read_max_active = 0xa zfs_vdev_sync_write_min_active = 
> 0xa zfs_vdev_sync_write_max_active = 0xa 
> zfs_vdev_async_read_min_active = 0x1 zfs_vdev_async_read_max_active = 
> 0x3 zfs_vdev_async_write_min_active = 0x1 
> zfs_vdev_async_write_max_active = 0xa zfs_vdev_scrub_min_active = 0x1 
> zfs_vdev_scrub_max_active = 0x2 
> zfs_vdev_async_write_active_min_dirty_percent = 0x1e 
> zfs_vdev_async_write_active_max_dirty_percent = 0x3c
> mdb: variable reference_tracking_enable not found: unknown symbol name
> mdb: variable reference_history not found: unknown symbol name 
> zfs_vdev_cache_max = 0x4000 zfs_vdev_cache_size = 0x0 
> zfs_vdev_cache_bshift = 0x10 vdev_mirror_shift = 0x15 
> zfs_vdev_aggregation_limit = 0x20000
> 
> Rune
> 
> -----Original Message-----
> From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] 
> On Behalf Of Rune Tipsmark
> Sent: Thursday, October 09, 2014 3:33 PM
> To: Dan McDonald
> Cc: omnios-discuss
> Subject: Re: [OmniOS-discuss] ZFS pool allocation remains after 
> removing all files
> 
> Is there a command I can run to check?
> 
> Rune
> 
> -----Original Message-----
> From: Dan McDonald [mailto:danmcd at omniti.com]
> Sent: Thursday, October 09, 2014 11:51 AM
> To: Rune Tipsmark
> Cc: omnios-discuss
> Subject: Re: [OmniOS-discuss] ZFS pool allocation remains after 
> removing all files
> 
> 
> On Oct 9, 2014, at 2:38 PM, Rune Tipsmark <rt at steait.net> wrote:
> 
>> So if I just upgrade to latest it should be supported?
> 
> It should be available in r151010!  That's why I'm surprised.
> 
> Dan
> 
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

From a.shvayakov at yahoo.com  Wed Nov 12 17:44:45 2014
From: a.shvayakov at yahoo.com (Shvayakov A.)
Date: Wed, 12 Nov 2014 20:44:45 +0300
Subject: [OmniOS-discuss] I can't add additional network card
Message-ID: <54639C8D.3000106@yahoo.com>

Hello,

Likely I don't know something important about solaris...

Can you help me?

I tried to add the additional network adapter, but my system (OmniOS v11 
r151012) don't see him

dladm show-phys
LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
interlink0   Ethernet             up         1000   full      igb0
extnet0      Ethernet             up         1000   full      igb1
ixgbe0       Ethernet             up         10000  full      ixgbe0
ixgbe1       Ethernet             up         10000  full      ixgbe1

But I have 6 NICs and want to see 6 NICs

prtconf -d | grep Gigabit
             pci15d9,1528 (pciex8086,1528) [Intel Corporation Ethernet 
Controller 10-Gigabit X540-AT2], instance #0
             pci15d9,1528 (pciex8086,1528) [Intel Corporation Ethernet 
Controller 10-Gigabit X540-AT2], instance #1
             pci8086,a03c (pciex8086,10c9) [Intel Corporation 82576 
Gigabit Network Connection], instance #0
             pci8086,a03c (pciex8086,10c9) [Intel Corporation 82576 
Gigabit Network Connection], instance #1
             pci8086,a03c (pciex8086,10c9) [Intel Corporation 82576 
Gigabit Network Connection] (driver not attached)
             pci8086,a03c (pciex8086,10c9) [Intel Corporation 82576 
Gigabit Network Connection] (driver not attached)

prtconf -dD | grep Giga
             pci15d9,1528 (pciex8086,1528) [Intel Corporation Ethernet 
Controller 10-Gigabit X540-AT2], instance #0 (driver name: ixgbe)
             pci15d9,1528 (pciex8086,1528) [Intel Corporation Ethernet 
Controller 10-Gigabit X540-AT2], instance #1 (driver name: ixgbe)
             pci8086,a03c (pciex8086,10c9) [Intel Corporation 82576 
Gigabit Network Connection], instance #0 (driver name: igb)
             pci8086,a03c (pciex8086,10c9) [Intel Corporation 82576 
Gigabit Network Connection], instance #1 (driver name: igb)
             pci8086,a03c (pciex8086,10c9) [Intel Corporation 82576 
Gigabit Network Connection] (driver name: igb)
             pci8086,a03c (pciex8086,10c9) [Intel Corporation 82576 
Gigabit Network Connection] (driver name: igb)


I tried this:
1.
touch /reconfigure
reboot

2.
mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix 
scsi_vhci zfs sata sd ip hook neti sockfs arp usba stmf stmf_sbd lofs 
random idm ipc crypto mpt_sas cpc nfs ufs logindmux ptm ]
 > ::devbindings -q igb
ffffff198bd4d000 pciex8086,10c9, instance #0 (driver name: igb)
ffffff198bd48d50 pciex8086,10c9, instance #1 (driver name: igb)
ffffff198bd8ad48 pciex8086,10c9 (driver not attached)
ffffff198bd8aaa0 pciex8086,10c9 (driver not attached)
 > $q

update_drv -d -i 'pciex8086,10c9' igb
reboot

update_drv -a -i 'pciex8086,10c9' igb


It doesn't help.

The cfgadm doesn't work:

cfgadm pci
Ap_Id                          Type         Receptacle Occupant     
Condition
pci: No matching library found
( https://docs.oracle.com/cd/E19253-01/817-5093/devconfig2-55/index.html )

In the LiveCD system (OmniOS v11 r151012) I see everything NIC

What else can I do?

Thanks


From danmcd at omniti.com  Wed Nov 12 18:16:45 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Wed, 12 Nov 2014 12:16:45 -0600
Subject: [OmniOS-discuss] I can't add additional network card
In-Reply-To: <54639C8D.3000106@yahoo.com>
References: <54639C8D.3000106@yahoo.com>
Message-ID: <A00EFCA5-B3A9-41CA-AC87-1CF3A78A6D83@omniti.com>

So let me get this straight --> booting the r151012 live CD lets you see two ixgbe ports and four igb ports?  But the installed system only lets you see two igb ports and two ixgbe ports?

What does "dladm show-ether" have to say?  And since two of the igbs appear not to be attached, can you "grep igb /var/adm/messages*" and see if there are any kernel complaints about it?

Thanks,
Dan


From danmcd at omniti.com  Wed Nov 12 18:19:04 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Wed, 12 Nov 2014 12:19:04 -0600
Subject: [OmniOS-discuss] omnios and mpip
In-Reply-To: <20141108135104.17afacf5@sleipner.datanom.net>
References: <20141108135104.17afacf5@sleipner.datanom.net>
Message-ID: <8F1DC00F-09D3-4954-A20E-1E1BB5658B9A@omniti.com>


> On Nov 8, 2014, at 6:51 AM, Michael Rasmussen <mir at miras.org> wrote:
> 
> Hi all,
> 
> How does omnios support mpip?

You mean IPMP, right?

Please share the output of "ifconfig -a", "dladm show-link", and "netstat -rnv" for starters.

Dan


From mir at miras.org  Wed Nov 12 18:27:34 2014
From: mir at miras.org (Michael Rasmussen)
Date: Wed, 12 Nov 2014 19:27:34 +0100
Subject: [OmniOS-discuss] omnios and mpip
In-Reply-To: <8F1DC00F-09D3-4954-A20E-1E1BB5658B9A@omniti.com>
References: <20141108135104.17afacf5@sleipner.datanom.net>
	<8F1DC00F-09D3-4954-A20E-1E1BB5658B9A@omniti.com>
Message-ID: <20141112192734.6b6a89eb@sleipner.datanom.net>

On Wed, 12 Nov 2014 12:19:04 -0600
Dan McDonald <danmcd at omniti.com> wrote:

> 
> > On Nov 8, 2014, at 6:51 AM, Michael Rasmussen <mir at miras.org> wrote:
> > 
> > Hi all,
> > 
> > How does omnios support mpip?
> 
> You mean IPMP, right?
> 
Yes:-)

> Please share the output of "ifconfig -a", "dladm show-link", and "netstat -rnv" for starters.
> 
To make it work I have assigned each port(pFFFF.ibp2, pFFFF.ibp3) to
separate subnets

ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu
8232 index 1 inet 127.0.0.1 netmask ff000000 
vnic1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500
index 2 inet 192.168.2.11 netmask ffffff00 broadcast 192.168.2.255
vlan20: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu
1500 index 3 inet 172.16.2.2 netmask ffffff00 broadcast 172.16.2.255
pFFFF.ibp2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu
65520 index 4 inet 10.0.1.10 netmask ffffff00 broadcast 10.0.1.255
pFFFF.ibp3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu
65520 index 5 inet 10.0.2.10 netmask ffffff00 broadcast 10.0.2.255
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu
8252 index 1 inet6 ::1/128 
vnic1: flags=20002000840<RUNNING,MULTICAST,IPv6> mtu 1500 index 2
        inet6 ::/0 
vlan20: flags=20202000840<RUNNING,MULTICAST,IPv6,CoS> mtu 1500 index 3
        inet6 ::/0 
pFFFF.ibp2: flags=20002000840<RUNNING,MULTICAST,IPv6> mtu 65520 index 4
        inet6 ::/0 
pFFFF.ibp3: flags=20002000840<RUNNING,MULTICAST,IPv6> mtu 65520 index 5
        inet6 ::/0 

dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
rge0        phys      1500   unknown  --         --
e1000g1     phys      1500   up       --         --
aggr0       aggr      1500   up       --         e1000g1 e1000g2
vnic1       vnic      1500   up       --         aggr0
vlan20      vlan      1500   up       --         aggr0
e1000g2     phys      1500   up       --         --
ibp2        phys      65520  up       --         --
ibp3        phys      65520  up       --         --
pFFFF.ibp2  part      65520  up       --         ibp2
pFFFF.ibp3  part      65520  up       --         ibp3

netstat -rnv

IRE Table: IPv4
  Destination             Mask           Gateway          Device  MTU
Ref Flg  Out  In/Fwd -------------------- ---------------
-------------------- ------ ----- --- --- ----- ------
default              0.0.0.0         192.168.2.1
0   3 UG     878      0 10.0.1.0             255.255.255.0
10.0.1.10            pFFFF.ibp2 65520   3 U   6649452      0
10.0.2.0             255.255.255.0   10.0.2.10            pFFFF.ibp3
65520   3 U   9655794      0 127.0.0.1            255.255.255.255
127.0.0.1            lo0     8232   3 UH   89498  89498
172.16.2.0           255.255.255.0   172.16.2.2           vlan20
1500   5 U   5195904      0 192.168.2.0          255.255.255.0
192.168.2.11         vnic1   1500   4 U    44561      0 

IRE Table: IPv6
  Destination/Mask            Gateway                    If    MTU  Ref
Flags  Out   In/Fwd ---------------------------
--------------------------- ----- ----- --- ----- ------
------ ::1                         ::1                         lo0
8252   2 UH         0      0 
-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
I'm QUIETLY reading the latest issue of "BOWLING WORLD" while my wife
and two children stand QUIETLY BY ...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141112/6ca755a7/attachment.bin>

From danmcd at omniti.com  Wed Nov 12 18:42:12 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Wed, 12 Nov 2014 12:42:12 -0600
Subject: [OmniOS-discuss] omnios and mpip
In-Reply-To: <20141112192734.6b6a89eb@sleipner.datanom.net>
References: <20141108135104.17afacf5@sleipner.datanom.net>
	<8F1DC00F-09D3-4954-A20E-1E1BB5658B9A@omniti.com>
	<20141112192734.6b6a89eb@sleipner.datanom.net>
Message-ID: <CBF4320E-9A17-480E-A5E1-DEAA9AA4B792@omniti.com>

You're not using IPMP, it appears you're using link aggregation.

"dladm show-aggr" would be useful too.

Dan


From mir at miras.org  Wed Nov 12 18:58:16 2014
From: mir at miras.org (Michael Rasmussen)
Date: Wed, 12 Nov 2014 19:58:16 +0100
Subject: [OmniOS-discuss] omnios and mpip
In-Reply-To: <CBF4320E-9A17-480E-A5E1-DEAA9AA4B792@omniti.com>
References: <20141108135104.17afacf5@sleipner.datanom.net>
	<8F1DC00F-09D3-4954-A20E-1E1BB5658B9A@omniti.com>
	<20141112192734.6b6a89eb@sleipner.datanom.net>
	<CBF4320E-9A17-480E-A5E1-DEAA9AA4B792@omniti.com>
Message-ID: <20141112195816.1dd049af@sleipner.datanom.net>

On Wed, 12 Nov 2014 12:42:12 -0600
Dan McDonald <danmcd at omniti.com> wrote:

> You're not using IPMP, it appears you're using link aggregation.
> 
No, I had to drop IPMP on infiniband since I were only able to have it work in
active/passive mode.

Link aggregation is used over the 2 Intel Gb nics

> "dladm show-aggr" would be useful too.
> 
dladm show-aggr 
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr0           L4       auto                 active        short       -----

dladm show-phys
LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
rge0         Ethernet             unknown    0      unknown   rge0
e1000g1      Ethernet             up         1000   full      e1000g1
e1000g2      Ethernet             up         1000   full      e1000g2
ibp2         Infiniband           up         8000   unknown   ibp2
ibp3         Infiniband           up         8000   unknown   ibp3

dladm show-ib
LINK         HCAGUID         PORTGUID        PORT STATE  PKEYS
ibp2         8F104039A6254   8F104039A6255   1    up     FFFF
ibp3         8F104039A6254   8F104039A6256   2    up     FFFF

dladm show-ether
LINK            PTYPE    STATE    AUTO  SPEED-DUPLEX                    PAUSE
rge0            current  unknown  no    0M                              none
e1000g1         current  up       yes   1G-f                            bi
e1000g2         current  up       yes   1G-f                            bi

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
While it may be true that a watched pot never boils, the one you don't
keep an eye on can make an awful mess of your stove.
		-- Edward Stevenson
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141112/28705352/attachment.bin>

From rt at steait.net  Wed Nov 12 22:35:46 2014
From: rt at steait.net (Rune Tipsmark)
Date: Wed, 12 Nov 2014 22:35:46 +0000
Subject: [OmniOS-discuss] infiniband
In-Reply-To: <20141110080206.50e6f07a@sleipner.datanom.net>
References: <20141107234951.26ef4aa0@sleipner.datanom.net>
	<20141110000626.1d1b7424@sleipner.datanom.net>
	<20141110022041.427e1505@sleipner.datanom.net>
	<95f5a26aa2fd438eb6dc0719349e91c3@EX1301.steait.net>
	<20141110080206.50e6f07a@sleipner.datanom.net>
Message-ID: <649b4fab1100440197fcc2ba70040f0c@EX1301.steait.net>

Just applied it on my server, it doubled my write speeds over iSER from ESX - basically they are par with SRP now, just worse latency.
Now I just gotta find out if there are tweaks to be done to ESX  to improve further.

Did you try creating more partitions say 4 partitions to see if it performs better than 2? I see about twice the speed using round robin with two paths instead of just a single path.

Br,
Rune

-----Original Message-----
From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Michael Rasmussen
Sent: Sunday, November 09, 2014 11:02 PM
To: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] infiniband

On Mon, 10 Nov 2014 04:38:23 +0000
Rune Tipsmark <rt at steait.net> wrote:

> What network throughput were you looking at before the tweaking?
> 
Raised my performance from 5.2 gbps to 7.9 gbps (50% performance
increase)

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
Who needs friends when you can sit alone in your room and drink?

From rt at steait.net  Wed Nov 12 22:49:27 2014
From: rt at steait.net (Rune Tipsmark)
Date: Wed, 12 Nov 2014 22:49:27 +0000
Subject: [OmniOS-discuss] slog limits write speed more than it should
Message-ID: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>

Hi all,

Got a problem... with my pool using sync=always I see a max write speed of about 6000 IOPS (64KB block size) during storage vMotion.
I doesn't matter if I have one, two or three SLOGs, if  I use one it will just do ~6000 w/s, 0% busy (SLC IO Drive), if I use two of these each will do ~3000 w/s or so... if I use 3  each will  do ~2000 w/s or so... the underlying disks in the pool are maybe 10-15% busy tops...
If I disable sync I see a tasks such as vMotion for a 40GB vm drop from 3-4 minutes to less than 1 minute.

I can't see  where the problem is, each SLOG device can easily handle much more than 6000 IOPS and vSphere can easily get vMotion done faster when sync is disabled.... So essentially SLOGs aren't busy, disks aren't busy... sync just kills the speed... what parameters can I  tweak to push this performance up significantly? I tried 512B, 4KB, 64KB and 128KB block sizes, 64KB (default) seems to have the best performance.

Any ideas welcome.
Br,
Rune



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141112/23eeb799/attachment.html>

From bfriesen at simple.dallas.tx.us  Wed Nov 12 23:16:49 2014
From: bfriesen at simple.dallas.tx.us (Bob Friesenhahn)
Date: Wed, 12 Nov 2014 17:16:49 -0600 (CST)
Subject: [OmniOS-discuss] slog limits write speed more than it should
In-Reply-To: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>
References: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>
Message-ID: <alpine.GSO.2.01.1411121714550.4045@freddy.simplesystems.org>

On Wed, 12 Nov 2014, Rune Tipsmark wrote:
> 
> each SLOG device can easily handle much more than 6000 IOPS

Where may we find the specifications for the SSDs you are using?

6000 IOPS sounds like it might be quite a lot, depending on the SSD 
used.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

From rt at steait.net  Wed Nov 12 23:21:54 2014
From: rt at steait.net (Rune Tipsmark)
Date: Wed, 12 Nov 2014 23:21:54 +0000
Subject: [OmniOS-discuss] slog limits write speed more than it should
In-Reply-To: <alpine.GSO.2.01.1411121714550.4045@freddy.simplesystems.org>
References: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>
	<alpine.GSO.2.01.1411121714550.4045@freddy.simplesystems.org>
Message-ID: <8a28a83fd09b474da1cfc9d1d14af37f@EX1301.steait.net>

http://www.fusionio.com/products/iodrive

160GB SLC

-----Original Message-----
From: Bob Friesenhahn [mailto:bfriesen at simple.dallas.tx.us] 
Sent: Wednesday, November 12, 2014 3:17 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] slog limits write speed more than it should

On Wed, 12 Nov 2014, Rune Tipsmark wrote:
> 
> each SLOG device can easily handle much more than 6000 IOPS

Where may we find the specifications for the SSDs you are using?

6000 IOPS sounds like it might be quite a lot, depending on the SSD used.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

From mir at miras.org  Wed Nov 12 23:47:53 2014
From: mir at miras.org (Michael Rasmussen)
Date: Thu, 13 Nov 2014 00:47:53 +0100
Subject: [OmniOS-discuss] infiniband
In-Reply-To: <763137907c994ee4b2f03d8f819b4eec@EX1301.steait.net>
References: <20141107234951.26ef4aa0@sleipner.datanom.net>
	<20141110000626.1d1b7424@sleipner.datanom.net>
	<20141110022041.427e1505@sleipner.datanom.net>
	<95f5a26aa2fd438eb6dc0719349e91c3@EX1301.steait.net>
	<20141110080206.50e6f07a@sleipner.datanom.net>
	<649b4fab1100440197fcc2ba70040f0c@EX1301.steait.net>
	<20141113000000.477941ed@sleipner.datanom.net>
	<763137907c994ee4b2f03d8f819b4eec@EX1301.steait.net>
Message-ID: <20141113004753.08c41de6@sleipner.datanom.net>

How exactly do you configure this?
Is a switch required?

On Wed, 12 Nov 2014 23:19:24 +0000
Rune Tipsmark <rt at steait.net> wrote:

> Multipath, round robin.
> You can create multiple IPoIB partitions... if you want 4 on each port
> 
> dladm create-part -l ibp0 -P 0xFFFF pFFFF.ibp0
> dladm create-part -l ibp0 -P 0xFFFF pFFFF.ibp2
> dladm create-part -l ibp0 -P 0xFFFF pFFFF.ibp4
> dladm create-part -l ibp0 -P 0xFFFF pFFFF.ibp6
> dladm create-part -l ibp1 -P 0xFFFF pFFFF.ibp1
> dladm create-part -l ibp1 -P 0xFFFF pFFFF.ibp3
> dladm create-part -l ibp1 -P 0xFFFF pFFFF.ibp5
> dladm create-part -l ibp1 -P 0xFFFF pFFFF.ibp7
> 
> 
> -----Original Message-----
> From: Michael Rasmussen [mailto:mir at miras.org] 
> Sent: Wednesday, November 12, 2014 3:00 PM
> To: Rune Tipsmark
> Subject: Re: [OmniOS-discuss] infiniband
> 
> On Wed, 12 Nov 2014 22:35:46 +0000
> Rune Tipsmark <rt at steait.net> wrote:
> 
> > 
> > Did you try creating more partitions say 4 partitions to see if it performs better than 2? I see about twice the speed using round robin with two paths instead of just a single path.
> > 
> What do you mean by partitions?
> By two paths do you mean multipath or bonding?
> 


-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
index, n.:
	Alphabetical list of words of no possible interest where an
	alphabetical list of subjects with references ought to be.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141113/ef66edc9/attachment.bin>

From rt at steait.net  Thu Nov 13 00:22:40 2014
From: rt at steait.net (Rune Tipsmark)
Date: Thu, 13 Nov 2014 00:22:40 +0000
Subject: [OmniOS-discuss] infiniband
In-Reply-To: <20141113004753.08c41de6@sleipner.datanom.net>
References: <20141107234951.26ef4aa0@sleipner.datanom.net>
	<20141110000626.1d1b7424@sleipner.datanom.net>
	<20141110022041.427e1505@sleipner.datanom.net>
	<95f5a26aa2fd438eb6dc0719349e91c3@EX1301.steait.net>
	<20141110080206.50e6f07a@sleipner.datanom.net>
	<649b4fab1100440197fcc2ba70040f0c@EX1301.steait.net>
	<20141113000000.477941ed@sleipner.datanom.net>
	<763137907c994ee4b2f03d8f819b4eec@EX1301.steait.net>
	<20141113004753.08c41de6@sleipner.datanom.net>
Message-ID: <509c9ff8b05f42c185de71e891345172@EX1301.steait.net>

Nope, just a subnet manager...

I would do something like this

dladm create-part -l ibp0 -P 0xFFFF pFFFF.ibp0
dladm create-part -l ibp0 -P 0xFFFF pFFFF.ibp2
dladm create-part -l ibp0 -P 0xFFFF pFFFF.ibp4
dladm create-part -l ibp0 -P 0xFFFF pFFFF.ibp6
dladm create-part -l ibp1 -P 0xFFFF pFFFF.ibp1
dladm create-part -l ibp1 -P 0xFFFF pFFFF.ibp3
dladm create-part -l ibp1 -P 0xFFFF pFFFF.ibp5
dladm create-part -l ibp1 -P 0xFFFF pFFFF.ibp7

ipadm create-if pFFFF.ibp0
ipadm create-if pFFFF.ibp1
ipadm create-if pFFFF.ibp2
ipadm create-if pFFFF.ibp3
ipadm create-if pFFFF.ibp4
ipadm create-if pFFFF.ibp5
ipadm create-if pFFFF.ibp6
ipadm create-if pFFFF.ibp7

ipadm create-addr -T static -a 10.98.0.10 pFFFF.ibp0/ipv4
ipadm create-addr -T static -a 10.99.0.10 pFFFF.ibp1/ipv4
ipadm create-addr -T static -a 10.98.0.12 pFFFF.ibp2/ipv4
ipadm create-addr -T static -a 10.99.0.12 pFFFF.ibp3/ipv4
ipadm create-addr -T static -a 10.98.0.14 pFFFF.ibp4/ipv4
ipadm create-addr -T static -a 10.99.0.14 pFFFF.ibp5/ipv4
ipadm create-addr -T static -a 10.98.0.16 pFFFF.ibp6/ipv4
ipadm create-addr -T static -a 10.99.0.16 pFFFF.ibp7/ipv4


-----Original Message-----
From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Michael Rasmussen
Sent: Wednesday, November 12, 2014 3:48 PM
To: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] infiniband

How exactly do you configure this?
Is a switch required?

On Wed, 12 Nov 2014 23:19:24 +0000
Rune Tipsmark <rt at steait.net> wrote:

> Multipath, round robin.
> You can create multiple IPoIB partitions... if you want 4 on each port
> 
> dladm create-part -l ibp0 -P 0xFFFF pFFFF.ibp0 dladm create-part -l 
> ibp0 -P 0xFFFF pFFFF.ibp2 dladm create-part -l ibp0 -P 0xFFFF 
> pFFFF.ibp4 dladm create-part -l ibp0 -P 0xFFFF pFFFF.ibp6 dladm 
> create-part -l ibp1 -P 0xFFFF pFFFF.ibp1 dladm create-part -l ibp1 -P 
> 0xFFFF pFFFF.ibp3 dladm create-part -l ibp1 -P 0xFFFF pFFFF.ibp5 dladm 
> create-part -l ibp1 -P 0xFFFF pFFFF.ibp7
> 
> 
> -----Original Message-----
> From: Michael Rasmussen [mailto:mir at miras.org]
> Sent: Wednesday, November 12, 2014 3:00 PM
> To: Rune Tipsmark
> Subject: Re: [OmniOS-discuss] infiniband
> 
> On Wed, 12 Nov 2014 22:35:46 +0000
> Rune Tipsmark <rt at steait.net> wrote:
> 
> > 
> > Did you try creating more partitions say 4 partitions to see if it performs better than 2? I see about twice the speed using round robin with two paths instead of just a single path.
> > 
> What do you mean by partitions?
> By two paths do you mean multipath or bonding?
> 


--
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
index, n.:
	Alphabetical list of words of no possible interest where an
	alphabetical list of subjects with references ought to be.

From mir at miras.org  Thu Nov 13 00:44:56 2014
From: mir at miras.org (Michael Rasmussen)
Date: Thu, 13 Nov 2014 01:44:56 +0100
Subject: [OmniOS-discuss] infiniband
In-Reply-To: <509c9ff8b05f42c185de71e891345172@EX1301.steait.net>
References: <20141107234951.26ef4aa0@sleipner.datanom.net>
	<20141110000626.1d1b7424@sleipner.datanom.net>
	<20141110022041.427e1505@sleipner.datanom.net>
	<95f5a26aa2fd438eb6dc0719349e91c3@EX1301.steait.net>
	<20141110080206.50e6f07a@sleipner.datanom.net>
	<649b4fab1100440197fcc2ba70040f0c@EX1301.steait.net>
	<20141113000000.477941ed@sleipner.datanom.net>
	<763137907c994ee4b2f03d8f819b4eec@EX1301.steait.net>
	<20141113004753.08c41de6@sleipner.datanom.net>
	<509c9ff8b05f42c185de71e891345172@EX1301.steait.net>
Message-ID: <20141113014456.738afed6@sleipner.datanom.net>

On Thu, 13 Nov 2014 00:22:40 +0000
Rune Tipsmark <rt at steait.net> wrote:

> 
> ipadm create-addr -T static -a 10.98.0.10 pFFFF.ibp0/ipv4
> ipadm create-addr -T static -a 10.99.0.10 pFFFF.ibp1/ipv4
> ipadm create-addr -T static -a 10.98.0.12 pFFFF.ibp2/ipv4
> ipadm create-addr -T static -a 10.99.0.12 pFFFF.ibp3/ipv4
> ipadm create-addr -T static -a 10.98.0.14 pFFFF.ibp4/ipv4
> ipadm create-addr -T static -a 10.99.0.14 pFFFF.ibp5/ipv4
> ipadm create-addr -T static -a 10.98.0.16 pFFFF.ibp6/ipv4
> ipadm create-addr -T static -a 10.99.0.16 pFFFF.ibp7/ipv4
> 
What is the subnet, /16 or /24

From my findings you cannot have both ports on the same subnet so I guess this means /24,
right?

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
You will receive a legacy which will place you above want.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141113/09ff7102/attachment-0001.bin>

From mir at miras.org  Thu Nov 13 00:55:30 2014
From: mir at miras.org (Michael Rasmussen)
Date: Thu, 13 Nov 2014 01:55:30 +0100
Subject: [OmniOS-discuss] infiniband
In-Reply-To: <509c9ff8b05f42c185de71e891345172@EX1301.steait.net>
References: <20141107234951.26ef4aa0@sleipner.datanom.net>
	<20141110000626.1d1b7424@sleipner.datanom.net>
	<20141110022041.427e1505@sleipner.datanom.net>
	<95f5a26aa2fd438eb6dc0719349e91c3@EX1301.steait.net>
	<20141110080206.50e6f07a@sleipner.datanom.net>
	<649b4fab1100440197fcc2ba70040f0c@EX1301.steait.net>
	<20141113000000.477941ed@sleipner.datanom.net>
	<763137907c994ee4b2f03d8f819b4eec@EX1301.steait.net>
	<20141113004753.08c41de6@sleipner.datanom.net>
	<509c9ff8b05f42c185de71e891345172@EX1301.steait.net>
Message-ID: <20141113015530.48f9b7d8@sleipner.datanom.net>

On Thu, 13 Nov 2014 00:22:40 +0000
Rune Tipsmark <rt at steait.net> wrote:

> 
> ipadm create-addr -T static -a 10.98.0.10 pFFFF.ibp0/ipv4
> ipadm create-addr -T static -a 10.99.0.10 pFFFF.ibp1/ipv4
> ipadm create-addr -T static -a 10.98.0.12 pFFFF.ibp2/ipv4
> ipadm create-addr -T static -a 10.99.0.12 pFFFF.ibp3/ipv4
> ipadm create-addr -T static -a 10.98.0.14 pFFFF.ibp4/ipv4
> ipadm create-addr -T static -a 10.99.0.14 pFFFF.ibp5/ipv4
> ipadm create-addr -T static -a 10.98.0.16 pFFFF.ibp6/ipv4
> ipadm create-addr -T static -a 10.99.0.16 pFFFF.ibp7/ipv4
> 
What are the next steps? IPMP? or something else?

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
Win95 is not a virus; a virus does something.
		-- unknown source
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141113/87873bb3/attachment.bin>

From rt at steait.net  Thu Nov 13 01:12:19 2014
From: rt at steait.net (Rune Tipsmark)
Date: Thu, 13 Nov 2014 01:12:19 +0000
Subject: [OmniOS-discuss] infiniband
In-Reply-To: <20141113014456.738afed6@sleipner.datanom.net>
References: <20141107234951.26ef4aa0@sleipner.datanom.net>
	<20141110000626.1d1b7424@sleipner.datanom.net>
	<20141110022041.427e1505@sleipner.datanom.net>
	<95f5a26aa2fd438eb6dc0719349e91c3@EX1301.steait.net>
	<20141110080206.50e6f07a@sleipner.datanom.net>
	<649b4fab1100440197fcc2ba70040f0c@EX1301.steait.net>
	<20141113000000.477941ed@sleipner.datanom.net>
	<763137907c994ee4b2f03d8f819b4eec@EX1301.steait.net>
	<20141113004753.08c41de6@sleipner.datanom.net>
	<509c9ff8b05f42c185de71e891345172@EX1301.steait.net>
	<20141113014456.738afed6@sleipner.datanom.net>
Message-ID: <ec6a5d6f56fd420a90213c49694f0640@EX1301.steait.net>

Yes /24 and I also do the same as you, diff subnets for each port, else its game over...

-----Original Message-----
From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Michael Rasmussen
Sent: Wednesday, November 12, 2014 4:45 PM
To: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] infiniband

On Thu, 13 Nov 2014 00:22:40 +0000
Rune Tipsmark <rt at steait.net> wrote:

> 
> ipadm create-addr -T static -a 10.98.0.10 pFFFF.ibp0/ipv4 ipadm 
> create-addr -T static -a 10.99.0.10 pFFFF.ibp1/ipv4 ipadm create-addr 
> -T static -a 10.98.0.12 pFFFF.ibp2/ipv4 ipadm create-addr -T static -a 
> 10.99.0.12 pFFFF.ibp3/ipv4 ipadm create-addr -T static -a 10.98.0.14 
> pFFFF.ibp4/ipv4 ipadm create-addr -T static -a 10.99.0.14 
> pFFFF.ibp5/ipv4 ipadm create-addr -T static -a 10.98.0.16 
> pFFFF.ibp6/ipv4 ipadm create-addr -T static -a 10.99.0.16 
> pFFFF.ibp7/ipv4
> 
What is the subnet, /16 or /24

>From my findings you cannot have both ports on the same subnet so I guess this means /24, right?

--
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
You will receive a legacy which will place you above want.

From rt at steait.net  Thu Nov 13 01:59:20 2014
From: rt at steait.net (Rune Tipsmark)
Date: Thu, 13 Nov 2014 01:59:20 +0000
Subject: [OmniOS-discuss] infiniband
In-Reply-To: <20141113015530.48f9b7d8@sleipner.datanom.net>
References: <20141107234951.26ef4aa0@sleipner.datanom.net>
	<20141110000626.1d1b7424@sleipner.datanom.net>
	<20141110022041.427e1505@sleipner.datanom.net>
	<95f5a26aa2fd438eb6dc0719349e91c3@EX1301.steait.net>
	<20141110080206.50e6f07a@sleipner.datanom.net>
	<649b4fab1100440197fcc2ba70040f0c@EX1301.steait.net>
	<20141113000000.477941ed@sleipner.datanom.net>
	<763137907c994ee4b2f03d8f819b4eec@EX1301.steait.net>
	<20141113004753.08c41de6@sleipner.datanom.net>
	<509c9ff8b05f42c185de71e891345172@EX1301.steait.net>
	<20141113015530.48f9b7d8@sleipner.datanom.net>
Message-ID: <fdfb3a3e94984b61927d6f6484afe39f@EX1301.steait.net>

Not sure about IPMP, I would just use the same number of partitions on my ESX and round robin the whole thing...

Do you have more info on the network tweaks you did earlier? With iSER I see a slightly higher response time than with SRP but lower spikes.
Throughput is 10% higher and CPU 50% lower, so I might just stick with iSER from now on.


-----Original Message-----
From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Michael Rasmussen
Sent: Wednesday, November 12, 2014 4:56 PM
To: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] infiniband

On Thu, 13 Nov 2014 00:22:40 +0000
Rune Tipsmark <rt at steait.net> wrote:

> 
> ipadm create-addr -T static -a 10.98.0.10 pFFFF.ibp0/ipv4 ipadm 
> create-addr -T static -a 10.99.0.10 pFFFF.ibp1/ipv4 ipadm create-addr 
> -T static -a 10.98.0.12 pFFFF.ibp2/ipv4 ipadm create-addr -T static -a 
> 10.99.0.12 pFFFF.ibp3/ipv4 ipadm create-addr -T static -a 10.98.0.14 
> pFFFF.ibp4/ipv4 ipadm create-addr -T static -a 10.99.0.14 
> pFFFF.ibp5/ipv4 ipadm create-addr -T static -a 10.98.0.16 
> pFFFF.ibp6/ipv4 ipadm create-addr -T static -a 10.99.0.16 
> pFFFF.ibp7/ipv4
> 
What are the next steps? IPMP? or something else?

--
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
Win95 is not a virus; a virus does something.
		-- unknown source

From mir at miras.org  Thu Nov 13 02:15:01 2014
From: mir at miras.org (Michael Rasmussen)
Date: Thu, 13 Nov 2014 03:15:01 +0100
Subject: [OmniOS-discuss] infiniband
In-Reply-To: <fdfb3a3e94984b61927d6f6484afe39f@EX1301.steait.net>
References: <20141107234951.26ef4aa0@sleipner.datanom.net>
	<20141110000626.1d1b7424@sleipner.datanom.net>
	<20141110022041.427e1505@sleipner.datanom.net>
	<95f5a26aa2fd438eb6dc0719349e91c3@EX1301.steait.net>
	<20141110080206.50e6f07a@sleipner.datanom.net>
	<649b4fab1100440197fcc2ba70040f0c@EX1301.steait.net>
	<20141113000000.477941ed@sleipner.datanom.net>
	<763137907c994ee4b2f03d8f819b4eec@EX1301.steait.net>
	<20141113004753.08c41de6@sleipner.datanom.net>
	<509c9ff8b05f42c185de71e891345172@EX1301.steait.net>
	<20141113015530.48f9b7d8@sleipner.datanom.net>
	<fdfb3a3e94984b61927d6f6484afe39f@EX1301.steait.net>
Message-ID: <20141113031501.6172f257@sleipner.datanom.net>

On Thu, 13 Nov 2014 01:59:20 +0000
Rune Tipsmark <rt at steait.net> wrote:

> Not sure about IPMP, I would just use the same number of partitions on my ESX and round robin the whole thing...
> 
I don't use ESX so I will have to find out how this is done in Linux.
> Do you have more info on the network tweaks you did earlier? With iSER I see a slightly higher response time than with SRP but lower spikes.
> Throughput is 10% higher and CPU 50% lower, so I might just stick with iSER from now on.
> 
The network tweaks was something I found via a google search


-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
<Teller> where am I and what am I doing in this handbasket?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141113/a88a4288/attachment.bin>

From headlesscow at gmail.com  Thu Nov 13 05:24:28 2014
From: headlesscow at gmail.com (Nicholas George)
Date: Thu, 13 Nov 2014 00:24:28 -0500
Subject: [OmniOS-discuss] slog limits write speed more than it should
In-Reply-To: <8a28a83fd09b474da1cfc9d1d14af37f@EX1301.steait.net>
References: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>
	<alpine.GSO.2.01.1411121714550.4045@freddy.simplesystems.org>
	<8a28a83fd09b474da1cfc9d1d14af37f@EX1301.steait.net>
Message-ID: <CAEyWAj1q3=VxnxcciNVRxHGfDRAjnAEj__QJQmhwq=RNT2kt9A@mail.gmail.com>

>From what I understand, ZFS writes to the intent log in a single threaded
synchronous fashion. Adding three drives just means that the first block is
written to the first drive synchronously, then the second block to the
second drive synchronously, then the third to the third drive and then back
around to the first. You don't actually increase your throughput beyond the
performance of a single drive.

http://nex7.blogspot.com/2013/04/zfs-intent-log.html
http://www.nexentastor.org/boards/5/topics/6179

On Wed, Nov 12, 2014 at 6:21 PM, Rune Tipsmark <rt at steait.net> wrote:

> http://www.fusionio.com/products/iodrive
>
> 160GB SLC
>
> -----Original Message-----
> From: Bob Friesenhahn [mailto:bfriesen at simple.dallas.tx.us]
> Sent: Wednesday, November 12, 2014 3:17 PM
> To: Rune Tipsmark
> Cc: omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] slog limits write speed more than it should
>
> On Wed, 12 Nov 2014, Rune Tipsmark wrote:
> >
> > each SLOG device can easily handle much more than 6000 IOPS
>
> Where may we find the specifications for the SSDs you are using?
>
> 6000 IOPS sounds like it might be quite a lot, depending on the SSD used.
>
> Bob
> --
> Bob Friesenhahn
> bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141113/a6e176b5/attachment.html>

From a.shvayakov at yahoo.com  Thu Nov 13 09:24:48 2014
From: a.shvayakov at yahoo.com (Shvayakov A.)
Date: Thu, 13 Nov 2014 12:24:48 +0300
Subject: [OmniOS-discuss] I can't add additional network card
In-Reply-To: <A00EFCA5-B3A9-41CA-AC87-1CF3A78A6D83@omniti.com>
References: <54639C8D.3000106@yahoo.com>
	<A00EFCA5-B3A9-41CA-AC87-1CF3A78A6D83@omniti.com>
Message-ID: <546478E0.8050503@yahoo.com>


On 12.11.2014 21:16, Dan McDonald wrote:
 > So let me get this straight --> booting the r151012 live CD lets you 
see two ixgbe ports and four igb ports?  But the installed system only 
lets you see two igb ports and two ixgbe ports?
Yes, and this is a problem.

I don't know how to control it.

I tried to copy the file /etc/path_to_inst from live CD, butit didn't help
 >
 > What does "dladm show-ether" have to say?
dladm show-ether
LINK            PTYPE    STATE    AUTO SPEED-DUPLEX                    PAUSE
interlink0      current  up       yes 1G-f                            bi
extnet0         current  up       yes 1G-f                            bi
ixgbe0          current  up       yes 10G-f                           bi
ixgbe1          current  up       yes 10G-f                           bi

 > And since two of the igbs appear not to be attached, can you "grep 
igb /var/adm/messages*" and see if there are any kernel complaints about it?
Nov 12 17:19:18 ad999 mac: [ID 469746 kern.info] NOTICE: igb0 registered
Nov 12 17:19:18 ad999 igb: [ID 611667 kern.info] NOTICE: igb0: igb 2.3.8-ish
Nov 12 17:19:19 ad999 mac: [ID 469746 kern.info] NOTICE: igb1 registered
Nov 12 17:19:19 ad999 igb: [ID 611667 kern.info] NOTICE: igb1: igb 2.3.8-ish
Nov 12 17:19:21 ad999 mac: [ID 435574 kern.info] NOTICE: igb0 link up, 
1000 Mbps, full duplex
Nov 12 17:19:22 ad999 mac: [ID 435574 kern.info] NOTICE: igb1 link up, 
1000 Mbps, full duplex
 >
 > Thanks,
 > Dan
 >



From danmcd at omniti.com  Thu Nov 13 14:20:18 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Thu, 13 Nov 2014 09:20:18 -0500
Subject: [OmniOS-discuss] I can't add additional network card
In-Reply-To: <546478E0.8050503@yahoo.com>
References: <54639C8D.3000106@yahoo.com>
	<A00EFCA5-B3A9-41CA-AC87-1CF3A78A6D83@omniti.com>
	<546478E0.8050503@yahoo.com>
Message-ID: <D3C37AFD-A944-40E6-9622-AEE7B59D758C@omniti.com>


> On Nov 13, 2014, at 4:24 AM, Shvayakov A. <a.shvayakov at yahoo.com> wrote:
> 
> 
> On 12.11.2014 21:16, Dan McDonald wrote:
> > So let me get this straight --> booting the r151012 live CD lets you see two ixgbe ports and four igb ports?  But the installed system only lets you see two igb ports and two ixgbe ports?
> Yes, and this is a problem.
> 
> I don't know how to control it.
> 
> I tried to copy the file /etc/path_to_inst from live CD, butit didn't help

Utter "devfsadm" on your system, and see if there's any change, or any additional output in /etc/driver_aliases.

There has to be some reason the driver isn't attaching...

Dan


From rt at steait.net  Fri Nov 14 07:58:34 2014
From: rt at steait.net (Rune Tipsmark)
Date: Fri, 14 Nov 2014 07:58:34 +0000
Subject: [OmniOS-discuss] slog limits write speed more than it should
In-Reply-To: <CAEyWAj1q3=VxnxcciNVRxHGfDRAjnAEj__QJQmhwq=RNT2kt9A@mail.gmail.com>
References: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>
	<alpine.GSO.2.01.1411121714550.4045@freddy.simplesystems.org>
	<8a28a83fd09b474da1cfc9d1d14af37f@EX1301.steait.net>
	<CAEyWAj1q3=VxnxcciNVRxHGfDRAjnAEj__QJQmhwq=RNT2kt9A@mail.gmail.com>
Message-ID: <62474cc7631c4c9e8484be553d0efb91@EX1301.steait.net>

Looking at the first link he clearly states that ZIL is written to in a round robin fashion, I was under the impression that 2 log devices would then be faster than 1?unless mirrored of course.

If this is not true, what is the point of allowing more than 1 vdev as log device at all?

How can I test the max sped of my log device directly in OmniOS?

Br,
Rune

From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Nicholas George
Sent: Wednesday, November 12, 2014 9:24 PM
To: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] slog limits write speed more than it should

From what I understand, ZFS writes to the intent log in a single threaded synchronous fashion. Adding three drives just means that the first block is written to the first drive synchronously, then the second block to the second drive synchronously, then the third to the third drive and then back around to the first. You don't actually increase your throughput beyond the performance of a single drive.

http://nex7.blogspot.com/2013/04/zfs-intent-log.html
http://www.nexentastor.org/boards/5/topics/6179

On Wed, Nov 12, 2014 at 6:21 PM, Rune Tipsmark <rt at steait.net<mailto:rt at steait.net>> wrote:
http://www.fusionio.com/products/iodrive

160GB SLC

-----Original Message-----
From: Bob Friesenhahn [mailto:bfriesen at simple.dallas.tx.us<mailto:bfriesen at simple.dallas.tx.us>]
Sent: Wednesday, November 12, 2014 3:17 PM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Re: [OmniOS-discuss] slog limits write speed more than it should

On Wed, 12 Nov 2014, Rune Tipsmark wrote:
>
> each SLOG device can easily handle much more than 6000 IOPS

Where may we find the specifications for the SSDs you are using?

6000 IOPS sounds like it might be quite a lot, depending on the SSD used.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us<mailto:bfriesen at simple.dallas.tx.us>, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141114/a57459e5/attachment.html>

From a.shvayakov at yahoo.com  Fri Nov 14 10:06:44 2014
From: a.shvayakov at yahoo.com (Shvayakov A.)
Date: Fri, 14 Nov 2014 13:06:44 +0300
Subject: [OmniOS-discuss] Is this ZFS bug?
Message-ID: <5465D434.2000706@yahoo.com>

Hello,

OmniOS v11 r151010


Can somebody help determine the cause of the growth of the file to this 
size 1.6P?

Without reasons happened increasing file to unbelievably large size
This is qcow2 image of the virtual machine. The KVM hypervisor.
I copied normal file from a snapshot, but a few days again grew size
This was repeated 3 times.
This is repeated with only one this file
I'm worried

Monitoring doesn't show anything similar to loading of  large amount of data

My diagnosis - this is file system error

But scrub does not detect errors
scan: scrub repaired 0 in 17h32m with 0 errors on Thu Nov 13 04:29:25 2014

ls -lsh /GIGO1/p1/eb06e74c-4405-4a03-b5a3-ca7a61648222.saved
7.0G -rwxr--r-- 1 root root 1.6P Nov 11 19:45 
/GIGO1/p1/eb06e74c-4405-4a03-b5a3-ca7a61648222.saved


zdb -U /opt/HAC/RSF-1/etc/volume-cache/GIGO1.cache -dddd GIGO1/p1 84
Dataset GIGO1/p1 [ZPL], ID 42, cr_txg 84, 3.80T, 287 objects, rootbp 
DVA[0]=<1:23259e3da00:200> DVA[1]=<2:1671aa8f800:200> [L0 DMU objset] 
fletcher4 lzjb LE contiguous unique double size=800L/200P 
birth=10292996L/10292996P fill=287 
cksum=18dde56a53:7b65ea8cfe3:14c1ab20eacd2:27d1fa7035c252

     Object  lvl   iblk   dblk  dsize  lsize   %full  type
         84    6    16K   128K  6.94G  1.57P    0.00  ZFS plain file
                                         168   bonus  System attributes
         dnode flags: USED_BYTES USERUSED_ACCOUNTED
         dnode maxblkid: 13466419270
         path    /eb06e74c-4405-4a03-b5a3-ca7a61648222.saved
         uid     0
         gid     0
         atime   Wed Sep 24 16:50:25 2014
         mtime   Tue Nov 11 19:45:41 2014
         ctime   Tue Nov 11 22:00:00 2014
         crtime  Wed Sep 24 16:50:25 2014
         gen     7044582
         mode    100744
         size    1765070506688512
         parent  4
         links   1
         pflags  40800000004



zpool list GIGO1
NAME    SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
GIGO1  8.16T  4.33T  3.83T         -    53%  1.00x  ONLINE  -


zfs get all GIGO1/p1
NAME      PROPERTY 
VALUE                                                    SOURCE
GIGO1/p1  type filesystem                                               -
GIGO1/p1  creation              Mon Sep 16 17:49 
2013                                    -
GIGO1/p1  used 4.23T                                                    -
GIGO1/p1  available 
2.86T                                                    -
GIGO1/p1  referenced 
3.80T                                                    -
GIGO1/p1  compressratio 
1.47x                                                    -
GIGO1/p1  mounted yes                                                      -
GIGO1/p1  quota none                                                     
default
GIGO1/p1  reservation 
none                                                     default
GIGO1/p1  recordsize 
128K                                                     default
GIGO1/p1  mountpoint 
/GIGO1/p1                                                default
GIGO1/p1  sharenfs anon=nobody,sec=sys,rw=@XXXXX.0/24,root=@XXXX/24  local
GIGO1/p1  checksum 
on                                                       default
GIGO1/p1  compression 
off                                                      local
GIGO1/p1  atime off                                                      
inherited from GIGO1
GIGO1/p1  devices 
off                                                      inherited from 
GIGO1
GIGO1/p1  exec off                                                      
inherited from GIGO1
GIGO1/p1  setuid 
off                                                      inherited from 
GIGO1
GIGO1/p1  readonly 
off                                                      default
GIGO1/p1  zoned off                                                      
default
GIGO1/p1  snapdir 
hidden                                                   default
GIGO1/p1  aclmode 
discard                                                  default
GIGO1/p1  aclinherit 
restricted                                               default
GIGO1/p1  canmount 
on                                                       default
GIGO1/p1  xattr off                                                      
inherited from GIGO1
GIGO1/p1  copies 
1                                                        default
GIGO1/p1  version 5                                                        -
GIGO1/p1  utf8only 
off                                                      -
GIGO1/p1  normalization 
none                                                     -
GIGO1/p1  casesensitivity 
sensitive                                                -
GIGO1/p1  vscan off                                                      
default
GIGO1/p1  nbmand 
off                                                      default
GIGO1/p1  sharesmb 
off                                                      default
GIGO1/p1  refquota 
none                                                     default
GIGO1/p1  refreservation 
none                                                     default
GIGO1/p1  primarycache 
all                                                      default
GIGO1/p1  secondarycache 
all                                                      default
GIGO1/p1  usedbysnapshots 
442G                                                     -
GIGO1/p1  usedbydataset 
3.80T                                                    -
GIGO1/p1  usedbychildren 
0                                                        -
GIGO1/p1  usedbyrefreservation 
0                                                        -
GIGO1/p1  logbias 
latency                                                  default
GIGO1/p1  dedup off                                                      
default
GIGO1/p1  mlslabel 
none                                                     default
GIGO1/p1  sync standard                                                 
default
GIGO1/p1  refcompressratio 
1.43x                                                    -
GIGO1/p1  written 59.5G                                                    -
GIGO1/p1  logicalused 
6.22T                                                    -
GIGO1/p1  logicalreferenced 
5.44T                                                    -

From bfriesen at simple.dallas.tx.us  Fri Nov 14 14:47:47 2014
From: bfriesen at simple.dallas.tx.us (Bob Friesenhahn)
Date: Fri, 14 Nov 2014 08:47:47 -0600 (CST)
Subject: [OmniOS-discuss] slog limits write speed more than it should
In-Reply-To: <62474cc7631c4c9e8484be553d0efb91@EX1301.steait.net>
References: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>
	<alpine.GSO.2.01.1411121714550.4045@freddy.simplesystems.org>
	<8a28a83fd09b474da1cfc9d1d14af37f@EX1301.steait.net>
	<CAEyWAj1q3=VxnxcciNVRxHGfDRAjnAEj__QJQmhwq=RNT2kt9A@mail.gmail.com>
	<62474cc7631c4c9e8484be553d0efb91@EX1301.steait.net>
Message-ID: <alpine.GSO.2.01.1411140844060.18732@freddy.simplesystems.org>

On Fri, 14 Nov 2014, Rune Tipsmark wrote:

> 
> Looking at the first link he clearly states that ZIL is written to in a round robin fashion, I was under the
> impression that 2 log devices would then be faster than 1?unless mirrored of course.

This could be a mistaken impression.  The slog commits each write 
before proceeding to the next write.  If the involved SSDs have a 
fixed minimum write latency, then this would limit the maximum 
transaction rate regardless of the number of SSDs involved.  There 
could be an advantage to more SSDs if the additional time between 
writes allows the SSD to more effectively prepare for the next write 
and reduce the write latency.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

From rt at steait.net  Fri Nov 14 17:46:33 2014
From: rt at steait.net (Rune Tipsmark)
Date: Fri, 14 Nov 2014 17:46:33 +0000
Subject: [OmniOS-discuss] slog limits write speed more than it should
In-Reply-To: <alpine.GSO.2.01.1411140844060.18732@freddy.simplesystems.org>
References: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>
	<alpine.GSO.2.01.1411121714550.4045@freddy.simplesystems.org>
	<8a28a83fd09b474da1cfc9d1d14af37f@EX1301.steait.net>
	<CAEyWAj1q3=VxnxcciNVRxHGfDRAjnAEj__QJQmhwq=RNT2kt9A@mail.gmail.com>
	<62474cc7631c4c9e8484be553d0efb91@EX1301.steait.net>
	<alpine.GSO.2.01.1411140844060.18732@freddy.simplesystems.org>
Message-ID: <730e31bc0c654d71b91edeffd39c9692@EX1301.steait.net>

Well that sucks... I guess one more reason to move to NV-Dimms to replace slow SLC cards.
Br,
Rune

-----Original Message-----
From: Bob Friesenhahn [mailto:bfriesen at simple.dallas.tx.us] 
Sent: Friday, November 14, 2014 6:48 AM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] slog limits write speed more than it should

On Fri, 14 Nov 2014, Rune Tipsmark wrote:

> 
> Looking at the first link he clearly states that ZIL is written to in 
> a round robin fashion, I was under the impression that 2 log devices would then be faster than 1?unless mirrored of course.

This could be a mistaken impression.  The slog commits each write before proceeding to the next write.  If the involved SSDs have a fixed minimum write latency, then this would limit the maximum transaction rate regardless of the number of SSDs involved.  There could be an advantage to more SSDs if the additional time between writes allows the SSD to more effectively prepare for the next write and reduce the write latency.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

From rt at steait.net  Fri Nov 14 19:53:19 2014
From: rt at steait.net (Rune Tipsmark)
Date: Fri, 14 Nov 2014 19:53:19 +0000
Subject: [OmniOS-discuss] slog limits write speed more than it should
In-Reply-To: <730e31bc0c654d71b91edeffd39c9692@EX1301.steait.net>
References: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>
	<alpine.GSO.2.01.1411121714550.4045@freddy.simplesystems.org>
	<8a28a83fd09b474da1cfc9d1d14af37f@EX1301.steait.net>
	<CAEyWAj1q3=VxnxcciNVRxHGfDRAjnAEj__QJQmhwq=RNT2kt9A@mail.gmail.com>
	<62474cc7631c4c9e8484be553d0efb91@EX1301.steait.net>
	<alpine.GSO.2.01.1411140844060.18732@freddy.simplesystems.org>
	<730e31bc0c654d71b91edeffd39c9692@EX1301.steait.net>
Message-ID: <c05908b42bf04de98dca40c78c99a92c@EX1301.steait.net>

Is there a way to stripe two block devices and use them as log?
I tried following this one https://blogs.oracle.com/bilke/entry/raid_0_stripe_on_solaris
But I cannot use the device in ZFS - getting the following error: cannot use '/dev/md/rdsk/zil1d0': must be a block device or regular file

Br,
Rune

-----Original Message-----
From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Rune Tipsmark
Sent: Friday, November 14, 2014 9:47 AM
To: Bob Friesenhahn
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] slog limits write speed more than it should

Well that sucks... I guess one more reason to move to NV-Dimms to replace slow SLC cards.
Br,
Rune

-----Original Message-----
From: Bob Friesenhahn [mailto:bfriesen at simple.dallas.tx.us]
Sent: Friday, November 14, 2014 6:48 AM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] slog limits write speed more than it should

On Fri, 14 Nov 2014, Rune Tipsmark wrote:

> 
> Looking at the first link he clearly states that ZIL is written to in 
> a round robin fashion, I was under the impression that 2 log devices would then be faster than 1?unless mirrored of course.

This could be a mistaken impression.  The slog commits each write before proceeding to the next write.  If the involved SSDs have a fixed minimum write latency, then this would limit the maximum transaction rate regardless of the number of SSDs involved.  There could be an advantage to more SSDs if the additional time between writes allows the SSD to more effectively prepare for the next write and reduce the write latency.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

From rt at steait.net  Fri Nov 14 23:06:04 2014
From: rt at steait.net (Rune Tipsmark)
Date: Fri, 14 Nov 2014 23:06:04 +0000
Subject: [OmniOS-discuss] slog limits write speed more than it should
In-Reply-To: <c05908b42bf04de98dca40c78c99a92c@EX1301.steait.net>
References: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>
	<alpine.GSO.2.01.1411121714550.4045@freddy.simplesystems.org>
	<8a28a83fd09b474da1cfc9d1d14af37f@EX1301.steait.net>
	<CAEyWAj1q3=VxnxcciNVRxHGfDRAjnAEj__QJQmhwq=RNT2kt9A@mail.gmail.com>
	<62474cc7631c4c9e8484be553d0efb91@EX1301.steait.net>
	<alpine.GSO.2.01.1411140844060.18732@freddy.simplesystems.org>
	<730e31bc0c654d71b91edeffd39c9692@EX1301.steait.net>
	<c05908b42bf04de98dca40c78c99a92c@EX1301.steait.net>
Message-ID: <bf78eb041097447cb27b0bb67b15407d@EX1301.steait.net>

I get only about half the bandwidth with Sync=Always compared to Sync=Disabled.

Using an SLC device that should perform better, its rated as 750 MB/sec, I only get something like 60% of that at best of times. If there is a way to stripe a ZIL it would be great. I have enough to mirror two stripes and get the speed I want.



Copying ~60GB from one LUN to another on same ZFS box.



Sync=Always:

[cid:image001.png at 01D0001C.7E3289F0]



Sync=Disabled

[cid:image002.png at 01D0001C.7E3289F0]



-----Original Message-----
From: Rune Tipsmark
Sent: Friday, November 14, 2014 11:53 AM
To: Rune Tipsmark; Bob Friesenhahn
Cc: omnios-discuss at lists.omniti.com
Subject: RE: [OmniOS-discuss] slog limits write speed more than it should



Is there a way to stripe two block devices and use them as log?

I tried following this one https://blogs.oracle.com/bilke/entry/raid_0_stripe_on_solaris

But I cannot use the device in ZFS - getting the following error: cannot use '/dev/md/rdsk/zil1d0': must be a block device or regular file



Br,

Rune



-----Original Message-----

From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Rune Tipsmark

Sent: Friday, November 14, 2014 9:47 AM

To: Bob Friesenhahn

Cc: omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>

Subject: Re: [OmniOS-discuss] slog limits write speed more than it should



Well that sucks... I guess one more reason to move to NV-Dimms to replace slow SLC cards.

Br,

Rune



-----Original Message-----

From: Bob Friesenhahn [mailto:bfriesen at simple.dallas.tx.us]

Sent: Friday, November 14, 2014 6:48 AM

To: Rune Tipsmark

Cc: omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>

Subject: Re: [OmniOS-discuss] slog limits write speed more than it should



On Fri, 14 Nov 2014, Rune Tipsmark wrote:



>

> Looking at the first link he clearly states that ZIL is written to in

> a round robin fashion, I was under the impression that 2 log devices would then be faster than 1?unless mirrored of course.



This could be a mistaken impression.  The slog commits each write before proceeding to the next write.  If the involved SSDs have a fixed minimum write latency, then this would limit the maximum transaction rate regardless of the number of SSDs involved.  There could be an advantage to more SSDs if the additional time between writes allows the SSD to more effectively prepare for the next write and reduce the write latency.



Bob

--

Bob Friesenhahn

bfriesen at simple.dallas.tx.us<mailto:bfriesen at simple.dallas.tx.us>, http://www.simplesystems.org/users/bfriesen/

GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

_______________________________________________

OmniOS-discuss mailing list

OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>

http://lists.omniti.com/mailman/listinfo/omnios-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141114/3941cc88/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 12899 bytes
Desc: image001.png
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141114/3941cc88/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 13179 bytes
Desc: image002.png
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141114/3941cc88/attachment-0003.png>

From mir at miras.org  Fri Nov 14 23:49:25 2014
From: mir at miras.org (Michael Rasmussen)
Date: Sat, 15 Nov 2014 00:49:25 +0100
Subject: [OmniOS-discuss] slog limits write speed more than it should
In-Reply-To: <bf78eb041097447cb27b0bb67b15407d@EX1301.steait.net>
References: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>
	<alpine.GSO.2.01.1411121714550.4045@freddy.simplesystems.org>
	<8a28a83fd09b474da1cfc9d1d14af37f@EX1301.steait.net>
	<CAEyWAj1q3=VxnxcciNVRxHGfDRAjnAEj__QJQmhwq=RNT2kt9A@mail.gmail.com>
	<62474cc7631c4c9e8484be553d0efb91@EX1301.steait.net>
	<alpine.GSO.2.01.1411140844060.18732@freddy.simplesystems.org>
	<730e31bc0c654d71b91edeffd39c9692@EX1301.steait.net>
	<c05908b42bf04de98dca40c78c99a92c@EX1301.steait.net>
	<bf78eb041097447cb27b0bb67b15407d@EX1301.steait.net>
Message-ID: <20141115004925.4f4eaa12@sleipner.datanom.net>

On Fri, 14 Nov 2014 23:06:04 +0000
Rune Tipsmark <rt at steait.net> wrote:

> I get only about half the bandwidth with Sync=Always compared to Sync=Disabled.
> 
> Using an SLC device that should perform better, its rated as 750 MB/sec, I only get something like 60% of that at best of times. If there is a way to stripe a ZIL it would be great. I have enough to mirror two stripes and get the speed I want.
> 
Have you tried sync=standard

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
Do not remove tag under penalty of law.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141115/bd00bb95/attachment.bin>

From richard.elling at richardelling.com  Fri Nov 14 23:51:09 2014
From: richard.elling at richardelling.com (Richard Elling)
Date: Fri, 14 Nov 2014 15:51:09 -0800
Subject: [OmniOS-discuss] slog limits write speed more than it should
In-Reply-To: <CAEyWAj1q3=VxnxcciNVRxHGfDRAjnAEj__QJQmhwq=RNT2kt9A@mail.gmail.com>
References: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>
	<alpine.GSO.2.01.1411121714550.4045@freddy.simplesystems.org>
	<8a28a83fd09b474da1cfc9d1d14af37f@EX1301.steait.net>
	<CAEyWAj1q3=VxnxcciNVRxHGfDRAjnAEj__QJQmhwq=RNT2kt9A@mail.gmail.com>
Message-ID: <DFCB3802-9A94-4D6A-A905-1CE42C0C6677@richardelling.com>

On Nov 12, 2014, at 9:24 PM, Nicholas George <headlesscow at gmail.com> wrote:

> From what I understand, ZFS writes to the intent log in a single threaded synchronous fashion. Adding three drives just means that the first block is written to the first drive synchronously, then the second block to the second drive synchronously, then the third to the third drive and then back around to the first. You don't actually increase your throughput beyond the performance of a single drive.

This information is not quite right. ZFS has several algorithms at play and recall that there 
is not one ZIL, there is one ZIL per dataset. In practice, a busy server will see work spread
out across all slogs and adding more can improve performance. There is always a point of
marginal return, but these are often determined by physical choices.

The best approach is to measure and experiment on your hardware under your workload.
 -- richard

> 
> http://nex7.blogspot.com/2013/04/zfs-intent-log.html
> http://www.nexentastor.org/boards/5/topics/6179
> 
> On Wed, Nov 12, 2014 at 6:21 PM, Rune Tipsmark <rt at steait.net> wrote:
> http://www.fusionio.com/products/iodrive
> 
> 160GB SLC
> 
> -----Original Message-----
> From: Bob Friesenhahn [mailto:bfriesen at simple.dallas.tx.us]
> Sent: Wednesday, November 12, 2014 3:17 PM
> To: Rune Tipsmark
> Cc: omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] slog limits write speed more than it should
> 
> On Wed, 12 Nov 2014, Rune Tipsmark wrote:
> >
> > each SLOG device can easily handle much more than 6000 IOPS
> 
> Where may we find the specifications for the SSDs you are using?
> 
> 6000 IOPS sounds like it might be quite a lot, depending on the SSD used.
> 
> Bob
> --
> Bob Friesenhahn
> bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> 
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141114/c36cf8fe/attachment.html>

From richard.elling at richardelling.com  Sat Nov 15 00:05:09 2014
From: richard.elling at richardelling.com (Richard Elling)
Date: Fri, 14 Nov 2014 16:05:09 -0800
Subject: [OmniOS-discuss] slog limits write speed more than it should
In-Reply-To: <bf78eb041097447cb27b0bb67b15407d@EX1301.steait.net>
References: <4fa7e899cd8c4a70b60da83099f692d2@EX1301.steait.net>
	<alpine.GSO.2.01.1411121714550.4045@freddy.simplesystems.org>
	<8a28a83fd09b474da1cfc9d1d14af37f@EX1301.steait.net>
	<CAEyWAj1q3=VxnxcciNVRxHGfDRAjnAEj__QJQmhwq=RNT2kt9A@mail.gmail.com>
	<62474cc7631c4c9e8484be553d0efb91@EX1301.steait.net>
	<alpine.GSO.2.01.1411140844060.18732@freddy.simplesystems.org>
	<730e31bc0c654d71b91edeffd39c9692@EX1301.steait.net>
	<c05908b42bf04de98dca40c78c99a92c@EX1301.steait.net>
	<bf78eb041097447cb27b0bb67b15407d@EX1301.steait.net>
Message-ID: <71FBAB96-833D-42E0-B387-791401BF7D55@richardelling.com>


On Nov 14, 2014, at 3:06 PM, Rune Tipsmark <rt at steait.net> wrote:

> I get only about half the bandwidth with Sync=Always compared to Sync=Disabled.
> Using an SLC device that should perform better, its rated as 750 MB/sec, I only get something like 60% of that at best of times. If there is a way to stripe a ZIL it would be great. I have enough to mirror two stripes and get the speed I want.

Technically, you can't mirror stripes in ZFS. You can stripe mirrors. For write workloads (eg slog)
a mirror is always at least slightly slower than unmirrored disks. For read workloads, mirrors are
demonstrably faster than unmirrored disks.
 -- richard

>  
> Copying ~60GB from one LUN to another on same ZFS box.
>  
> Sync=Always:
> <image001.png>
>  
> Sync=Disabled
> <image002.png>
>  
> -----Original Message-----
> From: Rune Tipsmark 
> Sent: Friday, November 14, 2014 11:53 AM
> To: Rune Tipsmark; Bob Friesenhahn
> Cc: omnios-discuss at lists.omniti.com
> Subject: RE: [OmniOS-discuss] slog limits write speed more than it should
>  
> Is there a way to stripe two block devices and use them as log?
> I tried following this one https://blogs.oracle.com/bilke/entry/raid_0_stripe_on_solaris
> But I cannot use the device in ZFS - getting the following error: cannot use '/dev/md/rdsk/zil1d0': must be a block device or regular file
>  
> Br,
> Rune
>  
> -----Original Message-----
> From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Rune Tipsmark
> Sent: Friday, November 14, 2014 9:47 AM
> To: Bob Friesenhahn
> Cc: omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] slog limits write speed more than it should
>  
> Well that sucks... I guess one more reason to move to NV-Dimms to replace slow SLC cards.
> Br,
> Rune
>  
> -----Original Message-----
> From: Bob Friesenhahn [mailto:bfriesen at simple.dallas.tx.us]
> Sent: Friday, November 14, 2014 6:48 AM
> To: Rune Tipsmark
> Cc: omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] slog limits write speed more than it should
>  
> On Fri, 14 Nov 2014, Rune Tipsmark wrote:
>  
> >
> > Looking at the first link he clearly states that ZIL is written to in
> > a round robin fashion, I was under the impression that 2 log devices would then be faster than 1?unless mirrored of course.
>  
> This could be a mistaken impression.  The slog commits each write before proceeding to the next write.  If the involved SSDs have a fixed minimum write latency, then this would limit the maximum transaction rate regardless of the number of SSDs involved.  There could be an advantage to more SSDs if the additional time between writes allows the SSD to more effectively prepare for the next write and reduce the write latency.
>  
> Bob
> --
> Bob Friesenhahn
> bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141114/803689ff/attachment-0001.html>

From cj.keist at colostate.edu  Sat Nov 15 14:57:10 2014
From: cj.keist at colostate.edu (CJ Keist)
Date: Sat, 15 Nov 2014 07:57:10 -0700
Subject: [OmniOS-discuss] Identifying physical disk
Message-ID: <546769C6.50805@colostate.edu>

Hello,
    Need some help identifying physical disk on my system.  My system 
got hung and last thing in the /var/adm/message file before the crash is 
follows:

Nov 14 13:31:12 projects2 scsi: [ID 107833 kern.warning] WARNING: 
/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
Nov 14 13:31:12 projects2       Disconnected command timeout for Target 48
Nov 14 13:31:12 projects2 scsi: [ID 243001 kern.warning] WARNING: 
/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
Nov 14 13:31:12 projects2       mptsas_handle_event_sync: 
IOCStatus=0x8000, IOCLogInfo=0x31170000
Nov 14 13:31:16 projects2 scsi: [ID 365881 kern.info] 
/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
Nov 14 13:31:16 projects2       Log info 0x31140000 received for target 48.
Nov 14 13:31:16 projects2       scsi_status=0x0, ioc_status=0x8048, 
scsi_state=0xc
Nov 14 13:31:16 projects2 scsi: [ID 107833 kern.warning] WARNING: 
/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
Nov 14 13:31:16 projects2       mptsas_check_task_mgt: Task 0x3 failed. 
IOCStatus=0x4a IOCLogInfo=0x0 target=48
Nov 14 13:31:16 projects2 scsi: [ID 107833 kern.warning] WARNING: 
/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
Nov 14 13:31:16 projects2       mptsas_ioc_task_management failed try to 
reset ioc to recovery!

Then using the command "cfgadm -alv" shows that the 
/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 is the controller in front of all 
the disks.  The output above does say "Target 48", I'm hoping that is 
the disk target so would be the one listed in the cfgadm command here:

c6::w50014ee2b2e8dc6a,0        connected    configured   unknown Client 
Device: /dev/dsk/c3t50014EE2B2E8DC6Ad0s0(sd48)
unavailable  disk-path    n 
/devices/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0/iport at f:scsi::w50014ee2b2e8dc6a,0

Would this be the disk being reported in /var/adm/messages?

-- 
C. J. Keist                     Email: cj.keist at colostate.edu
Systems Group Manager           Solaris 10 OS (SAI)
Engineering Network Services    Phone: 970-491-0630
College of Engineering, CSU     Fax:   970-491-5569
Ft. Collins, CO 80523-1301

All I want is a chance to prove 'Money can't buy happiness'


From mir at miras.org  Sat Nov 15 15:09:46 2014
From: mir at miras.org (Michael Rasmussen)
Date: Sat, 15 Nov 2014 16:09:46 +0100
Subject: [OmniOS-discuss] package request
Message-ID: <20141115160946.3c677648@sleipner.datanom.net>

Hi Dan,

Would it be possible to have the userland tools for infiniband ported to omnios?
There exists a pkg configured repository here:
https://github.com/syoyo/solaris-infiniband-tools

Unfortunately I have not be able to run a succesful pkgadd -d since the metadata apparently
is not properly identified by 151012.

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
Why are there always boycotts?  Shouldn't there be girlcotts too?
		-- argon on #Linux
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141115/aaa7b692/attachment.bin>

From richard.elling at richardelling.com  Sat Nov 15 15:49:49 2014
From: richard.elling at richardelling.com (Richard Elling)
Date: Sat, 15 Nov 2014 07:49:49 -0800
Subject: [OmniOS-discuss] Identifying physical disk
In-Reply-To: <546769C6.50805@colostate.edu>
References: <546769C6.50805@colostate.edu>
Message-ID: <8AD5738F-27B7-4A54-B8B0-D0464BE21084@RichardElling.com>

Hi CJ,
I'm away from my notes at the moment, but know that an mptsas instance's reported target number is not the same as the sd driver instance number. The most expeditious way to cross reference is to use sasinfo or, failing that, 

echo ::mptsas -t | mdb -k

which dumps the sas WWN to target port info.

If that still doesn't narrow the field, look at the per-disk error counters: 
iostat -E

Note: resets are tricky and you can be sure sd is trying to reset devices to wake them up. 

More later, when I get back to the office...
  -- richard



> On Nov 15, 2014, at 6:57 AM, CJ Keist <cj.keist at colostate.edu> wrote:
> 
> Hello,
>   Need some help identifying physical disk on my system.  My system got hung and last thing in the /var/adm/message file before the crash is follows:
> 
> Nov 14 13:31:12 projects2 scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
> Nov 14 13:31:12 projects2       Disconnected command timeout for Target 48
> Nov 14 13:31:12 projects2 scsi: [ID 243001 kern.warning] WARNING: /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
> Nov 14 13:31:12 projects2       mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31170000
> Nov 14 13:31:16 projects2 scsi: [ID 365881 kern.info] /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
> Nov 14 13:31:16 projects2       Log info 0x31140000 received for target 48.
> Nov 14 13:31:16 projects2       scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc
> Nov 14 13:31:16 projects2 scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
> Nov 14 13:31:16 projects2       mptsas_check_task_mgt: Task 0x3 failed. IOCStatus=0x4a IOCLogInfo=0x0 target=48
> Nov 14 13:31:16 projects2 scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
> Nov 14 13:31:16 projects2       mptsas_ioc_task_management failed try to reset ioc to recovery!
> 
> Then using the command "cfgadm -alv" shows that the /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 is the controller in front of all the disks.  The output above does say "Target 48", I'm hoping that is the disk target so would be the one listed in the cfgadm command here:
> 
> c6::w50014ee2b2e8dc6a,0        connected    configured   unknown Client Device: /dev/dsk/c3t50014EE2B2E8DC6Ad0s0(sd48)
> unavailable  disk-path    n /devices/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0/iport at f:scsi::w50014ee2b2e8dc6a,0
> 
> Would this be the disk being reported in /var/adm/messages?
> 
> -- 
> C. J. Keist                     Email: cj.keist at colostate.edu
> Systems Group Manager           Solaris 10 OS (SAI)
> Engineering Network Services    Phone: 970-491-0630
> College of Engineering, CSU     Fax:   970-491-5569
> Ft. Collins, CO 80523-1301
> 
> All I want is a chance to prove 'Money can't buy happiness'
> 
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

From richard.elling at richardelling.com  Sat Nov 15 16:54:34 2014
From: richard.elling at richardelling.com (Richard Elling)
Date: Sat, 15 Nov 2014 08:54:34 -0800
Subject: [OmniOS-discuss] Identifying physical disk
In-Reply-To: <8AD5738F-27B7-4A54-B8B0-D0464BE21084@RichardElling.com>
References: <546769C6.50805@colostate.edu>
	<8AD5738F-27B7-4A54-B8B0-D0464BE21084@RichardElling.com>
Message-ID: <6C2D1555-E02C-400E-9920-3E740815FADC@RichardElling.com>


> On Nov 15, 2014, at 7:49 AM, Richard Elling <richard.elling at richardelling.com> wrote:
> 
> Hi CJ,
> I'm away from my notes at the moment, but know that an mptsas instance's reported target number is not the same as the sd driver instance number. The most expeditious way to cross reference is to use sasinfo or, failing that, 
> 
> echo ::mptsas -t | mdb -k
> 
> which dumps the sas WWN to target port info.

in this output, the ?devhdl? is the number you need to match
 ? richard

> 
> If that still doesn't narrow the field, look at the per-disk error counters: 
> iostat -E
> 
> Note: resets are tricky and you can be sure sd is trying to reset devices to wake them up. 
> 
> More later, when I get back to the office...
>  -- richard
> 
> 
> 
>> On Nov 15, 2014, at 6:57 AM, CJ Keist <cj.keist at colostate.edu> wrote:
>> 
>> Hello,
>>  Need some help identifying physical disk on my system.  My system got hung and last thing in the /var/adm/message file before the crash is follows:
>> 
>> Nov 14 13:31:12 projects2 scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>> Nov 14 13:31:12 projects2       Disconnected command timeout for Target 48
>> Nov 14 13:31:12 projects2 scsi: [ID 243001 kern.warning] WARNING: /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>> Nov 14 13:31:12 projects2       mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31170000
>> Nov 14 13:31:16 projects2 scsi: [ID 365881 kern.info] /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>> Nov 14 13:31:16 projects2       Log info 0x31140000 received for target 48.
>> Nov 14 13:31:16 projects2       scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc
>> Nov 14 13:31:16 projects2 scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>> Nov 14 13:31:16 projects2       mptsas_check_task_mgt: Task 0x3 failed. IOCStatus=0x4a IOCLogInfo=0x0 target=48
>> Nov 14 13:31:16 projects2 scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>> Nov 14 13:31:16 projects2       mptsas_ioc_task_management failed try to reset ioc to recovery!
>> 
>> Then using the command "cfgadm -alv" shows that the /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 is the controller in front of all the disks.  The output above does say "Target 48", I'm hoping that is the disk target so would be the one listed in the cfgadm command here:
>> 
>> c6::w50014ee2b2e8dc6a,0        connected    configured   unknown Client Device: /dev/dsk/c3t50014EE2B2E8DC6Ad0s0(sd48)
>> unavailable  disk-path    n /devices/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0/iport at f:scsi::w50014ee2b2e8dc6a,0
>> 
>> Would this be the disk being reported in /var/adm/messages?
>> 
>> -- 
>> C. J. Keist                     Email: cj.keist at colostate.edu
>> Systems Group Manager           Solaris 10 OS (SAI)
>> Engineering Network Services    Phone: 970-491-0630
>> College of Engineering, CSU     Fax:   970-491-5569
>> Ft. Collins, CO 80523-1301
>> 
>> All I want is a chance to prove 'Money can't buy happiness'
>> 
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss

--

Richard.Elling at RichardElling.com
+1-760-896-4422



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141115/f3e04b35/attachment-0001.html>

From cj.keist at colostate.edu  Sat Nov 15 17:25:36 2014
From: cj.keist at colostate.edu (CJ Keist)
Date: Sat, 15 Nov 2014 10:25:36 -0700
Subject: [OmniOS-discuss] Identifying physical disk
In-Reply-To: <6C2D1555-E02C-400E-9920-3E740815FADC@RichardElling.com>
References: <546769C6.50805@colostate.edu>
	<8AD5738F-27B7-4A54-B8B0-D0464BE21084@RichardElling.com>
	<6C2D1555-E02C-400E-9920-3E740815FADC@RichardElling.com>
Message-ID: <54678C90.8090909@colostate.edu>

Richard,
    Thank you very much for this!  Unfortunately, since the system has 
been rebooted all the disk error counts are back to zero.  My system 
didn't have the sasinfo command, do you know what package I need to 
install?  The echo::mptasas command worked.



On 11/15/14, 9:54 AM, Richard Elling wrote:
>
>> On Nov 15, 2014, at 7:49 AM, Richard Elling 
>> <richard.elling at richardelling.com 
>> <mailto:richard.elling at richardelling.com>> wrote:
>>
>> Hi CJ,
>> I'm away from my notes at the moment, but know that an mptsas 
>> instance's reported target number is not the same as the sd driver 
>> instance number. The most expeditious way to cross reference is to 
>> use sasinfo or, failing that,
>>
>> echo ::mptsas -t | mdb -k
>>
>> which dumps the sas WWN to target port info.
>
> in this output, the ?devhdl? is the number you need to match
>  ? richard
>
>>
>> If that still doesn't narrow the field, look at the per-disk error 
>> counters:
>> iostat -E
>>
>> Note: resets are tricky and you can be sure sd is trying to reset 
>> devices to wake them up.
>>
>> More later, when I get back to the office...
>>  -- richard
>>
>>
>>
>>> On Nov 15, 2014, at 6:57 AM, CJ Keist <cj.keist at colostate.edu 
>>> <mailto:cj.keist at colostate.edu>> wrote:
>>>
>>> Hello,
>>>  Need some help identifying physical disk on my system.  My system 
>>> got hung and last thing in the /var/adm/message file before the 
>>> crash is follows:
>>>
>>> Nov 14 13:31:12 projects2 scsi: [ID 107833 kern.warning] WARNING: 
>>> /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>>> Nov 14 13:31:12 projects2       Disconnected command timeout for 
>>> Target 48
>>> Nov 14 13:31:12 projects2 scsi: [ID 243001 kern.warning] WARNING: 
>>> /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>>> Nov 14 13:31:12 projects2       mptsas_handle_event_sync: 
>>> IOCStatus=0x8000, IOCLogInfo=0x31170000
>>> Nov 14 13:31:16 projects2 scsi: [ID 365881 kern.info] 
>>> /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>>> Nov 14 13:31:16 projects2       Log info 0x31140000 received for 
>>> target 48.
>>> Nov 14 13:31:16 projects2       scsi_status=0x0, ioc_status=0x8048, 
>>> scsi_state=0xc
>>> Nov 14 13:31:16 projects2 scsi: [ID 107833 kern.warning] WARNING: 
>>> /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>>> Nov 14 13:31:16 projects2       mptsas_check_task_mgt: Task 0x3 
>>> failed. IOCStatus=0x4a IOCLogInfo=0x0 target=48
>>> Nov 14 13:31:16 projects2 scsi: [ID 107833 kern.warning] WARNING: 
>>> /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>>> Nov 14 13:31:16 projects2       mptsas_ioc_task_management failed 
>>> try to reset ioc to recovery!
>>>
>>> Then using the command "cfgadm -alv" shows that the 
>>> /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 is the controller in front of 
>>> all the disks.  The output above does say "Target 48", I'm hoping 
>>> that is the disk target so would be the one listed in the cfgadm 
>>> command here:
>>>
>>> c6::w50014ee2b2e8dc6a,0        connected    configured   unknown 
>>> Client Device: /dev/dsk/c3t50014EE2B2E8DC6Ad0s0(sd48)
>>> unavailable  disk-path    n 
>>> /devices/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0/iport at f:scsi::w50014ee2b2e8dc6a,0
>>>
>>> Would this be the disk being reported in /var/adm/messages?
>>>
>>> -- 
>>> C. J. Keist                     Email: cj.keist at colostate.edu 
>>> <mailto:cj.keist at colostate.edu>
>>> Systems Group Manager           Solaris 10 OS (SAI)
>>> Engineering Network Services    Phone: 970-491-0630
>>> College of Engineering, CSU     Fax:   970-491-5569
>>> Ft. Collins, CO 80523-1301
>>>
>>> All I want is a chance to prove 'Money can't buy happiness'
>>>
>>> _______________________________________________
>>> OmniOS-discuss mailing list
>>> OmniOS-discuss at lists.omniti.com <mailto:OmniOS-discuss at lists.omniti.com>
>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
> --
>
> Richard.Elling at RichardElling.com <mailto:Richard.Elling at RichardElling.com>
> +1-760-896-4422
>
>
>

-- 
C. J. Keist                     Email: cj.keist at colostate.edu
Systems Group Manager           Solaris 10 OS (SAI)
Engineering Network Services    Phone: 970-491-0630
College of Engineering, CSU     Fax:   970-491-5569
Ft. Collins, CO 80523-1301

All I want is a chance to prove 'Money can't buy happiness'

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141115/dced435c/attachment.html>

From danmcd at omniti.com  Sat Nov 15 18:29:47 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Sat, 15 Nov 2014 13:29:47 -0500
Subject: [OmniOS-discuss] package request
In-Reply-To: <20141115160946.3c677648@sleipner.datanom.net>
References: <20141115160946.3c677648@sleipner.datanom.net>
Message-ID: <8E7DDA99-775B-4624-AA2E-E3C12E7112FE@omniti.com>

You are welcome to pull from omnios-build and start the port.  I have other fish to fry at the moment, however (pkg(5) and Illumos work that will affect bloody and the next stable).

Sorry,
Dan

Sent from my iPhone (typos, autocorrect, and all)

> On Nov 15, 2014, at 10:09 AM, Michael Rasmussen <mir at miras.org> wrote:
> 
> Hi Dan,
> 
> Would it be possible to have the userland tools for infiniband ported to omnios?
> There exists a pkg configured repository here:
> https://github.com/syoyo/solaris-infiniband-tools
> 
> Unfortunately I have not be able to run a succesful pkgadd -d since the metadata apparently
> is not properly identified by 151012.
> 
> -- 
> Hilsen/Regards
> Michael Rasmussen
> 
> Get my public GnuPG keys:
> michael <at> rasmussen <dot> cc
> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
> mir <at> datanom <dot> net
> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
> mir <at> miras <dot> org
> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
> --------------------------------------------------------------
> /usr/games/fortune -es says:
> Why are there always boycotts?  Shouldn't there be girlcotts too?
>        -- argon on #Linux
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

From richard.elling at richardelling.com  Sat Nov 15 18:36:58 2014
From: richard.elling at richardelling.com (Richard Elling)
Date: Sat, 15 Nov 2014 10:36:58 -0800
Subject: [OmniOS-discuss] Identifying physical disk
In-Reply-To: <54678C90.8090909@colostate.edu>
References: <546769C6.50805@colostate.edu>
	<8AD5738F-27B7-4A54-B8B0-D0464BE21084@RichardElling.com>
	<6C2D1555-E02C-400E-9920-3E740815FADC@RichardElling.com>
	<54678C90.8090909@colostate.edu>
Message-ID: <4BB86A0F-DFD8-4CE6-B8AB-8E8C344CC843@RichardElling.com>


> On Nov 15, 2014, at 9:25 AM, CJ Keist <cj.keist at colostate.edu> wrote:
> 
> Richard,
>    Thank you very much for this!  Unfortunately, since the system has been rebooted all the disk error counts are back to zero.  My system didn't have the sasinfo command, do you know what package I need to install?  The echo::mptasas command worked.

sasinfo is in the sasinfo package, perhaps not installed by default.

More forensics?
In most cases, when mptsas complains, it is accompanied by error reports logged by FMA. To see
these in great detail, use:
	fmdump -eV

The problem with resets is that one device can result in a reset of the HBA, which causes ereports
from potentially all outstanding requests (aka spam). The trick is to filter out the root cause from the
recovery.
 ? richard


> 
> 
> 
> On 11/15/14, 9:54 AM, Richard Elling wrote:
>> 
>>> On Nov 15, 2014, at 7:49 AM, Richard Elling <richard.elling at richardelling.com> wrote:
>>> 
>>> Hi CJ,
>>> I'm away from my notes at the moment, but know that an mptsas instance's reported target number is not the same as the sd driver instance number. The most expeditious way to cross reference is to use sasinfo or, failing that, 
>>> 
>>> echo ::mptsas -t | mdb -k
>>> 
>>> which dumps the sas WWN to target port info.
>> 
>> in this output, the ?devhdl? is the number you need to match
>>  ? richard
>> 
>>> 
>>> If that still doesn't narrow the field, look at the per-disk error counters: 
>>> iostat -E
>>> 
>>> Note: resets are tricky and you can be sure sd is trying to reset devices to wake them up. 
>>> 
>>> More later, when I get back to the office...
>>>  -- richard
>>> 
>>> 
>>> 
>>>> On Nov 15, 2014, at 6:57 AM, CJ Keist <cj.keist at colostate.edu> wrote:
>>>> 
>>>> Hello,
>>>>  Need some help identifying physical disk on my system.  My system got hung and last thing in the /var/adm/message file before the crash is follows:
>>>> 
>>>> Nov 14 13:31:12 projects2 scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>>>> Nov 14 13:31:12 projects2       Disconnected command timeout for Target 48
>>>> Nov 14 13:31:12 projects2 scsi: [ID 243001 kern.warning] WARNING: /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>>>> Nov 14 13:31:12 projects2       mptsas_handle_event_sync: IOCStatus=0x8000, IOCLogInfo=0x31170000
>>>> Nov 14 13:31:16 projects2 scsi: [ID 365881 kern.info] /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>>>> Nov 14 13:31:16 projects2       Log info 0x31140000 received for target 48.
>>>> Nov 14 13:31:16 projects2       scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc
>>>> Nov 14 13:31:16 projects2 scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>>>> Nov 14 13:31:16 projects2       mptsas_check_task_mgt: Task 0x3 failed. IOCStatus=0x4a IOCLogInfo=0x0 target=48
>>>> Nov 14 13:31:16 projects2 scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 (mpt_sas0):
>>>> Nov 14 13:31:16 projects2       mptsas_ioc_task_management failed try to reset ioc to recovery!
>>>> 
>>>> Then using the command "cfgadm -alv" shows that the /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0 is the controller in front of all the disks.  The output above does say "Target 48", I'm hoping that is the disk target so would be the one listed in the cfgadm command here:
>>>> 
>>>> c6::w50014ee2b2e8dc6a,0        connected    configured   unknown Client Device: /dev/dsk/c3t50014EE2B2E8DC6Ad0s0(sd48)
>>>> unavailable  disk-path    n /devices/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0/iport at f:scsi::w50014ee2b2e8dc6a,0
>>>> 
>>>> Would this be the disk being reported in /var/adm/messages?
>>>> 
>>>> -- 
>>>> C. J. Keist                     Email: cj.keist at colostate.edu
>>>> Systems Group Manager           Solaris 10 OS (SAI)
>>>> Engineering Network Services    Phone: 970-491-0630
>>>> College of Engineering, CSU     Fax:   970-491-5569
>>>> Ft. Collins, CO 80523-1301
>>>> 
>>>> All I want is a chance to prove 'Money can't buy happiness'
>>>> 
>>>> _______________________________________________
>>>> OmniOS-discuss mailing list
>>>> OmniOS-discuss at lists.omniti.com
>>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>> 
>> --
>> 
>> Richard.Elling at RichardElling.com
>> +1-760-896-4422
>> 
>> 
>> 
> 
> -- 
> C. J. Keist                     Email: 
> cj.keist at colostate.edu
> 
> Systems Group Manager           Solaris 10 OS (SAI)
> Engineering Network Services    Phone: 970-491-0630
> College of Engineering, CSU     Fax:   970-491-5569
> Ft. Collins, CO 80523-1301
> 
> All I want is a chance to prove 'Money can't buy happiness'
> 

-- 

ZFS storage and performance consulting at http://www.RichardElling.com







From gate03 at landcroft.co.uk  Sun Nov 16 22:51:17 2014
From: gate03 at landcroft.co.uk (Michael Mounteney)
Date: Mon, 17 Nov 2014 08:51:17 +1000
Subject: [OmniOS-discuss] Building Haskell on OmniOS
Message-ID: <20141117085117.7e0e3e77@emeritus>

Last year there was a thread about bootstrapping GHC, mainly by Paul
Henson and Matthieu Paindavoine.  I'd like to pick it up, because
I, like Matthieu, want to build the compiler.

There were two separate problems.  One was that a stray 'link' call was
causing a directory under /tmp to be undeletable, even though it was
empty.  That was likely a problem with tmpfs, according to
http://lists.omniti.com/pipermail/omnios-discuss/2013-May/000779.html
so that can be worked-around.

Then there's the problem with the build, most likely down to some
presumed Linuxism:
http://lists.omniti.com/pipermail/omnios-discuss/2013-May/000844.html

Is there any solution for this?  I tried it yesterday with an
up-to-date stable (r151010) and the above build failure still obtains.

Michael.

From tobi at oetiker.ch  Mon Nov 17 08:31:16 2014
From: tobi at oetiker.ch (Tobias Oetiker)
Date: Mon, 17 Nov 2014 09:31:16 +0100 (CET)
Subject: [OmniOS-discuss] best guide to tftp ?
In-Reply-To: <20141109063015.71b9614b@emerior>
References: <20141025065313.4b67dcbc@punda-mlia>
	<CAHBEJzWx06iwAiVK3zKtG+XVmb3N-1krXe3CgoYwhcAG9XEMLw@mail.gmail.com>
	<20141108124524.6f42889a@emerior> <20141109063015.71b9614b@emerior>
Message-ID: <alpine.DEB.2.10.1411170929490.26293@engelberg.oetiker.ch>

Hi Michael,

Nov 9 Michael Mounteney wrote:

> [Later] just running
>
> # /usr/sbin/in.tftpd -p -d -s /tftpboot
>
> doesn't work either.  A client-side
>
> $ tftp cortex -c get text
>
> times out.

we found that tftpd on omnios stoped working with the upgrade to
r10 .. we have investigated a bit but not found the cause and
switched to another machine (linux) for your tftp needs ... but it
would certainly be nice to know what broke it ...

cheers
tobi

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch tobi at oetiker.ch +41 62 775 9902


From danmcd at omniti.com  Mon Nov 17 15:12:50 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Mon, 17 Nov 2014 10:12:50 -0500
Subject: [OmniOS-discuss] Who here had lockd/nlockmgr problems?
References: <968CF721-8839-49E2-8F04-9FD912E78E68@omniti.com>
Message-ID: <9A9651B5-B71B-44D3-90C1-BCF96B4ECCE8@omniti.com>

ISTR someone here had a problem where his/her nlockmgr SMF service wouldn't start.  I've encountered this problem myself recently (on a OI VM, but it's running the same new open-source lockd that's in OmniOS with r151010 and later), and wanted to share a conversation from the illumos developer's list.

FYI,
Dan


> Begin forwarded message:
> 
> Subject: Re: [developer] lockd not starting?
> From: Dan McDonald <danmcd at omniti.com>
> Date: November 17, 2014 at 10:10:38 AM EST
> Cc: illumos Developer <developer at lists.illumos.org>, Matt Amdur <matt.amdur at delphix.com>
> To: Sebastien Roy <sebastien.roy at delphix.com>
> 
> 
>> On Nov 16, 2014, at 7:54 PM, Sebastien Roy <sebastien.roy at delphix.com> wrote:
>> 
>> Hey Dan,
>> 
>> I think we're looking into a similar problem here at Delphix. We've noticed that the nfs/nlockmgr service goes into maintenance mode after a timeout of the SM_CRASH call to statd.
>> 
>> Upon startup, the statd daemon is blocked notifying clients. These clients are cached in /var/statmon/sm.bak/... If any of these clients are unreachable, the SM_CRASH call issued by klm will timeout (this timeout is much shorter than that which statd uses to give up on client notifications), and the nfs/nlockmgr service will then end up in maintenance state.
>> 
>> We've been discussing possible fixes to this problem. One easy fix that Matt Amdur (Cc'ed) proposed would be to shorten the timeout that statd uses to notify clients to reduce the chance that lockd's SM_CRASH call would timeout in the event of an unreachable client. We haven't implemented the fix yet.
>> 
>> A workaround is to clear statd's cached clients.
> 
> Okay!  This makes some modicum of sense.  Here's what I have now:
> 
> # ls -lt /var/statmon/sm.bak/
> total 2
> lrwxrwxrwx   1 daemon   daemon        10 Nov 12 19:55 ipv4.10.0.1.68 -> Everywhere
> lrwxrwxrwx   1 daemon   daemon        28 Nov 11 12:44 ipv4.10.8.3.241 -> everywhere.office.omniti.com
> # nslookup everywhere
> Server:         10.8.3.1
> Address:        10.8.3.1#53
> 
> Name:   everywhere.office.omniti.com
> Address: 10.8.3.241
> 
> # 
> 
> While travelling last week, I had to park myself on a 10.0.1.0/24 network.  My VMs are configured to be link-sharers (same-link peers), so I was using NFS over the same link to my VM.  Once I renumbered back to the intended OmniTI networks, things started going screwy.
> 
> I'm going to remove the 10.0.1.68 one and try again... BINGO!
> 
> # svcs -xv nlockmgr
> svc:/network/nfs/nlockmgr:default (NFS lock manager)
> State: maintenance since November 16, 2014 04:00:05 PM EST
> Reason: Start method failed repeatedly, last exited with status 1.
>   See: http://illumos.org/msg/SMF-8000-KS
>   See: man -M /usr/share/man -s 1M lockd
>   See: /var/svc/log/network-nfs-nlockmgr:default.log
> Impact: 1 dependent service is not running:
>        svc:/network/nfs/server:default
> # rm /var/statmon/sm.bak/ipv4.10.0.1.68 
> # svcadm clear nlockmgr
> # svcs -xv nlockmgr
> svc:/network/nfs/nlockmgr:default (NFS lock manager)
> State: online since November 17, 2014 10:08:53 AM EST
>   See: man -M /usr/share/man -s 1M lockd
>   See: /var/svc/log/network-nfs-nlockmgr:default.log
> Impact: None.
> # 
> 
> This is actually good, in that I believe the circumstances for this bug are now reproducible.
> 
> Thanks!
> Dan
> 
> 


From gate03 at landcroft.co.uk  Mon Nov 17 21:16:17 2014
From: gate03 at landcroft.co.uk (Michael Mounteney)
Date: Tue, 18 Nov 2014 07:16:17 +1000
Subject: [OmniOS-discuss] Building Haskell on OmniOS
In-Reply-To: <5B4B585D-2335-4FA1-8E8D-8A504B6F4EBB@omniti.com>
References: <20141117085117.7e0e3e77@emeritus>
	<5B4B585D-2335-4FA1-8E8D-8A504B6F4EBB@omniti.com>
Message-ID: <20141118071617.7851b06c@emeritus>

On Sun, 16 Nov 2014 17:57:25 -0500
Dan McDonald <danmcd at omniti.com> wrote:

> Have you tried 151012, the current stable release?

That yields a different failure, for which I have no ready explanation:

~/ghc-7.4.2 # gmake install
[...]
/opt/gcc-4.8.1/bin/gcc -E  -undef -traditional -P -DINSTALLING -DLIB_DIR='"/opt/ghc-7.4.2/lib/ghc-7.4.2"' -DINCLUDE_DIR='"/opt/ghc-7.4.2/lib/ghc-7.4.2/include"' -DPAPI_INCLUDE_DIR="" -DPAPI_LIB_DIR="" -x c -Iincludes rts/package.conf.in -o rts/package.conf.install.raw
grep -v '^#pragma GCC' rts/package.conf.install.raw | sed -e 's/""//g' -e 's/:[         ]*,/: /g' >rts/package.conf.install
"inplace/bin/ghc-stage1" -fPIC -dynamic  -H32m -O    -package-name integer-gmp-0.4.0.0 -hide-all-packages -i -ilibraries/integer-gmp/. -ilibraries/integer-gmp/dist-install/build -ilibraries/integer-gmp/dist-install/build/autogen -Ilibraries/integer-gmp/dist-install/build -Ilibraries/integer-gmp/dist-install/build/autogen -Ilibraries/integer-gmp/.    -optP-include -optPlibraries/integer-gmp/dist-install/build/autogen/cabal_macros.h -package ghc-prim-0.2.0.0  -package-name integer-gmp -XHaskell98 -XCPP -XMagicHash -XUnboxedTuples -XNoImplicitPrelude -XForeignFunctionInterface -XUnliftedFFITypes -O2  -no-user-package-conf -rtsopts     -odir libraries/integer-gmp/dist-install/build -hidir libraries/integer-gmp/dist-install/build -stubdir libraries/integer-gmp/dist-install/build -hisuf dyn_hi -osuf  dyn_o -hcsuf dyn_hc libraries/integer-gmp/dist-install/build/GHC/Integer.dyn_o libraries/integer-gmp/dist-install/build/GHC/Integer/GMP/Internals.dyn_o libraries/integer-gmp/dist-insta
 ll/build/GHC/Integer/GMP/Prim.dyn_o libraries/integer-gmp/dist-install/build/GHC/Integer/Logarithms.dyn_o libraries/integer-gmp/dist-install/build/GHC/Integer/Logarithms/Internals.dyn_o libraries/integer-gmp/dist-install/build/GHC/Integer/Type.dyn_o libraries/integer-gmp/dist-install/build/cbits/gmp-wrappers.dyn_o  libraries/integer-gmp/dist-install/build/cbits/cbits.dyn_o   libraries/integer-gmp/gmp/objs/*.o -shared -dynamic -dynload deploy -dylib-install-name /opt/ghc-7.4.2/lib/ghc-7.4.2/`basename "libraries/integer-gmp/dist-install/build/libHSinteger-gmp-0.4.0.0-ghc7.4.2.so" | sed 's/^libHS//;s/[-]ghc.*//'`/`basename "libraries/integer-gmp/dist-install/build/libHSinteger-gmp-0.4.0.0-ghc7.4.2.so"` -no-auto-link-packages -o libraries/integer-gmp/dist-install/build/libHSinteger-gmp-0.4.0.0-ghc7.4.2.so
Undefined                       first referenced
 symbol                             in file
g0                                  libraries/integer-gmp/dist-install/build/cbits/gmp-wrappers.dyn_o
__int_encodeFloat                   libraries/integer-gmp/dist-install/build/GHC/Integer/Type.dyn_o
stg_gc_noregs                       libraries/integer-gmp/dist-install/build/GHC/Integer/Logarithms/Internals.dyn_o
stg_gc_unbx_r1                      libraries/integer-gmp/dist-install/build/GHC/Integer/Type.dyn_o
stg_upd_frame_info                  libraries/integer-gmp/dist-install/build/GHC/Integer/Logarithms.dyn_o
allocate                            libraries/integer-gmp/dist-install/build/cbits/cbits.dyn_o
stg_newByteArrayzh                  libraries/integer-gmp/dist-install/build/GHC/Integer/Logarithms/Internals.dyn_o
stg_CAF_BLACKHOLE_info              libraries/integer-gmp/dist-install/build/GHC/Integer/Logarithms/Internals.dyn_o
stg_gc_gen                          libraries/integer-gmp/dist-install/build/cbits/gmp-wrappers.dyn_o
newCAF                              libraries/integer-gmp/dist-install/build/GHC/Integer/Logarithms/Internals.dyn_o
stg_bh_upd_frame_info               libraries/integer-gmp/dist-install/build/GHC/Integer/Logarithms/Internals.dyn_o
stg_ARR_WORDS_info                  libraries/integer-gmp/dist-install/build/cbits/gmp-wrappers.dyn_o
__int_encodeDouble                  libraries/integer-gmp/dist-install/build/GHC/Integer/Type.dyn_o
large_alloc_lim                     libraries/integer-gmp/dist-install/build/cbits/gmp-wrappers.dyn_o
stg_gc_ut                           libraries/integer-gmp/dist-install/build/GHC/Integer/Logarithms/Internals.dyn_o
rts_unsafeGetMyCapability           libraries/integer-gmp/dist-install/build/cbits/cbits.dyn_o
stg_gc_gen_hp                       libraries/integer-gmp/dist-install/build/cbits/gmp-wrappers.dyn_o
ld: warning: symbol referencing errors
Text relocation remains                         referenced
    against symbol                  offset      in file
.text (section)                     0x2e        libraries/integer-gmp/gmp/objs/add_n.o
.text (section)                     0x6e        libraries/integer-gmp/gmp/objs/add_n.o
__gmpz_add                          0x2d5       libraries/integer-gmp/gmp/objs/aors.o
__gmpz_sub                          0x2e1       libraries/integer-gmp/gmp/objs/aors.o
[.. thousands more like it..]

vsnprintf                           0x2e    libraries/integer-gmp/gmp/objs/vasprintf.o
vsprintf                            0x62    libraries/integer-gmp/gmp/objs/sprintffuns.o
sscanf                              0x51    libraries/integer-gmp/gmp/objs/sscanffuns.o
ld: fatal: relocations remain against allocatable but non-writable sections
collect2: error: ld returned 1 exit status
gmake[1]: *** [libraries/integer-gmp/dist-install/build/libHSinteger-gmp-0.4.0.0-ghc7.4.2.so] Error 1
gmake: *** [install] Error 2

From rt at steait.net  Wed Nov 19 14:36:55 2014
From: rt at steait.net (Rune Tipsmark)
Date: Wed, 19 Nov 2014 14:36:55 +0000
Subject: [OmniOS-discuss] need to change c17d0 to c15d0
Message-ID: <d0ee326b448f41efaa3fcef7c541b028@EX1301.steait.net>

I moved one of my PCI-E IOdrives and the disks changed from c14d0 and  c15d0 to c16d0 and c17d0
How do I change it back so I can get my pool back online?

32. c16d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c02 at 1/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
33. c17d0 <Unknown-Unknown-0001-298.02GB>
          /pci at 79,0/pci8086,3c02 at 1/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0

Br,
Rune

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141119/3391e6b1/attachment.html>

From slefevre at indy.rr.com  Wed Nov 19 14:47:16 2014
From: slefevre at indy.rr.com (Scott LeFevre)
Date: Wed, 19 Nov 2014 09:47:16 -0500
Subject: [OmniOS-discuss] need to change c17d0 to c15d0
In-Reply-To: <d0ee326b448f41efaa3fcef7c541b028@EX1301.steait.net>
References: <d0ee326b448f41efaa3fcef7c541b028@EX1301.steait.net>
Message-ID: <1416408436.3020.3.camel@indy.rr.com>

Put the drive/hba back and then export your pool (zpool export
pool-name). Then move the drive/hba and import the pool (zpool import
pool-name).
If putting the drive/hba back isn't an option, you could try a forced
import with 'zpool import -f pool-name'.

Cheers,
-- 
Scott LeFevre
317-696-1010 

On Wed, 2014-11-19 at 14:36 +0000, Rune Tipsmark wrote:
> I moved one of my PCI-E IOdrives and the disks changed from c14d0 and
> c15d0 to c16d0 and c17d0
> 
> How do I change it back so I can get my pool back online?
> 
>  
> 
> 32. c16d0 <Unknown-Unknown-0001-298.02GB>
> 
>           /pci at 79,0/pci8086,3c02 at 1/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
> 
> 33. c17d0 <Unknown-Unknown-0001-298.02GB>
> 
>           /pci at 79,0/pci8086,3c02 at 1/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
> 
>  
> 
> Br,
> 
> Rune
> 
>  
> 
> 
> 
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141119/d3137dc6/attachment.html>

From rt at steait.net  Wed Nov 19 14:50:46 2014
From: rt at steait.net (Rune Tipsmark)
Date: Wed, 19 Nov 2014 14:50:46 +0000
Subject: [OmniOS-discuss] need to change c17d0 to c15d0
In-Reply-To: <1416408436.3020.3.camel@indy.rr.com>
References: <d0ee326b448f41efaa3fcef7c541b028@EX1301.steait.net>
	<1416408436.3020.3.camel@indy.rr.com>
Message-ID: <acdaabe8f3cd4f65a9c5531e405b9aca@EX1301.steait.net>

Ok, no other option? I am physically not near and its ?just? a test pool so I can kill it and start over.
If I try the import option, should it  recognize the metadata of the devices and reuse it?
Br,
Rune

From: Scott LeFevre [mailto:slefevre at indy.rr.com]
Sent: Wednesday, November 19, 2014 6:47 AM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] need to change c17d0 to c15d0

Put the drive/hba back and then export your pool (zpool export pool-name). Then move the drive/hba and import the pool (zpool import pool-name).
If putting the drive/hba back isn't an option, you could try a forced import with 'zpool import -f pool-name'.

Cheers,
--
Scott LeFevre
317-696-1010

On Wed, 2014-11-19 at 14:36 +0000, Rune Tipsmark wrote:
I moved one of my PCI-E IOdrives and the disks changed from c14d0 and  c15d0 to c16d0 and c17d0

How do I change it back so I can get my pool back online?



32. c16d0 <Unknown-Unknown-0001-298.02GB>

          /pci at 79,0/pci8086,3c02 at 1/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0

33. c17d0 <Unknown-Unknown-0001-298.02GB>

          /pci at 79,0/pci8086,3c02 at 1/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0



Br,

Rune






_______________________________________________

OmniOS-discuss mailing list

OmniOS-discuss at lists.omniti.com<mailto:OmniOS-discuss at lists.omniti.com>

http://lists.omniti.com/mailman/listinfo/omnios-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141119/faac5733/attachment-0001.html>

From richard.elling at richardelling.com  Wed Nov 19 14:58:42 2014
From: richard.elling at richardelling.com (Richard Elling)
Date: Wed, 19 Nov 2014 06:58:42 -0800
Subject: [OmniOS-discuss] need to change c17d0 to c15d0
In-Reply-To: <d0ee326b448f41efaa3fcef7c541b028@EX1301.steait.net>
References: <d0ee326b448f41efaa3fcef7c541b028@EX1301.steait.net>
Message-ID: <AB576C79-A8C9-47A7-BD3D-5E5F4C271349@richardelling.com>


On Nov 19, 2014, at 6:36 AM, Rune Tipsmark <rt at steait.net> wrote:

> I moved one of my PCI-E IOdrives and the disks changed from c14d0 and  c15d0 to c16d0 and c17d0
> How do I change it back so I can get my pool back online?

In most cases, you don't need to change it, just import.
 -- richard

>  
> 32. c16d0 <Unknown-Unknown-0001-298.02GB>
>           /pci at 79,0/pci8086,3c02 at 1/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
> 33. c17d0 <Unknown-Unknown-0001-298.02GB>
>           /pci at 79,0/pci8086,3c02 at 1/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
>  
> Br,
> Rune
>  
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141119/70730c4f/attachment.html>

From bfriesen at simple.dallas.tx.us  Wed Nov 19 15:00:23 2014
From: bfriesen at simple.dallas.tx.us (Bob Friesenhahn)
Date: Wed, 19 Nov 2014 09:00:23 -0600 (CST)
Subject: [OmniOS-discuss] need to change c17d0 to c15d0
In-Reply-To: <acdaabe8f3cd4f65a9c5531e405b9aca@EX1301.steait.net>
References: <d0ee326b448f41efaa3fcef7c541b028@EX1301.steait.net>
	<1416408436.3020.3.camel@indy.rr.com>
	<acdaabe8f3cd4f65a9c5531e405b9aca@EX1301.steait.net>
Message-ID: <alpine.GSO.2.01.1411190856170.18732@freddy.simplesystems.org>

On Wed, 19 Nov 2014, Rune Tipsmark wrote:

> 
> Ok, no other option? I am physically not near and its ?just? a test pool so I can kill it and start over.
> 
> If I try the import option, should it? recognize the metadata of the devices and reuse it?

Zfs puts identifying information on all the pool disks so that they 
may be properly imported by pool name without regard for device 
identifiers.  'zpool import' scans all of the available devices, 
searching for devices which are in that pool.

If you don't want to physically move the drives again, then just 
re-import the pool.

It is always a good idea to export a pool before moving its drives 
around and then import the pool afterward.  Usually it just takes a 
few seconds to import the pool.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

From rt at steait.net  Wed Nov 19 15:09:14 2014
From: rt at steait.net (Rune Tipsmark)
Date: Wed, 19 Nov 2014 15:09:14 +0000
Subject: [OmniOS-discuss] need to change c17d0 to c15d0
In-Reply-To: <alpine.GSO.2.01.1411190856170.18732@freddy.simplesystems.org>
References: <d0ee326b448f41efaa3fcef7c541b028@EX1301.steait.net>
	<1416408436.3020.3.camel@indy.rr.com>
	<acdaabe8f3cd4f65a9c5531e405b9aca@EX1301.steait.net>
	<alpine.GSO.2.01.1411190856170.18732@freddy.simplesystems.org>
Message-ID: <93f3c3b1e010444fbc981702beb6c981@EX1301.steait.net>

Ok makes sense, but can I just destroy my current pool in the failed state and then import it with the -f option and it should show up with all 4 drives...

pool: pool02
 state: UNAVAIL
status: One or more devices could not be opened.  There are insufficient
	replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-3C
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM      CAP            Product /napp-it   IOstat mess
	pool02      UNAVAIL      0     0     0  insufficient replicas
	  c12d0     ONLINE       0     0     0                       
	  c13d0     ONLINE       0     0     0                       
	  c14d0     UNAVAIL      0     0     0  cannot open                       
	  c15d0     UNAVAIL      0     0     0  cannot open            


-----Original Message-----
From: Bob Friesenhahn [mailto:bfriesen at simple.dallas.tx.us] 
Sent: Wednesday, November 19, 2014 7:00 AM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] need to change c17d0 to c15d0

On Wed, 19 Nov 2014, Rune Tipsmark wrote:

> 
> Ok, no other option? I am physically not near and its ?just? a test pool so I can kill it and start over.
> 
> If I try the import option, should it? recognize the metadata of the devices and reuse it?

Zfs puts identifying information on all the pool disks so that they may be properly imported by pool name without regard for device identifiers.  'zpool import' scans all of the available devices, searching for devices which are in that pool.

If you don't want to physically move the drives again, then just re-import the pool.

It is always a good idea to export a pool before moving its drives around and then import the pool afterward.  Usually it just takes a few seconds to import the pool.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

From bfriesen at simple.dallas.tx.us  Wed Nov 19 15:23:57 2014
From: bfriesen at simple.dallas.tx.us (Bob Friesenhahn)
Date: Wed, 19 Nov 2014 09:23:57 -0600 (CST)
Subject: [OmniOS-discuss] need to change c17d0 to c15d0
In-Reply-To: <93f3c3b1e010444fbc981702beb6c981@EX1301.steait.net>
References: <d0ee326b448f41efaa3fcef7c541b028@EX1301.steait.net>
	<1416408436.3020.3.camel@indy.rr.com>
	<acdaabe8f3cd4f65a9c5531e405b9aca@EX1301.steait.net>
	<alpine.GSO.2.01.1411190856170.18732@freddy.simplesystems.org>
	<93f3c3b1e010444fbc981702beb6c981@EX1301.steait.net>
Message-ID: <alpine.GSO.2.01.1411190921540.18732@freddy.simplesystems.org>

On Wed, 19 Nov 2014, Rune Tipsmark wrote:

> Ok makes sense, but can I just destroy my current pool in the failed 
> state and then import it with the -f option and it should show up 
> with all 4 drives...

That is actually a bit more work for you since two of the drives would 
be orphaned and look like they come from the former pool.  They will 
not be "destroyed" while they are still in state UNAVAIL.

It will take a bit more work to re-use these orphaned drives since zfs 
has precautions to avoid accidentally overwriting drives which are 
already in a pool.

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

From rt at steait.net  Wed Nov 19 15:30:48 2014
From: rt at steait.net (Rune Tipsmark)
Date: Wed, 19 Nov 2014 15:30:48 +0000
Subject: [OmniOS-discuss] need to change c17d0 to c15d0
In-Reply-To: <alpine.GSO.2.01.1411190921540.18732@freddy.simplesystems.org>
References: <d0ee326b448f41efaa3fcef7c541b028@EX1301.steait.net>
	<1416408436.3020.3.camel@indy.rr.com>
	<acdaabe8f3cd4f65a9c5531e405b9aca@EX1301.steait.net>
	<alpine.GSO.2.01.1411190856170.18732@freddy.simplesystems.org>
	<93f3c3b1e010444fbc981702beb6c981@EX1301.steait.net>
	<alpine.GSO.2.01.1411190921540.18732@freddy.simplesystems.org>
Message-ID: <a709b3182c454ccfba2cae09a606fdd2@EX1301.steait.net>

Zpool destroy pool02 and then zpool import -f pool02 worked.
Br,
Rune

-----Original Message-----
From: Bob Friesenhahn [mailto:bfriesen at simple.dallas.tx.us] 
Sent: Wednesday, November 19, 2014 7:24 AM
To: Rune Tipsmark
Cc: omnios-discuss at lists.omniti.com
Subject: RE: [OmniOS-discuss] need to change c17d0 to c15d0

On Wed, 19 Nov 2014, Rune Tipsmark wrote:

> Ok makes sense, but can I just destroy my current pool in the failed 
> state and then import it with the -f option and it should show up with 
> all 4 drives...

That is actually a bit more work for you since two of the drives would be orphaned and look like they come from the former pool.  They will not be "destroyed" while they are still in state UNAVAIL.

It will take a bit more work to re-use these orphaned drives since zfs has precautions to avoid accidentally overwriting drives which are already in a pool.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

From danmcd at omniti.com  Wed Nov 19 16:02:32 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Wed, 19 Nov 2014 11:02:32 -0500
Subject: [OmniOS-discuss] need to change c17d0 to c15d0
In-Reply-To: <acdaabe8f3cd4f65a9c5531e405b9aca@EX1301.steait.net>
References: <d0ee326b448f41efaa3fcef7c541b028@EX1301.steait.net>
	<1416408436.3020.3.camel@indy.rr.com>
	<acdaabe8f3cd4f65a9c5531e405b9aca@EX1301.steait.net>
Message-ID: <3AA91BE4-0415-4ADA-92E2-401FA42A352D@omniti.com>

Did you try "zpool replace" ?

Dan

Sent from my iPhone (typos, autocorrect, and all)

> On Nov 19, 2014, at 9:50 AM, Rune Tipsmark <rt at steait.net> wrote:
> 
> Ok, no other option? I am physically not near and its ?just? a test pool so I can kill it and start over.
> If I try the import option, should it  recognize the metadata of the devices and reuse it?
> Br,
> Rune
>  
> From: Scott LeFevre [mailto:slefevre at indy.rr.com] 
> Sent: Wednesday, November 19, 2014 6:47 AM
> To: Rune Tipsmark
> Cc: omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] need to change c17d0 to c15d0
>  
> Put the drive/hba back and then export your pool (zpool export pool-name). Then move the drive/hba and import the pool (zpool import pool-name).
> If putting the drive/hba back isn't an option, you could try a forced import with 'zpool import -f pool-name'.
> 
> Cheers,
> -- 
> Scott LeFevre
> 317-696-1010
> 
> On Wed, 2014-11-19 at 14:36 +0000, Rune Tipsmark wrote:
> I moved one of my PCI-E IOdrives and the disks changed from c14d0 and  c15d0 to c16d0 and c17d0
> 
> How do I change it back so I can get my pool back online?
> 
>  
> 
> 32. c16d0 <Unknown-Unknown-0001-298.02GB>
> 
>           /pci at 79,0/pci8086,3c02 at 1/pci10b5,8616 at 0/pci10b5,8616 at 5/pci103c,178e at 0
> 
> 33. c17d0 <Unknown-Unknown-0001-298.02GB>
> 
>           /pci at 79,0/pci8086,3c02 at 1/pci10b5,8616 at 0/pci10b5,8616 at 6/pci103c,178e at 0
> 
>  
> 
> Br,
> 
> Rune
> 
>  
> 
> 
>  
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141119/71ffedaf/attachment-0001.html>

From jboren at drakecooper.com  Wed Nov 19 23:35:51 2014
From: jboren at drakecooper.com (Joseph Boren)
Date: Wed, 19 Nov 2014 16:35:51 -0700
Subject: [OmniOS-discuss] list of know-compatible motherboards?
Message-ID: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>

Is anyone aware of a list, even a short list, of motherboards that are
known to be compatible with OmniOS?  The illumos HCL doesn't list any
motherboards.

Thanks,

-jb-
*Joseph Boren*

IT Specialist
*DRAKE COOPER*
+ c: (208) 891-2128 + o: (208) 342-0925
+ 416 S. 8th St., Boise, ID 83702
+ w: drakecooper.com + f: /drakecooper <http://facebook.com/drakecooper> +
t: @drakecooper <http://twitter.com/drakecooper>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141119/64df8e9a/attachment.html>

From fabio at fabiorabelo.wiki.br  Wed Nov 19 23:48:36 2014
From: fabio at fabiorabelo.wiki.br (=?UTF-8?Q?F=C3=A1bio_Rabelo?=)
Date: Wed, 19 Nov 2014 21:48:36 -0200
Subject: [OmniOS-discuss] list of know-compatible motherboards?
In-Reply-To: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>
References: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>
Message-ID: <CAEekY65+jM9aAhymmvQz2mywECH6s6nVYqczXdGkNRbmNTbZmA@mail.gmail.com>

I can show you what motherboards I have installed and fully working in the
customers of mine :

http://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8DG6-F.cfm

http://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8DG6.cfm

http://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8DGi-F.cfm

http://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8DGi.cfm

http://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8SGL-F.cfm

http://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8SGL.cfm

The ones with LSI SAS controler needs to be flashed with IT firmware, you
can find them in the official Supermicro FTP site :

ftp://ftp.supermicro.com/Driver/SAS/LSI/

Opterons from 8 to 24 cores, no issue whats soever ...

Some of them are up and running for over an year !!!


F?bio Rabelo

2014-11-19 21:35 GMT-02:00 Joseph Boren <jboren at drakecooper.com>:

> Is anyone aware of a list, even a short list, of motherboards that are
> known to be compatible with OmniOS?  The illumos HCL doesn't list any
> motherboards.
>
> Thanks,
>
> -jb-
> *Joseph Boren*
>
> IT Specialist
> *DRAKE COOPER*
> + c: (208) 891-2128 + o: (208) 342-0925
> + 416 S. 8th St., Boise, ID 83702
> + w: drakecooper.com + f: /drakecooper <http://facebook.com/drakecooper> +
>  t: @drakecooper <http://twitter.com/drakecooper>
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141119/f9c082ed/attachment.html>

From jboren at drakecooper.com  Wed Nov 19 23:54:26 2014
From: jboren at drakecooper.com (Joseph Boren)
Date: Wed, 19 Nov 2014 16:54:26 -0700
Subject: [OmniOS-discuss] list of know-compatible motherboards?
In-Reply-To: <CAEekY65+jM9aAhymmvQz2mywECH6s6nVYqczXdGkNRbmNTbZmA@mail.gmail.com>
References: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>
	<CAEekY65+jM9aAhymmvQz2mywECH6s6nVYqczXdGkNRbmNTbZmA@mail.gmail.com>
Message-ID: <CAGoaUhPwSz=4PwHZYEhkobW5=ZPzTp4+oWon6VFCOpPLj_PiFg@mail.gmail.com>

Wow, F?bio, thanks so much, that is very helpful.  I was looking at
supermicro motherboards, so your info is perfect.

I will have a look at those, I'm guessing I can find something that fits my
use case.  Thanks again, the help is much appreciated.

Best regards,

-jb-
*Joseph Boren*

IT Specialist
*DRAKE COOPER*
+ c: (208) 891-2128 + o: (208) 342-0925
+ 416 S. 8th St., Boise, ID 83702
+ w: drakecooper.com + f: /drakecooper <http://facebook.com/drakecooper> +
t: @drakecooper <http://twitter.com/drakecooper>


On Wed, Nov 19, 2014 at 4:48 PM, F?bio Rabelo <fabio at fabiorabelo.wiki.br>
wrote:

> I can show you what motherboards I have installed and fully working in the
> customers of mine :
>
> http://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8DG6-F.cfm
>
> http://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8DG6.cfm
>
> http://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8DGi-F.cfm
>
> http://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8DGi.cfm
>
> http://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8SGL-F.cfm
>
> http://www.supermicro.com/Aplus/motherboard/Opteron6000/SR56x0/H8SGL.cfm
>
> The ones with LSI SAS controler needs to be flashed with IT firmware, you
> can find them in the official Supermicro FTP site :
>
> ftp://ftp.supermicro.com/Driver/SAS/LSI/
>
> Opterons from 8 to 24 cores, no issue whats soever ...
>
> Some of them are up and running for over an year !!!
>
>
> F?bio Rabelo
>
> 2014-11-19 21:35 GMT-02:00 Joseph Boren <jboren at drakecooper.com>:
>
>> Is anyone aware of a list, even a short list, of motherboards that are
>> known to be compatible with OmniOS?  The illumos HCL doesn't list any
>> motherboards.
>>
>> Thanks,
>>
>> -jb-
>> *Joseph Boren*
>>
>> IT Specialist
>> *DRAKE COOPER*
>> + c: (208) 891-2128 + o: (208) 342-0925
>> + 416 S. 8th St., Boise, ID 83702
>> + w: drakecooper.com + f: /drakecooper <http://facebook.com/drakecooper>
>> + t: @drakecooper <http://twitter.com/drakecooper>
>>
>>
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141119/6e791504/attachment.html>

From doug at will.to  Wed Nov 19 23:58:44 2014
From: doug at will.to (Doug Hughes)
Date: Wed, 19 Nov 2014 18:58:44 -0500
Subject: [OmniOS-discuss] list of know-compatible motherboards?
In-Reply-To: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>
References: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>
Message-ID: <546D2EB4.9080806@will.to>


There are probably more that work than not. It might be easier to have a 
hardware incompatibility list.

On 11/19/2014 6:35 PM, Joseph Boren wrote:
> Is anyone aware of a list, even a short list, of motherboards that are 
> known to be compatible with OmniOS?  The illumos HCL doesn't list any 
> motherboards.
>
> Thanks,
>
> -jb-
>
> *Joseph Boren*
>
> IT Specialist
>
> *DRAKE COOPER*
> +c: (208) 891-2128 + o: (208) 342-0925
> +416 S. 8th St., Boise, ID 83702
> +w: drakecooper.com <http://drakecooper.com/> + f:/drakecooper 
> <http://facebook.com/drakecooper> + t:@drakecooper 
> <http://twitter.com/drakecooper>
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141119/e253c1f7/attachment-0001.html>

From guorong.koh at gmail.com  Thu Nov 20 00:12:06 2014
From: guorong.koh at gmail.com (Guo-Rong Koh)
Date: Thu, 20 Nov 2014 10:42:06 +1030
Subject: [OmniOS-discuss] list of know-compatible motherboards?
In-Reply-To: <CAGoaUhPwSz=4PwHZYEhkobW5=ZPzTp4+oWon6VFCOpPLj_PiFg@mail.gmail.com>
References: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>
	<CAEekY65+jM9aAhymmvQz2mywECH6s6nVYqczXdGkNRbmNTbZmA@mail.gmail.com>
	<CAGoaUhPwSz=4PwHZYEhkobW5=ZPzTp4+oWon6VFCOpPLj_PiFg@mail.gmail.com>
Message-ID: <1416442326.30851.2.camel@gmail.com>

On Wed, 2014-11-19 at 16:54 -0700, Joseph Boren wrote:
> Wow, F?bio, thanks so much, that is very helpful.  I was looking at
> supermicro motherboards, so your info is perfect.  
> 
> 
> I will have a look at those, I'm guessing I can find something that
> fits my use case.  Thanks again, the help is much appreciated.

I recently installed on a Supermicro X10SL7-F:
http://www.supermicro.com/products/motherboard/Xeon/C220/X10SL7-F.cfm

and have not had any issues (besides needing to flash the IT firmware
for the onboard LSI).

regards,
Guo-Rong



From jboren at drakecooper.com  Thu Nov 20 00:18:39 2014
From: jboren at drakecooper.com (Joseph Boren)
Date: Wed, 19 Nov 2014 17:18:39 -0700
Subject: [OmniOS-discuss] list of know-compatible motherboards?
In-Reply-To: <1416442326.30851.2.camel@gmail.com>
References: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>
	<CAEekY65+jM9aAhymmvQz2mywECH6s6nVYqczXdGkNRbmNTbZmA@mail.gmail.com>
	<CAGoaUhPwSz=4PwHZYEhkobW5=ZPzTp4+oWon6VFCOpPLj_PiFg@mail.gmail.com>
	<1416442326.30851.2.camel@gmail.com>
Message-ID: <CAGoaUhN2NC+3tfO0e6ZQODo+d8j4W58T5_v8D+dRw1hW=qnN_g@mail.gmail.com>

Hi Guo-Rong,

Thanks, that's a good one too.  I think it might be a little low on memory
capacity for my current project, but I have another build coming up that
might be perfect for.  Thanks for the help, much appreciated.

Best regards,

-jb-
*Joseph Boren*

IT Specialist
*DRAKE COOPER*
+ c: (208) 891-2128 + o: (208) 342-0925
+ 416 S. 8th St., Boise, ID 83702
+ w: drakecooper.com + f: /drakecooper <http://facebook.com/drakecooper> +
t: @drakecooper <http://twitter.com/drakecooper>


On Wed, Nov 19, 2014 at 5:12 PM, Guo-Rong Koh <guorong.koh at gmail.com> wrote:

> On Wed, 2014-11-19 at 16:54 -0700, Joseph Boren wrote:
> > Wow, F?bio, thanks so much, that is very helpful.  I was looking at
> > supermicro motherboards, so your info is perfect.
> >
> >
> > I will have a look at those, I'm guessing I can find something that
> > fits my use case.  Thanks again, the help is much appreciated.
>
> I recently installed on a Supermicro X10SL7-F:
> http://www.supermicro.com/products/motherboard/Xeon/C220/X10SL7-F.cfm
>
> and have not had any issues (besides needing to flash the IT firmware
> for the onboard LSI).
>
> regards,
> Guo-Rong
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141119/67186790/attachment.html>

From henson at acm.org  Thu Nov 20 00:47:08 2014
From: henson at acm.org (Paul B. Henson)
Date: Wed, 19 Nov 2014 16:47:08 -0800
Subject: [OmniOS-discuss] list of know-compatible motherboards?
In-Reply-To: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>
References: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>
Message-ID: <20141120004708.GV20693@bender.unx.csupomona.edu>

On Wed, Nov 19, 2014 at 04:35:51PM -0700, Joseph Boren wrote:
> Is anyone aware of a list, even a short list, of motherboards that are
> known to be compatible with OmniOS?  The illumos HCL doesn't list any
> motherboards.

Joyent publishes everything they use, mostly Supermicro gear:

http://eng.joyent.com/manufacturing/bom.html

If it's on their list, it's a pretty good certainty it works well :).
Obviously they use SmartOS, but I don't think there's much divergence in
terms of hardward support from OmniOS.


From mir at miras.org  Thu Nov 20 00:52:02 2014
From: mir at miras.org (Michael Rasmussen)
Date: Thu, 20 Nov 2014 01:52:02 +0100
Subject: [OmniOS-discuss] list of know-compatible motherboards?
In-Reply-To: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>
References: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>
Message-ID: <20141120015202.63259322@sleipner.datanom.net>

On Wed, 19 Nov 2014 16:35:51 -0700
Joseph Boren <jboren at drakecooper.com> wrote:

> Is anyone aware of a list, even a short list, of motherboards that are
> known to be compatible with OmniOS?  The illumos HCL doesn't list any
> motherboards.
> 
The generel rule:
Any server grade equipment from Intel and AMD is supported

For desktop grade equipment the support is more uncertain. I have succeeded using AMD
Hudson chipset (A75 socket FM1) based desktop board.

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
In Brooklyn, we had such great pennant races, it made the World Series
just something that came later.
		-- Walter O'Malley, Dodgers owner

----

This mail was virus scanned and spam checked before delivery.
This mail is also DKIM signed. See header dkim-signature.


From jboren at drakecooper.com  Thu Nov 20 01:01:00 2014
From: jboren at drakecooper.com (Joseph Boren)
Date: Wed, 19 Nov 2014 18:01:00 -0700
Subject: [OmniOS-discuss] list of know-compatible motherboards?
In-Reply-To: <20141120004708.GV20693@bender.unx.csupomona.edu>
References: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>
	<20141120004708.GV20693@bender.unx.csupomona.edu>
Message-ID: <CAGoaUhN7h-8ZiJRVNS1WFeX2CV0rW_oqt3ZNmNTdSNcz=+=Anw@mail.gmail.com>

Hi Paul,

Thanks a ton, holy cow, that's a detailed list, right down to heatsinks. :)
 Very helpful for stuff besides motherboards too.  Thank you very much.

Best regards,

-jb-
*Joseph Boren*

IT Specialist
*DRAKE COOPER*
+ c: (208) 891-2128 + o: (208) 342-0925
+ 416 S. 8th St., Boise, ID 83702
+ w: drakecooper.com + f: /drakecooper <http://facebook.com/drakecooper> +
t: @drakecooper <http://twitter.com/drakecooper>


On Wed, Nov 19, 2014 at 5:47 PM, Paul B. Henson <henson at acm.org> wrote:

> On Wed, Nov 19, 2014 at 04:35:51PM -0700, Joseph Boren wrote:
> > Is anyone aware of a list, even a short list, of motherboards that are
> > known to be compatible with OmniOS?  The illumos HCL doesn't list any
> > motherboards.
>
> Joyent publishes everything they use, mostly Supermicro gear:
>
> http://eng.joyent.com/manufacturing/bom.html
>
> If it's on their list, it's a pretty good certainty it works well :).
> Obviously they use SmartOS, but I don't think there's much divergence in
> terms of hardward support from OmniOS.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141119/1c62072f/attachment.html>

From danmcd at omniti.com  Thu Nov 20 20:29:25 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Thu, 20 Nov 2014 15:29:25 -0500
Subject: [OmniOS-discuss] OmniOS Bloody November 20th update
Message-ID: <76899095-B8AC-4366-98DA-A043B7ADD468@omniti.com>

New with this release (r151013-20141120):

- omnios-build master branch, revision 31f07b8

- illumos-omnios master branch, revision 51f5c62

- more man page fixes

- curl update to 7.39.0

- Addition of intltool.

- smbadm(1M) join may now use regular RPC (see Illumos #5316), as well as other SMB fixes.

- mpt_sas has improved auto request sense (see Illumos #5016)

- ZFS can now support larger blocks (see Illumos #5027) - USE WITH CARE.

No install media update for this one, use "pkg update", even after a fresh via-media installation.

Happy updating!
Dan


From mark0x01 at gmail.com  Fri Nov 21 07:44:55 2014
From: mark0x01 at gmail.com (Mark)
Date: Fri, 21 Nov 2014 20:44:55 +1300
Subject: [OmniOS-discuss] list of know-compatible motherboards?
In-Reply-To: <CAGoaUhN2NC+3tfO0e6ZQODo+d8j4W58T5_v8D+dRw1hW=qnN_g@mail.gmail.com>
References: <CAGoaUhMgnSUZK+ub_avHaaub0UZROmGiWnF+FKF7X9dVnuqYpg@mail.gmail.com>	<CAEekY65+jM9aAhymmvQz2mywECH6s6nVYqczXdGkNRbmNTbZmA@mail.gmail.com>	<CAGoaUhPwSz=4PwHZYEhkobW5=ZPzTp4+oWon6VFCOpPLj_PiFg@mail.gmail.com>	<1416442326.30851.2.camel@gmail.com>
	<CAGoaUhN2NC+3tfO0e6ZQODo+d8j4W58T5_v8D+dRw1hW=qnN_g@mail.gmail.com>
Message-ID: <546EED77.3030809@gmail.com>

The only issues I have had with Supermicro is the IMM's remote console 
on the SIMSO+ addin card.
The usb interface to kvm and virtual media can have problems.



On 20/11/2014 1:18 p.m., Joseph Boren wrote:
> Hi Guo-Rong,
>
> Thanks, that's a good one too.  I think it might be a little low on
> memory capacity for my current project, but I have another build coming
> up that might be perfect for.  Thanks for the help, much appreciated.
>
> Best regards,
>
> -jb-
>
> *Joseph Boren*
>
> IT Specialist
>
> *DRAKE COOPER*
> +c: (208) 891-2128 + o: (208) 342-0925
> +416 S. 8th St., Boise, ID 83702
> +w: drakecooper.com <http://drakecooper.com/> + f:/drakecooper
> <http://facebook.com/drakecooper> + t: @drakecooper
> <http://twitter.com/drakecooper>
>
>
> On Wed, Nov 19, 2014 at 5:12 PM, Guo-Rong Koh <guorong.koh at gmail.com
> <mailto:guorong.koh at gmail.com>> wrote:
>
>     On Wed, 2014-11-19 at 16:54 -0700, Joseph Boren wrote:
>     > Wow, F?bio, thanks so much, that is very helpful.  I was looking at
>     > supermicro motherboards, so your info is perfect.
>     >
>     >
>     > I will have a look at those, I'm guessing I can find something that
>     > fits my use case.  Thanks again, the help is much appreciated.
>
>     I recently installed on a Supermicro X10SL7-F:
>     http://www.supermicro.com/products/motherboard/Xeon/C220/X10SL7-F.cfm
>
>     and have not had any issues (besides needing to flash the IT firmware
>     for the onboard LSI).
>
>     regards,
>     Guo-Rong
>
>
>
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>


From hasslerd at gmx.li  Fri Nov 21 16:03:18 2014
From: hasslerd at gmx.li (Dominik Hassler)
Date: Fri, 21 Nov 2014 17:03:18 +0100
Subject: [OmniOS-discuss] HP Z230 as a home server
In-Reply-To: <alpine.UW2.2.11.1410251227390.8378@server01.int.multitalents.net>
References: <5446B314.9020600@gmx.li>
	<E5756AE2-0F13-4640-9E5B-C00656FF673C@omniti.com>
	<52DCA0DF-14A9-44F8-8B62-E29749D3645D@clozure.com>
	<5447E43D.5070303@gmx.li>
	<alpine.UW2.2.11.1410251227390.8378@server01.int.multitalents.net>
Message-ID: <546F6246.8080109@gmx.li>

got all the hardware and set up the system. it works great.

wanted to share the final HW setup in case someone plans to build a
similar system:

- MBD-X10SL7-F (LSI 2308 SAS flashed to IT firmware)
- Xeon E3-1231v3
- 2x WD Xe (SAS) 300G (rpool mirror)
- 2x WD Re (SAS) 4T   (data mirror)
- 32G samsung ECC ram according to supermicro compatibility list
- Lian Li PC-V358 case

thank you all for your help!


On 10/25/2014 09:35 PM, Tim Rice wrote:
> On Wed, 22 Oct 2014, Dominik Hassler wrote:
> 
> [snip]
>> - MBD-X10SLM-F-O mainboard
>> - Xeon E3-1231v3 (this time w/ HT)
>> - 4x 8 GB KTH-PL316EK4/32G
>> - 2x SSD DC S3500 (rpool mirror)
>> - 2x WD RE 4T (data mirror)
>> - Lian Li PC-V358 case
> [snip]
>> does anyone have (good or bad) experience with the kingston RAM
>> mentioned above?
> 
> Kingston RAM didn't work on a MBD-X9SCM-F-0 board here. Had to buy new ram.
> I recommend you stick with whatever RAM supermicro says is tested
> on your board.
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0xF9ECC6A5.asc
Type: application/pgp-keys
Size: 2686 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141121/c0705c78/attachment.bin>

From mir at miras.org  Fri Nov 21 16:24:18 2014
From: mir at miras.org (Michael Rasmussen)
Date: Fri, 21 Nov 2014 17:24:18 +0100
Subject: [OmniOS-discuss] HP Z230 as a home server
In-Reply-To: <546F6246.8080109@gmx.li>
References: <5446B314.9020600@gmx.li>
	<E5756AE2-0F13-4640-9E5B-C00656FF673C@omniti.com>
	<52DCA0DF-14A9-44F8-8B62-E29749D3645D@clozure.com>
	<5447E43D.5070303@gmx.li>
	<alpine.UW2.2.11.1410251227390.8378@server01.int.multitalents.net>
	<546F6246.8080109@gmx.li>
Message-ID: <20141121172418.0b05ea43@sleipner.datanom.net>

On Fri, 21 Nov 2014 17:03:18 +0100
Dominik Hassler <hasslerd at gmx.li> wrote:

> - 2x WD Re (SAS) 4T   (data mirror)
Are you sure its SAS and not SATA? I have not seen 2/4T SAS disks before.

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
If you lose your temper at a newspaper columnist, he'll get rich,
or famous or both.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141121/17e89c7f/attachment.bin>

From danmcd at omniti.com  Fri Nov 21 16:29:57 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Fri, 21 Nov 2014 11:29:57 -0500
Subject: [OmniOS-discuss] HP Z230 as a home server
In-Reply-To: <20141121172418.0b05ea43@sleipner.datanom.net>
References: <5446B314.9020600@gmx.li>
	<E5756AE2-0F13-4640-9E5B-C00656FF673C@omniti.com>
	<52DCA0DF-14A9-44F8-8B62-E29749D3645D@clozure.com>
	<5447E43D.5070303@gmx.li>
	<alpine.UW2.2.11.1410251227390.8378@server01.int.multitalents.net>
	<546F6246.8080109@gmx.li>
	<20141121172418.0b05ea43@sleipner.datanom.net>
Message-ID: <5587478E-CD1A-4FDA-88C8-1BF8061AFB45@omniti.com>


> On Nov 21, 2014, at 11:24 AM, Michael Rasmussen <mir at miras.org> wrote:
> 
> On Fri, 21 Nov 2014 17:03:18 +0100
> Dominik Hassler <hasslerd at gmx.li> wrote:
> 
>> - 2x WD Re (SAS) 4T   (data mirror)
> Are you sure its SAS and not SATA? I have not seen 2/4T SAS disks before.
> 

They exist:

	http://www.newegg.com/Product/Product.aspx?Item=N82E16822236353

Dan


From hasslerd at gmx.li  Fri Nov 21 16:31:14 2014
From: hasslerd at gmx.li (Dominik Hassler)
Date: Fri, 21 Nov 2014 17:31:14 +0100
Subject: [OmniOS-discuss] HP Z230 as a home server
In-Reply-To: <20141121172418.0b05ea43@sleipner.datanom.net>
References: <5446B314.9020600@gmx.li>	<E5756AE2-0F13-4640-9E5B-C00656FF673C@omniti.com>	<52DCA0DF-14A9-44F8-8B62-E29749D3645D@clozure.com>	<5447E43D.5070303@gmx.li>	<alpine.UW2.2.11.1410251227390.8378@server01.int.multitalents.net>	<546F6246.8080109@gmx.li>
	<20141121172418.0b05ea43@sleipner.datanom.net>
Message-ID: <546F68D2.8050302@gmx.li>

On 11/21/2014 05:24 PM, Michael Rasmussen wrote:
> On Fri, 21 Nov 2014 17:03:18 +0100
> Dominik Hassler <hasslerd at gmx.li> wrote:
> 
>> - 2x WD Re (SAS) 4T   (data mirror)
> Are you sure its SAS and not SATA? I have not seen 2/4T SAS disks before.

yeah, am sure:
- WD4001FYYG (
http://www.wdc.com/global/products/specs/?driveID=1184&language=1 )
> 
> 
> 
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0xF9ECC6A5.asc
Type: application/pgp-keys
Size: 2686 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141121/181a049b/attachment.bin>

From mir at miras.org  Fri Nov 21 17:12:44 2014
From: mir at miras.org (Michael Rasmussen)
Date: Fri, 21 Nov 2014 18:12:44 +0100
Subject: [OmniOS-discuss] HP Z230 as a home server
In-Reply-To: <546F68D2.8050302@gmx.li>
References: <5446B314.9020600@gmx.li>
	<E5756AE2-0F13-4640-9E5B-C00656FF673C@omniti.com>
	<52DCA0DF-14A9-44F8-8B62-E29749D3645D@clozure.com>
	<5447E43D.5070303@gmx.li>
	<alpine.UW2.2.11.1410251227390.8378@server01.int.multitalents.net>
	<546F6246.8080109@gmx.li>
	<20141121172418.0b05ea43@sleipner.datanom.net>
	<546F68D2.8050302@gmx.li>
Message-ID: <20141121181244.007711fa@sleipner.datanom.net>

On Fri, 21 Nov 2014 17:31:14 +0100
Dominik Hassler <hasslerd at gmx.li> wrote:

> 
> yeah, am sure:
> - WD4001FYYG (
> http://www.wdc.com/global/products/specs/?driveID=1184&language=1 )
> > 
Nice. Hopefully our SAN guys droppes SATA for the slow-tier in our EMC SAN then;-)

-- 
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
  An empty cab drove up and Sarah Bernhardt got out. -Arthur Baer,
  American comic and columnist
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141121/be8bd680/attachment.bin>

From contact at jacobvosmaer.nl  Sun Nov 23 19:35:01 2014
From: contact at jacobvosmaer.nl (Jacob Vosmaer)
Date: Sun, 23 Nov 2014 20:35:01 +0100
Subject: [OmniOS-discuss] Who here had lockd/nlockmgr problems?
In-Reply-To: <9A9651B5-B71B-44D3-90C1-BCF96B4ECCE8@omniti.com>
References: <968CF721-8839-49E2-8F04-9FD912E78E68@omniti.com>
	<9A9651B5-B71B-44D3-90C1-BCF96B4ECCE8@omniti.com>
Message-ID: <CAMcGgP2JBSzae3HBkMo0Wrg40MTor0dcFsACbUWbLiSBc0SFhQ@mail.gmail.com>

Hi Dan,

Thanks for sharing! I run OmniOS v11 r151012 on a home NFS server that gets
turned on and off a lot; I was pretty consistently getting the nlockmgr
maintenance state just after boot. I even went as far as to set the start
timeout to 10 minutes:

svccfg -s nlockmgr setprop \start/timeout_seconds=count: 600

And then I edited the /lib/svc/method/nlockmgr startup script to retry up
to 100 times:

diff --git a/nlockmgr b/nlockmgr
index 3a543d6..0b24bbc 100755
--- a/nlockmgr
+++ b/nlockmgr
@@ -61,4 +61,12 @@ then
        fi
 fi

-exec /usr/lib/nfs/lockd
+i=0
+while ! (echo "$0: starting lockd (attempt $i)"; /usr/lib/nfs/lockd) ; do
+  if [ $i -ge 100 ] ; then
+    exit 1
+  fi
+  i=$((i + 1))
+done
+
+exit 0

An ugly hack but at least it freed me from having to baby-sit the NFS
server after boot.

But then I saw your email and I decided to just remove the contents of
/var/statmon/sm.bak:

rm /var/statmon/sm.bak/*

I have not had any nlockmgr trouble since. Yay! I only did the sm.bak
cleanup today though.

Best wishes,

Jacob



2014-11-17 16:12 GMT+01:00 Dan McDonald <danmcd at omniti.com>:

> ISTR someone here had a problem where his/her nlockmgr SMF service
> wouldn't start.  I've encountered this problem myself recently (on a OI VM,
> but it's running the same new open-source lockd that's in OmniOS with
> r151010 and later), and wanted to share a conversation from the illumos
> developer's list.
>
> FYI,
> Dan
>
>
> > Begin forwarded message:
> >
> > Subject: Re: [developer] lockd not starting?
> > From: Dan McDonald <danmcd at omniti.com>
> > Date: November 17, 2014 at 10:10:38 AM EST
> > Cc: illumos Developer <developer at lists.illumos.org>, Matt Amdur <
> matt.amdur at delphix.com>
> > To: Sebastien Roy <sebastien.roy at delphix.com>
> >
> >
> >> On Nov 16, 2014, at 7:54 PM, Sebastien Roy <sebastien.roy at delphix.com>
> wrote:
> >>
> >> Hey Dan,
> >>
> >> I think we're looking into a similar problem here at Delphix. We've
> noticed that the nfs/nlockmgr service goes into maintenance mode after a
> timeout of the SM_CRASH call to statd.
> >>
> >> Upon startup, the statd daemon is blocked notifying clients. These
> clients are cached in /var/statmon/sm.bak/... If any of these clients are
> unreachable, the SM_CRASH call issued by klm will timeout (this timeout is
> much shorter than that which statd uses to give up on client
> notifications), and the nfs/nlockmgr service will then end up in
> maintenance state.
> >>
> >> We've been discussing possible fixes to this problem. One easy fix that
> Matt Amdur (Cc'ed) proposed would be to shorten the timeout that statd uses
> to notify clients to reduce the chance that lockd's SM_CRASH call would
> timeout in the event of an unreachable client. We haven't implemented the
> fix yet.
> >>
> >> A workaround is to clear statd's cached clients.
> >
> > Okay!  This makes some modicum of sense.  Here's what I have now:
> >
> > # ls -lt /var/statmon/sm.bak/
> > total 2
> > lrwxrwxrwx   1 daemon   daemon        10 Nov 12 19:55 ipv4.10.0.1.68 ->
> Everywhere
> > lrwxrwxrwx   1 daemon   daemon        28 Nov 11 12:44 ipv4.10.8.3.241 ->
> everywhere.office.omniti.com
> > # nslookup everywhere
> > Server:         10.8.3.1
> > Address:        10.8.3.1#53
> >
> > Name:   everywhere.office.omniti.com
> > Address: 10.8.3.241
> >
> > #
> >
> > While travelling last week, I had to park myself on a 10.0.1.0/24
> network.  My VMs are configured to be link-sharers (same-link peers), so I
> was using NFS over the same link to my VM.  Once I renumbered back to the
> intended OmniTI networks, things started going screwy.
> >
> > I'm going to remove the 10.0.1.68 one and try again... BINGO!
> >
> > # svcs -xv nlockmgr
> > svc:/network/nfs/nlockmgr:default (NFS lock manager)
> > State: maintenance since November 16, 2014 04:00:05 PM EST
> > Reason: Start method failed repeatedly, last exited with status 1.
> >   See: http://illumos.org/msg/SMF-8000-KS
> >   See: man -M /usr/share/man -s 1M lockd
> >   See: /var/svc/log/network-nfs-nlockmgr:default.log
> > Impact: 1 dependent service is not running:
> >        svc:/network/nfs/server:default
> > # rm /var/statmon/sm.bak/ipv4.10.0.1.68
> > # svcadm clear nlockmgr
> > # svcs -xv nlockmgr
> > svc:/network/nfs/nlockmgr:default (NFS lock manager)
> > State: online since November 17, 2014 10:08:53 AM EST
> >   See: man -M /usr/share/man -s 1M lockd
> >   See: /var/svc/log/network-nfs-nlockmgr:default.log
> > Impact: None.
> > #
> >
> > This is actually good, in that I believe the circumstances for this bug
> are now reproducible.
> >
> > Thanks!
> > Dan
> >
> >
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141123/54d7af35/attachment-0001.html>

From danmcd at omniti.com  Sun Nov 23 19:50:16 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Sun, 23 Nov 2014 14:50:16 -0500
Subject: [OmniOS-discuss] Who here had lockd/nlockmgr problems?
In-Reply-To: <CAMcGgP2JBSzae3HBkMo0Wrg40MTor0dcFsACbUWbLiSBc0SFhQ@mail.gmail.com>
References: <968CF721-8839-49E2-8F04-9FD912E78E68@omniti.com>
	<9A9651B5-B71B-44D3-90C1-BCF96B4ECCE8@omniti.com>
	<CAMcGgP2JBSzae3HBkMo0Wrg40MTor0dcFsACbUWbLiSBc0SFhQ@mail.gmail.com>
Message-ID: <7074558C-08DE-4647-9D29-A06854838189@omniti.com>


> On Nov 23, 2014, at 2:35 PM, Jacob Vosmaer <contact at jacobvosmaer.nl> wrote:
> 
> Hi Dan,
> 
> Thanks for sharing! I run OmniOS v11 r151012 on a home NFS server that gets turned on and off a lot; I was pretty consistently getting the nlockmgr maintenance state just after boot. I even went as far as to set the start timeout to 10 minutes:

<SNIP!>

> But then I saw your email and I decided to just remove the contents of /var/statmon/sm.bak:
> 
> rm /var/statmon/sm.bak/*
> 
> I have not had any nlockmgr trouble since. Yay! I only did the sm.bak cleanup today though.

This is still an ugly hack, BTW.  Just perhaps less ugly than your old one.  :)

I'd highly recommend you track the illumos bug in question, just in case someone comes up with a better solution.

Thanks for the additional confirmation, though!
Dan


From henson at acm.org  Mon Nov 24 18:58:59 2014
From: henson at acm.org (Paul B. Henson)
Date: Mon, 24 Nov 2014 10:58:59 -0800
Subject: [OmniOS-discuss] r151012 new zpool features
Message-ID: <20141124185859.GA29549@bender.unx.csupomona.edu>

I just updated my storage server to r151012 and am evaluating which new
zpool features to enable. Looks like the new ones include enabled_txg,
hole_birth, embedded_data, bookmarks, and filesystem_limits.

I don't think I'd get much use out of any of them but embedded_data. I
was wondering which ones people who've already updated to r151012 have
enabled and if anyone has noticed any issues with any of them. I think
last time I updated I just ended up enabling all of them so zpool status
would stop whining about missing features :). Is there any reason not to
just turn them all on and be done with it?

Thanks...


From danmcd at omniti.com  Mon Nov 24 19:12:08 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Mon, 24 Nov 2014 14:12:08 -0500
Subject: [OmniOS-discuss] r151012 new zpool features
In-Reply-To: <20141124185859.GA29549@bender.unx.csupomona.edu>
References: <20141124185859.GA29549@bender.unx.csupomona.edu>
Message-ID: <01238CE4-DCBB-400A-B478-30C20E29036F@omniti.com>


> On Nov 24, 2014, at 1:58 PM, Paul B. Henson <henson at acm.org> wrote:
> 
> Is there any reason not to
> just turn them all on and be done with it?

The ONLY reason to not turn them all on is if you think the pool(s) in question will be used on a pre-012 system.

If you don't think that's the case, then you definitely should enable all the new feature flags.

Dan


From henson at acm.org  Mon Nov 24 19:18:32 2014
From: henson at acm.org (Paul B. Henson)
Date: Mon, 24 Nov 2014 11:18:32 -0800
Subject: [OmniOS-discuss] r151012 new zpool features
In-Reply-To: <01238CE4-DCBB-400A-B478-30C20E29036F@omniti.com>
References: <20141124185859.GA29549@bender.unx.csupomona.edu>
	<01238CE4-DCBB-400A-B478-30C20E29036F@omniti.com>
Message-ID: <20141124191832.GB29549@bender.unx.csupomona.edu>

On Mon, Nov 24, 2014 at 02:12:08PM -0500, Dan McDonald wrote:

> The ONLY reason to not turn them all on is if you think the pool(s) in
> question will be used on a pre-012 system.
> 
> If you don't think that's the case, then you definitely should enable
> all the new feature flags.

Ah, ok, thanks... In that case, I think I'll let it burn in on the new
version for a few weeks to make sure nothing pops up that would require
reverting back to the previous BE and then flip them on.

From danmcd at omniti.com  Mon Nov 24 20:18:44 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Mon, 24 Nov 2014 15:18:44 -0500
Subject: [OmniOS-discuss] r151012 new zpool features
In-Reply-To: <5bdba129b4dea92256fd6cbcd45fe56f@blackdot.be>
References: <20141124185859.GA29549@bender.unx.csupomona.edu>
	<01238CE4-DCBB-400A-B478-30C20E29036F@omniti.com>
	<20141124191832.GB29549@bender.unx.csupomona.edu>
	<5bdba129b4dea92256fd6cbcd45fe56f@blackdot.be>
Message-ID: <19E9F077-1AFE-4E3B-8048-729761C4F357@omniti.com>


> On Nov 24, 2014, at 3:17 PM, Jorge Schrauwen <sjorge+ml at blackdot.be> wrote:
> 
> Is a 'zpool upgrade <pool>/ sufficient? Or do they need to be turned on manually?

"zpool upgrade <poolname-no-trailing-slash>"  is sufficient to enable all the feature flags.

Dan


From sjorge+ml at blackdot.be  Mon Nov 24 20:19:43 2014
From: sjorge+ml at blackdot.be (Jorge Schrauwen)
Date: Mon, 24 Nov 2014 21:19:43 +0100
Subject: [OmniOS-discuss] r151012 new zpool features
In-Reply-To: <19E9F077-1AFE-4E3B-8048-729761C4F357@omniti.com>
References: <20141124185859.GA29549@bender.unx.csupomona.edu>
	<01238CE4-DCBB-400A-B478-30C20E29036F@omniti.com>
	<20141124191832.GB29549@bender.unx.csupomona.edu>
	<5bdba129b4dea92256fd6cbcd45fe56f@blackdot.be>
	<19E9F077-1AFE-4E3B-8048-729761C4F357@omniti.com>
Message-ID: <612d88b9b1a841bbe8c4bf6d0c3c5d7e@blackdot.be>

Cool, then I am good :)

And that was an EPHONE, that slash was supose to be a ' :p

On 2014-11-24 21:18, Dan McDonald wrote:
>> On Nov 24, 2014, at 3:17 PM, Jorge Schrauwen <sjorge+ml at blackdot.be> 
>> wrote:
>> 
>> Is a 'zpool upgrade <pool>/ sufficient? Or do they need to be turned 
>> on manually?
> 
> "zpool upgrade <poolname-no-trailing-slash>"  is sufficient to enable
> all the feature flags.
> 
> Dan

From sjorge+ml at blackdot.be  Mon Nov 24 20:17:35 2014
From: sjorge+ml at blackdot.be (Jorge Schrauwen)
Date: Mon, 24 Nov 2014 21:17:35 +0100
Subject: [OmniOS-discuss] r151012 new zpool features
In-Reply-To: <20141124191832.GB29549@bender.unx.csupomona.edu>
References: <20141124185859.GA29549@bender.unx.csupomona.edu>
	<01238CE4-DCBB-400A-B478-30C20E29036F@omniti.com>
	<20141124191832.GB29549@bender.unx.csupomona.edu>
Message-ID: <5bdba129b4dea92256fd6cbcd45fe56f@blackdot.be>

Is a 'zpool upgrade <pool>/ sufficient? Or do they need to be turned on 
manually?



On 2014-11-24 20:18, Paul B. Henson wrote:
> On Mon, Nov 24, 2014 at 02:12:08PM -0500, Dan McDonald wrote:
> 
>> The ONLY reason to not turn them all on is if you think the pool(s) in
>> question will be used on a pre-012 system.
>> 
>> If you don't think that's the case, then you definitely should enable
>> all the new feature flags.
> 
> Ah, ok, thanks... In that case, I think I'll let it burn in on the new
> version for a few weeks to make sure nothing pops up that would require
> reverting back to the previous BE and then flip them on.
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

From filip.marvan at aira.cz  Tue Nov 25 15:00:59 2014
From: filip.marvan at aira.cz (Filip Marvan)
Date: Tue, 25 Nov 2014 16:00:59 +0100
Subject: [OmniOS-discuss] Parity error on path mpt_sas2
Message-ID: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAAD0@AIRA-SRV.aira.local>

Hi,

 

I have OmniOS 12 with three RAIDZ pools.

In /var/adm/messages a have these messages about every few seconds:

 

Nov 25 15:30:52 servername scsi: [ID 107833 kern.notice]
/pci at 0,0/pci8086,e08 at 3/pci1000,3020 at 0 (mpt_sas0):

Nov 25 15:30:52 servername      Aborted_command!

Nov 25 15:30:52 servername scsi: [ID 243001 kern.warning] WARNING:
/scsi_vhci (scsi_vhci0):

Nov 25 15:30:52 servername      /scsi_vhci/disk at g50014ee209cf0116 (sd13):
Parity Error on path mpt_sas2/disk at w50014ee209cf0116,0

Nov 25 15:30:52 servername scsi: [ID 107833 kern.notice]
/pci at 0,0/pci8086,e08 at 3/pci1000,3020 at 0 (mpt_sas0):

Nov 25 15:30:52 servername      Aborted_command!

Nov 25 15:30:53 servername scsi: [ID 107833 kern.notice]
/pci at 0,0/pci8086,e08 at 3/pci1000,3020 at 0 (mpt_sas0):

Nov 25 15:30:53 servername      Aborted_command!

 

In this log messages there is error on sd13, but there are similar parity
errors on four other disks too. All disks with parity errors are in the same
RAIDZ pool. 

All disks seems to be OK in zpool status and there are no errors on disks
reported by smartctl. There are no problems reported by fmdump 

 

All three pools using the same LSI 9211-8i HBA with firmware version 20.

 

Sometimes that pool have very slow responses even on "zpool status" command,
sometimes works normaly.

 

Do you have any idea where could be a problem? What else I can check?

 

Thank you very much for any help,

 

Filip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141125/43d06166/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6247 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141125/43d06166/attachment-0001.bin>

From danmcd at omniti.com  Tue Nov 25 15:26:01 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Tue, 25 Nov 2014 10:26:01 -0500
Subject: [OmniOS-discuss] Parity error on path mpt_sas2
In-Reply-To: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAAD0@AIRA-SRV.aira.local>
References: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAAD0@AIRA-SRV.aira.local>
Message-ID: <1439760B-8D8B-40F9-8A62-4CFE4A9483C1@omniti.com>


> On Nov 25, 2014, at 10:00 AM, Filip Marvan <filip.marvan at aira.cz> wrote:
> 
> Hi,
>  
> I have OmniOS 12 with three RAIDZ pools.
> In /var/adm/messages a have these messages about every few seconds:
>  
> Nov 25 15:30:52 servername scsi: [ID 107833 kern.notice] /pci at 0,0/pci8086,e08 at 3/pci1000,3020 at 0 (mpt_sas0):
> Nov 25 15:30:52 servername      Aborted_command!
> Nov 25 15:30:52 servername scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
> Nov 25 15:30:52 servername      /scsi_vhci/disk at g50014ee209cf0116 (sd13): Parity Error on path mpt_sas2/disk at w50014ee209cf0116,0
> Nov 25 15:30:52 servername scsi: [ID 107833 kern.notice] /pci at 0,0/pci8086,e08 at 3/pci1000,3020 at 0 (mpt_sas0):
> Nov 25 15:30:52 servername      Aborted_command!
> Nov 25 15:30:53 servername scsi: [ID 107833 kern.notice] /pci at 0,0/pci8086,e08 at 3/pci1000,3020 at 0 (mpt_sas0):
> Nov 25 15:30:53 servername      Aborted_command!
>  
> In this log messages there is error on sd13, but there are similar parity errors on four other disks too. All disks with parity errors are in the same RAIDZ pool. 
> All disks seems to be OK in zpool status and there are no errors on disks reported by smartctl. There are no problems reported by fmdump 
>  
> All three pools using the same LSI 9211-8i HBA with firmware version 20.
>  
> Sometimes that pool have very slow responses even on "zpool status" command, sometimes works normaly.
>  
> Do you have any idea where could be a problem? What else I can check?
>  

You're not being clear enough.  Is sd13/g50014ee209cf0116 the only disk reporting errors?  If so, replace it with a new disk.  If there are other disks reporting errors, you might want to replace them as well.  It looks like the SCSI commands are failing, but perhaps succeeding upon retry.  I'd check your cabling and see which disks are reporting the problems.  If you bought them in batches, maybe you have a bad lot?

Dan


From filip.marvan at aira.cz  Tue Nov 25 15:30:39 2014
From: filip.marvan at aira.cz (Filip Marvan)
Date: Tue, 25 Nov 2014 16:30:39 +0100
Subject: [OmniOS-discuss] Parity error on path mpt_sas2
In-Reply-To: <1439760B-8D8B-40F9-8A62-4CFE4A9483C1@omniti.com>
References: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAAD0@AIRA-SRV.aira.local>
	<1439760B-8D8B-40F9-8A62-4CFE4A9483C1@omniti.com>
Message-ID: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAADA@AIRA-SRV.aira.local>

Hi Dan,

thanks for reply.
Yes, errors are on all 5 disks in the same RAIDZ pool. No problem on other
disks in different pool but on the same SAS cable.
So maybe I only had bad luck with that disks.

Filip

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6247 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141125/d4c9a0ad/attachment.bin>

From asc1111 at gmail.com  Tue Nov 25 16:45:31 2014
From: asc1111 at gmail.com (Aaron Curry)
Date: Tue, 25 Nov 2014 09:45:31 -0700
Subject: [OmniOS-discuss] Parity error on path mpt_sas2
In-Reply-To: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAADA@AIRA-SRV.aira.local>
References: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAAD0@AIRA-SRV.aira.local>
	<1439760B-8D8B-40F9-8A62-4CFE4A9483C1@omniti.com>
	<3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAADA@AIRA-SRV.aira.local>
Message-ID: <CAOqBcP_12FaH77o+3nrGVm7KU3x1XCk3B-9uenKRRL63t5wdyg@mail.gmail.com>

I had this exact same problem recently when setting up a new home server...
same controller, same firmware (P20). The errors were on all disks attached
to the controller, but only on high read activity. Writes did not generate
the errors. I downgraded the firmware to P18 (which is what we are using at
work) and the errors went away. Has anyone had success with this controller
and the P20 firmware?

Aaron

On Tue, Nov 25, 2014 at 8:30 AM, Filip Marvan <filip.marvan at aira.cz> wrote:

> Hi Dan,
>
> thanks for reply.
> Yes, errors are on all 5 disks in the same RAIDZ pool. No problem on other
> disks in different pool but on the same SAS cable.
> So maybe I only had bad luck with that disks.
>
> Filip
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141125/7e430a8a/attachment.html>

From slefevre at indy.rr.com  Tue Nov 25 17:45:49 2014
From: slefevre at indy.rr.com (Scott LeFevre)
Date: Tue, 25 Nov 2014 12:45:49 -0500
Subject: [OmniOS-discuss] Parity error on path mpt_sas2
In-Reply-To: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAADA@AIRA-SRV.aira.local>
References: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAAD0@AIRA-SRV.aira.local>
	<1439760B-8D8B-40F9-8A62-4CFE4A9483C1@omniti.com>
	<3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAADA@AIRA-SRV.aira.local>
Message-ID: <1416937549.10798.11.camel@indy.rr.com>

On Tue, 2014-11-25 at 16:30 +0100, Filip Marvan wrote:

> Hi Dan,
> 
> thanks for reply.
> Yes, errors are on all 5 disks in the same RAIDZ pool. No problem on other
> disks in different pool but on the same SAS cable.
> So maybe I only had bad luck with that disks.
> 
> Filip
> 
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss


Run 'sasinfo hba-port -vly' and look at the output for Link Error
Statistic counters.  You may need to 'pkg install
system/storage/sasinfo' first.

I've got the same hba (not sure of the firmware rev.) but ran into the
exact same issue.  My issue was the power supply and the small gauge
wires used to power the drives.  I installed a new power supply w/ the
same wattage but heavier wires and all my issues disappeared.

I stumbled upon the solution by connecting a separate power supply to
one of the drives and hot wiring the MB connector to power it on (i.e.
green wire to any black w/ paper clip).  All the issues disappeared for
the drive on the separate power supply while the other continued to have
issues.

All my drives have run error free for a months since the new power
supply. YMMV!

Cheers,
-- 
Scott LeFevre
317-696-1010 



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141125/c104b957/attachment.html>

From steve at linuxsuite.org  Tue Nov 25 19:31:36 2014
From: steve at linuxsuite.org (steve at linuxsuite.org)
Date: Tue, 25 Nov 2014 14:31:36 -0500
Subject: [OmniOS-discuss] Scrub behavour and set readonly=on
Message-ID: <ffa8c9a3d6e514320f01e9112ec540a0.squirrel@emailmg.netfirms.com>


 Howdy

       zfs get all

        AND

      zpool get all

        have settings for readonly

        readonly       off         default

     if I set readonly=on

       will this affect scrub and its ability to fix errors?

       For both zfs and zpool settings?

      If I set
         zfs set readonly=on

      scrub will start and not complain... can it still fix errors?

            What about for

              zpool set readonly=on

          thanx - steve 	omnios-discuss at lists.omniti.com



From gate03 at landcroft.co.uk  Tue Nov 25 19:51:13 2014
From: gate03 at landcroft.co.uk (Michael Mounteney)
Date: Wed, 26 Nov 2014 05:51:13 +1000
Subject: [OmniOS-discuss] Building Haskell on OmniOS
In-Reply-To: <20141118071617.7851b06c@emeritus>
References: <20141117085117.7e0e3e77@emeritus>
	<5B4B585D-2335-4FA1-8E8D-8A504B6F4EBB@omniti.com>
	<20141118071617.7851b06c@emeritus>
Message-ID: <20141126055113.65750c96@punda-mlia>

On Tue, 18 Nov 2014 07:16:17 +1000
Michael Mounteney <gate03 at landcroft.co.uk> wrote:

> On Sun, 16 Nov 2014 17:57:25 -0500
> Dan McDonald <danmcd at omniti.com> wrote:
> 
> > Have you tried 151012, the current stable release?
> 
> That yields a different failure, for which I have no ready
> explanation:

[...]

Same with bloody.  I assume the symbols have been moved into other
libraries, and the ghc configure script hasn't caught up yet.

Michael.

From chip at innovates.com  Tue Nov 25 21:04:35 2014
From: chip at innovates.com (Schweiss, Chip)
Date: Tue, 25 Nov 2014 15:04:35 -0600
Subject: [OmniOS-discuss] Installer dependency on a Floppy
Message-ID: <CALeZrrQzSR0hqb1=xw=2jPERRgu1bBejbjo19B7NFWyP1yvPKQ@mail.gmail.com>

Started an installation a new Haswell based Supermicro server.  (*X10DRU-i+
* <http://www.supermicro.com/products/motherboard/Xeon/C600/X10DRU-i_.cfm>motherboard)


The floppy controller is completely gone from these systems and causing the
installer to panic when loading.

Anyone aware of a workaround?

-Chip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141125/d14dbd2a/attachment.html>

From jim.oltman at gmail.com  Tue Nov 25 21:17:19 2014
From: jim.oltman at gmail.com (Jim Oltman)
Date: Tue, 25 Nov 2014 14:17:19 -0700
Subject: [OmniOS-discuss] Installer dependency on a Floppy
In-Reply-To: <CALeZrrQzSR0hqb1=xw=2jPERRgu1bBejbjo19B7NFWyP1yvPKQ@mail.gmail.com>
References: <CALeZrrQzSR0hqb1=xw=2jPERRgu1bBejbjo19B7NFWyP1yvPKQ@mail.gmail.com>
Message-ID: <CA+Rs3Oe2p-Cr5XS0XM0WxvUKzSkq-3XR3p6D3gofAT6o4rLVKQ@mail.gmail.com>

On ESXi, I had to add the floppy controller in order to get past the panic.

On Tue, Nov 25, 2014 at 2:04 PM, Schweiss, Chip <chip at innovates.com> wrote:

> Started an installation a new Haswell based Supermicro server.  (*X10DRU-i+
> * <http://www.supermicro.com/products/motherboard/Xeon/C600/X10DRU-i_.cfm>motherboard)
>
>
> The floppy controller is completely gone from these systems and causing
> the installer to panic when loading.
>
> Anyone aware of a workaround?
>
> -Chip
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141125/d70876e5/attachment.html>

From chip at innovates.com  Tue Nov 25 21:22:54 2014
From: chip at innovates.com (Schweiss, Chip)
Date: Tue, 25 Nov 2014 15:22:54 -0600
Subject: [OmniOS-discuss] Installer dependency on a Floppy
In-Reply-To: <CA+Rs3Oe2p-Cr5XS0XM0WxvUKzSkq-3XR3p6D3gofAT6o4rLVKQ@mail.gmail.com>
References: <CALeZrrQzSR0hqb1=xw=2jPERRgu1bBejbjo19B7NFWyP1yvPKQ@mail.gmail.com>
	<CA+Rs3Oe2p-Cr5XS0XM0WxvUKzSkq-3XR3p6D3gofAT6o4rLVKQ@mail.gmail.com>
Message-ID: <CALeZrrSMkZNye4BBBjGW2sR=nQpu5hFcNSDvh=_F5ja_RyPQjg@mail.gmail.com>

On Tue, Nov 25, 2014 at 3:17 PM, Jim Oltman <jim.oltman at gmail.com> wrote:

> On ESXi, I had to add the floppy controller in order to get past the panic.
>

Same here.  Unfortunately this is real hardware with no floppy controller
of any kind.   Even the IPMI has dropped the virtual floppy.



> On Tue, Nov 25, 2014 at 2:04 PM, Schweiss, Chip <chip at innovates.com>
> wrote:
>
>> Started an installation a new Haswell based Supermicro server.  (*X10DRU-i+
>> *
>> <http://www.supermicro.com/products/motherboard/Xeon/C600/X10DRU-i_.cfm>motherboard)
>>
>>
>> The floppy controller is completely gone from these systems and causing
>> the installer to panic when loading.
>>
>> Anyone aware of a workaround?
>>
>> -Chip
>>
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141125/920506a3/attachment.html>

From henson at acm.org  Tue Nov 25 21:34:03 2014
From: henson at acm.org (Paul B. Henson)
Date: Tue, 25 Nov 2014 13:34:03 -0800
Subject: [OmniOS-discuss] Installer dependency on a Floppy
In-Reply-To: <CALeZrrQzSR0hqb1=xw=2jPERRgu1bBejbjo19B7NFWyP1yvPKQ@mail.gmail.com>
References: <CALeZrrQzSR0hqb1=xw=2jPERRgu1bBejbjo19B7NFWyP1yvPKQ@mail.gmail.com>
Message-ID: <20141125213403.GE29549@bender.unx.csupomona.edu>

On Tue, Nov 25, 2014 at 03:04:35PM -0600, Schweiss, Chip wrote:

> The floppy controller is completely gone from these systems and causing the
> installer to panic when loading.
> 
> Anyone aware of a workaround?

Check the BIOS and see if it says anything about a floppy. For vmware
guests, even if you don't have a virtual floppy configured, the BIOS
still shows that a floppy is installed. Most people recommend adding a
virtual floppy to work around the issue, but you can also go into the
virtual BIOS and change the floppy to not installed, which also allows
the installer to work.

So if the BIOS mentions floppies, make sure they're configured as not
installed...


From chip at innovates.com  Tue Nov 25 21:41:35 2014
From: chip at innovates.com (Schweiss, Chip)
Date: Tue, 25 Nov 2014 15:41:35 -0600
Subject: [OmniOS-discuss] Installer dependency on a Floppy
In-Reply-To: <20141125213403.GE29549@bender.unx.csupomona.edu>
References: <CALeZrrQzSR0hqb1=xw=2jPERRgu1bBejbjo19B7NFWyP1yvPKQ@mail.gmail.com>
	<20141125213403.GE29549@bender.unx.csupomona.edu>
Message-ID: <CALeZrrQB0UC25tZFvWsyBggE0AcubF3QvLtFZU2KmD1_P-WsZg@mail.gmail.com>

On Tue, Nov 25, 2014 at 3:34 PM, Paul B. Henson <henson at acm.org> wrote:

> On Tue, Nov 25, 2014 at 03:04:35PM -0600, Schweiss, Chip wrote:
>
> > The floppy controller is completely gone from these systems and causing
> the
> > installer to panic when loading.
> >
> > Anyone aware of a workaround?
>
> Check the BIOS and see if it says anything about a floppy. For vmware
> guests, even if you don't have a virtual floppy configured, the BIOS
> still shows that a floppy is installed. Most people recommend adding a
> virtual floppy to work around the issue, but you can also go into the
> virtual BIOS and change the floppy to not installed, which also allows
> the installer to work.
>
> So if the BIOS mentions floppies, make sure they're configured as not
> installed...
>

That was my first place to look.    Floppy is truly non-existent.

I just tried plugging a USB floppy in.  That didn't help either.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141125/9dce2ca3/attachment.html>

From chip at innovates.com  Tue Nov 25 21:49:46 2014
From: chip at innovates.com (Schweiss, Chip)
Date: Tue, 25 Nov 2014 15:49:46 -0600
Subject: [OmniOS-discuss] Installer dependency on a Floppy
In-Reply-To: <CALeZrrQB0UC25tZFvWsyBggE0AcubF3QvLtFZU2KmD1_P-WsZg@mail.gmail.com>
References: <CALeZrrQzSR0hqb1=xw=2jPERRgu1bBejbjo19B7NFWyP1yvPKQ@mail.gmail.com>
	<20141125213403.GE29549@bender.unx.csupomona.edu>
	<CALeZrrQB0UC25tZFvWsyBggE0AcubF3QvLtFZU2KmD1_P-WsZg@mail.gmail.com>
Message-ID: <CALeZrrT6ceek-0g86t5S5nSqtnt2N8EnED29_V7mB6qE5RDvCA@mail.gmail.com>

Beginning to suspect this is something else.  OpenIndiana's ISO panics
also.   It works on VMware without a floppy.



On Tue, Nov 25, 2014 at 3:41 PM, Schweiss, Chip <chip at innovates.com> wrote:

> On Tue, Nov 25, 2014 at 3:34 PM, Paul B. Henson <henson at acm.org> wrote:
>
>> On Tue, Nov 25, 2014 at 03:04:35PM -0600, Schweiss, Chip wrote:
>>
>> > The floppy controller is completely gone from these systems and causing
>> the
>> > installer to panic when loading.
>> >
>> > Anyone aware of a workaround?
>>
>> Check the BIOS and see if it says anything about a floppy. For vmware
>> guests, even if you don't have a virtual floppy configured, the BIOS
>> still shows that a floppy is installed. Most people recommend adding a
>> virtual floppy to work around the issue, but you can also go into the
>> virtual BIOS and change the floppy to not installed, which also allows
>> the installer to work.
>>
>> So if the BIOS mentions floppies, make sure they're configured as not
>> installed...
>>
>
> That was my first place to look.    Floppy is truly non-existent.
>
> I just tried plugging a USB floppy in.  That didn't help either.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141125/a55c9ad1/attachment.html>

From chip at innovates.com  Tue Nov 25 22:12:29 2014
From: chip at innovates.com (Schweiss, Chip)
Date: Tue, 25 Nov 2014 16:12:29 -0600
Subject: [OmniOS-discuss] Installer dependency on a Floppy
In-Reply-To: <CALeZrrT6ceek-0g86t5S5nSqtnt2N8EnED29_V7mB6qE5RDvCA@mail.gmail.com>
References: <CALeZrrQzSR0hqb1=xw=2jPERRgu1bBejbjo19B7NFWyP1yvPKQ@mail.gmail.com>
	<20141125213403.GE29549@bender.unx.csupomona.edu>
	<CALeZrrQB0UC25tZFvWsyBggE0AcubF3QvLtFZU2KmD1_P-WsZg@mail.gmail.com>
	<CALeZrrT6ceek-0g86t5S5nSqtnt2N8EnED29_V7mB6qE5RDvCA@mail.gmail.com>
Message-ID: <CALeZrrQDF7bDbQJEMtaGn3+Oyru=PdGZ29ukDLXwQ2hQay=wRA@mail.gmail.com>

X2APCI is the culprit.  Needs to be disabled in the BIOs.  Worked without
the floppy at all.

Hopefully this will be useful to someone else.

-Chip

On Tue, Nov 25, 2014 at 3:49 PM, Schweiss, Chip <chip at innovates.com> wrote:

> Beginning to suspect this is something else.  OpenIndiana's ISO panics
> also.   It works on VMware without a floppy.
>
>
>
> On Tue, Nov 25, 2014 at 3:41 PM, Schweiss, Chip <chip at innovates.com>
> wrote:
>
>> On Tue, Nov 25, 2014 at 3:34 PM, Paul B. Henson <henson at acm.org> wrote:
>>
>>> On Tue, Nov 25, 2014 at 03:04:35PM -0600, Schweiss, Chip wrote:
>>>
>>> > The floppy controller is completely gone from these systems and
>>> causing the
>>> > installer to panic when loading.
>>> >
>>> > Anyone aware of a workaround?
>>>
>>> Check the BIOS and see if it says anything about a floppy. For vmware
>>> guests, even if you don't have a virtual floppy configured, the BIOS
>>> still shows that a floppy is installed. Most people recommend adding a
>>> virtual floppy to work around the issue, but you can also go into the
>>> virtual BIOS and change the floppy to not installed, which also allows
>>> the installer to work.
>>>
>>> So if the BIOS mentions floppies, make sure they're configured as not
>>> installed...
>>>
>>
>> That was my first place to look.    Floppy is truly non-existent.
>>
>> I just tried plugging a USB floppy in.  That didn't help either.
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141125/e586789b/attachment-0001.html>

From danmcd at omniti.com  Tue Nov 25 22:20:29 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Tue, 25 Nov 2014 17:20:29 -0500
Subject: [OmniOS-discuss] Installer dependency on a Floppy
In-Reply-To: <CALeZrrQDF7bDbQJEMtaGn3+Oyru=PdGZ29ukDLXwQ2hQay=wRA@mail.gmail.com>
References: <CALeZrrQzSR0hqb1=xw=2jPERRgu1bBejbjo19B7NFWyP1yvPKQ@mail.gmail.com>
	<20141125213403.GE29549@bender.unx.csupomona.edu>
	<CALeZrrQB0UC25tZFvWsyBggE0AcubF3QvLtFZU2KmD1_P-WsZg@mail.gmail.com>
	<CALeZrrT6ceek-0g86t5S5nSqtnt2N8EnED29_V7mB6qE5RDvCA@mail.gmail.com>
	<CALeZrrQDF7bDbQJEMtaGn3+Oyru=PdGZ29ukDLXwQ2hQay=wRA@mail.gmail.com>
Message-ID: <A1EB9FD1-6DE8-4CAF-9529-1059E2934299@omniti.com>


> On Nov 25, 2014, at 5:12 PM, Schweiss, Chip <chip at innovates.com> wrote:
> 
> X2APCI is the culprit.  Needs to be disabled in the BIOs.  Worked without the floppy at all.
> 
> Hopefully this will be useful to someone else.

Is it specifically x2apic?

One thing every illumos user SHOULD/MUST disable in the BIOS is deep-C-states.  Maybe that's it, or perhaps the IOMMU.  There's a blog post by Jeff Sipek that's relevant:

	http://blahg.josefsipek.net/?p=488

Dan


From brogyi at gmail.com  Wed Nov 26 00:03:23 2014
From: brogyi at gmail.com (=?ISO-8859-1?Q?Brogy=E1nyi_J=F3zsef?=)
Date: Wed, 26 Nov 2014 01:03:23 +0100
Subject: [OmniOS-discuss] Parity error on path mpt_sas2
In-Reply-To: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAAD0@AIRA-SRV.aira.local>
References: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAAD0@AIRA-SRV.aira.local>
Message-ID: <547518CB.5040305@gmail.com>

Filip

Some days ago I had same issue. My problem was very easy. Of course I 
already known it.
I had another problem too. The smartctl is show me the next error what 
is growing up and up.
199 UDMA_CRC_Error_Count
Anyway the solution is very easy. I have two PCIe slot and I tried the 
another slot and that's it.
Please write it back if it works for you. May be the FW downgrade is 
good idea.
ps
1 PCIe 1x16 (x16, PCIe 3.0)
2 PCIe  1x16 (x4, PCIe 2.0)
HBA FW is 20.

BR
Brogyi

2014.11.25. 16:00 keltez?ssel, Filip Marvan ?rta:
>
> Hi,
>
> I have OmniOS 12 with three RAIDZ pools.
>
> In /var/adm/messages a have these messages about every few seconds:
>
> Nov 25 15:30:52 servername scsi: [ID 107833 kern.notice] 
> /pci at 0,0/pci8086,e08 at 3/pci1000,3020 at 0 (mpt_sas0):
>
> Nov 25 15:30:52 servername Aborted_command!
>
> Nov 25 15:30:52 servername scsi: [ID 243001 kern.warning] WARNING: 
> /scsi_vhci (scsi_vhci0):
>
> Nov 25 15:30:52 servername /scsi_vhci/disk at g50014ee209cf0116 (sd13): 
> Parity Error on path mpt_sas2/disk at w50014ee209cf0116,0
>
> Nov 25 15:30:52 servername scsi: [ID 107833 kern.notice] 
> /pci at 0,0/pci8086,e08 at 3/pci1000,3020 at 0 (mpt_sas0):
>
> Nov 25 15:30:52 servername Aborted_command!
>
> Nov 25 15:30:53 servername scsi: [ID 107833 kern.notice] 
> /pci at 0,0/pci8086,e08 at 3/pci1000,3020 at 0 (mpt_sas0):
>
> Nov 25 15:30:53 servername Aborted_command!
>
> In this log messages there is error on sd13, but there are similar 
> parity errors on four other disks too. All disks with parity errors 
> are in the same RAIDZ pool.
>
> All disks seems to be OK in zpool status and there are no errors on 
> disks reported by smartctl. There are no problems reported by fmdump
>
> All three pools using the same LSI 9211-8i HBA with firmware version 20.
>
> Sometimes that pool have very slow responses even on "zpool status" 
> command, sometimes works normaly.
>
> Do you have any idea where could be a problem? What else I can check?
>
> Thank you very much for any help,
>
> Filip
>
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141126/2205a330/attachment.html>

From bfriesen at simple.dallas.tx.us  Wed Nov 26 15:33:14 2014
From: bfriesen at simple.dallas.tx.us (Bob Friesenhahn)
Date: Wed, 26 Nov 2014 09:33:14 -0600 (CST)
Subject: [OmniOS-discuss] Installer dependency on a Floppy
In-Reply-To: <A1EB9FD1-6DE8-4CAF-9529-1059E2934299@omniti.com>
References: <CALeZrrQzSR0hqb1=xw=2jPERRgu1bBejbjo19B7NFWyP1yvPKQ@mail.gmail.com>
	<20141125213403.GE29549@bender.unx.csupomona.edu>
	<CALeZrrQB0UC25tZFvWsyBggE0AcubF3QvLtFZU2KmD1_P-WsZg@mail.gmail.com>
	<CALeZrrT6ceek-0g86t5S5nSqtnt2N8EnED29_V7mB6qE5RDvCA@mail.gmail.com>
	<CALeZrrQDF7bDbQJEMtaGn3+Oyru=PdGZ29ukDLXwQ2hQay=wRA@mail.gmail.com>
	<A1EB9FD1-6DE8-4CAF-9529-1059E2934299@omniti.com>
Message-ID: <alpine.GSO.2.01.1411260931300.18732@freddy.simplesystems.org>

On Tue, 25 Nov 2014, Dan McDonald wrote:
>
> One thing every illumos user SHOULD/MUST disable in the BIOS is 
> deep-C-states.  Maybe that's it, or perhaps the IOMMU.  There's a 
> blog post by Jeff Sipek that's relevant:

Is there documentation available for the appropriate way to disable 
"deep-C-states" on modern SuperMicro hardware?

Bob
-- 
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

From filip.marvan at aira.cz  Thu Nov 27 15:03:19 2014
From: filip.marvan at aira.cz (Filip Marvan)
Date: Thu, 27 Nov 2014 16:03:19 +0100
Subject: [OmniOS-discuss] Parity error on path mpt_sas2
In-Reply-To: <CAOqBcP_12FaH77o+3nrGVm7KU3x1XCk3B-9uenKRRL63t5wdyg@mail.gmail.com>
References: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAAD0@AIRA-SRV.aira.local>
	<1439760B-8D8B-40F9-8A62-4CFE4A9483C1@omniti.com>
	<3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAADA@AIRA-SRV.aira.local>
	<CAOqBcP_12FaH77o+3nrGVm7KU3x1XCk3B-9uenKRRL63t5wdyg@mail.gmail.com>
Message-ID: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DACD1@AIRA-SRV.aira.local>

Thank you for your help Aaron! I downgraded firmware to P18 and there are no 
more errors today.

It seems, that there is something bad with P20 firmware!



You saved me a lot of time :)



Filip





From: Aaron Curry [mailto:asc1111 at gmail.com]
Sent: Tuesday, November 25, 2014 5:46 PM
To: Filip Marvan
Cc: Dan McDonald; omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] Parity error on path mpt_sas2



I had this exact same problem recently when setting up a new home server... 
same controller, same firmware (P20). The errors were on all disks attached to 
the controller, but only on high read activity. Writes did not generate the 
errors. I downgraded the firmware to P18 (which is what we are using at work) 
and the errors went away. Has anyone had success with this controller and the 
P20 firmware?



Aaron



On Tue, Nov 25, 2014 at 8:30 AM, Filip Marvan <filip.marvan at aira.cz 
<mailto:filip.marvan at aira.cz> > wrote:

Hi Dan,

thanks for reply.
Yes, errors are on all 5 disks in the same RAIDZ pool. No problem on other
disks in different pool but on the same SAS cable.
So maybe I only had bad luck with that disks.

Filip


_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com <mailto:OmniOS-discuss at lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141127/771fbb84/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2700 bytes
Desc: not available
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141127/771fbb84/attachment-0001.bin>

From rt at steait.net  Thu Nov 27 14:35:16 2014
From: rt at steait.net (Rune Tipsmark)
Date: Thu, 27 Nov 2014 14:35:16 +0000
Subject: [OmniOS-discuss] Active-Active vSphere
Message-ID: <d04b17a9c37a4de3841496979104f73a@EX1301.steait.net>

Hi guys,
Does anyone know if Active/Active and Round Robin is supported from vSphere towards OmniOS ZFS on Fiber Channel?
Br
Rune
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141127/b0e5e26a/attachment.html>

From johan.kragsterman at capvert.se  Thu Nov 27 17:33:46 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Thu, 27 Nov 2014 18:33:46 +0100
Subject: [OmniOS-discuss] Ang:  Active-Active vSphere
In-Reply-To: <d04b17a9c37a4de3841496979104f73a@EX1301.steait.net>
References: <d04b17a9c37a4de3841496979104f73a@EX1301.steait.net>
Message-ID: <OFD22DAEDA.4FE34FE1-ONC1257D9D.00601EBF-C1257D9D.006079B4@inse.com>



-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev: -----
Till: "omnios-discuss at lists.omniti.com" <omnios-discuss at lists.omniti.com>
Fr?n: Rune Tipsmark 
S?nt av: "OmniOS-discuss" 
Datum: 2014-11-27 16:06
?rende: [OmniOS-discuss] Active-Active vSphere

Hi guys,

Does anyone know if Active/Active and Round Robin is supported from vSphere towards OmniOS ZFS on Fiber Channel?

Br




You got to explain a little bit more. What is it you want to be active/active? Pool access? LUN access? Controller access? In my world active/active means you got two active controllers(SAN Controllers/targets/servers), that both serves the SAN actively on the same time with LU's. Though they don't serve the same LU's at the same time, though during a downtime on one of them, the other can serve all the LU's from both controllers.
 Round Robin is default fibre channel setting when using multipath. Check mpathadm.


Johan





Rune

_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss



From skiselkov.ml at gmail.com  Thu Nov 27 20:47:22 2014
From: skiselkov.ml at gmail.com (Saso Kiselkov)
Date: Thu, 27 Nov 2014 21:47:22 +0100
Subject: [OmniOS-discuss] Active-Active vSphere
In-Reply-To: <d04b17a9c37a4de3841496979104f73a@EX1301.steait.net>
References: <d04b17a9c37a4de3841496979104f73a@EX1301.steait.net>
Message-ID: <54778DDA.4080807@gmail.com>

On 11/27/14 3:35 PM, Rune Tipsmark wrote:
> Hi guys,
> 
> Does anyone know if Active/Active and Round Robin is supported from
> vSphere towards OmniOS ZFS on Fiber Channel?

The short answer is: yes, but you wouldn't want to employ Round-Robin on
it. A ZFS pool can only be imported on a single node, never
simultaneously on two or more. However, using ALUA it's possible to make
the LUs on it visible from two OmniOS nodes - this is NOT round robin,
though. The initiator will see two paths to the LU, but only one should
be active at any one time (the one that has the pool holding the LU
imported). Access to the LU over the secondary target will be possible,
but slow. Upon failover, the secondary would grab the pool and become
the preferred ALUA path, so it all works out OK. Google "ALUA" for more
on the theory behind it.

It is a complicated to set up, though, so be aware of that.

Cheers,
-- 
Saso

From mailinglists at qutic.com  Thu Nov 27 21:51:26 2014
From: mailinglists at qutic.com (qutic development)
Date: Thu, 27 Nov 2014 22:51:26 +0100
Subject: [OmniOS-discuss] announcement znapzend a new zfs backup tool
In-Reply-To: <alpine.DEB.2.02.1407291748500.6752@froburg.oetiker.ch>
References: <alpine.DEB.2.02.1407291748500.6752@froburg.oetiker.ch>
Message-ID: <A9DD60A2-CE24-4E2C-91C1-DD4FC52C8B0C@qutic.com>

Hi Tobi,

is there a way to specify ssh on another port than 22?

Thanks
Stefan

On 29.07.2014, at 17:50, Tobias Oetiker <tobi at oetiker.ch> wrote:

> Just out:
> 
> ZnapZend a Multilevel Backuptool for ZFS
> 
> It is on Github. Check out
> 
> http://www.znapzend.org
> 
> cheers
> tobi


From tobi at oetiker.ch  Thu Nov 27 21:59:31 2014
From: tobi at oetiker.ch (Tobi Oetiker)
Date: Thu, 27 Nov 2014 22:59:31 +0100
Subject: [OmniOS-discuss] announcement znapzend a new zfs backup tool
In-Reply-To: <A9DD60A2-CE24-4E2C-91C1-DD4FC52C8B0C@qutic.com>
References: <alpine.DEB.2.02.1407291748500.6752@froburg.oetiker.ch>
	<A9DD60A2-CE24-4E2C-91C1-DD4FC52C8B0C@qutic.com>
Message-ID: <D25FC962-8296-48FA-8A17-25611A4864BE@oetiker.ch>

hi stefan

> On 27.11.2014, at 22:51, qutic development <mailinglists at qutic.com> wrote:
> 
> Hi Tobi,
> 
> is there a way to specify ssh on another port than 22?
> 


I would just setup your .ssh/config file appropriately

cheers
tobi


> Thanks
> Stefan
> 
>> On 29.07.2014, at 17:50, Tobias Oetiker <tobi at oetiker.ch> wrote:
>> 
>> Just out:
>> 
>> ZnapZend a Multilevel Backuptool for ZFS
>> 
>> It is on Github. Check out
>> 
>> http://www.znapzend.org
>> 
>> cheers
>> tobi
> 

From rt at steait.net  Thu Nov 27 22:40:57 2014
From: rt at steait.net (Rune Tipsmark)
Date: Thu, 27 Nov 2014 22:40:57 +0000
Subject: [OmniOS-discuss] Active-Active vSphere
In-Reply-To: <54778DDA.4080807@gmail.com>
References: <d04b17a9c37a4de3841496979104f73a@EX1301.steait.net>
	<54778DDA.4080807@gmail.com>
Message-ID: <ed4e38efa095444db0f20eb5b900f562@EX1301.steait.net>

so to simplify, say we have one esxi host that has two FC ports, one omnios/zfs server has two FC ports as well.
Should it run ALUA with Round Robin instead of the default ALUA with MRU (most recently used). RR has traffic on both paths (says Active I/O on both) and MRU only on one...


-----Original Message-----
From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Saso Kiselkov
Sent: Thursday, November 27, 2014 12:47 PM
To: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] Active-Active vSphere

On 11/27/14 3:35 PM, Rune Tipsmark wrote:
> Hi guys,
> 
> Does anyone know if Active/Active and Round Robin is supported from 
> vSphere towards OmniOS ZFS on Fiber Channel?

The short answer is: yes, but you wouldn't want to employ Round-Robin on it. A ZFS pool can only be imported on a single node, never simultaneously on two or more. However, using ALUA it's possible to make the LUs on it visible from two OmniOS nodes - this is NOT round robin, though. The initiator will see two paths to the LU, but only one should be active at any one time (the one that has the pool holding the LU imported). Access to the LU over the secondary target will be possible, but slow. Upon failover, the secondary would grab the pool and become the preferred ALUA path, so it all works out OK. Google "ALUA" for more on the theory behind it.

It is a complicated to set up, though, so be aware of that.

Cheers,
--
Saso
_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

From skiselkov.ml at gmail.com  Thu Nov 27 23:29:25 2014
From: skiselkov.ml at gmail.com (Saso Kiselkov)
Date: Fri, 28 Nov 2014 00:29:25 +0100
Subject: [OmniOS-discuss] Active-Active vSphere
In-Reply-To: <ed4e38efa095444db0f20eb5b900f562@EX1301.steait.net>
References: <d04b17a9c37a4de3841496979104f73a@EX1301.steait.net>
	<54778DDA.4080807@gmail.com>
	<ed4e38efa095444db0f20eb5b900f562@EX1301.steait.net>
Message-ID: <5477B3D5.5000704@gmail.com>

On 11/27/14 11:40 PM, Rune Tipsmark wrote:
> so to simplify, say we have one esxi host that has two FC ports, one omnios/zfs server has two FC ports as well.
> Should it run ALUA with Round Robin instead of the default ALUA with MRU (most recently used). RR has traffic on both paths (says Active I/O on both) and MRU only on one...

Ah, now this is a different story altogether, if it's just one storage
head node. In that case, Round-Robin will of course work exactly as
expected, be it over FC or iSCSI and any number of links. The important
thing is that the backing ZFS pool is only imported on one machine.

Cheers,
-- 
Saso

From gate03 at landcroft.co.uk  Fri Nov 28 09:17:03 2014
From: gate03 at landcroft.co.uk (Michael Mounteney)
Date: Fri, 28 Nov 2014 19:17:03 +1000
Subject: [OmniOS-discuss] common-factor key exchange
Message-ID: <20141128191703.00a9df99@punda-mlia>

After a recent upgrade to my Gentoo Linux installations, I can no
longer ssh into them from OmniOS:

-bash-4.3$ ssh 192.168.1.48
no common kex alg: client
'diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1', server
'curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1'

The client being OmniOS and the server being Gentoo Linux.  Now I could
downgrade what Gentoo offers, but it would be nicer to upgrade what
OmniOS offers.  Is there a way ?

Michael.

From gate03 at landcroft.co.uk  Fri Nov 28 09:18:21 2014
From: gate03 at landcroft.co.uk (Michael Mounteney)
Date: Fri, 28 Nov 2014 19:18:21 +1000
Subject: [OmniOS-discuss] Postgres on Stable
Message-ID: <20141128191821.43382469@punda-mlia>

Hello, went from LTS to stable to-day and was somewhat taken aback that
the highest Postgres version is 8.3 whereas on LTS, 9.3 is available.
Is there a way to get 9.3 on stable ?

Michael.

From richard at netbsd.org  Fri Nov 28 11:56:29 2014
From: richard at netbsd.org (Richard PALO)
Date: Fri, 28 Nov 2014 12:56:29 +0100
Subject: [OmniOS-discuss] common-factor key exchange
In-Reply-To: <20141128191703.00a9df99@punda-mlia>
References: <20141128191703.00a9df99@punda-mlia>
Message-ID: <547862ED.8050701@netbsd.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Le 28/11/14 10:17, Michael Mounteney a ?crit :
| After a recent upgrade to my Gentoo Linux installations, I can no
| longer ssh into them from OmniOS:
|
| -bash-4.3$ ssh 192.168.1.48 no common kex alg: client
| 'diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1',
| server
|
'curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1'
|
|  The client being OmniOS and the server being Gentoo Linux.  Now I
| could downgrade what Gentoo offers, but it would be nicer to
| upgrade what OmniOS offers.  Is there a way ?
|
| Michael.
|
you may need to verify which ssh you have installed.
| # pkg list -a |grep ssh network/openssh (omnios)
| 6.6.1-0.151013             --- network/openssh-server (omnios)
| 6.6.1-0.151013             --- network/ssh
| 0.5.11-0.151013            i-- network/ssh/ssh-key
| 0.5.11-0.151013            i-- service/network/ssh
| 0.5.11-0.151013            i--

if you have openssh installed (omnios or, for example pkgsrc) then you
should be able to query the algorithms supported.
With pkgsrc openssh:
| # ssh -Q kex diffie-hellman-group1-sha1
| diffie-hellman-group14-sha1 diffie-hellman-group-exchange-sha1
| diffie-hellman-group-exchange-sha256 ecdh-sha2-nistp256
| ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group1-sha1
| curve25519-sha256 at libssh.org

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJUeGLqAAoJECAB22fHtp27hjoH/inE8Abemhr4zjy+MqE6HMu2
sxtGKHDN4mR6DmQpAx3EOyfOOO6It1NnyQxuze99qHYAbm3rVNER9M+ATwkTN4Nd
/IH7m2XK7OjNkLvy8Bz0OBqwJXQ4aiwfbGtHnGHLv4m8KF463eV+wwJDEkh632VX
siZiuLmBQZ1tngv8La4QvvXVpCzbXo3jr/LjyYBGOWpAJL6kI+s2aH0uVhUz2qjx
i1a2FwkqYLhABW06dwrnFFwY5/EBy4c8iXCgt/rgzUjFrU65Dr4kxqyA0WOMRsbO
9OhUzRNx7Jn7mvhoNpWbl2t8Gj6zpBftS09fqgkJOPa6aHPTysuJjaSPfa8vG54=
=krP7
-----END PGP SIGNATURE-----


From rt at steait.net  Fri Nov 28 14:34:56 2014
From: rt at steait.net (Rune Tipsmark)
Date: Fri, 28 Nov 2014 14:34:56 +0000
Subject: [OmniOS-discuss] Active-Active vSphere
In-Reply-To: <5477B3D5.5000704@gmail.com>
References: <d04b17a9c37a4de3841496979104f73a@EX1301.steait.net>
	<54778DDA.4080807@gmail.com>
	<ed4e38efa095444db0f20eb5b900f562@EX1301.steait.net>,
	<5477B3D5.5000704@gmail.com>
Message-ID: <E0E35376-8C80-4F04-B015-C3F077E24145@steait.net>

Okay, i noticed alua support is not enabled per default on omnios. How do i enable that?



> On Nov 27, 2014, at 11:29 PM, Saso Kiselkov <skiselkov.ml at gmail.com> wrote:
> 
>> On 11/27/14 11:40 PM, Rune Tipsmark wrote:
>> so to simplify, say we have one esxi host that has two FC ports, one omnios/zfs server has two FC ports as well.
>> Should it run ALUA with Round Robin instead of the default ALUA with MRU (most recently used). RR has traffic on both paths (says Active I/O on both) and MRU only on one...
> 
> Ah, now this is a different story altogether, if it's just one storage
> head node. In that case, Round-Robin will of course work exactly as
> expected, be it over FC or iSCSI and any number of links. The important
> thing is that the backing ZFS pool is only imported on one machine.
> 
> Cheers,
> -- 
> Saso

From skiselkov.ml at gmail.com  Fri Nov 28 14:53:35 2014
From: skiselkov.ml at gmail.com (Saso Kiselkov)
Date: Fri, 28 Nov 2014 15:53:35 +0100
Subject: [OmniOS-discuss] Active-Active vSphere
In-Reply-To: <E0E35376-8C80-4F04-B015-C3F077E24145@steait.net>
References: <d04b17a9c37a4de3841496979104f73a@EX1301.steait.net>
	<54778DDA.4080807@gmail.com>
	<ed4e38efa095444db0f20eb5b900f562@EX1301.steait.net>,
	<5477B3D5.5000704@gmail.com>
	<E0E35376-8C80-4F04-B015-C3F077E24145@steait.net>
Message-ID: <54788C6F.4000805@gmail.com>

On 11/28/14 3:34 PM, Rune Tipsmark wrote:
> Okay, i noticed alua support is not enabled per default on omnios. How do i enable that?

The process is complicated, not very well documented and it isn't meant
for your scenario anyway. If you only have one storage node, then you
don't need ALUA, even if you have multiple connections to the node. So
no need to worry about it.

Cheers,
-- 
Saso


From richard at netbsd.org  Fri Nov 28 15:19:36 2014
From: richard at netbsd.org (Richard PALO)
Date: Fri, 28 Nov 2014 16:19:36 +0100
Subject: [OmniOS-discuss] common-factor key exchange
In-Reply-To: <20141128191703.00a9df99@punda-mlia>
References: <20141128191703.00a9df99@punda-mlia>
Message-ID: <54789288.20302@netbsd.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Le 28/11/14 10:17, Michael Mounteney a ?crit :
> After a recent upgrade to my Gentoo Linux installations, I can no
> longer ssh into them from OmniOS:
> 
> -bash-4.3$ ssh 192.168.1.48
> no common kex alg: client
> 'diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1', server
> 'curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1'
> 
> The client being OmniOS and the server being Gentoo Linux.  Now I could
> downgrade what Gentoo offers, but it would be nicer to upgrade what
> OmniOS offers.  Is there a way ?
> 
> Michael.
> 
Also, this seems to be a known issue for sun ssh.
https://www.illumos.org/issues/5283
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJUeJKIAAoJECAB22fHtp27hPUH/i0g5Aamp/OHxxthwEOAN5HH
XAoXT+AbPYSsXhVx8YTJb6yXmFuvo2aJZEsZi+x115SWcqwQdXwWYhvbv9fEEuGi
RJI0Hpl9SLbwuDv1dh7/uJCuooMED2U8OmojimYuJfSnI1dlzn02z1vhJCmOuV+0
eXDbqIVxCF4w9G1Rddm+xdeSRptbx+OkhrouGydnFI9lRRXQ01WIcP5t1JA8XkS/
bOaCDdWXI9EViuvcvHkvkH3ig8YUzwfEFKicObq79oQnnlyIImJFNBXM4s5tEGFH
2y2lYG4k1L8SyMxjirL13vxLeXCv01uyT9KiPmxSI8rkvKnwCyaaMryr5KKZM1g=
=ktnX
-----END PGP SIGNATURE-----


From jesus at omniti.com  Fri Nov 28 17:02:16 2014
From: jesus at omniti.com (Theo Schlossnagle)
Date: Fri, 28 Nov 2014 18:02:16 +0100
Subject: [OmniOS-discuss] Postgres on Stable
In-Reply-To: <20141128191821.43382469@punda-mlia>
References: <20141128191821.43382469@punda-mlia>
Message-ID: <CACLsApuL0vfHhn7FAQw8OU+uPd=j=qEGRUVWAtCtA1VXoo+TVA@mail.gmail.com>

The OmniTI DBA team just builds things out of the omniti-ms build repo on
demand.  The build scripts are there for later releases, so you can build
your own packages.  I'm guessing they've just never had to deploy anything
newer on stuff other than LTS.  I know all our database servers run LTS,
I'd bet the ones they manage do as well.

https://github.com/omniti-labs/omnios-build/tree/omniti-ms

On Fri, Nov 28, 2014 at 10:18 AM, Michael Mounteney <gate03 at landcroft.co.uk>
wrote:

> Hello, went from LTS to stable to-day and was somewhat taken aback that
> the highest Postgres version is 8.3 whereas on LTS, 9.3 is available.
> Is there a way to get 9.3 on stable ?
>
> Michael.
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>



-- 

Theo Schlossnagle

http://omniti.com/is/theo-schlossnagle
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141128/83641dc1/attachment.html>

From nagela at informatik.uni-tuebingen.de  Fri Nov 28 18:59:38 2014
From: nagela at informatik.uni-tuebingen.de (Andreas Nagel)
Date: Fri, 28 Nov 2014 19:59:38 +0100
Subject: [OmniOS-discuss] ufsdump/ufsrestore: Segmentation Fault ?
Message-ID: <5478C61A.7080005@informatik.uni-tuebingen.de>

Hi,

under Solaris 10 ufsrestore do as requested, but under the actual Omnios 
r151012 it fails with the message "Segmentation Fault". Have i make a 
mistake or what is the reason for this behavior? :

root at kayak:/tmp# zfs create -V 100m rpool/test
root at kayak:/tmp# newfs /dev/zvol/rdsk/rpool/test
newfs: construct a new file system /dev/zvol/rdsk/rpool/test: (y/n)? y
Warning: 4130 sector(s) in last cylinder unallocated
/dev/zvol/rdsk/rpool/test:      204766 sectors in 34 cylinders of 48 
tracks, 128 sectors
         100.0MB in 3 cyl groups (14 c/g, 42.00MB/g, 20160 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
  32, 86176, 172320,

root at kayak:/tmp# mkdir /mnt/test
root at kayak:/tmp# mount -F ufs /dev/zvol/dsk/rpool/test /mnt/test
root at kayak:/tmp# ufsdump 0f test.dump /mnt/test
   DUMP: Date of this level 0 dump: November 28, 2014 07:54:46 PM CET
   DUMP: Date of last level 0 dump: the epoch
   DUMP: Dumping /dev/zvol/rdsk/rpool/test (kayak:/mnt/test) to test.dump.
   DUMP: Mapping (Pass I) [regular files]
   DUMP: Mapping (Pass II) [directories]
   DUMP: Writing 32 Kilobyte records
   DUMP: Estimated 348 blocks (174KB).
   DUMP: Dumping (Pass III) [directories]
   DUMP: Dumping (Pass IV) [regular files]
   DUMP: 318 blocks (159KB) on 1 volume at 39750 KB/sec
   DUMP: DUMP IS DONE
root at kayak:/tmp# ufsrestore ivf ./test.dump
Verify volume and initialize maps
Media block size is 126
Dump   date: November 28, 2014 07:54:46 PM CET
Dumped from: the epoch
Level 0 dump of /mnt/test on kayak:/dev/zvol/dsk/rpool/test
Label: none
Extract directories from tape
Initialize symbol table.
Segmentation Fault
root at kayak:/tmp#

From robert at omniti.com  Fri Nov 28 19:41:02 2014
From: robert at omniti.com (Robert Treat)
Date: Fri, 28 Nov 2014 14:41:02 -0500
Subject: [OmniOS-discuss] Postgres on Stable
In-Reply-To: <CACLsApuL0vfHhn7FAQw8OU+uPd=j=qEGRUVWAtCtA1VXoo+TVA@mail.gmail.com>
References: <20141128191821.43382469@punda-mlia>
	<CACLsApuL0vfHhn7FAQw8OU+uPd=j=qEGRUVWAtCtA1VXoo+TVA@mail.gmail.com>
Message-ID: <CA+FJ0zsFukbMSdnTRxqdO1fU_zR69_=bH4UDJkh3mdB+pmtABQ@mail.gmail.com>

FWIW, we do have newer postgres running on newer omnios, so if you build
your own there shouldn't be any issues. I can poke folks about what's
published where, but it might be a few days before getting a response.

Robert Treat

On Friday, November 28, 2014, Theo Schlossnagle <jesus at omniti.com> wrote:

> The OmniTI DBA team just builds things out of the omniti-ms build repo on
> demand.  The build scripts are there for later releases, so you can build
> your own packages.  I'm guessing they've just never had to deploy anything
> newer on stuff other than LTS.  I know all our database servers run LTS,
> I'd bet the ones they manage do as well.
>
> https://github.com/omniti-labs/omnios-build/tree/omniti-ms
>
> On Fri, Nov 28, 2014 at 10:18 AM, Michael Mounteney <
> gate03 at landcroft.co.uk
> <javascript:_e(%7B%7D,'cvml','gate03 at landcroft.co.uk');>> wrote:
>
>> Hello, went from LTS to stable to-day and was somewhat taken aback that
>> the highest Postgres version is 8.3 whereas on LTS, 9.3 is available.
>> Is there a way to get 9.3 on stable ?
>>
>> Michael.
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> <javascript:_e(%7B%7D,'cvml','OmniOS-discuss at lists.omniti.com');>
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>
>
>
>
> --
>
> Theo Schlossnagle
>
> http://omniti.com/is/theo-schlossnagle
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141128/6af7bb84/attachment-0001.html>

From zmalone at omniti.com  Fri Nov 28 20:06:03 2014
From: zmalone at omniti.com (Zach Malone)
Date: Fri, 28 Nov 2014 15:06:03 -0500
Subject: [OmniOS-discuss] Postgres on Stable
In-Reply-To: <CA+FJ0zsFukbMSdnTRxqdO1fU_zR69_=bH4UDJkh3mdB+pmtABQ@mail.gmail.com>
References: <20141128191821.43382469@punda-mlia>
	<CACLsApuL0vfHhn7FAQw8OU+uPd=j=qEGRUVWAtCtA1VXoo+TVA@mail.gmail.com>
	<CA+FJ0zsFukbMSdnTRxqdO1fU_zR69_=bH4UDJkh3mdB+pmtABQ@mail.gmail.com>
Message-ID: <CALLZ=vXs6XDheAGuE2aDJigNaHti4bVe7npAc2qiiqK45zh0sQ@mail.gmail.com>

We have pretty much abandoned 9.3, and are using 9.2 in production,
with plans to evaluate 9.4 when it's released.

That said, the 9.2 builds, despite being built on 151006, should work
on 151010 and 151012, we're running them on a 010 box now, and expect
to update it to 012 soon.

If you want to build your own repo, so you can manage your own
versions and patches, you just need to clone
https://github.com/omniti-labs/omnios-build , use the omniti-ms
branch, and update
https://github.com/omniti-labs/omnios-build/blob/omniti-ms/lib/site.sh
to publish to either your own http pkg repo, or to a local file based
repo.

After that's done, you can run build.sh inside the postgres
directories as an unprivileged user (for example,
https://github.com/omniti-labs/omnios-build/tree/omniti-ms/build/postgres93
) and the build scripts will point out if you need to install gcc,
package build dependencies, or anything else.




On Fri, Nov 28, 2014 at 2:41 PM, Robert Treat <robert at omniti.com> wrote:
> FWIW, we do have newer postgres running on newer omnios, so if you build
> your own there shouldn't be any issues. I can poke folks about what's
> published where, but it might be a few days before getting a response.
>
> Robert Treat
>
> On Friday, November 28, 2014, Theo Schlossnagle <jesus at omniti.com> wrote:
>>
>> The OmniTI DBA team just builds things out of the omniti-ms build repo on
>> demand.  The build scripts are there for later releases, so you can build
>> your own packages.  I'm guessing they've just never had to deploy anything
>> newer on stuff other than LTS.  I know all our database servers run LTS, I'd
>> bet the ones they manage do as well.
>>
>> https://github.com/omniti-labs/omnios-build/tree/omniti-ms
>>
>> On Fri, Nov 28, 2014 at 10:18 AM, Michael Mounteney
>> <gate03 at landcroft.co.uk> wrote:
>>>
>>> Hello, went from LTS to stable to-day and was somewhat taken aback that
>>> the highest Postgres version is 8.3 whereas on LTS, 9.3 is available.
>>> Is there a way to get 9.3 on stable ?
>>>
>>> Michael.
>>> _______________________________________________
>>> OmniOS-discuss mailing list
>>> OmniOS-discuss at lists.omniti.com
>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>
>>
>>
>>
>> --
>>
>> Theo Schlossnagle
>>
>> http://omniti.com/is/theo-schlossnagle
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>

From zmalone at omniti.com  Fri Nov 28 20:08:16 2014
From: zmalone at omniti.com (Zach Malone)
Date: Fri, 28 Nov 2014 15:08:16 -0500
Subject: [OmniOS-discuss] Postgres on Stable
In-Reply-To: <CALLZ=vXs6XDheAGuE2aDJigNaHti4bVe7npAc2qiiqK45zh0sQ@mail.gmail.com>
References: <20141128191821.43382469@punda-mlia>
	<CACLsApuL0vfHhn7FAQw8OU+uPd=j=qEGRUVWAtCtA1VXoo+TVA@mail.gmail.com>
	<CA+FJ0zsFukbMSdnTRxqdO1fU_zR69_=bH4UDJkh3mdB+pmtABQ@mail.gmail.com>
	<CALLZ=vXs6XDheAGuE2aDJigNaHti4bVe7npAc2qiiqK45zh0sQ@mail.gmail.com>
Message-ID: <CALLZ=vXQ=4qn=4SWAtAnBksf=dSVXkzFRrfWffe3HyoLB9b+1g@mail.gmail.com>

Oh yeah, http://omnios.omniti.com/wiki.php/CreatingRepos has
instructions on creating pkg repos, in case you missed that.

omniti-ms is intended as an unsupported internal repo, although it's
clearly useful to the outside world, so we don't bother keeping it
private.

What repos do you currently have configured, and if you specify
postgres-92 or 93 as a package to install, what prevents it from
installing (if you use the -v option to pkg install)?
--Zach

On Fri, Nov 28, 2014 at 3:06 PM, Zach Malone <zmalone at omniti.com> wrote:
> We have pretty much abandoned 9.3, and are using 9.2 in production,
> with plans to evaluate 9.4 when it's released.
>
> That said, the 9.2 builds, despite being built on 151006, should work
> on 151010 and 151012, we're running them on a 010 box now, and expect
> to update it to 012 soon.
>
> If you want to build your own repo, so you can manage your own
> versions and patches, you just need to clone
> https://github.com/omniti-labs/omnios-build , use the omniti-ms
> branch, and update
> https://github.com/omniti-labs/omnios-build/blob/omniti-ms/lib/site.sh
> to publish to either your own http pkg repo, or to a local file based
> repo.
>
> After that's done, you can run build.sh inside the postgres
> directories as an unprivileged user (for example,
> https://github.com/omniti-labs/omnios-build/tree/omniti-ms/build/postgres93
> ) and the build scripts will point out if you need to install gcc,
> package build dependencies, or anything else.
>
>
>
>
> On Fri, Nov 28, 2014 at 2:41 PM, Robert Treat <robert at omniti.com> wrote:
>> FWIW, we do have newer postgres running on newer omnios, so if you build
>> your own there shouldn't be any issues. I can poke folks about what's
>> published where, but it might be a few days before getting a response.
>>
>> Robert Treat
>>
>> On Friday, November 28, 2014, Theo Schlossnagle <jesus at omniti.com> wrote:
>>>
>>> The OmniTI DBA team just builds things out of the omniti-ms build repo on
>>> demand.  The build scripts are there for later releases, so you can build
>>> your own packages.  I'm guessing they've just never had to deploy anything
>>> newer on stuff other than LTS.  I know all our database servers run LTS, I'd
>>> bet the ones they manage do as well.
>>>
>>> https://github.com/omniti-labs/omnios-build/tree/omniti-ms
>>>
>>> On Fri, Nov 28, 2014 at 10:18 AM, Michael Mounteney
>>> <gate03 at landcroft.co.uk> wrote:
>>>>
>>>> Hello, went from LTS to stable to-day and was somewhat taken aback that
>>>> the highest Postgres version is 8.3 whereas on LTS, 9.3 is available.
>>>> Is there a way to get 9.3 on stable ?
>>>>
>>>> Michael.
>>>> _______________________________________________
>>>> OmniOS-discuss mailing list
>>>> OmniOS-discuss at lists.omniti.com
>>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Theo Schlossnagle
>>>
>>> http://omniti.com/is/theo-schlossnagle
>>
>>
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>

From gate03 at landcroft.co.uk  Fri Nov 28 21:18:35 2014
From: gate03 at landcroft.co.uk (Michael Mounteney)
Date: Sat, 29 Nov 2014 07:18:35 +1000
Subject: [OmniOS-discuss] Postgres on Stable
In-Reply-To: <CALLZ=vXQ=4qn=4SWAtAnBksf=dSVXkzFRrfWffe3HyoLB9b+1g@mail.gmail.com>
References: <20141128191821.43382469@punda-mlia>
	<CACLsApuL0vfHhn7FAQw8OU+uPd=j=qEGRUVWAtCtA1VXoo+TVA@mail.gmail.com>
	<CA+FJ0zsFukbMSdnTRxqdO1fU_zR69_=bH4UDJkh3mdB+pmtABQ@mail.gmail.com>
	<CALLZ=vXs6XDheAGuE2aDJigNaHti4bVe7npAc2qiiqK45zh0sQ@mail.gmail.com>
	<CALLZ=vXQ=4qn=4SWAtAnBksf=dSVXkzFRrfWffe3HyoLB9b+1g@mail.gmail.com>
Message-ID: <20141129071835.1daa0799@punda-mlia>

On Fri, 28 Nov 2014 15:08:16 -0500
Zach Malone <zmalone at omniti.com> wrote:

> We have pretty much abandoned 9.3, and are using 9.2 in production,
> with plans to evaluate 9.4 when it's released.

May I ask why ?  The problem is that I have databases built with 9.3,
and I don't know to downgrade or even if it's possible.

> If you want to build your own repo, so you can manage your own
> versions and patches, you just need to clone
> [...]
> Oh yeah, http://omnios.omniti.com/wiki.php/CreatingRepos has
> instructions on creating pkg repos, in case you missed that.

Yes;  I'd avoided getting involved in my own repository building so
far, but it's a bridge I'll have to cross.

> omniti-ms is intended as an unsupported internal repo, although it's
> clearly useful to the outside world, so we don't bother keeping it
> private.

It keeps users like me in lazy habits.

> What repos do you currently have configured, and if you specify
> postgres-92 or 93 as a package to install, what prevents it from
> installing (if you use the -v option to pkg install)?

root at world:/root# pkg -R /zone/diener/root install -v postgresql-934
Creating Plan \                      
pkg install: No solution was found to satisfy constraints

maintained incorporations:

  pkg://omnios/entire at 11,5.11-0.151012:20141027T191658Z
  pkg://omnios/incorporation/jeos/illumos-gate at 11,5.11-0.151012:20140913T032317Z
  pkg://omnios/consolidation/osnet/osnet-incorporation at 0.5.11,5.11-0.151012:20140913T033401Z
  pkg://omnios/incorporation/jeos/omnios-userland at 11,5.11-0.151012:20141027T191659Z

Plan Creation: dependency error(s) in proposed packages:

  No suitable version of required package pkg://ms.omniti.com/omniti/database/postgresql-934 at 9.3.4,5.11-0.151008:20140603T183939Z found:
    Reject:  pkg://ms.omniti.com/omniti/database/postgresql-934 at 9.3.4,5.11-0.151008:20140603T183939Z
    Reason:  A version for 'incorporate' dependency on pkg:/entire at 11,5.11-0.151008 cannot be found
  No suitable version of required package pkg://ms.omniti.com/omniti/database/postgresql-934 at 9.3.4,5.11-0.151010:20140804T183201Z found:
    Reject:  pkg://ms.omniti.com/omniti/database/postgresql-934 at 9.3.4,5.11-0.151010:20140804T183201Z
    Reason:  A version for 'incorporate' dependency on pkg:/entire at 11,5.11-0.151010 cannot be found
  No suitable version of required package pkg://ms.omniti.com/omniti/database/postgresql-934 at 9.3.4,5.11-0.151006:20140423T181948Z found:
    Reject:  pkg://ms.omniti.com/omniti/database/postgresql-934 at 9.3.4,5.11-0.151006:20140423T181948Z
    Reason:  A version for 'incorporate' dependency on pkg:/entire at 11,5.11-0.151006 cannot be found

Plan Creation: Errors in installed packages due to proposed changes:

  No suitable version of installed package pkg://omnios/entire at 11,5.11-0.151012:20141027T191658Z found
    Reject:  pkg://omnios/entire at 11,5.11-0.151012:20141027T191658Z
    Reason:  Excluded by proposed incorporation 'omniti/database/postgresql-934'
             This version is excluded by installed incorporation pkg://omnios/entire at 11,5.11-0.151012:20141027T191658Z

Michael.

From zmalone at omniti.com  Fri Nov 28 21:29:19 2014
From: zmalone at omniti.com (Zach Malone)
Date: Fri, 28 Nov 2014 16:29:19 -0500
Subject: [OmniOS-discuss] Postgres on Stable
In-Reply-To: <20141129071835.1daa0799@punda-mlia>
References: <20141128191821.43382469@punda-mlia>
	<CACLsApuL0vfHhn7FAQw8OU+uPd=j=qEGRUVWAtCtA1VXoo+TVA@mail.gmail.com>
	<CA+FJ0zsFukbMSdnTRxqdO1fU_zR69_=bH4UDJkh3mdB+pmtABQ@mail.gmail.com>
	<CALLZ=vXs6XDheAGuE2aDJigNaHti4bVe7npAc2qiiqK45zh0sQ@mail.gmail.com>
	<CALLZ=vXQ=4qn=4SWAtAnBksf=dSVXkzFRrfWffe3HyoLB9b+1g@mail.gmail.com>
	<20141129071835.1daa0799@punda-mlia>
Message-ID: <CALLZ=vU4MNQ2cZQgTHguzq4Ok-UywT7HTr5c6Pm05g4QyR8=7Q@mail.gmail.com>

On Fri, Nov 28, 2014 at 4:18 PM, Michael Mounteney
<gate03 at landcroft.co.uk> wrote:
> On Fri, 28 Nov 2014 15:08:16 -0500
> Zach Malone <zmalone at omniti.com> wrote:
>
>> We have pretty much abandoned 9.3, and are using 9.2 in production,
>> with plans to evaluate 9.4 when it's released.
>
> May I ask why ?  The problem is that I have databases built with 9.3,
> and I don't know to downgrade or even if it's possible.

9.3 has had a number of bugs that resulted in data corruption.  The
general feel from the PG community was that it was better to stay on
9.2.

>
>> If you want to build your own repo, so you can manage your own
>> versions and patches, you just need to clone
>> [...]
>> Oh yeah, http://omnios.omniti.com/wiki.php/CreatingRepos has
>> instructions on creating pkg repos, in case you missed that.
>
> Yes;  I'd avoided getting involved in my own repository building so
> far, but it's a bridge I'll have to cross.
>

I completely understand.

> root at world:/root# pkg -R /zone/diener/root install -v postgresql-934
> Creating Plan \
> pkg install: No solution was found to satisfy constraints
>
> maintained incorporations:
>
>   pkg://omnios/entire at 11,5.11-0.151012:20141027T191658Z
>   pkg://omnios/incorporation/jeos/illumos-gate at 11,5.11-0.151012:20140913T032317Z
>   pkg://omnios/consolidation/osnet/osnet-incorporation at 0.5.11,5.11-0.151012:20140913T033401Z
>   pkg://omnios/incorporation/jeos/omnios-userland at 11,5.11-0.151012:20141027T191659Z
>
> Plan Creation: dependency error(s) in proposed packages:
>
>   No suitable version of required package pkg://ms.omniti.com/omniti/database/postgresql-934 at 9.3.4,5.11-0.151008:20140603T183939Z found:
>     Reject:  pkg://ms.omniti.com/omniti/database/postgresql-934 at 9.3.4,5.11-0.151008:20140603T183939Z
>     Reason:  A version for 'incorporate' dependency on pkg:/entire at 11,5.11-0.151008 cannot be found
>   No suitable version of required package pkg://ms.omniti.com/omniti/database/postgresql-934 at 9.3.4,5.11-0.151010:20140804T183201Z found:
>     Reject:  pkg://ms.omniti.com/omniti/database/postgresql-934 at 9.3.4,5.11-0.151010:20140804T183201Z
>     Reason:  A version for 'incorporate' dependency on pkg:/entire at 11,5.11-0.151010 cannot be found
>   No suitable version of required package pkg://ms.omniti.com/omniti/database/postgresql-934 at 9.3.4,5.11-0.151006:20140423T181948Z found:
>     Reject:  pkg://ms.omniti.com/omniti/database/postgresql-934 at 9.3.4,5.11-0.151006:20140423T181948Z
>     Reason:  A version for 'incorporate' dependency on pkg:/entire at 11,5.11-0.151006 cannot be found
>
> Plan Creation: Errors in installed packages due to proposed changes:
>
>   No suitable version of installed package pkg://omnios/entire at 11,5.11-0.151012:20141027T191658Z found
>     Reject:  pkg://omnios/entire at 11,5.11-0.151012:20141027T191658Z
>     Reason:  Excluded by proposed incorporation 'omniti/database/postgresql-934'
>              This version is excluded by installed incorporation pkg://omnios/entire at 11,5.11-0.151012:20141027T191658Z

I hadn't realized we incorporated against the OS for postgres, I'll
spin a new postgresql-934 or 935 on a 012 machine when I get a chance.
It looks like 012 doesn't have a Vagrant box published yet either,
which will make this take a little longer.

8.3.4 could install because it predates us having a OS incorporation,
so it was willing to install onto any omnios system.
--Zach

From zmalone at omniti.com  Fri Nov 28 23:37:09 2014
From: zmalone at omniti.com (Zach Malone)
Date: Fri, 28 Nov 2014 18:37:09 -0500
Subject: [OmniOS-discuss] Postgres on Stable
In-Reply-To: <CALLZ=vU4MNQ2cZQgTHguzq4Ok-UywT7HTr5c6Pm05g4QyR8=7Q@mail.gmail.com>
References: <20141128191821.43382469@punda-mlia>
	<CACLsApuL0vfHhn7FAQw8OU+uPd=j=qEGRUVWAtCtA1VXoo+TVA@mail.gmail.com>
	<CA+FJ0zsFukbMSdnTRxqdO1fU_zR69_=bH4UDJkh3mdB+pmtABQ@mail.gmail.com>
	<CALLZ=vXs6XDheAGuE2aDJigNaHti4bVe7npAc2qiiqK45zh0sQ@mail.gmail.com>
	<CALLZ=vXQ=4qn=4SWAtAnBksf=dSVXkzFRrfWffe3HyoLB9b+1g@mail.gmail.com>
	<20141129071835.1daa0799@punda-mlia>
	<CALLZ=vU4MNQ2cZQgTHguzq4Ok-UywT7HTr5c6Pm05g4QyR8=7Q@mail.gmail.com>
Message-ID: <CALLZ=vV6u2H1WC6D+a_a9qiUSpOs0jBxwq=YjvRVnCDVowNLdA@mail.gmail.com>

>>   No suitable version of installed package pkg://omnios/entire at 11,5.11-0.151012:20141027T191658Z found
>>     Reject:  pkg://omnios/entire at 11,5.11-0.151012:20141027T191658Z
>>     Reason:  Excluded by proposed incorporation 'omniti/database/postgresql-934'
>>              This version is excluded by installed incorporation pkg://omnios/entire at 11,5.11-0.151012:20141027T191658Z
>
> I hadn't realized we incorporated against the OS for postgres, I'll
> spin a new postgresql-934 or 935 on a 012 machine when I get a chance.
> It looks like 012 doesn't have a Vagrant box published yet either,
> which will make this take a little longer.

I built postgresql-935 for you on a r151012 system, and published it
to the omniti-ms repo.  Want to give it a shot?  Past installing it, I
have not tried it on a production system (I'm planning on moving some
systems to r151012 this month), and I have not rolled any of the other
postgres libraries or modules that we publish for 9.2.

I also noticed that we're missing a Vagrant box for r151012, I'll see
if I can get that up next week.
--Zach

From gate03 at landcroft.co.uk  Sat Nov 29 00:47:45 2014
From: gate03 at landcroft.co.uk (Michael Mounteney)
Date: Sat, 29 Nov 2014 10:47:45 +1000
Subject: [OmniOS-discuss] Postgres on Stable
In-Reply-To: <CALLZ=vV6u2H1WC6D+a_a9qiUSpOs0jBxwq=YjvRVnCDVowNLdA@mail.gmail.com>
References: <20141128191821.43382469@punda-mlia>
	<CACLsApuL0vfHhn7FAQw8OU+uPd=j=qEGRUVWAtCtA1VXoo+TVA@mail.gmail.com>
	<CA+FJ0zsFukbMSdnTRxqdO1fU_zR69_=bH4UDJkh3mdB+pmtABQ@mail.gmail.com>
	<CALLZ=vXs6XDheAGuE2aDJigNaHti4bVe7npAc2qiiqK45zh0sQ@mail.gmail.com>
	<CALLZ=vXQ=4qn=4SWAtAnBksf=dSVXkzFRrfWffe3HyoLB9b+1g@mail.gmail.com>
	<20141129071835.1daa0799@punda-mlia>
	<CALLZ=vU4MNQ2cZQgTHguzq4Ok-UywT7HTr5c6Pm05g4QyR8=7Q@mail.gmail.com>
	<CALLZ=vV6u2H1WC6D+a_a9qiUSpOs0jBxwq=YjvRVnCDVowNLdA@mail.gmail.com>
Message-ID: <20141129104745.4f6d903f@punda-mlia>

On Fri, 28 Nov 2014 18:37:09 -0500
Zach Malone <zmalone at omniti.com> wrote:

> I built postgresql-935 for you on a r151012 system, and published it
> to the omniti-ms repo.  Want to give it a shot?  Past installing it, I
> have not tried it on a production system (I'm planning on moving some
> systems to r151012 this month), and I have not rolled any of the other
> postgres libraries or modules that we publish for 9.2.

Gosh, that's extremly decent of you Zack.  Thank you very much.  It's
installed and running here now.  My only complaint is that you have
allowed me to continue to be lazy.

Michael.

From danmcd at omniti.com  Sat Nov 29 01:28:39 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Fri, 28 Nov 2014 20:28:39 -0500
Subject: [OmniOS-discuss] Parity error on path mpt_sas2
In-Reply-To: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DACD1@AIRA-SRV.aira.local>
References: <3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAAD0@AIRA-SRV.aira.local>
	<1439760B-8D8B-40F9-8A62-4CFE4A9483C1@omniti.com>
	<3BE0DEED8863E5429BAE4CAEDF624565039C1B0DAADA@AIRA-SRV.aira.local>
	<CAOqBcP_12FaH77o+3nrGVm7KU3x1XCk3B-9uenKRRL63t5wdyg@mail.gmail.com>
	<3BE0DEED8863E5429BAE4CAEDF624565039C1B0DACD1@AIRA-SRV.aira.local>
Message-ID: <6309477E-045F-4265-BB87-7EB4BC0DEE80@omniti.com>


> On Nov 27, 2014, at 10:03 AM, Filip Marvan <filip.marvan at aira.cz> wrote:
> 
> Thank you for your help Aaron! I downgraded firmware to P18 and there are no more errors today.
> It seems, that there is something bad with P20 firmware!
>  

VERY interesting to know that the P20 IT firmware has problems.  May I share this on the illumos developer's list as well?

Thanks,
Dan


From danmcd at omniti.com  Sat Nov 29 02:05:39 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Fri, 28 Nov 2014 21:05:39 -0500
Subject: [OmniOS-discuss] ufsdump/ufsrestore: Segmentation Fault ?
In-Reply-To: <5478C61A.7080005@informatik.uni-tuebingen.de>
References: <5478C61A.7080005@informatik.uni-tuebingen.de>
Message-ID: <1F3FDF9A-1575-446D-83C5-008B9ABD5D39@omniti.com>

Wow.  I didn't know anyone still used UFS.  (Perhaps to restore prior to migration to ZFS?)

Wait a minute...


> root at kayak:/tmp# zfs create -V 100m rpool/test
> root at kayak:/tmp# newfs /dev/zvol/rdsk/rpool/test
> newfs: construct a new file system /dev/zvol/rdsk/rpool/test: (y/n)? y
> Warning: 4130 sector(s) in last cylinder unallocated
> /dev/zvol/rdsk/rpool/test:      204766 sectors in 34 cylinders of 48 tracks, 128 sectors
>        100.0MB in 3 cyl groups (14 c/g, 42.00MB/g, 20160 i/g)
> super-block backups (for fsck -F ufs -o b=#) at:
> 32, 86176, 172320,


I'm not sure if we support UFS on zvols officially.  Having said that... I tried to reproduce your bug with a raw disk, and it fails the same way:


# newfs /dev/rdsk/c5t0d0s0 
newfs: /dev/rdsk/c5t0d0s0 last mounted as /mnt
newfs: construct a new file system /dev/rdsk/c5t0d0s0: (y/n)? y
Warning: 290 sector(s) in last cylinder unallocated
/dev/rdsk/c5t0d0s0:	16760542 sectors in 2728 cylinders of 48 tracks, 128 sectors
	8183.9MB in 171 cyl groups (16 c/g, 48.00MB/g, 5824 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 98464, 196896, 295328, 393760, 492192, 590624, 689056, 787488, 885920,
 15831200, 15929632, 16028064, 16126496, 16224928, 16323360, 16421792,
 16520224, 16618656, 16717088
# mount -F ufs /dev/dsk/c5t0d0s0 /mnt
# ufsdump 0f test.dump /mnt
  DUMP: Date of this level 0 dump: November 28, 2014 09:03:39 PM EST
  DUMP: Date of last level 0 dump: the epoch
  DUMP: Dumping /dev/rdsk/c5t0d0s0 (danmcd-oi:/mnt) to test.dump.
  DUMP: Mapping (Pass I) [regular files]
  DUMP: Mapping (Pass II) [directories]
  DUMP: Writing 32 Kilobyte records
  DUMP: Estimated 604 blocks (302KB).
  DUMP: Dumping (Pass III) [directories]
  DUMP: Dumping (Pass IV) [regular files]
  DUMP: 574 blocks (287KB) on 1 volume at 17937 KB/sec
  DUMP: DUMP IS DONE
# ufsrestore ifv ./test.dump
Verify volume and initialize maps
Media block size is 126
Dump   date: November 28, 2014 09:03:39 PM EST
Dumped from: the epoch
Level 0 dump of /mnt on danmcd-oi:/dev/dsk/c5t0d0s0
Label: none
Extract directories from tape
Initialize symbol table.
Segmentation fault
# 

It fails in the function lookupparent():

	http://src.illumos.org/source/xref/illumos-gate/usr/src/cmd/backup/restore/symtab.c#191

I don't claim to begin to know how this crufty code works, especially as its change history comes from pre-OpenSolaris, never mind anything in Illumos.

Sorry I can't be of more immediate assistance, however.

Dan


From johan.kragsterman at capvert.se  Sat Nov 29 09:10:14 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Sat, 29 Nov 2014 10:10:14 +0100
Subject: [OmniOS-discuss] Illumos/omnios ALUA support was:: Active-Active
	vSphere
Message-ID: <OFB92E2269.62156AE9-ONC1257D9E.005A0AE4-C1257D9F.00326067@inse.com>

So, since this subject seams very much to be around ALUA understanding, this might be a good opportunity to discuss Illumos/OmniOS ALUA implementation..?

So I change/alter the subject, if you folks don't mind...?

If you mind, I can start a new thread...?

I continue further down...



-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev: -----
Till: Rune Tipsmark <rt at steait.net>
Fr?n: Saso Kiselkov 
S?nt av: "OmniOS-discuss" 
Datum: 2014-11-28 15:54
Kopia: "omnios-discuss at lists.omniti.com" <omnios-discuss at lists.omniti.com>
?rende: Re: [OmniOS-discuss] Active-Active vSphere

On 11/28/14 3:34 PM, Rune Tipsmark wrote:
> Okay, i noticed alua support is not enabled per default on omnios. How do i enable that?

The process is complicated, not very well documented and it isn't meant
for your scenario anyway. If you only have one storage node, then you
don't need ALUA, even if you have multiple connections to the node. So
no need to worry about it.




When you say it is poorly documented, I suppose you mean ALUA on Illumos dists?

If you mean on Illumos dists, I haven't found any documentation, apart from your(Saso) patch Feature #3775. If you know of any such documentation, pls give us a link to it.

There are some documentation though, from Nexenta and high-availability.com, but I find this difficult to use in OmniOS, since it's only Nexenta focus(of coarse).


One thing I don't understand is if the Nexenta implementation of ALUA use access through the "not prefered path" to the LUN(-s), even through the node that doesn't own it?? In my understanding, that would need a complicated interconnect between the head nodes, as well as a LU pass-through handling through the "not prefered path-node", wouldn't it? And I haven't seen that anywhere in Illumos/Nexenta, but of coarse, I might be wrong here...?

If I'm right here, the Nexenta implementation of ALUA is not providing the connection to the LUN(-s) through the "not prefered path", it only provides access after failover, when the "not prefered path" has become the "prefered path"?

If I'm wrong here, and have failed to understand something, pls enlighten me...



I'm also curious about the NMC, Nexenta Management Console. Is that one also in illumos-gate, or any equivalent software...?

If not, we would need to add and check the target group members on both(all) nodes manually, right?


How is this related to your implementation of stmf-ha, Saso?


Regards Johan







Cheers,
-- 
Saso

_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss



From asc1111 at gmail.com  Sat Nov 29 19:58:52 2014
From: asc1111 at gmail.com (Aaron Curry)
Date: Sat, 29 Nov 2014 12:58:52 -0700
Subject: [OmniOS-discuss] Illumos/omnios ALUA support was::
	Active-Active vSphere
In-Reply-To: <OFB92E2269.62156AE9-ONC1257D9E.005A0AE4-C1257D9F.00326067@inse.com>
References: <OFB92E2269.62156AE9-ONC1257D9E.005A0AE4-C1257D9F.00326067@inse.com>
Message-ID: <CAOqBcP-yVxq=FaMhVw9nPmJuRcVXMLKWKTrjH6AKvfJV7T1+bw@mail.gmail.com>

Johan,

It looks to me like your understanding of ALUA is correct.

Bottom line: ALUA is a way to present a LUN to a host while advertising all
possible paths. These "possible paths" include the local target HBAs as
well as any other target HBAs in the ALUA group. This is necessary in a
high availability storage solution!

Without ALUA, the host will see only the paths for the LUN to the pool
owner. Simply moving the pool to a secondary storage server and recreating
the LUNs/viwes (no ALUA involved) will cause the host to see it as a new
storage device. This setup can reduce downtime in the case of an outage or
planned maintenance but will cause any active IO to fail. The whole point
of highly available storage is for there to be no interruption in IO when a
pool moves between storage servers. Keep in mind that a zfs pool can only
be imported on one server at a time. To import it on two servers at the
same time will corrupt the pool.

I will respectfully disagree with Saso here and say that getting ALUA
itself up and running is not all that complicated (I'll qualify this
statement further down). The problem, as you pointed out, is the lack of
information on how to do it. I won't go into the details here, but the
basics of getting up and running are as follows:

1) Configure the stmf_proxy service similar to what high-availability.com
has documented here
<http://www.high-availability.com/FAQ/index.php?action=artikel&cat=3&id=62&artlang=en>.
Obviously there are some changes you have to make from that document, but
this is the most reliable information I have found on the matter.
2) Start the stmf_proxy service.
3) Create a target group with the same name on all storage nodes in your
cluster.
4) Add all target HBAs from all nodes to the target group on all servers.
So, if you have two servers with two target HBAs, the target group on both
servers will have four target group members.
5) Create a LUN and view. The stmf_proxy service will automagically
replicate the LUN/view information to the other nodes in the ALUA group.
6) Rescan the storage on the host.

When all that is done, the host will see the LUN with all paths configured
in the ALUA group. The paths to the pool owner will show as active while
the paths to the other storage servers will show as "standby." The
multipathing software will recognize the standby paths for what they are
and not attempt to perform any IO on those paths... ever. Configuring your
multipathing software to use the standby paths will result in IO timing out
on a standby path and then retrying on an active path. This will make the
IO appear to be really slow as Saso mentioned. However, with the
mulipathing software I have worked with (VMware, Windows... can't say I've
tried it with Linux), the round-robin setting recognizes a standby path and
simply ignores it. So, to back to the original poster's question: on ESXi,
Round-Robin works just fine... even when using an ALUA configuration.

With all that said, while I don't think that getting ALUA up and running is
that complicated (assuming you can get enough information to help you out
with the process) the problem is, where do you go from there? Highly
available storage is only good if a secondary node can take over when the
primary node fails. How do you manage that? How do the nodes in a cluster
monitor each other? How does a secondary node import a pool and change the
configured LUNs from standby to active (changing from standby to active is
not actually handled by ALUA)? And then what do you do when the primary
comes back online? How do you keep it from trying to take over? How do you
avoid split-brain scenarios? This is where the cluster management software
comes into play, and where things get really complicated. There are lots of
moving parts here to try to manage and coordinate. You can try to write up
all the scripts yourself. I have seen references to a set of open source
utilities that people have used with varying degrees of success. My
personal opinion is that if you truly need an HA storage solution, you are
much better off spending the money on a supported solution for the whole
stack. My preference for the FC HA storage stack:

Hardware: Supermicro Storage Bridge Bay system (purchased through a system
integrator who provides warranty/support). Its a little more expensive, but
definitely worth it.
OS: OmniOS. Great product with a great community. Plus there's a real, for
profit company backing it and providing professional support if needed.
Cluster: RSF-1 (high-availability.com). This is the only supported ZFS
cluster software that I am aware of (Nexenta actually uses it in their HA
cluster solution). It works great and doesn't cost much when you compare it
to buying a brand-name storage box.

There are, of course, other ways to implement redundant storage, such as
mirroring storage at the host level, but there are limitations to where
this can be used and it adds overhead to the host.

Sorry for this being so long. If you're still reading at this point... wow,
you must be really interested in this topic. For those who have made it
this far, the bottom line to the ALUA question is this: ALUA is only needed
in an HA setup. And, if you need HA and want to use OmniOS, then you're
much better off buying a license for RSF-1.

Aaron

On Sat, Nov 29, 2014 at 2:10 AM, Johan Kragsterman <
johan.kragsterman at capvert.se> wrote:

> So, since this subject seams very much to be around ALUA understanding,
> this might be a good opportunity to discuss Illumos/OmniOS ALUA
> implementation..?
>
> So I change/alter the subject, if you folks don't mind...?
>
> If you mind, I can start a new thread...?
>
> I continue further down...
>
>
>
> -----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev:
> -----
> Till: Rune Tipsmark <rt at steait.net>
> Fr?n: Saso Kiselkov
> S?nt av: "OmniOS-discuss"
> Datum: 2014-11-28 15:54
> Kopia: "omnios-discuss at lists.omniti.com" <omnios-discuss at lists.omniti.com>
> ?rende: Re: [OmniOS-discuss] Active-Active vSphere
>
> On 11/28/14 3:34 PM, Rune Tipsmark wrote:
> > Okay, i noticed alua support is not enabled per default on omnios. How
> do i enable that?
>
> The process is complicated, not very well documented and it isn't meant
> for your scenario anyway. If you only have one storage node, then you
> don't need ALUA, even if you have multiple connections to the node. So
> no need to worry about it.
>
>
>
>
> When you say it is poorly documented, I suppose you mean ALUA on Illumos
> dists?
>
> If you mean on Illumos dists, I haven't found any documentation, apart
> from your(Saso) patch Feature #3775. If you know of any such documentation,
> pls give us a link to it.
>
> There are some documentation though, from Nexenta and
> high-availability.com, but I find this difficult to use in OmniOS, since
> it's only Nexenta focus(of coarse).
>
>
> One thing I don't understand is if the Nexenta implementation of ALUA use
> access through the "not prefered path" to the LUN(-s), even through the
> node that doesn't own it?? In my understanding, that would need a
> complicated interconnect between the head nodes, as well as a LU
> pass-through handling through the "not prefered path-node", wouldn't it?
> And I haven't seen that anywhere in Illumos/Nexenta, but of coarse, I might
> be wrong here...?
>
> If I'm right here, the Nexenta implementation of ALUA is not providing the
> connection to the LUN(-s) through the "not prefered path", it only provides
> access after failover, when the "not prefered path" has become the
> "prefered path"?
>
> If I'm wrong here, and have failed to understand something, pls enlighten
> me...
>
>
>
> I'm also curious about the NMC, Nexenta Management Console. Is that one
> also in illumos-gate, or any equivalent software...?
>
> If not, we would need to add and check the target group members on
> both(all) nodes manually, right?
>
>
> How is this related to your implementation of stmf-ha, Saso?
>
>
> Regards Johan
>
>
>
>
>
>
>
> Cheers,
> --
> Saso
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141129/a62a6e86/attachment-0001.html>

From gearboxes at outlook.com  Sat Nov 29 23:50:50 2014
From: gearboxes at outlook.com (Machine Man)
Date: Sat, 29 Nov 2014 18:50:50 -0500
Subject: [OmniOS-discuss] Release 12 - KVM poor network performance
Message-ID: <BAY167-W576E805BCA6E92E076512EAE7F0@phx.gbl>

Hello,We experience slow network performance after upgrading for virtual machines.
Copying from a system (physical machine) to the VM the copy speed drop from 85MB/s on release 10 to between 9 and 14MB/s. Both systems I upgraded experience the exact same symptom and use the igb driver.Both have Supermicro mainboards;  X9DRD-7LN4F-JBOD and X10SLL-F. Copy tests are done between windows 8.1 and a Windows 2012R2 VM.
Changing to virtio for nic as suggested in a previous email had no impact other than causing kvm to crash after copying ~400MB.   
Anyone else seeing this behavior? I do see the X10SLL board has an e1000g nic alsoLINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
e1000g0      Ethernet             unknown    0      half      e1000g0
igb0         Ethernet             up         1000   full      igb0
I will switch over to that and test again. 
 
Thanks, 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141129/0fb24de0/attachment.html>

From danmcd at omniti.com  Sun Nov 30 00:57:12 2014
From: danmcd at omniti.com (Dan McDonald)
Date: Sat, 29 Nov 2014 19:57:12 -0500
Subject: [OmniOS-discuss] Release 12 - KVM poor network performance
In-Reply-To: <BAY167-W576E805BCA6E92E076512EAE7F0@phx.gbl>
References: <BAY167-W576E805BCA6E92E076512EAE7F0@phx.gbl>
Message-ID: <1F1549D7-C1AC-41DA-972F-8A63CB3BD8D9@omniti.com>

You mention 012, but you do not mention what you're upgrading from.  That can be helpful.

Dan

Sent from my iPhone (typos, autocorrect, and all)

> On Nov 29, 2014, at 6:50 PM, Machine Man <gearboxes at outlook.com> wrote:
> 
> Hello,
> We experience slow network performance after upgrading for virtual machines.
> Copying from a system (physical machine) to the VM the copy speed drop from 85MB/s on release 10 to between 9 and 14MB/s. Both systems I upgraded experience the exact same symptom and use the igb driver.
> Both have Supermicro mainboards;  X9DRD-7LN4F-JBOD and X10SLL-F. Copy tests are done between windows 8.1 and a Windows 2012R2 VM.
> Changing to virtio for nic as suggested in a previous email had no impact other than causing kvm to crash after copying ~400MB. 
>  
> Anyone else seeing this behavior?
>  
> I do see the X10SLL board has an e1000g nic also
> LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
> e1000g0      Ethernet             unknown    0      half      e1000g0
> igb0         Ethernet             up         1000   full      igb0
> I will switch over to that and test again. 
>  
> Thanks,
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141129/7d5b697c/attachment.html>

From gearboxes at outlook.com  Sun Nov 30 02:23:28 2014
From: gearboxes at outlook.com (Machine Man)
Date: Sat, 29 Nov 2014 21:23:28 -0500
Subject: [OmniOS-discuss] Release 12 - KVM poor network performance
Message-ID: <BAY405-EAS70FEAB6E9692210A8F91D4AE7C0@phx.gbl>

>From 010.
________________________________
From: Dan McDonald<mailto:danmcd at omniti.com>
Sent: ?11/?29/?2014 19:57
To: Machine Man<mailto:gearboxes at outlook.com>
Cc: omnios-discuss at lists.omniti.com<mailto:omnios-discuss at lists.omniti.com>
Subject: Re: [OmniOS-discuss] Release 12 - KVM poor network performance

You mention 012, but you do not mention what you're upgrading from.  That can be helpful.

Dan

Sent from my iPhone (typos, autocorrect, and all)

> On Nov 29, 2014, at 6:50 PM, Machine Man <gearboxes at outlook.com> wrote:
>
> Hello,
> We experience slow network performance after upgrading for virtual machines.
> Copying from a system (physical machine) to the VM the copy speed drop from 85MB/s on release 10 to between 9 and 14MB/s. Both systems I upgraded experience the exact same symptom and use the igb driver.
> Both have Supermicro mainboards;  X9DRD-7LN4F-JBOD and X10SLL-F. Copy tests are done between windows 8.1 and a Windows 2012R2 VM.
> Changing to virtio for nic as suggested in a previous email had no impact other than causing kvm to crash after copying ~400MB.
>
> Anyone else seeing this behavior?
>
> I do see the X10SLL board has an e1000g nic also
> LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
> e1000g0      Ethernet             unknown    0      half      e1000g0
> igb0         Ethernet             up         1000   full      igb0
> I will switch over to that and test again.
>
> Thanks,
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141129/9a546d96/attachment.html>

From henson at acm.org  Sun Nov 30 02:55:57 2014
From: henson at acm.org (Paul B. Henson)
Date: Sat, 29 Nov 2014 18:55:57 -0800
Subject: [OmniOS-discuss] common-factor key exchange
In-Reply-To: <20141128191703.00a9df99@punda-mlia>
References: <20141128191703.00a9df99@punda-mlia>
Message-ID: <20141130025557.GI29549@bender.unx.csupomona.edu>

On Fri, Nov 28, 2014 at 07:17:03PM +1000, Michael Mounteney wrote:
> After a recent upgrade to my Gentoo Linux installations, I can no
> longer ssh into them from OmniOS:
> 
> -bash-4.3$ ssh 192.168.1.48
> no common kex alg: client
> 'diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1', server
> 'curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1'

What is your sshd_config on the gentoo side? I have no problems
connecting from an r151012 omnios client to a gentoo 6.6_p1-r1 server.
Unless you've modified the ssh server config, out of the box a gentoo
ssh server should be willing to accept diffie-hellman-group1-sha1. Per
the sshd_config manpage under Gentoo:

     KexAlgorithms
             Specifies the available KEX (Key Exchange) algorithms.  Multiple
             algorithms must be comma-separated.  The default is

                   curve25519-sha256 at libssh.org,
                   ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,
                   diffie-hellman-group-exchange-sha256,
                   diffie-hellman-group-exchange-sha1,
                   diffie-hellman-group14-sha1,
                   diffie-hellman-group1-sha1

Given your server is not offering diffie-hellman-group1-sha1, my guess is
there must be an explicit KexAlgorithms configuration in place removing
it...


From henson at acm.org  Sun Nov 30 03:06:52 2014
From: henson at acm.org (Paul B. Henson)
Date: Sat, 29 Nov 2014 19:06:52 -0800
Subject: [OmniOS-discuss] common-factor key exchange
In-Reply-To: <54789288.20302@netbsd.org>
References: <20141128191703.00a9df99@punda-mlia>
 <54789288.20302@netbsd.org>
Message-ID: <20141130030652.GJ29549@bender.unx.csupomona.edu>

On Fri, Nov 28, 2014 at 04:19:36PM +0100, Richard PALO wrote:

> this seems to be a known issue for sun ssh.

I don't think anybody is maintaining the sun ssh fork in illumos :(. So
it's just going to get more and more obsolete.

Personally, I think the sun fork should just get dropped from
illumos-gate and openssh be added in by distributions. As I recall,
there were three motivations for the original sun fork; sun engineers
didn't agree with the way upstream decided to do privsep, they added in
RBAC support, and more support for solaris auditing features.

I'm not really qualified to comment on the difference of opinion
regarding privsep, but I think I'd take an upstream maintained version
over an abandoned one 8-/. And I don't use any of the RBAC/audit
features, although perhaps somebody does.

Hmm, interesting, I see omnios does have network/openssh and
network/openssh-server packages in the OS repo, I didn't realize that.
Any thoughts on making those the default for an install rather than the
native illumos ssh packages? Perhaps I'm biased by my own needs, but it
seems an up to date ssh implementation would be of more general use than
an outdated one with a few sun specific features.


From richard.elling at richardelling.com  Sun Nov 30 03:25:01 2014
From: richard.elling at richardelling.com (Richard Elling)
Date: Sat, 29 Nov 2014 19:25:01 -0800
Subject: [OmniOS-discuss] Scrub behavour and set readonly=on
In-Reply-To: <ffa8c9a3d6e514320f01e9112ec540a0.squirrel@emailmg.netfirms.com>
References: <ffa8c9a3d6e514320f01e9112ec540a0.squirrel@emailmg.netfirms.com>
Message-ID: <8F314FCE-EEE4-4493-BB9F-9D81D2344704@richardelling.com>


On Nov 25, 2014, at 11:31 AM, steve at linuxsuite.org wrote:

> 
> Howdy
> 
>       zfs get all
> 
>        AND
> 
>      zpool get all
> 
>        have settings for readonly
> 
>        readonly       off         default
> 
>     if I set readonly=on
> 
>       will this affect scrub and its ability to fix errors?
> 
>       For both zfs and zpool settings?
> 
>      If I set
>         zfs set readonly=on
> 
>      scrub will start and not complain... can it still fix errors?

yes

> 
>            What about for
> 
>              zpool set readonly=on

If the pool is imported readonly, then there should be no writes at all, and hence, no
data correction.
 -- richard

> 
>          thanx - steve 	omnios-discuss at lists.omniti.com
> 
> 
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss


From johan.kragsterman at capvert.se  Sun Nov 30 06:02:47 2014
From: johan.kragsterman at capvert.se (Johan Kragsterman)
Date: Sun, 30 Nov 2014 07:02:47 +0100
Subject: [OmniOS-discuss] Ang: Re: Illumos/omnios ALUA support was::
 Active-Active vSphere
In-Reply-To: <CAOqBcP-yVxq=FaMhVw9nPmJuRcVXMLKWKTrjH6AKvfJV7T1+bw@mail.gmail.com>
References: <CAOqBcP-yVxq=FaMhVw9nPmJuRcVXMLKWKTrjH6AKvfJV7T1+bw@mail.gmail.com>,
	<OFB92E2269.62156AE9-ONC1257D9E.005A0AE4-C1257D9F.00326067@inse.com>
Message-ID: <OF595EA113.DCDD225F-ONC1257DA0.001D353A-C1257DA0.002136F4@inse.com>

Hi, Aron and others interested in this!


Thanks for your respons, and don?t worry about it length, since it is interesting.

I'll continue further down...



-----Aaron Curry <asc1111 at gmail.com> skrev: -----
Till: Johan Kragsterman <johan.kragsterman at capvert.se>
Fr?n: Aaron Curry <asc1111 at gmail.com>
Datum: 2014-11-29 20:59
Kopia: Saso Kiselkov <skiselkov.ml at gmail.com>, "omnios-discuss at lists.omniti.com" <omnios-discuss at lists.omniti.com>
?rende: Re: [OmniOS-discuss] Illumos/omnios ALUA support was:: Active-Active vSphere

Johan,

It looks to me like your understanding of ALUA is correct.

Bottom line: ALUA is a way to present a LUN to a host while advertising all possible paths. These "possible paths" include the local target HBAs as well as any other target HBAs in the ALUA group. This is necessary in a high availability storage solution!




What I would like to know here, is which software is responsible for advertising the LUN(-s) as prefered  or not prefered? Would that be the stmf proxy service?







With all that said, while I don't think that getting ALUA up and running is that complicated (assuming you can get enough information to help you out with the process) the problem is, where do you go from there? Highly available storage is only good if a secondary node can take over when the primary node fails. How do you manage that? How do the nodes in a cluster monitor each other? How does a secondary node import a pool and change the configured LUNs from standby to active (changing from standby to active is not actually handled by ALUA)? And then what do you do when the primary comes back online? How do you keep it from trying to take over? How do you avoid split-brain scenarios? This is where the cluster management software comes into play, and where things get really complicated. There are lots of moving parts here to try to manage and coordinate. You can try to write up all the scripts yourself. I have seen references to a set of open source utilities that people have used with varying degrees of success.





Saso has a cluster solution here, that I still haven't tried, but I know other has:

http://zfs-create.blogspot.se/2014/05/building-zfs-storage-appliance-part-2.html

That is one of the reasons I went in to this discussion, that I wanted to be able to understand how to get ALUA into that solution.


Saso, do you got any software for your stmf-ha that is comparable to the RFS-1 stmf proxy service?

I saw this "Feature #3775 Upstreaming of ALUA use-after-free fixes from Nexenta", is that something that is usable for your solution?

If so, got any docs for how to implement that?







Sorry for this being so long. If you're still reading at this point... wow, you must be really interested in this topic. For those who have made it this far, the bottom line to the ALUA question is this: ALUA is only needed in an HA setup. And, if you need HA and want to use OmniOS, then you're much better off buying a license for RSF-1.








I agree about buying an RFS-1 license for important production clusters, but there are many other scenarious where you would like to have access to this: In education, in your own process of understanding how things work, for small companies that don't have the money yet, but want to give it a shot, etc.

 For me, I'd like to set it up and get it running to get a good understanding how it all works. I want to understand the software, I want to undestand the processes.




Regards Johan







Aaron

On Sat, Nov 29, 2014 at 2:10 AM, Johan Kragsterman <johan.kragsterman at capvert.se> wrote:
So, since this subject seams very much to be around ALUA understanding, this might be a good opportunity to discuss Illumos/OmniOS ALUA implementation..?

So I change/alter the subject, if you folks don't mind...?

If you mind, I can start a new thread...?

I continue further down...



-----"OmniOS-discuss" <omnios-discuss-bounces at lists.omniti.com> skrev: -----
Till: Rune Tipsmark <rt at steait.net>
Fr?n: Saso Kiselkov
S?nt av: "OmniOS-discuss"
Datum: 2014-11-28 15:54
Kopia: "omnios-discuss at lists.omniti.com" <omnios-discuss at lists.omniti.com>
?rende: Re: [OmniOS-discuss] Active-Active vSphere

On 11/28/14 3:34 PM, Rune Tipsmark wrote:
> Okay, i noticed alua support is not enabled per default on omnios. How do i enable that?

The process is complicated, not very well documented and it isn't meant
for your scenario anyway. If you only have one storage node, then you
don't need ALUA, even if you have multiple connections to the node. So
no need to worry about it.




When you say it is poorly documented, I suppose you mean ALUA on Illumos dists?

If you mean on Illumos dists, I haven't found any documentation, apart from your(Saso) patch Feature #3775. If you know of any such documentation, pls give us a link to it.

There are some documentation though, from Nexenta and high-availability.com, but I find this difficult to use in OmniOS, since it's only Nexenta focus(of coarse).


One thing I don't understand is if the Nexenta implementation of ALUA use access through the "not prefered path" to the LUN(-s), even through the node that doesn't own it?? In my understanding, that would need a complicated interconnect between the head nodes, as well as a LU pass-through handling through the "not prefered path-node", wouldn't it? And I haven't seen that anywhere in Illumos/Nexenta, but of coarse, I might be wrong here...?

If I'm right here, the Nexenta implementation of ALUA is not providing the connection to the LUN(-s) through the "not prefered path", it only provides access after failover, when the "not prefered path" has become the "prefered path"?

If I'm wrong here, and have failed to understand something, pls enlighten me...



I'm also curious about the NMC, Nexenta Management Console. Is that one also in illumos-gate, or any equivalent software...?

If not, we would need to add and check the target group members on both(all) nodes manually, right?


How is this related to your implementation of stmf-ha, Saso?


Regards Johan







Cheers,
--
Saso

_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss


_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss




From gate03 at landcroft.co.uk  Sun Nov 30 19:56:54 2014
From: gate03 at landcroft.co.uk (Michael Mounteney)
Date: Mon, 1 Dec 2014 05:56:54 +1000
Subject: [OmniOS-discuss] common-factor key exchange
In-Reply-To: <20141130025557.GI29549@bender.unx.csupomona.edu>
References: <20141128191703.00a9df99@punda-mlia>
	<20141130025557.GI29549@bender.unx.csupomona.edu>
Message-ID: <20141201055654.2a3e23f8@punda-mlia>

On Sat, 29 Nov 2014 18:55:57 -0800
"Paul B. Henson" <henson at acm.org> wrote:

> What is your sshd_config on the gentoo side?

Port {something other than 22}
PasswordAuthentication no
PermitEmptyPasswords no
UsePAM no
PrintMotd no
PrintLastLog no
UsePrivilegeSeparation sandbox          # Default for new installations.
ClientAliveInterval 61
Subsystem       sftp    /usr/lib/misc/sftp-server
AcceptEnv LANG LC_*

From heinrich.vanriel at outlook.com  Sat Nov 29 20:42:59 2014
From: heinrich.vanriel at outlook.com (Machine Man)
Date: Sat, 29 Nov 2014 15:42:59 -0500
Subject: [OmniOS-discuss] Release 12 KVM poor network performance
Message-ID: <BAY167-W945359F73CCDEE97B19CBE837F0@phx.gbl>

Experience slow network performance after upgrading.
Copying from another windows system (physical machine) it drops form the 85MB/s on R10 to between 9 and 14MB/s. Two systems experience the exact same symptom. Both make use of igb driver. 
Copy tests are done between windows 8.1 and Windows 2012R2 VM.
Changing to virtio for nic has no impact other than causing kvm to crash after copying ~400MB. 
 
Any ideas? 
 
Thanks,
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141129/757e6642/attachment.html>

From valo at valo.at  Fri Nov 28 09:55:04 2014
From: valo at valo.at (Christian Kivalo)
Date: Fri, 28 Nov 2014 10:55:04 +0100
Subject: [OmniOS-discuss] common-factor key exchange
In-Reply-To: <20141128191703.00a9df99@punda-mlia>
References: <20141128191703.00a9df99@punda-mlia>
Message-ID: <4fcd8a9de3d5ab64e565365c78cb3105@kivalo.at>

Hello,


On Fri, 28 Nov 2014 19:17:03 +1000 Michael Mounteney <gate03 at landcroft.co.uk>
wrote

> After a recent upgrade to my Gentoo Linux installations, I can no
> longer ssh into them from OmniOS:
> 
> -bash-4.3$ ssh 192.168.1.48
> no common kex alg: client
> 'diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1', server
> 'curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2
> -nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1'
> 

This seems to be a recurring error message with current linux distributions.

I have noticed this error message after recently upgrading to Debian 8, where i
currently have lowered the sshd defaults on Debian to allow the OmniOS to
connect.

I would be interestent in replacing the default ssh package with OpenSSH on the
OmniOS installation, but the ssh package seems to be bound to @entire so this
looks to me as if this is not as easy as uninstall/install to replace the ssh
daemon/client...

> The client being OmniOS and the server being Gentoo Linux.  Now I could
> downgrade what Gentoo offers, but it would be nicer to upgrade what
> OmniOS offers.  Is there a way ?

It should be possible to create a special host configuration to offer specific
kex algorithms only to the the OmniOS client but had no time to test that
configuration.

> 
> Michael.

- christian

> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss



