`<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">That was a secondary thought, maybe worth testing one day.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Primarily I was looking at a way of speeding up zfs send-recv.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Guess it’s a no go on a single HCA…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Johan Kragsterman [mailto:johan.kragsterman@capvert.se]
<br>
<b>Sent:</b> Monday, November 03, 2014 12:41 AM<br>
<b>To:</b> Rune Tipsmark<br>
<b>Cc:</b> omnios-discuss@lists.omniti.com<br>
<b>Subject:</b> Ang: RE: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br>
Hej!<br>
<br>
<br>
If you want to mirror a local volume with a remote LU, I don't think it is a good idea. Asking for trubble...<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">I understand you want to get an HA solution. But I believe HA solutions needs other strategies than that.<br>
<br>
<br>
Best regards from/Med vänliga hälsningar från<br>
<br>
Johan Kragsterman<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br>
Capvert<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br>
<br>
</span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#990099">-----Rune Tipsmark <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#990099"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#990099">>
skrev: -----</span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p>
<div>
<div style="border:none;border-left:solid black 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">Till: Johan Kragsterman <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:johan.kragsterman@capvert.se"><span lang="DA">johan.kragsterman@capvert.se</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br>
Från: Rune Tipsmark <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br>
Datum: 2014-11-03 08:37<br>
Kopia: "</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:omnios-discuss@lists.omniti.com"><span lang="DA">omnios-discuss@lists.omniti.com</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">"
<</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:omnios-discuss@lists.omniti.com"><span lang="DA">omnios-discuss@lists.omniti.com</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br>
Ärende: RE: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Tjenare,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Maybe its not possible, would explain why I cannot see the shared lun on the other system.
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I don’t have any spare PCI-E slots on either, all filled with IODrives etc.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Br,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Rune</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>]
<br>
<b>Sent:</b> Sunday, November 02, 2014 11:24 PM<br>
<b>To:</b> Rune Tipsmark<br>
<b>Cc:</b> <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br>
<b>Subject:</b> Ang: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br>
Hej!<br>
<br>
<br>
Hmm, how about the target/initiiator configuration of the HCA's?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> When I think about it, I have never done this that you're trying: To use SRP both as target and initiator on the
same system. I don't even know if it is possible. Since an HCA is entirely(with all ports) a target, you need to set one HCA as a target, and one HCA as an initiator. I've done it on fibre channel, but never on IB/SRP.<br>
<br>
<br>
Best regards from/Med vänliga hälsningar från<br>
<br>
Johan Kragsterman</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br>
Capvert</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br>
<br>
</span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#990099">-----Rune Tipsmark <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#990099"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#990099">>
skrev: -----</span><o:p></o:p></p>
<div>
<div style="border:none;border-left:solid black 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">Till: Johan Kragsterman <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:johan.kragsterman@capvert.se"><span lang="DA">johan.kragsterman@capvert.se</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br>
Från: Rune Tipsmark <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br>
Datum: 2014-11-03 08:09<br>
Kopia: "</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:omnios-discuss@lists.omniti.com"><span lang="DA">omnios-discuss@lists.omniti.com</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">"
<</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:omnios-discuss@lists.omniti.com"><span lang="DA">omnios-discuss@lists.omniti.com</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br>
Ärende: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">connectX2 and drivers are loaded, both OmniOS servers have LUNs I can access from both ESX and Windows…
just the conection between them that I cant figure out.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Br,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Rune</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>]
<br>
<b>Sent:</b> Sunday, November 02, 2014 10:49 PM<br>
<b>To:</b> Rune Tipsmark<br>
<b>Cc:</b> <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br>
<b>Subject:</b> Ang: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br>
Hej!</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">I can see I forgot the list in previous mail...</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Anyway, are you using connectx adapters?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Have you checked if the driver is loaded in the host system?</span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br>
<br>
Best regards from/Med vänliga hälsningar från<br>
<br>
Johan Kragsterman</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br>
Capvert</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br>
<br>
</span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#990099">-----Rune Tipsmark <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#990099"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif;color:#990099">>
skrev: -----</span><o:p></o:p></p>
<div>
<div style="border:none;border-left:solid black 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">Till: Johan Kragsterman <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:johan.kragsterman@capvert.se"><span lang="DA">johan.kragsterman@capvert.se</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br>
Från: Rune Tipsmark <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br>
Datum: 2014-11-03 00:07<br>
Ärende: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-family:"Courier New"">It doesn't show up in stmfadm list-target.... only my other servers such as ESX and Windows show up...<br>
<br>
Subnet Manager is OpenSM running on the ESX hosts, has been working OK ever since I installed then, prior to that I had a couple of Cisco(Topspin) DDR Switches with inbuilt SM.<br>
<br>
Any other idea? All things I configured so far on both ZFS boxes,windows and esx has worked as expected... just this darn thing that's not doing what its supposed to...<br>
<br>
It should be enough to just add the HCA from ZFS10 onto the host-group on ZFS00 for ZFS10 to see a LUN on ZFS00 granted that there is a view configured on ZFS00 as well... or am I missing a step?
<br>
<br>
Br<br>
Rune<br>
<br>
-----Original Message-----<br>
From: Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>]
<br>
Sent: Sunday, November 02, 2014 11:58 AM<br>
To: Rune Tipsmark<br>
Subject: Ang: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br>
<br>
<br>
-----Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>> skrev: -----<br>
Till: Johan Kragsterman <<a href="mailto:johan.kragsterman@capvert.se">johan.kragsterman@capvert.se</a>><br>
Från: Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>><br>
Datum: 2014-11-02 20:05<br>
Kopia: David Bomba <<a href="mailto:turbo124@gmail.com">turbo124@gmail.com</a>>, "<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>" <<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>><br>
Ärende: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br>
<br>
I know, but how do I initiate a session from ZFS10?<br>
<br>
<br>
<br>
If a session doesn't show up in "stmfadm list-targte -v", then you got something wrong in the IB fabric, if the view is right. Do you use a switch? Where do you have your IB Storage Manager?<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
Br,<br>
<br>
Rune<br>
<br>
<br>
<br>
From: Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>]<br>
Sent: Sunday, November 02, 2014 10:33 AM<br>
To: Rune Tipsmark<br>
Cc: David Bomba; <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br>
Subject: Ang: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br>
<br>
<br>
<br>
<br>
<br>
<br>
-----Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>> skrev: -----<br>
<br>
Till: Johan Kragsterman <<a href="mailto:johan.kragsterman@capvert.se">johan.kragsterman@capvert.se</a>><br>
Från: Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>><br>
Datum: 2014-11-02 19:11<br>
Kopia: David Bomba <<a href="mailto:turbo124@gmail.com">turbo124@gmail.com</a>>, "<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>" <<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>><br>
Ärende: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br>
<br>
Hi Johan<br>
<br>
Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine..<br>
<br>
What I can't figure out is how to share a LUN with the other zfs box... see pasted info below...<br>
<br>
ZFS00: (the box I want to receive my zfs snapshot) The below are all ESX servers, cannot see the other ZFS box<br>
<br>
root@zfs00:/pool03# stmfadm list-target -v<br>
Target: eui.0002C90300095E7C<br>
Operational Status: Online<br>
Provider Name : srpt<br>
Alias : -<br>
Protocol : SRP<br>
Sessions : 8<br>
Initiator: eui.0002C903000F397C<br>
Alias: 8102c90300095e7e:0002c903000f397c<br>
Logged in since: Sun Nov 2 02:09:41 2014<br>
Initiator: eui.0002C903000F397B<br>
Alias: 8102c90300095e7d:0002c903000f397b<br>
Logged in since: Sun Nov 2 02:09:40 2014<br>
Initiator: eui.0002C903000D3D04<br>
Alias: 8102c90300095e7e:0002c903000d3d04<br>
Logged in since: Sat Nov 1 21:14:47 2014<br>
Initiator: eui.0002C90300104F47<br>
Alias: 8102c90300095e7d:0002c90300104f47<br>
Logged in since: Sat Nov 1 21:12:54 2014<br>
Initiator: eui.0002C903000D3D03<br>
Alias: 8102c90300095e7d:0002c903000d3d03<br>
Logged in since: Sat Nov 1 21:12:32 2014<br>
Initiator: eui.0002C90300104F48<br>
Alias: 8102c90300095e7e:0002c90300104f48<br>
Logged in since: Sat Nov 1 21:10:45 2014<br>
Initiator: eui.0002C903000A48FA<br>
Alias: 8102c90300095e7e:0002c903000a48fa<br>
Logged in since: Sat Nov 1 21:10:40 2014<br>
Initiator: eui.0002C903000D3CA0<br>
Alias: 8102c90300095e7e:0002c903000d3ca0<br>
Logged in since: Sat Nov 1 21:10:39 2014<br>
Target: iqn.2010-09.org.napp-it:1394106801<br>
Operational Status: Online<br>
Provider Name : iscsit<br>
Alias : 03.06.2014<br>
Protocol : iSCSI<br>
Sessions : 0<br>
<br>
root@zfs00:/pool03# stmfadm list-lu -v<br>
LU Name: 600144F007780B7F00005455EDD50002<br>
Operational Status: Online<br>
Provider Name : sbd<br>
Alias : /pool03/LU11<br>
View Entry Count : 1<br>
Data File : /pool03/LU11<br>
Meta File : not set<br>
Size : 2199023255552<br>
Block Size : 512<br>
Management URL : not set<br>
Vendor ID : SUN<br>
Product ID : COMSTAR<br>
Serial Num : not set<br>
Write Protect : Disabled<br>
Writeback Cache : Disabled<br>
Access State : Active<br>
<br>
root@zfs00:/pool03# stmfadm list-hg -v<br>
Host Group: ESX<br>
Host Group: Windows<br>
Host Group: ESX-iSER<br>
Host Group: OmniOS<br>
Member: eui.0002C903000923E6 <<--- The other ZFS box<br>
Member: iqn.2010-09.org.napp-it:1402013225<br>
Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d<br>
<br>
root@zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002 View Entry: 0<br>
Host group : OmniOS<br>
Target group : All<br>
LUN : 0<br>
<br>
ZFS10: (the sending box where I want to see the LUN from ZFS00) No disk show up from ZFS00...<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
Hi!<br>
<br>
<br>
<br>
<br>
<br>
You got eui.0002C903000923E6 in host group OmniOS, but you don't have a session from that eui to the target.<br>
<br>
<br>
<br>
Rgrds Johan<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
root@zfs10:/root# format<br>
Searching for disks...done<br>
<br>
<br>
AVAILABLE DISK SELECTIONS:<br>
0. c4t5000C50055FC9533d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c50055fc9533<br>
1. c4t5000C50055FE6A63d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c50055fe6a63<br>
2. c4t5000C500625B7EA7d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c500625b7ea7<br>
3. c4t5000C500625B86E3d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c500625b86e3<br>
4. c4t5000C500625B886Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c500625b886f<br>
5. c4t5000C500625B8137d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c500625b8137<br>
6. c4t5000C500625B8427d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c500625b8427<br>
7. c4t5000C500625BB773d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c500625bb773<br>
8. c4t5000C500625BC2C3d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c500625bc2c3<br>
9. c4t5000C500625BD3EBd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c500625bd3eb<br>
10. c4t5000C50057085A6Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c50057085a6b<br>
11. c4t5000C50057086B67d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c50057086b67<br>
12. c4t5000C50062878C0Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c50062878c0b<br>
13. c4t5000C50062878C43d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c50062878c43<br>
14. c4t5000C500570858EFd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c500570858ef<br>
15. c4t5000C500570870D3d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c500570870d3<br>
16. c4t5000C5005708351Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c5005708351b<br>
17. c4t5000C5005708296Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c5005708296f<br>
18. c4t5000C50057089753d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c50057089753<br>
19. c4t5000C50057086307d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c50057086307<br>
20. c4t5000C50062879687d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c50062879687<br>
21. c4t5000C50062879707d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c50062879707<br>
22. c4t5000C50062879723d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c50062879723<br>
23. c4t5000C50062879787d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br>
/scsi_vhci/disk@g5000c50062879787<br>
24. c7t0d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56><br>
/pci@0,0/pci15d9,704@1f,2/disk@0,0<br>
25. c7t1d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56><br>
/pci@0,0/pci15d9,704@1f,2/disk@1,0<br>
26. c7t4d0 <ATA-InnoLite SATADOM-19-14.91GB><br>
/pci@0,0/pci15d9,704@1f,2/disk@4,0<br>
27. c7t5d0 <ATA-InnoLite SATADOM-19-14.91GB><br>
/pci@0,0/pci15d9,704@1f,2/disk@5,0<br>
28. c10d0 <Unknown-Unknown-0001-149.00GB><br>
/pci@0,0/pci8086,3c08@3/pci103c,178b@0<br>
29. c11d0 <Unknown-Unknown-0001-149.00GB><br>
/pci@0,0/pci8086,3c0a@3,2/pci103c,178b@0<br>
30. c12d0 <Unknown-Unknown-0001-298.02GB><br>
/pci@79,0/pci8086,3c04@2/pci10b5,8616@0/pci10b5,8616@5/pci103c,178e@0<br>
31. c13d0 <Unknown-Unknown-0001-298.02GB><br>
/pci@79,0/pci8086,3c04@2/pci10b5,8616@0/pci10b5,8616@6/pci103c,178e@0<br>
32. c14d0 <Unknown-Unknown-0001-298.02GB><br>
/pci@79,0/pci8086,3c08@3/pci10b5,8616@0/pci10b5,8616@5/pci103c,178e@0<br>
33. c15d0 <Unknown-Unknown-0001-298.02GB><br>
/pci@79,0/pci8086,3c08@3/pci10b5,8616@0/pci10b5,8616@6/pci103c,178e@0<br>
Specify disk (enter its number):<br>
<br>
<br>
<br>
-----Original Message-----<br>
From: Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>]<br>
Sent: Sunday, November 02, 2014 9:56 AM<br>
To: Rune Tipsmark<br>
Cc: David Bomba; <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br>
Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br>
<br>
<br>
-----"OmniOS-discuss" <<a href="mailto:omnios-discuss-bounces@lists.omniti.com">omnios-discuss-bounces@lists.omniti.com</a>> skrev: -----<br>
Till: David Bomba <<a href="mailto:turbo124@gmail.com">turbo124@gmail.com</a>><br>
Från: Rune Tipsmark <br>
Sänt av: "OmniOS-discuss" <br>
Datum: 2014-11-02 07:55<br>
Kopia: "<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>" <<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>><br>
Ärende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br>
<br>
Sounds sensible, how do I do that?<br>
<br>
I tried creating a view for a thin lu with my other zfs box, but how do I detect it?<br>
<br>
<br>
<br>
<br>
<br>
First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application.<br>
<br>
Did you create SRP target service using COMSTAR?<br>
<br>
# svcadm enable -r ibsrp/target<br>
<br>
What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v"<br>
<br>
Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link?<br>
<br>
If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA.<br>
<br>
<br>
Rgrds Johan<br>
<br>
<br>
<br>
<br>
<br>
<br>
I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article
<a href="http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html">
http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html</a><br>
<br>
<br>
<br>
Br,<br>
<br>
Rune<br>
<br>
<br>
<br>
From: David Bomba [<a href="mailto:turbo124@gmail.com">mailto:turbo124@gmail.com</a>]
<br>
Sent: Saturday, November 01, 2014 6:01 PM<br>
To: Rune Tipsmark<br>
Cc: <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br>
Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br>
<br>
<br>
<br>
I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential. <br>
<br>
<br>
<br>
<br>
<br>
On 2 Nov 2014, at 11:45 am, Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>> wrote:<br>
<br>
<br>
<br>
Hi all,<br>
<br>
<br>
<br>
Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.<br>
<br>
<br>
<br>
If not, is there a way I can aggregate say 8 or 16 IPoIB partitions and push throughput to a more reasonable speed…<br>
<br>
<br>
<br>
Br,<br>
<br>
Rune<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
OmniOS-discuss mailing list<br>
<a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br>
<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><br>
<br>
<br>
<br>
_______________________________________________<br>
OmniOS-discuss mailing list<br>
<a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br>
<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><br>
<br>
<br>
</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>
`