`<font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"><div><br><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2">Hej!<br><br><br>I use that method a lot for accessing a backup server over SAN, but mainly over FC.<br><br><br>Best regards from/Med vänliga hälsningar från<br><br>Johan Kragsterman</font><div><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"><br>Capvert</font></div></div><br><br><font color="#990099">-----Rune Tipsmark <rt@steait.net> skrev: -----</font><div class="iNotesHistory" style="padding-left:5px;"><div style="padding-right:0px;padding-left:5px;border-left:solid black 2px;">Till: Johan Kragsterman <johan.kragsterman@capvert.se><br>Från: Rune Tipsmark <rt@steait.net><br>Datum: 2014-11-03 09:51<br>Kopia: "omnios-discuss@lists.omniti.com" <omnios-discuss@lists.omniti.com><br>Ärende: RE: RE: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br><br> <!--Notes ACF <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">--> <!--[if gte mso 9]><xml> <o:shapedefaults v:ext="edit" spidmax="1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext="edit"> <o:idmap v:ext="edit" data="1" /> </o:shapelayout></xml><![endif]--> <div class="WordSection1"> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">That was a secondary thought, maybe worth testing one day.<o:p></o:p></font></span></p> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Primarily I was looking at a way of speeding up zfs send-recv.<o:p></o:p></font></span></p> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d"><o:p> </o:p></font></span></p> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Guess it’s a no go on a single HCA…<o:p></o:p></font></span></p> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d"><o:p> </o:p></font></span></p> <p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Johan Kragsterman [mailto:johan.kragsterman@capvert.se] <br> <b>Sent:</b> Monday, November 03, 2014 12:41 AM<br> <b>To:</b> Rune Tipsmark<br> <b>Cc:</b> omnios-discuss@lists.omniti.com<br> <b>Subject:</b> Ang: RE: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<o:p></o:p></span></p> <p class="MsoNormal"><o:p> </o:p></p> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> Hej!<br> <br> <br> If you want to mirror a local volume with a remote LU, I don't think it is a good idea. Asking for trubble...<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">I understand you want to get an HA solution. But I believe HA solutions needs other strategies than that.<br> <br> <br> Best regards from/Med vänliga hälsningar från<br> <br> Johan Kragsterman<o:p></o:p></span></p> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> Capvert<o:p></o:p></span></p> </div> </div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> <br> </span><span lang="DA" style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099">-----Rune Tipsmark <</font></span><span style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></font></span><span lang="DA" style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099">> skrev: -----</font></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> <div> <div style="border:none;border-left:solid black 1.5pt;padding:0in 0in 0in 4.0pt"> <p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">Till: Johan Kragsterman <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:johan.kragsterman@capvert.se"><span lang="DA">johan.kragsterman@capvert.se</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Från: Rune Tipsmark <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Datum: 2014-11-03 08:37<br> Kopia: "</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:omnios-discuss@lists.omniti.com"><span lang="DA">omnios-discuss@lists.omniti.com</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">" <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:omnios-discuss@lists.omniti.com"><span lang="DA">omnios-discuss@lists.omniti.com</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Ärende: RE: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<o:p></o:p></span></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Tjenare,</font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d"> </font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Maybe its not possible, would explain why I cannot see the shared lun on the other system. </font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">I don’t have any spare PCI-E slots on either, all filled with IODrives etc.</font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d"> </font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Br,</font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Rune</font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d"> </font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>] <br> <b>Sent:</b> Sunday, November 02, 2014 11:24 PM<br> <b>To:</b> Rune Tipsmark<br> <b>Cc:</b> <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br> <b>Subject:</b> Ang: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> Hej!<br> <br> <br> Hmm, how about the target/initiiator configuration of the HCA's?</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> </span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> When I think about it, I have never done this that you're trying: To use SRP both as target and initiator on the same system. I don't even know if it is possible. Since an HCA is entirely(with all ports) a target, you need to set one HCA as a target, and one HCA as an initiator. I've done it on fibre channel, but never on IB/SRP.<br> <br> <br> Best regards from/Med vänliga hälsningar från<br> <br> Johan Kragsterman</span><o:p></o:p></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> Capvert</span><o:p></o:p></p> </div> </div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> <br> </span><span lang="DA" style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099">-----Rune Tipsmark <</font></span><span style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></font></span><span lang="DA" style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099">> skrev: -----</font></span><o:p></o:p></p> <div> <div style="border:none;border-left:solid black 1.5pt;padding:0in 0in 0in 4.0pt"> <p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">Till: Johan Kragsterman <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:johan.kragsterman@capvert.se"><span lang="DA">johan.kragsterman@capvert.se</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Från: Rune Tipsmark <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Datum: 2014-11-03 08:09<br> Kopia: "</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:omnios-discuss@lists.omniti.com"><span lang="DA">omnios-discuss@lists.omniti.com</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">" <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:omnios-discuss@lists.omniti.com"><span lang="DA">omnios-discuss@lists.omniti.com</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Ärende: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol</span><o:p></o:p></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">connectX2 and drivers are loaded, both OmniOS servers have LUNs I can access from both ESX and Windows… just the conection between them that I cant figure out.</font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Br,</font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Rune</font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d"> </font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>] <br> <b>Sent:</b> Sunday, November 02, 2014 10:49 PM<br> <b>To:</b> Rune Tipsmark<br> <b>Cc:</b> <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br> <b>Subject:</b> Ang: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> Hej!</span><o:p></o:p></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">I can see I forgot the list in previous mail...</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> </span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Anyway, are you using connectx adapters?</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> </span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Have you checked if the driver is loaded in the host system?</span><o:p></o:p></p> </div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> <br> Best regards from/Med vänliga hälsningar från<br> <br> Johan Kragsterman</span><o:p></o:p></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> Capvert</span><o:p></o:p></p> </div> </div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> <br> </span><span lang="DA" style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099">-----Rune Tipsmark <</font></span><span style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></font></span><span lang="DA" style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099">> skrev: -----</font></span><o:p></o:p></p> <div> <div style="border:none;border-left:solid black 1.5pt;padding:0in 0in 0in 4.0pt"> <p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">Till: Johan Kragsterman <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:johan.kragsterman@capvert.se"><span lang="DA">johan.kragsterman@capvert.se</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Från: Rune Tipsmark <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Datum: 2014-11-03 00:07<br> Ärende: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol</span><o:p></o:p></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-family:"Courier New"">It doesn't show up in stmfadm list-target.... only my other servers such as ESX and Windows show up...<br> <br> Subnet Manager is OpenSM running on the ESX hosts, has been working OK ever since I installed then, prior to that I had a couple of Cisco(Topspin) DDR Switches with inbuilt SM.<br> <br> Any other idea? All things I configured so far on both ZFS boxes,windows and esx has worked as expected... just this darn thing that's not doing what its supposed to...<br> <br> It should be enough to just add the HCA from ZFS10 onto the host-group on ZFS00 for ZFS10 to see a LUN on ZFS00 granted that there is a view configured on ZFS00 as well... or am I missing a step? <br> <br> Br<br> Rune<br> <br> -----Original Message-----<br> From: Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>] <br> Sent: Sunday, November 02, 2014 11:58 AM<br> To: Rune Tipsmark<br> Subject: Ang: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> <br> -----Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>> skrev: -----<br> Till: Johan Kragsterman <<a href="mailto:johan.kragsterman@capvert.se">johan.kragsterman@capvert.se</a>><br> Från: Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>><br> Datum: 2014-11-02 20:05<br> Kopia: David Bomba <<a href="mailto:turbo124@gmail.com">turbo124@gmail.com</a>>, "<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>" <<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>><br> Ärende: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> I know, but how do I initiate a session from ZFS10?<br> <br> <br> <br> If a session doesn't show up in "stmfadm list-targte -v", then you got something wrong in the IB fabric, if the view is right. Do you use a switch? Where do you have your IB Storage Manager?<br> <br> <br> <br> <br> <br> <br> <br> <br> Br,<br> <br> Rune<br> <br> <br> <br> From: Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>]<br> Sent: Sunday, November 02, 2014 10:33 AM<br> To: Rune Tipsmark<br> Cc: David Bomba; <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br> Subject: Ang: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> <br> <br> <br> <br> <br> -----Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>> skrev: -----<br> <br> Till: Johan Kragsterman <<a href="mailto:johan.kragsterman@capvert.se">johan.kragsterman@capvert.se</a>><br> Från: Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>><br> Datum: 2014-11-02 19:11<br> Kopia: David Bomba <<a href="mailto:turbo124@gmail.com">turbo124@gmail.com</a>>, "<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>" <<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>><br> Ärende: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> Hi Johan<br> <br> Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine..<br> <br> What I can't figure out is how to share a LUN with the other zfs box... see pasted info below...<br> <br> ZFS00: (the box I want to receive my zfs snapshot) The below are all ESX servers, cannot see the other ZFS box<br> <br> root@zfs00:/pool03# stmfadm list-target -v<br> Target: eui.0002C90300095E7C<br> Operational Status: Online<br> Provider Name : srpt<br> Alias : -<br> Protocol : SRP<br> Sessions : 8<br> Initiator: eui.0002C903000F397C<br> Alias: 8102c90300095e7e:0002c903000f397c<br> Logged in since: Sun Nov 2 02:09:41 2014<br> Initiator: eui.0002C903000F397B<br> Alias: 8102c90300095e7d:0002c903000f397b<br> Logged in since: Sun Nov 2 02:09:40 2014<br> Initiator: eui.0002C903000D3D04<br> Alias: 8102c90300095e7e:0002c903000d3d04<br> Logged in since: Sat Nov 1 21:14:47 2014<br> Initiator: eui.0002C90300104F47<br> Alias: 8102c90300095e7d:0002c90300104f47<br> Logged in since: Sat Nov 1 21:12:54 2014<br> Initiator: eui.0002C903000D3D03<br> Alias: 8102c90300095e7d:0002c903000d3d03<br> Logged in since: Sat Nov 1 21:12:32 2014<br> Initiator: eui.0002C90300104F48<br> Alias: 8102c90300095e7e:0002c90300104f48<br> Logged in since: Sat Nov 1 21:10:45 2014<br> Initiator: eui.0002C903000A48FA<br> Alias: 8102c90300095e7e:0002c903000a48fa<br> Logged in since: Sat Nov 1 21:10:40 2014<br> Initiator: eui.0002C903000D3CA0<br> Alias: 8102c90300095e7e:0002c903000d3ca0<br> Logged in since: Sat Nov 1 21:10:39 2014<br> Target: iqn.2010-09.org.napp-it:1394106801<br> Operational Status: Online<br> Provider Name : iscsit<br> Alias : 03.06.2014<br> Protocol : iSCSI<br> Sessions : 0<br> <br> root@zfs00:/pool03# stmfadm list-lu -v<br> LU Name: 600144F007780B7F00005455EDD50002<br> Operational Status: Online<br> Provider Name : sbd<br> Alias : /pool03/LU11<br> View Entry Count : 1<br> Data File : /pool03/LU11<br> Meta File : not set<br> Size : 2199023255552<br> Block Size : 512<br> Management URL : not set<br> Vendor ID : SUN<br> Product ID : COMSTAR<br> Serial Num : not set<br> Write Protect : Disabled<br> Writeback Cache : Disabled<br> Access State : Active<br> <br> root@zfs00:/pool03# stmfadm list-hg -v<br> Host Group: ESX<br> Host Group: Windows<br> Host Group: ESX-iSER<br> Host Group: OmniOS<br> Member: eui.0002C903000923E6 <<--- The other ZFS box<br> Member: iqn.2010-09.org.napp-it:1402013225<br> Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d<br> <br> root@zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002 View Entry: 0<br> Host group : OmniOS<br> Target group : All<br> LUN : 0<br> <br> ZFS10: (the sending box where I want to see the LUN from ZFS00) No disk show up from ZFS00...<br> <br> <br> <br> <br> <br> <br> <br> Hi!<br> <br> <br> <br> <br> <br> You got eui.0002C903000923E6 in host group OmniOS, but you don't have a session from that eui to the target.<br> <br> <br> <br> Rgrds Johan<br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> root@zfs10:/root# format<br> Searching for disks...done<br> <br> <br> AVAILABLE DISK SELECTIONS:<br> 0. c4t5000C50055FC9533d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50055fc9533<br> 1. c4t5000C50055FE6A63d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50055fe6a63<br> 2. c4t5000C500625B7EA7d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625b7ea7<br> 3. c4t5000C500625B86E3d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625b86e3<br> 4. c4t5000C500625B886Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625b886f<br> 5. c4t5000C500625B8137d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625b8137<br> 6. c4t5000C500625B8427d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625b8427<br> 7. c4t5000C500625BB773d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625bb773<br> 8. c4t5000C500625BC2C3d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625bc2c3<br> 9. c4t5000C500625BD3EBd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625bd3eb<br> 10. c4t5000C50057085A6Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50057085a6b<br> 11. c4t5000C50057086B67d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50057086b67<br> 12. c4t5000C50062878C0Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50062878c0b<br> 13. c4t5000C50062878C43d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50062878c43<br> 14. c4t5000C500570858EFd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500570858ef<br> 15. c4t5000C500570870D3d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500570870d3<br> 16. c4t5000C5005708351Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c5005708351b<br> 17. c4t5000C5005708296Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c5005708296f<br> 18. c4t5000C50057089753d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50057089753<br> 19. c4t5000C50057086307d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50057086307<br> 20. c4t5000C50062879687d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50062879687<br> 21. c4t5000C50062879707d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50062879707<br> 22. c4t5000C50062879723d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50062879723<br> 23. c4t5000C50062879787d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50062879787<br> 24. c7t0d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56><br> /pci@0,0/pci15d9,704@1f,2/disk@0,0<br> 25. c7t1d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56><br> /pci@0,0/pci15d9,704@1f,2/disk@1,0<br> 26. c7t4d0 <ATA-InnoLite SATADOM-19-14.91GB><br> /pci@0,0/pci15d9,704@1f,2/disk@4,0<br> 27. c7t5d0 <ATA-InnoLite SATADOM-19-14.91GB><br> /pci@0,0/pci15d9,704@1f,2/disk@5,0<br> 28. c10d0 <Unknown-Unknown-0001-149.00GB><br> /pci@0,0/pci8086,3c08@3/pci103c,178b@0<br> 29. c11d0 <Unknown-Unknown-0001-149.00GB><br> /pci@0,0/pci8086,3c0a@3,2/pci103c,178b@0<br> 30. c12d0 <Unknown-Unknown-0001-298.02GB><br> /pci@79,0/pci8086,3c04@2/pci10b5,8616@0/pci10b5,8616@5/pci103c,178e@0<br> 31. c13d0 <Unknown-Unknown-0001-298.02GB><br> /pci@79,0/pci8086,3c04@2/pci10b5,8616@0/pci10b5,8616@6/pci103c,178e@0<br> 32. c14d0 <Unknown-Unknown-0001-298.02GB><br> /pci@79,0/pci8086,3c08@3/pci10b5,8616@0/pci10b5,8616@5/pci103c,178e@0<br> 33. c15d0 <Unknown-Unknown-0001-298.02GB><br> /pci@79,0/pci8086,3c08@3/pci10b5,8616@0/pci10b5,8616@6/pci103c,178e@0<br> Specify disk (enter its number):<br> <br> <br> <br> -----Original Message-----<br> From: Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>]<br> Sent: Sunday, November 02, 2014 9:56 AM<br> To: Rune Tipsmark<br> Cc: David Bomba; <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br> Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> <br> -----"OmniOS-discuss" <<a href="mailto:omnios-discuss-bounces@lists.omniti.com">omnios-discuss-bounces@lists.omniti.com</a>> skrev: -----<br> Till: David Bomba <<a href="mailto:turbo124@gmail.com">turbo124@gmail.com</a>><br> Från: Rune Tipsmark <br> Sänt av: "OmniOS-discuss" <br> Datum: 2014-11-02 07:55<br> Kopia: "<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>" <<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>><br> Ärende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> Sounds sensible, how do I do that?<br> <br> I tried creating a view for a thin lu with my other zfs box, but how do I detect it?<br> <br> <br> <br> <br> <br> First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application.<br> <br> Did you create SRP target service using COMSTAR?<br> <br> # svcadm enable -r ibsrp/target<br> <br> What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v"<br> <br> Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link?<br> <br> If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA.<br> <br> <br> Rgrds Johan<br> <br> <br> <br> <br> <br> <br> I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article <a href="http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html"> http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html</a><br> <br> <br> <br> Br,<br> <br> Rune<br> <br> <br> <br> From: David Bomba [<a href="mailto:turbo124@gmail.com">mailto:turbo124@gmail.com</a>] <br> Sent: Saturday, November 01, 2014 6:01 PM<br> To: Rune Tipsmark<br> Cc: <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br> Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> <br> <br> I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential. <br> <br> <br> <br> <br> <br> On 2 Nov 2014, at 11:45 am, Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>> wrote:<br> <br> <br> <br> Hi all,<br> <br> <br> <br> Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.<br> <br> <br> <br> If not, is there a way I can aggregate say 8 or 16 IPoIB partitions and push throughput to a more reasonable speed…<br> <br> <br> <br> Br,<br> <br> Rune<br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> _______________________________________________<br> OmniOS-discuss mailing list<br> <a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br> <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><br> <br> <br> <br> _______________________________________________<br> OmniOS-discuss mailing list<br> <a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br> <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><br> <br> <br> </span><o:p></o:p></p> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div></div><div></div></font>
`