`<font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"><div><br><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2">Tjenare, Rune!<br><br><br> Ahaa, are you trying to run SRPT on one port, and SRP initiator on the other port, on the same HCA? That is not supported! At least to my knowledge...it is COMSTAR hat assigns the whole HCA to act as SRP target.</font></div><div><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"><br></font></div><div><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"> If it is so, we have found the solution, finally!<br><br><br>Best regards from/Med vänliga hälsningar från<br><br>Johan Kragsterman</font><div><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"><br>Capvert</font></div></div><br><br><font color="#990099">-----Rune Tipsmark <rt@steait.net> skrev: -----</font><div class="iNotesHistory" style="padding-left:5px;"><div style="padding-right:0px;padding-left:5px;border-left:solid black 2px;">Till: Johan Kragsterman <johan.kragsterman@capvert.se><br>Från: Rune Tipsmark <rt@steait.net><br>Datum: 2014-11-03 08:37<br>Kopia: "omnios-discuss@lists.omniti.com" <omnios-discuss@lists.omniti.com><br>Ärende: RE: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br><br> <!--Notes ACF <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">--> <!--[if gte mso 9]><xml> <o:shapedefaults v:ext="edit" spidmax="1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext="edit"> <o:idmap v:ext="edit" data="1" /> </o:shapelayout></xml><![endif]--> <div class="WordSection1"> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Tjenare,<o:p></o:p></font></span></p> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d"><o:p> </o:p></font></span></p> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Maybe its not possible, would explain why I cannot see the shared lun on the other system. <o:p></o:p></font></span></p> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">I don’t have any spare PCI-E slots on either, all filled with IODrives etc.<o:p></o:p></font></span></p> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d"><o:p> </o:p></font></span></p> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Br,<o:p></o:p></font></span></p> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Rune<o:p></o:p></font></span></p> <p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d"><o:p> </o:p></font></span></p> <p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Johan Kragsterman [mailto:johan.kragsterman@capvert.se] <br> <b>Sent:</b> Sunday, November 02, 2014 11:24 PM<br> <b>To:</b> Rune Tipsmark<br> <b>Cc:</b> omnios-discuss@lists.omniti.com<br> <b>Subject:</b> Ang: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<o:p></o:p></span></p> <p class="MsoNormal"><o:p> </o:p></p> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> Hej!<br> <br> <br> Hmm, how about the target/initiiator configuration of the HCA's?<o:p></o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p> </o:p></span></p> </div> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> When I think about it, I have never done this that you're trying: To use SRP both as target and initiator on the same system. I don't even know if it is possible. Since an HCA is entirely(with all ports) a target, you need to set one HCA as a target, and one HCA as an initiator. I've done it on fibre channel, but never on IB/SRP.<br> <br> <br> Best regards from/Med vänliga hälsningar från<br> <br> Johan Kragsterman<o:p></o:p></span></p> <div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> Capvert<o:p></o:p></span></p> </div> </div> <p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> <br> </span><span lang="DA" style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099">-----Rune Tipsmark <</font></span><span style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></font></span><span lang="DA" style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099">> skrev: -----</font></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif"><o:p></o:p></span></p> <div> <div style="border:none;border-left:solid black 1.5pt;padding:0in 0in 0in 4.0pt"> <p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">Till: Johan Kragsterman <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:johan.kragsterman@capvert.se"><span lang="DA">johan.kragsterman@capvert.se</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Från: Rune Tipsmark <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Datum: 2014-11-03 08:09<br> Kopia: "</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:omnios-discuss@lists.omniti.com"><span lang="DA">omnios-discuss@lists.omniti.com</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">" <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:omnios-discuss@lists.omniti.com"><span lang="DA">omnios-discuss@lists.omniti.com</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Ärende: RE: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<o:p></o:p></span></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">connectX2 and drivers are loaded, both OmniOS servers have LUNs I can access from both ESX and Windows… just the conection between them that I cant figure out.</font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Br,</font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d">Rune</font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size: 11pt; font-family: Calibri, sans-serif;"><font color="#1f497d"> </font></span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>] <br> <b>Sent:</b> Sunday, November 02, 2014 10:49 PM<br> <b>To:</b> Rune Tipsmark<br> <b>Cc:</b> <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br> <b>Subject:</b> Ang: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol</span><o:p></o:p></p> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> Hej!</span><o:p></o:p></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">I can see I forgot the list in previous mail...</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> </span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Anyway, are you using connectx adapters?</span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"> </span><o:p></o:p></p> </div> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif">Have you checked if the driver is loaded in the host system?</span><o:p></o:p></p> </div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> <br> Best regards from/Med vänliga hälsningar från<br> <br> Johan Kragsterman</span><o:p></o:p></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> Capvert</span><o:p></o:p></p> </div> </div> <p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><br> <br> </span><span lang="DA" style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099">-----Rune Tipsmark <</font></span><span style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></font></span><span lang="DA" style="font-size: 10pt; font-family: Verdana, sans-serif;"><font color="#990099">> skrev: -----</font></span><o:p></o:p></p> <div> <div style="border:none;border-left:solid black 1.5pt;padding:0in 0in 0in 4.0pt"> <p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">Till: Johan Kragsterman <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:johan.kragsterman@capvert.se"><span lang="DA">johan.kragsterman@capvert.se</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Från: Rune Tipsmark <</span><span style="font-size:10.0pt;font-family:"Verdana",sans-serif"><a href="mailto:rt@steait.net"><span lang="DA">rt@steait.net</span></a></span><span lang="DA" style="font-size:10.0pt;font-family:"Verdana",sans-serif">><br> Datum: 2014-11-03 00:07<br> Ärende: RE: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol</span><o:p></o:p></p> <div> <p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-family:"Courier New"">It doesn't show up in stmfadm list-target.... only my other servers such as ESX and Windows show up...<br> <br> Subnet Manager is OpenSM running on the ESX hosts, has been working OK ever since I installed then, prior to that I had a couple of Cisco(Topspin) DDR Switches with inbuilt SM.<br> <br> Any other idea? All things I configured so far on both ZFS boxes,windows and esx has worked as expected... just this darn thing that's not doing what its supposed to...<br> <br> It should be enough to just add the HCA from ZFS10 onto the host-group on ZFS00 for ZFS10 to see a LUN on ZFS00 granted that there is a view configured on ZFS00 as well... or am I missing a step? <br> <br> Br<br> Rune<br> <br> -----Original Message-----<br> From: Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>] <br> Sent: Sunday, November 02, 2014 11:58 AM<br> To: Rune Tipsmark<br> Subject: Ang: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> <br> -----Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>> skrev: -----<br> Till: Johan Kragsterman <<a href="mailto:johan.kragsterman@capvert.se">johan.kragsterman@capvert.se</a>><br> Från: Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>><br> Datum: 2014-11-02 20:05<br> Kopia: David Bomba <<a href="mailto:turbo124@gmail.com">turbo124@gmail.com</a>>, "<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>" <<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>><br> Ärende: RE: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> I know, but how do I initiate a session from ZFS10?<br> <br> <br> <br> If a session doesn't show up in "stmfadm list-targte -v", then you got something wrong in the IB fabric, if the view is right. Do you use a switch? Where do you have your IB Storage Manager?<br> <br> <br> <br> <br> <br> <br> <br> <br> Br,<br> <br> Rune<br> <br> <br> <br> From: Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>]<br> Sent: Sunday, November 02, 2014 10:33 AM<br> To: Rune Tipsmark<br> Cc: David Bomba; <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br> Subject: Ang: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> <br> <br> <br> <br> <br> -----Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>> skrev: -----<br> <br> Till: Johan Kragsterman <<a href="mailto:johan.kragsterman@capvert.se">johan.kragsterman@capvert.se</a>><br> Från: Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>><br> Datum: 2014-11-02 19:11<br> Kopia: David Bomba <<a href="mailto:turbo124@gmail.com">turbo124@gmail.com</a>>, "<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>" <<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>><br> Ärende: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> Hi Johan<br> <br> Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine..<br> <br> What I can't figure out is how to share a LUN with the other zfs box... see pasted info below...<br> <br> ZFS00: (the box I want to receive my zfs snapshot) The below are all ESX servers, cannot see the other ZFS box<br> <br> root@zfs00:/pool03# stmfadm list-target -v<br> Target: eui.0002C90300095E7C<br> Operational Status: Online<br> Provider Name : srpt<br> Alias : -<br> Protocol : SRP<br> Sessions : 8<br> Initiator: eui.0002C903000F397C<br> Alias: 8102c90300095e7e:0002c903000f397c<br> Logged in since: Sun Nov 2 02:09:41 2014<br> Initiator: eui.0002C903000F397B<br> Alias: 8102c90300095e7d:0002c903000f397b<br> Logged in since: Sun Nov 2 02:09:40 2014<br> Initiator: eui.0002C903000D3D04<br> Alias: 8102c90300095e7e:0002c903000d3d04<br> Logged in since: Sat Nov 1 21:14:47 2014<br> Initiator: eui.0002C90300104F47<br> Alias: 8102c90300095e7d:0002c90300104f47<br> Logged in since: Sat Nov 1 21:12:54 2014<br> Initiator: eui.0002C903000D3D03<br> Alias: 8102c90300095e7d:0002c903000d3d03<br> Logged in since: Sat Nov 1 21:12:32 2014<br> Initiator: eui.0002C90300104F48<br> Alias: 8102c90300095e7e:0002c90300104f48<br> Logged in since: Sat Nov 1 21:10:45 2014<br> Initiator: eui.0002C903000A48FA<br> Alias: 8102c90300095e7e:0002c903000a48fa<br> Logged in since: Sat Nov 1 21:10:40 2014<br> Initiator: eui.0002C903000D3CA0<br> Alias: 8102c90300095e7e:0002c903000d3ca0<br> Logged in since: Sat Nov 1 21:10:39 2014<br> Target: iqn.2010-09.org.napp-it:1394106801<br> Operational Status: Online<br> Provider Name : iscsit<br> Alias : 03.06.2014<br> Protocol : iSCSI<br> Sessions : 0<br> <br> root@zfs00:/pool03# stmfadm list-lu -v<br> LU Name: 600144F007780B7F00005455EDD50002<br> Operational Status: Online<br> Provider Name : sbd<br> Alias : /pool03/LU11<br> View Entry Count : 1<br> Data File : /pool03/LU11<br> Meta File : not set<br> Size : 2199023255552<br> Block Size : 512<br> Management URL : not set<br> Vendor ID : SUN<br> Product ID : COMSTAR<br> Serial Num : not set<br> Write Protect : Disabled<br> Writeback Cache : Disabled<br> Access State : Active<br> <br> root@zfs00:/pool03# stmfadm list-hg -v<br> Host Group: ESX<br> Host Group: Windows<br> Host Group: ESX-iSER<br> Host Group: OmniOS<br> Member: eui.0002C903000923E6 <<--- The other ZFS box<br> Member: iqn.2010-09.org.napp-it:1402013225<br> Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d<br> <br> root@zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002 View Entry: 0<br> Host group : OmniOS<br> Target group : All<br> LUN : 0<br> <br> ZFS10: (the sending box where I want to see the LUN from ZFS00) No disk show up from ZFS00...<br> <br> <br> <br> <br> <br> <br> <br> Hi!<br> <br> <br> <br> <br> <br> You got eui.0002C903000923E6 in host group OmniOS, but you don't have a session from that eui to the target.<br> <br> <br> <br> Rgrds Johan<br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> root@zfs10:/root# format<br> Searching for disks...done<br> <br> <br> AVAILABLE DISK SELECTIONS:<br> 0. c4t5000C50055FC9533d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50055fc9533<br> 1. c4t5000C50055FE6A63d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50055fe6a63<br> 2. c4t5000C500625B7EA7d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625b7ea7<br> 3. c4t5000C500625B86E3d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625b86e3<br> 4. c4t5000C500625B886Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625b886f<br> 5. c4t5000C500625B8137d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625b8137<br> 6. c4t5000C500625B8427d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625b8427<br> 7. c4t5000C500625BB773d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625bb773<br> 8. c4t5000C500625BC2C3d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625bc2c3<br> 9. c4t5000C500625BD3EBd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500625bd3eb<br> 10. c4t5000C50057085A6Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50057085a6b<br> 11. c4t5000C50057086B67d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50057086b67<br> 12. c4t5000C50062878C0Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50062878c0b<br> 13. c4t5000C50062878C43d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50062878c43<br> 14. c4t5000C500570858EFd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500570858ef<br> 15. c4t5000C500570870D3d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c500570870d3<br> 16. c4t5000C5005708351Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c5005708351b<br> 17. c4t5000C5005708296Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c5005708296f<br> 18. c4t5000C50057089753d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50057089753<br> 19. c4t5000C50057086307d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50057086307<br> 20. c4t5000C50062879687d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50062879687<br> 21. c4t5000C50062879707d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50062879707<br> 22. c4t5000C50062879723d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50062879723<br> 23. c4t5000C50062879787d0 <SEAGATE-ST4000NM0023-0004-3.64TB><br> /scsi_vhci/disk@g5000c50062879787<br> 24. c7t0d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56><br> /pci@0,0/pci15d9,704@1f,2/disk@0,0<br> 25. c7t1d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56><br> /pci@0,0/pci15d9,704@1f,2/disk@1,0<br> 26. c7t4d0 <ATA-InnoLite SATADOM-19-14.91GB><br> /pci@0,0/pci15d9,704@1f,2/disk@4,0<br> 27. c7t5d0 <ATA-InnoLite SATADOM-19-14.91GB><br> /pci@0,0/pci15d9,704@1f,2/disk@5,0<br> 28. c10d0 <Unknown-Unknown-0001-149.00GB><br> /pci@0,0/pci8086,3c08@3/pci103c,178b@0<br> 29. c11d0 <Unknown-Unknown-0001-149.00GB><br> /pci@0,0/pci8086,3c0a@3,2/pci103c,178b@0<br> 30. c12d0 <Unknown-Unknown-0001-298.02GB><br> /pci@79,0/pci8086,3c04@2/pci10b5,8616@0/pci10b5,8616@5/pci103c,178e@0<br> 31. c13d0 <Unknown-Unknown-0001-298.02GB><br> /pci@79,0/pci8086,3c04@2/pci10b5,8616@0/pci10b5,8616@6/pci103c,178e@0<br> 32. c14d0 <Unknown-Unknown-0001-298.02GB><br> /pci@79,0/pci8086,3c08@3/pci10b5,8616@0/pci10b5,8616@5/pci103c,178e@0<br> 33. c15d0 <Unknown-Unknown-0001-298.02GB><br> /pci@79,0/pci8086,3c08@3/pci10b5,8616@0/pci10b5,8616@6/pci103c,178e@0<br> Specify disk (enter its number):<br> <br> <br> <br> -----Original Message-----<br> From: Johan Kragsterman [<a href="mailto:johan.kragsterman@capvert.se">mailto:johan.kragsterman@capvert.se</a>]<br> Sent: Sunday, November 02, 2014 9:56 AM<br> To: Rune Tipsmark<br> Cc: David Bomba; <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br> Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> <br> -----"OmniOS-discuss" <<a href="mailto:omnios-discuss-bounces@lists.omniti.com">omnios-discuss-bounces@lists.omniti.com</a>> skrev: -----<br> Till: David Bomba <<a href="mailto:turbo124@gmail.com">turbo124@gmail.com</a>><br> Från: Rune Tipsmark <br> Sänt av: "OmniOS-discuss" <br> Datum: 2014-11-02 07:55<br> Kopia: "<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>" <<a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a>><br> Ärende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> Sounds sensible, how do I do that?<br> <br> I tried creating a view for a thin lu with my other zfs box, but how do I detect it?<br> <br> <br> <br> <br> <br> First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application.<br> <br> Did you create SRP target service using COMSTAR?<br> <br> # svcadm enable -r ibsrp/target<br> <br> What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v"<br> <br> Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link?<br> <br> If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA.<br> <br> <br> Rgrds Johan<br> <br> <br> <br> <br> <br> <br> I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article <a href="http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html"> http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html</a><br> <br> <br> <br> Br,<br> <br> Rune<br> <br> <br> <br> From: David Bomba [<a href="mailto:turbo124@gmail.com">mailto:turbo124@gmail.com</a>] <br> Sent: Saturday, November 01, 2014 6:01 PM<br> To: Rune Tipsmark<br> Cc: <a href="mailto:omnios-discuss@lists.omniti.com">omnios-discuss@lists.omniti.com</a><br> Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol<br> <br> <br> <br> I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential. <br> <br> <br> <br> <br> <br> On 2 Nov 2014, at 11:45 am, Rune Tipsmark <<a href="mailto:rt@steait.net">rt@steait.net</a>> wrote:<br> <br> <br> <br> Hi all,<br> <br> <br> <br> Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.<br> <br> <br> <br> If not, is there a way I can aggregate say 8 or 16 IPoIB partitions and push throughput to a more reasonable speed…<br> <br> <br> <br> Br,<br> <br> Rune<br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> <br> _______________________________________________<br> OmniOS-discuss mailing list<br> <a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br> <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><br> <br> <br> <br> _______________________________________________<br> OmniOS-discuss mailing list<br> <a href="mailto:OmniOS-discuss@lists.omniti.com">OmniOS-discuss@lists.omniti.com</a><br> <a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><br> <br> <br> <br> </span><o:p></o:p></p> </div> </div> </div> </div> </div> </div> </div> </div></div><div></div></font>
`