`<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Cool. Sorry I've still not looked at your lockstat data. Been buried under other tasks. I will try to find some time in the next day or so. <br><br>Sent from my iPhone</div><div><br>On Mar 8, 2015, at 3:58 PM, W Verb <<a href="mailto:wverb73@gmail.com">wverb73@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr"><div><div><div><div><div>Hello,<br><br></div>I was able to perform my last round of testing last night. The tests were done with a single host, while enabling one or two 1G ports.<br><br><font face="monospace,monospace"> </font><span style="font-family:monospace,monospace">1 port (Read) 2 ports (Read)<br></span></div><span style="font-family:monospace,monospace">Baseline: 130MB/s Read 30MB/s<br></span></div><span style="font-family:monospace,monospace">Disable LRO: 90MB/s 27MB/s<br></span></div><span style="font-family:monospace,monospace">Disable LRO/LSO 88MB/s 27MB/s<br><br></span></div><span style="font-family:monospace,monospace">LRO/LSO enabled, TCP window size varies<br></span><div><span style="font-family:monospace,monospace">Default iscsid (256k max)<br><br></span></div><div><span style="font-family:monospace,monospace">64k Window 96MB/s 28MB/s<br></span></div><div><span style="font-family:monospace,monospace">32k Window 72MB/s 22MB/s<br></span></div><div><span style="font-family:monospace,monospace">16k Window 61MB/s 17MB/s<br><br></span></div><div><span style="font-family:monospace,monospace"><font face="arial,helvetica,sans-serif">I then set everything back to the default and captured exactly what happens when I start a single port transfer then enable a second port in the middle. It's pretty illustrative. The server chokes a bit, then strangely sends TDS protocol packets saying "exception occurred". I didn't know TDS had anything to do with iSCSI.<br><br></font></span></div><div><span style="font-family:monospace,monospace"><font face="arial,helvetica,sans-serif">Captures from both interfaces here:<br></font></span><span style="font-family:monospace,monospace"></span><a href="https://drive.google.com/open?id=0BwyUMjibonYQMG8zZnNWbk40Ymc&authuser=0" target="_blank">https://drive.google.com/open?id=0BwyUMjibonYQMG8zZnNWbk40Ymc&authuser=0</a><br><br></div><div>So it seems that window size isn't the limiting factor here.<br><br></div><div>I am in the middle of implementing infiniband now. I can highly recommend the Silverstorm (QLogic) 9024CU 20G 4096MTU (with latest firmware) switch for the lab. The fans run very quietly at normal temps, and they are very inexpensive ($250 on eBay). It supports ethernet out-of-band management, as well as a subnet manager web app hosted from the switch itself. Creating the serial console cable was mildly irritating.<br><br></div><div>The latest firmware can be retrieved from the QLogic site via a Google search, you won't find a link on their support frontpage.<br><br></div><div>I'll report back once I have iSER / SRP results. <br><br><br></div><div>-Warren V<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 4, 2015 at 9:14 AM, Mallory, Rob <span dir="ltr"><<a href="mailto:rmallory@qualcomm.com" target="_blank">rmallory@qualcomm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Hi Warren,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">[ …no objections here if you want to take this thread off-line to a smaller group… I wanted to post this to the larger groups for benefit of others,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">And maybe if you find success in the end you can post back to the larger groups with a summary ]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> Your recent success case going to 10GbE end to end seem to back up my theory of overload.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> I noticed yesterday a couple things: in the packet capture of the sender, it is apparent that large send offload LSO is being used (notice the size up to
64k packets). Among other things, this makes it a bit harder to tune and understand what is happening from packet captures on the hosts. It also lets the NIC “tightly pack” the outgoing packet stream without much gap between the MTU sized packets. You need
to get a 3<sup>rd</sup>-party snoop on the wire to see what the wire sees. Same thing on the ESXi server. I suspect it is using LRO or RSC, also ganging up the packets, and making it difficult to diagnose from a tcpdump on a VM.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I still stand by my original inclination. And the data you have shown also seems to back this. (smaller MTU = less drops/pauses and then the latest: equal size
pipes on send and receive make it work fine.) I was recommending that you either/both decrease the rwin on the client side (limiting it to an absurdly small, but not for this case, 17KB) or on the server.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">It makes more sense to control the window-size on the server, (to tune just for these small-bandwidth clients) and use a host-route with –sendpipe (or is it –ssthresh)
because then the server will limit (and hopefully set some timers on the LSO part of the ixgbe) the amount of in-flight so-as the last-switch hop does not drop those important initial packets on tcp-slowstart.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">So my original recommendation still stands: limit the rwin on the client to 17K, if you want to continue to use a 1GbE interface on it.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">(your BDP @ 150us * 1GbE is about 24k, I’d pick a max receive window size smaller than this to be more conservative in the case of two interfaces)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">(and yes, only 2 x 9K jumbo frames fit in that BDP, so those 3k jumbos had less loss for a good reason, 1500MTU is probably best in your case)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">And two other things to help identify/understand the situation (in the mode you had it before with the quad-1GbE in the client)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">You can turn off LSO (in ixbge.conf) and also on the ESX, and the client VM side you can turn off LRO or RSC.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Note: there was (maybe still is) a long-standing bug in the S10 and S11 TCP stack which I know of only second-hand from a reliable source,
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">It has very similar conditions that you describe here, including high bandwidth servers, low bandwidth clients, multiple hops.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">The workaround is to disable RSC on the linux clients.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">good to hear that you can configure the system end-to-end 10GbE. That’s the obvious best case if you stick eith ethernet, and you don’t have to go to extremes
like above. Note that the lossless fabric of Infiniband will completely hide these effects of TCP loss. Also, you can use much more efficient transport such as SRP (RDMA) which I think would fit really well if you can afford the additional complexity. (I’ve
done this, and it’s really not that hard on small scale).<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Cheers, Rob<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> W Verb [mailto:<a href="mailto:wverb73@gmail.com" target="_blank">wverb73@gmail.com</a>]
<br>
<b>Sent:</b> Tuesday, March 03, 2015 9:22 PM<br>
<b>To:</b> Mallory, Rob; illumos-dev<br>
<b>Cc:</b> Garrett D'Amore; Joerg Goltermann; <a href="mailto:omnios-discuss@lists.omniti.com" target="_blank">omnios-discuss@lists.omniti.com</a></span></p><div><div class="h5"><br>
<b>Subject:</b> Re: [developer] Re: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello all,<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">This is probably the last message in this thread.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I pulled the quad-gig NIC out of one of my hosts, and installed an X520. I then set a single 10G port on the server to be on the same VLAN as the host, and defined a vswitch, vmknic, etc on the host.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I set the MTU to be 9000 on both sides, then ran my tests.<u></u><u></u></p>
</div>
<p class="MsoNormal">Read: 130 MB/s.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Write: 156 MB/s.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Additionally, at higher MTUs, the NIC would periodically lock up until I performed an "ipadm disable-if -t ixgbe0" and re-enabled it. I tried your updated driver, Jeorg, but unfortunately it failed quite often.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I then disabled stmf, enabled NFS (v3 only) on the server, and shared a dataset on the zpool with "share -f nfs /ppool/testy".<br>
I then mounted the server dataset on the host via NFS, and copied my test VM from the iSCSI zvol to the NFS dataset. I also removed the binding of the 10G port on the host from the sw iscsi interface.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Running the same tests on the VM over NFSv3 yielded:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Read: 650MB/s<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Write: 306MB/s<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">This is getting within 10% of the throughput I consistently get on dd operations local on the server, so I'm pretty happy that I'm getting as good as I'm going to get until I add more drives. Additionally, I haven't experienced any NIC
hangs.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I tried varying the settings in ixgbe.conf, the MTU, and disabling LSO on the host and server, but nothing really made that much of a difference (except reducing the MTU made things about 20-30% slower).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">mpstat during both NFS and iSCSI transfers showed all processors as getting roughly the same number of interrupts, etc, although I did see a varying number of spins on reader/writer locks during the iSCSI transfers.
The NFS showed no srws at all.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Here is a pretty representative example of a 1s mpstat during an iSCSI transfer:<br>
<br>
<span style="font-family:"Courier New"">CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl set<br>
0 0 0 0 3246 2690 8739 6 772 5967 2 0 0 11 0 89 0<br>
1 0 0 0 2366 2249 7910 8 988 5563 2 302 0 9 0 91 0<br>
2 0 0 0 2455 2344 5584 5 687 5656 3 66 0 9 0 91 0<br>
3 0 0 25 248 12 6210 1 885 5679 2 0 0 9 0 91 0<br>
4 0 0 0 284 7 5450 2 861 5751 1 0 0 8 0 92 0<br>
5 0 0 0 232 3 4513 0 547 5733 3 0 0 7 0 93 0<br>
6 0 0 0 322 8 6084 1 836 6295 2 0 0 8 0 92 0<br>
7 0 0 0 3114 2848 8229 4 648 4966 2 0 0 10 0 90 0</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">So, it seems that it's COMSTAR/iSCSI that's broke as hell, not ixgbe. My apologies to anyone I may have offended with my pre-judgement.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The consequences of this performance issue are significant:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">1: Instead of being able to utilize the existing quad-port NICs I have in my hosts, I must use dual 10G cards for redundancy purposes.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2: I must build out a full 10G switching infrastructure.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">3: The network traffic is inherently less secure, as it is essentially impossible to do real security with NFSv3 (that is supported by ESXi).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">In the short run, I have already ordered some relatively cheap 20G infiniband gear that will hopefully push up the cost/performance ratio. However, I have received all sorts of advice about how painful it can
be to build and maintain infiniband, and if iSCSI over 10G ethernet is this painful, I'm not hopeful that infiniband will "just work".<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">The last option, of course, is to bail out of the Solaris derivatives and move to ZoL or ZoBSD. The drawbacks of this are:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">1: ZoL doesn't easily support booting off of mirrored USB flash drives, let alone running the root filesystem and swap on them. FreeNAS, by way of comparison, puts a 2G swap partition on each zdev, which (strangely
enough) causes it to often crash when a zdev experiences a failure under load.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2: Neither ZoL or FreeNAS have good, stable, kernel-based iSCSI implementations. FreeNAS is indeed testing istgt, but it proved unstable for my purposes in recent builds. Unfortunately, stmf hasn't proved itself any better.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">There are other minor differences, but these are the ones that brought me to OmniOS in the first place. We'll just have to wait and see how well the infiniband stuff works.<br>
<br>
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hopefully this exercise will help prevent others from going down the same rabbit-hole that I did.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">-Warren V<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Tue, Mar 3, 2015 at 3:45 PM, W Verb <<a href="mailto:wverb73@gmail.com" target="_blank">wverb73@gmail.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello Rob et al,<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Thank you for taking the time to look at this problem with me. I completely understand your inclination to look at the network as the most probable source of my issue, but I believe that this is a pretty clear-cut
case of server-side issues.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">1: I did run ping RTT tests during both read and write operations with multiple interfaces enabled, and the RTT stayed at ~.2ms regardless of whether traffic was actively being transmitted/received or not.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">2: I am not seeing the TCP window size bouncing around, and I am certainly not seeing starvation and delay in my packet captures. It is true that I do see delayed ACKs and retransmissions when I bump the MTU
to 9000 on both sides, but I stopped testing with high MTU as soon as I saw it happening because I have a good understanding of incast. All of my recent testing has been with MTUs between 1000 and 3000 bytes.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">3: When testing with MTUs between 1000 and 3000 bytes, I do not see lost packets and retransmission in captures on either the server or client side. I only see staggered transmission delays on the part of the
server.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">4: The client is consistently advertising a large window size (20k+), so the TCP throttling mechanism does not appear to play into this.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">5: As mentioned previously, layer 2 flow control is not enabled anywhere in the network, so there are no lower-level mechanisms at work.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">6: Upon checking buffer and queue sizes (and doing the appropriate research into documentation on the C3560E's buffer sizes), I do not see large numbers of frames being dropped by the switch. It does happen
at larger MTUs, but not very often (and not consistently) during transfers at 1000-3000 byte MTUs. I do not have QoS, policing, or rate-shaping enabled.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">7: Network interface stats on both the server and the ESXi client show no errors of any kind. This is via netstat on the server, and esxcli / Vsphere client on the ESXi box.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">8: When looking at captures taken simultaneously on the server and client side, the server-side transmission pauses are consistently seen and reproducible, even after multiple server rebuilds, ESXi rebuilds, vSphere reinstallations (down
to wiping the SQL db), various COMSTAR configuration variations, multiple 10G NICs with different NIC chipsets, multiple switches (I tried both a 48-port and 24-port C3560E), multiple IOS revisions (12.2 and 15.0), OmniOS versions (r151012 and previous) multiple
cables, transceivers, etc etc etc etc etc <u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
For your review, I have uploaded the actual packet captures to Google Drive:<br>
<br>
<span style="font-family:"Courier New""><a href="https://drive.google.com/file/d/0BwyUMjibonYQZS03dHJ2ZjJvTEE/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQZS03dHJ2ZjJvTEE/view?usp=sharing</a> 2 int write - ESXi vmk5<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQTlNTQ2M5bjlxZ00/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQTlNTQ2M5bjlxZ00/view?usp=sharing</a> 2 int write - ESXi vmk1<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQUEFsSVJCYXBVX3c/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQUEFsSVJCYXBVX3c/view?usp=sharing</a> 2 int read - server ixgbe0<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQT3FBbElnNFpJTzQ/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQT3FBbElnNFpJTzQ/view?usp=sharing</a> 2 int read - ESXi vmk5<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQU1hXdFRLM2cxSTA/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQU1hXdFRLM2cxSTA/view?usp=sharing</a> 2 int read - ESXi vmk1<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQNEFZSHVZdFNweDA/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQNEFZSHVZdFNweDA/view?usp=sharing</a> 1 int write - ESXi vmk1<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQM3FpTmloQm5iMGc/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQM3FpTmloQm5iMGc/view?usp=sharing</a> 1 int read - ESXi vmk1</span><u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Regards,<u></u><u></u></p>
</div>
<p class="MsoNormal">Warren V<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">On Mon, Mar 2, 2015 at 1:11 PM, Mallory, Rob <<a href="mailto:rmallory@qualcomm.com" target="_blank">rmallory@qualcomm.com</a>> wrote:<u></u><u></u></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Just an EWAG, and forgive me for not following closely, I just saw this in my inbox, and looked at
it and the screenshots for 2 minutes.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> But this looks like the typical incast problem.. see
<a href="http://www.pdl.cmu.edu/Incast/" target="_blank">http://www.pdl.cmu.edu/Incast/</a> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">where your storage servers (there are effectively two with ISCSI/MPIO if round-robin is working) have
networks which are 20:1 oversubscribed to your 1GbE host interfaces. (although one of the tcpdumps shows only one server so it may be choked out completely)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">What is your BDP? I’m guessing .150ms * 1GbE. For single-link that gets you to a MSS of 18700 or
so.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">On your 1GbE connected clients, leave MTU at 9k, set the following in sysctl.conf,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">And reboot.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">net.ipv4.tcp_rmem = 4096 8938 17876</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">If MPIO from the server is indeed round-robining properly, this will “make things fit” much better.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Note that your tcp_wmem can and should stay high, since you are not oversubscribed going from client</span><span style="font-size:11.0pt;font-family:Wingdings;color:#1f497d">à</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">server
; you only need to tweak the tcp receive window size.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I’ve not done it in quite some time, but IIRC, You can also set these from the server side with:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Route add -sendpipe 8930 or –ssthresh</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">And I think you can see the hash-table with computed BDP per client with ndd.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I would try playing with those before delving deep into potential bugs in the TCP, nic driver, zfs,
or vm.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">-Rob</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> W Verb via illumos-developer
[mailto:<a href="mailto:developer@lists.illumos.org" target="_blank">developer@lists.illumos.org</a>]
<br>
<b>Sent:</b> Monday, March 02, 2015 12:20 PM<br>
<b>To:</b> Garrett D'Amore<br>
<b>Cc:</b> Joerg Goltermann; illumos-dev; <a href="mailto:omnios-discuss@lists.omniti.com" target="_blank">
omnios-discuss@lists.omniti.com</a><br>
<b>Subject:</b> Re: [developer] Re: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello,<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">vmstat seems pretty boring. Certainly nothing going to swap.<br>
<br>
<span style="font-family:"Courier New"">root@sanbox:/root# vmstat<br>
kthr memory page disk faults cpu<br>
r b w swap free re mf pi po fr de sr po ro s0 s2 in sy cs us sy id<br>
0 0 0 34631632 30728068 175 215 0 0 0 0 963 275 4 6 140 3301 796 6681 0 1 99</span><u></u><u></u></p>
<div>
<p class="MsoNormal">Here is the "taskq_dispatch_ent" output from "lockstat -s 5 -kWP sleep 30" during the "fast" write operation.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Courier New"">-------------------------------------------------------------------------------<br>
Count indv cuml rcnt nsec Hottest Lock Caller<br>
50934 3% 79% 0.00 3437 0xffffff093145ba40 taskq_dispatch_ent<br>
<br>
nsec ------ Time Distribution ------ count Stack<br>
128 | 7 spa_taskq_dispatch_ent<br>
256 |@@ 4333 zio_taskq_dispatch<br>
512 |@@ 3863 zio_issue_async<br>
1024 |@@@@@ 9717 zio_execute<br>
2048 |@@@@@@@@@ 15904<br>
4096 |@@@@ 7595<br>
8192 |@@ 4498<br>
16384 |@ 2662<br>
32768 |@ 1886<br>
65536 | 434<br>
131072 | 34<br>
262144 | 1<br>
-------------------------------------------------------------------------------</span><br>
<br>
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">However, the truly "broken" function is a read operation:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Top lock 1st try:<br>
<span style="font-family:"Courier New"">-------------------------------------------------------------------------------<br>
Count indv cuml rcnt nsec Hottest Lock Caller<br>
474 15% 15% 0.00 7031 0xffffff093145b6f8 cv_wait<br>
<br>
nsec ------ Time Distribution ------ count Stack<br>
256 |@ 29 taskq_thread_wait<br>
512 |@@@@@@ 100 taskq_thread<br>
1024 |@@@@ 72 thread_start<br>
2048 |@@@@ 69<br>
4096 |@@@ 51<br>
8192 |@@ 47<br>
16384 |@@ 44<br>
32768 |@@ 32<br>
65536 |@ 25<br>
131072 | 5<br>
-------------------------------------------------------------------------------</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Top lock 2nd try:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Courier New"">-------------------------------------------------------------------------------<br>
Count indv cuml rcnt nsec Hottest Lock Caller<br>
174 39% 39% 0.00 103909 0xffffff0943f116a0 dmu_zfetch_find<br>
<br>
nsec ------ Time Distribution ------ count Stack<br>
2048 | 2 dmu_zfetch<br>
4096 | 3 dbuf_read<br>
8192 | 4 dmu_buf_hold_array_by_dnode<br>
16384 | 3 dmu_buf_hold_array<br>
32768 |@ 7<br>
65536 |@@ 14<br>
131072 |@@@@@@@@@@@@@@@@@@@@ 116<br>
262144 |@@@ 19<br>
524288 | 4<br>
1048576 | 2<br>
-------------------------------------------------------------------------------</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Top lock 3rd try:<br>
<br>
<span style="font-family:"Courier New"">-------------------------------------------------------------------------------<br>
Count indv cuml rcnt nsec Hottest Lock Caller<br>
283 55% 55% 0.00 94602 0xffffff0943ff5a68 dmu_zfetch_find<br>
<br>
nsec ------ Time Distribution ------ count Stack<br>
512 | 1 dmu_zfetch<br>
1024 | 1 dbuf_read<br>
2048 | 0 dmu_buf_hold_array_by_dnode<br>
4096 | 5 dmu_buf_hold_array<br>
8192 | 2<br>
16384 | 7<br>
32768 | 4<br>
65536 |@@@ 33<br>
131072 |@@@@@@@@@@@@@@@@@@@@ 198<br>
262144 |@@ 27<br>
524288 | 2<br>
1048576 | 3<br>
-------------------------------------------------------------------------------</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">As for the MTU question- setting the MTU to 9000 makes read operations grind almost to a halt at 5MB/s transfer rate.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">-Warren V<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Mon, Mar 2, 2015 at 11:30 AM, Garrett D'Amore <<a href="mailto:garrett@damore.org" target="_blank">garrett@damore.org</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Here’s a theory. You are using small (relatively) MTUs (3000 is less than the smallest ZFS block size.) So, when you go multipathing this way, might a single upper layer transaction
(ZFS block transfer request, or for that matter COMSTAR block request) get routed over different paths. This sounds like a potentially pathological condition to me.<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">What happens if you increase the MTU to 9000? Have you tried it? I’m sort of thinking that this will permit each transaction to be issued in a single IP frame, which may alleviate
certain tragic code paths. (That said, I’m not sure how aware COMSTAR is of the IP MTU. If it is ignorant, then it shouldn’t matter *that* much, since TCP should do the right thing here and a single TCP stream should stick to a single underlying NIC. But
if COMSTAR is aware of the MTU, it may do some really screwball things as it tries to break requests up into single frames.)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Your read spin really looks like only about 22 msec of wait out of a total run of 30 sec. (That’s not *great*, but neither does it sound tragic.) Your write is interesting because
that looks like it is going a wildly different path. You should be aware that the locks you see are *not* necessarily related in call order, but rather are ordered by instance count. The write code path hitting the task_thread as hard as it does is really,
really weird. Something is pounding on a taskq lock super hard. The number of taskq_dispatch_ent calls is interesting here. I’m starting to wonder if it’s something as stupid as a spin where if the taskq is “full” (max size reached), a caller just is spinning
trying to dispatch jobs to the taskq. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The taskq_dispatch_ent code is super simple, and it should be almost impossible to have contention on that lock — barring a thread spinning hard on taskq_dispatch (or taskq_dispatch_ent
as I think is happening here). Looking at the various call sites, there are places in both COMSTAR (iscsit) and in ZFS where this could be coming from. To know which, we really need to have the back trace associated. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">lockstat can give this — try giving “-s 5” to give a short backtrace from this, that will probably give us a little more info about the guilty caller. :-)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- Garrett<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">On Mar 2, 2015, at 11:07 AM, W Verb via illumos-developer <<a href="mailto:developer@lists.illumos.org" target="_blank">developer@lists.illumos.org</a>> wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hello all,<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I am not using layer 2 flow control. The switch carries line-rate 10G traffic without error.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I think I have found the issue via lockstat. The first lockstat is taken during a multipath read:<u></u><u></u></p>
</div>
<p class="MsoNormal">lockstat -kWP sleep 30<u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Adaptive mutex spin: 21331 events in 30.020 seconds (711 events/sec)<br>
<br>
Count indv cuml rcnt nsec Hottest Lock Caller<br>
-------------------------------------------------------------------------------<br>
9306 44% 44% 0.00 1557 htable_mutex+0x370 htable_release<br>
6307 23% 68% 0.00 1207 htable_mutex+0x108 htable_lookup<br>
596 7% 75% 0.00 4100 0xffffff0931705188 cv_wait<br>
349 5% 80% 0.00 4437 0xffffff0931705188 taskq_thread<br>
704 2% 82% 0.00 995 0xffffff0935de3c50 dbuf_create<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The hash table being read here I would guess is the tcp connection hash table.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">When lockstat is run during a multipath write operation, I get:<br>
<br>
Adaptive mutex spin: 1097341 events in 30.016 seconds (36558 events/sec)<br>
<br>
Count indv cuml rcnt nsec Hottest Lock Caller<br>
-------------------------------------------------------------------------------<br>
210752 28% 28% 0.00 4781 0xffffff0931705188 taskq_thread<br>
174471 22% 50% 0.00 4476 0xffffff0931705188 cv_wait<br>
127183 10% 61% 0.00 2871 0xffffff096f29b510 zio_notify_parent<br>
176066 10% 70% 0.00 1922 0xffffff0931705188 taskq_dispatch_ent<br>
105134 9% 80% 0.00 3110 0xffffff096ffdbf10 zio_remove_child<br>
67512 4% 83% 0.00 1938 0xffffff096f3db4b0 zio_add_child<br>
45736 3% 86% 0.00 2239 0xffffff0935de3c50 dbuf_destroy<br>
27781 3% 89% 0.00 3416 0xffffff0935de3c50 dbuf_create<br>
38536 2% 91% 0.00 2122 0xffffff0935de3b70 dnode_rele<br>
27841 2% 93% 0.00 2423 0xffffff0935de3b70 dnode_diduse_space<br>
19020 2% 95% 0.00 3046 0xffffff09d9e305e0 dbuf_rele<br>
14627 1% 96% 0.00 3632 dbuf_hash_table+0x4f8 dbuf_find<br>
<br>
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Writes are not performing htable lookups, while reads are.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">-Warren V<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<br>
<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">On Mon, Mar 2, 2015 at 3:14 AM, Joerg Goltermann <<a href="mailto:jg@osn.de" target="_blank">jg@osn.de</a>> wrote:<u></u><u></u></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">Hi,<br>
<br>
I would try *one* TPG which includes both interface addresses<br>
and I would double check for packet drops on the Catalyst.<br>
<br>
The 3560 supports only receive flow control which means, that<br>
a sending 10Gbit port can easily overload a 1Gbit port.<br>
Do you have flow control enabled?<br>
<br>
- Joerg<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
<br>
On 02.03.2015 09:22, W Verb via illumos-developer wrote:<u></u><u></u></p>
</div>
</div>
</div>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hello Garrett,<br>
<br>
No, no 802.3ad going on in this config.<br>
<br>
Here is a basic schematic:<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQVkVqcE5OQUJyUUU/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQVkVqcE5OQUJyUUU/view?usp=sharing</a><br>
<br>
Here is the Nexenta MPIO iSCSI Setup Document that I used as a guide:<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQbjEyUTBjN2tTNWM/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQbjEyUTBjN2tTNWM/view?usp=sharing</a><br>
<br>
Note that I am using an MTU of 3000 on both the 10G and 1G NICs. The<br>
switch is set to allow 9148-byte frames, and I'm not seeing any<br>
errors/buffer overruns on the switch.<br>
<br>
Here is a screenshot of a packet capture from a read operation on the<br>
guest OS (from it's local drive, which is actually a VMDK file on the<br>
storage server). In this example, only a single 1G ESXi kernel interface<br>
(vmk1) is bound to the software iSCSI initiator.<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQa2NYdXhpZkpkbU0/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQa2NYdXhpZkpkbU0/view?usp=sharing</a><br>
<br>
Note that there's a nice, well-behaved window sizing process taking<br>
place. The ESXi decreases the scaled window by 11 or 12 for each ACK,<br>
then bumps it back up to 512.<br>
<br>
Here is a similar screenshot of a single-interface write operation:<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQbU1RZHRnakxDSFU/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQbU1RZHRnakxDSFU/view?usp=sharing</a><br>
<br>
There are no pauses or gaps in the transmission rate in the<br>
single-interface transfers.<br>
<br>
<br>
In the next screenshots, I have enabled an additional 1G interface on<br>
the ESXi host, and bound it to the iSCSI initiator. The new interface is<br>
bound to a separate physical port, uses a different VLAN on the switch,<br>
and talks to a different 10G port on the storage server.<br>
<br>
First, let's look at a write operation on the guest OS, which happily<br>
pumps data at near-line-rate to the storage server.<br>
<br>
Here is a sequence number trace diagram. Note how the transfer has a<br>
nice, smooth increment rate over the entire transfer.<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQWHNIa0drWnNxMmM/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQWHNIa0drWnNxMmM/view?usp=sharing</a><br>
<br>
Here are screenshots from packet captures on both 1G interfaces:<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQRWhyVVQ4djNaU3c/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQRWhyVVQ4djNaU3c/view?usp=sharing</a><br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQaTVjTEtTRloyR2c/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQaTVjTEtTRloyR2c/view?usp=sharing</a><br>
<br>
Note how we again see nice, smooth window adjustment, and no gaps in<br>
transmission.<br>
<br>
<br>
But now, let's look at the problematic two-interface Read operation.<br>
First, the sequence graph:<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQTzdFVWdQMWZ6LUU/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQTzdFVWdQMWZ6LUU/view?usp=sharing</a><br>
<br>
As you can see, there are gaps and jumps in the transmission throughout<br>
the transfer.<br>
It is very illustrative to look at captures of the gaps, which are<br>
occurring on both interfaces:<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQc0VISXN6eVFwQzg/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQc0VISXN6eVFwQzg/view?usp=sharing</a><br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQVFREUHp3TGFiUU0/view?usp=sharing" target="_blank">https://drive.google.com/file/d/0BwyUMjibonYQVFREUHp3TGFiUU0/view?usp=sharing</a><br>
<br>
As you can see, there are ~.4 second pauses in transmission from the<br>
storage server, which kills the transfer rate.<br>
It's clear that the ESXi box ACKs the prior iSCSI operation to<br>
completion, then makes a new LUN request, which the storage server<br>
immediately replies to. The ESXi ACKs the response packet from the<br>
storage server, then waits...and waits....and waits... until eventually<br>
the storage server starts transmitting again.<br>
<br>
Because the pause happens while the ESXi client is waiting for a packet<br>
from the storage server, that tells me that the gaps are not an artifact<br>
of traffic being switched between both active interfaces, but are<br>
actually indicative of short hangs occurring on the server.<br>
<br>
Having a pause or two in transmission is no big deal, but in my case, it<br>
is happening constantly, and dropping my overall read transfer rate down<br>
to 20-60MB/s, which is slower than the single interface transfer rate<br>
(~90-100MB/s).<br>
<br>
Decreasing the MTU makes the pauses shorter, increasing them makes the<br>
pauses longer.<br>
<br>
Another interesting thing is that if I set the multipath io interval to<br>
3 operations instead of 1, I get better throughput. In other words, the<br>
less frequently I swap IP addresses on my iSCSI requests from the ESXi<br>
unit, the fewer pauses I see.<br>
<br>
Basically, COMSTAR seems to choke each time an iSCSI request from a new<br>
IP arrives.<br>
<br>
Because the single interface transfer is near line rate, that tells me<br>
that the storage system (mpt_sas, zfs, etc) is working fine. It's only<br>
when multiple paths are attempted that iSCSI falls on its face during reads.<br>
<br>
All of these captures were taken without a cache device being attached<br>
to the storage zpool, so this isn't looking like some kind of ZFS ARC<br>
problem. As mentioned previously, local transfers to/from the zpool are<br>
showing ~300-500 MB/s rates over long transfers (10G+).<br>
<br>
-Warren V<br>
<br>
On Sun, Mar 1, 2015 at 9:11 PM, Garrett D'Amore <<a href="mailto:garrett@damore.org" target="_blank">garrett@damore.org</a><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><mailto:<a href="mailto:garrett@damore.org" target="_blank">garrett@damore.org</a>>> wrote:<br>
<br>
I’m not sure I’ve followed properly. You have *two* interfaces.<br>
You are not trying to provision these in an aggr are you? As far as<br>
I’m aware, VMware does not support 802.3ad link aggregations. (Its<br>
possible that you can make it work with ESXi if you give the entire<br>
NIC to the guest — but I’m skeptical.) The problem is that if you<br>
try to use link aggregation, some packets (up to half!) will be<br>
lost. TCP and other protocols fare poorly in this situation.<br>
<br>
Its possible I’ve totally misunderstood what you’re trying to do, in<br>
which case I apologize.<br>
<br>
The idle thing is a red-herring — the cpu is waiting for work to do,<br>
probably because packets haven’t arrived (or where dropped by the<br>
hypervisor!) I wouldn’t read too much into that except that your<br>
network stack is in trouble. I’d look a bit more closely at the<br>
kstats for tcp — I suspect you’ll see retransmits or out of order<br>
values that are unusually high — if so this may help validate my<br>
theory above.<br>
<br>
- Garrett<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"> On Mar 1, 2015, at 9:03 PM, W Verb via illumos-developer<br>
<<a href="mailto:developer@lists.illumos.org" target="_blank">developer@lists.illumos.org</a> <mailto:<a href="mailto:developer@lists.illumos.org" target="_blank">developer@lists.illumos.org</a>>><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
wrote:<br>
<br>
Hello all,<br>
<br>
<br>
Well, I no longer blame the ixgbe driver for the problems I'm seeing.<br>
<br>
<br>
I tried Joerg's updated driver, which didn't improve the issue. So<br>
I went back to the drawing board and rebuilt the server from scratch.<br>
<br>
What I noted is that if I have only a single 1-gig physical<br>
interface active on the ESXi host, everything works as expected.<br>
As soon as I enable two interfaces, I start seeing the performance<br>
problems I've described.<br>
<br>
Response pauses from the server that I see in TCPdumps are still<br>
leading me to believe the problem is delay on the server side, so<br>
I ran a series of kernel dtraces and produced some flamegraphs.<br>
<br>
<br>
This was taken during a read operation with two active 10G<br>
interfaces on the server, with a single target being shared by two<br>
tpgs- one tpg for each 10G physical port. The host device has two<br>
1G ports enabled, with VLANs separating the active ports into<br>
10G/1G pairs. ESXi is set to multipath using both VLANS with a<br>
round-robin IO interval of 1.<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQd3ZYOGh4d2pteGs/view?usp=sharing" target="_blank">
https://drive.google.com/file/d/0BwyUMjibonYQd3ZYOGh4d2pteGs/view?usp=sharing</a><br>
<br>
<br>
This was taken during a write operation:<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQMnBtU1Q2SXM2ams/view?usp=sharing" target="_blank">
https://drive.google.com/file/d/0BwyUMjibonYQMnBtU1Q2SXM2ams/view?usp=sharing</a><br>
<br>
<br>
I then rebooted the server and disabled C-State, ACPI T-State, and<br>
general EIST (Turbo boost) functionality in the CPU.<br>
<br>
I when I attempted to boot my guest VM, the iSCSI transfer<br>
gradually ground to a halt during the boot loading process, and<br>
the guest OS never did complete its boot process.<br>
<br>
Here is a flamegraph taken while iSCSI is slowly dying:<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQM21JeFZPX3dZWTg/view?usp=sharing" target="_blank">
https://drive.google.com/file/d/0BwyUMjibonYQM21JeFZPX3dZWTg/view?usp=sharing</a><br>
<br>
<br>
I edited out cpu_idle_adaptive from the dtrace output and<br>
regenerated the slowdown graph:<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQbTVwV3NvXzlPS1E/view?usp=sharing" target="_blank">
https://drive.google.com/file/d/0BwyUMjibonYQbTVwV3NvXzlPS1E/view?usp=sharing</a><br>
<br>
<br>
I then edited cpu_idle_adaptive out of the speedy write operation<br>
and regenerated that graph:<br>
<br>
<a href="https://drive.google.com/file/d/0BwyUMjibonYQeWFYM0pCMDZ1X2s/view?usp=sharing" target="_blank">
https://drive.google.com/file/d/0BwyUMjibonYQeWFYM0pCMDZ1X2s/view?usp=sharing</a><br>
<br>
<br>
I have zero experience with interpreting flamegraphs, but the most<br>
significant difference I see between the slow read example and the<br>
fast write example is in unix`thread_start --> unix`idle. There's<br>
a good chunk of "unix`i86_mwait" in the read example that is not<br>
present in the write example at all.<br>
<br>
Disabling the l2arc cache device didn't make a difference, and I<br>
had to reenable EIST support on the CPU to get my VMs to boot.<br>
<br>
I am seeing a variety of bug reports going back to 2010 regarding<br>
excessive mwait operations, with the suggested solutions usually<br>
being to set "cpupm enable poll-mode" in power.conf. That change<br>
also had no effect on speed.<br>
<br>
-Warren V<br>
<br>
<br>
<br>
<br>
-----Original Message-----<br>
<br>
From: Chris Siebenmann [mailto:<a href="mailto:cks@cs.toronto.edu" target="_blank">cks@cs.toronto.edu</a>]<br>
<br>
Sent: Monday, February 23, 2015 8:30 AM<br>
<br>
To: W Verb<br>
<br>
Cc: <a href="mailto:omnios-discuss@lists.omniti.com" target="_blank">omnios-discuss@lists.omniti.com</a><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <mailto:<a href="mailto:omnios-discuss@lists.omniti.com" target="_blank">omnios-discuss@lists.omniti.com</a>>;
<a href="mailto:cks@cs.toronto.edu" target="_blank">cks@cs.toronto.edu</a><br>
<mailto:<a href="mailto:cks@cs.toronto.edu" target="_blank">cks@cs.toronto.edu</a>><br>
<br>
Subject: Re: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and<br>
the Greek economy<br>
<br>
<br>
> Chris, thanks for your specific details. I'd appreciate it if you<br>
<br>
> could tell me which copper NIC you tried, as well as to pass on the<br>
<br>
> iSCSI tuning parameters.<br>
<br>
<br>
Our copper NIC experience is with onboard X540-AT2 ports on<br>
SuperMicro hardware (which have the guaranteed 10-20 msec lock<br>
hold) and dual-port 82599EB TN cards (which have some sort of<br>
driver/hardware failure under load that eventually leads to<br>
2-second lock holds). I can't recommend either with the current<br>
driver; we had to revert to 1G networking in order to get stable<br>
servers.<br>
<br>
<br>
The iSCSI parameter modifications we do, across both initiators<br>
and targets, are:<br>
<br>
<br>
initialr2tno<br>
<br>
firstburstlength128k<br>
<br>
maxrecvdataseglen128k[only on Linux backends]<br>
<br>
maxxmitdataseglen128k[only on Linux backends]<br>
<br>
<br>
The OmniOS initiator doesn't need tuning for more than the first<br>
two parameters; on the Linux backends we tune up all four. My<br>
extended thoughts on these tuning parameters and why we touch them<br>
can be found<br>
<br>
here:<br>
<br>
<br>
<a href="http://utcc.utoronto.ca/~cks/space/blog/tech/UnderstandingiSCSIProtocol" target="_blank">
http://utcc.utoronto.ca/~cks/space/blog/tech/UnderstandingiSCSIProtocol</a><br>
<br>
<a href="http://utcc.utoronto.ca/~cks/space/blog/tech/LikelyISCSITuning" target="_blank">
http://utcc.utoronto.ca/~cks/space/blog/tech/LikelyISCSITuning</a><br>
<br>
<br>
The short version is that these parameters probably only make a<br>
small difference but their overall goal is to do 128KB ZFS reads<br>
and writes in single iSCSI operations (although they will be<br>
fragmented at the TCP<br>
<br>
layer) and to do iSCSI writes without a back-and-forth delay<br>
between initiator and target (that's 'initialr2t no').<br>
<br>
<br>
I think basically everyone should use InitialR2T set to no and in<br>
fact that it should be the software default. These days only<br>
unusually limited iSCSI targets should need it to be otherwise and<br>
they can change their setting for it (initiator and target must<br>
both agree to it being 'yes', so either can veto it).<br>
<br>
<br>
- cks<br>
<br>
<br>
<br>
On Mon, Feb 23, 2015 at 8:21 AM, Joerg Goltermann <<a href="mailto:jg@osn.de" target="_blank">jg@osn.de</a><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"> <mailto:<a href="mailto:jg@osn.de" target="_blank">jg@osn.de</a>>> wrote:<br>
<br>
Hi,<br>
<br>
I think your problem is caused by your link properties or your<br>
switch settings. In general the standard ixgbe seems to perform<br>
well.<br>
<br>
I had trouble after changing the default flow control settings<br>
to "bi"<br>
and this was my motivation to update the ixgbe driver a long<br>
time ago.<br>
After I have updated our systems to ixgbe 2.5.8 I never had any<br>
problems ....<br>
<br>
Make sure your switch has support for jumbo frames and you use<br>
the same mtu on all ports, otherwise the smallest will be used.<br>
<br>
What switch do you use? I can tell you nice horror stories about<br>
different vendors....<br>
<br>
- Joerg<br>
<br>
On 23.02.2015 10:31, W Verb wrote:<br>
<br>
Thank you Joerg,<br>
<br>
I've downloaded the package and will try it tomorrow.<br>
<br>
The only thing I can add at this point is that upon review<br>
of my<br>
testing, I may have performed my "pkg -u" between the<br>
initial quad-gig<br>
performance test and installing the 10G NIC. So this may<br>
be a new<br>
problem introduced in the latest updates.<br>
<br>
Those of you who are running 10G and have not upgraded to<br>
the latest<br>
kernel, etc, might want to do some additional testing<br>
before running the<br>
update.<br>
<br>
-Warren V<br>
<br>
On Mon, Feb 23, 2015 at 1:15 AM, Joerg Goltermann<br>
<<a href="mailto:jg@osn.de" target="_blank">jg@osn.de</a> <mailto:<a href="mailto:jg@osn.de" target="_blank">jg@osn.de</a>><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <mailto:<a href="mailto:jg@osn.de" target="_blank">jg@osn.de</a> <mailto:<a href="mailto:jg@osn.de" target="_blank">jg@osn.de</a>>>> wrote:<br>
<br>
Hi,<br>
<br>
I remember there was a problem with the flow control<br>
settings in the<br>
ixgbe<br>
driver, so I updated it a long time ago for our<br>
internal servers to<br>
2.5.8.<br>
Last weekend I integrated the latest changes from the<br>
FreeBSD driver<br>
to bring<br>
the illumos ixgbe to 2.5.25 but I had no time to test<br>
it, so it's<br>
completely<br>
untested!<br>
<br>
<br>
If you would like to give the latest driver a try you<br>
can fetch the<br>
kernel modules from<br>
<a href="https://cloud.osn.de/index.____php/s/Fb4so9RsNnXA7r9" target="_blank">
https://cloud.osn.de/index.____php/s/Fb4so9RsNnXA7r9</a><br>
<<a href="https://cloud.osn.de/index.__php/s/Fb4so9RsNnXA7r9" target="_blank">https://cloud.osn.de/index.__php/s/Fb4so9RsNnXA7r9</a>><br>
<<a href="https://cloud.osn.de/index.__php/s/Fb4so9RsNnXA7r9" target="_blank">https://cloud.osn.de/index.__php/s/Fb4so9RsNnXA7r9</a><br>
<<a href="https://cloud.osn.de/index.php/s/Fb4so9RsNnXA7r9" target="_blank">https://cloud.osn.de/index.php/s/Fb4so9RsNnXA7r9</a>>><br>
<br>
Clone your boot environment, place the modules in the<br>
new environment<br>
and update the boot-archive of the new BE.<br>
<br>
- Joerg<br>
<br>
<br>
<br>
<br>
<br>
On 23.02.2015 02:54, W Verb wrote:<br>
<br>
By the way, to those of you who have working<br>
setups: please send me<br>
your pool/volume settings, interface linkprops,<br>
and any kernel<br>
tuning<br>
parameters you may have set.<br>
<br>
Thanks,<br>
Warren V<br>
<br>
On Sat, Feb 21, 2015 at 7:59 AM, Schweiss, Chip<br>
<<a href="mailto:chip@innovates.com" target="_blank">chip@innovates.com</a> <mailto:<a href="mailto:chip@innovates.com" target="_blank">chip@innovates.com</a>><br>
<mailto:<a href="mailto:chip@innovates.com" target="_blank">chip@innovates.com</a> <mailto:<a href="mailto:chip@innovates.com" target="_blank">chip@innovates.com</a>>>><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
wrote:<br>
<br>
I can't say I totally agree with your performance<br>
assessment. I run Intel<br>
X520 in all my OmniOS boxes.<br>
<br>
Here is a capture of nfssvrtop I made while<br>
running many<br>
storage vMotions<br>
between two OmniOS boxes hosting NFS<br>
datastores. This is a<br>
10 host VMware<br>
cluster. Both OmniOS boxes are dual 10G<br>
connected with<br>
copper twin-ax to<br>
the in rack Nexus 5010.<br>
<br>
VMware does 100% sync writes, I use ZeusRAM<br>
SSDs for log<br>
devices.<br>
<br>
-Chip<br>
<br>
2014 Apr 24 08:05:51, load: 12.64, read:<br>
17330243 KB,<br>
swrite: 15985 KB,<br>
awrite: 1875455 KB<br>
<br>
Ver Client NFSOPS Reads<br>
SWrites AWrites<br>
Commits Rd_bw<br>
SWr_bw AWr_bw Rd_t SWr_t AWr_t<br>
Com_t Align%<br>
<br>
4 10.28.17.105 0 0<br>
0 0<br>
0 0<br>
0 0 0 0 0 0 0<br>
<br>
4 10.28.17.215 0 0<br>
0 0<br>
0 0<br>
0 0 0 0 0 0 0<br>
<br>
4 10.28.17.213 0 0<br>
0 0<br>
0 0<br>
0 0 0 0 0 0 0<br>
<br>
4 10.28.16.151 0 0<br>
0 0<br>
0 0<br>
0 0 0 0 0 0 0<br>
<br>
4 all 1 0<br>
0 0<br>
0 0<br>
0 0 0 0 0 0 0<br>
<br>
3 10.28.16.175 3 0<br>
3 0<br>
0 1<br>
11 0 4806 48 0 0 85<br>
<br>
3 10.28.16.183 6 0<br>
6 0<br>
0 3<br>
162 0 549 124 0 0<br>
73<br>
<br>
3 10.28.16.180 11 0<br>
10 0<br>
0 3<br>
27 0 776 89 0 0 67<br>
<br>
3 10.28.16.176 28 2<br>
26 0<br>
0 10<br>
405 0 2572 198 0 0<br>
100<br>
<br>
3 10.28.16.178 4606 4602<br>
4 0<br>
0 294534<br>
3 0 723 49 0 0 99<br>
<br>
3 10.28.16.179 4905 4879<br>
26 0<br>
0 312208<br>
311 0 735 271 0 0<br>
99<br>
<br>
3 10.28.16.181 5515 5502<br>
13 0<br>
0 352107<br>
77 0 89 87 0 0 99<br>
<br>
3 10.28.16.184 12095 12059<br>
10 0<br>
0 763014<br>
39 0 249 147 0 0 99<br>
<br>
3 10.28.58.1 15401 6040<br>
116 6354<br>
53 191605<br>
474 202346 192 96 144 83<br>
99<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> 3 all
<a href="tel:42574%2033086" target="_blank">42574 33086</a> <<a href="tel:42574%2033086" target="_blank">tel:42574%2033086</a>><br>
<<a href="tel:42574%20%20%2033086" target="_blank">tel:42574%20%20%2033086</a>> 217<br>
6354 53 1913488<br>
1582 202300 348 138 153 105<br>
99<br>
<br>
<br>
<br>
<br>
<br>
On Fri, Feb 20, 2015 at 11:46 PM, W Verb<br>
<<a href="mailto:wverb73@gmail.com" target="_blank">wverb73@gmail.com</a> <mailto:<a href="mailto:wverb73@gmail.com" target="_blank">wverb73@gmail.com</a>><br>
<mailto:<a href="mailto:wverb73@gmail.com" target="_blank">wverb73@gmail.com</a><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
<mailto:<a href="mailto:wverb73@gmail.com" target="_blank">wverb73@gmail.com</a>>>> wrote:<br>
<br>
<br>
Hello All,<br>
<br>
Thank you for your replies.<br>
I tried a few things, and found the following:<br>
<br>
1: Disabling hyperthreading support in the<br>
BIOS drops<br>
performance overall<br>
by a factor of 4.<br>
2: Disabling VT support also seems to have<br>
some effect,<br>
although it<br>
appears to be minor. But this has the<br>
amusing side<br>
effect of fixing the<br>
hangs I've been experiencing with fast<br>
reboot. Probably<br>
by disabling kvm.<br>
3: The performance tests are a bit tricky<br>
to quantify<br>
because of caching<br>
effects. In fact, I'm not entirely sure<br>
what is<br>
happening here. It's just<br>
best to describe what I'm seeing:<br>
<br>
The commands I'm using to test are<br>
dd if=/dev/zero of=./test.dd bs=2M count=5000<br>
dd of=/dev/null if=./test.dd bs=2M count=5000<br>
The host vm is running Centos 6.6, and has<br>
the latest<br>
vmtools installed.<br>
There is a host cache on an SSD local to<br>
the host that<br>
is also in place.<br>
Disabling the host cache didn't<br>
immediately have an<br>
effect as far as I could<br>
see.<br>
<br>
The host MTU set to 3000 on all iSCSI<br>
interfaces for all<br>
tests.<br>
<br>
Test 1: Right after reboot, with an ixgbe<br>
MTU of 9000,<br>
the write test<br>
yields an average speed over three tests<br>
of 137MB/s. The<br>
read test yields an<br>
average over three tests of 5MB/s.<br>
<br>
Test 2: After setting "ifconfig ixgbe0 mtu<br>
3000", the<br>
write tests yield<br>
140MB/s, and the read tests yield 53MB/s.<br>
It's important<br>
to note here that<br>
if I cut the read test short at only<br>
2-3GB, I get<br>
results upwards of<br>
350MB/s, which I assume is local<br>
cache-related distortion.<br>
<br>
Test 3: MTU of 1500. Read tests are up to<br>
156 MB/s.<br>
Write tests yield<br>
about 142MB/s.<br>
Test 4: MTU of 1000: Read test at 182MB/s.<br>
Test 5: MTU of 900: Read test at 130 MB/s.<br>
Test 6: MTU of 1000: Read test at 160MB/s.<br>
Write tests<br>
are now<br>
consistently at about 300MB/s.<br>
Test 7: MTU of 1200: Read test at 124MB/s.<br>
Test 8: MTU of 1000: Read test at 161MB/s.<br>
Write at 261MB/s.<br>
<br>
A few final notes:<br>
L1ARC grabs about 10GB of RAM during the<br>
tests, so<br>
there's definitely some<br>
read cachi</p></div></div></blockquote></div></div></blockquote></blockquote></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></div></div></div></div></blockquote></div></div></blockquote></div></div></div></div></div></div>...<br><br>[Message clipped] </blockquote></div><br></div>
</div></blockquote></body></html>`