`<div dir="ltr"><div><div><div><div>Thank you Joerg,<br><br></div>I've downloaded the package and will try it tomorrow.<br><br></div>The only thing I can add at this point is that upon review of my testing, I may have performed my "pkg -u" between the initial quad-gig performance test and installing the 10G NIC. So this may be a new problem introduced in the latest updates.<br><br></div>Those of you who are running 10G and have not upgraded to the latest kernel, etc, might want to do some additional testing before running the update.<br><br></div>-Warren V<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 23, 2015 at 1:15 AM, Joerg Goltermann <span dir="ltr"><<a href="mailto:jg@osn.de" target="_blank">jg@osn.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I remember there was a problem with the flow control settings in the ixgbe<br>
driver, so I updated it a long time ago for our internal servers to 2.5.8.<br>
Last weekend I integrated the latest changes from the FreeBSD driver to bring<br>
the illumos ixgbe to 2.5.25 but I had no time to test it, so it's completely<br>
untested!<br>
<br>
<br>
If you would like to give the latest driver a try you can fetch the<br>
kernel modules from <a href="https://cloud.osn.de/index.php/s/Fb4so9RsNnXA7r9" target="_blank">https://cloud.osn.de/index.<u></u>php/s/Fb4so9RsNnXA7r9</a><br>
<br>
Clone your boot environment, place the modules in the new environment<br>
and update the boot-archive of the new BE.<br>
<br>
- Joerg<div class="HOEnZb"><div class="h5"><br>
<br>
<br>
<br>
<br>
On 23.02.2015 02:54, W Verb wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
By the way, to those of you who have working setups: please send me<br>
your pool/volume settings, interface linkprops, and any kernel tuning<br>
parameters you may have set.<br>
<br>
Thanks,<br>
Warren V<br>
<br>
On Sat, Feb 21, 2015 at 7:59 AM, Schweiss, Chip <<a href="mailto:chip@innovates.com" target="_blank">chip@innovates.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I can't say I totally agree with your performance assessment. I run Intel<br>
X520 in all my OmniOS boxes.<br>
<br>
Here is a capture of nfssvrtop I made while running many storage vMotions<br>
between two OmniOS boxes hosting NFS datastores. This is a 10 host VMware<br>
cluster. Both OmniOS boxes are dual 10G connected with copper twin-ax to<br>
the in rack Nexus 5010.<br>
<br>
VMware does 100% sync writes, I use ZeusRAM SSDs for log devices.<br>
<br>
-Chip<br>
<br>
2014 Apr 24 08:05:51, load: 12.64, read: 17330243 KB, swrite: 15985 KB,<br>
awrite: 1875455 KB<br>
<br>
Ver Client NFSOPS Reads SWrites AWrites Commits Rd_bw<br>
SWr_bw AWr_bw Rd_t SWr_t AWr_t Com_t Align%<br>
<br>
4 10.28.17.105 0 0 0 0 0 0<br>
0 0 0 0 0 0 0<br>
<br>
4 10.28.17.215 0 0 0 0 0 0<br>
0 0 0 0 0 0 0<br>
<br>
4 10.28.17.213 0 0 0 0 0 0<br>
0 0 0 0 0 0 0<br>
<br>
4 10.28.16.151 0 0 0 0 0 0<br>
0 0 0 0 0 0 0<br>
<br>
4 all 1 0 0 0 0 0<br>
0 0 0 0 0 0 0<br>
<br>
3 10.28.16.175 3 0 3 0 0 1<br>
11 0 4806 48 0 0 85<br>
<br>
3 10.28.16.183 6 0 6 0 0 3<br>
162 0 549 124 0 0 73<br>
<br>
3 10.28.16.180 11 0 10 0 0 3<br>
27 0 776 89 0 0 67<br>
<br>
3 10.28.16.176 28 2 26 0 0 10<br>
405 0 2572 198 0 0 100<br>
<br>
3 10.28.16.178 4606 4602 4 0 0 294534<br>
3 0 723 49 0 0 99<br>
<br>
3 10.28.16.179 4905 4879 26 0 0 312208<br>
311 0 735 271 0 0 99<br>
<br>
3 10.28.16.181 5515 5502 13 0 0 352107<br>
77 0 89 87 0 0 99<br>
<br>
3 10.28.16.184 12095 12059 10 0 0 763014<br>
39 0 249 147 0 0 99<br>
<br>
3 10.28.58.1 15401 6040 116 6354 53 191605<br>
474 202346 192 96 144 83 99<br>
<br>
3 all <a href="tel:42574%20%20%2033086" value="+14257433086" target="_blank">42574 33086</a> 217 6354 53 1913488<br>
1582 202300 348 138 153 105 99<br>
<br>
<br>
<br>
<br>
<br>
On Fri, Feb 20, 2015 at 11:46 PM, W Verb <<a href="mailto:wverb73@gmail.com" target="_blank">wverb73@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Hello All,<br>
<br>
Thank you for your replies.<br>
I tried a few things, and found the following:<br>
<br>
1: Disabling hyperthreading support in the BIOS drops performance overall<br>
by a factor of 4.<br>
2: Disabling VT support also seems to have some effect, although it<br>
appears to be minor. But this has the amusing side effect of fixing the<br>
hangs I've been experiencing with fast reboot. Probably by disabling kvm.<br>
3: The performance tests are a bit tricky to quantify because of caching<br>
effects. In fact, I'm not entirely sure what is happening here. It's just<br>
best to describe what I'm seeing:<br>
<br>
The commands I'm using to test are<br>
dd if=/dev/zero of=./test.dd bs=2M count=5000<br>
dd of=/dev/null if=./test.dd bs=2M count=5000<br>
The host vm is running Centos 6.6, and has the latest vmtools installed.<br>
There is a host cache on an SSD local to the host that is also in place.<br>
Disabling the host cache didn't immediately have an effect as far as I could<br>
see.<br>
<br>
The host MTU set to 3000 on all iSCSI interfaces for all tests.<br>
<br>
Test 1: Right after reboot, with an ixgbe MTU of 9000, the write test<br>
yields an average speed over three tests of 137MB/s. The read test yields an<br>
average over three tests of 5MB/s.<br>
<br>
Test 2: After setting "ifconfig ixgbe0 mtu 3000", the write tests yield<br>
140MB/s, and the read tests yield 53MB/s. It's important to note here that<br>
if I cut the read test short at only 2-3GB, I get results upwards of<br>
350MB/s, which I assume is local cache-related distortion.<br>
<br>
Test 3: MTU of 1500. Read tests are up to 156 MB/s. Write tests yield<br>
about 142MB/s.<br>
Test 4: MTU of 1000: Read test at 182MB/s.<br>
Test 5: MTU of 900: Read test at 130 MB/s.<br>
Test 6: MTU of 1000: Read test at 160MB/s. Write tests are now<br>
consistently at about 300MB/s.<br>
Test 7: MTU of 1200: Read test at 124MB/s.<br>
Test 8: MTU of 1000: Read test at 161MB/s. Write at 261MB/s.<br>
<br>
A few final notes:<br>
L1ARC grabs about 10GB of RAM during the tests, so there's definitely some<br>
read caching going on.<br>
The write operations are easier to observe with iostat, and I'm seeing io<br>
rates that closely correlate with the network write speeds.<br>
<br>
<br>
Chris, thanks for your specific details. I'd appreciate it if you could<br>
tell me which copper NIC you tried, as well as to pass on the iSCSI tuning<br>
parameters.<br>
<br>
I've ordered an Intel EXPX9502AFXSR, which uses the 82598 chip instead of<br>
the 82599 in the X520. If I get similar results with my fiber transcievers,<br>
I'll see if I can get a hold of copper ones.<br>
<br>
But I should mention that I did indeed look at PHY/MAC error rates, and<br>
they are nil.<br>
<br>
-Warren V<br>
<br>
On Fri, Feb 20, 2015 at 7:25 PM, Chris Siebenmann <<a href="mailto:cks@cs.toronto.edu" target="_blank">cks@cs.toronto.edu</a>><br>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
After installation and configuration, I observed all kinds of bad<br>
behavior<br>
in the network traffic between the hosts and the server. All of this<br>
bad<br>
behavior is traced to the ixgbe driver on the storage server. Without<br>
going<br>
into the full troubleshooting process, here are my takeaways:<br>
</blockquote>
[...]<br>
<br>
For what it's worth, we managed to achieve much better line rates on<br>
copper 10G ixgbe hardware of various descriptions between OmniOS<br>
and CentOS 7 (I don't think we ever tested OmniOS to OmniOS). I don't<br>
believe OmniOS could do TCP at full line rate but I think we managed 700+<br>
Mbytes/sec on both transmit and receive and we got basically disk-limited<br>
speeds with iSCSI (across multiple disks on multi-disk mirrored pools,<br>
OmniOS iSCSI initiator, Linux iSCSI targets).<br>
<br>
I don't believe we did any specific kernel tuning (and in fact some of<br>
our attempts to fiddle ixgbe driver parameters blew up in our face).<br>
We did tune iSCSI connection parameters to increase various buffer<br>
sizes so that ZFS could do even large single operations in single iSCSI<br>
transactions. (More details available if people are interested.)<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
10: At the wire level, the speed problems are clearly due to pauses in<br>
response time by omnios. At 9000 byte frame sizes, I see a good number<br>
of duplicate ACKs and fast retransmits during read operations (when<br>
omnios is transmitting). But below about a 4100-byte MTU on omnios<br>
(which seems to correlate to 4096-byte iSCSI block transfers), the<br>
transmission errors fade away and we only see the transmission pause<br>
problem.<br>
</blockquote>
<br>
This is what really attracted my attention. In our OmniOS setup, our<br>
specific Intel hardware had ixgbe driver issues that could cause<br>
activity stalls during once-a-second link heartbeat checks. This<br>
obviously had an effect at the TCP and iSCSI layers. My initial message<br>
to illumos-developer sparked a potentially interesting discussion:<br>
<br>
<br>
<a href="http://www.listbox.com/member/archive/182179/2014/10/sort/time_rev/page/16/entry/6:405/20141003125035:6357079A-4B1D-11E4-A39C-D534381BA44D/" target="_blank">http://www.listbox.com/member/<u></u>archive/182179/2014/10/sort/<u></u>time_rev/page/16/entry/6:405/<u></u>20141003125035:6357079A-4B1D-<u></u>11E4-A39C-D534381BA44D/</a><br>
<br>
If you think this is a possibility in your setup, I've put the DTrace<br>
script I used to hunt for this up on the web:<br>
<br>
<a href="http://www.cs.toronto.edu/~cks/src/omnios-ixgbe/ixgbe_delay.d" target="_blank">http://www.cs.toronto.edu/~<u></u>cks/src/omnios-ixgbe/ixgbe_<u></u>delay.d</a><br>
<br>
This isn't the only potential source of driver stalls by any means, it's<br>
just the one I found. You may also want to look at lockstat in general,<br>
as information it reported is what led us to look specifically at the<br>
ixgbe code here.<br>
<br>
(If you suspect kernel/driver issues, lockstat combined with kernel<br>
source is a really excellent resource.)<br>
<br>
- cks<br>
</blockquote>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
OmniOS-discuss mailing list<br>
<a href="mailto:OmniOS-discuss@lists.omniti.com" target="_blank">OmniOS-discuss@lists.omniti.<u></u>com</a><br>
<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">http://lists.omniti.com/<u></u>mailman/listinfo/omnios-<u></u>discuss</a><br>
<br>
</blockquote>
<br>
</blockquote>
______________________________<u></u>_________________<br>
OmniOS-discuss mailing list<br>
<a href="mailto:OmniOS-discuss@lists.omniti.com" target="_blank">OmniOS-discuss@lists.omniti.<u></u>com</a><br>
<a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" target="_blank">http://lists.omniti.com/<u></u>mailman/listinfo/omnios-<u></u>discuss</a><br>
<br>
</blockquote>
<br></div></div><span class="HOEnZb"><font color="#888888">
-- <br>
OSN Online Service Nuernberg GmbH, Bucher Str. 78, 90408 Nuernberg<br>
Tel: <a href="tel:%2B49%20911%2039905-0" value="+49911399050" target="_blank">+49 911 39905-0</a> - Fax: <a href="tel:%2B49%20911%2039905-55" value="+499113990555" target="_blank">+49 911 39905-55</a> - <a href="http://www.osn.de" target="_blank">http://www.osn.de</a><br>
HRB 15022 Nuernberg, USt-Id: DE189301263, GF: Joerg Goltermann<br>
</font></span></blockquote></div><br></div>
`