# [OmniOS-discuss] Emulex 10Gbit adapter speeds

Ales ales at ovca.be
Fri Sep 12 05:52:06 UTC 2014

- Previous message: [OmniOS-discuss] Fibre Target problems
- Next message: [OmniOS-discuss] sed on bloody crashes when compiling cdrtools
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

    Hey guys (and girls),

    I've got a few servers with and Emulex OneConnect 10Gb (I think it's
    OCe11102 chipset based), running on OmniOS (08 and 010) an Linux (OpenSuSE).

    Out of the box the performance is pretty shitty but once you tweak the
    windows sizes and buffers it takes off.

    But here's what's bothering me, I can reach 9,2Gbit/s on linux but only
    around 6.3Gbit/s on OmniOS. The machines are identical dual xeons
    E5-2630 so the CPU power isn't an issue (loaded with SSDs).

    I'm testing my through put with iperf and netperf both show similar speeds.

    Below are my tweaks to the netstack an I'd really like your input on
    10Gbit tweaks to reach the full potential.

    ipadm set-prop -p recv_buf=87380  tcp
    ipadm set-prop -p send_buf=87380  tcp
    ipadm set-prop -p send_buf=87380  udp
    ipadm set-prop -p recv_buf=87380  udp

    (both enabled by default but it never hurts)

    ndd -set /dev/tcp tcp_wscale_always 1
    ndd -set /dev/tcp tcp_tstamp_if_wscale 1

    ndd -set /dev/tcp tcp_recv_hiwat 1048576
    ndd -set /dev/tcp tcp_xmit_hiwat 1048576
    ndd -set /dev/udp udp_max_buf 16777216
    ndd -set /dev/tcp tcp_max_buf 16777216
    ndd -set /dev/tcp tcp_cwnd_max 8388608
    ndd -set /dev/tcp tcp_time_wait_interval 10000
    ndd -set /dev/tcp tcp_conn_req_max_q 8000
    ndd -set /dev/tcp tcp_conn_req_max_q0 300000

    P.S.
    The MTU is 1500, I can't use jumbo frames due to support (or lack of) on
    endclients. But doesn't seem to bother linux.
    The speeds reached are TCP, UDP floats around 4,5-5Gbit/s

- Previous message: [OmniOS-discuss] Fibre Target problems
- Next message: [OmniOS-discuss] sed on bloody crashes when compiling cdrtools
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the OmniOS-discuss mailing list