`mlxcx` -> QSFP28 ================= [root@server2 (us-mobile-1) /zones/global/mlxcx]# iperf3 -c 10.100.4.2 -P 4 Connecting to host 10.100.4.2, port 5201 [ 5] local 10.100.4.1 port 58779 connected to 10.100.4.2 port 5201 [ 7] local 10.100.4.1 port 38540 connected to 10.100.4.2 port 5201 [ 9] local 10.100.4.1 port 45219 connected to 10.100.4.2 port 5201 [ 11] local 10.100.4.1 port 63555 connected to 10.100.4.2 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.01 sec 120 MBytes 997 Mbits/sec [ 7] 0.00-1.01 sec 89.6 MBytes 748 Mbits/sec [ 9] 0.00-1.01 sec 121 MBytes 1.01 Gbits/sec [ 11] 0.00-1.01 sec 124 MBytes 1.04 Gbits/sec [SUM] 0.00-1.01 sec 455 MBytes 3.79 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.01-2.00 sec 138 MBytes 1.16 Gbits/sec [ 7] 1.01-2.00 sec 108 MBytes 910 Mbits/sec [ 9] 1.01-2.00 sec 143 MBytes 1.20 Gbits/sec [ 11] 1.01-2.00 sec 144 MBytes 1.21 Gbits/sec [SUM] 1.01-2.00 sec 533 MBytes 4.48 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 189 MBytes 1.58 Gbits/sec [ 7] 2.00-3.00 sec 154 MBytes 1.29 Gbits/sec [ 9] 2.00-3.00 sec 195 MBytes 1.63 Gbits/sec [ 11] 2.00-3.00 sec 193 MBytes 1.62 Gbits/sec [SUM] 2.00-3.00 sec 731 MBytes 6.13 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 210 MBytes 1.76 Gbits/sec [ 7] 3.00-4.00 sec 170 MBytes 1.43 Gbits/sec [ 9] 3.00-4.00 sec 210 MBytes 1.77 Gbits/sec [ 11] 3.00-4.00 sec 209 MBytes 1.75 Gbits/sec [SUM] 3.00-4.00 sec 799 MBytes 6.71 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 204 MBytes 1.71 Gbits/sec [ 7] 4.00-5.00 sec 168 MBytes 1.41 Gbits/sec [ 9] 4.00-5.00 sec 204 MBytes 1.71 Gbits/sec [ 11] 4.00-5.00 sec 204 MBytes 1.71 Gbits/sec [SUM] 4.00-5.00 sec 780 MBytes 6.53 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 202 MBytes 1.71 Gbits/sec [ 7] 5.00-6.00 sec 172 MBytes 1.45 Gbits/sec [ 9] 5.00-6.00 sec 205 MBytes 1.73 Gbits/sec [ 11] 5.00-6.00 sec 204 MBytes 1.72 Gbits/sec [SUM] 5.00-6.00 sec 784 MBytes 6.60 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 217 MBytes 1.81 Gbits/sec [ 7] 6.00-7.00 sec 173 MBytes 1.45 Gbits/sec [ 9] 6.00-7.00 sec 206 MBytes 1.72 Gbits/sec [ 11] 6.00-7.00 sec 212 MBytes 1.78 Gbits/sec [SUM] 6.00-7.00 sec 808 MBytes 6.76 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 204 MBytes 1.71 Gbits/sec [ 7] 7.00-8.00 sec 166 MBytes 1.39 Gbits/sec [ 9] 7.00-8.00 sec 206 MBytes 1.72 Gbits/sec [ 11] 7.00-8.00 sec 207 MBytes 1.74 Gbits/sec [SUM] 7.00-8.00 sec 783 MBytes 6.56 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 205 MBytes 1.72 Gbits/sec [ 7] 8.00-9.00 sec 165 MBytes 1.38 Gbits/sec [ 9] 8.00-9.00 sec 205 MBytes 1.72 Gbits/sec [ 11] 8.00-9.00 sec 207 MBytes 1.74 Gbits/sec [SUM] 8.00-9.00 sec 783 MBytes 6.57 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 205 MBytes 1.72 Gbits/sec [ 7] 9.00-10.00 sec 173 MBytes 1.45 Gbits/sec [ 9] 9.00-10.00 sec 205 MBytes 1.72 Gbits/sec [ 11] 9.00-10.00 sec 204 MBytes 1.71 Gbits/sec [SUM] 9.00-10.00 sec 788 MBytes 6.60 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.85 GBytes 1.59 Gbits/sec sender [ 5] 0.00-10.02 sec 1.85 GBytes 1.58 Gbits/sec receiver [ 7] 0.00-10.00 sec 1.50 GBytes 1.29 Gbits/sec sender [ 7] 0.00-10.02 sec 1.50 GBytes 1.29 Gbits/sec receiver [ 9] 0.00-10.00 sec 1.86 GBytes 1.59 Gbits/sec sender [ 9] 0.00-10.02 sec 1.86 GBytes 1.59 Gbits/sec receiver [ 11] 0.00-10.00 sec 1.86 GBytes 1.60 Gbits/sec sender [ 11] 0.00-10.02 sec 1.86 GBytes 1.60 Gbits/sec receiver [SUM] 0.00-10.00 sec 7.07 GBytes 6.07 Gbits/sec sender [SUM] 0.00-10.02 sec 7.07 GBytes 6.06 Gbits/sec receiver iperf Done. `mlxcx` -> QSFP56 ================= [root@server2 (us-mobile-1) /zones/global/mlxcx]# iperf3 -c 10.100.2.2 -P 4 Connecting to host 10.100.2.2, port 5201 [ 5] local 10.100.2.1 port 44147 connected to 10.100.2.2 port 5201 [ 7] local 10.100.2.1 port 43150 connected to 10.100.2.2 port 5201 [ 9] local 10.100.2.1 port 61703 connected to 10.100.2.2 port 5201 [ 11] local 10.100.2.1 port 49088 connected to 10.100.2.2 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 193 MBytes 1.62 Gbits/sec [ 7] 0.00-1.00 sec 199 MBytes 1.67 Gbits/sec [ 9] 0.00-1.00 sec 153 MBytes 1.29 Gbits/sec [ 11] 0.00-1.00 sec 136 MBytes 1.14 Gbits/sec [SUM] 0.00-1.00 sec 682 MBytes 5.72 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 207 MBytes 1.73 Gbits/sec [ 7] 1.00-2.00 sec 212 MBytes 1.78 Gbits/sec [ 9] 1.00-2.00 sec 169 MBytes 1.42 Gbits/sec [ 11] 1.00-2.00 sec 138 MBytes 1.16 Gbits/sec [SUM] 1.00-2.00 sec 726 MBytes 6.09 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 201 MBytes 1.69 Gbits/sec [ 7] 2.00-3.00 sec 198 MBytes 1.66 Gbits/sec [ 9] 2.00-3.00 sec 198 MBytes 1.66 Gbits/sec [ 11] 2.00-3.00 sec 144 MBytes 1.20 Gbits/sec [SUM] 2.00-3.00 sec 741 MBytes 6.21 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 205 MBytes 1.72 Gbits/sec [ 7] 3.00-4.00 sec 205 MBytes 1.72 Gbits/sec [ 9] 3.00-4.00 sec 203 MBytes 1.70 Gbits/sec [ 11] 3.00-4.00 sec 144 MBytes 1.21 Gbits/sec [SUM] 3.00-4.00 sec 758 MBytes 6.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 210 MBytes 1.76 Gbits/sec [ 7] 4.00-5.00 sec 207 MBytes 1.74 Gbits/sec [ 9] 4.00-5.00 sec 201 MBytes 1.69 Gbits/sec [ 11] 4.00-5.00 sec 150 MBytes 1.26 Gbits/sec [SUM] 4.00-5.00 sec 768 MBytes 6.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 199 MBytes 1.67 Gbits/sec [ 7] 5.00-6.00 sec 198 MBytes 1.66 Gbits/sec [ 9] 5.00-6.00 sec 196 MBytes 1.65 Gbits/sec [ 11] 5.00-6.00 sec 144 MBytes 1.21 Gbits/sec [SUM] 5.00-6.00 sec 737 MBytes 6.19 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 204 MBytes 1.71 Gbits/sec [ 7] 6.00-7.00 sec 205 MBytes 1.72 Gbits/sec [ 9] 6.00-7.00 sec 201 MBytes 1.69 Gbits/sec [ 11] 6.00-7.00 sec 142 MBytes 1.19 Gbits/sec [SUM] 6.00-7.00 sec 752 MBytes 6.32 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 198 MBytes 1.66 Gbits/sec [ 7] 7.00-8.00 sec 200 MBytes 1.68 Gbits/sec [ 9] 7.00-8.00 sec 195 MBytes 1.64 Gbits/sec [ 11] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec [SUM] 7.00-8.00 sec 732 MBytes 6.15 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 208 MBytes 1.74 Gbits/sec [ 7] 8.00-9.00 sec 205 MBytes 1.72 Gbits/sec [ 9] 8.00-9.00 sec 207 MBytes 1.73 Gbits/sec [ 11] 8.00-9.00 sec 150 MBytes 1.26 Gbits/sec [SUM] 8.00-9.00 sec 770 MBytes 6.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 211 MBytes 1.77 Gbits/sec [ 7] 9.00-10.00 sec 211 MBytes 1.77 Gbits/sec [ 9] 9.00-10.00 sec 206 MBytes 1.73 Gbits/sec [ 11] 9.00-10.00 sec 144 MBytes 1.21 Gbits/sec [SUM] 9.00-10.00 sec 772 MBytes 6.48 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.99 GBytes 1.71 Gbits/sec sender [ 5] 0.00-10.00 sec 1.99 GBytes 1.71 Gbits/sec receiver [ 7] 0.00-10.00 sec 1.99 GBytes 1.71 Gbits/sec sender [ 7] 0.00-10.00 sec 1.99 GBytes 1.71 Gbits/sec receiver [ 9] 0.00-10.00 sec 1.88 GBytes 1.62 Gbits/sec sender [ 9] 0.00-10.00 sec 1.88 GBytes 1.62 Gbits/sec receiver [ 11] 0.00-10.00 sec 1.40 GBytes 1.20 Gbits/sec sender [ 11] 0.00-10.00 sec 1.40 GBytes 1.20 Gbits/sec receiver [SUM] 0.00-10.00 sec 7.27 GBytes 6.24 Gbits/sec sender [SUM] 0.00-10.00 sec 7.27 GBytes 6.24 Gbits/sec receiver iperf Done. `mlxcx` Lockstat on QSFP56 [root@server2 (us-mobile-1) /zones/global/mlxcx]# lockstat iperf3 -c 10.100.2.2 -P 4 Connecting to host 10.100.2.2, port 5201 [ 6] local 10.100.2.1 port 51109 connected to 10.100.2.2 port 5201 [ 8] local 10.100.2.1 port 60896 connected to 10.100.2.2 port 5201 [ 10] local 10.100.2.1 port 38689 connected to 10.100.2.2 port 5201 [ 12] local 10.100.2.1 port 57239 connected to 10.100.2.2 port 5201 [ ID] Interval Transfer Bitrate [ 6] 0.00-1.00 sec 186 MBytes 1.56 Gbits/sec [ 8] 0.00-1.00 sec 178 MBytes 1.49 Gbits/sec [ 10] 0.00-1.00 sec 165 MBytes 1.38 Gbits/sec [ 12] 0.00-1.00 sec 178 MBytes 1.49 Gbits/sec [SUM] 0.00-1.00 sec 706 MBytes 5.92 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 1.00-2.00 sec 190 MBytes 1.59 Gbits/sec [ 8] 1.00-2.00 sec 174 MBytes 1.46 Gbits/sec [ 10] 1.00-2.00 sec 158 MBytes 1.33 Gbits/sec [ 12] 1.00-2.00 sec 193 MBytes 1.62 Gbits/sec [SUM] 1.00-2.00 sec 715 MBytes 5.99 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 2.00-3.00 sec 179 MBytes 1.50 Gbits/sec [ 8] 2.00-3.00 sec 177 MBytes 1.48 Gbits/sec [ 10] 2.00-3.00 sec 159 MBytes 1.34 Gbits/sec [ 12] 2.00-3.00 sec 191 MBytes 1.60 Gbits/sec [SUM] 2.00-3.00 sec 705 MBytes 5.92 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 3.00-4.00 sec 172 MBytes 1.44 Gbits/sec [ 8] 3.00-4.00 sec 172 MBytes 1.44 Gbits/sec [ 10] 3.00-4.00 sec 157 MBytes 1.32 Gbits/sec [ 12] 3.00-4.00 sec 184 MBytes 1.54 Gbits/sec [SUM] 3.00-4.00 sec 685 MBytes 5.74 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 4.00-5.00 sec 185 MBytes 1.55 Gbits/sec [ 8] 4.00-5.00 sec 169 MBytes 1.42 Gbits/sec [ 10] 4.00-5.00 sec 161 MBytes 1.35 Gbits/sec [ 12] 4.00-5.00 sec 194 MBytes 1.63 Gbits/sec [SUM] 4.00-5.00 sec 708 MBytes 5.94 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 5.00-6.00 sec 181 MBytes 1.52 Gbits/sec [ 8] 5.00-6.00 sec 181 MBytes 1.52 Gbits/sec [ 10] 5.00-6.00 sec 168 MBytes 1.41 Gbits/sec [ 12] 5.00-6.00 sec 177 MBytes 1.49 Gbits/sec [SUM] 5.00-6.00 sec 708 MBytes 5.94 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 6.00-7.00 sec 197 MBytes 1.65 Gbits/sec [ 8] 6.00-7.00 sec 186 MBytes 1.56 Gbits/sec [ 10] 6.00-7.00 sec 171 MBytes 1.43 Gbits/sec [ 12] 6.00-7.00 sec 196 MBytes 1.64 Gbits/sec [SUM] 6.00-7.00 sec 750 MBytes 6.29 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 7.00-8.00 sec 199 MBytes 1.67 Gbits/sec [ 8] 7.00-8.00 sec 191 MBytes 1.60 Gbits/sec [ 10] 7.00-8.00 sec 179 MBytes 1.50 Gbits/sec [ 12] 7.00-8.00 sec 200 MBytes 1.68 Gbits/sec [SUM] 7.00-8.00 sec 769 MBytes 6.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 8.00-9.00 sec 170 MBytes 1.42 Gbits/sec [ 8] 8.00-9.00 sec 172 MBytes 1.44 Gbits/sec [ 10] 8.00-9.00 sec 160 MBytes 1.35 Gbits/sec [ 12] 8.00-9.00 sec 166 MBytes 1.39 Gbits/sec [SUM] 8.00-9.00 sec 668 MBytes 5.60 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 9.00-10.00 sec 185 MBytes 1.55 Gbits/sec [ 8] 9.00-10.00 sec 176 MBytes 1.48 Gbits/sec [ 10] 9.00-10.00 sec 166 MBytes 1.39 Gbits/sec [ 12] 9.00-10.00 sec 197 MBytes 1.65 Gbits/sec [SUM] 9.00-10.00 sec 724 MBytes 6.07 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 6] 0.00-10.00 sec 1.80 GBytes 1.55 Gbits/sec sender [ 6] 0.00-20.05 sec 1.80 GBytes 771 Mbits/sec receiver [ 8] 0.00-10.00 sec 1.73 GBytes 1.49 Gbits/sec sender [ 8] 0.00-20.05 sec 1.73 GBytes 743 Mbits/sec receiver [ 10] 0.00-10.00 sec 1.60 GBytes 1.38 Gbits/sec sender [ 10] 0.00-20.05 sec 1.60 GBytes 687 Mbits/sec receiver [ 12] 0.00-10.00 sec 1.83 GBytes 1.57 Gbits/sec sender [ 12] 0.00-20.05 sec 1.83 GBytes 784 Mbits/sec receiver [SUM] 0.00-10.00 sec 6.97 GBytes 5.99 Gbits/sec sender [SUM] 0.00-20.05 sec 6.97 GBytes 2.99 Gbits/sec receiver iperf Done. Adaptive mutex spin: 831580 events in 20.963 seconds (39670 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 125256 15% 15% 0.00 540 0xfffffe2cdc58eb98 mlxcx_buf_return+0x8c 119820 14% 29% 0.00 494 0xfffffe2cdc59fb58 mlxcx_buf_return+0x8c 103245 12% 42% 0.00 1203 0xfffffe2c90b24680 hook_run+0xd1 101802 12% 54% 0.00 488 0xfffffe2cdc58ed18 mlxcx_buf_return+0x8c 97869 12% 66% 0.00 504 0xfffffe2cd8bb5758 mlxcx_buf_return+0x8c 80879 10% 76% 0.00 1227 0xfffffe2c90b24680 hook_run+0x34 28582 3% 79% 0.00 833 0xfffffe2cdc58eb98 mlxcx_buf_take+0x2f 25140 3% 82% 0.00 604 0xfffffe2cdc59fb58 mlxcx_buf_take+0x2f 20262 2% 85% 0.00 630 0xfffffe2cd8bb5758 mlxcx_buf_take+0x2f 20171 2% 87% 0.00 617 0xfffffe2cdc58ed18 mlxcx_buf_take+0x2f 16087 2% 89% 0.00 4313 0xfffffe2c863ad0f8 kmem_slab_free+0x39 10266 1% 90% 0.00 465 0xfffffe2cdbac4538 mlxcx_process_cq+0x207 9399 1% 91% 0.00 402 0xfffffe2cdc5973b8 mlxcx_process_cq+0x207 9302 1% 92% 0.00 568 0xfffffe2cdc5971f8 mlxcx_process_cq+0x207 9276 1% 93% 0.00 542 0xfffffe2cd8bb1338 mlxcx_process_cq+0x207 7219 1% 94% 0.00 778 0xfffffe2c90eab950 squeue_enter+0x53 5607 1% 95% 0.00 8496 0xfffffe2c863ad0f8 kmem_slab_alloc+0x39 4839 1% 96% 0.00 447 0xfffffe2cdc5971f8 mlxcx_sq_add_buffer+0x469 4591 1% 96% 0.00 407 0xfffffe2cdbac4538 mlxcx_sq_add_buffer+0x469 3953 0% 97% 0.00 428 0xfffffe2cdc5973b8 mlxcx_sq_add_buffer+0x469 3937 0% 97% 0.00 419 0xfffffe2cd8bb1338 mlxcx_sq_add_buffer+0x469 1263 0% 97% 0.00 1661 0xfffffe2c860cc300 kmem_cache_free+0x78 1211 0% 97% 0.00 746 0xfffffe2c90eabd10 squeue_enter+0x53 1169 0% 98% 0.00 501 0xfffffe2c90eab950 squeue_enter+0x37a 1057 0% 98% 0.00 567 0xfffffe2c90eab950 squeue_drain+0x18b 1046 0% 98% 0.00 777 0xfffffe2cdef4a558 mlxcx_buf_loan+0x75 968 0% 98% 0.00 674 0xfffffe2cdef4a018 mlxcx_buf_loan+0x75 914 0% 98% 0.00 449 0xfffffe2c90eabd10 squeue_enter+0x37a 895 0% 98% 0.00 804 0xfffffe2cdef4a258 mlxcx_buf_loan+0x75 848 0% 98% 0.00 929 0xfffffe2cdef4a558 mlxcx_buf_return+0x8c 812 0% 98% 0.00 648 0xfffffe2cdfdfdc58 mlxcx_buf_loan+0x75 786 0% 98% 0.00 941 0xfffffe2cdef4a018 mlxcx_buf_return+0x8c 770 0% 99% 0.00 1006 0xfffffe2cdef4a258 mlxcx_buf_return+0x8c 683 0% 99% 0.00 499 0xfffffe2f547b4840 squeue_drain+0x255 651 0% 99% 0.00 689 0xfffffe2f547b4840 ipcl_classify_v4+0x25d 644 0% 99% 0.00 568 0xfffffe2c9a1e3040 ipcl_classify_v4+0x25d 591 0% 99% 0.00 751 0xfffffe2f30959800 ipcl_classify_v4+0x25d 580 0% 99% 0.00 561 0xfffffe2ef39bd7c0 squeue_drain+0x255 553 0% 99% 0.00 785 0xfffffe2ef39bd7c0 ipcl_classify_v4+0x25d 492 0% 99% 0.00 432 0xfffffe2f547b4840 tcp_sendmsg+0xb8 487 0% 99% 0.00 658 0xfffffe2f30959800 squeue_drain+0x255 473 0% 99% 0.00 482 0xfffffe2c90eabd10 squeue_drain+0x18b 458 0% 99% 0.00 469 0xfffffe2ef39bd7c0 tcp_sendmsg+0xb8 450 0% 99% 0.00 554 0xfffffe2f30959800 tcp_sendmsg+0xb8 322 0% 99% 0.00 543 0xfffffe2f547b4f38 tcp_output+0x63 311 0% 99% 0.00 2114 0xfffffe2d1ed957f8 cv_timedwait_sig_hires+0x2f1 287 0% 99% 0.00 439 0xfffffe2c9a1e3040 squeue_drain+0x255 223 0% 99% 0.00 496 0xfffffe2f547b4f38 tcp_sendmsg+0x83 199 0% 99% 0.00 486 0xfffffe2c9a1e3738 tcp_output+0x63 193 0% 100% 0.00 446 0xfffffe2f547b4f38 tcp_wput_data+0x401 192 0% 100% 0.00 498 0xfffffe2c860e01d0 kmem_depot_alloc+0x88 179 0% 100% 0.00 5708 0xfffffe2d1ca58bd8 taskq_thread+0x2cd 163 0% 100% 0.00 6538 0xfffffe2d1ca58bd8 taskq_dispatch_ent+0x51 157 0% 100% 0.00 11913 overlay_target_lock overlay_target_lookup_request+0x47 156 0% 100% 0.00 2188 0xfffffe2c860cc280 kmem_cache_free+0x78 154 0% 100% 0.00 436 0xfffffe2c9a1e3040 squeue_enter+0x4cd 149 0% 100% 0.00 7696 0xfffffe2d1ca58bd8 cv_wait+0x78 145 0% 100% 0.00 2627 0xfffffe2ddbb53700 cv_wait+0x78 142 0% 100% 0.00 711 0xfffffe2c9a1e3738 tcp_sendmsg+0x83 138 0% 100% 0.00 2166 0xfffffe2c860cc400 kmem_cache_free+0x78 128 0% 100% 0.00 586 0xfffffe2c9a1e3040 tcp_sendmsg+0xb8 127 0% 100% 0.00 2748 0xfffffe2ddbb59200 cv_wait+0x78 115 0% 100% 0.00 448 0xfffffe2c9a1e3738 tcp_wput_data+0x401 104 0% 100% 0.00 630 0xfffffe2f30959ef8 tcp_output+0x63 93 0% 100% 0.00 8708 0xfffffe2c8684fe88 vdev_queue_io+0x66 90 0% 100% 0.00 8945 overlay_target_lock cv_timedwait_hires+0xf0 83 0% 100% 0.00 1483 0xfffffe2d1ed957f8 pollwakeup+0x193 83 0% 100% 0.00 2615 0xfffffe2c860e02c0 kmem_cache_free+0x78 79 0% 100% 0.00 547 0xfffffe2f30959ef8 tcp_sendmsg+0x83 69 0% 100% 0.00 2401 0xfffffe2f4f10cdc0 cv_wait+0x78 62 0% 100% 0.00 402 0xfffffe2ef39bdeb8 tcp_output+0x63 60 0% 100% 0.00 574 0xfffffe2fa9bd3ec8 i_dls_head_hold+0x1f 57 0% 100% 0.00 367 0xfffffe2f547b4840 squeue_enter+0x4cd 53 0% 100% 0.00 410 0xfffffe2fa9bd3ec8 i_dls_head_rele+0x18 52 0% 100% 0.00 14556 0xfffffe2c8edfa268 cv_wait+0x78 48 0% 100% 0.00 3291 0xfffffe2d192e8800 getf_gen+0x63 45 0% 100% 0.00 421 0xfffffe2f30959ef8 tcp_wput_data+0x401 44 0% 100% 0.00 10907 0xfffffe2d1ca583f8 cv_wait+0x78 43 0% 100% 0.00 2238 0xfffffe2f4f10cdc0 mac_rx_soft_ring_process+0x53 43 0% 100% 0.00 633 0xfffffe2ef39bdeb8 tcp_sendmsg+0x83 42 0% 100% 0.00 4884 0xfffffe2d1ca583f8 taskq_thread+0x2cd 39 0% 100% 0.00 2664 0xfffffe2c863ad340 kmem_cache_alloc+0x3d 33 0% 100% 0.00 477 0xfffffe2c860cc1d0 kmem_depot_alloc+0x88 32 0% 100% 0.00 973 sqset_lock ip_squeue_random+0x22 31 0% 100% 0.00 481 0xfffffe2f30959800 squeue_enter+0x4cd 31 0% 100% 0.00 3249 0xfffffe2cd9b8d500 cv_wait+0x78 28 0% 100% 0.00 524 0xfffffe2ef39bdeb8 tcp_wput_data+0x401 26 0% 100% 0.00 368 0xfffffe2dae6b0c90 dls_accept_common+0x80 24 0% 100% 0.00 4286 0xfffffe2c8684fe88 vdev_queue_io_done+0x75 18 0% 100% 0.00 2392 0xfffffe2f4f10ab00 cv_wait+0x78 18 0% 100% 0.00 2996 0xfffffe2deb1a7240 cv_wait+0x78 18 0% 100% 0.00 3270 0xfffffe2d192e8800 releasef+0x53 18 0% 100% 0.00 7051 0xfffffe2cd9b8d500 taskq_thread+0x2cd 18 0% 100% 0.00 2636 0xfffffe2c988dc9c0 ahci_tran_start+0x75 18 0% 100% 0.00 5573 0xfffffe2c8684fe88 vdev_queue_io_done+0x30 16 0% 100% 0.00 448 0xfffffe2ef39bd7c0 squeue_enter+0x4cd 16 0% 100% 0.00 2017 0xfffffe2d190d87c0 cv_wait+0x78 15 0% 100% 0.00 689 0xfffffe2f30959800 conn_clrqfull+0x59 15 0% 100% 0.00 393 0xfffffe2c860e01d0 kmem_depot_free+0x35 15 0% 100% 0.00 527 0xfffffe2c860dc1d0 kmem_depot_alloc+0x88 14 0% 100% 0.00 72603 0xfffffe2c8e643590 gcpu_ntv_mca_poll_wrapper+0x72 14 0% 100% 0.00 2101 0xfffffe2c860e0300 kmem_cache_free+0x78 13 0% 100% 0.00 3802 0xfffffe2d1cb8d2e0 taskq_thread+0x2cd 13 0% 100% 0.00 614 0xfffffe2c863ad1d0 kmem_depot_alloc+0x88 12 0% 100% 0.00 2235 0xfffffe2f4f109c00 cv_wait+0x78 12 0% 100% 0.00 351 0xfffffe2ef39bd7c0 conn_clrqfull+0x59 12 0% 100% 0.00 4934 0xfffffe2cd9b8d500 taskq_dispatch_ent+0x51 12 0% 100% 0.00 2082 0xfffffe2c860e0400 kmem_cache_free+0x78 11 0% 100% 0.00 11264 0xfffffe2d1ca583f8 taskq_dispatch+0x174 11 0% 100% 0.00 661 0xfffffe2ca2f3c4c0 sd_buf_iodone+0x54 10 0% 100% 0.00 348 0xfffffe2f547b4840 conn_clrqfull+0x59 10 0% 100% 0.00 1046 0xfffffe2c988dc9c0 ahci_port_intr+0x62 9 0% 100% 0.00 8182 0xfffffe2d1cb8d2e0 cv_wait+0x78 9 0% 100% 0.00 1043 0xfffffe2ca2f3c4c0 sd_start_cmds+0x105 9 0% 100% 0.00 1518 0xfffffe2c868a0340 cv_timedwait_hires+0xf0 9 0% 100% 0.00 6542 0xfffffe2c8684fe88 vdev_queue_aggregate+0x43c 9 0% 100% 0.00 2812 0xfffffe2c863ad2c0 kmem_cache_alloc+0x3d 8 0% 100% 0.00 572 0xfffffe2f547e57a8 ire_route_recursive_dstonly_v4+0x68 8 0% 100% 0.00 487 0xfffffe2f547b4f38 tcp_output+0x18d 8 0% 100% 0.00 3461 0xfffffe2deb1a1740 cv_wait+0x78 8 0% 100% 0.00 1242 0xfffffe2ddbb59200 mac_rx_soft_ring_drain+0x115 8 0% 100% 0.00 3085 0xfffffe2c97dca000 metaslab_group_alloc_normal+0x1b4 7 0% 100% 0.00 1913 0xfffffe2f4f10ab00 mac_rx_soft_ring_process+0x53 7 0% 100% 0.00 759 0xfffffe2d1cc95940 metaslab_group_alloc+0x63 7 0% 100% 0.00 1751 0xfffffe2c988dc9c0 ahci_intr_ncq_events+0x59 7 0% 100% 0.00 503 0xfffffe2c8edfac88 taskq_dispatch+0x174 7 0% 100% 0.00 2586 0xfffffe2c860da2c0 kmem_cache_alloc+0x3d 7 0% 100% 0.00 366 0xfffffe2c860da1d0 kmem_depot_alloc+0x88 6 0% 100% 0.00 724 0xfffffe2ca2f3c4c0 sd_return_command+0x12f 6 0% 100% 0.00 789 0xfffffe2ca2f3c4c0 sd_core_iostart+0x40 6 0% 100% 0.00 452 0xfffffe2c860cc1d0 kmem_depot_free+0x35 6 0% 100% 0.00 752 0xfffffe2c86003020 vmem_xalloc+0x2fa 5 0% 100% 0.00 914 0xfffffe2cdfdf9c08 mlxcx_intr_n+0x10b 5 0% 100% 0.00 399 0xfffffe2cdef4a018 mlxcx_buf_take_n+0x36 5 0% 100% 0.00 534 0xfffffe2ca2f3c4c0 sd_initpkt_for_buf+0x1c9 5 0% 100% 0.00 1191 0xfffffe2c9857a000 metaslab_group_alloc_normal+0x1b4 5 0% 100% 0.00 2491 0xfffffe2c860da340 kmem_cache_alloc+0x3d 5 0% 100% 0.00 906 0xfffffe2c86020020 vmem_xfree+0x39 5 0% 100% 0.00 768 0xfffffe2c86003020 vmem_alloc+0x77 4 0% 100% 0.00 382 vph_mutex+0x4000 page_hashout+0x122 4 0% 100% 0.00 539 0xfffffe2f4f109c00 mac_rx_soft_ring_process+0x53 4 0% 100% 0.00 1007 0xfffffe2deb1a7240 mac_rx_soft_ring_drain+0x115 4 0% 100% 0.00 1479 0xfffffe2ddbb53700 mac_rx_soft_ring_drain+0x115 4 0% 100% 0.00 1575 0xfffffe2d1ca583f8 taskq_ent_alloc+0xe4 4 0% 100% 0.00 6113 0xfffffe2d1ca583f8 taskq_ent_free+0x6d 4 0% 100% 0.00 950 0xfffffe2d18030070 spa_config_enter+0x43 4 0% 100% 0.00 378 0xfffffe2cdfdfdc58 mlxcx_buf_take_n+0x36 4 0% 100% 0.00 379 0xfffffe2cdfdf9c00 mlxcx_mac_ring_rx_poll+0x34 4 0% 100% 0.00 672 0xfffffe2cdef4a558 mlxcx_buf_take_n+0x36 4 0% 100% 0.00 438 0xfffffe2cdef4a258 mlxcx_buf_take_n+0x36 4 0% 100% 0.00 592 0xfffffe2ca2f3c4c0 sdstrategy+0x5b 4 0% 100% 0.00 775 0xfffffe2ca2f28388 sata_hba_start+0xc4 4 0% 100% 0.00 989 0xfffffe2c9a1e3738 tcp_output+0x18d 4 0% 100% 0.00 1083 0xfffffe2c988dc9c0 ahci_flush_doneq+0x78 4 0% 100% 0.00 1595 0xfffffe2c919c9800 metaslab_group_alloc_normal+0x1b4 4 0% 100% 0.00 878 0xfffffe2c8e4e4300 untimeout_generic+0x65 4 0% 100% 0.00 1638 0xfffffe2c868a03c0 cv_timedwait_hires+0xf0 4 0% 100% 0.00 825 0xfffffe2c868a0340 taskq_d_thread+0x21a 4 0% 100% 0.00 2712 0xfffffe2c868a0340 taskq_bucket_dispatch+0x32 3 0% 100% 0.00 1519 0xfffffe2d1cc95940 metaslab_group_alloc_normal+0x15f 3 0% 100% 0.00 406 0xfffffe2cdfe06a08 mlxcx_mac_ring_intr_enable+0x2b 3 0% 100% 0.00 369 0xfffffe2cdfe064c8 mlxcx_mac_ring_intr_enable+0x2b 3 0% 100% 0.00 379 0xfffffe2cdfe064c0 mlxcx_mac_ring_rx_poll+0x34 3 0% 100% 0.00 474 0xfffffe2cdef3f2c0 mlxcx_mac_ring_rx_poll+0x34 3 0% 100% 0.00 1996 0xfffffe2c9bce4f40 mac_tx+0x27d 3 0% 100% 0.00 365 0xfffffe2c9857a000 metaslab_free_concrete+0xc3 3 0% 100% 0.00 2845 0xfffffe2c91377318 cpudrv_monitor+0x3a 3 0% 100% 0.00 842 0xfffffe2c8e4e46c0 callout_list_expire+0xa1 3 0% 100% 0.00 1469 0xfffffe2c8e4e4180 untimeout_generic+0x65 3 0% 100% 0.00 390 0xfffffe2c863ad1d0 kmem_depot_free+0x35 3 0% 100% 0.00 393 0xfffffe2c860dc1d0 kmem_depot_free+0x35 2 0% 100% 0.00 325 vph_mutex+0x4000 page_hashin+0x4d 2 0% 100% 0.00 1077 0xfffffe2f4f109c00 mac_rx_soft_ring_drain+0x98 2 0% 100% 0.00 809 0xfffffe2e4b7d3e70 ip_xmit+0x76c 2 0% 100% 0.00 550 0xfffffe2d1c4dc3e0 dnode_diduse_space+0x35 2 0% 100% 0.00 494 0xfffffe2d190d87c0 mac_rx_soft_ring_process+0x53 2 0% 100% 0.00 450 0xfffffe2cdfdf9c08 mlxcx_mac_ring_intr_enable+0x2b 2 0% 100% 0.00 365 0xfffffe2cdef3f2c8 mlxcx_mac_ring_intr_enable+0x2b 2 0% 100% 0.00 411 0xfffffe2ca2f3c4c0 sdintr+0x3d 2 0% 100% 0.00 502 0xfffffe2ca2f28388 sata_scsi_start+0x62 2 0% 100% 0.00 926 0xfffffe2c990047c0 zio_dva_throttle+0xa8 2 0% 100% 0.00 1939 0xfffffe2c97dca000 metaslab_free_concrete+0xc3 2 0% 100% 0.00 1403 0xfffffe2c91443c80 cpudrv_power+0x5b 2 0% 100% 0.00 2550 0xfffffe2c90eab950 cv_wait+0x78 2 0% 100% 0.00 1041 0xfffffe2c8edfac88 taskq_thread+0x2cd 2 0% 100% 0.00 2751 0xfffffe2c8edfa268 taskq_dispatch_ent+0x51 2 0% 100% 0.00 922 0xfffffe2c8e4e4840 callout_list_expire+0xa1 2 0% 100% 0.00 1090 0xfffffe2c8e4e46c0 untimeout_generic+0x65 2 0% 100% 0.00 3242 0xfffffe2c868a03c0 taskq_d_thread+0x21a 2 0% 100% 0.00 626 0xfffffe2c868a02c0 taskq_d_thread+0x21a 2 0% 100% 0.00 418 0xfffffe2c860da1d0 kmem_depot_free+0x35 2 0% 100% 0.00 374 0xfffffe2c860310f8 kmem_slab_alloc+0x39 1 0% 100% 0.00 435 zone_pdata zfs_zone_zio_start+0x3c 1 0% 100% 0.00 366 freemem_lock page_unresv+0x1f 1 0% 100% 0.00 280 freemem_lock page_xresv+0x29 1 0% 100% 0.00 340 ph_mutex+0x6fc0 page_find+0x89 1 0% 100% 0.00 472 0xfffffe2f4f10cdc0 mac_rx_soft_ring_drain+0x98 1 0% 100% 0.00 338 0xfffffe2f30959800 tcp_timeout+0x60 1 0% 100% 0.00 371 0xfffffe2f30959ef8 tcp_output+0x18d 1 0% 100% 0.00 388 0xfffffe2ef39bd7c0 tcp_timeout+0x60 1 0% 100% 0.00 334 0xfffffe2ef39bdeb8 tcp_output+0x18d 1 0% 100% 0.00 483 0xfffffe2deb1a7240 mac_rx_srs_poll_ring+0x173 1 0% 100% 0.00 1155 0xfffffe2deb1a1740 mac_rx_soft_ring_drain+0x115 1 0% 100% 0.00 466 0xfffffe2d1f6360e8 zio_remove_child+0x47 1 0% 100% 0.00 3253 0xfffffe2ddbb53700 mac_rx_srs_process+0x5c 1 0% 100% 0.00 4702 0xfffffe2d1cb8d2e0 taskq_dispatch+0x174 1 0% 100% 0.00 423 0xfffffe2d1ed957f8 poll_common+0x144 1 0% 100% 0.00 514 0xfffffe2cdfe06a00 mlxcx_mac_ring_rx_poll+0x34 1 0% 100% 0.00 438 0xfffffe2cdfe064c8 mlxcx_intr_n+0x10b 1 0% 100% 0.00 398 0xfffffe2d18030070 spa_config_exit+0x4b 1 0% 100% 0.00 444 0xfffffe2cdfe06a08 mlxcx_intr_n+0x10b 1 0% 100% 0.00 549 0xfffffe2d18030030 spa_config_exit+0x4b 1 0% 100% 0.00 1281 0xfffffe2d190d87c0 mac_rx_soft_ring_drain+0x98 1 0% 100% 0.00 434 0xfffffe2d1c4a0510 taskq_dispatch_ent+0x51 1 0% 100% 0.00 483 0xfffffe2d1947ce48 zio_wait_for_children+0x3b 1 0% 100% 0.00 389 0xfffffe2ca3c29d80 ddi_xbuf_qstrategy+0x2a 1 0% 100% 0.00 517 0xfffffe2ca3c29d80 ddi_xbuf_done+0x36 1 0% 100% 0.00 351 0xfffffe2ca2f28388 sata_scsi_init_pkt+0xe6 1 0% 100% 0.00 492 0xfffffe2cdef3f2c8 mlxcx_intr_n+0x10b 1 0% 100% 0.00 352 0xfffffe2cde161898 mlxcx_buf_take+0x2f 1 0% 100% 0.00 402 0xfffffe2c8e4e40c0 callout_list_expire+0xa1 1 0% 100% 0.00 495 0xfffffe2c8e4e4000 timeout_generic+0x72 1 0% 100% 0.00 418 0xfffffe2c8e4e4000 untimeout_generic+0x65 1 0% 100% 0.00 306 0xfffffe2c868a03c0 taskq_bucket_dispatch+0x32 1 0% 100% 0.00 1711 0xfffffe2c869aa4b0 ppm_lock_all+0x100 1 0% 100% 0.00 349 0xfffffe2c868a02c0 cv_timedwait_hires+0xf0 1 0% 100% 0.00 1185 0xfffffe2c868a0040 cv_timedwait_hires+0xf0 1 0% 100% 0.00 2629 0xfffffe2c860e0340 kmem_cache_free+0x78 1 0% 100% 0.00 1884 0xfffffe2c863ad340 kmem_cache_free+0x78 1 0% 100% 0.00 340 0xfffffe2c860391d0 kmem_depot_alloc+0x88 1 0% 100% 0.00 2188 0xfffffe2c86002020 vmem_alloc+0x77 1 0% 100% 0.00 287 0xfffffe003c5945b0 page_ctr_add+0x6b 1 0% 100% 0.00 471 0xfffffe003c5945a0 page_ctr_add+0x6b 1 0% 100% 0.00 1037 0xfffffe2c86003020 vmem_xalloc+0xef 1 0% 100% 0.00 1236 0xfffffe2c860dc0f8 kmem_slab_free+0x39 1 0% 100% 0.00 2251 0xfffffe2c860cc340 kmem_cache_free+0x78 1 0% 100% 0.00 2727 0xfffffe2c860e0280 kmem_cache_free+0x78 1 0% 100% 0.00 374 0xfffffe2c8f025b00 multilist_sublist_lock+0x27 1 0% 100% 0.00 1560 0xfffffe2c8ef43440 anon_resvmem+0x67 1 0% 100% 0.00 492 0xfffffe2c8ef42340 clock_tick_process+0xd0 1 0% 100% 0.00 2662 0xfffffe2c8ef42340 issig_forreal+0x40 1 0% 100% 0.00 2212 0xfffffe2c8edfac88 cv_wait+0x78 1 0% 100% 0.00 516 0xfffffe2c8e613948 ndi_devi_exit+0x3c 1 0% 100% 0.00 1386 0xfffffe2c8e613480 pm_scan+0x2e 1 0% 100% 0.00 436 0xfffffe2c8e4e4b40 callout_list_expire+0xa1 1 0% 100% 0.00 389 0xfffffe2c8e613948 ndi_devi_tryenter+0x4c 1 0% 100% 0.00 362 0xfffffe2c8e4e4600 untimeout_generic+0x65 1 0% 100% 0.00 542 0xfffffe2c8e4e4300 callout_list_expire+0xa1 1 0% 100% 0.00 2149 0xfffffe2c8e4e4180 callout_realtime+0x18 1 0% 100% 0.00 397 0xfffffe2c8e4e4240 callout_list_expire+0xa1 1 0% 100% 0.00 491 0xfffffe2c8e4e4240 callout_execute+0x18 1 0% 100% 0.00 878 0xfffffe2c8e4e4a80 untimeout_generic+0x65 1 0% 100% 0.00 333 0xfffffe2c8e4e4900 untimeout_generic+0x65 1 0% 100% 0.00 450 0xfffffe2c8e4e4b40 timeout_generic+0x72 1 0% 100% 0.00 2541 0xfffffe2c8e4e4b40 untimeout_generic+0x65 1 0% 100% 0.00 435 0xfffffe2c91377318 cpudrv_power+0x5b 1 0% 100% 0.00 346 0xfffffe2c912a4038 taskq_thread+0x2cd 1 0% 100% 0.00 566 0xfffffe2c8f025bc0 multilist_sublist_lock+0x27 1 0% 100% 0.00 386 0xfffffe2c91541b80 taskq_thread+0x2cd 1 0% 100% 0.00 6059 0xfffffe2c91443c80 cpudrv_monitor+0x3a 1 0% 100% 0.00 659 0xfffffe2c91541a60 taskq_thread+0x2cd 1 0% 100% 0.00 1301 0xfffffe2c97eec000 metaslab_class_evict_old+0xec 1 0% 100% 0.00 1458 0xfffffe2c97e13000 metaslab_group_alloc_normal+0x1b4 1 0% 100% 0.00 321 0xfffffe2c98549218 igb_get_free_list+0x3f ------------------------------------------------------------------------------- Adaptive mutex block: 9459 events in 20.963 seconds (451 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 2491 26% 26% 0.00 115367 0xfffffe2c90b24680 hook_run+0x34 1263 13% 40% 0.00 2552 0xfffffe2c860cc300 kmem_cache_free+0x78 1210 13% 52% 0.00 2866 0xfffffe2cd8bb1338 mlxcx_process_cq+0x207 872 9% 62% 0.00 15134 0xfffffe2cdc5971f8 mlxcx_process_cq+0x207 680 7% 69% 0.00 28002 0xfffffe2c90b24680 hook_run+0xd1 548 6% 75% 0.00 4691 0xfffffe2cdbac4538 mlxcx_process_cq+0x207 295 3% 78% 0.00 8839 0xfffffe2cdc58eb98 mlxcx_buf_return+0x8c 243 3% 80% 0.00 4687 0xfffffe2cdc5973b8 mlxcx_process_cq+0x207 208 2% 83% 0.00 6666 0xfffffe2d1ed957f8 cv_timedwait_sig_hires+0x2f1 164 2% 84% 0.00 5180 0xfffffe2cd8bb5758 mlxcx_buf_return+0x8c 156 2% 86% 0.00 4120 0xfffffe2c860cc280 kmem_cache_free+0x78 138 1% 87% 0.00 3831 0xfffffe2c860cc400 kmem_cache_free+0x78 129 1% 89% 0.00 12071 0xfffffe2ddbb53700 cv_wait+0x78 114 1% 90% 0.00 12114 0xfffffe2ddbb59200 cv_wait+0x78 86 1% 91% 0.00 17585 0xfffffe2c863ad0f8 kmem_slab_free+0x39 83 1% 92% 0.00 5201 0xfffffe2c860e02c0 kmem_cache_free+0x78 78 1% 93% 0.00 11362 0xfffffe2c90eab950 squeue_enter+0x53 73 1% 93% 0.00 9244 0xfffffe2cdc59fb58 mlxcx_buf_return+0x8c 62 1% 94% 0.00 59168 0xfffffe2f4f10cdc0 cv_wait+0x78 53 1% 95% 0.00 8858 0xfffffe2c863ad0f8 kmem_slab_alloc+0x39 38 0% 95% 0.00 7002 0xfffffe2cdc58ed18 mlxcx_buf_return+0x8c 27 0% 96% 0.00 11617 0xfffffe2f4f10cdc0 mac_rx_soft_ring_process+0x53 20 0% 96% 0.00 5936 0xfffffe2cdef4a258 mlxcx_buf_loan+0x75 18 0% 96% 0.00 347838 0xfffffe2f4f10ab00 cv_wait+0x78 16 0% 96% 0.00 7173 0xfffffe2f30959800 tcp_sendmsg+0xb8 15 0% 96% 0.00 5121 0xfffffe2deb1a7240 cv_wait+0x78 15 0% 97% 0.00 14452 0xfffffe2c90eabd10 squeue_drain+0x18b 15 0% 97% 0.00 8719 0xfffffe2c90eabd10 squeue_enter+0x53 14 0% 97% 0.00 5706 0xfffffe2f30959800 ipcl_classify_v4+0x25d 14 0% 97% 0.00 6132 0xfffffe2c860e0300 kmem_cache_free+0x78 12 0% 97% 0.00 5140 0xfffffe2d190d87c0 cv_wait+0x78 12 0% 97% 0.00 4033 0xfffffe2c860e0400 kmem_cache_free+0x78 11 0% 97% 0.00 12269 overlay_target_lock overlay_target_lookup_request+0x47 11 0% 98% 0.00 12824 0xfffffe2c90eabd10 squeue_enter+0x37a 11 0% 98% 0.00 9977 0xfffffe2c860e01d0 kmem_depot_alloc+0x88 10 0% 98% 0.00 22086 0xfffffe2f547b4840 squeue_drain+0x255 10 0% 98% 0.00 4668 0xfffffe2f4f109c00 cv_wait+0x78 10 0% 98% 0.00 19657 0xfffffe2d1ed957f8 pollwakeup+0x193 10 0% 98% 0.00 54420 0xfffffe2c90eab950 squeue_enter+0x37a 9 0% 98% 0.00 6931 0xfffffe2cdfdfdc58 mlxcx_buf_loan+0x75 9 0% 98% 0.00 23573 0xfffffe2c90eab950 squeue_drain+0x18b 9 0% 98% 0.00 5419 0xfffffe2c863ad2c0 kmem_cache_alloc+0x3d 8 0% 98% 0.00 4912 0xfffffe2deb1a1740 cv_wait+0x78 8 0% 98% 0.00 14220 0xfffffe2cdfdfdc58 mlxcx_buf_return+0x8c 7 0% 99% 0.00 5173 sqset_lock ip_squeue_random+0x22 7 0% 99% 0.00 9571 0xfffffe2c9a1e3040 squeue_drain+0x255 7 0% 99% 0.00 50474 0xfffffe2c9a1e3040 ipcl_classify_v4+0x25d 7 0% 99% 0.00 5445 0xfffffe2c860da2c0 kmem_cache_alloc+0x3d 6 0% 99% 0.00 7275 0xfffffe2ef39bd7c0 ipcl_classify_v4+0x25d 6 0% 99% 0.00 5284 0xfffffe2cdef4a558 mlxcx_buf_loan+0x75 5 0% 99% 0.00 7996 0xfffffe2f547b4f38 tcp_wput_data+0x401 5 0% 99% 0.00 32000 0xfffffe2ef39bd7c0 squeue_drain+0x255 5 0% 99% 0.00 2123367 0xfffffe2c9a1e3738 tcp_wput_data+0x401 5 0% 99% 0.00 14783 0xfffffe2c9a1e3040 tcp_sendmsg+0xb8 5 0% 99% 0.00 3816 0xfffffe2c860da340 kmem_cache_alloc+0x3d 4 0% 99% 0.00 35431 overlay_target_lock cv_timedwait_hires+0xf0 4 0% 99% 0.00 5161 0xfffffe2fa9bd3ec8 i_dls_head_hold+0x1f 4 0% 99% 0.00 49856 0xfffffe2f547b4f38 tcp_output+0x63 4 0% 99% 0.00 15861 0xfffffe2f547b4840 ipcl_classify_v4+0x25d 4 0% 99% 0.00 6846 0xfffffe2d1ca58bd8 taskq_dispatch_ent+0x51 4 0% 99% 0.00 36679 0xfffffe2c8684fe88 vdev_queue_io_done+0x30 3 0% 99% 0.00 32129 0xfffffe2ddbb59200 mac_rx_soft_ring_drain+0x115 3 0% 100% 0.00 8040 0xfffffe2cdef4a018 mlxcx_buf_return+0x8c 3 0% 100% 0.00 3041 0xfffffe2c9bce4f40 mac_tx+0x27d 3 0% 100% 0.00 9762 0xfffffe2c9a1e3040 squeue_enter+0x4cd 3 0% 100% 0.00 20853 0xfffffe2c8684fe88 vdev_queue_io+0x66 2 0% 100% 0.00 16774 0xfffffe2f4f10ab00 mac_rx_soft_ring_process+0x53 2 0% 100% 0.00 7460 0xfffffe2cdef4a018 mlxcx_buf_loan+0x75 2 0% 100% 0.00 11339 0xfffffe2cdc59fb58 mlxcx_buf_take+0x2f 2 0% 100% 0.00 9595 0xfffffe2cdc58eb98 mlxcx_buf_take+0x2f 2 0% 100% 0.00 8261 0xfffffe2c9a1e3738 tcp_output+0x63 2 0% 100% 0.00 22396 0xfffffe2c8edfa268 cv_wait+0x78 1 0% 100% 0.00 17040 0xfffffe2fa9bd3ec8 i_dls_head_rele+0x18 1 0% 100% 0.00 3007 0xfffffe2f547e57a8 ire_route_recursive_dstonly_v4+0x68 1 0% 100% 0.00 4622 0xfffffe2f30959ef8 tcp_output+0x63 1 0% 100% 0.00 10528 0xfffffe2ef39bdeb8 tcp_wput_data+0x401 1 0% 100% 0.00 98230 0xfffffe2f30959800 squeue_drain+0x255 1 0% 100% 0.00 8341 0xfffffe2ddbb53700 mac_rx_soft_ring_drain+0x115 1 0% 100% 0.00 184314 0xfffffe2cdfdf9c08 mlxcx_intr_n+0x10b 1 0% 100% 0.00 1068647 0xfffffe2ddbb53700 mac_rx_srs_process+0x5c 1 0% 100% 0.00 8437 0xfffffe2cdc58ed18 mlxcx_buf_take+0x2f 1 0% 100% 0.00 18236 0xfffffe2c9a1e3738 tcp_sendmsg+0x83 1 0% 100% 0.00 16916 0xfffffe2cd8bb5758 mlxcx_buf_take+0x2f 1 0% 100% 0.00 6849 0xfffffe2c860e0340 kmem_cache_free+0x78 1 0% 100% 0.00 5068 0xfffffe2c860e0280 kmem_cache_free+0x78 1 0% 100% 0.00 3854 0xfffffe2c860cc340 kmem_cache_free+0x78 1 0% 100% 0.00 8217 0xfffffe2c860dc1d0 kmem_depot_alloc+0x88 1 0% 100% 0.00 12406 0xfffffe2c8684fe88 vdev_queue_io_done+0x75 1 0% 100% 0.00 3551 0xfffffe2c863ad340 kmem_cache_free+0x78 1 0% 100% 0.00 13601 0xfffffe2c863ad1d0 kmem_depot_alloc+0x88 1 0% 100% 0.00 6859 0xfffffe2c8e4e4180 untimeout_generic+0x65 1 0% 100% 0.00 9874 0xfffffe2c868a03c0 cv_timedwait_hires+0xf0 1 0% 100% 0.00 11115 0xfffffe2c868a0340 taskq_bucket_dispatch+0x32 1 0% 100% 0.00 4124 0xfffffe2c8e4e4180 callout_realtime+0x18 1 0% 100% 0.00 7930 0xfffffe2c8edfac88 cv_wait+0x78 1 0% 100% 0.00 3723 0xfffffe2c86002020 vmem_alloc+0x77 1 0% 100% 0.00 8607 0xfffffe2c860cc1d0 kmem_depot_alloc+0x88 1 0% 100% 0.00 9358 0xfffffe2c86020020 vmem_xfree+0x39 ------------------------------------------------------------------------------- Spin lock spin: 4260 events in 20.963 seconds (203 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 779 18% 18% 0.00 4901 cp_default disp_lock_enter+0x26 545 13% 31% 0.00 3114 0xfffffe2c90ec23e8 disp_lock_enter+0x26 494 12% 43% 0.00 1347 turnstile_table+0xc38 disp_lock_enter+0x26 482 11% 54% 0.00 884 0xfffffe2c9136cb68 disp_lock_enter+0x26 458 11% 65% 0.00 4527 0xfffffe2c9135d400 disp_lock_enter+0x26 367 9% 73% 0.00 810 0xfffffe2c9128e5b0 disp_lock_enter+0x26 292 7% 80% 0.00 4171 0xfffffe2c9135dba8 disp_lock_enter+0x26 169 4% 84% 0.00 2002 0xfffffe2c9128eb28 disp_lock_enter+0x26 150 4% 88% 0.00 1310 0xfffffe2c9136c3f8 disp_lock_enter+0x26 121 3% 91% 0.00 6633 cp_default disp_lock_enter_high+0x11 75 2% 92% 0.00 1334 cpu0_disp disp_lock_enter+0x26 59 1% 94% 0.00 557 turnstile_table+0xa98 disp_lock_enter+0x26 48 1% 95% 0.00 843 0xfffffe2c9128e5b0 disp_lock_enter_high+0x11 47 1% 96% 0.00 1555 turnstile_table+0xa68 disp_lock_enter+0x26 26 1% 97% 0.00 4053 0xfffffe2c98300008 disp_lock_enter_high+0x11 19 0% 97% 0.00 759 0xfffffe2c9136cb68 disp_lock_enter_high+0x11 16 0% 97% 0.00 2264 0xfffffe2c9136c3f8 disp_lock_enter_high+0x11 16 0% 98% 0.00 3632 0xfffffe2c9135dba8 disp_lock_enter_high+0x11 15 0% 98% 0.00 21070 0xfffffe2c8699c2f1 mutex_vector_enter+0x431 11 0% 98% 0.00 8900 turnstile_table+0xc78 disp_lock_enter+0x26 10 0% 99% 0.00 792 turnstile_table+0xb08 disp_lock_enter+0x26 10 0% 99% 0.00 1152 0xfffffe2c90ec23e8 disp_lock_enter_high+0x11 7 0% 99% 0.00 326 turnstile_table+0xdb8 disp_lock_enter+0x26 6 0% 99% 0.00 2527 0xfffffe2c9135d400 disp_lock_enter_high+0x11 5 0% 99% 0.00 369 turnstile_table+0xb68 disp_lock_enter+0x26 4 0% 99% 0.00 306 turnstile_table+0xd38 disp_lock_enter+0x26 4 0% 99% 0.00 1509 cpu0_disp disp_lock_enter_high+0x11 3 0% 99% 0.00 1911 turnstile_table+0xd98 disp_lock_enter+0x26 3 0% 100% 0.00 379 turnstile_table+0xc98 disp_lock_enter+0x26 3 0% 100% 0.00 852 turnstile_table+0xcc8 disp_lock_enter+0x26 3 0% 100% 0.00 3188 turnstile_table+0xa18 disp_lock_enter+0x26 2 0% 100% 0.00 380 turnstile_table+0xbb8 disp_lock_enter+0x26 2 0% 100% 0.00 1186 0xfffffe2c9128eb28 disp_lock_enter_high+0x11 1 0% 100% 0.00 294 turnstile_table+0xbc8 disp_lock_enter+0x26 1 0% 100% 0.00 416 turnstile_table+0xa78 disp_lock_enter+0x26 1 0% 100% 0.00 248 turnstile_table+0xb48 disp_lock_enter+0x26 1 0% 100% 0.00 338 0xfffffe2c985c7b59 mutex_vector_enter+0x431 1 0% 100% 0.00 272 turnstile_table+0x808 disp_lock_enter+0x26 1 0% 100% 0.00 6172 sleepq_head+0xe18 disp_lock_enter_high+0x11 1 0% 100% 0.00 1210 turnstile_table+0x848 disp_lock_enter+0x26 1 0% 100% 0.00 363 turnstile_table+0xa18 turnstile_interlock+0x148 1 0% 100% 0.00 241 turnstile_table+0xa28 disp_lock_enter+0x26 ------------------------------------------------------------------------------- Thread lock spin: 738 events in 20.963 seconds (35 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 192 26% 26% 0.00 470 cpu[6]+0xe8 post_syscall+0x392 171 23% 49% 0.00 354 sleepq_head+0xe18 setrun+0x18 82 11% 60% 0.00 476 cpu[1]+0xe8 post_syscall+0x392 78 11% 71% 0.00 439 cpu[7]+0xe8 post_syscall+0x392 65 9% 80% 0.00 173 cpu[6]+0xe8 fss_tick+0x24 37 5% 85% 0.00 411 cpu[4]+0xe8 post_syscall+0x392 31 4% 89% 0.00 192 cpu[1]+0xe8 fss_tick+0x24 27 4% 93% 0.00 205 cpu[0]+0xe8 fss_tick+0x24 18 2% 95% 0.00 222 cpu[7]+0xe8 fss_tick+0x24 17 2% 97% 0.00 474 cpu[4]+0xe8 fss_tick+0x24 3 0% 98% 0.00 1017 cpu[4]+0xe8 preempt+0x47 3 0% 98% 0.00 200 cpu[2]+0xe8 post_syscall+0x392 3 0% 99% 0.00 266 cpu[2]+0xe8 fss_tick+0x24 2 0% 99% 0.00 1663 cpu[7]+0xe8 preempt+0x47 1 0% 99% 0.00 581 cpu[0]+0xe8 fss_update_list+0x53 1 0% 99% 0.00 4487 cpu[6]+0xe8 post_syscall+0x538 1 0% 99% 0.00 779 cpu[6]+0xe8 preempt+0x47 1 0% 99% 0.00 1601 cpu[7]+0xe8 sema_p+0x7e 1 0% 99% 0.00 123 cpu[7]+0xe8 turnstile_pi_waive+0x1b 1 0% 100% 0.00 302 cpu[7]+0xe8 turnstile_block+0x91 1 0% 100% 0.00 177 cpu[7]+0xe8 cv_timedwait_sig_hires+0x205 1 0% 100% 0.00 686 0xfffffe2c9136c3f8 sysdc_update+0x150 1 0% 100% 0.00 161 cpu[1]+0xe8 cv_timedwait_sig_hires+0x205 ------------------------------------------------------------------------------- R/W writer blocked by writer: 1 events in 20.963 seconds (0 events/sec) ------------------------------------------------------------------------------- 1 100% 100% 0.00 149779 0xfffffe2eabaef778 as_unmap+0x6a ------------------------------------------------------------------------------- R/W writer blocked by readers: 5 events in 20.963 seconds (0 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 3 60% 60% 0.00 19457 0xfffffe2eabaef778 as_map+0x4a 2 40% 100% 0.00 19931 0xfffffe2eabaef778 as_unmap+0x6a ------------------------------------------------------------------------------- R/W reader blocked by writer: 14 events in 20.963 seconds (1 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 14 100% 100% 0.00 184236 0xfffffe2eabaef778 as_fault+0xf1 -------------------------------------------------------------------------------