`master` -> QSFP28 ================== [root@server2 (us-mobile-1) /zones/global/mlxcx]# iperf3 -c 10.100.4.2 -P 4 Connecting to host 10.100.4.2, port 5201 [ 5] local 10.100.4.1 port 47034 connected to 10.100.4.2 port 5201 [ 7] local 10.100.4.1 port 33217 connected to 10.100.4.2 port 5201 [ 9] local 10.100.4.1 port 52069 connected to 10.100.4.2 port 5201 [ 11] local 10.100.4.1 port 45735 connected to 10.100.4.2 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 156 MBytes 1.31 Gbits/sec [ 7] 0.00-1.00 sec 171 MBytes 1.43 Gbits/sec [ 9] 0.00-1.00 sec 142 MBytes 1.19 Gbits/sec [ 11] 0.00-1.00 sec 170 MBytes 1.42 Gbits/sec [SUM] 0.00-1.00 sec 639 MBytes 5.36 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 159 MBytes 1.33 Gbits/sec [ 7] 1.00-2.00 sec 181 MBytes 1.51 Gbits/sec [ 9] 1.00-2.00 sec 148 MBytes 1.23 Gbits/sec [ 11] 1.00-2.00 sec 178 MBytes 1.49 Gbits/sec [SUM] 1.00-2.00 sec 665 MBytes 5.56 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 196 MBytes 1.65 Gbits/sec [ 7] 2.00-3.00 sec 205 MBytes 1.72 Gbits/sec [ 9] 2.00-3.00 sec 190 MBytes 1.59 Gbits/sec [ 11] 2.00-3.00 sec 202 MBytes 1.69 Gbits/sec [SUM] 2.00-3.00 sec 792 MBytes 6.66 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 186 MBytes 1.56 Gbits/sec [ 7] 3.00-4.00 sec 192 MBytes 1.61 Gbits/sec [ 9] 3.00-4.00 sec 181 MBytes 1.51 Gbits/sec [ 11] 3.00-4.00 sec 194 MBytes 1.63 Gbits/sec [SUM] 3.00-4.00 sec 753 MBytes 6.30 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 196 MBytes 1.64 Gbits/sec [ 7] 4.00-5.00 sec 200 MBytes 1.68 Gbits/sec [ 9] 4.00-5.00 sec 180 MBytes 1.51 Gbits/sec [ 11] 4.00-5.00 sec 194 MBytes 1.62 Gbits/sec [SUM] 4.00-5.00 sec 770 MBytes 6.45 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.01 sec 188 MBytes 1.58 Gbits/sec [ 7] 5.00-6.01 sec 192 MBytes 1.61 Gbits/sec [ 9] 5.00-6.01 sec 178 MBytes 1.49 Gbits/sec [ 11] 5.00-6.01 sec 195 MBytes 1.63 Gbits/sec [SUM] 5.00-6.01 sec 753 MBytes 6.31 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.01-7.00 sec 196 MBytes 1.65 Gbits/sec [ 7] 6.01-7.00 sec 218 MBytes 1.84 Gbits/sec [ 9] 6.01-7.00 sec 192 MBytes 1.62 Gbits/sec [ 11] 6.01-7.00 sec 216 MBytes 1.83 Gbits/sec [SUM] 6.01-7.00 sec 823 MBytes 6.95 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 204 MBytes 1.71 Gbits/sec [ 7] 7.00-8.00 sec 217 MBytes 1.82 Gbits/sec [ 9] 7.00-8.00 sec 197 MBytes 1.65 Gbits/sec [ 11] 7.00-8.00 sec 221 MBytes 1.85 Gbits/sec [SUM] 7.00-8.00 sec 839 MBytes 7.03 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 185 MBytes 1.55 Gbits/sec [ 7] 8.00-9.00 sec 193 MBytes 1.62 Gbits/sec [ 9] 8.00-9.00 sec 185 MBytes 1.54 Gbits/sec [ 11] 8.00-9.00 sec 196 MBytes 1.64 Gbits/sec [SUM] 8.00-9.00 sec 760 MBytes 6.35 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 183 MBytes 1.54 Gbits/sec [ 7] 9.00-10.00 sec 194 MBytes 1.64 Gbits/sec [ 9] 9.00-10.00 sec 178 MBytes 1.50 Gbits/sec [ 11] 9.00-10.00 sec 190 MBytes 1.60 Gbits/sec [SUM] 9.00-10.00 sec 746 MBytes 6.28 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.81 GBytes 1.55 Gbits/sec sender [ 5] 0.00-20.05 sec 1.81 GBytes 774 Mbits/sec receiver [ 7] 0.00-10.00 sec 1.92 GBytes 1.65 Gbits/sec sender [ 7] 0.00-20.05 sec 1.92 GBytes 822 Mbits/sec receiver [ 9] 0.00-10.00 sec 1.73 GBytes 1.49 Gbits/sec sender [ 9] 0.00-20.05 sec 1.73 GBytes 741 Mbits/sec receiver [ 11] 0.00-10.00 sec 1.91 GBytes 1.64 Gbits/sec sender [ 11] 0.00-20.05 sec 1.91 GBytes 818 Mbits/sec receiver [SUM] 0.00-10.00 sec 7.36 GBytes 6.33 Gbits/sec sender [SUM] 0.00-20.05 sec 7.36 GBytes 3.16 Gbits/sec receiver iperf Done. `master` -> QSFP56 ================== [root@server2 (us-mobile-1) /zones/global/mlxcx]# iperf3 -c 10.100.2.2 -P 4 Connecting to host 10.100.2.2, port 5201 [ 5] local 10.100.2.1 port 37136 connected to 10.100.2.2 port 5201 [ 7] local 10.100.2.1 port 37920 connected to 10.100.2.2 port 5201 [ 9] local 10.100.2.1 port 63630 connected to 10.100.2.2 port 5201 [ 11] local 10.100.2.1 port 48654 connected to 10.100.2.2 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.01 sec 115 MBytes 956 Mbits/sec [ 7] 0.00-1.01 sec 112 MBytes 934 Mbits/sec [ 9] 0.00-1.01 sec 110 MBytes 914 Mbits/sec [ 11] 0.00-1.01 sec 110 MBytes 921 Mbits/sec [SUM] 0.00-1.01 sec 447 MBytes 3.72 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.01-2.00 sec 106 MBytes 898 Mbits/sec [ 7] 1.01-2.00 sec 106 MBytes 896 Mbits/sec [ 9] 1.01-2.00 sec 106 MBytes 896 Mbits/sec [ 11] 1.01-2.00 sec 104 MBytes 877 Mbits/sec [SUM] 1.01-2.00 sec 423 MBytes 3.57 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 174 MBytes 1.46 Gbits/sec [ 7] 2.00-3.00 sec 174 MBytes 1.46 Gbits/sec [ 9] 2.00-3.00 sec 171 MBytes 1.43 Gbits/sec [ 11] 2.00-3.00 sec 169 MBytes 1.42 Gbits/sec [SUM] 2.00-3.00 sec 689 MBytes 5.77 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 184 MBytes 1.54 Gbits/sec [ 7] 3.00-4.00 sec 183 MBytes 1.53 Gbits/sec [ 9] 3.00-4.00 sec 183 MBytes 1.53 Gbits/sec [ 11] 3.00-4.00 sec 179 MBytes 1.50 Gbits/sec [SUM] 3.00-4.00 sec 730 MBytes 6.11 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 196 MBytes 1.65 Gbits/sec [ 7] 4.00-5.00 sec 195 MBytes 1.64 Gbits/sec [ 9] 4.00-5.00 sec 196 MBytes 1.65 Gbits/sec [ 11] 4.00-5.00 sec 194 MBytes 1.63 Gbits/sec [SUM] 4.00-5.00 sec 781 MBytes 6.57 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 196 MBytes 1.64 Gbits/sec [ 7] 5.00-6.00 sec 195 MBytes 1.63 Gbits/sec [ 9] 5.00-6.00 sec 196 MBytes 1.64 Gbits/sec [ 11] 5.00-6.00 sec 195 MBytes 1.64 Gbits/sec [SUM] 5.00-6.00 sec 783 MBytes 6.55 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 196 MBytes 1.65 Gbits/sec [ 7] 6.00-7.00 sec 196 MBytes 1.64 Gbits/sec [ 9] 6.00-7.00 sec 195 MBytes 1.64 Gbits/sec [ 11] 6.00-7.00 sec 195 MBytes 1.63 Gbits/sec [SUM] 6.00-7.00 sec 782 MBytes 6.56 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 168 MBytes 1.41 Gbits/sec [ 7] 7.00-8.00 sec 168 MBytes 1.41 Gbits/sec [ 9] 7.00-8.00 sec 167 MBytes 1.40 Gbits/sec [ 11] 7.00-8.00 sec 166 MBytes 1.40 Gbits/sec [SUM] 7.00-8.00 sec 670 MBytes 5.62 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 189 MBytes 1.58 Gbits/sec [ 7] 8.00-9.00 sec 188 MBytes 1.58 Gbits/sec [ 9] 8.00-9.00 sec 188 MBytes 1.58 Gbits/sec [ 11] 8.00-9.00 sec 189 MBytes 1.59 Gbits/sec [SUM] 8.00-9.00 sec 754 MBytes 6.32 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 197 MBytes 1.65 Gbits/sec [ 7] 9.00-10.00 sec 195 MBytes 1.63 Gbits/sec [ 9] 9.00-10.00 sec 200 MBytes 1.67 Gbits/sec [ 11] 9.00-10.00 sec 202 MBytes 1.69 Gbits/sec [SUM] 9.00-10.00 sec 794 MBytes 6.65 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.68 GBytes 1.44 Gbits/sec sender [ 5] 0.00-10.00 sec 1.68 GBytes 1.44 Gbits/sec receiver [ 7] 0.00-10.00 sec 1.67 GBytes 1.44 Gbits/sec sender [ 7] 0.00-10.00 sec 1.67 GBytes 1.44 Gbits/sec receiver [ 9] 0.00-10.00 sec 1.67 GBytes 1.44 Gbits/sec sender [ 9] 0.00-10.00 sec 1.67 GBytes 1.44 Gbits/sec receiver [ 11] 0.00-10.00 sec 1.66 GBytes 1.43 Gbits/sec sender [ 11] 0.00-10.00 sec 1.66 GBytes 1.43 Gbits/sec receiver [SUM] 0.00-10.00 sec 6.69 GBytes 5.74 Gbits/sec sender [SUM] 0.00-10.00 sec 6.69 GBytes 5.74 Gbits/sec receiver iperf Done. `master` Lockstat on QSFP56 [root@server2 (us-mobile-1) /zones/global/mlxcx]# lockstat iperf3 -c 10.100.2.2 -P 4 Connecting to host 10.100.2.2, port 5201 [ 6] local 10.100.2.1 port 44701 connected to 10.100.2.2 port 5201 [ 8] local 10.100.2.1 port 41817 connected to 10.100.2.2 port 5201 [ 10] local 10.100.2.1 port 59189 connected to 10.100.2.2 port 5201 [ 12] local 10.100.2.1 port 37864 connected to 10.100.2.2 port 5201 [ ID] Interval Transfer Bitrate [ 6] 0.00-1.00 sec 177 MBytes 1.48 Gbits/sec [ 8] 0.00-1.00 sec 136 MBytes 1.14 Gbits/sec [ 10] 0.00-1.00 sec 182 MBytes 1.52 Gbits/sec [ 12] 0.00-1.00 sec 181 MBytes 1.51 Gbits/sec [SUM] 0.00-1.00 sec 676 MBytes 5.65 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 1.00-2.00 sec 176 MBytes 1.48 Gbits/sec [ 8] 1.00-2.00 sec 135 MBytes 1.13 Gbits/sec [ 10] 1.00-2.00 sec 181 MBytes 1.52 Gbits/sec [ 12] 1.00-2.00 sec 180 MBytes 1.52 Gbits/sec [SUM] 1.00-2.00 sec 672 MBytes 5.65 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 2.00-3.00 sec 166 MBytes 1.40 Gbits/sec [ 8] 2.00-3.00 sec 127 MBytes 1.07 Gbits/sec [ 10] 2.00-3.00 sec 175 MBytes 1.47 Gbits/sec [ 12] 2.00-3.00 sec 175 MBytes 1.47 Gbits/sec [SUM] 2.00-3.00 sec 644 MBytes 5.40 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 3.00-4.00 sec 178 MBytes 1.49 Gbits/sec [ 8] 3.00-4.00 sec 136 MBytes 1.14 Gbits/sec [ 10] 3.00-4.00 sec 187 MBytes 1.57 Gbits/sec [ 12] 3.00-4.00 sec 188 MBytes 1.58 Gbits/sec [SUM] 3.00-4.00 sec 689 MBytes 5.78 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 4.00-5.00 sec 197 MBytes 1.65 Gbits/sec [ 8] 4.00-5.00 sec 157 MBytes 1.32 Gbits/sec [ 10] 4.00-5.00 sec 206 MBytes 1.73 Gbits/sec [ 12] 4.00-5.00 sec 213 MBytes 1.79 Gbits/sec [SUM] 4.00-5.00 sec 772 MBytes 6.48 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 5.00-6.00 sec 189 MBytes 1.59 Gbits/sec [ 8] 5.00-6.00 sec 156 MBytes 1.32 Gbits/sec [ 10] 5.00-6.00 sec 201 MBytes 1.69 Gbits/sec [ 12] 5.00-6.00 sec 199 MBytes 1.67 Gbits/sec [SUM] 5.00-6.00 sec 746 MBytes 6.27 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 6.00-7.00 sec 191 MBytes 1.61 Gbits/sec [ 8] 6.00-7.00 sec 162 MBytes 1.36 Gbits/sec [ 10] 6.00-7.00 sec 211 MBytes 1.77 Gbits/sec [ 12] 6.00-7.00 sec 206 MBytes 1.73 Gbits/sec [SUM] 6.00-7.00 sec 771 MBytes 6.47 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 7.00-8.00 sec 184 MBytes 1.54 Gbits/sec [ 8] 7.00-8.00 sec 150 MBytes 1.25 Gbits/sec [ 10] 7.00-8.00 sec 198 MBytes 1.65 Gbits/sec [ 12] 7.00-8.00 sec 196 MBytes 1.64 Gbits/sec [SUM] 7.00-8.00 sec 728 MBytes 6.08 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 8.00-9.01 sec 175 MBytes 1.46 Gbits/sec [ 8] 8.00-9.01 sec 141 MBytes 1.18 Gbits/sec [ 10] 8.00-9.01 sec 187 MBytes 1.57 Gbits/sec [ 12] 8.00-9.01 sec 188 MBytes 1.57 Gbits/sec [SUM] 8.00-9.01 sec 691 MBytes 5.79 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 6] 9.01-10.00 sec 191 MBytes 1.61 Gbits/sec [ 8] 9.01-10.00 sec 153 MBytes 1.29 Gbits/sec [ 10] 9.01-10.00 sec 213 MBytes 1.79 Gbits/sec [ 12] 9.01-10.00 sec 213 MBytes 1.79 Gbits/sec [SUM] 9.01-10.00 sec 770 MBytes 6.47 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 6] 0.00-10.00 sec 1.78 GBytes 1.53 Gbits/sec sender [ 6] 0.00-20.00 sec 1.78 GBytes 765 Mbits/sec receiver [ 8] 0.00-10.00 sec 1.42 GBytes 1.22 Gbits/sec sender [ 8] 0.00-20.00 sec 1.42 GBytes 610 Mbits/sec receiver [ 10] 0.00-10.00 sec 1.89 GBytes 1.63 Gbits/sec sender [ 10] 0.00-20.00 sec 1.89 GBytes 814 Mbits/sec receiver [ 12] 0.00-10.00 sec 1.89 GBytes 1.63 Gbits/sec sender [ 12] 0.00-20.00 sec 1.89 GBytes 813 Mbits/sec receiver [SUM] 0.00-10.00 sec 6.99 GBytes 6.00 Gbits/sec sender [SUM] 0.00-20.00 sec 6.99 GBytes 3.00 Gbits/sec receiver iperf Done. Adaptive mutex spin: 850634 events in 20.818 seconds (40861 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 183736 22% 22% 0.00 1760 0xfffffe2c90b24680 hook_run+0xd1 156543 18% 40% 0.00 1841 0xfffffe2c90b24680 hook_run+0x34 94605 11% 51% 0.00 469 0xfffffe2cdda2dd50 mlxcx_buf_return+0x8c 92082 11% 62% 0.00 664 0xfffffe2cdda2d350 mlxcx_buf_return+0x8c 78086 9% 71% 0.00 488 0xfffffe2cdbbb0480 mlxcx_buf_return+0x8c 77780 9% 80% 0.00 496 0xfffffe2cdda16a18 mlxcx_buf_return+0x8c 33981 4% 84% 0.00 754 0xfffffe2cdda2d350 mlxcx_buf_take+0x2f 18443 2% 86% 0.00 632 0xfffffe2cdda2dd50 mlxcx_buf_take+0x2f 18356 2% 89% 0.00 635 0xfffffe2cdbbb0480 mlxcx_buf_take+0x2f 16875 2% 91% 0.00 679 0xfffffe2cdda16a18 mlxcx_buf_take+0x2f 9739 1% 92% 0.00 674 0xfffffe2cdda1bef8 mlxcx_process_cq+0x207 9386 1% 93% 0.00 493 0xfffffe2cdbbaa378 mlxcx_process_cq+0x207 8992 1% 94% 0.00 416 0xfffffe2cdda29eb8 mlxcx_process_cq+0x207 7386 1% 95% 0.00 445 0xfffffe2cdda1b2b8 mlxcx_process_cq+0x207 4753 1% 95% 0.00 436 0xfffffe2cdbbaa378 mlxcx_sq_add_buffer+0x469 4507 1% 96% 0.00 508 0xfffffe2cdda1bef8 mlxcx_sq_add_buffer+0x469 4059 0% 96% 0.00 407 0xfffffe2cdda29eb8 mlxcx_sq_add_buffer+0x469 3102 0% 97% 0.00 452 0xfffffe2cdda1b2b8 mlxcx_sq_add_buffer+0x469 3026 0% 97% 0.00 902 0xfffffe2c90eabad0 squeue_enter+0x53 2164 0% 97% 0.00 671 0xfffffe2c90eabad0 squeue_enter+0x37a 1278 0% 97% 0.00 706 0xfffffe2c90eabad0 squeue_drain+0x18b 1078 0% 98% 0.00 981 0xfffffe2cdf606528 mlxcx_buf_return+0x8c 1076 0% 98% 0.00 3160 0xfffffe2f6d891280 cv_wait+0x78 1061 0% 98% 0.00 682 0xfffffe2c90eabb90 squeue_enter+0x37a 1006 0% 98% 0.00 451 0xfffffe2c90eabd10 squeue_enter+0x37a 901 0% 98% 0.00 951 0xfffffe2cdf606528 mlxcx_buf_loan+0x76 895 0% 98% 0.00 759 0xfffffe2c90eabd10 squeue_enter+0x53 769 0% 98% 0.00 930 0xfffffe2c90eabb90 squeue_enter+0x53 690 0% 98% 0.00 3127 0xfffffe2f7cfb5000 cv_wait+0x78 635 0% 98% 0.00 936 0xfffffe2cdf6063e8 mlxcx_buf_return+0x8c 551 0% 98% 0.00 1017 0xfffffe2cdf606ca8 mlxcx_buf_return+0x8c 533 0% 99% 0.00 2250 0xfffffe2f7cfb66c0 cv_wait+0x78 505 0% 99% 0.00 767 0xfffffe2cdf606ca8 mlxcx_buf_loan+0x76 498 0% 99% 0.00 658 0xfffffe2c90eabb90 squeue_drain+0x18b 465 0% 99% 0.00 835 0xfffffe2cdf6063e8 mlxcx_buf_loan+0x76 454 0% 99% 0.00 559 0xfffffe2c90eabd10 squeue_drain+0x18b 402 0% 99% 0.00 659 0xfffffe2cdf6065c8 mlxcx_buf_loan+0x76 389 0% 99% 0.00 771 0xfffffe2cdf6065c8 mlxcx_buf_return+0x8c 346 0% 99% 0.00 538 sqset_lock ip_squeue_random+0x22 325 0% 99% 0.00 680 0xfffffe30e47a2840 ipcl_classify_v4+0x25d 298 0% 99% 0.00 652 0xfffffe30e6a527c0 ipcl_classify_v4+0x25d 291 0% 99% 0.00 602 0xfffffe30e833f840 ipcl_classify_v4+0x25d 284 0% 99% 0.00 590 0xfffffe30e47a2840 squeue_drain+0x255 273 0% 99% 0.00 734 0xfffffe30e91a4840 ipcl_classify_v4+0x25d 272 0% 99% 0.00 2166 0xfffffe2c860e0340 kmem_cache_free+0x78 264 0% 99% 0.00 554 0xfffffe2d19355c08 i_dls_head_hold+0x1f 252 0% 99% 0.00 1935 0xfffffe2c860cc400 kmem_cache_free+0x78 241 0% 99% 0.00 625 0xfffffe2d19355c08 i_dls_head_rele+0x18 226 0% 99% 0.00 8591 0xfffffe2d1cc72760 cv_wait+0x78 218 0% 99% 0.00 588 0xfffffe30e91a4840 squeue_drain+0x255 214 0% 99% 0.00 415 0xfffffe30e833f840 squeue_drain+0x255 208 0% 99% 0.00 1832 0xfffffe2c860e0280 kmem_cache_free+0x78 206 0% 99% 0.00 12376 overlay_target_lock overlay_target_lookup_request+0x47 195 0% 99% 0.00 6893 0xfffffe2d1cc72760 taskq_thread+0x2cd 184 0% 99% 0.00 587 0xfffffe30e6a527c0 tcp_sendmsg+0xb8 184 0% 99% 0.00 2985 0xfffffe30cf563200 mac_rx_soft_ring_process+0x53 181 0% 99% 0.00 588 0xfffffe30e47a2840 tcp_sendmsg+0xb8 175 0% 99% 0.00 629 0xfffffe30e6a527c0 squeue_drain+0x255 175 0% 99% 0.00 8973 0xfffffe2d1cc72760 taskq_dispatch_ent+0x51 166 0% 100% 0.00 781 0xfffffe30e6a527c0 squeue_enter+0x4cd 163 0% 100% 0.00 674 0xfffffe30e47a2840 squeue_enter+0x4cd 146 0% 100% 0.00 494 0xfffffe2c860e01d0 kmem_depot_alloc+0x88 145 0% 100% 0.00 12332 overlay_target_lock cv_timedwait_hires+0xf0 143 0% 100% 0.00 539 0xfffffe30e833ff38 tcp_sendmsg+0x83 142 0% 100% 0.00 405 0xfffffe30e833f840 squeue_enter+0x4cd 141 0% 100% 0.00 592 0xfffffe30e91a4f38 tcp_sendmsg+0x83 136 0% 100% 0.00 536 0xfffffe30e91a4f38 tcp_output+0x63 127 0% 100% 0.00 452 0xfffffe30e833f840 tcp_sendmsg+0xb8 122 0% 100% 0.00 665 0xfffffe2dae54ea30 dls_accept_common+0x80 121 0% 100% 0.00 2452 0xfffffe2c860e0400 kmem_cache_free+0x78 116 0% 100% 0.00 549 0xfffffe30e6a52eb8 tcp_sendmsg+0x83 114 0% 100% 0.00 495 0xfffffe30e91a4840 tcp_sendmsg+0xb8 114 0% 100% 0.00 451 0xfffffe30e833ff38 tcp_output+0x63 113 0% 100% 0.00 2000 0xfffffe2c860e0240 kmem_cache_free+0x78 111 0% 100% 0.00 647 0xfffffe30e47a2f38 tcp_sendmsg+0x83 109 0% 100% 0.00 445 0xfffffe30e47a2f38 tcp_output+0x63 109 0% 100% 0.00 2508 0xfffffe30d9ce6780 mac_rx_soft_ring_process+0x53 101 0% 100% 0.00 17434 0xfffffe2c9aa60e88 vdev_queue_io+0x66 91 0% 100% 0.00 488 0xfffffe30e6a52eb8 tcp_output+0x63 89 0% 100% 0.00 2078 0xfffffe30d9ce5dc0 cv_wait+0x78 78 0% 100% 0.00 2540 0xfffffe30d9ce6c00 cv_wait+0x78 74 0% 100% 0.00 595 0xfffffe30e91a4840 squeue_enter+0x4cd 69 0% 100% 0.00 1642 0xfffffe30cf563200 cv_wait+0x78 67 0% 100% 0.00 2549 0xfffffe30d9ce6c00 mac_rx_soft_ring_process+0x53 65 0% 100% 0.00 21803 0xfffffe2d1cc73758 cv_wait+0x78 62 0% 100% 0.00 1448 0xfffffe30d9ce6780 cv_wait+0x78 57 0% 100% 0.00 7186 0xfffffe2d1cc73758 taskq_thread+0x2cd 54 0% 100% 0.00 3394 0xfffffe2d193b0a00 getf_gen+0x63 50 0% 100% 0.00 13970 0xfffffe2c8edfa268 cv_wait+0x78 46 0% 100% 0.00 8832 0xfffffe2c9aa60e88 vdev_queue_io_done+0x30 41 0% 100% 0.00 588 0xfffffe30e91a4f38 tcp_wput_data+0x401 38 0% 100% 0.00 649 0xfffffe30e6a7d7a8 ire_route_recursive_dstonly_v4+0x68 35 0% 100% 0.00 7447 0xfffffe2c9aa60e88 vdev_queue_io_done+0x75 31 0% 100% 0.00 2024 0xfffffe2c863ad240 kmem_cache_alloc+0x3d 29 0% 100% 0.00 5671 0xfffffe2d1cc721c0 taskq_thread+0x2cd 25 0% 100% 0.00 5292 0xfffffe2d1c639aa8 cv_wait+0x78 22 0% 100% 0.00 1815 0xfffffe2c989fecc0 ahci_tran_start+0x75 22 0% 100% 0.00 366 0xfffffe2c860cc1d0 kmem_depot_alloc+0x88 20 0% 100% 0.00 12417 0xfffffe2c9aa60e88 vdev_queue_aggregate+0x43c 20 0% 100% 0.00 3549 0xfffffe2c860e03c0 kmem_cache_free+0x78 19 0% 100% 0.00 383 0xfffffe30e47a2f38 tcp_wput_data+0x401 19 0% 100% 0.00 608 0xfffffe2ce04c5a08 mlxcx_mac_ring_intr_enable+0x2b 19 0% 100% 0.00 2973 0xfffffe2c863ad3c0 kmem_cache_free+0x78 19 0% 100% 0.00 2152 0xfffffe2c860e0380 kmem_cache_free+0x78 18 0% 100% 0.00 870 0xfffffe30d9ce5dc0 mac_rx_soft_ring_process+0x53 18 0% 100% 0.00 3811 0xfffffe2d193b0a00 releasef+0x53 17 0% 100% 0.00 637 0xfffffe30e6a52eb8 tcp_wput_data+0x401 17 0% 100% 0.00 2647 0xfffffe2f8c709200 cv_wait+0x78 17 0% 100% 0.00 1417 0xfffffe2d1c639aa8 taskq_dispatch_ent+0x51 16 0% 100% 0.00 579 0xfffffe2ce04c5a00 mlxcx_mac_ring_rx_poll+0x34 16 0% 100% 0.00 849 0xfffffe2cdf60b280 mlxcx_mac_ring_rx_poll+0x34 16 0% 100% 0.00 1353 0xfffffe2c986078c0 sd_buf_iodone+0x54 16 0% 100% 0.00 432 0xfffffe2c860e01d0 kmem_depot_free+0x35 15 0% 100% 0.00 377 0xfffffe30e833ff38 tcp_wput_data+0x401 15 0% 100% 0.00 954 0xfffffe2c986078c0 sd_start_cmds+0x105 15 0% 100% 0.00 2155 0xfffffe2c986078c0 sd_initpkt_for_buf+0x1c9 14 0% 100% 0.00 903 0xfffffe2f7cfb5000 mac_rx_soft_ring_drain+0x115 14 0% 100% 0.00 66919 0xfffffe2c8e643590 gcpu_ntv_mca_poll_wrapper+0x72 13 0% 100% 0.00 2248 0xfffffe2f7cfb66c0 mac_rx_soft_ring_drain+0x115 13 0% 100% 0.00 1912 0xfffffe2d1ec2a2d8 pollwakeup+0x193 13 0% 100% 0.00 2952 0xfffffe2d1cce4680 metaslab_group_alloc+0x63 13 0% 100% 0.00 2348 0xfffffe2d1cce4680 metaslab_group_alloc_normal+0x15f 13 0% 100% 0.00 2408 0xfffffe2d1cc73758 taskq_dispatch+0x174 13 0% 100% 0.00 11128 0xfffffe2d1cc721c0 cv_wait+0x78 12 0% 100% 0.00 1624 0xfffffe2d1ec2a2d8 cv_timedwait_sig_hires+0x2f1 12 0% 100% 0.00 5409 0xfffffe2d1c639aa8 taskq_thread+0x2cd 12 0% 100% 0.00 544 0xfffffe2ce04c5680 mlxcx_mac_ring_rx_poll+0x34 11 0% 100% 0.00 634 0xfffffe2cdf60b288 mlxcx_mac_ring_intr_enable+0x2b 11 0% 100% 0.00 1733 0xfffffe2c989fecc0 ahci_flush_doneq+0x78 10 0% 100% 0.00 1020 0xfffffe2f8c709200 mac_rx_soft_ring_drain+0x115 10 0% 100% 0.00 716 0xfffffe2cdf606528 mlxcx_buf_take_n+0x36 10 0% 100% 0.00 1711 0xfffffe2c989fecc0 ahci_intr_ncq_events+0x59 10 0% 100% 0.00 1030 0xfffffe2c986078c0 sd_core_iostart+0x40 8 0% 100% 0.00 859 0xfffffe2f6d891280 mac_rx_soft_ring_drain+0x115 8 0% 100% 0.00 6295 0xfffffe2d1cc73758 taskq_ent_free+0x6d 8 0% 100% 0.00 1539 0xfffffe2c986078c0 sd_return_command+0x12f 8 0% 100% 0.00 4297 0xfffffe2c97f17800 metaslab_free_concrete+0xc3 8 0% 100% 0.00 3241 0xfffffe2c868a0340 cv_timedwait_hires+0xf0 8 0% 100% 0.00 3278 0xfffffe2c863ad3c0 kmem_cache_alloc+0x3d 7 0% 100% 0.00 362 0xfffffe2ce04c5bc8 mlxcx_mac_ring_intr_enable+0x2b 7 0% 100% 0.00 399 0xfffffe2cdf60b288 mlxcx_intr_n+0x10b 7 0% 100% 0.00 1166 0xfffffe2c989fecc0 ahci_port_intr+0x62 7 0% 100% 0.00 4411 0xfffffe2c97ee7000 metaslab_group_alloc_normal+0x1b4 7 0% 100% 0.00 2048 0xfffffe2c860da240 kmem_cache_alloc+0x3d 7 0% 100% 0.00 817 0xfffffe003c5945f8 page_ctr_sub+0x6b 6 0% 100% 0.00 3199 0xfffffe30d9ce6780 mac_rx_soft_ring_drain+0x98 6 0% 100% 0.00 4714 0xfffffe2c97ee7000 metaslab_free_concrete+0xc3 6 0% 100% 0.00 1289 0xfffffe2c97e20000 metaslab_group_alloc_normal+0x1b4 6 0% 100% 0.00 8009 0xfffffe2c90ecc980 cv_timedwait_hires+0xf0 6 0% 100% 0.00 2244 0xfffffe2c863ad340 kmem_cache_alloc+0x3d 5 0% 100% 0.00 3433 zone_pdata zfs_zone_zio_done+0xa9 5 0% 100% 0.00 1228 0xfffffe30e91a4840 conn_clrqfull+0x59 5 0% 100% 0.00 573 0xfffffe2ce04c5688 mlxcx_mac_ring_intr_enable+0x2b 5 0% 100% 0.00 667 0xfffffe2c986078c0 sdintr+0x3d 5 0% 100% 0.00 907 0xfffffe2c986078c0 sdstrategy+0x5b 5 0% 100% 0.00 326 0xfffffe2c863ad1d0 kmem_depot_alloc+0x88 4 0% 100% 0.00 1382 0xfffffe30da592d30 lwp_unpark+0x54 4 0% 100% 0.00 5682 0xfffffe2d1cc73758 taskq_ent_alloc+0xe4 4 0% 100% 0.00 1386 0xfffffe2ca2fc6408 sata_hba_start+0xc4 4 0% 100% 0.00 1499 0xfffffe2c97f17800 metaslab_group_alloc_normal+0x1b4 4 0% 100% 0.00 3013 0xfffffe2c8ef43040 stop+0x2eb 4 0% 100% 0.00 1462 0xfffffe2c868a03c0 cv_timedwait_hires+0xf0 4 0% 100% 0.00 466 0xfffffe2c860dc1d0 kmem_depot_alloc+0x88 4 0% 100% 0.00 2386 0xfffffe2c860da340 kmem_cache_alloc+0x3d 4 0% 100% 0.00 2489 0xfffffe2c860cc340 kmem_cache_free+0x78 3 0% 100% 0.00 540 0xfffffe2f8c709200 mac_rx_srs_poll_ring+0x173 3 0% 100% 0.00 10397 0xfffffe2d18b0f070 spa_config_exit+0x4b 3 0% 100% 0.00 361 0xfffffe2ce04c5bc0 mlxcx_mac_ring_rx_poll+0x34 3 0% 100% 0.00 384 0xfffffe2ce04c5a08 mlxcx_intr_n+0x10b 3 0% 100% 0.00 534 0xfffffe2cdf606ca8 mlxcx_buf_take_n+0x36 3 0% 100% 0.00 961 0xfffffe2c97e20000 metaslab_free_concrete+0xc3 3 0% 100% 0.00 388 0xfffffe2c8edfac88 taskq_dispatch+0x174 3 0% 100% 0.00 863 0xfffffe2c8e4e4840 callout_execute+0x18 3 0% 100% 0.00 1881 0xfffffe2c8e4e4840 callout_list_expire+0xa1 3 0% 100% 0.00 3886 0xfffffe2c868a0340 taskq_bucket_dispatch+0x32 3 0% 100% 0.00 3009 0xfffffe2c860e02c0 kmem_cache_free+0x78 2 0% 100% 0.00 2289 0xfffffe2f8c709200 mac_rx_srs_process+0x5c 2 0% 100% 0.00 2986 0xfffffe2f7cfb66c0 mac_rx_srs_process+0x5c 2 0% 100% 0.00 572 0xfffffe2f6d891280 mac_rx_srs_drain+0x22d 2 0% 100% 0.00 629 0xfffffe2f6d891280 mac_rx_srs_poll_ring+0x173 2 0% 100% 0.00 4774 0xfffffe2d1cc73758 taskq_wait+0x2f 2 0% 100% 0.00 648 0xfffffe2cdf6063e8 mlxcx_buf_take_n+0x36 2 0% 100% 0.00 488 0xfffffe2ca3c9fd80 ddi_xbuf_done+0x36 2 0% 100% 0.00 923 0xfffffe2ca2fc6408 sata_scsi_start+0x62 2 0% 100% 0.00 474 0xfffffe2ca2fc6408 sata_scsi_init_pkt+0xe6 2 0% 100% 0.00 3265 0xfffffe2c8ef43040 syslwp_continue+0x28 2 0% 100% 0.00 549 0xfffffe2c8edfac88 taskq_thread+0x2cd 2 0% 100% 0.00 1798 0xfffffe2c8e613948 ndi_devi_tryenter+0x4c 2 0% 100% 0.00 1453 0xfffffe2c8e4e4b40 untimeout_generic+0x65 2 0% 100% 0.00 782 0xfffffe2c8e4e4a80 untimeout_generic+0x65 2 0% 100% 0.00 2008 0xfffffe2c8e4e4840 untimeout_generic+0x65 2 0% 100% 0.00 1642 0xfffffe2c8e4e4600 untimeout_generic+0x65 2 0% 100% 0.00 964 0xfffffe2c868a0340 taskq_d_thread+0x21a 2 0% 100% 0.00 1385 0xfffffe2c868a02c0 cv_timedwait_hires+0xf0 2 0% 100% 0.00 2923 0xfffffe2c863ad2c0 kmem_cache_alloc+0x3d 2 0% 100% 0.00 1752 0xfffffe2c860da280 kmem_cache_alloc+0x3d 2 0% 100% 0.00 521 0xfffffe2c860da1d0 kmem_depot_alloc+0x88 1 0% 100% 0.00 1610 zone_pdata add_zone_iop+0x2d 1 0% 100% 0.00 1914 pidlock thread_free+0x1aa 1 0% 100% 0.00 627 ph_mutex+0x280 page_lookup_create+0x1bc 1 0% 100% 0.00 1459 htable_mutex+0x2c8 htable_release+0x103 1 0% 100% 0.00 346 htable_mutex+0x2c8 htable_lookup+0x7e 1 0% 100% 0.00 499 zfs_disk_lock zfs_zone_zio_done+0xe1 1 0% 100% 0.00 304 0xfffffe30e91a4f38 tcp_output+0x18d 1 0% 100% 0.00 573 0xfffffe30e73e7e70 ip_xmit+0x76c 1 0% 100% 0.00 463 0xfffffe30e5751978 stream_runservice+0x79 1 0% 100% 0.00 1545 0xfffffe30e6a527c0 conn_clrqfull+0x59 1 0% 100% 0.00 335 0xfffffe30e47a2f38 tcp_output+0x18d 1 0% 100% 0.00 393 0xfffffe30dacfa250 lwp_unpark+0x54 1 0% 100% 0.00 909 0xfffffe30da592d30 lwp_park+0xfb 1 0% 100% 0.00 902 0xfffffe30cf563200 mac_rx_soft_ring_drain+0x98 1 0% 100% 0.00 430 0xfffffe30d9ce5dc0 mac_rx_soft_ring_drain+0x98 1 0% 100% 0.00 505 0xfffffe2d1dde1af8 dbuf_find+0x94 1 0% 100% 0.00 2729 0xfffffe2d1cce4308 do_userquota_cacheflush+0x1c1 1 0% 100% 0.00 417 0xfffffe2d1cce4308 do_userquota_cacheflush+0x72 1 0% 100% 0.00 435 0xfffffe2d1cc72d00 taskq_dispatch_ent+0x51 1 0% 100% 0.00 1231 0xfffffe2d1ec2a2d8 poll_common+0x144 1 0% 100% 0.00 814 0xfffffe2f7cfb5000 mac_rx_srs_drain+0x22d 1 0% 100% 0.00 1811 0xfffffe2f7cfb66c0 mac_rx_srs_poll_ring+0x173 1 0% 100% 0.00 388 0xfffffe2ca3c9fd80 xbuf_iostart+0x118 1 0% 100% 0.00 2242 0xfffffe2c9d678ec0 mac_tx+0x27d 1 0% 100% 0.00 2325 0xfffffe2c9d678f40 mac_tx+0x27d 1 0% 100% 0.00 523 0xfffffe2ce04c5bc8 mlxcx_intr_n+0x10b 1 0% 100% 0.00 363 0xfffffe2ce04c5688 mlxcx_intr_n+0x10b 1 0% 100% 0.00 3137 0xfffffe2d19513de8 cv_wait+0x78 1 0% 100% 0.00 1259 0xfffffe2d195b3898 zio_remove_child+0x47 1 0% 100% 0.00 639 0xfffffe2d1c7bf228 dbuf_rele+0x27 1 0% 100% 0.00 1413 0xfffffe2d1cc721c0 taskq_dispatch+0x174 1 0% 100% 0.00 382 0xfffffe2c8e4e4540 callout_execute+0x18 1 0% 100% 0.00 603 0xfffffe2c8e4e4540 untimeout_generic+0x65 1 0% 100% 0.00 7358 0xfffffe2c8e4e4840 timeout_generic+0x72 1 0% 100% 0.00 902 0xfffffe2c868a03c0 taskq_d_thread+0x21a 1 0% 100% 0.00 2865 0xfffffe2c868a03c0 taskq_bucket_dispatch+0x32 1 0% 100% 0.00 289 0xfffffe2c8e4e4300 timeout_generic+0x72 1 0% 100% 0.00 340 0xfffffe2c8e4e40c0 callout_execute+0x18 1 0% 100% 0.00 978 0xfffffe2c8e4e43c0 callout_execute+0x18 1 0% 100% 0.00 329 0xfffffe2c8e4e4540 timeout_generic+0x72 1 0% 100% 0.00 27541 0xfffffe2c868a0040 cv_timedwait_hires+0xf0 1 0% 100% 0.00 477 0xfffffe2c867cd158 zio_wait_for_children+0x3b 1 0% 100% 0.00 2446 0xfffffe2c863ad400 kmem_cache_free+0x78 1 0% 100% 0.00 1089 0xfffffe2c868a0240 taskq_bucket_dispatch+0x32 1 0% 100% 0.00 3444 0xfffffe2c868a00c0 cv_timedwait_hires+0xf0 1 0% 100% 0.00 1335 0xfffffe2c868a00c0 taskq_d_thread+0x21a 1 0% 100% 0.00 440 0xfffffe2c868a02c0 taskq_bucket_dispatch+0x32 1 0% 100% 0.00 3913 0xfffffe2c860da3c0 kmem_cache_alloc+0x3d 1 0% 100% 0.00 1343 0xfffffe003c58e218 page_get_mnode_freelist+0x12b 1 0% 100% 0.00 40806 0xfffffe003c58e200 page_get_mnode_freelist+0x12b 1 0% 100% 0.00 556 0xfffffe003c58e1c8 page_get_mnode_freelist+0x12b 1 0% 100% 0.00 456 0xfffffe003c5945f8 page_ctr_add+0x6b 1 0% 100% 0.00 339 0xfffffe2c860cc1d0 kmem_depot_free+0x35 1 0% 100% 0.00 2592 0xfffffe2c860da400 kmem_cache_alloc+0x3d 1 0% 100% 0.00 563 0xfffffe2c863ad1d0 kmem_depot_free+0x35 1 0% 100% 0.00 2090 0xfffffe2c863ad280 kmem_cache_free+0x78 1 0% 100% 0.00 4374 0xfffffe2c91541a60 cv_wait+0x78 1 0% 100% 0.00 1398 0xfffffe2c91541940 taskq_thread+0x2cd 1 0% 100% 0.00 7360 0xfffffe2c91443c80 cpudrv_monitor+0x3a 1 0% 100% 0.00 1638 0xfffffe2c90ecc980 compute_min_max_threads+0x14 1 0% 100% 0.00 944 0xfffffe2c91377958 cpudrv_monitor+0x3a 1 0% 100% 0.00 338 0xfffffe2c90eabad0 cv_wait+0x78 1 0% 100% 0.00 2119 0xfffffe2c8edfa268 taskq_dispatch_ent+0x51 1 0% 100% 0.00 584 0xfffffe2c8ef43340 sigaction32+0x9f 1 0% 100% 0.00 2367 0xfffffe2c8ef43340 cv_wait_sig+0x166 1 0% 100% 0.00 1279 0xfffffe2c9857f488 igb_tx+0x2f2 1 0% 100% 0.00 2207 0xfffffe2c98299d00 metaslab_class_throttle_unreserve+0x1d 1 0% 100% 0.00 1233 0xfffffe2c98499000 metaslab_class_evict_old+0xec 1 0% 100% 0.00 569 0xfffffe2c9818ace0 dsl_pool_mos_diduse_space+0x47 1 0% 100% 0.00 1861 0xfffffe2c97dfb800 metaslab_group_alloc_normal+0x1b4 1 0% 100% 0.00 364 0xfffffe2c91a21800 metaslab_free_concrete+0xc3 1 0% 100% 0.00 9506 0xfffffe2c91541b80 cv_wait+0x78 1 0% 100% 0.00 5125 0xfffffe2c97dfb800 metaslab_free_concrete+0xc3 1 0% 100% 0.00 1724 0xfffffe2c9d678e40 mac_tx+0x27d ------------------------------------------------------------------------------- Adaptive mutex block: 15190 events in 20.818 seconds (730 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 4711 31% 31% 0.00 10027 0xfffffe2c90b24680 hook_run+0x34 1635 11% 42% 0.00 4259 0xfffffe2cdda1bef8 mlxcx_process_cq+0x207 1552 10% 52% 0.00 9662 0xfffffe2c90b24680 hook_run+0xd1 949 6% 58% 0.00 13225 0xfffffe2f6d891280 cv_wait+0x78 817 5% 64% 0.00 4308 0xfffffe2cdbbaa378 mlxcx_process_cq+0x207 562 4% 67% 0.00 10928 0xfffffe2f7cfb5000 cv_wait+0x78 498 3% 71% 0.00 2899 0xfffffe2c860cc280 kmem_cache_free+0x78 478 3% 74% 0.00 10157 0xfffffe2f7cfb66c0 cv_wait+0x78 370 2% 76% 0.00 4269 0xfffffe2cdda1b2b8 mlxcx_process_cq+0x207 358 2% 79% 0.00 22181 0xfffffe2cdda29eb8 mlxcx_process_cq+0x207 272 2% 80% 0.00 4781 0xfffffe2c860e0340 kmem_cache_free+0x78 260 2% 82% 0.00 5861 0xfffffe2cdda2d350 mlxcx_buf_return+0x8c 252 2% 84% 0.00 3498 0xfffffe2c860cc400 kmem_cache_free+0x78 207 1% 85% 0.00 4994 0xfffffe2c860e0280 kmem_cache_free+0x78 186 1% 86% 0.00 5744 0xfffffe2cdbbb0480 mlxcx_buf_return+0x8c 183 1% 87% 0.00 10698 0xfffffe2c90eabad0 squeue_enter+0x37a 168 1% 89% 0.00 6913 0xfffffe2cdda2dd50 mlxcx_buf_return+0x8c 148 1% 90% 0.00 12523 0xfffffe30cf563200 mac_rx_soft_ring_process+0x53 132 1% 90% 0.00 11296 0xfffffe2c90eabb90 squeue_enter+0x37a 121 1% 91% 0.00 7155 0xfffffe2c860e0400 kmem_cache_free+0x78 113 1% 92% 0.00 4538 0xfffffe2c860e0240 kmem_cache_free+0x78 83 1% 93% 0.00 5745 0xfffffe2cdda16a18 mlxcx_buf_return+0x8c 78 1% 93% 0.00 17272 0xfffffe2c90eabad0 squeue_drain+0x18b 74 0% 94% 0.00 7064 0xfffffe2c90eabad0 squeue_enter+0x53 72 0% 94% 0.00 80574 0xfffffe30d9ce5dc0 cv_wait+0x78 70 0% 94% 0.00 17068 0xfffffe30d9ce6c00 cv_wait+0x78 67 0% 95% 0.00 9953 0xfffffe30d9ce6780 mac_rx_soft_ring_process+0x53 59 0% 95% 0.00 8986 0xfffffe30d9ce6c00 mac_rx_soft_ring_process+0x53 36 0% 96% 0.00 5336 0xfffffe30cf563200 cv_wait+0x78 34 0% 96% 0.00 10864 0xfffffe2c90eabb90 squeue_drain+0x18b 31 0% 96% 0.00 3710 0xfffffe2c863ad240 kmem_cache_alloc+0x3d 29 0% 96% 0.00 6041 0xfffffe30d9ce6780 cv_wait+0x78 27 0% 96% 0.00 31545 0xfffffe2d1cc72760 cv_wait+0x78 26 0% 96% 0.00 13534 0xfffffe2d1cc72760 taskq_dispatch_ent+0x51 23 0% 97% 0.00 12222 sqset_lock ip_squeue_random+0x22 22 0% 97% 0.00 10563 0xfffffe30e47a2840 squeue_enter+0x4cd 22 0% 97% 0.00 9436 0xfffffe2c90eabb90 squeue_enter+0x53 20 0% 97% 0.00 9095 0xfffffe2c860e03c0 kmem_cache_free+0x78 19 0% 97% 0.00 5574 0xfffffe2c863ad3c0 kmem_cache_free+0x78 19 0% 97% 0.00 7147 0xfffffe2c860e0380 kmem_cache_free+0x78 18 0% 97% 0.00 10319 0xfffffe30e6a527c0 squeue_enter+0x4cd 18 0% 98% 0.00 11802 0xfffffe30e47a2840 squeue_drain+0x255 16 0% 98% 0.00 8067 0xfffffe2d19355c08 i_dls_head_hold+0x1f 15 0% 98% 0.00 13492 0xfffffe30e91a4840 squeue_drain+0x255 15 0% 98% 0.00 11556 0xfffffe2dae54ea30 dls_accept_common+0x80 15 0% 98% 0.00 62547 0xfffffe2c9aa60e88 vdev_queue_io+0x66 14 0% 98% 0.00 5018 0xfffffe2c90eabd10 squeue_enter+0x53 13 0% 98% 0.00 70604 overlay_target_lock cv_timedwait_hires+0xf0 12 0% 98% 0.00 6159 0xfffffe30e833f840 ipcl_classify_v4+0x25d 12 0% 98% 0.00 10729 0xfffffe30e6a527c0 squeue_drain+0x255 12 0% 98% 0.00 8652 0xfffffe2d19355c08 i_dls_head_rele+0x18 11 0% 98% 0.00 7689 0xfffffe30e6a527c0 ipcl_classify_v4+0x25d 11 0% 99% 0.00 8342 0xfffffe2f7cfb66c0 mac_rx_soft_ring_drain+0x115 11 0% 99% 0.00 30538 0xfffffe2d1cc72760 taskq_thread+0x2cd 11 0% 99% 0.00 7571 0xfffffe2cdf6065c8 mlxcx_buf_loan+0x76 10 0% 99% 0.00 32564 overlay_target_lock overlay_target_lookup_request+0x47 9 0% 99% 0.00 6499 0xfffffe30e47a2840 ipcl_classify_v4+0x25d 8 0% 99% 0.00 6370 0xfffffe2f8c709200 cv_wait+0x78 8 0% 99% 0.00 5563 0xfffffe2c863ad3c0 kmem_cache_alloc+0x3d 7 0% 99% 0.00 10170 0xfffffe30e91a4840 squeue_enter+0x4cd 7 0% 99% 0.00 4918 0xfffffe30e91a4840 ipcl_classify_v4+0x25d 7 0% 99% 0.00 6572 0xfffffe2d1ec2a2d8 cv_timedwait_sig_hires+0x2f1 7 0% 99% 0.00 65145 0xfffffe2c9aa60e88 vdev_queue_io_done+0x30 6 0% 99% 0.00 9300 0xfffffe30e6a527c0 tcp_sendmsg+0xb8 6 0% 99% 0.00 6666 0xfffffe2cdf6065c8 mlxcx_buf_return+0x8c 6 0% 99% 0.00 5134 0xfffffe2c863ad340 kmem_cache_alloc+0x3d 6 0% 99% 0.00 7688 0xfffffe2c860e01d0 kmem_depot_alloc+0x88 5 0% 99% 0.00 39913 zone_pdata zfs_zone_zio_done+0xa9 5 0% 99% 0.00 10907 0xfffffe30e47a2840 tcp_sendmsg+0xb8 5 0% 99% 0.00 7003 0xfffffe2cdf6063e8 mlxcx_buf_loan+0x76 4 0% 99% 0.00 6148 0xfffffe30e6a7d7a8 ire_route_recursive_dstonly_v4+0x68 4 0% 99% 0.00 6171 0xfffffe2d1cc73758 cv_wait+0x78 4 0% 99% 0.00 4799 0xfffffe2cdf606ca8 mlxcx_buf_loan+0x76 4 0% 99% 0.00 5409 0xfffffe2c860da340 kmem_cache_alloc+0x3d 4 0% 100% 0.00 4677 0xfffffe2c860cc340 kmem_cache_free+0x78 3 0% 100% 0.00 7800 0xfffffe30e91a4f38 tcp_sendmsg+0x83 3 0% 100% 0.00 15090 0xfffffe2cdda2dd50 mlxcx_buf_take+0x2f 3 0% 100% 0.00 7957 0xfffffe2cdbbb0480 mlxcx_buf_take+0x2f 3 0% 100% 0.00 56625 0xfffffe2c9aa60e88 vdev_queue_io_done+0x75 3 0% 100% 0.00 46071 0xfffffe2c97ee7000 metaslab_free_concrete+0xc3 3 0% 100% 0.00 12832 0xfffffe2c860e02c0 kmem_cache_free+0x78 2 0% 100% 0.00 5005 0xfffffe30e91a4f38 tcp_wput_data+0x401 2 0% 100% 0.00 5347 0xfffffe30e833f840 squeue_enter+0x4cd 2 0% 100% 0.00 6504 0xfffffe30e47a2f38 tcp_sendmsg+0x83 2 0% 100% 0.00 69241 0xfffffe2f7cfb66c0 mac_rx_srs_process+0x5c 2 0% 100% 0.00 299755 0xfffffe2f8c709200 mac_rx_soft_ring_drain+0x115 2 0% 100% 0.00 12907 0xfffffe2f8c709200 mac_rx_srs_process+0x5c 2 0% 100% 0.00 5565 0xfffffe2d1cc73758 taskq_thread+0x2cd 2 0% 100% 0.00 5538 0xfffffe2d1ec2a2d8 pollwakeup+0x193 2 0% 100% 0.00 11091 0xfffffe2d193b0a00 getf_gen+0x63 2 0% 100% 0.00 5940 0xfffffe2cdf606ca8 mlxcx_buf_return+0x8c 2 0% 100% 0.00 9066 0xfffffe2cdf606528 mlxcx_buf_loan+0x76 2 0% 100% 0.00 14286 0xfffffe2cdda16a18 mlxcx_buf_take+0x2f 2 0% 100% 0.00 8610 0xfffffe2c863ad2c0 kmem_cache_alloc+0x3d 2 0% 100% 0.00 5860 0xfffffe2c860da280 kmem_cache_alloc+0x3d 1 0% 100% 0.00 10235 0xfffffe30e91a4840 tcp_sendmsg+0xb8 1 0% 100% 0.00 14631 0xfffffe30e91a4840 conn_clrqfull+0x59 1 0% 100% 0.00 10739 0xfffffe30e833f840 tcp_sendmsg+0xb8 1 0% 100% 0.00 23252 0xfffffe30e6a527c0 conn_clrqfull+0x59 1 0% 100% 0.00 7542 0xfffffe2d1c639aa8 cv_wait+0x78 1 0% 100% 0.00 7329 0xfffffe2d19513de8 cv_wait+0x78 1 0% 100% 0.00 8924 0xfffffe2d1c639aa8 taskq_dispatch_ent+0x51 1 0% 100% 0.00 47485 0xfffffe2cdf6063e8 mlxcx_buf_return+0x8c 1 0% 100% 0.00 7548 0xfffffe2cdda2d350 mlxcx_buf_take+0x2f 1 0% 100% 0.00 7047 0xfffffe2cdf60b280 mlxcx_mac_ring_rx_poll+0x34 1 0% 100% 0.00 13395 0xfffffe2c989fecc0 ahci_flush_doneq+0x78 1 0% 100% 0.00 5649 0xfffffe2c989fecc0 ahci_port_intr+0x62 1 0% 100% 0.00 9135 0xfffffe2c98299d00 metaslab_class_throttle_unreserve+0x1d 1 0% 100% 0.00 16158 0xfffffe2c989fecc0 ahci_tran_start+0x75 1 0% 100% 0.00 47287 0xfffffe2c97f17800 metaslab_free_concrete+0xc3 1 0% 100% 0.00 4207 0xfffffe2c9d678f40 mac_tx+0x27d 1 0% 100% 0.00 48362 0xfffffe2c9d678ec0 mac_tx+0x27d 1 0% 100% 0.00 2780 0xfffffe2c9d678e40 mac_tx+0x27d 1 0% 100% 0.00 40998 0xfffffe2c9aa60e88 vdev_queue_aggregate+0x43c 1 0% 100% 0.00 11175 0xfffffe2cdda1b2b8 mlxcx_sq_add_buffer+0x469 1 0% 100% 0.00 7591 0xfffffe2c860da3c0 kmem_cache_alloc+0x3d 1 0% 100% 0.00 12076 0xfffffe2c860da400 kmem_cache_alloc+0x3d 1 0% 100% 0.00 10483 0xfffffe2c863ad280 kmem_cache_free+0x78 1 0% 100% 0.00 6869 0xfffffe2c863ad400 kmem_cache_free+0x78 1 0% 100% 0.00 3623 0xfffffe2c868a02c0 cv_timedwait_hires+0xf0 1 0% 100% 0.00 4652 0xfffffe2c868a00c0 cv_timedwait_hires+0xf0 1 0% 100% 0.00 9465 0xfffffe2c8e4e4600 untimeout_generic+0x65 1 0% 100% 0.00 9656 0xfffffe2c8ef43040 stop+0x2eb ------------------------------------------------------------------------------- Spin lock spin: 12172 events in 20.818 seconds (585 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 2889 24% 24% 0.00 1792 0xfffffe2c90ec23e8 disp_lock_enter+0x26 2051 17% 41% 0.00 1929 turnstile_table+0xc38 disp_lock_enter+0x26 1944 16% 57% 0.00 3110 0xfffffe2c9135d400 disp_lock_enter+0x26 1484 12% 69% 0.00 4030 cpu0_disp disp_lock_enter+0x26 1105 9% 78% 0.00 5419 cp_default disp_lock_enter+0x26 758 6% 84% 0.00 3434 0xfffffe2c9128eb28 disp_lock_enter+0x26 506 4% 88% 0.00 2214 0xfffffe2c9136c3f8 disp_lock_enter+0x26 394 3% 91% 0.00 5532 0xfffffe2c9135dba8 disp_lock_enter+0x26 267 2% 94% 0.00 2347 0xfffffe2c9136cb68 disp_lock_enter+0x26 142 1% 95% 0.00 8538 cp_default disp_lock_enter_high+0x11 136 1% 96% 0.00 2591 0xfffffe2c90ec23e8 disp_lock_enter_high+0x11 126 1% 97% 0.00 2325 0xfffffe2c9128e5b0 disp_lock_enter+0x26 74 1% 98% 0.00 1290 0xfffffe2c9135d400 disp_lock_enter_high+0x11 37 0% 98% 0.00 2567 cpu0_disp disp_lock_enter_high+0x11 36 0% 98% 0.00 5154 0xfffffe2c98394008 disp_lock_enter_high+0x11 31 0% 98% 0.00 3864 turnstile_table+0xeb8 disp_lock_enter+0x26 23 0% 99% 0.00 488 0xfffffe2c9135dba8 disp_lock_enter_high+0x11 20 0% 99% 0.00 893 0xfffffe2c9128eb28 disp_lock_enter_high+0x11 17 0% 99% 0.00 1071 0xfffffe2c9136c3f8 disp_lock_enter_high+0x11 16 0% 99% 0.00 11455 turnstile_table+0xab8 disp_lock_enter+0x26 16 0% 99% 0.00 1100 turnstile_table+0xa98 disp_lock_enter+0x26 14 0% 99% 0.00 28788 0xfffffe2c8699c2f1 mutex_vector_enter+0x431 12 0% 99% 0.00 381 turnstile_table+0xa88 disp_lock_enter+0x26 12 0% 99% 0.00 559 turnstile_table+0x918 disp_lock_enter+0x26 7 0% 100% 0.00 1102 turnstile_table+0xc28 disp_lock_enter+0x26 7 0% 100% 0.00 808 0xfffffe2c9128e5b0 disp_lock_enter_high+0x11 6 0% 100% 0.00 607 turnstile_table+0xc18 disp_lock_enter+0x26 5 0% 100% 0.00 336 turnstile_table+0xbd8 disp_lock_enter+0x26 4 0% 100% 0.00 472 turnstile_table+0xbb8 disp_lock_enter+0x26 4 0% 100% 0.00 337 turnstile_table+0xa28 disp_lock_enter+0x26 3 0% 100% 0.00 13835 turnstile_table+0xf18 disp_lock_enter+0x26 3 0% 100% 0.00 5232 turnstile_table+0xb88 disp_lock_enter+0x26 3 0% 100% 0.00 4547 turnstile_table+0x858 disp_lock_enter+0x26 3 0% 100% 0.00 1129 0xfffffe2c9136cb68 disp_lock_enter_high+0x11 2 0% 100% 0.00 633 turnstile_table+0x998 disp_lock_enter+0x26 2 0% 100% 0.00 279 turnstile_table+0x808 disp_lock_enter+0x26 2 0% 100% 0.00 356 sleepq_head+0x6868 disp_lock_enter_high+0x11 1 0% 100% 0.00 562 turnstile_table+0xf78 disp_lock_enter+0x26 1 0% 100% 0.00 306 turnstile_table+0xf58 disp_lock_enter+0x26 1 0% 100% 0.00 440 turnstile_table+0xf48 disp_lock_enter+0x26 1 0% 100% 0.00 4598 turnstile_table+0xc98 disp_lock_enter+0x26 1 0% 100% 0.00 423 turnstile_table+0xe58 disp_lock_enter+0x26 1 0% 100% 0.00 461 turnstile_table+0xb48 disp_lock_enter+0x26 1 0% 100% 0.00 430 turnstile_table+0xaf8 disp_lock_enter+0x26 1 0% 100% 0.00 490 turnstile_table+0xb08 disp_lock_enter+0x26 1 0% 100% 0.00 471 turnstile_table+0x938 disp_lock_enter+0x26 1 0% 100% 0.00 345 turnstile_table+0x9d8 disp_lock_enter+0x26 1 0% 100% 0.00 438 turnstile_table+0xa48 disp_lock_enter+0x26 ------------------------------------------------------------------------------- Thread lock spin: 970 events in 20.818 seconds (47 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 224 23% 23% 0.00 433 sleepq_head+0x6868 setrun+0x18 173 18% 41% 0.00 440 cpu[6]+0xe8 post_syscall+0x392 155 16% 57% 0.00 465 cpu[5]+0xe8 post_syscall+0x392 124 13% 70% 0.00 423 cpu[3]+0xe8 post_syscall+0x392 60 6% 76% 0.00 257 cpu[3]+0xe8 fss_tick+0x24 55 6% 82% 0.00 199 cpu[6]+0xe8 fss_tick+0x24 47 5% 86% 0.00 200 cpu[5]+0xe8 fss_tick+0x24 29 3% 89% 0.00 583 cpu[1]+0xe8 post_syscall+0x392 14 1% 91% 0.00 220 cpu[1]+0xe8 fss_tick+0x24 12 1% 92% 0.00 198 0xfffffe2c90ec23e8 fss_tick+0x24 11 1% 93% 0.00 774 turnstile_table+0xc38 fss_tick+0x24 7 1% 94% 0.00 253 cpu[2]+0xe8 fss_tick+0x24 6 1% 95% 0.00 921 cpu[5]+0xe8 preempt+0x47 5 1% 95% 0.00 250 cpu[4]+0xe8 post_syscall+0x392 5 1% 96% 0.00 709 cpu[2]+0xe8 post_syscall+0x392 5 1% 96% 0.00 526 cpu[3]+0xe8 turnstile_block+0x91 5 1% 97% 0.00 776 cpu[3]+0xe8 preempt+0x47 4 0% 97% 0.00 448 sleepq_head+0x1a88 setrun+0x18 4 0% 97% 0.00 199 transition_lock fss_tick+0x24 3 0% 98% 0.00 169 cpu[0]+0xe8 fss_tick+0x24 3 0% 98% 0.00 347 cpu[7]+0xe8 fss_tick+0x24 2 0% 98% 0.00 879 turnstile_table+0xc28 fss_tick+0x24 2 0% 98% 0.00 722 cpu[7]+0xe8 turnstile_block+0x91 2 0% 99% 0.00 200 0xfffffe2c9136cb68 fss_tick+0x24 2 0% 99% 0.00 806 cpu[1]+0xe8 preempt+0x47 1 0% 99% 0.00 410 sleepq_head+0x3cf8 fss_tick+0x24 1 0% 99% 0.00 138 cpu0_disp fss_tick+0x24 1 0% 99% 0.00 167 cp_default fss_tick+0x24 1 0% 99% 0.00 201 cpu[6]+0xe8 fss_update_list+0x53 1 0% 99% 0.00 804 cpu[6]+0xe8 preempt+0x47 1 0% 99% 0.00 254 cpu[1]+0xe8 turnstile_pi_waive+0x1b 1 0% 100% 0.00 2953 cpu[2]+0xe8 cv_timedwait_hires+0xaa 1 0% 100% 0.00 406 cpu[4]+0xe8 fss_tick+0x24 1 0% 100% 0.00 434 cpu[4]+0xe8 turnstile_pi_waive+0x1b 1 0% 100% 0.00 211 cpu[4]+0xe8 turnstile_block+0x91 1 0% 100% 0.00 653 0xfffffe2c9135d400 fss_tick+0x24 -------------------------------------------------------------------------------