Print this page
NEX-20178 Heavy read load using 10G i40e causes network disconnect
MFV illumos-joyent@83a8d0d616db36010b59cc850d1926c0f6a30de1
OS-7457 i40e Tx freezes on zero descriptors
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Rob Johnston <rob.johnston@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
MFV illumos-joyent@0d3f2b61dcfb18edace4fd257054f6fdbe07c99c
OS-7492 i40e Tx freeze when b_cont chain exceeds 8 descriptors
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Rob Johnston <rob.johnston@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
MFV illumos-joyent@b4bede175d4c50ac1b36078a677b69388f6fb59f
OS-7577 initialize FC for i40e
Reviewed by: Robert Mustacchi <rm@joyent.com>
Approved by: Rob Johnston <rob.johnston@joyent.com>
MFV illumos-joyent@83a8d0d616db36010b59cc850d1926c0f6a30de1
OS-7457 i40e Tx freezes on zero descriptors
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Rob Johnston <rob.johnston@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
MFV: illumos-joyent@61dc3dec4f82a3e13e94609a0a83d5f66c64e760
OS-6846 want i40e multi-group support
OS-7372 i40e_alloc_ring_mem() unwinds when it shouldn't
Reviewed by: Robert Mustacchi <rm@joyent.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: Ryan Zezeski <rpz@joyent.com>
MFV: illumos-joyent@757454db6669c1186f60bc625510c1b67217aae6
OS-7082 i40e: blown assert in i40e_tx_cleanup_ring()
OS-7086 i40e: add mdb dcmd to dump info on tx descriptor rings
OS-7101 i40e: add kstat to track TX DMA bind failures
Reviewed by: Ryan Zezeski <rpz@joyent.com>
Reviewed by: Robert Mustacchi <rm@joyent.com>
Approved by: Patrick Mooney <patrick.mooney@joyent.com>
Author: Rob Johnston <rob.johnston@joyent.com>
MFV: illumos-joyent@9e30beee2f0c127bf41868db46257124206e28d6
OS-5225 Want Fortville TSO support
Reviewed by: Ryan Zezeski <rpz@joyent.com>
Reviewed by: Robert Mustacchi <rm@joyent.com>
Approved by: Patrick Mooney <patrick.mooney@joyent.com>
Author: Rob Johnston <rob.johnston@joyent.com>

Split Close
Expand all
Collapse all
          --- old/usr/src/uts/common/io/i40e/i40e_stats.c
          +++ new/usr/src/uts/common/io/i40e/i40e_stats.c
↓ open down ↓ 3 lines elided ↑ open up ↑
   4    4   * You may only use this file in accordance with the terms of version
   5    5   * 1.0 of the CDDL.
   6    6   *
   7    7   * A full copy of the text of the CDDL should have accompanied this
   8    8   * source.  A copy of the CDDL is also available via the Internet at
   9    9   * http://www.illumos.org/license/CDDL.
  10   10   */
  11   11  
  12   12  /*
  13   13   * Copyright 2015 OmniTI Computer Consulting, Inc. All rights reserved.
  14      - * Copyright 2016 Joyent, Inc.
       14 + * Copyright 2019 Joyent, Inc.
  15   15   */
  16   16  
  17   17  #include "i40e_sw.h"
  18   18  
  19   19  /*
  20   20   * -------------------
  21   21   * Statistics Overview
  22   22   * -------------------
  23   23   *
  24   24   * As part of managing the driver and understanding what's going on, we keep
↓ open down ↓ 37 lines elided ↑ open up ↑
  62   62   * that these are labeled based on their local index, which may mean that
  63   63   * different instances have overlapping sets of queues. This isn't a problem as
  64   64   * the kstats will always use the instance number of the pf to distinguish it in
  65   65   * the kstat tuple.
  66   66   *
  67   67   * ---------------------
  68   68   * Hardware Arrangements
  69   69   * ---------------------
  70   70   *
  71   71   * The hardware keeps statistics at each physical function/MAC (PF) and it keeps
  72      - * statistics on each virtual station interface (VSI). Currently we only use one
  73      - * VSI per PF (see the i40e_main.c theory statement). The hardware has a limited
  74      - * number of statistics units available. While every PF is guaranteed to have a
  75      - * statistics unit, it is possible that we will run out for a given VSI. We'll
  76      - * have to figure out an appropriate strategy here when we end up supporting
  77      - * multiple VSIs.
       72 + * statistics on each virtual station interface (VSI).
  78   73   *
  79   74   * The hardware keeps these statistics as 32-bit and 48-bit counters. We are
  80   75   * required to read them and then compute the differences between them. The
  81   76   * 48-bit counters span more than one 32-bit register in the BAR. The hardware
  82   77   * suggests that to read them, we perform 64-bit reads of the lower of the two
  83   78   * registers that make up a 48-bit stat. The hardware guarantees that the reads
  84   79   * of those two registers will be atomic and we'll get a consistent value, not a
  85   80   * property it has for every read of two registers.
  86   81   *
  87   82   * For every kstat we have based on this, we have a corresponding uint64_t that
↓ open down ↓ 5 lines elided ↑ open up ↑
  93   88   * i40e_stat_get_uint48().
  94   89   *
  95   90   * The only unfortunate thing here is that the hardware doesn't give us any kind
  96   91   * of overflow counter. It just tries to make sure that the uint32_t and
  97   92   * uint48_t counters are large enough to hopefully not overflow right away. This
  98   93   * isn't the most reassuring statement and we should investigate ways of
  99   94   * ensuring that if a system is active, but not actively measured, we don't lose
 100   95   * data.
 101   96   *
 102   97   * The pf kstats data is stored in the i40e_t`i40e_pf_kstat. It is backed by the
 103      - * i40e_t`i40e_pf_stat structure. Similarly the VSI related kstat is in
 104      - * i40e_t`i40e_vsi_kstat and the data is backed in the i40e_t`i40e_vsi_stat. All
 105      - * of this data is protected by the i40e_stat_lock, which should be taken last,
 106      - * when acquiring locks.
       98 + * i40e_t`i40e_pf_stat structure. Similarly the VSI related kstats are in
       99 + * i40e_t`i40e_vsis[idx].iv_kstats and the data is backed in the
      100 + * i40e_t`i40e_vsis[idx].iv_stats. All of this data is protected by the
      101 + * i40e_stat_lock, which should be taken last, when acquiring locks.
 107  102   */
 108  103  
 109  104  static void
 110  105  i40e_stat_get_uint48(i40e_t *i40e, uintptr_t reg, kstat_named_t *kstat,
 111  106      uint64_t *base, boolean_t init)
 112  107  {
 113  108          i40e_hw_t *hw = &i40e->i40e_hw_space;
 114  109          uint64_t raw, delta;
 115  110  
 116  111          ASSERT(MUTEX_HELD(&i40e->i40e_stat_lock));
↓ open down ↓ 45 lines elided ↑ open up ↑
 162  157          } else {
 163  158                  delta = 0x100000000ULL - *base + raw;
 164  159          }
 165  160  
 166  161          kstat->value.ui64 += delta;
 167  162          *base = raw;
 168  163  
 169  164  }
 170  165  
 171  166  static void
 172      -i40e_stat_vsi_update(i40e_t *i40e, boolean_t init)
      167 +i40e_stat_vsi_update(i40e_t *i40e, uint_t idx, boolean_t init)
 173  168  {
 174  169          i40e_vsi_stats_t *ivs;
 175  170          i40e_vsi_kstats_t *ivk;
 176      -        int id = i40e->i40e_vsi_stat_id;
      171 +        uint16_t id = i40e->i40e_vsis[idx].iv_stats_id;
 177  172  
 178      -        ASSERT(i40e->i40e_vsi_kstat != NULL);
 179      -        ivs = &i40e->i40e_vsi_stat;
 180      -        ivk = i40e->i40e_vsi_kstat->ks_data;
      173 +        ASSERT3P(i40e->i40e_vsis[idx].iv_kstats, !=, NULL);
      174 +        ivs = &i40e->i40e_vsis[idx].iv_stats;
      175 +        ivk = i40e->i40e_vsis[idx].iv_kstats->ks_data;
 181  176  
 182  177          mutex_enter(&i40e->i40e_stat_lock);
 183  178  
 184  179          i40e_stat_get_uint48(i40e, I40E_GLV_GORCL(id), &ivk->ivk_rx_bytes,
 185  180              &ivs->ivs_rx_bytes, init);
 186  181          i40e_stat_get_uint48(i40e, I40E_GLV_UPRCL(id), &ivk->ivk_rx_unicast,
 187  182              &ivs->ivs_rx_unicast, init);
 188  183          i40e_stat_get_uint48(i40e, I40E_GLV_MPRCL(id), &ivk->ivk_rx_multicast,
 189  184              &ivs->ivs_rx_multicast, init);
 190  185          i40e_stat_get_uint48(i40e, I40E_GLV_BPRCL(id), &ivk->ivk_rx_broadcast,
↓ open down ↓ 33 lines elided ↑ open up ↑
 224  219  
 225  220  static int
 226  221  i40e_stat_vsi_kstat_update(kstat_t *ksp, int rw)
 227  222  {
 228  223          i40e_t *i40e;
 229  224  
 230  225          if (rw == KSTAT_WRITE)
 231  226                  return (EACCES);
 232  227  
 233  228          i40e = ksp->ks_private;
 234      -        i40e_stat_vsi_update(i40e, B_FALSE);
      229 +        for (uint_t i = 0; i < i40e->i40e_num_rx_groups; i++)
      230 +                i40e_stat_vsi_update(i40e, i, B_FALSE);
      231 +
 235  232          return (0);
 236  233  }
 237  234  
 238  235  void
 239      -i40e_stat_vsi_fini(i40e_t *i40e)
      236 +i40e_stat_vsi_fini(i40e_t *i40e, uint_t idx)
 240  237  {
 241      -        if (i40e->i40e_vsi_kstat != NULL) {
 242      -                kstat_delete(i40e->i40e_vsi_kstat);
 243      -                i40e->i40e_vsi_kstat = NULL;
      238 +        if (i40e->i40e_vsis[idx].iv_kstats != NULL) {
      239 +                kstat_delete(i40e->i40e_vsis[idx].iv_kstats);
      240 +                i40e->i40e_vsis[idx].iv_kstats = NULL;
 244  241          }
 245  242  }
 246  243  
 247  244  boolean_t
 248      -i40e_stat_vsi_init(i40e_t *i40e)
      245 +i40e_stat_vsi_init(i40e_t *i40e, uint_t idx)
 249  246  {
 250  247          kstat_t *ksp;
 251  248          i40e_vsi_kstats_t *ivk;
 252  249          char buf[64];
      250 +        uint16_t vsi_id = i40e->i40e_vsis[idx].iv_seid;
 253  251  
 254      -        (void) snprintf(buf, sizeof (buf), "vsi_%d", i40e->i40e_vsi_id);
      252 +        (void) snprintf(buf, sizeof (buf), "vsi_%u", vsi_id);
 255  253  
 256  254          ksp = kstat_create(I40E_MODULE_NAME, ddi_get_instance(i40e->i40e_dip),
 257  255              buf, "net", KSTAT_TYPE_NAMED,
 258  256              sizeof (i40e_vsi_kstats_t) / sizeof (kstat_named_t), 0);
 259  257  
 260  258          if (ksp == NULL) {
 261      -                i40e_error(i40e, "Failed to create kstats for VSI %d",
 262      -                    i40e->i40e_vsi_id);
      259 +                i40e_error(i40e, "Failed to create kstats for VSI %u", vsi_id);
 263  260                  return (B_FALSE);
 264  261          }
 265  262  
 266      -        i40e->i40e_vsi_kstat = ksp;
      263 +        i40e->i40e_vsis[idx].iv_kstats = ksp;
 267  264          ivk = ksp->ks_data;
 268  265          ksp->ks_update = i40e_stat_vsi_kstat_update;
 269  266          ksp->ks_private = i40e;
 270  267  
 271  268          kstat_named_init(&ivk->ivk_rx_bytes, "rx_bytes",
 272  269              KSTAT_DATA_UINT64);
 273  270          kstat_named_init(&ivk->ivk_rx_unicast, "rx_unicast",
 274  271              KSTAT_DATA_UINT64);
 275  272          kstat_named_init(&ivk->ivk_rx_multicast, "rx_multicast",
 276  273              KSTAT_DATA_UINT64);
↓ open down ↓ 7 lines elided ↑ open up ↑
 284  281              KSTAT_DATA_UINT64);
 285  282          kstat_named_init(&ivk->ivk_tx_unicast, "tx_unicast",
 286  283              KSTAT_DATA_UINT64);
 287  284          kstat_named_init(&ivk->ivk_tx_multicast, "tx_multicast",
 288  285              KSTAT_DATA_UINT64);
 289  286          kstat_named_init(&ivk->ivk_tx_broadcast, "tx_broadcast",
 290  287              KSTAT_DATA_UINT64);
 291  288          kstat_named_init(&ivk->ivk_tx_errors, "tx_errors",
 292  289              KSTAT_DATA_UINT64);
 293  290  
 294      -        bzero(&i40e->i40e_vsi_stat, sizeof (i40e_vsi_stats_t));
 295      -        i40e_stat_vsi_update(i40e, B_TRUE);
 296      -        kstat_install(i40e->i40e_vsi_kstat);
      291 +        bzero(&i40e->i40e_vsis[idx].iv_stats, sizeof (i40e_vsi_stats_t));
      292 +        i40e_stat_vsi_update(i40e, idx, B_TRUE);
      293 +        kstat_install(i40e->i40e_vsis[idx].iv_kstats);
 297  294  
 298  295          return (B_TRUE);
 299  296  }
 300  297  
 301  298  static void
 302  299  i40e_stat_pf_update(i40e_t *i40e, boolean_t init)
 303  300  {
 304  301          i40e_pf_stats_t *ips;
 305  302          i40e_pf_kstats_t *ipk;
 306  303          int port = i40e->i40e_hw_space.port;
↓ open down ↓ 356 lines elided ↑ open up ↑
 663  660          i40e_stat_pf_update(i40e, B_TRUE);
 664  661  
 665  662          kstat_install(i40e->i40e_pf_kstat);
 666  663  
 667  664          return (B_TRUE);
 668  665  }
 669  666  
 670  667  void
 671  668  i40e_stats_fini(i40e_t *i40e)
 672  669  {
 673      -        ASSERT(i40e->i40e_vsi_kstat == NULL);
      670 +#ifdef DEBUG
      671 +        for (uint_t i = 0; i < i40e->i40e_num_rx_groups; i++) {
      672 +                ASSERT3P(i40e->i40e_vsis[i].iv_kstats, ==, NULL);
      673 +        }
      674 +#endif
      675 +
 674  676          if (i40e->i40e_pf_kstat != NULL) {
 675  677                  kstat_delete(i40e->i40e_pf_kstat);
 676  678                  i40e->i40e_pf_kstat = NULL;
 677  679          }
 678  680  
 679  681          mutex_destroy(&i40e->i40e_stat_lock);
 680  682  }
 681  683  
 682  684  boolean_t
 683  685  i40e_stats_init(i40e_t *i40e)
↓ open down ↓ 539 lines elided ↑ open up ↑
1223 1225          tsp->itxs_bytes.value.ui64 = 0;
1224 1226          kstat_named_init(&tsp->itxs_packets, "tx_packets",
1225 1227              KSTAT_DATA_UINT64);
1226 1228          tsp->itxs_packets.value.ui64 = 0;
1227 1229          kstat_named_init(&tsp->itxs_descriptors, "tx_descriptors",
1228 1230              KSTAT_DATA_UINT64);
1229 1231          tsp->itxs_descriptors.value.ui64 = 0;
1230 1232          kstat_named_init(&tsp->itxs_recycled, "tx_recycled",
1231 1233              KSTAT_DATA_UINT64);
1232 1234          tsp->itxs_recycled.value.ui64 = 0;
     1235 +        kstat_named_init(&tsp->itxs_force_copy, "tx_force_copy",
     1236 +            KSTAT_DATA_UINT64);
     1237 +        tsp->itxs_force_copy.value.ui64 = 0;
     1238 +        kstat_named_init(&tsp->itxs_tso_force_copy, "tx_tso_force_copy",
     1239 +            KSTAT_DATA_UINT64);
     1240 +        tsp->itxs_tso_force_copy.value.ui64 = 0;
1233 1241  
1234 1242          kstat_named_init(&tsp->itxs_hck_meoifail, "tx_hck_meoifail",
1235 1243              KSTAT_DATA_UINT64);
1236 1244          tsp->itxs_hck_meoifail.value.ui64 = 0;
1237 1245          kstat_named_init(&tsp->itxs_hck_nol2info, "tx_hck_nol2info",
1238 1246              KSTAT_DATA_UINT64);
1239 1247          tsp->itxs_hck_nol2info.value.ui64 = 0;
1240 1248          kstat_named_init(&tsp->itxs_hck_nol3info, "tx_hck_nol3info",
1241 1249              KSTAT_DATA_UINT64);
1242 1250          tsp->itxs_hck_nol3info.value.ui64 = 0;
1243 1251          kstat_named_init(&tsp->itxs_hck_nol4info, "tx_hck_nol4info",
1244 1252              KSTAT_DATA_UINT64);
1245 1253          tsp->itxs_hck_nol4info.value.ui64 = 0;
1246 1254          kstat_named_init(&tsp->itxs_hck_badl3, "tx_hck_badl3",
1247 1255              KSTAT_DATA_UINT64);
1248 1256          tsp->itxs_hck_badl3.value.ui64 = 0;
1249 1257          kstat_named_init(&tsp->itxs_hck_badl4, "tx_hck_badl4",
1250 1258              KSTAT_DATA_UINT64);
1251 1259          tsp->itxs_hck_badl4.value.ui64 = 0;
     1260 +        kstat_named_init(&tsp->itxs_lso_nohck, "tx_lso_nohck",
     1261 +            KSTAT_DATA_UINT64);
     1262 +        tsp->itxs_lso_nohck.value.ui64 = 0;
     1263 +        kstat_named_init(&tsp->itxs_bind_fails, "tx_bind_fails",
     1264 +            KSTAT_DATA_UINT64);
     1265 +        tsp->itxs_bind_fails.value.ui64 = 0;
1252 1266          kstat_named_init(&tsp->itxs_err_notcb, "tx_err_notcb",
1253 1267              KSTAT_DATA_UINT64);
1254 1268          tsp->itxs_err_notcb.value.ui64 = 0;
1255 1269          kstat_named_init(&tsp->itxs_err_nodescs, "tx_err_nodescs",
1256 1270              KSTAT_DATA_UINT64);
1257 1271          tsp->itxs_err_nodescs.value.ui64 = 0;
1258 1272          kstat_named_init(&tsp->itxs_err_context, "tx_err_context",
1259 1273              KSTAT_DATA_UINT64);
1260 1274          tsp->itxs_err_context.value.ui64 = 0;
1261 1275          kstat_named_init(&tsp->itxs_num_unblocked, "tx_num_unblocked",
↓ open down ↓ 61 lines elided ↑ open up ↑
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX