Print this page
NEX-17397 Getting panic: kernel heap corruption detected when trying to create big count of iSCSI targets and mappings
Reviewed by: Evan Layton <evan.layton@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
Reviewed by: Dan Fields <dan.fields@nexenta.com>
Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
NEX-18878 Getting BAD TRAP panic when payload to iSCSI or FC shared zvol from client
    Reviewed by: tim Jacobson <tim.jacobson@nexenta.com>
    Reviewed by: Joyce McIntosh <joyce.mcintosh@nexenta.com>
    Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
    Reviewed by: Evan Layton <evan.layton@nexenta.com>
NEX-16937 Poor performance observed during large delete / SCSI UNMAP operations (Comstar portion)
NEX-16711 STMF task workers do not scale correctly
Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
NEX-15346 COMSTAR hang with thousands of threads waiting in idm_refcnt_wait_ref() from iscsit_conn_lost()
Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Dan Fields <dan.fields@nexenta.com>
NEX-15346 COMSTAR hang with thousands of threads waiting in idm_refcnt_wait_ref() from iscsit_conn_lost()
Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Dan Fields <dan.fields@nexenta.com>
Running stmfadm remove-hg-member caused a NULL pointer dereference panic in stmf_remove_lu_from_session
Reviewed by: Dan Fields <dan.fields@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
Running stmfadm remove-hg-member caused a NULL pointer dereference panic in stmf_remove_lu_from_session
Reviewed by: Dan Fields <dan.fields@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
NEX-9567 Deadlock: cycle in blocking chain panic
Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
NEX-7273 nbmand makes NFSv4 RENAME of an open file inoperable
Reviewed by: Matt Barden <matt.barden@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
NEX-7681 System panics: Deadlock: cycle in blocking chain
Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
Reviewed by: Steve Peng <steve.peng@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
Reviewed by: Bayard Bell <bayard.bell@nexenta.com>
NEX-6018 Return of the walking dead idm_refcnt_wait_ref comstar threads
Reviewed by:  Rick McNeal <rick.mcneal@nexenta.com>
Reviewed by:  Evan Layton <evan.layton@nexenta.com>
NEX-5428 Backout the 5.0 changes
NEX-2937 Continuous write_same starves all other commands
Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
Reviewed by: Steve Peng <steve.peng@nexenta.com>
NEX-4953 want uintptr32_t in sys/types32.h
Reviewed by: Josef 'Jeff' Sipek <josef.sipek@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
NEX-4905 While Deleting an iSCSI target, the appliance
         gets in to panic mode(restart)
Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
NEX-4928 Kernel panic in stfm_deregister_lu function
Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
SUP-770 deadlock between thread acquiring iss->iss_lockp in stmf_task_free() and thread holding sl->sl_pgr->pgr_lock from sbd_pgr_remove_it_handle()
Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
NEX-3622 COMSTAR should have per remote port kstats for I/O and latency
Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
NEX-4026 panic in smb_session_send
NEX-3738 Should support SMB2_CAP_LARGE_MTU (uio bug)
Reviewed by: Alek Pinchuk <alek.pinchuk@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
NEX-3566 assertion failed: lu == dlun0 || (ilu->ilu_state != STMF_STATE_OFFLINING && ilu->ilu_state != STMF_STATE_OFFLINE), file: ../../common/io /comstar/stmf/stmf.c, line: 4063
Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
NEX-3204 Panic doing FC rescan from ESXi 5.5u1 with VAAI enabled
Reviewed by: Steve Peng <steve.peng@nexenta.com>
Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
Reviewed by: Gordon Ross <gordon.ross@nexenta.com>
NEX-3217 Panic running benchmark at ESX VM
NEX-3204 Panic doing FC rescan from ESXi 5.5u1 with VAAI enabled
        Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
        Reviewed by: Tony Nguyen <tony.nguyen@nexenta.com>
NEX-2529 NFS service fails to start after hostname change
Reviewed by: Gordon Ross <gordon.ross@nexenta.com>
NEX-3259 COMSTAR has insufficient task workers
        Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
NEX-3169 STMF has duplicate code in 6 places which is error prone.
Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
Reviewby by: Steve Ping <steve.ping@nexenta.com>
NEX-3111 Comstar does not pass cstyle and hdrchk
        Reviewed by: Jean McCormack <jean.mccormack@nexenta.com>
        Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
        Reviewed by: Tony Nguyen <tony.nguyen@nexenta.com>
NEX-3104 Panic in stmf_dlun0_done
        Review by: Rick McNeal <rick.mcneal@nexenta.com>
NEX-3023 Panics and hangs when using write_same and compare_and_write
Review by: Bayard Bell <bayard.bell@nexenta.com>
Review by: Rick McNeal <rick.mcneal@nexenta.com>
Review by: Jean McCormack <jean.mccormack@nexenta.com>
Approved by: Jean McCormack <jean.mccormack@nexenta.com>
Related bug: NEX-2723 Kernel panic in xfer_completion code for write_same (0x93) and compare_and_write (0x89)
SUP-772 COMSTAR task waiting pressure doesn't increase worker thread pool sizing above initial value
SUP-765 When a Windows Clustered Shared Volume is placed on a pool under Nexenta HA Cluster control the clustered shared disk looses its PGR3 reservation to the presented zvol.
Reviewed by: Bayard Bell <bayard.bell@nexenta.com>
Reviewed by: Tony Nguyen <tony.nguyen@nexenta.com>
Reviewed by: Josef Sipek <josef.sipek@nexenta.com>
NEX-988 itask_lu_[read|write]_time was inadvertently removed by the Illumos 3862 fix
SUP-540 panic on page fault in stmf_task_lu_free()
re #13796 OSX FC Initiator cannot attach to LUN if LUN id is different then other OSX
re #7936 rb3706 Support for COMSTAR/OEM
re #8002 rb3706 Allow setting iSCSI vendor ID via stmf_sbd.conf
re #11454 rb3750 Fix inconsistent vid/pid in stmf
re #7550 rb2134 lint-clean nza-kernel
Re #6790 backspace should perform delete on console
VAAI (XXX ATS support for COMSTAR, YYY Block-copy support for COMSTAR)

Split Close
Expand all
Collapse all
          --- old/usr/src/uts/common/io/comstar/stmf/stmf.c
          +++ new/usr/src/uts/common/io/comstar/stmf/stmf.c
↓ open down ↓ 10 lines elided ↑ open up ↑
  11   11   * and limitations under the License.
  12   12   *
  13   13   * When distributing Covered Code, include this CDDL HEADER in each
  14   14   * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
  15   15   * If applicable, add the following below this CDDL HEADER, with the
  16   16   * fields enclosed by brackets "[]" replaced with your own identifying
  17   17   * information: Portions Copyright [yyyy] [name of copyright owner]
  18   18   *
  19   19   * CDDL HEADER END
  20   20   */
       21 +
  21   22  /*
  22   23   * Copyright (c) 2008, 2010, Oracle and/or its affiliates. All rights reserved.
  23   24   */
  24   25  /*
  25      - * Copyright 2012, Nexenta Systems, Inc. All rights reserved.
       26 + * Copyright 2019 Nexenta Systems, Inc.  All rights reserved.
  26   27   * Copyright (c) 2013 by Delphix. All rights reserved.
  27   28   * Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
  28   29   */
  29   30  
  30   31  #include <sys/conf.h>
  31   32  #include <sys/file.h>
  32   33  #include <sys/ddi.h>
  33   34  #include <sys/sunddi.h>
  34   35  #include <sys/modctl.h>
  35   36  #include <sys/scsi/scsi.h>
↓ open down ↓ 81 lines elided ↑ open up ↑
 117  118  int stmf_load_ppd_ioctl(stmf_ppioctl_data_t *ppi, uint64_t *ppi_token,
 118  119      uint32_t *err_ret);
 119  120  int stmf_delete_ppd_ioctl(stmf_ppioctl_data_t *ppi);
 120  121  int stmf_get_ppd_ioctl(stmf_ppioctl_data_t *ppi, stmf_ppioctl_data_t *ppi_out,
 121  122      uint32_t *err_ret);
 122  123  void stmf_delete_ppd(stmf_pp_data_t *ppd);
 123  124  void stmf_delete_all_ppds();
 124  125  void stmf_trace_clear();
 125  126  void stmf_worker_init();
 126  127  stmf_status_t stmf_worker_fini();
 127      -void stmf_worker_mgmt();
 128  128  void stmf_worker_task(void *arg);
 129  129  static void stmf_task_lu_free(scsi_task_t *task, stmf_i_scsi_session_t *iss);
 130  130  static stmf_status_t stmf_ic_lu_reg(stmf_ic_reg_dereg_lun_msg_t *msg,
 131  131      uint32_t type);
 132  132  static stmf_status_t stmf_ic_lu_dereg(stmf_ic_reg_dereg_lun_msg_t *msg);
 133  133  static stmf_status_t stmf_ic_rx_scsi_status(stmf_ic_scsi_status_msg_t *msg);
 134  134  static stmf_status_t stmf_ic_rx_status(stmf_ic_status_msg_t *msg);
 135  135  static stmf_status_t stmf_ic_rx_scsi_data(stmf_ic_scsi_data_msg_t *msg);
 136  136  void stmf_task_lu_killall(stmf_lu_t *lu, scsi_task_t *tm_task, stmf_status_t s);
 137  137  
↓ open down ↓ 19 lines elided ↑ open up ↑
 157  157  
 158  158  static void stmf_lport_xfer_start(stmf_i_scsi_task_t *itask,
 159  159      stmf_data_buf_t *dbuf);
 160  160  static void stmf_lport_xfer_done(stmf_i_scsi_task_t *itask,
 161  161      stmf_data_buf_t *dbuf);
 162  162  
 163  163  static void stmf_update_kstat_lu_q(scsi_task_t *, void());
 164  164  static void stmf_update_kstat_lport_q(scsi_task_t *, void());
 165  165  static void stmf_update_kstat_lu_io(scsi_task_t *, stmf_data_buf_t *);
 166  166  static void stmf_update_kstat_lport_io(scsi_task_t *, stmf_data_buf_t *);
      167 +static hrtime_t stmf_update_rport_timestamps(hrtime_t *start_tstamp,
      168 +    hrtime_t *done_tstamp, stmf_i_scsi_task_t *itask);
 167  169  
 168  170  static int stmf_irport_compare(const void *void_irport1,
 169  171      const void *void_irport2);
      172 +static void stmf_create_kstat_rport(stmf_i_remote_port_t *irport);
      173 +static void stmf_destroy_kstat_rport(stmf_i_remote_port_t *irport);
      174 +static int stmf_kstat_rport_update(kstat_t *ksp, int rw);
 170  175  static stmf_i_remote_port_t *stmf_irport_create(scsi_devid_desc_t *rport_devid);
 171  176  static void stmf_irport_destroy(stmf_i_remote_port_t *irport);
 172  177  static stmf_i_remote_port_t *stmf_irport_register(
 173  178      scsi_devid_desc_t *rport_devid);
 174  179  static stmf_i_remote_port_t *stmf_irport_lookup_locked(
 175  180      scsi_devid_desc_t *rport_devid);
 176  181  static void stmf_irport_deregister(stmf_i_remote_port_t *irport);
 177  182  
 178  183  extern struct mod_ops mod_driverops;
 179  184  
 180  185  /* =====[ Tunables ]===== */
 181  186  /* Internal tracing */
 182      -volatile int    stmf_trace_on = 1;
      187 +volatile int    stmf_trace_on = 0;
 183  188  volatile int    stmf_trace_buf_size = (1 * 1024 * 1024);
 184  189  /*
 185  190   * The reason default task timeout is 75 is because we want the
 186  191   * host to timeout 1st and mostly host timeout is 60 seconds.
 187  192   */
 188  193  volatile int    stmf_default_task_timeout = 75;
 189  194  /*
 190  195   * Setting this to one means, you are responsible for config load and keeping
 191  196   * things in sync with persistent database.
 192  197   */
 193  198  volatile int    stmf_allow_modunload = 0;
 194  199  
 195      -volatile int stmf_max_nworkers = 256;
 196      -volatile int stmf_min_nworkers = 4;
 197      -volatile int stmf_worker_scale_down_delay = 20;
      200 +volatile int stmf_nworkers = 512;
      201 +volatile int stmf_worker_warn = 0;
 198  202  
 199  203  /* === [ Debugging and fault injection ] === */
 200  204  #ifdef  DEBUG
 201      -volatile uint32_t stmf_drop_task_counter = 0;
 202      -volatile uint32_t stmf_drop_buf_counter = 0;
      205 +volatile int stmf_drop_task_counter = 0;
      206 +volatile int stmf_drop_buf_counter = 0;
 203  207  
 204  208  #endif
 205  209  
 206  210  stmf_state_t            stmf_state;
 207  211  static stmf_lu_t        *dlun0;
 208  212  
 209  213  static uint8_t stmf_first_zero[] =
 210  214          { 0, 1, 0, 2, 0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0, 0xff };
 211  215  static uint8_t stmf_first_one[] =
 212  216          { 0xff, 0, 1, 0, 2, 0, 1, 0, 3, 0, 1, 0, 2, 0, 1, 0 };
↓ open down ↓ 1 lines elided ↑ open up ↑
 214  218  static kmutex_t trace_buf_lock;
 215  219  static int      trace_buf_size;
 216  220  static int      trace_buf_curndx;
 217  221  caddr_t stmf_trace_buf;
 218  222  
 219  223  static enum {
 220  224          STMF_WORKERS_DISABLED = 0,
 221  225          STMF_WORKERS_ENABLING,
 222  226          STMF_WORKERS_ENABLED
 223  227  } stmf_workers_state = STMF_WORKERS_DISABLED;
 224      -static int stmf_i_max_nworkers;
 225      -static int stmf_i_min_nworkers;
 226      -static int stmf_nworkers_cur;           /* # of workers currently running */
 227      -static int stmf_nworkers_needed;        /* # of workers need to be running */
      228 +static kmutex_t stmf_worker_sel_mx;
      229 +volatile uint32_t stmf_nworkers_cur = 0; /* # of workers currently running */
 228  230  static int stmf_worker_sel_counter = 0;
 229  231  static uint32_t stmf_cur_ntasks = 0;
 230      -static clock_t stmf_wm_last = 0;
 231      -/*
 232      - * This is equal to stmf_nworkers_cur while we are increasing # workers and
 233      - * stmf_nworkers_needed while we are decreasing the worker count.
 234      - */
      232 +static clock_t stmf_wm_next = 0;
 235  233  static int stmf_nworkers_accepting_cmds;
 236  234  static stmf_worker_t *stmf_workers = NULL;
 237      -static clock_t stmf_worker_mgmt_delay = 2;
 238  235  static clock_t stmf_worker_scale_down_timer = 0;
 239  236  static int stmf_worker_scale_down_qd = 0;
 240  237  
 241  238  static struct cb_ops stmf_cb_ops = {
 242  239          stmf_open,                      /* open */
 243  240          stmf_close,                     /* close */
 244  241          nodev,                          /* strategy */
 245  242          nodev,                          /* print */
 246  243          nodev,                          /* dump */
 247  244          nodev,                          /* read */
↓ open down ↓ 18 lines elided ↑ open up ↑
 266  263          nulldev,                /* identify */
 267  264          nulldev,                /* probe */
 268  265          stmf_attach,
 269  266          stmf_detach,
 270  267          nodev,                  /* reset */
 271  268          &stmf_cb_ops,
 272  269          NULL,                   /* bus_ops */
 273  270          NULL                    /* power */
 274  271  };
 275  272  
 276      -#define STMF_NAME               "COMSTAR STMF"
      273 +
 277  274  #define STMF_MODULE_NAME        "stmf"
 278  275  
      276 +#ifdef  DEBUG
      277 +#define STMF_NAME               "COMSTAR STMF D " __DATE__ " " __TIME__
      278 +#else
      279 +#define STMF_NAME               "COMSTAR STMF"
      280 +#endif
      281 +
 279  282  static struct modldrv modldrv = {
 280  283          &mod_driverops,
 281  284          STMF_NAME,
 282  285          &stmf_ops
 283  286  };
 284  287  
 285  288  static struct modlinkage modlinkage = {
 286  289          MODREV_1,
 287  290          &modldrv,
 288  291          NULL
↓ open down ↓ 4 lines elided ↑ open up ↑
 293  296  {
 294  297          int ret;
 295  298  
 296  299          ret = mod_install(&modlinkage);
 297  300          if (ret)
 298  301                  return (ret);
 299  302          stmf_trace_buf = kmem_zalloc(stmf_trace_buf_size, KM_SLEEP);
 300  303          trace_buf_size = stmf_trace_buf_size;
 301  304          trace_buf_curndx = 0;
 302  305          mutex_init(&trace_buf_lock, NULL, MUTEX_DRIVER, 0);
      306 +        mutex_init(&stmf_worker_sel_mx, NULL, MUTEX_ADAPTIVE, 0);
 303  307          bzero(&stmf_state, sizeof (stmf_state_t));
 304  308          /* STMF service is off by default */
 305  309          stmf_state.stmf_service_running = 0;
 306  310          /* default lu/lport states are online */
 307  311          stmf_state.stmf_default_lu_state = STMF_STATE_ONLINE;
 308  312          stmf_state.stmf_default_lport_state = STMF_STATE_ONLINE;
 309  313          mutex_init(&stmf_state.stmf_lock, NULL, MUTEX_DRIVER, NULL);
 310  314          cv_init(&stmf_state.stmf_cv, NULL, CV_DRIVER, NULL);
 311  315          stmf_session_counter = (uint64_t)ddi_get_lbolt();
 312  316          avl_create(&stmf_state.stmf_irportlist,
↓ open down ↓ 50 lines elided ↑ open up ↑
 363  367          while ((irport = avl_destroy_nodes(&stmf_state.stmf_irportlist,
 364  368              &avl_dest_cookie)) != NULL)
 365  369                  stmf_irport_destroy(irport);
 366  370          avl_destroy(&stmf_state.stmf_irportlist);
 367  371          id_space_destroy(stmf_state.stmf_ilport_inst_space);
 368  372          id_space_destroy(stmf_state.stmf_irport_inst_space);
 369  373  
 370  374          kmem_free(stmf_trace_buf, stmf_trace_buf_size);
 371  375          mutex_destroy(&trace_buf_lock);
 372  376          mutex_destroy(&stmf_state.stmf_lock);
      377 +        mutex_destroy(&stmf_worker_sel_mx);
 373  378          cv_destroy(&stmf_state.stmf_cv);
 374  379          return (ret);
 375  380  }
 376  381  
 377  382  int
 378  383  _info(struct modinfo *modinfop)
 379  384  {
 380  385          return (mod_info(&modlinkage, modinfop));
 381  386  }
 382  387  
↓ open down ↓ 80 lines elided ↑ open up ↑
 463  468                  stmf_view_clear_config();
 464  469                  stmf_view_init();
 465  470          }
 466  471          stmf_state.stmf_exclusive_open = 0;
 467  472          mutex_exit(&stmf_state.stmf_lock);
 468  473          return (0);
 469  474  }
 470  475  
 471  476  int
 472  477  stmf_copyin_iocdata(intptr_t data, int mode, stmf_iocdata_t **iocd,
 473      -                                                void **ibuf, void **obuf)
      478 +    void **ibuf, void **obuf)
 474  479  {
 475  480          int ret;
 476  481  
 477  482          *ibuf = NULL;
 478  483          *obuf = NULL;
 479  484          *iocd = kmem_zalloc(sizeof (stmf_iocdata_t), KM_SLEEP);
 480  485  
 481  486          ret = ddi_copyin((void *)data, *iocd, sizeof (stmf_iocdata_t), mode);
 482  487          if (ret)
 483  488                  return (EFAULT);
↓ open down ↓ 38 lines elided ↑ open up ↑
 522  527          }
 523  528          ret = ddi_copyout(iocd, (void *)data, sizeof (stmf_iocdata_t), mode);
 524  529          if (ret)
 525  530                  return (EFAULT);
 526  531          return (0);
 527  532  }
 528  533  
 529  534  /* ARGSUSED */
 530  535  static int
 531  536  stmf_ioctl(dev_t dev, int cmd, intptr_t data, int mode,
 532      -        cred_t *credp, int *rval)
      537 +    cred_t *credp, int *rval)
 533  538  {
 534  539          stmf_iocdata_t *iocd;
 535  540          void *ibuf = NULL, *obuf = NULL;
 536  541          slist_lu_t *luid_list;
 537  542          slist_target_port_t *lportid_list;
 538  543          stmf_i_lu_t *ilu;
 539  544          stmf_i_local_port_t *ilport;
 540  545          stmf_i_scsi_session_t *iss;
 541  546          slist_scsi_session_t *iss_list;
 542  547          sioc_lu_props_t *lup;
↓ open down ↓ 1105 lines elided ↑ open up ↑
1648 1653          }
1649 1654  
1650 1655          if (ilu == NULL) {
1651 1656                  mutex_exit(&stmf_state.stmf_lock);
1652 1657                  return (NULL);
1653 1658          }
1654 1659  
1655 1660          mutex_enter(&ilu->ilu_task_lock);
1656 1661          for (itask = ilu->ilu_tasks; itask != NULL;
1657 1662              itask = itask->itask_lu_next) {
     1663 +                mutex_enter(&itask->itask_mutex);
1658 1664                  if (itask->itask_flags & (ITASK_IN_FREE_LIST |
1659 1665                      ITASK_BEING_ABORTED)) {
     1666 +                        mutex_exit(&itask->itask_mutex);
1660 1667                          continue;
1661 1668                  }
     1669 +                mutex_exit(&itask->itask_mutex);
1662 1670                  if (itask->itask_proxy_msg_id == task_msgid) {
1663 1671                          break;
1664 1672                  }
1665 1673          }
1666 1674          mutex_exit(&ilu->ilu_task_lock);
1667 1675          mutex_exit(&stmf_state.stmf_lock);
1668 1676  
1669 1677          if (itask != NULL) {
1670 1678                  return (itask->itask_task);
1671 1679          } else {
↓ open down ↓ 224 lines elided ↑ open up ↑
1896 1904  
1897 1905          /*
1898 1906           * stmf will now take over the task handling for this task
1899 1907           * but it still needs to be treated differently from other
1900 1908           * default handled tasks, hence the ITASK_PROXY_TASK.
1901 1909           * If this is a task management function, we're really just
1902 1910           * duping the command to the peer. Set the TM bit so that
1903 1911           * we can recognize this on return since we won't be completing
1904 1912           * the proxied task in that case.
1905 1913           */
     1914 +        mutex_enter(&itask->itask_mutex);
1906 1915          if (task->task_mgmt_function) {
1907 1916                  itask->itask_proxy_msg_id |= MSG_ID_TM_BIT;
1908 1917          } else {
1909      -                uint32_t new, old;
1910      -                do {
1911      -                        new = old = itask->itask_flags;
1912      -                        if (new & ITASK_BEING_ABORTED)
1913      -                                return (STMF_FAILURE);
1914      -                        new |= ITASK_DEFAULT_HANDLING | ITASK_PROXY_TASK;
1915      -                } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     1918 +                if (itask->itask_flags & ITASK_BEING_ABORTED) {
     1919 +                        mutex_exit(&itask->itask_mutex);
     1920 +                        return (STMF_FAILURE);
     1921 +                }
     1922 +                itask->itask_flags |= ITASK_DEFAULT_HANDLING | ITASK_PROXY_TASK;
1916 1923          }
1917 1924          if (dbuf) {
1918 1925                  ic_cmd_msg = ic_scsi_cmd_msg_alloc(itask->itask_proxy_msg_id,
1919 1926                      task, dbuf->db_data_size, dbuf->db_sglist[0].seg_addr,
1920 1927                      itask->itask_proxy_msg_id);
1921 1928          } else {
1922 1929                  ic_cmd_msg = ic_scsi_cmd_msg_alloc(itask->itask_proxy_msg_id,
1923 1930                      task, 0, NULL, itask->itask_proxy_msg_id);
1924 1931          }
     1932 +        mutex_exit(&itask->itask_mutex);
1925 1933          if (ic_cmd_msg) {
1926 1934                  ic_ret = ic_tx_msg(ic_cmd_msg);
1927 1935                  if (ic_ret == STMF_IC_MSG_SUCCESS) {
1928 1936                          ret = STMF_SUCCESS;
1929 1937                  }
1930 1938          }
1931 1939          return (ret);
1932 1940  }
1933 1941  
1934 1942  
↓ open down ↓ 583 lines elided ↑ open up ↑
2518 2526                  }
2519 2527          }
2520 2528  
2521 2529          if ((ret = nvlist_unpack((char *)ppi->ppi_data,
2522 2530              (size_t)ppi->ppi_data_size, &nv, KM_NOSLEEP)) != 0) {
2523 2531                  mutex_exit(&stmf_state.stmf_lock);
2524 2532                  return (ret);
2525 2533          }
2526 2534  
2527 2535          /* Free any existing lists and add this one to the ppd */
2528      -        nvlist_free(ppd->ppd_nv);
     2536 +        if (ppd->ppd_nv)
     2537 +                nvlist_free(ppd->ppd_nv);
2529 2538          ppd->ppd_nv = nv;
2530 2539  
2531 2540          /* set the token for writes */
2532 2541          ppd->ppd_token++;
2533 2542          /* return token to caller */
2534 2543          if (ppi_token) {
2535 2544                  *ppi_token = ppd->ppd_token;
2536 2545          }
2537 2546  
2538 2547          /* If there is a provider registered, do the notifications */
↓ open down ↓ 51 lines elided ↑ open up ↑
2590 2599          for (pppd = &stmf_state.stmf_ppdlist; *pppd != NULL;
2591 2600              pppd = &((*pppd)->ppd_next)) {
2592 2601                  if (*pppd == ppd)
2593 2602                          break;
2594 2603          }
2595 2604  
2596 2605          if (*pppd == NULL)
2597 2606                  return;
2598 2607  
2599 2608          *pppd = ppd->ppd_next;
2600      -        nvlist_free(ppd->ppd_nv);
     2609 +        if (ppd->ppd_nv)
     2610 +                nvlist_free(ppd->ppd_nv);
2601 2611  
2602 2612          kmem_free(ppd, ppd->ppd_alloc_size);
2603 2613  }
2604 2614  
2605 2615  int
2606 2616  stmf_delete_ppd_ioctl(stmf_ppioctl_data_t *ppi)
2607 2617  {
2608 2618          stmf_pp_data_t *ppd;
2609 2619          int ret = ENOENT;
2610 2620  
↓ open down ↓ 88 lines elided ↑ open up ↑
2699 2709                  stmf_delete_ppd(ppd);
2700 2710          }
2701 2711  }
2702 2712  
2703 2713  /*
2704 2714   * 16 is the max string length of a protocol_ident, increase
2705 2715   * the size if needed.
2706 2716   */
2707 2717  #define STMF_KSTAT_LU_SZ        (STMF_GUID_INPUT + 1 + 256)
2708 2718  #define STMF_KSTAT_TGT_SZ       (256 * 2 + 16)
     2719 +#define STMF_KSTAT_RPORT_DATAMAX        (sizeof (stmf_kstat_rport_info_t) / \
     2720 +                                            sizeof (kstat_named_t))
2709 2721  
2710 2722  /*
2711 2723   * This array matches the Protocol Identifier in stmf_ioctl.h
2712 2724   */
2713 2725  #define MAX_PROTO_STR_LEN       32
2714 2726  
2715 2727  char *protocol_ident[PROTOCOL_ANY] = {
2716 2728          "Fibre Channel",
2717 2729          "Parallel SCSI",
2718 2730          "SSA",
↓ open down ↓ 57 lines elided ↑ open up ↑
2776 2788                  kip = KSTAT_IO_PTR(ilp->ilport_kstat_io);
2777 2789                  if (kip != NULL) {
2778 2790                          mutex_enter(ilp->ilport_kstat_io->ks_lock);
2779 2791                          STMF_UPDATE_KSTAT_IO(kip, dbuf);
2780 2792                          mutex_exit(ilp->ilport_kstat_io->ks_lock);
2781 2793                  }
2782 2794          }
2783 2795  }
2784 2796  
2785 2797  static void
     2798 +stmf_update_kstat_rport_io(scsi_task_t *task, stmf_data_buf_t *dbuf)
     2799 +{
     2800 +        stmf_i_scsi_session_t   *iss;
     2801 +        stmf_i_remote_port_t    *irport;
     2802 +        kstat_io_t              *kip;
     2803 +
     2804 +        iss = task->task_session->ss_stmf_private;
     2805 +        irport = iss->iss_irport;
     2806 +        if (irport->irport_kstat_io != NULL) {
     2807 +                kip = KSTAT_IO_PTR(irport->irport_kstat_io);
     2808 +                mutex_enter(irport->irport_kstat_io->ks_lock);
     2809 +                STMF_UPDATE_KSTAT_IO(kip, dbuf);
     2810 +                mutex_exit(irport->irport_kstat_io->ks_lock);
     2811 +        }
     2812 +}
     2813 +
     2814 +static void
     2815 +stmf_update_kstat_rport_estat(scsi_task_t *task)
     2816 +{
     2817 +        stmf_i_scsi_task_t              *itask;
     2818 +        stmf_i_scsi_session_t           *iss;
     2819 +        stmf_i_remote_port_t            *irport;
     2820 +        stmf_kstat_rport_estat_t        *ks_estat;
     2821 +        hrtime_t                        lat = 0;
     2822 +        uint32_t                        n = 0;
     2823 +
     2824 +        itask = task->task_stmf_private;
     2825 +        iss = task->task_session->ss_stmf_private;
     2826 +        irport = iss->iss_irport;
     2827 +
     2828 +        if (irport->irport_kstat_estat == NULL)
     2829 +                return;
     2830 +
     2831 +        ks_estat = (stmf_kstat_rport_estat_t *)KSTAT_NAMED_PTR(
     2832 +            irport->irport_kstat_estat);
     2833 +
     2834 +        mutex_enter(irport->irport_kstat_estat->ks_lock);
     2835 +
     2836 +        if (task->task_flags & TF_READ_DATA)
     2837 +                n = atomic_dec_32_nv(&irport->irport_nread_tasks);
     2838 +        else if (task->task_flags & TF_WRITE_DATA)
     2839 +                n = atomic_dec_32_nv(&irport->irport_nwrite_tasks);
     2840 +
     2841 +        if (itask->itask_read_xfer > 0) {
     2842 +                ks_estat->i_nread_tasks.value.ui64++;
     2843 +                lat = stmf_update_rport_timestamps(
     2844 +                    &irport->irport_rdstart_timestamp,
     2845 +                    &irport->irport_rddone_timestamp, itask);
     2846 +                if (n == 0)
     2847 +                        ks_estat->i_rport_read_latency.value.ui64 += lat;
     2848 +        } else if ((itask->itask_write_xfer > 0) ||
     2849 +            (task->task_flags & TF_INITIAL_BURST)) {
     2850 +                ks_estat->i_nwrite_tasks.value.ui64++;
     2851 +                lat = stmf_update_rport_timestamps(
     2852 +                    &irport->irport_wrstart_timestamp,
     2853 +                    &irport->irport_wrdone_timestamp, itask);
     2854 +                if (n == 0)
     2855 +                        ks_estat->i_rport_write_latency.value.ui64 += lat;
     2856 +        }
     2857 +
     2858 +        if (n == 0) {
     2859 +                if (task->task_flags & TF_READ_DATA) {
     2860 +                        irport->irport_rdstart_timestamp = LLONG_MAX;
     2861 +                        irport->irport_rddone_timestamp = 0;
     2862 +                } else if (task->task_flags & TF_WRITE_DATA) {
     2863 +                        irport->irport_wrstart_timestamp = LLONG_MAX;
     2864 +                        irport->irport_wrdone_timestamp = 0;
     2865 +                }
     2866 +        }
     2867 +
     2868 +        mutex_exit(irport->irport_kstat_estat->ks_lock);
     2869 +}
     2870 +
     2871 +static hrtime_t
     2872 +stmf_update_rport_timestamps(hrtime_t *start_tstamp, hrtime_t *done_tstamp,
     2873 +    stmf_i_scsi_task_t *itask)
     2874 +{
     2875 +        *start_tstamp = MIN(*start_tstamp, itask->itask_start_timestamp);
     2876 +        if ((*done_tstamp == 0) &&
     2877 +            (itask->itask_xfer_done_timestamp == 0)) {
     2878 +                *done_tstamp = *start_tstamp;
     2879 +        } else {
     2880 +                *done_tstamp = MAX(*done_tstamp,
     2881 +                    itask->itask_xfer_done_timestamp);
     2882 +        }
     2883 +
     2884 +        return (*done_tstamp - *start_tstamp);
     2885 +}
     2886 +
     2887 +static void
2786 2888  stmf_update_kstat_lu_io(scsi_task_t *task, stmf_data_buf_t *dbuf)
2787 2889  {
2788 2890          stmf_i_lu_t             *ilu;
2789 2891          kstat_io_t              *kip;
2790 2892  
2791 2893          ilu = (stmf_i_lu_t *)task->task_lu->lu_stmf_private;
2792 2894          if (ilu != NULL && ilu->ilu_kstat_io != NULL) {
2793 2895                  kip = KSTAT_IO_PTR(ilu->ilu_kstat_io);
2794 2896                  if (kip != NULL) {
2795 2897                          mutex_enter(ilu->ilu_kstat_io->ks_lock);
↓ open down ↓ 640 lines elided ↑ open up ↑
3436 3538                  kmem_free(irport, alloc_len);
3437 3539                  return (NULL);
3438 3540          }
3439 3541  
3440 3542          irport->irport_id =
3441 3543              (struct scsi_devid_desc *)(irport + 1); /* Ptr. Arith. */
3442 3544          bcopy(rport_devid, irport->irport_id,
3443 3545              sizeof (scsi_devid_desc_t) + rport_devid->ident_length - 1);
3444 3546          irport->irport_refcnt = 1;
3445 3547          mutex_init(&irport->irport_mutex, NULL, MUTEX_DEFAULT, NULL);
     3548 +        irport->irport_rdstart_timestamp = LLONG_MAX;
     3549 +        irport->irport_wrstart_timestamp = LLONG_MAX;
3446 3550  
3447 3551          return (irport);
3448 3552  }
3449 3553  
3450 3554  static void
3451 3555  stmf_irport_destroy(stmf_i_remote_port_t *irport)
3452 3556  {
     3557 +        stmf_destroy_kstat_rport(irport);
3453 3558          id_free(stmf_state.stmf_irport_inst_space, irport->irport_instance);
3454 3559          mutex_destroy(&irport->irport_mutex);
3455 3560          kmem_free(irport, sizeof (*irport) + sizeof (scsi_devid_desc_t) +
3456 3561              irport->irport_id->ident_length - 1);
3457 3562  }
3458 3563  
     3564 +static void
     3565 +stmf_create_kstat_rport(stmf_i_remote_port_t *irport)
     3566 +{
     3567 +        scsi_devid_desc_t *id = irport->irport_id;
     3568 +        char ks_nm[KSTAT_STRLEN];
     3569 +        stmf_kstat_rport_info_t *ks_info;
     3570 +        stmf_kstat_rport_estat_t *ks_estat;
     3571 +        char *ident = NULL;
     3572 +
     3573 +        ks_info = kmem_zalloc(sizeof (*ks_info), KM_NOSLEEP);
     3574 +        if (ks_info == NULL)
     3575 +                goto err_out;
     3576 +
     3577 +        (void) snprintf(ks_nm, KSTAT_STRLEN, "stmf_rport_%"PRIxPTR"",
     3578 +            (uintptr_t)irport);
     3579 +        irport->irport_kstat_info = kstat_create(STMF_MODULE_NAME, 0,
     3580 +            ks_nm, "misc", KSTAT_TYPE_NAMED,
     3581 +            STMF_KSTAT_RPORT_DATAMAX - STMF_RPORT_INFO_LIMIT,
     3582 +            KSTAT_FLAG_VIRTUAL | KSTAT_FLAG_VAR_SIZE);
     3583 +        if (irport->irport_kstat_info == NULL) {
     3584 +                kmem_free(ks_info, sizeof (*ks_info));
     3585 +                goto err_out;
     3586 +        }
     3587 +
     3588 +        irport->irport_kstat_info->ks_data = ks_info;
     3589 +        irport->irport_kstat_info->ks_private = irport;
     3590 +        irport->irport_kstat_info->ks_update = stmf_kstat_rport_update;
     3591 +        ident = kmem_alloc(id->ident_length + 1, KM_NOSLEEP);
     3592 +        if (ident == NULL) {
     3593 +                kstat_delete(irport->irport_kstat_info);
     3594 +                irport->irport_kstat_info = NULL;
     3595 +                kmem_free(ks_info, sizeof (*ks_info));
     3596 +                goto err_out;
     3597 +        }
     3598 +
     3599 +        (void) memcpy(ident, id->ident, id->ident_length);
     3600 +        ident[id->ident_length] = '\0';
     3601 +        kstat_named_init(&ks_info->i_rport_name, "name", KSTAT_DATA_STRING);
     3602 +        kstat_named_init(&ks_info->i_protocol, "protocol",
     3603 +            KSTAT_DATA_STRING);
     3604 +
     3605 +        kstat_named_setstr(&ks_info->i_rport_name, ident);
     3606 +        kstat_named_setstr(&ks_info->i_protocol,
     3607 +            protocol_ident[irport->irport_id->protocol_id]);
     3608 +        irport->irport_kstat_info->ks_lock = &irport->irport_mutex;
     3609 +        irport->irport_info_dirty = B_TRUE;
     3610 +        kstat_install(irport->irport_kstat_info);
     3611 +
     3612 +        (void) snprintf(ks_nm, KSTAT_STRLEN, "stmf_rport_io_%"PRIxPTR"",
     3613 +            (uintptr_t)irport);
     3614 +        irport->irport_kstat_io = kstat_create(STMF_MODULE_NAME, 0, ks_nm,
     3615 +            "io", KSTAT_TYPE_IO, 1, 0);
     3616 +        if (irport->irport_kstat_io == NULL)
     3617 +                goto err_out;
     3618 +
     3619 +        irport->irport_kstat_io->ks_lock = &irport->irport_mutex;
     3620 +        kstat_install(irport->irport_kstat_io);
     3621 +
     3622 +        (void) snprintf(ks_nm, KSTAT_STRLEN, "stmf_rport_st_%"PRIxPTR"",
     3623 +            (uintptr_t)irport);
     3624 +        irport->irport_kstat_estat = kstat_create(STMF_MODULE_NAME, 0, ks_nm,
     3625 +            "misc", KSTAT_TYPE_NAMED,
     3626 +            sizeof (*ks_estat) / sizeof (kstat_named_t), 0);
     3627 +        if (irport->irport_kstat_estat == NULL)
     3628 +                goto err_out;
     3629 +
     3630 +        ks_estat = (stmf_kstat_rport_estat_t *)KSTAT_NAMED_PTR(
     3631 +            irport->irport_kstat_estat);
     3632 +        kstat_named_init(&ks_estat->i_rport_read_latency,
     3633 +            "rlatency", KSTAT_DATA_UINT64);
     3634 +        kstat_named_init(&ks_estat->i_rport_write_latency,
     3635 +            "wlatency", KSTAT_DATA_UINT64);
     3636 +        kstat_named_init(&ks_estat->i_nread_tasks, "rntasks",
     3637 +            KSTAT_DATA_UINT64);
     3638 +        kstat_named_init(&ks_estat->i_nwrite_tasks, "wntasks",
     3639 +            KSTAT_DATA_UINT64);
     3640 +        irport->irport_kstat_estat->ks_lock = &irport->irport_mutex;
     3641 +        kstat_install(irport->irport_kstat_estat);
     3642 +
     3643 +        return;
     3644 +
     3645 +err_out:
     3646 +        (void) memcpy(ks_nm, id->ident, MAX(KSTAT_STRLEN - 1,
     3647 +            id->ident_length));
     3648 +        ks_nm[id->ident_length] = '\0';
     3649 +        cmn_err(CE_WARN, "STMF: remote port kstat creation failed: %s", ks_nm);
     3650 +}
     3651 +
     3652 +static void
     3653 +stmf_destroy_kstat_rport(stmf_i_remote_port_t *irport)
     3654 +{
     3655 +        if (irport->irport_kstat_io != NULL) {
     3656 +                kstat_delete(irport->irport_kstat_io);
     3657 +        }
     3658 +        if (irport->irport_kstat_info != NULL) {
     3659 +                stmf_kstat_rport_info_t *ks_info;
     3660 +                kstat_named_t *knp;
     3661 +                void *ptr;
     3662 +                int i;
     3663 +
     3664 +                ks_info = (stmf_kstat_rport_info_t *)KSTAT_NAMED_PTR(
     3665 +                    irport->irport_kstat_info);
     3666 +                kstat_delete(irport->irport_kstat_info);
     3667 +                ptr = KSTAT_NAMED_STR_PTR(&ks_info->i_rport_name);
     3668 +                kmem_free(ptr, KSTAT_NAMED_STR_BUFLEN(&ks_info->i_rport_name));
     3669 +
     3670 +                for (i = 0, knp = ks_info->i_rport_uinfo;
     3671 +                    i < STMF_RPORT_INFO_LIMIT; i++, knp++) {
     3672 +                        ptr = KSTAT_NAMED_STR_PTR(knp);
     3673 +                        if (ptr != NULL)
     3674 +                        kmem_free(ptr, KSTAT_NAMED_STR_BUFLEN(knp));
     3675 +                }
     3676 +                kmem_free(ks_info, sizeof (*ks_info));
     3677 +        }
     3678 +}
     3679 +
3459 3680  static stmf_i_remote_port_t *
3460 3681  stmf_irport_register(scsi_devid_desc_t *rport_devid)
3461 3682  {
3462 3683          stmf_i_remote_port_t    *irport;
3463 3684  
3464 3685          mutex_enter(&stmf_state.stmf_lock);
3465 3686  
3466 3687          /*
3467 3688           * Lookup will bump the refcnt if there's an existing rport
3468 3689           * context for this identifier.
↓ open down ↓ 2 lines elided ↑ open up ↑
3471 3692                  mutex_exit(&stmf_state.stmf_lock);
3472 3693                  return (irport);
3473 3694          }
3474 3695  
3475 3696          irport = stmf_irport_create(rport_devid);
3476 3697          if (irport == NULL) {
3477 3698                  mutex_exit(&stmf_state.stmf_lock);
3478 3699                  return (NULL);
3479 3700          }
3480 3701  
     3702 +        stmf_create_kstat_rport(irport);
3481 3703          avl_add(&stmf_state.stmf_irportlist, irport);
3482 3704          mutex_exit(&stmf_state.stmf_lock);
3483 3705  
3484 3706          return (irport);
3485 3707  }
3486 3708  
3487 3709  static stmf_i_remote_port_t *
3488 3710  stmf_irport_lookup_locked(scsi_devid_desc_t *rport_devid)
3489 3711  {
3490 3712          stmf_i_remote_port_t    *irport;
↓ open down ↓ 103 lines elided ↑ open up ↑
3594 3816          iss->iss_creation_time = ddi_get_time();
3595 3817          ss->ss_session_id = atomic_inc_64_nv(&stmf_session_counter);
3596 3818          iss->iss_flags &= ~ISS_BEING_CREATED;
3597 3819          /* XXX should we remove ISS_LUN_INVENTORY_CHANGED on new session? */
3598 3820          iss->iss_flags &= ~ISS_LUN_INVENTORY_CHANGED;
3599 3821          DTRACE_PROBE2(session__online, stmf_local_port_t *, lport,
3600 3822              stmf_scsi_session_t *, ss);
3601 3823          return (STMF_SUCCESS);
3602 3824  }
3603 3825  
     3826 +stmf_status_t
     3827 +stmf_add_rport_info(stmf_scsi_session_t *ss,
     3828 +    const char *prop_name, const char *prop_value)
     3829 +{
     3830 +        stmf_i_scsi_session_t *iss = ss->ss_stmf_private;
     3831 +        stmf_i_remote_port_t *irport = iss->iss_irport;
     3832 +        kstat_named_t *knp;
     3833 +        char *s;
     3834 +        int i;
     3835 +
     3836 +        s = strdup(prop_value);
     3837 +
     3838 +        mutex_enter(irport->irport_kstat_info->ks_lock);
     3839 +        /* Make sure the caller doesn't try to add already existing property */
     3840 +        knp = KSTAT_NAMED_PTR(irport->irport_kstat_info);
     3841 +        for (i = 0; i < STMF_KSTAT_RPORT_DATAMAX; i++, knp++) {
     3842 +                if (KSTAT_NAMED_STR_PTR(knp) == NULL)
     3843 +                        break;
     3844 +
     3845 +                ASSERT(strcmp(knp->name, prop_name) != 0);
     3846 +        }
     3847 +
     3848 +        if (i == STMF_KSTAT_RPORT_DATAMAX) {
     3849 +                mutex_exit(irport->irport_kstat_info->ks_lock);
     3850 +                kmem_free(s, strlen(s) + 1);
     3851 +                return (STMF_FAILURE);
     3852 +        }
     3853 +
     3854 +        irport->irport_info_dirty = B_TRUE;
     3855 +        kstat_named_init(knp, prop_name, KSTAT_DATA_STRING);
     3856 +        kstat_named_setstr(knp, s);
     3857 +        mutex_exit(irport->irport_kstat_info->ks_lock);
     3858 +
     3859 +        return (STMF_SUCCESS);
     3860 +}
     3861 +
3604 3862  void
     3863 +stmf_remove_rport_info(stmf_scsi_session_t *ss,
     3864 +    const char *prop_name)
     3865 +{
     3866 +        stmf_i_scsi_session_t *iss = ss->ss_stmf_private;
     3867 +        stmf_i_remote_port_t *irport = iss->iss_irport;
     3868 +        kstat_named_t *knp;
     3869 +        char *s;
     3870 +        int i;
     3871 +        uint32_t len;
     3872 +
     3873 +        mutex_enter(irport->irport_kstat_info->ks_lock);
     3874 +        knp = KSTAT_NAMED_PTR(irport->irport_kstat_info);
     3875 +        for (i = 0; i < STMF_KSTAT_RPORT_DATAMAX; i++, knp++) {
     3876 +                if ((knp->name != NULL) && (strcmp(knp->name, prop_name) == 0))
     3877 +                        break;
     3878 +        }
     3879 +
     3880 +        if (i == STMF_KSTAT_RPORT_DATAMAX) {
     3881 +                mutex_exit(irport->irport_kstat_info->ks_lock);
     3882 +                return;
     3883 +        }
     3884 +
     3885 +        s = KSTAT_NAMED_STR_PTR(knp);
     3886 +        len = KSTAT_NAMED_STR_BUFLEN(knp);
     3887 +
     3888 +        for (; i < STMF_KSTAT_RPORT_DATAMAX - 1; i++, knp++) {
     3889 +                kstat_named_init(knp, knp[1].name, KSTAT_DATA_STRING);
     3890 +                kstat_named_setstr(knp, KSTAT_NAMED_STR_PTR(&knp[1]));
     3891 +        }
     3892 +        kstat_named_init(knp, "", KSTAT_DATA_STRING);
     3893 +
     3894 +        irport->irport_info_dirty = B_TRUE;
     3895 +        mutex_exit(irport->irport_kstat_info->ks_lock);
     3896 +        kmem_free(s, len);
     3897 +}
     3898 +
     3899 +static int
     3900 +stmf_kstat_rport_update(kstat_t *ksp, int rw)
     3901 +{
     3902 +        stmf_i_remote_port_t *irport = ksp->ks_private;
     3903 +        kstat_named_t *knp;
     3904 +        uint_t ndata = 0;
     3905 +        size_t dsize = 0;
     3906 +        int i;
     3907 +
     3908 +        if (rw == KSTAT_WRITE)
     3909 +                return (EACCES);
     3910 +
     3911 +        if (!irport->irport_info_dirty)
     3912 +                return (0);
     3913 +
     3914 +        knp = KSTAT_NAMED_PTR(ksp);
     3915 +        for (i = 0; i < STMF_KSTAT_RPORT_DATAMAX; i++, knp++) {
     3916 +                if (KSTAT_NAMED_STR_PTR(knp) == NULL)
     3917 +                        break;
     3918 +                ndata++;
     3919 +                dsize += KSTAT_NAMED_STR_BUFLEN(knp);
     3920 +        }
     3921 +
     3922 +        ksp->ks_ndata = ndata;
     3923 +        ksp->ks_data_size = sizeof (kstat_named_t) * ndata + dsize;
     3924 +        irport->irport_info_dirty = B_FALSE;
     3925 +
     3926 +        return (0);
     3927 +}
     3928 +
     3929 +void
3605 3930  stmf_deregister_scsi_session(stmf_local_port_t *lport, stmf_scsi_session_t *ss)
3606 3931  {
3607 3932          stmf_i_local_port_t *ilport = (stmf_i_local_port_t *)
3608 3933              lport->lport_stmf_private;
3609 3934          stmf_i_scsi_session_t *iss, **ppss;
3610 3935          int found = 0;
3611 3936          stmf_ic_msg_t *ic_session_dereg;
3612 3937          stmf_status_t ic_ret = STMF_FAILURE;
     3938 +        stmf_lun_map_t *sm;
     3939 +        stmf_i_lu_t *ilu;
     3940 +        uint16_t n;
     3941 +        stmf_lun_map_ent_t *ent;
3613 3942  
3614 3943          DTRACE_PROBE2(session__offline, stmf_local_port_t *, lport,
3615 3944              stmf_scsi_session_t *, ss);
3616 3945  
3617 3946          iss = (stmf_i_scsi_session_t *)ss->ss_stmf_private;
3618 3947          if (ss->ss_rport_alias) {
3619 3948                  ss->ss_rport_alias = NULL;
3620 3949          }
3621 3950  
3622 3951  try_dereg_ss_again:
↓ open down ↓ 29 lines elided ↑ open up ↑
3652 3981                          break;
3653 3982                  }
3654 3983          }
3655 3984          if (!found) {
3656 3985                  cmn_err(CE_PANIC, "Deregister session called for non existent"
3657 3986                      " session");
3658 3987          }
3659 3988          ilport->ilport_nsessions--;
3660 3989  
3661 3990          stmf_irport_deregister(iss->iss_irport);
3662      -        (void) stmf_session_destroy_lun_map(ilport, iss);
     3991 +        /*
     3992 +         * to avoid conflict with updating session's map,
     3993 +         * which only grab stmf_lock
     3994 +         */
     3995 +        sm = iss->iss_sm;
     3996 +        iss->iss_sm = NULL;
     3997 +        iss->iss_hg = NULL;
     3998 +
3663 3999          rw_exit(&ilport->ilport_lock);
3664      -        mutex_exit(&stmf_state.stmf_lock);
3665 4000  
     4001 +        if (sm->lm_nentries) {
     4002 +                for (n = 0; n < sm->lm_nentries; n++) {
     4003 +                        if ((ent = (stmf_lun_map_ent_t *)sm->lm_plus[n])
     4004 +                            != NULL) {
     4005 +                                if (ent->ent_itl_datap) {
     4006 +                                        stmf_do_itl_dereg(ent->ent_lu,
     4007 +                                            ent->ent_itl_datap,
     4008 +                                            STMF_ITL_REASON_IT_NEXUS_LOSS);
     4009 +                                }
     4010 +                                ilu = (stmf_i_lu_t *)
     4011 +                                    ent->ent_lu->lu_stmf_private;
     4012 +                                atomic_dec_32(&ilu->ilu_ref_cnt);
     4013 +                                kmem_free(sm->lm_plus[n],
     4014 +                                    sizeof (stmf_lun_map_ent_t));
     4015 +                        }
     4016 +                }
     4017 +                kmem_free(sm->lm_plus,
     4018 +                    sizeof (stmf_lun_map_ent_t *) * sm->lm_nentries);
     4019 +        }
     4020 +        kmem_free(sm, sizeof (*sm));
     4021 +
3666 4022          if (iss->iss_flags & ISS_NULL_TPTID) {
3667 4023                  stmf_remote_port_free(ss->ss_rport);
3668 4024          }
     4025 +
     4026 +        mutex_exit(&stmf_state.stmf_lock);
3669 4027  }
3670 4028  
     4029 +
     4030 +
3671 4031  stmf_i_scsi_session_t *
3672 4032  stmf_session_id_to_issptr(uint64_t session_id, int stay_locked)
3673 4033  {
3674 4034          stmf_i_local_port_t *ilport;
3675 4035          stmf_i_scsi_session_t *iss;
3676 4036  
3677 4037          mutex_enter(&stmf_state.stmf_lock);
3678 4038          for (ilport = stmf_state.stmf_ilportlist; ilport != NULL;
3679 4039              ilport = ilport->ilport_next) {
3680 4040                  rw_enter(&ilport->ilport_lock, RW_WRITER);
↓ open down ↓ 175 lines elided ↑ open up ↑
3856 4216  
3857 4217          for (i = 0; i < nu; i++) {
3858 4218                  stmf_do_itl_dereg(lu, itl_list[i],
3859 4219                      STMF_ITL_REASON_DEREG_REQUEST);
3860 4220          }
3861 4221          kmem_free(itl_list, nmaps * sizeof (stmf_itl_data_t *));
3862 4222  
3863 4223          return (STMF_SUCCESS);
3864 4224  }
3865 4225  
3866      -stmf_status_t
3867      -stmf_get_itl_handle(stmf_lu_t *lu, uint8_t *lun, stmf_scsi_session_t *ss,
3868      -    uint64_t session_id, void **itl_handle_retp)
3869      -{
3870      -        stmf_i_scsi_session_t *iss;
3871      -        stmf_lun_map_ent_t *ent;
3872      -        stmf_lun_map_t *lm;
3873      -        stmf_status_t ret;
3874      -        int i;
3875      -        uint16_t n;
3876      -
3877      -        if (ss == NULL) {
3878      -                iss = stmf_session_id_to_issptr(session_id, 1);
3879      -                if (iss == NULL)
3880      -                        return (STMF_NOT_FOUND);
3881      -        } else {
3882      -                iss = (stmf_i_scsi_session_t *)ss->ss_stmf_private;
3883      -                rw_enter(iss->iss_lockp, RW_WRITER);
3884      -        }
3885      -
3886      -        ent = NULL;
3887      -        if (lun == NULL) {
3888      -                lm = iss->iss_sm;
3889      -                for (i = 0; i < lm->lm_nentries; i++) {
3890      -                        if (lm->lm_plus[i] == NULL)
3891      -                                continue;
3892      -                        ent = (stmf_lun_map_ent_t *)lm->lm_plus[i];
3893      -                        if (ent->ent_lu == lu)
3894      -                                break;
3895      -                }
3896      -        } else {
3897      -                n = ((uint16_t)lun[1] | (((uint16_t)(lun[0] & 0x3F)) << 8));
3898      -                ent = (stmf_lun_map_ent_t *)
3899      -                    stmf_get_ent_from_map(iss->iss_sm, n);
3900      -                if (lu && (ent->ent_lu != lu))
3901      -                        ent = NULL;
3902      -        }
3903      -        if (ent && ent->ent_itl_datap) {
3904      -                *itl_handle_retp = ent->ent_itl_datap->itl_handle;
3905      -                ret = STMF_SUCCESS;
3906      -        } else {
3907      -                ret = STMF_NOT_FOUND;
3908      -        }
3909      -
3910      -        rw_exit(iss->iss_lockp);
3911      -        return (ret);
3912      -}
3913      -
3914 4226  stmf_data_buf_t *
3915 4227  stmf_alloc_dbuf(scsi_task_t *task, uint32_t size, uint32_t *pminsize,
3916 4228      uint32_t flags)
3917 4229  {
3918 4230          stmf_i_scsi_task_t *itask =
3919 4231              (stmf_i_scsi_task_t *)task->task_stmf_private;
3920 4232          stmf_local_port_t *lport = task->task_lport;
3921 4233          stmf_data_buf_t *dbuf;
3922 4234          uint8_t ndx;
3923 4235  
↓ open down ↓ 80 lines elided ↑ open up ↑
4004 4316  
4005 4317          itask = (stmf_i_scsi_task_t *)task->task_stmf_private;
4006 4318          if (h > 3)
4007 4319                  return (NULL);
4008 4320          return (itask->itask_dbufs[h]);
4009 4321  }
4010 4322  
4011 4323  /* ARGSUSED */
4012 4324  struct scsi_task *
4013 4325  stmf_task_alloc(struct stmf_local_port *lport, stmf_scsi_session_t *ss,
4014      -                        uint8_t *lun, uint16_t cdb_length_in, uint16_t ext_id)
     4326 +    uint8_t *lun, uint16_t cdb_length_in, uint16_t ext_id)
4015 4327  {
4016 4328          stmf_lu_t *lu;
4017 4329          stmf_i_scsi_session_t *iss;
4018 4330          stmf_i_lu_t *ilu;
4019 4331          stmf_i_scsi_task_t *itask;
4020 4332          stmf_i_scsi_task_t **ppitask;
4021 4333          scsi_task_t *task;
4022 4334          uint8_t *l;
4023 4335          stmf_lun_map_ent_t *lun_map_ent;
4024 4336          uint16_t cdb_length;
↓ open down ↓ 12 lines elided ↑ open up ↑
4037 4349          iss = (stmf_i_scsi_session_t *)ss->ss_stmf_private;
4038 4350          luNbr = ((uint16_t)lun[1] | (((uint16_t)(lun[0] & 0x3F)) << 8));
4039 4351          rw_enter(iss->iss_lockp, RW_READER);
4040 4352          lun_map_ent =
4041 4353              (stmf_lun_map_ent_t *)stmf_get_ent_from_map(iss->iss_sm, luNbr);
4042 4354          if (!lun_map_ent) {
4043 4355                  lu = dlun0;
4044 4356          } else {
4045 4357                  lu = lun_map_ent->ent_lu;
4046 4358          }
     4359 +
4047 4360          ilu = lu->lu_stmf_private;
4048 4361          if (ilu->ilu_flags & ILU_RESET_ACTIVE) {
4049 4362                  rw_exit(iss->iss_lockp);
4050 4363                  return (NULL);
4051 4364          }
4052      -        ASSERT(lu == dlun0 || (ilu->ilu_state != STMF_STATE_OFFLINING &&
4053      -            ilu->ilu_state != STMF_STATE_OFFLINE));
     4365 +
     4366 +        /*
     4367 +         * if the LUN is being offlined or is offline then only command
     4368 +         * that are to query the LUN are allowed.  These are handled in
     4369 +         * stmf via the dlun0 vector.  It is possible that a race condition
     4370 +         * will cause other commands to arrive while the lun is in the
     4371 +         * process of being offlined.  Check for those and just let the
     4372 +         * protocol stack handle the error.
     4373 +         */
     4374 +        if ((ilu->ilu_state == STMF_STATE_OFFLINING) ||
     4375 +            (ilu->ilu_state == STMF_STATE_OFFLINE)) {
     4376 +                if (lu != dlun0) {
     4377 +                        rw_exit(iss->iss_lockp);
     4378 +                        return (NULL);
     4379 +                }
     4380 +        }
     4381 +
4054 4382          do {
4055 4383                  if (ilu->ilu_free_tasks == NULL) {
4056 4384                          new_task = 1;
4057 4385                          break;
4058 4386                  }
4059 4387                  mutex_enter(&ilu->ilu_task_lock);
4060 4388                  for (ppitask = &ilu->ilu_free_tasks; (*ppitask != NULL) &&
4061 4389                      ((*ppitask)->itask_cdb_buf_size < cdb_length);
4062 4390                      ppitask = &((*ppitask)->itask_lu_free_next))
4063 4391                          ;
↓ open down ↓ 27 lines elided ↑ open up ↑
4091 4419                  task->task_cdb = save_cdb;      /* restore */
4092 4420                  itask->itask_ncmds = 0;
4093 4421          } else {
4094 4422                  task = (scsi_task_t *)stmf_alloc(STMF_STRUCT_SCSI_TASK,
4095 4423                      cdb_length, AF_FORCE_NOSLEEP);
4096 4424                  if (task == NULL) {
4097 4425                          rw_exit(iss->iss_lockp);
4098 4426                          return (NULL);
4099 4427                  }
4100 4428                  task->task_lu = lu;
4101      -                l = task->task_lun_no;
4102      -                l[0] = lun[0];
4103      -                l[1] = lun[1];
4104      -                l[2] = lun[2];
4105      -                l[3] = lun[3];
4106      -                l[4] = lun[4];
4107      -                l[5] = lun[5];
4108      -                l[6] = lun[6];
4109      -                l[7] = lun[7];
4110 4429                  task->task_cdb = (uint8_t *)task->task_port_private;
4111 4430                  if ((ulong_t)(task->task_cdb) & 7ul) {
4112 4431                          task->task_cdb = (uint8_t *)(((ulong_t)
4113 4432                              (task->task_cdb) + 7ul) & ~(7ul));
4114 4433                  }
4115 4434                  itask = (stmf_i_scsi_task_t *)task->task_stmf_private;
4116 4435                  itask->itask_cdb_buf_size = cdb_length;
4117 4436                  mutex_init(&itask->itask_audit_mutex, NULL, MUTEX_DRIVER, NULL);
     4437 +                mutex_init(&itask->itask_mutex, NULL, MUTEX_DRIVER, NULL);
4118 4438          }
     4439 +
     4440 +        /*
     4441 +         * Since a LUN can be mapped as different LUN ids to different initiator
     4442 +         * groups, we need to set LUN id for a new task and reset LUN id for
     4443 +         * a reused task.
     4444 +         */
     4445 +        l = task->task_lun_no;
     4446 +        l[0] = lun[0];
     4447 +        l[1] = lun[1];
     4448 +        l[2] = lun[2];
     4449 +        l[3] = lun[3];
     4450 +        l[4] = lun[4];
     4451 +        l[5] = lun[5];
     4452 +        l[6] = lun[6];
     4453 +        l[7] = lun[7];
     4454 +
     4455 +        mutex_enter(&itask->itask_mutex);
4119 4456          task->task_session = ss;
4120 4457          task->task_lport = lport;
4121 4458          task->task_cdb_length = cdb_length_in;
4122 4459          itask->itask_flags = ITASK_IN_TRANSITION;
4123 4460          itask->itask_waitq_time = 0;
4124 4461          itask->itask_lu_read_time = itask->itask_lu_write_time = 0;
4125 4462          itask->itask_lport_read_time = itask->itask_lport_write_time = 0;
4126 4463          itask->itask_read_xfer = itask->itask_write_xfer = 0;
4127 4464          itask->itask_audit_index = 0;
     4465 +        bzero(&itask->itask_audit_records[0],
     4466 +            sizeof (stmf_task_audit_rec_t) * ITASK_TASK_AUDIT_DEPTH);
     4467 +        mutex_exit(&itask->itask_mutex);
4128 4468  
4129 4469          if (new_task) {
4130 4470                  if (lu->lu_task_alloc(task) != STMF_SUCCESS) {
4131 4471                          rw_exit(iss->iss_lockp);
4132 4472                          stmf_free(task);
4133 4473                          return (NULL);
4134 4474                  }
4135 4475                  mutex_enter(&ilu->ilu_task_lock);
4136 4476                  if (ilu->ilu_flags & ILU_RESET_ACTIVE) {
4137 4477                          mutex_exit(&ilu->ilu_task_lock);
↓ open down ↓ 20 lines elided ↑ open up ↑
4158 4498                  task->task_lu_itl_handle = itask->itask_itl_datap->itl_handle;
4159 4499          } else {
4160 4500                  itask->itask_itl_datap = NULL;
4161 4501                  task->task_lu_itl_handle = NULL;
4162 4502          }
4163 4503  
4164 4504          rw_exit(iss->iss_lockp);
4165 4505          return (task);
4166 4506  }
4167 4507  
     4508 +/* ARGSUSED */
4168 4509  static void
4169 4510  stmf_task_lu_free(scsi_task_t *task, stmf_i_scsi_session_t *iss)
4170 4511  {
4171 4512          stmf_i_scsi_task_t *itask =
4172 4513              (stmf_i_scsi_task_t *)task->task_stmf_private;
4173 4514          stmf_i_lu_t *ilu = (stmf_i_lu_t *)task->task_lu->lu_stmf_private;
4174 4515  
4175 4516          ASSERT(rw_lock_held(iss->iss_lockp));
     4517 +        ASSERT((itask->itask_flags & ITASK_IN_FREE_LIST) == 0);
     4518 +        ASSERT((itask->itask_flags & ITASK_IN_WORKER_QUEUE) == 0);
     4519 +        ASSERT((itask->itask_flags & ITASK_IN_TRANSITION) == 0);
     4520 +        ASSERT((itask->itask_flags & ITASK_KNOWN_TO_LU) == 0);
     4521 +        ASSERT(mutex_owned(&itask->itask_mutex));
     4522 +
4176 4523          itask->itask_flags = ITASK_IN_FREE_LIST;
     4524 +        itask->itask_ncmds = 0;
4177 4525          itask->itask_proxy_msg_id = 0;
     4526 +        atomic_dec_32(itask->itask_ilu_task_cntr);
     4527 +        itask->itask_worker_next = NULL;
     4528 +        mutex_exit(&itask->itask_mutex);
     4529 +
4178 4530          mutex_enter(&ilu->ilu_task_lock);
4179 4531          itask->itask_lu_free_next = ilu->ilu_free_tasks;
4180 4532          ilu->ilu_free_tasks = itask;
4181 4533          ilu->ilu_ntasks_free++;
4182 4534          if (ilu->ilu_ntasks == ilu->ilu_ntasks_free)
4183 4535                  cv_signal(&ilu->ilu_offline_pending_cv);
4184 4536          mutex_exit(&ilu->ilu_task_lock);
4185      -        atomic_dec_32(itask->itask_ilu_task_cntr);
4186 4537  }
4187 4538  
4188 4539  void
4189 4540  stmf_task_lu_check_freelist(stmf_i_lu_t *ilu)
4190 4541  {
4191 4542          uint32_t        num_to_release, ndx;
4192 4543          stmf_i_scsi_task_t *itask;
4193 4544          stmf_lu_t       *lu = ilu->ilu_lu;
4194 4545  
4195 4546          ASSERT(ilu->ilu_ntasks_min_free <= ilu->ilu_ntasks_free);
↓ open down ↓ 56 lines elided ↑ open up ↑
4252 4603                   */
4253 4604                  ilu->ilu_ntasks_min_free = ilu->ilu_ntasks_free;
4254 4605                  mutex_enter(&stmf_state.stmf_lock);
4255 4606                  ilu->ilu_flags &= ~ILU_STALL_DEREGISTER;
4256 4607                  cv_broadcast(&stmf_state.stmf_cv);
4257 4608                  if (ddi_get_lbolt() >= endtime)
4258 4609                          break;
4259 4610          }
4260 4611  }
4261 4612  
4262      -void
     4613 +/*
     4614 + * Since this method is looking to find tasks that are stuck, lost, or senile
     4615 + * it should be more willing to give up scaning during this time period. This
     4616 + * is why mutex_tryenter is now used instead of the standard mutex_enter.
     4617 + * There has been at least one case were the following occurred.
     4618 + *
     4619 + * 1) The iscsit_deferred() method is trying to register a session and
     4620 + *    needs the global lock which is held.
     4621 + * 2) Another thread which holds the global lock is trying to deregister a
     4622 + *    session and needs the session lock.
     4623 + * 3) A third thread is allocating a stmf task that has grabbed the session
     4624 + *    lock and is trying to grab the lun task lock.
     4625 + * 4) There's a timeout thread that has the lun task lock and is trying to grab
     4626 + *    a specific task lock.
     4627 + * 5) The thread that has the task lock is waiting for the ref count to go to
     4628 + *    zero.
     4629 + * 6) There's a task that would drop the count to zero, but it's in the task
     4630 + *    queue waiting to run and is stuck because of #1 is currently block.
     4631 + *
     4632 + * This method is number 4 in the above chain of events. Had this code
     4633 + * originally used mutex_tryenter the chain would have been broken and the
     4634 + * system wouldn't have hung. So, now this method uses mutex_tryenter and
     4635 + * you know why it does so.
     4636 + */
     4637 +/* ---- Only one thread calls stmf_do_ilu_timeouts so no lock required ---- */
     4638 +typedef struct stmf_bailout_cnt {
     4639 +        int     no_ilu_lock;
     4640 +        int     no_task_lock;
     4641 +        int     tasks_checked;
     4642 +} stmf_bailout_cnt_t;
     4643 +
     4644 +stmf_bailout_cnt_t stmf_bailout;
     4645 +
     4646 +static void
4263 4647  stmf_do_ilu_timeouts(stmf_i_lu_t *ilu)
4264 4648  {
4265 4649          clock_t l = ddi_get_lbolt();
4266 4650          clock_t ps = drv_usectohz(1000000);
4267 4651          stmf_i_scsi_task_t *itask;
4268 4652          scsi_task_t *task;
4269 4653          uint32_t to;
4270 4654  
4271      -        mutex_enter(&ilu->ilu_task_lock);
     4655 +        if (mutex_tryenter(&ilu->ilu_task_lock) == 0) {
     4656 +                stmf_bailout.no_ilu_lock++;
     4657 +                return;
     4658 +        }
     4659 +
4272 4660          for (itask = ilu->ilu_tasks; itask != NULL;
4273 4661              itask = itask->itask_lu_next) {
     4662 +                if (mutex_tryenter(&itask->itask_mutex) == 0) {
     4663 +                        stmf_bailout.no_task_lock++;
     4664 +                        continue;
     4665 +                }
     4666 +                stmf_bailout.tasks_checked++;
4274 4667                  if (itask->itask_flags & (ITASK_IN_FREE_LIST |
4275 4668                      ITASK_BEING_ABORTED)) {
     4669 +                        mutex_exit(&itask->itask_mutex);
4276 4670                          continue;
4277 4671                  }
4278 4672                  task = itask->itask_task;
4279 4673                  if (task->task_timeout == 0)
4280 4674                          to = stmf_default_task_timeout;
4281 4675                  else
4282 4676                          to = task->task_timeout;
4283      -                if ((itask->itask_start_time + (to * ps)) > l)
     4677 +
     4678 +                if ((itask->itask_start_time + (to * ps)) > l) {
     4679 +                        mutex_exit(&itask->itask_mutex);
4284 4680                          continue;
     4681 +                }
     4682 +                mutex_exit(&itask->itask_mutex);
4285 4683                  stmf_abort(STMF_QUEUE_TASK_ABORT, task,
4286 4684                      STMF_TIMEOUT, NULL);
4287 4685          }
4288 4686          mutex_exit(&ilu->ilu_task_lock);
4289 4687  }
4290 4688  
4291 4689  /*
4292 4690   * Called with stmf_lock held
4293 4691   */
4294 4692  void
↓ open down ↓ 34 lines elided ↑ open up ↑
4329 4727  /*
4330 4728   * Kills all tasks on a lu except tm_task
4331 4729   */
4332 4730  void
4333 4731  stmf_task_lu_killall(stmf_lu_t *lu, scsi_task_t *tm_task, stmf_status_t s)
4334 4732  {
4335 4733          stmf_i_lu_t *ilu = (stmf_i_lu_t *)lu->lu_stmf_private;
4336 4734          stmf_i_scsi_task_t *itask;
4337 4735  
4338 4736          mutex_enter(&ilu->ilu_task_lock);
4339      -
4340 4737          for (itask = ilu->ilu_tasks; itask != NULL;
4341 4738              itask = itask->itask_lu_next) {
4342      -                if (itask->itask_flags & ITASK_IN_FREE_LIST)
     4739 +                mutex_enter(&itask->itask_mutex);
     4740 +                if (itask->itask_flags & ITASK_IN_FREE_LIST) {
     4741 +                        mutex_exit(&itask->itask_mutex);
4343 4742                          continue;
     4743 +                }
     4744 +                mutex_exit(&itask->itask_mutex);
4344 4745                  if (itask->itask_task == tm_task)
4345 4746                          continue;
4346 4747                  stmf_abort(STMF_QUEUE_TASK_ABORT, itask->itask_task, s, NULL);
4347 4748          }
4348 4749          mutex_exit(&ilu->ilu_task_lock);
4349 4750  }
4350 4751  
4351 4752  void
4352 4753  stmf_free_task_bufs(stmf_i_scsi_task_t *itask, stmf_local_port_t *lport)
4353 4754  {
↓ open down ↓ 35 lines elided ↑ open up ↑
4389 4790  }
4390 4791  
4391 4792  void
4392 4793  stmf_task_free(scsi_task_t *task)
4393 4794  {
4394 4795          stmf_local_port_t *lport = task->task_lport;
4395 4796          stmf_i_scsi_task_t *itask = (stmf_i_scsi_task_t *)
4396 4797              task->task_stmf_private;
4397 4798          stmf_i_scsi_session_t *iss = (stmf_i_scsi_session_t *)
4398 4799              task->task_session->ss_stmf_private;
     4800 +        stmf_lu_t *lu = task->task_lu;
4399 4801  
4400 4802          stmf_task_audit(itask, TE_TASK_FREE, CMD_OR_IOF_NA, NULL);
4401      -
     4803 +        ASSERT(mutex_owned(&itask->itask_mutex));
     4804 +        if ((lu != NULL) && (lu->lu_task_done != NULL))
     4805 +                lu->lu_task_done(task);
4402 4806          stmf_free_task_bufs(itask, lport);
4403 4807          stmf_itl_task_done(itask);
4404 4808          DTRACE_PROBE2(stmf__task__end, scsi_task_t *, task,
4405 4809              hrtime_t,
4406 4810              itask->itask_done_timestamp - itask->itask_start_timestamp);
4407 4811          if (itask->itask_itl_datap) {
4408 4812                  if (atomic_dec_32_nv(&itask->itask_itl_datap->itl_counter) ==
4409 4813                      0) {
4410 4814                          stmf_release_itl_handle(task->task_lu,
4411 4815                              itask->itask_itl_datap);
4412 4816                  }
4413 4817          }
4414 4818  
     4819 +        /*
     4820 +         * To prevent a deadlock condition must release the itask_mutex,
     4821 +         * grab a reader lock on iss_lockp and then reacquire the itask_mutex.
     4822 +         */
     4823 +        mutex_exit(&itask->itask_mutex);
4415 4824          rw_enter(iss->iss_lockp, RW_READER);
     4825 +        mutex_enter(&itask->itask_mutex);
     4826 +
4416 4827          lport->lport_task_free(task);
4417 4828          if (itask->itask_worker) {
4418 4829                  atomic_dec_32(&stmf_cur_ntasks);
4419 4830                  atomic_dec_32(&itask->itask_worker->worker_ref_count);
4420 4831          }
4421 4832          /*
4422 4833           * After calling stmf_task_lu_free, the task pointer can no longer
4423 4834           * be trusted.
4424 4835           */
4425 4836          stmf_task_lu_free(task, iss);
4426 4837          rw_exit(iss->iss_lockp);
4427 4838  }
4428 4839  
4429 4840  void
4430 4841  stmf_post_task(scsi_task_t *task, stmf_data_buf_t *dbuf)
4431 4842  {
4432 4843          stmf_i_scsi_task_t *itask = (stmf_i_scsi_task_t *)
4433 4844              task->task_stmf_private;
4434 4845          stmf_i_lu_t *ilu = (stmf_i_lu_t *)task->task_lu->lu_stmf_private;
4435 4846          int nv;
4436      -        uint32_t old, new;
     4847 +        uint32_t new;
4437 4848          uint32_t ct;
4438      -        stmf_worker_t *w, *w1;
     4849 +        stmf_worker_t *w;
4439 4850          uint8_t tm;
4440 4851  
4441 4852          if (task->task_max_nbufs > 4)
4442 4853                  task->task_max_nbufs = 4;
4443 4854          task->task_cur_nbufs = 0;
4444 4855          /* Latest value of currently running tasks */
4445 4856          ct = atomic_inc_32_nv(&stmf_cur_ntasks);
4446 4857  
4447 4858          /* Select the next worker using round robin */
4448      -        nv = (int)atomic_inc_32_nv((uint32_t *)&stmf_worker_sel_counter);
4449      -        if (nv >= stmf_nworkers_accepting_cmds) {
4450      -                int s = nv;
4451      -                do {
4452      -                        nv -= stmf_nworkers_accepting_cmds;
4453      -                } while (nv >= stmf_nworkers_accepting_cmds);
4454      -                if (nv < 0)
4455      -                        nv = 0;
4456      -                /* Its ok if this cas fails */
4457      -                (void) atomic_cas_32((uint32_t *)&stmf_worker_sel_counter,
4458      -                    s, nv);
4459      -        }
4460      -        w = &stmf_workers[nv];
     4859 +        mutex_enter(&stmf_worker_sel_mx);
     4860 +        stmf_worker_sel_counter++;
     4861 +        if (stmf_worker_sel_counter >= stmf_nworkers)
     4862 +                stmf_worker_sel_counter = 0;
     4863 +        nv = stmf_worker_sel_counter;
4461 4864  
4462      -        /*
4463      -         * A worker can be pinned by interrupt. So select the next one
4464      -         * if it has lower load.
4465      -         */
4466      -        if ((nv + 1) >= stmf_nworkers_accepting_cmds) {
4467      -                w1 = stmf_workers;
4468      -        } else {
4469      -                w1 = &stmf_workers[nv + 1];
     4865 +        /* if the selected worker is not idle then bump to the next worker */
     4866 +        if (stmf_workers[nv].worker_queue_depth > 0) {
     4867 +                stmf_worker_sel_counter++;
     4868 +                if (stmf_worker_sel_counter >= stmf_nworkers)
     4869 +                        stmf_worker_sel_counter = 0;
     4870 +                nv = stmf_worker_sel_counter;
4470 4871          }
4471      -        if (w1->worker_queue_depth < w->worker_queue_depth)
4472      -                w = w1;
     4872 +        mutex_exit(&stmf_worker_sel_mx);
4473 4873  
     4874 +        w = &stmf_workers[nv];
     4875 +
     4876 +        mutex_enter(&itask->itask_mutex);
4474 4877          mutex_enter(&w->worker_lock);
4475      -        if (((w->worker_flags & STMF_WORKER_STARTED) == 0) ||
4476      -            (w->worker_flags & STMF_WORKER_TERMINATE)) {
4477      -                /*
4478      -                 * Maybe we are in the middle of a change. Just go to
4479      -                 * the 1st worker.
4480      -                 */
4481      -                mutex_exit(&w->worker_lock);
4482      -                w = stmf_workers;
4483      -                mutex_enter(&w->worker_lock);
4484      -        }
     4878 +
4485 4879          itask->itask_worker = w;
     4880 +
4486 4881          /*
4487 4882           * Track max system load inside the worker as we already have the
4488 4883           * worker lock (no point implementing another lock). The service
4489 4884           * thread will do the comparisons and figure out the max overall
4490 4885           * system load.
4491 4886           */
4492 4887          if (w->worker_max_sys_qdepth_pu < ct)
4493 4888                  w->worker_max_sys_qdepth_pu = ct;
4494 4889  
4495      -        do {
4496      -                old = new = itask->itask_flags;
4497      -                new |= ITASK_KNOWN_TO_TGT_PORT | ITASK_IN_WORKER_QUEUE;
4498      -                if (task->task_mgmt_function) {
4499      -                        tm = task->task_mgmt_function;
4500      -                        if ((tm == TM_TARGET_RESET) ||
4501      -                            (tm == TM_TARGET_COLD_RESET) ||
4502      -                            (tm == TM_TARGET_WARM_RESET)) {
4503      -                                new |= ITASK_DEFAULT_HANDLING;
4504      -                        }
4505      -                } else if (task->task_cdb[0] == SCMD_REPORT_LUNS) {
     4890 +        new = itask->itask_flags;
     4891 +        new |= ITASK_KNOWN_TO_TGT_PORT;
     4892 +        if (task->task_mgmt_function) {
     4893 +                tm = task->task_mgmt_function;
     4894 +                if ((tm == TM_TARGET_RESET) ||
     4895 +                    (tm == TM_TARGET_COLD_RESET) ||
     4896 +                    (tm == TM_TARGET_WARM_RESET)) {
4506 4897                          new |= ITASK_DEFAULT_HANDLING;
4507 4898                  }
4508      -                new &= ~ITASK_IN_TRANSITION;
4509      -        } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     4899 +        } else if (task->task_cdb[0] == SCMD_REPORT_LUNS) {
     4900 +                new |= ITASK_DEFAULT_HANDLING;
     4901 +        }
     4902 +        new &= ~ITASK_IN_TRANSITION;
     4903 +        itask->itask_flags = new;
4510 4904  
4511 4905          stmf_itl_task_start(itask);
4512 4906  
4513      -        itask->itask_worker_next = NULL;
4514      -        if (w->worker_task_tail) {
4515      -                w->worker_task_tail->itask_worker_next = itask;
4516      -        } else {
4517      -                w->worker_task_head = itask;
4518      -        }
4519      -        w->worker_task_tail = itask;
4520      -        if (++(w->worker_queue_depth) > w->worker_max_qdepth_pu) {
4521      -                w->worker_max_qdepth_pu = w->worker_queue_depth;
4522      -        }
4523      -        /* Measure task waitq time */
4524      -        itask->itask_waitq_enter_timestamp = gethrtime();
4525      -        atomic_inc_32(&w->worker_ref_count);
4526 4907          itask->itask_cmd_stack[0] = ITASK_CMD_NEW_TASK;
4527 4908          itask->itask_ncmds = 1;
     4909 +
     4910 +        if ((task->task_flags & TF_INITIAL_BURST) &&
     4911 +            !(curthread->t_flag & T_INTR_THREAD)) {
     4912 +                stmf_update_kstat_lu_io(task, dbuf);
     4913 +                stmf_update_kstat_lport_io(task, dbuf);
     4914 +                stmf_update_kstat_rport_io(task, dbuf);
     4915 +        }
     4916 +
4528 4917          stmf_task_audit(itask, TE_TASK_START, CMD_OR_IOF_NA, dbuf);
4529 4918          if (dbuf) {
4530 4919                  itask->itask_allocated_buf_map = 1;
4531 4920                  itask->itask_dbufs[0] = dbuf;
4532 4921                  dbuf->db_handle = 0;
4533 4922          } else {
4534 4923                  itask->itask_allocated_buf_map = 0;
4535 4924                  itask->itask_dbufs[0] = NULL;
4536 4925          }
4537 4926  
4538      -        if ((w->worker_flags & STMF_WORKER_ACTIVE) == 0) {
4539      -                w->worker_signal_timestamp = gethrtime();
4540      -                DTRACE_PROBE2(worker__signal, stmf_worker_t *, w,
4541      -                    scsi_task_t *, task);
4542      -                cv_signal(&w->worker_cv);
4543      -        }
     4927 +        STMF_ENQUEUE_ITASK(w, itask);
     4928 +
4544 4929          mutex_exit(&w->worker_lock);
     4930 +        mutex_exit(&itask->itask_mutex);
4545 4931  
4546 4932          /*
4547 4933           * This can only happen if during stmf_task_alloc(), ILU_RESET_ACTIVE
4548 4934           * was set between checking of ILU_RESET_ACTIVE and clearing of the
4549 4935           * ITASK_IN_FREE_LIST flag. Take care of these "sneaked-in" tasks here.
4550 4936           */
4551 4937          if (ilu->ilu_flags & ILU_RESET_ACTIVE) {
4552 4938                  stmf_abort(STMF_QUEUE_TASK_ABORT, task, STMF_ABORTED, NULL);
4553 4939          }
4554 4940  }
↓ open down ↓ 35 lines elided ↑ open up ↑
4590 4976  stmf_status_t
4591 4977  stmf_xfer_data(scsi_task_t *task, stmf_data_buf_t *dbuf, uint32_t ioflags)
4592 4978  {
4593 4979          stmf_status_t ret = STMF_SUCCESS;
4594 4980  
4595 4981          stmf_i_scsi_task_t *itask =
4596 4982              (stmf_i_scsi_task_t *)task->task_stmf_private;
4597 4983  
4598 4984          stmf_task_audit(itask, TE_XFER_START, ioflags, dbuf);
4599 4985  
     4986 +        mutex_enter(&itask->itask_mutex);
4600 4987          if (ioflags & STMF_IOF_LU_DONE) {
4601      -                uint32_t new, old;
4602      -                do {
4603      -                        new = old = itask->itask_flags;
4604      -                        if (new & ITASK_BEING_ABORTED)
4605      -                                return (STMF_ABORTED);
4606      -                        new &= ~ITASK_KNOWN_TO_LU;
4607      -                } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     4988 +                if (itask->itask_flags & ITASK_BEING_ABORTED) {
     4989 +                        mutex_exit(&itask->itask_mutex);
     4990 +                        return (STMF_ABORTED);
     4991 +                }
     4992 +                itask->itask_flags &= ~ITASK_KNOWN_TO_LU;
4608 4993          }
4609      -        if (itask->itask_flags & ITASK_BEING_ABORTED)
     4994 +        if ((itask->itask_flags & ITASK_BEING_ABORTED) != 0) {
     4995 +                mutex_exit(&itask->itask_mutex);
4610 4996                  return (STMF_ABORTED);
     4997 +        }
     4998 +        mutex_exit(&itask->itask_mutex);
     4999 +
4611 5000  #ifdef  DEBUG
4612 5001          if (!(ioflags & STMF_IOF_STATS_ONLY) && stmf_drop_buf_counter > 0) {
4613      -                if (atomic_dec_32_nv(&stmf_drop_buf_counter) == 1)
     5002 +                if (atomic_dec_32_nv((uint32_t *)&stmf_drop_buf_counter) == 1)
4614 5003                          return (STMF_SUCCESS);
4615 5004          }
4616 5005  #endif
4617 5006  
4618 5007          stmf_update_kstat_lu_io(task, dbuf);
4619 5008          stmf_update_kstat_lport_io(task, dbuf);
     5009 +        stmf_update_kstat_rport_io(task, dbuf);
4620 5010          stmf_lport_xfer_start(itask, dbuf);
4621 5011          if (ioflags & STMF_IOF_STATS_ONLY) {
4622 5012                  stmf_lport_xfer_done(itask, dbuf);
4623 5013                  return (STMF_SUCCESS);
4624 5014          }
4625 5015  
4626 5016          dbuf->db_flags |= DB_LPORT_XFER_ACTIVE;
4627 5017          ret = task->task_lport->lport_xfer_data(task, dbuf, ioflags);
4628 5018  
4629 5019          /*
↓ open down ↓ 9 lines elided ↑ open up ↑
4639 5029          return (ret);
4640 5030  }
4641 5031  
4642 5032  void
4643 5033  stmf_data_xfer_done(scsi_task_t *task, stmf_data_buf_t *dbuf, uint32_t iof)
4644 5034  {
4645 5035          stmf_i_scsi_task_t *itask =
4646 5036              (stmf_i_scsi_task_t *)task->task_stmf_private;
4647 5037          stmf_i_local_port_t *ilport;
4648 5038          stmf_worker_t *w = itask->itask_worker;
4649      -        uint32_t new, old;
     5039 +        uint32_t new;
4650 5040          uint8_t update_queue_flags, free_it, queue_it;
4651 5041  
4652 5042          stmf_lport_xfer_done(itask, dbuf);
4653 5043  
4654 5044          stmf_task_audit(itask, TE_XFER_DONE, iof, dbuf);
4655 5045  
4656 5046          /* Guard against unexpected completions from the lport */
4657 5047          if (dbuf->db_flags & DB_LPORT_XFER_ACTIVE) {
4658 5048                  dbuf->db_flags &= ~DB_LPORT_XFER_ACTIVE;
4659 5049          } else {
4660 5050                  /*
4661 5051                   * This should never happen.
4662 5052                   */
4663 5053                  ilport = task->task_lport->lport_stmf_private;
4664 5054                  ilport->ilport_unexpected_comp++;
4665 5055                  cmn_err(CE_PANIC, "Unexpected xfer completion task %p dbuf %p",
4666 5056                      (void *)task, (void *)dbuf);
4667 5057                  return;
4668 5058          }
4669 5059  
     5060 +        mutex_enter(&itask->itask_mutex);
4670 5061          mutex_enter(&w->worker_lock);
4671      -        do {
4672      -                new = old = itask->itask_flags;
4673      -                if (old & ITASK_BEING_ABORTED) {
4674      -                        mutex_exit(&w->worker_lock);
4675      -                        return;
4676      -                }
     5062 +        new = itask->itask_flags;
     5063 +        if (itask->itask_flags & ITASK_BEING_ABORTED) {
     5064 +                mutex_exit(&w->worker_lock);
     5065 +                mutex_exit(&itask->itask_mutex);
     5066 +                return;
     5067 +        }
     5068 +        free_it = 0;
     5069 +        if (iof & STMF_IOF_LPORT_DONE) {
     5070 +                new &= ~ITASK_KNOWN_TO_TGT_PORT;
     5071 +                task->task_completion_status = dbuf->db_xfer_status;
     5072 +                free_it = 1;
     5073 +        }
     5074 +        /*
     5075 +         * If the task is known to LU then queue it. But if
     5076 +         * it is already queued (multiple completions) then
     5077 +         * just update the buffer information by grabbing the
     5078 +         * worker lock. If the task is not known to LU,
     5079 +         * completed/aborted, then see if we need to
     5080 +         * free this task.
     5081 +         */
     5082 +        if (itask->itask_flags & ITASK_KNOWN_TO_LU) {
4677 5083                  free_it = 0;
4678      -                if (iof & STMF_IOF_LPORT_DONE) {
4679      -                        new &= ~ITASK_KNOWN_TO_TGT_PORT;
4680      -                        task->task_completion_status = dbuf->db_xfer_status;
4681      -                        free_it = 1;
4682      -                }
4683      -                /*
4684      -                 * If the task is known to LU then queue it. But if
4685      -                 * it is already queued (multiple completions) then
4686      -                 * just update the buffer information by grabbing the
4687      -                 * worker lock. If the task is not known to LU,
4688      -                 * completed/aborted, then see if we need to
4689      -                 * free this task.
4690      -                 */
4691      -                if (old & ITASK_KNOWN_TO_LU) {
4692      -                        free_it = 0;
4693      -                        update_queue_flags = 1;
4694      -                        if (old & ITASK_IN_WORKER_QUEUE) {
4695      -                                queue_it = 0;
4696      -                        } else {
4697      -                                queue_it = 1;
4698      -                                new |= ITASK_IN_WORKER_QUEUE;
4699      -                        }
4700      -                } else {
4701      -                        update_queue_flags = 0;
     5084 +                update_queue_flags = 1;
     5085 +                if (itask->itask_flags & ITASK_IN_WORKER_QUEUE) {
4702 5086                          queue_it = 0;
     5087 +                } else {
     5088 +                        queue_it = 1;
4703 5089                  }
4704      -        } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     5090 +        } else {
     5091 +                update_queue_flags = 0;
     5092 +                queue_it = 0;
     5093 +        }
     5094 +        itask->itask_flags = new;
4705 5095  
4706 5096          if (update_queue_flags) {
4707 5097                  uint8_t cmd = (dbuf->db_handle << 5) | ITASK_CMD_DATA_XFER_DONE;
4708 5098  
     5099 +                ASSERT((itask->itask_flags & ITASK_IN_FREE_LIST) == 0);
4709 5100                  ASSERT(itask->itask_ncmds < ITASK_MAX_NCMDS);
     5101 +
4710 5102                  itask->itask_cmd_stack[itask->itask_ncmds++] = cmd;
4711 5103                  if (queue_it) {
4712      -                        itask->itask_worker_next = NULL;
4713      -                        if (w->worker_task_tail) {
4714      -                                w->worker_task_tail->itask_worker_next = itask;
4715      -                        } else {
4716      -                                w->worker_task_head = itask;
4717      -                        }
4718      -                        w->worker_task_tail = itask;
4719      -                        /* Measure task waitq time */
4720      -                        itask->itask_waitq_enter_timestamp = gethrtime();
4721      -                        if (++(w->worker_queue_depth) >
4722      -                            w->worker_max_qdepth_pu) {
4723      -                                w->worker_max_qdepth_pu = w->worker_queue_depth;
4724      -                        }
4725      -                        if ((w->worker_flags & STMF_WORKER_ACTIVE) == 0)
4726      -                                cv_signal(&w->worker_cv);
     5104 +                        STMF_ENQUEUE_ITASK(w, itask);
4727 5105                  }
     5106 +                mutex_exit(&w->worker_lock);
     5107 +                mutex_exit(&itask->itask_mutex);
     5108 +                return;
4728 5109          }
4729      -        mutex_exit(&w->worker_lock);
4730 5110  
     5111 +        mutex_exit(&w->worker_lock);
4731 5112          if (free_it) {
4732 5113                  if ((itask->itask_flags & (ITASK_KNOWN_TO_LU |
4733 5114                      ITASK_KNOWN_TO_TGT_PORT | ITASK_IN_WORKER_QUEUE |
4734 5115                      ITASK_BEING_ABORTED)) == 0) {
4735 5116                          stmf_task_free(task);
     5117 +                        return;
4736 5118                  }
4737 5119          }
     5120 +        mutex_exit(&itask->itask_mutex);
4738 5121  }
4739 5122  
4740 5123  stmf_status_t
4741 5124  stmf_send_scsi_status(scsi_task_t *task, uint32_t ioflags)
4742 5125  {
4743 5126          DTRACE_PROBE1(scsi__send__status, scsi_task_t *, task);
4744 5127  
4745 5128          stmf_i_scsi_task_t *itask =
4746 5129              (stmf_i_scsi_task_t *)task->task_stmf_private;
4747 5130  
4748 5131          stmf_task_audit(itask, TE_SEND_STATUS, ioflags, NULL);
4749 5132  
     5133 +        mutex_enter(&itask->itask_mutex);
4750 5134          if (ioflags & STMF_IOF_LU_DONE) {
4751      -                uint32_t new, old;
4752      -                do {
4753      -                        new = old = itask->itask_flags;
4754      -                        if (new & ITASK_BEING_ABORTED)
4755      -                                return (STMF_ABORTED);
4756      -                        new &= ~ITASK_KNOWN_TO_LU;
4757      -                } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     5135 +                if (itask->itask_flags & ITASK_BEING_ABORTED) {
     5136 +                        mutex_exit(&itask->itask_mutex);
     5137 +                        return (STMF_ABORTED);
     5138 +                }
     5139 +                itask->itask_flags &= ~ITASK_KNOWN_TO_LU;
4758 5140          }
4759 5141  
4760 5142          if (!(itask->itask_flags & ITASK_KNOWN_TO_TGT_PORT)) {
     5143 +                mutex_exit(&itask->itask_mutex);
4761 5144                  return (STMF_SUCCESS);
4762 5145          }
4763 5146  
4764      -        if (itask->itask_flags & ITASK_BEING_ABORTED)
     5147 +        if (itask->itask_flags & ITASK_BEING_ABORTED) {
     5148 +                mutex_exit(&itask->itask_mutex);
4765 5149                  return (STMF_ABORTED);
     5150 +        }
     5151 +        mutex_exit(&itask->itask_mutex);
4766 5152  
4767 5153          if (task->task_additional_flags & TASK_AF_NO_EXPECTED_XFER_LENGTH) {
4768 5154                  task->task_status_ctrl = 0;
4769 5155                  task->task_resid = 0;
4770 5156          } else if (task->task_cmd_xfer_length >
4771 5157              task->task_expected_xfer_length) {
4772 5158                  task->task_status_ctrl = TASK_SCTRL_OVER;
4773 5159                  task->task_resid = task->task_cmd_xfer_length -
4774 5160                      task->task_expected_xfer_length;
4775 5161          } else if (task->task_nbytes_transferred <
↓ open down ↓ 7 lines elided ↑ open up ↑
4783 5169          }
4784 5170          return (task->task_lport->lport_send_status(task, ioflags));
4785 5171  }
4786 5172  
4787 5173  void
4788 5174  stmf_send_status_done(scsi_task_t *task, stmf_status_t s, uint32_t iof)
4789 5175  {
4790 5176          stmf_i_scsi_task_t *itask =
4791 5177              (stmf_i_scsi_task_t *)task->task_stmf_private;
4792 5178          stmf_worker_t *w = itask->itask_worker;
4793      -        uint32_t new, old;
     5179 +        uint32_t new;
4794 5180          uint8_t free_it, queue_it;
4795 5181  
4796 5182          stmf_task_audit(itask, TE_SEND_STATUS_DONE, iof, NULL);
4797 5183  
     5184 +        mutex_enter(&itask->itask_mutex);
4798 5185          mutex_enter(&w->worker_lock);
4799      -        do {
4800      -                new = old = itask->itask_flags;
4801      -                if (old & ITASK_BEING_ABORTED) {
4802      -                        mutex_exit(&w->worker_lock);
4803      -                        return;
4804      -                }
     5186 +        new = itask->itask_flags;
     5187 +        if (itask->itask_flags & ITASK_BEING_ABORTED) {
     5188 +                mutex_exit(&w->worker_lock);
     5189 +                mutex_exit(&itask->itask_mutex);
     5190 +                return;
     5191 +        }
     5192 +        free_it = 0;
     5193 +        if (iof & STMF_IOF_LPORT_DONE) {
     5194 +                new &= ~ITASK_KNOWN_TO_TGT_PORT;
     5195 +                free_it = 1;
     5196 +        }
     5197 +        /*
     5198 +         * If the task is known to LU then queue it. But if
     5199 +         * it is already queued (multiple completions) then
     5200 +         * just update the buffer information by grabbing the
     5201 +         * worker lock. If the task is not known to LU,
     5202 +         * completed/aborted, then see if we need to
     5203 +         * free this task.
     5204 +         */
     5205 +        if (itask->itask_flags & ITASK_KNOWN_TO_LU) {
4805 5206                  free_it = 0;
4806      -                if (iof & STMF_IOF_LPORT_DONE) {
4807      -                        new &= ~ITASK_KNOWN_TO_TGT_PORT;
4808      -                        free_it = 1;
     5207 +                queue_it = 1;
     5208 +                if (itask->itask_flags & ITASK_IN_WORKER_QUEUE) {
     5209 +                        cmn_err(CE_PANIC, "status completion received"
     5210 +                            " when task is already in worker queue "
     5211 +                            " task = %p", (void *)task);
4809 5212                  }
4810      -                /*
4811      -                 * If the task is known to LU then queue it. But if
4812      -                 * it is already queued (multiple completions) then
4813      -                 * just update the buffer information by grabbing the
4814      -                 * worker lock. If the task is not known to LU,
4815      -                 * completed/aborted, then see if we need to
4816      -                 * free this task.
4817      -                 */
4818      -                if (old & ITASK_KNOWN_TO_LU) {
4819      -                        free_it = 0;
4820      -                        queue_it = 1;
4821      -                        if (old & ITASK_IN_WORKER_QUEUE) {
4822      -                                cmn_err(CE_PANIC, "status completion received"
4823      -                                    " when task is already in worker queue "
4824      -                                    " task = %p", (void *)task);
4825      -                        }
4826      -                        new |= ITASK_IN_WORKER_QUEUE;
4827      -                } else {
4828      -                        queue_it = 0;
4829      -                }
4830      -        } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     5213 +        } else {
     5214 +                queue_it = 0;
     5215 +        }
     5216 +        itask->itask_flags = new;
4831 5217          task->task_completion_status = s;
4832 5218  
4833      -
4834 5219          if (queue_it) {
4835 5220                  ASSERT(itask->itask_ncmds < ITASK_MAX_NCMDS);
4836 5221                  itask->itask_cmd_stack[itask->itask_ncmds++] =
4837 5222                      ITASK_CMD_STATUS_DONE;
4838      -                itask->itask_worker_next = NULL;
4839      -                if (w->worker_task_tail) {
4840      -                        w->worker_task_tail->itask_worker_next = itask;
4841      -                } else {
4842      -                        w->worker_task_head = itask;
4843      -                }
4844      -                w->worker_task_tail = itask;
4845      -                /* Measure task waitq time */
4846      -                itask->itask_waitq_enter_timestamp = gethrtime();
4847      -                if (++(w->worker_queue_depth) > w->worker_max_qdepth_pu) {
4848      -                        w->worker_max_qdepth_pu = w->worker_queue_depth;
4849      -                }
4850      -                if ((w->worker_flags & STMF_WORKER_ACTIVE) == 0)
4851      -                        cv_signal(&w->worker_cv);
     5223 +
     5224 +                STMF_ENQUEUE_ITASK(w, itask);
     5225 +                mutex_exit(&w->worker_lock);
     5226 +                mutex_exit(&itask->itask_mutex);
     5227 +                return;
4852 5228          }
     5229 +
4853 5230          mutex_exit(&w->worker_lock);
4854 5231  
4855 5232          if (free_it) {
4856 5233                  if ((itask->itask_flags & (ITASK_KNOWN_TO_LU |
4857 5234                      ITASK_KNOWN_TO_TGT_PORT | ITASK_IN_WORKER_QUEUE |
4858 5235                      ITASK_BEING_ABORTED)) == 0) {
4859 5236                          stmf_task_free(task);
     5237 +                        return;
4860 5238                  } else {
4861 5239                          cmn_err(CE_PANIC, "LU is done with the task but LPORT "
4862 5240                              " is not done, itask %p itask_flags %x",
4863 5241                              (void *)itask, itask->itask_flags);
4864 5242                  }
4865 5243          }
     5244 +        mutex_exit(&itask->itask_mutex);
4866 5245  }
4867 5246  
4868 5247  void
4869 5248  stmf_task_lu_done(scsi_task_t *task)
4870 5249  {
4871 5250          stmf_i_scsi_task_t *itask =
4872 5251              (stmf_i_scsi_task_t *)task->task_stmf_private;
4873 5252          stmf_worker_t *w = itask->itask_worker;
4874      -        uint32_t new, old;
4875 5253  
     5254 +        mutex_enter(&itask->itask_mutex);
4876 5255          mutex_enter(&w->worker_lock);
4877      -        do {
4878      -                new = old = itask->itask_flags;
4879      -                if (old & ITASK_BEING_ABORTED) {
4880      -                        mutex_exit(&w->worker_lock);
4881      -                        return;
4882      -                }
4883      -                if (old & ITASK_IN_WORKER_QUEUE) {
4884      -                        cmn_err(CE_PANIC, "task_lu_done received"
4885      -                            " when task is in worker queue "
4886      -                            " task = %p", (void *)task);
4887      -                }
4888      -                new &= ~ITASK_KNOWN_TO_LU;
4889      -        } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     5256 +        if (itask->itask_flags & ITASK_BEING_ABORTED) {
     5257 +                mutex_exit(&w->worker_lock);
     5258 +                mutex_exit(&itask->itask_mutex);
     5259 +                return;
     5260 +        }
     5261 +        if (itask->itask_flags & ITASK_IN_WORKER_QUEUE) {
     5262 +                cmn_err(CE_PANIC, "task_lu_done received"
     5263 +                    " when task is in worker queue "
     5264 +                    " task = %p", (void *)task);
     5265 +        }
     5266 +        itask->itask_flags &= ~ITASK_KNOWN_TO_LU;
4890 5267  
4891 5268          mutex_exit(&w->worker_lock);
4892      -
4893 5269          if ((itask->itask_flags & (ITASK_KNOWN_TO_LU |
4894 5270              ITASK_KNOWN_TO_TGT_PORT | ITASK_IN_WORKER_QUEUE |
4895 5271              ITASK_BEING_ABORTED)) == 0) {
4896 5272                  stmf_task_free(task);
     5273 +                return;
4897 5274          } else {
4898 5275                  cmn_err(CE_PANIC, "stmf_lu_done should be the last stage but "
4899 5276                      " the task is still not done, task = %p", (void *)task);
4900 5277          }
     5278 +        mutex_exit(&itask->itask_mutex);
4901 5279  }
4902 5280  
4903 5281  void
4904 5282  stmf_queue_task_for_abort(scsi_task_t *task, stmf_status_t s)
4905 5283  {
4906 5284          stmf_i_scsi_task_t *itask =
4907 5285              (stmf_i_scsi_task_t *)task->task_stmf_private;
4908 5286          stmf_worker_t *w;
4909      -        uint32_t old, new;
4910 5287  
4911 5288          stmf_task_audit(itask, TE_TASK_ABORT, CMD_OR_IOF_NA, NULL);
4912 5289  
4913      -        do {
4914      -                old = new = itask->itask_flags;
4915      -                if ((old & ITASK_BEING_ABORTED) ||
4916      -                    ((old & (ITASK_KNOWN_TO_TGT_PORT |
4917      -                    ITASK_KNOWN_TO_LU)) == 0)) {
4918      -                        return;
4919      -                }
4920      -                new |= ITASK_BEING_ABORTED;
4921      -        } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     5290 +        mutex_enter(&itask->itask_mutex);
     5291 +        if ((itask->itask_flags & ITASK_BEING_ABORTED) ||
     5292 +            ((itask->itask_flags & (ITASK_KNOWN_TO_TGT_PORT |
     5293 +            ITASK_KNOWN_TO_LU)) == 0)) {
     5294 +                mutex_exit(&itask->itask_mutex);
     5295 +                return;
     5296 +        }
     5297 +        itask->itask_flags |= ITASK_BEING_ABORTED;
4922 5298          task->task_completion_status = s;
4923      -        itask->itask_start_time = ddi_get_lbolt();
4924 5299  
4925 5300          if (((w = itask->itask_worker) == NULL) ||
4926 5301              (itask->itask_flags & ITASK_IN_TRANSITION)) {
     5302 +                mutex_exit(&itask->itask_mutex);
4927 5303                  return;
4928 5304          }
4929 5305  
4930 5306          /* Queue it and get out */
4931      -        mutex_enter(&w->worker_lock);
4932 5307          if (itask->itask_flags & ITASK_IN_WORKER_QUEUE) {
4933      -                mutex_exit(&w->worker_lock);
     5308 +                mutex_exit(&itask->itask_mutex);
4934 5309                  return;
4935 5310          }
4936      -        atomic_or_32(&itask->itask_flags, ITASK_IN_WORKER_QUEUE);
4937      -        itask->itask_worker_next = NULL;
4938      -        if (w->worker_task_tail) {
4939      -                w->worker_task_tail->itask_worker_next = itask;
4940      -        } else {
4941      -                w->worker_task_head = itask;
4942      -        }
4943      -        w->worker_task_tail = itask;
4944      -        if (++(w->worker_queue_depth) > w->worker_max_qdepth_pu) {
4945      -                w->worker_max_qdepth_pu = w->worker_queue_depth;
4946      -        }
4947      -        if ((w->worker_flags & STMF_WORKER_ACTIVE) == 0)
4948      -                cv_signal(&w->worker_cv);
     5311 +        mutex_enter(&w->worker_lock);
     5312 +        STMF_ENQUEUE_ITASK(w, itask);
4949 5313          mutex_exit(&w->worker_lock);
     5314 +        mutex_exit(&itask->itask_mutex);
4950 5315  }
4951 5316  
4952 5317  void
4953 5318  stmf_abort(int abort_cmd, scsi_task_t *task, stmf_status_t s, void *arg)
4954 5319  {
4955 5320          stmf_i_scsi_task_t *itask = NULL;
4956      -        uint32_t old, new, f, rf;
     5321 +        uint32_t f, rf;
4957 5322  
4958 5323          DTRACE_PROBE2(scsi__task__abort, scsi_task_t *, task,
4959 5324              stmf_status_t, s);
4960 5325  
4961 5326          switch (abort_cmd) {
4962 5327          case STMF_QUEUE_ABORT_LU:
4963 5328                  stmf_task_lu_killall((stmf_lu_t *)arg, task, s);
4964 5329                  return;
4965 5330          case STMF_QUEUE_TASK_ABORT:
4966 5331                  stmf_queue_task_for_abort(task, s);
↓ open down ↓ 2 lines elided ↑ open up ↑
4969 5334                  rf = ITASK_TGT_PORT_ABORT_CALLED;
4970 5335                  f = ITASK_KNOWN_TO_TGT_PORT;
4971 5336                  break;
4972 5337          case STMF_REQUEUE_TASK_ABORT_LU:
4973 5338                  rf = ITASK_LU_ABORT_CALLED;
4974 5339                  f = ITASK_KNOWN_TO_LU;
4975 5340                  break;
4976 5341          default:
4977 5342                  return;
4978 5343          }
     5344 +
4979 5345          itask = (stmf_i_scsi_task_t *)task->task_stmf_private;
     5346 +        mutex_enter(&itask->itask_mutex);
4980 5347          f |= ITASK_BEING_ABORTED | rf;
4981      -        do {
4982      -                old = new = itask->itask_flags;
4983      -                if ((old & f) != f) {
4984      -                        return;
4985      -                }
4986      -                new &= ~rf;
4987      -        } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     5348 +
     5349 +        if ((itask->itask_flags & f) != f) {
     5350 +                mutex_exit(&itask->itask_mutex);
     5351 +                return;
     5352 +        }
     5353 +        itask->itask_flags &= ~rf;
     5354 +        mutex_exit(&itask->itask_mutex);
     5355 +
4988 5356  }
4989 5357  
     5358 +/*
     5359 + * NOTE: stmf_abort_task_offline will release and then reacquire the
     5360 + * itask_mutex. This is required to prevent a lock order violation.
     5361 + */
4990 5362  void
4991 5363  stmf_task_lu_aborted(scsi_task_t *task, stmf_status_t s, uint32_t iof)
4992 5364  {
4993 5365          char                     info[STMF_CHANGE_INFO_LEN];
4994 5366          stmf_i_scsi_task_t      *itask = TASK_TO_ITASK(task);
4995 5367          unsigned long long      st;
4996 5368  
4997 5369          stmf_task_audit(itask, TE_TASK_LU_ABORTED, iof, NULL);
4998      -
     5370 +        ASSERT(mutex_owned(&itask->itask_mutex));
4999 5371          st = s; /* gcc fix */
5000 5372          if ((s != STMF_ABORT_SUCCESS) && (s != STMF_NOT_FOUND)) {
5001 5373                  (void) snprintf(info, sizeof (info),
5002 5374                      "task %p, lu failed to abort ret=%llx", (void *)task, st);
5003 5375          } else if ((iof & STMF_IOF_LU_DONE) == 0) {
5004 5376                  (void) snprintf(info, sizeof (info),
5005 5377                      "Task aborted but LU is not finished, task ="
5006 5378                      "%p, s=%llx, iof=%x", (void *)task, st, iof);
5007 5379          } else {
5008 5380                  /*
5009 5381                   * LU abort successfully
5010 5382                   */
5011 5383                  atomic_and_32(&itask->itask_flags, ~ITASK_KNOWN_TO_LU);
5012 5384                  return;
5013 5385          }
5014 5386  
5015 5387          stmf_abort_task_offline(task, 1, info);
5016 5388  }
5017 5389  
     5390 +/*
     5391 + * NOTE: stmf_abort_task_offline will release and then reacquire the
     5392 + * itask_mutex. This is required to prevent a lock order violation.
     5393 + */
5018 5394  void
5019 5395  stmf_task_lport_aborted(scsi_task_t *task, stmf_status_t s, uint32_t iof)
5020 5396  {
5021 5397          char                    info[STMF_CHANGE_INFO_LEN];
5022 5398          stmf_i_scsi_task_t      *itask = TASK_TO_ITASK(task);
5023 5399          unsigned long long      st;
5024      -        uint32_t                old, new;
5025 5400  
     5401 +        ASSERT(mutex_owned(&itask->itask_mutex));
5026 5402          stmf_task_audit(itask, TE_TASK_LPORT_ABORTED, iof, NULL);
5027      -
5028 5403          st = s;
5029 5404          if ((s != STMF_ABORT_SUCCESS) && (s != STMF_NOT_FOUND)) {
5030 5405                  (void) snprintf(info, sizeof (info),
5031 5406                      "task %p, tgt port failed to abort ret=%llx", (void *)task,
5032 5407                      st);
5033 5408          } else if ((iof & STMF_IOF_LPORT_DONE) == 0) {
5034 5409                  (void) snprintf(info, sizeof (info),
5035 5410                      "Task aborted but tgt port is not finished, "
5036 5411                      "task=%p, s=%llx, iof=%x", (void *)task, st, iof);
5037 5412          } else {
5038 5413                  /*
5039 5414                   * LPORT abort successfully
5040 5415                   */
5041      -                do {
5042      -                        old = new = itask->itask_flags;
5043      -                        if (!(old & ITASK_KNOWN_TO_TGT_PORT))
5044      -                                return;
5045      -                        new &= ~ITASK_KNOWN_TO_TGT_PORT;
5046      -                } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     5416 +                atomic_and_32(&itask->itask_flags, ~ITASK_KNOWN_TO_TGT_PORT);
5047 5417                  return;
5048 5418          }
5049 5419  
5050 5420          stmf_abort_task_offline(task, 0, info);
5051 5421  }
5052 5422  
     5423 +void
     5424 +stmf_task_lport_aborted_unlocked(scsi_task_t *task, stmf_status_t s,
     5425 +    uint32_t iof)
     5426 +{
     5427 +        stmf_i_scsi_task_t      *itask = TASK_TO_ITASK(task);
     5428 +
     5429 +        mutex_enter(&itask->itask_mutex);
     5430 +        stmf_task_lport_aborted(task, s, iof);
     5431 +        mutex_exit(&itask->itask_mutex);
     5432 +}
     5433 +
5053 5434  stmf_status_t
5054 5435  stmf_task_poll_lu(scsi_task_t *task, uint32_t timeout)
5055 5436  {
5056 5437          stmf_i_scsi_task_t *itask = (stmf_i_scsi_task_t *)
5057 5438              task->task_stmf_private;
5058 5439          stmf_worker_t *w = itask->itask_worker;
5059 5440          int i;
5060 5441  
     5442 +        mutex_enter(&itask->itask_mutex);
5061 5443          ASSERT(itask->itask_flags & ITASK_KNOWN_TO_LU);
5062 5444          mutex_enter(&w->worker_lock);
5063 5445          if (itask->itask_ncmds >= ITASK_MAX_NCMDS) {
5064 5446                  mutex_exit(&w->worker_lock);
     5447 +                mutex_exit(&itask->itask_mutex);
5065 5448                  return (STMF_BUSY);
5066 5449          }
5067 5450          for (i = 0; i < itask->itask_ncmds; i++) {
5068 5451                  if (itask->itask_cmd_stack[i] == ITASK_CMD_POLL_LU) {
5069 5452                          mutex_exit(&w->worker_lock);
     5453 +                        mutex_exit(&itask->itask_mutex);
5070 5454                          return (STMF_SUCCESS);
5071 5455                  }
5072 5456          }
5073 5457          itask->itask_cmd_stack[itask->itask_ncmds++] = ITASK_CMD_POLL_LU;
5074 5458          if (timeout == ITASK_DEFAULT_POLL_TIMEOUT) {
5075 5459                  itask->itask_poll_timeout = ddi_get_lbolt() + 1;
5076 5460          } else {
5077 5461                  clock_t t = drv_usectohz(timeout * 1000);
5078 5462                  if (t == 0)
5079 5463                          t = 1;
5080 5464                  itask->itask_poll_timeout = ddi_get_lbolt() + t;
5081 5465          }
5082 5466          if ((itask->itask_flags & ITASK_IN_WORKER_QUEUE) == 0) {
5083      -                itask->itask_worker_next = NULL;
5084      -                if (w->worker_task_tail) {
5085      -                        w->worker_task_tail->itask_worker_next = itask;
5086      -                } else {
5087      -                        w->worker_task_head = itask;
5088      -                }
5089      -                w->worker_task_tail = itask;
5090      -                if (++(w->worker_queue_depth) > w->worker_max_qdepth_pu) {
5091      -                        w->worker_max_qdepth_pu = w->worker_queue_depth;
5092      -                }
5093      -                atomic_or_32(&itask->itask_flags, ITASK_IN_WORKER_QUEUE);
5094      -                if ((w->worker_flags & STMF_WORKER_ACTIVE) == 0)
5095      -                        cv_signal(&w->worker_cv);
     5467 +                STMF_ENQUEUE_ITASK(w, itask);
5096 5468          }
5097 5469          mutex_exit(&w->worker_lock);
     5470 +        mutex_exit(&itask->itask_mutex);
5098 5471          return (STMF_SUCCESS);
5099 5472  }
5100 5473  
5101 5474  stmf_status_t
5102 5475  stmf_task_poll_lport(scsi_task_t *task, uint32_t timeout)
5103 5476  {
5104 5477          stmf_i_scsi_task_t *itask = (stmf_i_scsi_task_t *)
5105 5478              task->task_stmf_private;
5106 5479          stmf_worker_t *w = itask->itask_worker;
5107 5480          int i;
5108 5481  
     5482 +        mutex_enter(&itask->itask_mutex);
5109 5483          ASSERT(itask->itask_flags & ITASK_KNOWN_TO_TGT_PORT);
5110 5484          mutex_enter(&w->worker_lock);
5111 5485          if (itask->itask_ncmds >= ITASK_MAX_NCMDS) {
5112 5486                  mutex_exit(&w->worker_lock);
     5487 +                mutex_exit(&itask->itask_mutex);
5113 5488                  return (STMF_BUSY);
5114 5489          }
5115 5490          for (i = 0; i < itask->itask_ncmds; i++) {
5116 5491                  if (itask->itask_cmd_stack[i] == ITASK_CMD_POLL_LPORT) {
5117 5492                          mutex_exit(&w->worker_lock);
     5493 +                        mutex_exit(&itask->itask_mutex);
5118 5494                          return (STMF_SUCCESS);
5119 5495                  }
5120 5496          }
5121 5497          itask->itask_cmd_stack[itask->itask_ncmds++] = ITASK_CMD_POLL_LPORT;
5122 5498          if (timeout == ITASK_DEFAULT_POLL_TIMEOUT) {
5123 5499                  itask->itask_poll_timeout = ddi_get_lbolt() + 1;
5124 5500          } else {
5125 5501                  clock_t t = drv_usectohz(timeout * 1000);
5126 5502                  if (t == 0)
5127 5503                          t = 1;
5128 5504                  itask->itask_poll_timeout = ddi_get_lbolt() + t;
5129 5505          }
5130 5506          if ((itask->itask_flags & ITASK_IN_WORKER_QUEUE) == 0) {
5131      -                itask->itask_worker_next = NULL;
5132      -                if (w->worker_task_tail) {
5133      -                        w->worker_task_tail->itask_worker_next = itask;
5134      -                } else {
5135      -                        w->worker_task_head = itask;
5136      -                }
5137      -                w->worker_task_tail = itask;
5138      -                if (++(w->worker_queue_depth) > w->worker_max_qdepth_pu) {
5139      -                        w->worker_max_qdepth_pu = w->worker_queue_depth;
5140      -                }
5141      -                if ((w->worker_flags & STMF_WORKER_ACTIVE) == 0)
5142      -                        cv_signal(&w->worker_cv);
     5507 +                STMF_ENQUEUE_ITASK(w, itask);
5143 5508          }
5144 5509          mutex_exit(&w->worker_lock);
     5510 +        mutex_exit(&itask->itask_mutex);
5145 5511          return (STMF_SUCCESS);
5146 5512  }
5147 5513  
5148 5514  void
5149 5515  stmf_do_task_abort(scsi_task_t *task)
5150 5516  {
5151 5517          stmf_i_scsi_task_t      *itask = TASK_TO_ITASK(task);
5152 5518          stmf_lu_t               *lu;
5153 5519          stmf_local_port_t       *lport;
5154 5520          unsigned long long       ret;
5155      -        uint32_t                 old, new;
     5521 +        uint32_t                 new = 0;
5156 5522          uint8_t                  call_lu_abort, call_port_abort;
5157 5523          char                     info[STMF_CHANGE_INFO_LEN];
5158 5524  
5159 5525          lu = task->task_lu;
5160 5526          lport = task->task_lport;
5161      -        do {
5162      -                old = new = itask->itask_flags;
5163      -                if ((old & (ITASK_KNOWN_TO_LU | ITASK_LU_ABORT_CALLED)) ==
5164      -                    ITASK_KNOWN_TO_LU) {
5165      -                        new |= ITASK_LU_ABORT_CALLED;
5166      -                        call_lu_abort = 1;
5167      -                } else {
5168      -                        call_lu_abort = 0;
5169      -                }
5170      -        } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     5527 +        mutex_enter(&itask->itask_mutex);
     5528 +        new = itask->itask_flags;
     5529 +        if ((itask->itask_flags & (ITASK_KNOWN_TO_LU |
     5530 +            ITASK_LU_ABORT_CALLED)) == ITASK_KNOWN_TO_LU) {
     5531 +                new |= ITASK_LU_ABORT_CALLED;
     5532 +                call_lu_abort = 1;
     5533 +        } else {
     5534 +                call_lu_abort = 0;
     5535 +        }
     5536 +        itask->itask_flags = new;
5171 5537  
5172 5538          if (call_lu_abort) {
5173 5539                  if ((itask->itask_flags & ITASK_DEFAULT_HANDLING) == 0) {
5174 5540                          ret = lu->lu_abort(lu, STMF_LU_ABORT_TASK, task, 0);
5175 5541                  } else {
5176 5542                          ret = dlun0->lu_abort(lu, STMF_LU_ABORT_TASK, task, 0);
5177 5543                  }
5178 5544                  if ((ret == STMF_ABORT_SUCCESS) || (ret == STMF_NOT_FOUND)) {
5179 5545                          stmf_task_lu_aborted(task, ret, STMF_IOF_LU_DONE);
5180 5546                  } else if (ret == STMF_BUSY) {
↓ open down ↓ 7 lines elided ↑ open up ↑
5188 5554          } else if (itask->itask_flags & ITASK_KNOWN_TO_LU) {
5189 5555                  if (ddi_get_lbolt() > (itask->itask_start_time +
5190 5556                      STMF_SEC2TICK(lu->lu_abort_timeout?
5191 5557                      lu->lu_abort_timeout : ITASK_DEFAULT_ABORT_TIMEOUT))) {
5192 5558                          (void) snprintf(info, sizeof (info),
5193 5559                              "lu abort timed out");
5194 5560                          stmf_abort_task_offline(itask->itask_task, 1, info);
5195 5561                  }
5196 5562          }
5197 5563  
5198      -        do {
5199      -                old = new = itask->itask_flags;
5200      -                if ((old & (ITASK_KNOWN_TO_TGT_PORT |
5201      -                    ITASK_TGT_PORT_ABORT_CALLED)) == ITASK_KNOWN_TO_TGT_PORT) {
5202      -                        new |= ITASK_TGT_PORT_ABORT_CALLED;
5203      -                        call_port_abort = 1;
5204      -                } else {
5205      -                        call_port_abort = 0;
5206      -                }
5207      -        } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     5564 +        /*
     5565 +         * NOTE: After the call to either stmf_abort_task_offline() or
     5566 +         * stmf_task_lu_abort() the itask_mutex was dropped and reacquired
     5567 +         * to avoid a deadlock situation with stmf_state.stmf_lock.
     5568 +         */
     5569 +
     5570 +        new = itask->itask_flags;
     5571 +        if ((itask->itask_flags & (ITASK_KNOWN_TO_TGT_PORT |
     5572 +            ITASK_TGT_PORT_ABORT_CALLED)) == ITASK_KNOWN_TO_TGT_PORT) {
     5573 +                new |= ITASK_TGT_PORT_ABORT_CALLED;
     5574 +                call_port_abort = 1;
     5575 +        } else {
     5576 +                call_port_abort = 0;
     5577 +        }
     5578 +        itask->itask_flags = new;
     5579 +
5208 5580          if (call_port_abort) {
5209 5581                  ret = lport->lport_abort(lport, STMF_LPORT_ABORT_TASK, task, 0);
5210 5582                  if ((ret == STMF_ABORT_SUCCESS) || (ret == STMF_NOT_FOUND)) {
5211 5583                          stmf_task_lport_aborted(task, ret, STMF_IOF_LPORT_DONE);
5212 5584                  } else if (ret == STMF_BUSY) {
5213 5585                          atomic_and_32(&itask->itask_flags,
5214 5586                              ~ITASK_TGT_PORT_ABORT_CALLED);
5215 5587                  } else if (ret != STMF_SUCCESS) {
5216 5588                          (void) snprintf(info, sizeof (info),
5217 5589                              "Abort failed by tgt port %p ret %llx",
↓ open down ↓ 3 lines elided ↑ open up ↑
5221 5593          } else if (itask->itask_flags & ITASK_KNOWN_TO_TGT_PORT) {
5222 5594                  if (ddi_get_lbolt() > (itask->itask_start_time +
5223 5595                      STMF_SEC2TICK(lport->lport_abort_timeout?
5224 5596                      lport->lport_abort_timeout :
5225 5597                      ITASK_DEFAULT_ABORT_TIMEOUT))) {
5226 5598                          (void) snprintf(info, sizeof (info),
5227 5599                              "lport abort timed out");
5228 5600                          stmf_abort_task_offline(itask->itask_task, 0, info);
5229 5601                  }
5230 5602          }
     5603 +        mutex_exit(&itask->itask_mutex);
5231 5604  }
5232 5605  
5233 5606  stmf_status_t
5234 5607  stmf_ctl(int cmd, void *obj, void *arg)
5235 5608  {
5236 5609          stmf_status_t                   ret;
5237 5610          stmf_i_lu_t                     *ilu;
5238 5611          stmf_i_local_port_t             *ilport;
5239 5612          stmf_state_change_info_t        *ssci = (stmf_state_change_info_t *)arg;
5240 5613  
↓ open down ↓ 251 lines elided ↑ open up ↑
5492 5865          return (STMF_SUCCESS);
5493 5866  
5494 5867  stmf_ctl_lock_exit:;
5495 5868          mutex_exit(&stmf_state.stmf_lock);
5496 5869          return (ret);
5497 5870  }
5498 5871  
5499 5872  /* ARGSUSED */
5500 5873  stmf_status_t
5501 5874  stmf_info_impl(uint32_t cmd, void *arg1, void *arg2, uint8_t *buf,
5502      -                                                uint32_t *bufsizep)
     5875 +    uint32_t *bufsizep)
5503 5876  {
5504 5877          return (STMF_NOT_SUPPORTED);
5505 5878  }
5506 5879  
5507 5880  /* ARGSUSED */
5508 5881  stmf_status_t
5509 5882  stmf_info(uint32_t cmd, void *arg1, void *arg2, uint8_t *buf,
5510      -                                                uint32_t *bufsizep)
     5883 +    uint32_t *bufsizep)
5511 5884  {
5512 5885          uint32_t cl = SI_GET_CLASS(cmd);
5513 5886  
5514 5887          if (cl == SI_STMF) {
5515 5888                  return (stmf_info_impl(cmd, arg1, arg2, buf, bufsizep));
5516 5889          }
5517 5890          if (cl == SI_LPORT) {
5518 5891                  return (((stmf_local_port_t *)arg1)->lport_info(cmd, arg1,
5519 5892                      arg2, buf, bufsizep));
5520 5893          } else if (cl == SI_LU) {
↓ open down ↓ 31 lines elided ↑ open up ↑
5552 5925  stmf_xfer_data_t *
5553 5926  stmf_prepare_tpgs_data(uint8_t ilu_alua)
5554 5927  {
5555 5928          stmf_xfer_data_t *xd;
5556 5929          stmf_i_local_port_t *ilport;
5557 5930          uint8_t *p;
5558 5931          uint32_t sz, asz, nports = 0, nports_standby = 0;
5559 5932  
5560 5933          mutex_enter(&stmf_state.stmf_lock);
5561 5934          /* check if any ports are standby and create second group */
5562      -        for (ilport = stmf_state.stmf_ilportlist; ilport;
     5935 +        for (ilport = stmf_state.stmf_ilportlist; ilport != NULL;
5563 5936              ilport = ilport->ilport_next) {
5564 5937                  if (ilport->ilport_standby == 1) {
5565 5938                          nports_standby++;
5566 5939                  } else {
5567 5940                          nports++;
5568 5941                  }
5569 5942          }
5570 5943  
5571      -        /* The spec only allows for 255 ports to be reported per group */
     5944 +        /*
     5945 +         * Section 6.25 REPORT TARGET PORT GROUPS
     5946 +         * The reply can contain many group replies. Each group is limited
     5947 +         * to 255 port identifiers so we'll need to limit the amount of
     5948 +         * data returned. For FC ports there's a physical limitation in
     5949 +         * machines that make reaching 255 ports very, very unlikely. For
     5950 +         * iSCSI on the other hand recent changes mean the port count could
     5951 +         * be as high as 4096 (current limit). Limiting the data returned
     5952 +         * for iSCSI isn't as bad as it sounds. This information is only
     5953 +         * important for ALUA, which isn't supported for iSCSI. iSCSI uses
     5954 +         * virtual IP addresses to deal with node fail over in a cluster.
     5955 +         */
5572 5956          nports = min(nports, 255);
5573 5957          nports_standby = min(nports_standby, 255);
     5958 +
     5959 +        /*
     5960 +         * The first 4 bytes of the returned data is the length. The
     5961 +         * size of the Target Port Group header is 8 bytes. So, that's where
     5962 +         * the 12 comes from. Each port entry is 4 bytes in size.
     5963 +         */
5574 5964          sz = (nports * 4) + 12;
5575      -        if (nports_standby && ilu_alua) {
     5965 +        if (nports_standby != 0 && ilu_alua != 0) {
     5966 +                /* --- Only add 8 bytes since it's just the Group header ----*/
5576 5967                  sz += (nports_standby * 4) + 8;
5577 5968          }
5578      -        asz = sz + sizeof (*xd) - 4;
     5969 +
     5970 +        /*
     5971 +         * The stmf_xfer_data structure contains 4 bytes that will be
     5972 +         * part of the data buffer. So, subtract the 4 bytes from the space
     5973 +         * needed.
     5974 +         */
     5975 +        asz = sizeof (*xd) + sz - 4;
5579 5976          xd = (stmf_xfer_data_t *)kmem_zalloc(asz, KM_NOSLEEP);
5580 5977          if (xd == NULL) {
5581 5978                  mutex_exit(&stmf_state.stmf_lock);
5582 5979                  return (NULL);
5583 5980          }
5584 5981          xd->alloc_size = asz;
5585 5982          xd->size_left = sz;
5586 5983  
5587 5984          p = xd->buf;
5588 5985  
     5986 +        /* ---- length values never include the field that holds the size ----*/
5589 5987          *((uint32_t *)p) = BE_32(sz - 4);
5590 5988          p += 4;
     5989 +
     5990 +        /* ---- Now fill out the first Target Group header ---- */
5591 5991          p[0] = 0x80;    /* PREF */
5592 5992          p[1] = 5;       /* AO_SUP, S_SUP */
5593 5993          if (stmf_state.stmf_alua_node == 1) {
5594 5994                  p[3] = 1;       /* Group 1 */
5595 5995          } else {
5596 5996                  p[3] = 0;       /* Group 0 */
5597 5997          }
5598 5998          p[7] = nports & 0xff;
5599 5999          p += 8;
5600      -        for (ilport = stmf_state.stmf_ilportlist; ilport;
     6000 +        for (ilport = stmf_state.stmf_ilportlist; ilport != NULL && nports != 0;
5601 6001              ilport = ilport->ilport_next) {
5602 6002                  if (ilport->ilport_standby == 1) {
5603 6003                          continue;
5604 6004                  }
5605 6005                  ((uint16_t *)p)[1] = BE_16(ilport->ilport_rtpid);
5606 6006                  p += 4;
     6007 +                nports--;
5607 6008          }
5608      -        if (nports_standby && ilu_alua) {
     6009 +        if (nports_standby != 0 && ilu_alua != 0) {
5609 6010                  p[0] = 0x02;    /* Non PREF, Standby */
5610 6011                  p[1] = 5;       /* AO_SUP, S_SUP */
5611 6012                  if (stmf_state.stmf_alua_node == 1) {
5612 6013                          p[3] = 0;       /* Group 0 */
5613 6014                  } else {
5614 6015                          p[3] = 1;       /* Group 1 */
5615 6016                  }
5616 6017                  p[7] = nports_standby & 0xff;
5617 6018                  p += 8;
5618      -                for (ilport = stmf_state.stmf_ilportlist; ilport;
5619      -                    ilport = ilport->ilport_next) {
     6019 +                for (ilport = stmf_state.stmf_ilportlist; ilport != NULL &&
     6020 +                    nports_standby != 0; ilport = ilport->ilport_next) {
5620 6021                          if (ilport->ilport_standby == 0) {
5621 6022                                  continue;
5622 6023                          }
5623 6024                          ((uint16_t *)p)[1] = BE_16(ilport->ilport_rtpid);
5624 6025                          p += 4;
     6026 +                        nports_standby--;
5625 6027                  }
5626 6028          }
5627 6029  
5628 6030          mutex_exit(&stmf_state.stmf_lock);
5629 6031  
5630 6032          return (xd);
5631 6033  }
5632 6034  
5633 6035  struct scsi_devid_desc *
5634 6036  stmf_scsilib_get_devid_desc(uint16_t rtpid)
↓ open down ↓ 220 lines elided ↑ open up ↑
5855 6257  void
5856 6258  stmf_scsilib_handle_report_tpgs(scsi_task_t *task, stmf_data_buf_t *dbuf)
5857 6259  {
5858 6260          stmf_i_scsi_task_t *itask =
5859 6261              (stmf_i_scsi_task_t *)task->task_stmf_private;
5860 6262          stmf_i_lu_t *ilu =
5861 6263              (stmf_i_lu_t *)task->task_lu->lu_stmf_private;
5862 6264          stmf_xfer_data_t *xd;
5863 6265          uint32_t sz, minsz;
5864 6266  
     6267 +        mutex_enter(&itask->itask_mutex);
5865 6268          itask->itask_flags |= ITASK_DEFAULT_HANDLING;
     6269 +
5866 6270          task->task_cmd_xfer_length =
5867 6271              ((((uint32_t)task->task_cdb[6]) << 24) |
5868 6272              (((uint32_t)task->task_cdb[7]) << 16) |
5869 6273              (((uint32_t)task->task_cdb[8]) << 8) |
5870 6274              ((uint32_t)task->task_cdb[9]));
5871 6275  
5872 6276          if (task->task_additional_flags &
5873 6277              TASK_AF_NO_EXPECTED_XFER_LENGTH) {
5874 6278                  task->task_expected_xfer_length =
5875 6279                      task->task_cmd_xfer_length;
5876 6280          }
     6281 +        mutex_exit(&itask->itask_mutex);
5877 6282  
5878 6283          if (task->task_cmd_xfer_length == 0) {
5879 6284                  stmf_scsilib_send_status(task, STATUS_GOOD, 0);
5880 6285                  return;
5881 6286          }
5882 6287          if (task->task_cmd_xfer_length < 4) {
5883 6288                  stmf_scsilib_send_status(task, STATUS_CHECK,
5884 6289                      STMF_SAA_INVALID_FIELD_IN_CDB);
5885 6290                  return;
5886 6291          }
↓ open down ↓ 85 lines elided ↑ open up ↑
5972 6377                  return;
5973 6378          }
5974 6379          atomic_or_32(&ilu->ilu_flags, ILU_RESET_ACTIVE);
5975 6380          mutex_exit(&stmf_state.stmf_lock);
5976 6381  
5977 6382          /*
5978 6383           * Mark this task as the one causing LU reset so that we know who
5979 6384           * was responsible for setting the ILU_RESET_ACTIVE. In case this
5980 6385           * task itself gets aborted, we will clear ILU_RESET_ACTIVE.
5981 6386           */
     6387 +        mutex_enter(&itask->itask_mutex);
5982 6388          itask->itask_flags |= ITASK_DEFAULT_HANDLING | ITASK_CAUSING_LU_RESET;
     6389 +        mutex_exit(&itask->itask_mutex);
5983 6390  
5984 6391          /* Initiatiate abort on all commands on this LU except this one */
5985 6392          stmf_abort(STMF_QUEUE_ABORT_LU, task, STMF_ABORTED, task->task_lu);
5986 6393  
5987 6394          /* Start polling on this task */
5988 6395          if (stmf_task_poll_lu(task, ITASK_DEFAULT_POLL_TIMEOUT)
5989 6396              != STMF_SUCCESS) {
5990 6397                  stmf_abort(STMF_QUEUE_TASK_ABORT, task, STMF_ALLOC_FAILURE,
5991 6398                      NULL);
5992 6399                  return;
↓ open down ↓ 55 lines elided ↑ open up ↑
6048 6455          if (lf == 0) {
6049 6456                  /* No luns in this session */
6050 6457                  atomic_and_32(&iss->iss_flags, ~ISS_RESET_ACTIVE);
6051 6458                  rw_exit(iss->iss_lockp);
6052 6459                  mutex_exit(&stmf_state.stmf_lock);
6053 6460                  stmf_scsilib_send_status(task, STATUS_GOOD, 0);
6054 6461                  return;
6055 6462          }
6056 6463  
6057 6464          /* ok, start the damage */
     6465 +        mutex_enter(&itask->itask_mutex);
6058 6466          itask->itask_flags |= ITASK_DEFAULT_HANDLING |
6059 6467              ITASK_CAUSING_TARGET_RESET;
     6468 +        mutex_exit(&itask->itask_mutex);
6060 6469          for (i = 0; i < lm->lm_nentries; i++) {
6061 6470                  if (lm->lm_plus[i] == NULL)
6062 6471                          continue;
6063 6472                  lm_ent = (stmf_lun_map_ent_t *)lm->lm_plus[i];
6064 6473                  ilu = (stmf_i_lu_t *)(lm_ent->ent_lu->lu_stmf_private);
6065 6474                  atomic_or_32(&ilu->ilu_flags, ILU_RESET_ACTIVE);
6066 6475          }
6067 6476  
6068 6477          for (i = 0; i < lm->lm_nentries; i++) {
6069 6478                  if (lm->lm_plus[i] == NULL)
↓ open down ↓ 37 lines elided ↑ open up ↑
6107 6516          }
6108 6517          stmf_scsilib_send_status(task, STATUS_CHECK,
6109 6518              STMF_SAA_REPORT_LUN_DATA_HAS_CHANGED);
6110 6519          return (1);
6111 6520  }
6112 6521  
6113 6522  void
6114 6523  stmf_worker_init()
6115 6524  {
6116 6525          uint32_t i;
     6526 +        stmf_worker_t *w;
6117 6527  
6118 6528          /* Make local copy of global tunables */
6119      -        stmf_i_max_nworkers = stmf_max_nworkers;
6120      -        stmf_i_min_nworkers = stmf_min_nworkers;
6121 6529  
     6530 +        /*
     6531 +         * Allow workers to be scaled down to a very low number for cases
     6532 +         * where the load is light.  If the number of threads gets below
     6533 +         * 4 assume it is a mistake and force the threads back to a
     6534 +         * reasonable number.  The low limit of 4 is simply legacy and
     6535 +         * may be too low.
     6536 +         */
6122 6537          ASSERT(stmf_workers == NULL);
6123      -        if (stmf_i_min_nworkers < 4) {
6124      -                stmf_i_min_nworkers = 4;
     6538 +        if (stmf_nworkers < 4) {
     6539 +                stmf_nworkers = 64;
6125 6540          }
6126      -        if (stmf_i_max_nworkers < stmf_i_min_nworkers) {
6127      -                stmf_i_max_nworkers = stmf_i_min_nworkers;
6128      -        }
     6541 +
6129 6542          stmf_workers = (stmf_worker_t *)kmem_zalloc(
6130      -            sizeof (stmf_worker_t) * stmf_i_max_nworkers, KM_SLEEP);
6131      -        for (i = 0; i < stmf_i_max_nworkers; i++) {
     6543 +            sizeof (stmf_worker_t) * stmf_nworkers, KM_SLEEP);
     6544 +        for (i = 0; i < stmf_nworkers; i++) {
6132 6545                  stmf_worker_t *w = &stmf_workers[i];
6133 6546                  mutex_init(&w->worker_lock, NULL, MUTEX_DRIVER, NULL);
6134 6547                  cv_init(&w->worker_cv, NULL, CV_DRIVER, NULL);
6135 6548          }
6136      -        stmf_worker_mgmt_delay = drv_usectohz(20 * 1000);
6137 6549          stmf_workers_state = STMF_WORKERS_ENABLED;
6138 6550  
6139      -        /* Workers will be started by stmf_worker_mgmt() */
     6551 +        /* Check if we are starting */
     6552 +        if (stmf_nworkers_cur < stmf_nworkers - 1) {
     6553 +                for (i = stmf_nworkers_cur; i < stmf_nworkers; i++) {
     6554 +                        w = &stmf_workers[i];
     6555 +                        w->worker_tid = thread_create(NULL, 0, stmf_worker_task,
     6556 +                            (void *)&stmf_workers[i], 0, &p0, TS_RUN,
     6557 +                            minclsyspri);
     6558 +                        stmf_nworkers_accepting_cmds++;
     6559 +                }
     6560 +                return;
     6561 +        }
6140 6562  
6141 6563          /* Lets wait for atleast one worker to start */
6142 6564          while (stmf_nworkers_cur == 0)
6143 6565                  delay(drv_usectohz(20 * 1000));
6144      -        stmf_worker_mgmt_delay = drv_usectohz(3 * 1000 * 1000);
6145 6566  }
6146 6567  
6147 6568  stmf_status_t
6148 6569  stmf_worker_fini()
6149 6570  {
6150 6571          int i;
6151 6572          clock_t sb;
6152 6573  
6153 6574          if (stmf_workers_state == STMF_WORKERS_DISABLED)
6154 6575                  return (STMF_SUCCESS);
6155 6576          ASSERT(stmf_workers);
6156 6577          stmf_workers_state = STMF_WORKERS_DISABLED;
6157      -        stmf_worker_mgmt_delay = drv_usectohz(20 * 1000);
6158 6578          cv_signal(&stmf_state.stmf_cv);
6159 6579  
6160 6580          sb = ddi_get_lbolt() + drv_usectohz(10 * 1000 * 1000);
6161 6581          /* Wait for all the threads to die */
6162 6582          while (stmf_nworkers_cur != 0) {
6163 6583                  if (ddi_get_lbolt() > sb) {
6164 6584                          stmf_workers_state = STMF_WORKERS_ENABLED;
6165 6585                          return (STMF_BUSY);
6166 6586                  }
6167 6587                  delay(drv_usectohz(100 * 1000));
6168 6588          }
6169      -        for (i = 0; i < stmf_i_max_nworkers; i++) {
     6589 +        for (i = 0; i < stmf_nworkers; i++) {
6170 6590                  stmf_worker_t *w = &stmf_workers[i];
6171 6591                  mutex_destroy(&w->worker_lock);
6172 6592                  cv_destroy(&w->worker_cv);
6173 6593          }
6174      -        kmem_free(stmf_workers, sizeof (stmf_worker_t) * stmf_i_max_nworkers);
     6594 +        kmem_free(stmf_workers, sizeof (stmf_worker_t) * stmf_nworkers);
6175 6595          stmf_workers = NULL;
6176 6596  
6177 6597          return (STMF_SUCCESS);
6178 6598  }
6179 6599  
6180 6600  void
6181 6601  stmf_worker_task(void *arg)
6182 6602  {
6183 6603          stmf_worker_t *w;
6184 6604          stmf_i_scsi_session_t *iss;
6185 6605          scsi_task_t *task;
6186 6606          stmf_i_scsi_task_t *itask;
6187 6607          stmf_data_buf_t *dbuf;
6188 6608          stmf_lu_t *lu;
6189 6609          clock_t wait_timer = 0;
6190 6610          clock_t wait_ticks, wait_delta = 0;
6191      -        uint32_t old, new;
6192 6611          uint8_t curcmd;
6193 6612          uint8_t abort_free;
6194 6613          uint8_t wait_queue;
6195 6614          uint8_t dec_qdepth;
6196 6615  
6197 6616          w = (stmf_worker_t *)arg;
6198 6617          wait_ticks = drv_usectohz(10000);
6199 6618  
6200 6619          DTRACE_PROBE1(worker__create, stmf_worker_t, w);
6201 6620          mutex_enter(&w->worker_lock);
6202 6621          w->worker_flags |= STMF_WORKER_STARTED | STMF_WORKER_ACTIVE;
6203      -stmf_worker_loop:;
     6622 +        atomic_inc_32(&stmf_nworkers_cur);
     6623 +
     6624 +stmf_worker_loop:
6204 6625          if ((w->worker_ref_count == 0) &&
6205 6626              (w->worker_flags & STMF_WORKER_TERMINATE)) {
6206 6627                  w->worker_flags &= ~(STMF_WORKER_STARTED |
6207 6628                      STMF_WORKER_ACTIVE | STMF_WORKER_TERMINATE);
6208 6629                  w->worker_tid = NULL;
6209 6630                  mutex_exit(&w->worker_lock);
6210 6631                  DTRACE_PROBE1(worker__destroy, stmf_worker_t, w);
     6632 +                atomic_dec_32(&stmf_nworkers_cur);
6211 6633                  thread_exit();
6212 6634          }
     6635 +
6213 6636          /* CONSTCOND */
6214 6637          while (1) {
     6638 +                /* worker lock is held at this point */
6215 6639                  dec_qdepth = 0;
6216 6640                  if (wait_timer && (ddi_get_lbolt() >= wait_timer)) {
6217 6641                          wait_timer = 0;
6218 6642                          wait_delta = 0;
6219 6643                          if (w->worker_wait_head) {
6220 6644                                  ASSERT(w->worker_wait_tail);
6221 6645                                  if (w->worker_task_head == NULL)
6222 6646                                          w->worker_task_head =
6223 6647                                              w->worker_wait_head;
6224 6648                                  else
6225 6649                                          w->worker_task_tail->itask_worker_next =
6226 6650                                              w->worker_wait_head;
6227 6651                                  w->worker_task_tail = w->worker_wait_tail;
6228 6652                                  w->worker_wait_head = w->worker_wait_tail =
6229 6653                                      NULL;
6230 6654                          }
6231 6655                  }
6232      -                if ((itask = w->worker_task_head) == NULL) {
     6656 +
     6657 +                STMF_DEQUEUE_ITASK(w, itask);
     6658 +                if (itask == NULL)
6233 6659                          break;
6234      -                }
     6660 +
     6661 +                ASSERT((itask->itask_flags & ITASK_IN_FREE_LIST) == 0);
6235 6662                  task = itask->itask_task;
6236 6663                  DTRACE_PROBE2(worker__active, stmf_worker_t, w,
6237 6664                      scsi_task_t *, task);
6238      -                w->worker_task_head = itask->itask_worker_next;
6239      -                if (w->worker_task_head == NULL)
6240      -                        w->worker_task_tail = NULL;
6241      -
6242 6665                  wait_queue = 0;
6243 6666                  abort_free = 0;
     6667 +                mutex_exit(&w->worker_lock);
     6668 +                mutex_enter(&itask->itask_mutex);
     6669 +                mutex_enter(&w->worker_lock);
     6670 +
6244 6671                  if (itask->itask_ncmds > 0) {
6245 6672                          curcmd = itask->itask_cmd_stack[itask->itask_ncmds - 1];
6246 6673                  } else {
6247 6674                          ASSERT(itask->itask_flags & ITASK_BEING_ABORTED);
6248 6675                  }
6249      -                do {
6250      -                        old = itask->itask_flags;
6251      -                        if (old & ITASK_BEING_ABORTED) {
6252      -                                itask->itask_ncmds = 1;
6253      -                                curcmd = itask->itask_cmd_stack[0] =
6254      -                                    ITASK_CMD_ABORT;
6255      -                                goto out_itask_flag_loop;
6256      -                        } else if ((curcmd & ITASK_CMD_MASK) ==
6257      -                            ITASK_CMD_NEW_TASK) {
6258      -                                /*
6259      -                                 * set ITASK_KSTAT_IN_RUNQ, this flag
6260      -                                 * will not reset until task completed
6261      -                                 */
6262      -                                new = old | ITASK_KNOWN_TO_LU |
6263      -                                    ITASK_KSTAT_IN_RUNQ;
6264      -                        } else {
6265      -                                goto out_itask_flag_loop;
6266      -                        }
6267      -                } while (atomic_cas_32(&itask->itask_flags, old, new) != old);
     6676 +                if (itask->itask_flags & ITASK_BEING_ABORTED) {
     6677 +                        itask->itask_ncmds = 1;
     6678 +                        curcmd = itask->itask_cmd_stack[0] =
     6679 +                            ITASK_CMD_ABORT;
     6680 +                        goto out_itask_flag_loop;
     6681 +                } else if ((curcmd & ITASK_CMD_MASK) == ITASK_CMD_NEW_TASK) {
     6682 +                        /*
     6683 +                         * set ITASK_KSTAT_IN_RUNQ, this flag
     6684 +                         * will not reset until task completed
     6685 +                         */
     6686 +                        itask->itask_flags |= ITASK_KNOWN_TO_LU |
     6687 +                            ITASK_KSTAT_IN_RUNQ;
     6688 +                } else {
     6689 +                        goto out_itask_flag_loop;
     6690 +                }
6268 6691  
6269 6692  out_itask_flag_loop:
6270 6693  
6271 6694                  /*
6272 6695                   * Decide if this task needs to go to a queue and/or if
6273 6696                   * we can decrement the itask_cmd_stack.
6274 6697                   */
6275 6698                  if (curcmd == ITASK_CMD_ABORT) {
6276 6699                          if (itask->itask_flags & (ITASK_KNOWN_TO_LU |
6277 6700                              ITASK_KNOWN_TO_TGT_PORT)) {
↓ open down ↓ 38 lines elided ↑ open up ↑
6316 6739                          dec_qdepth = 1;
6317 6740                          itask->itask_waitq_time +=
6318 6741                              gethrtime() - itask->itask_waitq_enter_timestamp;
6319 6742                  }
6320 6743  
6321 6744                  /* We made it here means we are going to call LU */
6322 6745                  if ((itask->itask_flags & ITASK_DEFAULT_HANDLING) == 0)
6323 6746                          lu = task->task_lu;
6324 6747                  else
6325 6748                          lu = dlun0;
     6749 +
6326 6750                  dbuf = itask->itask_dbufs[ITASK_CMD_BUF_NDX(curcmd)];
6327 6751                  mutex_exit(&w->worker_lock);
6328 6752                  curcmd &= ITASK_CMD_MASK;
6329 6753                  stmf_task_audit(itask, TE_PROCESS_CMD, curcmd, dbuf);
     6754 +                mutex_exit(&itask->itask_mutex);
     6755 +
6330 6756                  switch (curcmd) {
6331 6757                  case ITASK_CMD_NEW_TASK:
6332 6758                          iss = (stmf_i_scsi_session_t *)
6333 6759                              task->task_session->ss_stmf_private;
6334 6760                          stmf_itl_lu_new_task(itask);
6335 6761                          if (iss->iss_flags & ISS_LUN_INVENTORY_CHANGED) {
6336      -                                if (stmf_handle_cmd_during_ic(itask))
     6762 +                                if (stmf_handle_cmd_during_ic(itask)) {
6337 6763                                          break;
     6764 +                                }
6338 6765                          }
6339 6766  #ifdef  DEBUG
6340 6767                          if (stmf_drop_task_counter > 0) {
6341      -                                if (atomic_dec_32_nv(&stmf_drop_task_counter)
6342      -                                    == 1)
     6768 +                                if (atomic_dec_32_nv(
     6769 +                                    (uint32_t *)&stmf_drop_task_counter) == 1) {
6343 6770                                          break;
     6771 +                                }
6344 6772                          }
6345 6773  #endif
6346 6774                          DTRACE_PROBE1(scsi__task__start, scsi_task_t *, task);
6347 6775                          lu->lu_new_task(task, dbuf);
6348 6776                          break;
6349 6777                  case ITASK_CMD_DATA_XFER_DONE:
6350 6778                          lu->lu_dbuf_xfer_done(task, dbuf);
6351 6779                          break;
6352 6780                  case ITASK_CMD_STATUS_DONE:
6353 6781                          lu->lu_send_status_done(task);
6354 6782                          break;
6355 6783                  case ITASK_CMD_ABORT:
6356 6784                          if (abort_free) {
     6785 +                                mutex_enter(&itask->itask_mutex);
6357 6786                                  stmf_task_free(task);
6358 6787                          } else {
6359 6788                                  stmf_do_task_abort(task);
6360 6789                          }
6361 6790                          break;
6362 6791                  case ITASK_CMD_POLL_LU:
6363 6792                          if (!wait_queue) {
6364 6793                                  lu->lu_task_poll(task);
6365 6794                          }
6366 6795                          break;
6367 6796                  case ITASK_CMD_POLL_LPORT:
6368 6797                          if (!wait_queue)
6369 6798                                  task->task_lport->lport_task_poll(task);
6370 6799                          break;
6371 6800                  case ITASK_CMD_SEND_STATUS:
6372 6801                  /* case ITASK_CMD_XFER_DATA: */
6373 6802                          break;
6374 6803                  }
     6804 +
6375 6805                  mutex_enter(&w->worker_lock);
6376 6806                  if (dec_qdepth) {
6377 6807                          w->worker_queue_depth--;
6378 6808                  }
6379 6809          }
6380 6810          if ((w->worker_flags & STMF_WORKER_TERMINATE) && (wait_timer == 0)) {
6381 6811                  if (w->worker_ref_count == 0)
6382 6812                          goto stmf_worker_loop;
6383 6813                  else {
6384 6814                          wait_timer = ddi_get_lbolt() + 1;
↓ open down ↓ 7 lines elided ↑ open up ↑
6392 6822                      wait_delta, TR_CLOCK_TICK);
6393 6823          } else {
6394 6824                  DTRACE_PROBE1(worker__sleep, stmf_worker_t, w);
6395 6825                  cv_wait(&w->worker_cv, &w->worker_lock);
6396 6826          }
6397 6827          DTRACE_PROBE1(worker__wakeup, stmf_worker_t, w);
6398 6828          w->worker_flags |= STMF_WORKER_ACTIVE;
6399 6829          goto stmf_worker_loop;
6400 6830  }
6401 6831  
6402      -void
6403      -stmf_worker_mgmt()
6404      -{
6405      -        int i;
6406      -        int workers_needed;
6407      -        uint32_t qd;
6408      -        clock_t tps, d = 0;
6409      -        uint32_t cur_max_ntasks = 0;
6410      -        stmf_worker_t *w;
6411      -
6412      -        /* Check if we are trying to increase the # of threads */
6413      -        for (i = stmf_nworkers_cur; i < stmf_nworkers_needed; i++) {
6414      -                if (stmf_workers[i].worker_flags & STMF_WORKER_STARTED) {
6415      -                        stmf_nworkers_cur++;
6416      -                        stmf_nworkers_accepting_cmds++;
6417      -                } else {
6418      -                        /* Wait for transition to complete */
6419      -                        return;
6420      -                }
6421      -        }
6422      -        /* Check if we are trying to decrease the # of workers */
6423      -        for (i = (stmf_nworkers_cur - 1); i >= stmf_nworkers_needed; i--) {
6424      -                if ((stmf_workers[i].worker_flags & STMF_WORKER_STARTED) == 0) {
6425      -                        stmf_nworkers_cur--;
6426      -                        /*
6427      -                         * stmf_nworkers_accepting_cmds has already been
6428      -                         * updated by the request to reduce the # of workers.
6429      -                         */
6430      -                } else {
6431      -                        /* Wait for transition to complete */
6432      -                        return;
6433      -                }
6434      -        }
6435      -        /* Check if we are being asked to quit */
6436      -        if (stmf_workers_state != STMF_WORKERS_ENABLED) {
6437      -                if (stmf_nworkers_cur) {
6438      -                        workers_needed = 0;
6439      -                        goto worker_mgmt_trigger_change;
6440      -                }
6441      -                return;
6442      -        }
6443      -        /* Check if we are starting */
6444      -        if (stmf_nworkers_cur < stmf_i_min_nworkers) {
6445      -                workers_needed = stmf_i_min_nworkers;
6446      -                goto worker_mgmt_trigger_change;
6447      -        }
6448      -
6449      -        tps = drv_usectohz(1 * 1000 * 1000);
6450      -        if ((stmf_wm_last != 0) &&
6451      -            ((d = ddi_get_lbolt() - stmf_wm_last) > tps)) {
6452      -                qd = 0;
6453      -                for (i = 0; i < stmf_nworkers_accepting_cmds; i++) {
6454      -                        qd += stmf_workers[i].worker_max_qdepth_pu;
6455      -                        stmf_workers[i].worker_max_qdepth_pu = 0;
6456      -                        if (stmf_workers[i].worker_max_sys_qdepth_pu >
6457      -                            cur_max_ntasks) {
6458      -                                cur_max_ntasks =
6459      -                                    stmf_workers[i].worker_max_sys_qdepth_pu;
6460      -                        }
6461      -                        stmf_workers[i].worker_max_sys_qdepth_pu = 0;
6462      -                }
6463      -        }
6464      -        stmf_wm_last = ddi_get_lbolt();
6465      -        if (d <= tps) {
6466      -                /* still ramping up */
6467      -                return;
6468      -        }
6469      -        /* max qdepth cannot be more than max tasks */
6470      -        if (qd > cur_max_ntasks)
6471      -                qd = cur_max_ntasks;
6472      -
6473      -        /* See if we have more workers */
6474      -        if (qd < stmf_nworkers_accepting_cmds) {
6475      -                /*
6476      -                 * Since we dont reduce the worker count right away, monitor
6477      -                 * the highest load during the scale_down_delay.
6478      -                 */
6479      -                if (qd > stmf_worker_scale_down_qd)
6480      -                        stmf_worker_scale_down_qd = qd;
6481      -                if (stmf_worker_scale_down_timer == 0) {
6482      -                        stmf_worker_scale_down_timer = ddi_get_lbolt() +
6483      -                            drv_usectohz(stmf_worker_scale_down_delay *
6484      -                            1000 * 1000);
6485      -                        return;
6486      -                }
6487      -                if (ddi_get_lbolt() < stmf_worker_scale_down_timer) {
6488      -                        return;
6489      -                }
6490      -                /* Its time to reduce the workers */
6491      -                if (stmf_worker_scale_down_qd < stmf_i_min_nworkers)
6492      -                        stmf_worker_scale_down_qd = stmf_i_min_nworkers;
6493      -                if (stmf_worker_scale_down_qd > stmf_i_max_nworkers)
6494      -                        stmf_worker_scale_down_qd = stmf_i_max_nworkers;
6495      -                if (stmf_worker_scale_down_qd == stmf_nworkers_cur)
6496      -                        return;
6497      -                workers_needed = stmf_worker_scale_down_qd;
6498      -                stmf_worker_scale_down_qd = 0;
6499      -                goto worker_mgmt_trigger_change;
6500      -        }
6501      -        stmf_worker_scale_down_qd = 0;
6502      -        stmf_worker_scale_down_timer = 0;
6503      -        if (qd > stmf_i_max_nworkers)
6504      -                qd = stmf_i_max_nworkers;
6505      -        if (qd < stmf_i_min_nworkers)
6506      -                qd = stmf_i_min_nworkers;
6507      -        if (qd == stmf_nworkers_cur)
6508      -                return;
6509      -        workers_needed = qd;
6510      -        goto worker_mgmt_trigger_change;
6511      -
6512      -        /* NOTREACHED */
6513      -        return;
6514      -
6515      -worker_mgmt_trigger_change:
6516      -        ASSERT(workers_needed != stmf_nworkers_cur);
6517      -        if (workers_needed > stmf_nworkers_cur) {
6518      -                stmf_nworkers_needed = workers_needed;
6519      -                for (i = stmf_nworkers_cur; i < workers_needed; i++) {
6520      -                        w = &stmf_workers[i];
6521      -                        w->worker_tid = thread_create(NULL, 0, stmf_worker_task,
6522      -                            (void *)&stmf_workers[i], 0, &p0, TS_RUN,
6523      -                            minclsyspri);
6524      -                }
6525      -                return;
6526      -        }
6527      -        /* At this point we know that we are decreasing the # of workers */
6528      -        stmf_nworkers_accepting_cmds = workers_needed;
6529      -        stmf_nworkers_needed = workers_needed;
6530      -        /* Signal the workers that its time to quit */
6531      -        for (i = (stmf_nworkers_cur - 1); i >= stmf_nworkers_needed; i--) {
6532      -                w = &stmf_workers[i];
6533      -                ASSERT(w && (w->worker_flags & STMF_WORKER_STARTED));
6534      -                mutex_enter(&w->worker_lock);
6535      -                w->worker_flags |= STMF_WORKER_TERMINATE;
6536      -                if ((w->worker_flags & STMF_WORKER_ACTIVE) == 0)
6537      -                        cv_signal(&w->worker_cv);
6538      -                mutex_exit(&w->worker_lock);
6539      -        }
6540      -}
6541      -
6542 6832  /*
6543 6833   * Fills out a dbuf from stmf_xfer_data_t (contained in the db_lu_private).
6544 6834   * If all the data has been filled out, frees the xd and makes
6545 6835   * db_lu_private NULL.
6546 6836   */
6547 6837  void
6548 6838  stmf_xd_to_dbuf(stmf_data_buf_t *dbuf, int set_rel_off)
6549 6839  {
6550 6840          stmf_xfer_data_t *xd;
6551 6841          uint8_t *p;
↓ open down ↓ 96 lines elided ↑ open up ↑
6648 6938                   */
6649 6939  
6650 6940                  bzero(p, inq_page_length + 5);
6651 6941  
6652 6942                  p[0] = DPQ_SUPPORTED | DTYPE_UNKNOWN;
6653 6943                  p[2] = 5;
6654 6944                  p[3] = 0x12;
6655 6945                  p[4] = inq_page_length;
6656 6946                  p[6] = 0x80;
6657 6947  
6658      -                (void) strncpy((char *)p+8, "SUN     ", 8);
6659      -                (void) strncpy((char *)p+16, "COMSTAR          ", 16);
6660      -                (void) strncpy((char *)p+32, "1.0 ", 4);
     6948 +                (void) strncpy((char *)p+8, "NONE    ", 8);
     6949 +                (void) strncpy((char *)p+16, "NONE            ", 16);
     6950 +                (void) strncpy((char *)p+32, "NONE", 4);
6661 6951  
6662 6952                  dbuf->db_data_size = sz;
6663 6953                  dbuf->db_relative_offset = 0;
6664 6954                  dbuf->db_flags = DB_DIRECTION_TO_RPORT;
6665 6955                  (void) stmf_xfer_data(task, dbuf, 0);
6666 6956  
6667 6957                  return;
6668 6958  
6669 6959          case SCMD_REPORT_LUNS:
6670 6960                  task->task_cmd_xfer_length =
↓ open down ↓ 189 lines elided ↑ open up ↑
6860 7150  }
6861 7151  
6862 7152  /* ARGSUSED */
6863 7153  void
6864 7154  stmf_dlun0_ctl(struct stmf_lu *lu, int cmd, void *arg)
6865 7155  {
6866 7156          /* This function will never be called */
6867 7157          cmn_err(CE_WARN, "stmf_dlun0_ctl called with cmd %x", cmd);
6868 7158  }
6869 7159  
     7160 +/* ARGSUSED */
6870 7161  void
     7162 +stmf_dlun0_task_done(struct scsi_task *task)
     7163 +{
     7164 +}
     7165 +
     7166 +void
6871 7167  stmf_dlun_init()
6872 7168  {
6873 7169          stmf_i_lu_t *ilu;
6874 7170  
6875 7171          dlun0 = stmf_alloc(STMF_STRUCT_STMF_LU, 0, 0);
6876 7172          dlun0->lu_task_alloc = stmf_dlun0_task_alloc;
6877 7173          dlun0->lu_new_task = stmf_dlun0_new_task;
6878 7174          dlun0->lu_dbuf_xfer_done = stmf_dlun0_dbuf_done;
6879 7175          dlun0->lu_send_status_done = stmf_dlun0_status_done;
6880 7176          dlun0->lu_task_free = stmf_dlun0_task_free;
6881 7177          dlun0->lu_abort = stmf_dlun0_abort;
6882 7178          dlun0->lu_task_poll = stmf_dlun0_task_poll;
     7179 +        dlun0->lu_task_done = stmf_dlun0_task_done;
6883 7180          dlun0->lu_ctl = stmf_dlun0_ctl;
6884 7181  
6885 7182          ilu = (stmf_i_lu_t *)dlun0->lu_stmf_private;
6886 7183          ilu->ilu_cur_task_cntr = &ilu->ilu_task_cntr1;
6887 7184  }
6888 7185  
6889 7186  stmf_status_t
6890 7187  stmf_dlun_fini()
6891 7188  {
6892 7189          stmf_i_lu_t *ilu;
↓ open down ↓ 223 lines elided ↑ open up ↑
7116 7413  stmf_generate_lu_event(stmf_i_lu_t *ilu, int eventid, void *arg, uint32_t flags)
7117 7414  {
7118 7415          if (STMF_EVENT_ENABLED(ilu->ilu_event_hdl, eventid) &&
7119 7416              (ilu->ilu_lu->lu_event_handler != NULL)) {
7120 7417                  ilu->ilu_lu->lu_event_handler(ilu->ilu_lu, eventid, arg, flags);
7121 7418          }
7122 7419  }
7123 7420  
7124 7421  void
7125 7422  stmf_generate_lport_event(stmf_i_local_port_t *ilport, int eventid, void *arg,
7126      -                                uint32_t flags)
     7423 +    uint32_t flags)
7127 7424  {
7128 7425          if (STMF_EVENT_ENABLED(ilport->ilport_event_hdl, eventid) &&
7129 7426              (ilport->ilport_lport->lport_event_handler != NULL)) {
7130 7427                  ilport->ilport_lport->lport_event_handler(
7131 7428                      ilport->ilport_lport, eventid, arg, flags);
7132 7429          }
7133 7430  }
7134 7431  
7135 7432  /*
7136 7433   * With the possibility of having multiple itl sessions pointing to the
7137 7434   * same itl_kstat_info, the ilu_kstat_lock mutex is used to synchronize
7138 7435   * the kstat update of the ilu_kstat_io, itl_kstat_taskq and itl_kstat_lu_xfer
7139 7436   * statistics.
7140 7437   */
7141 7438  void
7142 7439  stmf_itl_task_start(stmf_i_scsi_task_t *itask)
7143 7440  {
7144 7441          stmf_itl_data_t *itl = itask->itask_itl_datap;
7145 7442          scsi_task_t     *task = itask->itask_task;
7146 7443          stmf_i_lu_t     *ilu;
     7444 +        stmf_i_scsi_session_t   *iss =
     7445 +            itask->itask_task->task_session->ss_stmf_private;
     7446 +        stmf_i_remote_port_t    *irport = iss->iss_irport;
7147 7447  
7148 7448          if (itl == NULL || task->task_lu == dlun0)
7149 7449                  return;
7150 7450          ilu = (stmf_i_lu_t *)task->task_lu->lu_stmf_private;
7151 7451          itask->itask_start_timestamp = gethrtime();
     7452 +        itask->itask_xfer_done_timestamp = 0;
7152 7453          if (ilu->ilu_kstat_io != NULL) {
7153 7454                  mutex_enter(ilu->ilu_kstat_io->ks_lock);
7154 7455                  stmf_update_kstat_lu_q(itask->itask_task, kstat_waitq_enter);
7155 7456                  mutex_exit(ilu->ilu_kstat_io->ks_lock);
7156 7457          }
7157 7458  
     7459 +        if (irport->irport_kstat_estat != NULL) {
     7460 +                if (task->task_flags & TF_READ_DATA)
     7461 +                        atomic_inc_32(&irport->irport_nread_tasks);
     7462 +                else if (task->task_flags & TF_WRITE_DATA)
     7463 +                        atomic_inc_32(&irport->irport_nwrite_tasks);
     7464 +        }
     7465 +
7158 7466          stmf_update_kstat_lport_q(itask->itask_task, kstat_waitq_enter);
7159 7467  }
7160 7468  
7161 7469  void
7162 7470  stmf_itl_lu_new_task(stmf_i_scsi_task_t *itask)
7163 7471  {
7164 7472          stmf_itl_data_t *itl = itask->itask_itl_datap;
7165 7473          scsi_task_t     *task = itask->itask_task;
7166 7474          stmf_i_lu_t     *ilu;
7167 7475  
↓ open down ↓ 18 lines elided ↑ open up ↑
7186 7494  
7187 7495          itask->itask_done_timestamp = gethrtime();
7188 7496  
7189 7497          if (itl == NULL || task->task_lu == dlun0)
7190 7498                  return;
7191 7499          ilu = (stmf_i_lu_t *)task->task_lu->lu_stmf_private;
7192 7500  
7193 7501          if (ilu->ilu_kstat_io == NULL)
7194 7502                  return;
7195 7503  
     7504 +        stmf_update_kstat_rport_estat(task);
     7505 +
7196 7506          mutex_enter(ilu->ilu_kstat_io->ks_lock);
7197 7507  
7198 7508          if (itask->itask_flags & ITASK_KSTAT_IN_RUNQ) {
7199 7509                  stmf_update_kstat_lu_q(task, kstat_runq_exit);
7200 7510                  mutex_exit(ilu->ilu_kstat_io->ks_lock);
7201 7511                  stmf_update_kstat_lport_q(task, kstat_runq_exit);
7202 7512          } else {
7203 7513                  stmf_update_kstat_lu_q(task, kstat_waitq_exit);
7204 7514                  mutex_exit(ilu->ilu_kstat_io->ks_lock);
7205 7515                  stmf_update_kstat_lport_q(task, kstat_waitq_exit);
7206 7516          }
7207 7517  }
7208 7518  
     7519 +void
     7520 +stmf_lu_xfer_done(scsi_task_t *task, boolean_t read, hrtime_t elapsed_time)
     7521 +{
     7522 +        stmf_i_scsi_task_t *itask = task->task_stmf_private;
     7523 +
     7524 +        if (task->task_lu == dlun0)
     7525 +                return;
     7526 +
     7527 +        if (read) {
     7528 +                atomic_add_64((uint64_t *)&itask->itask_lu_read_time,
     7529 +                    elapsed_time);
     7530 +        } else {
     7531 +                atomic_add_64((uint64_t *)&itask->itask_lu_write_time,
     7532 +                    elapsed_time);
     7533 +        }
     7534 +}
     7535 +
7209 7536  static void
7210 7537  stmf_lport_xfer_start(stmf_i_scsi_task_t *itask, stmf_data_buf_t *dbuf)
7211 7538  {
7212 7539          stmf_itl_data_t         *itl = itask->itask_itl_datap;
7213 7540  
7214 7541          if (itl == NULL)
7215 7542                  return;
7216 7543  
7217 7544          DTRACE_PROBE2(scsi__xfer__start, scsi_task_t *, itask->itask_task,
7218 7545              stmf_data_buf_t *, dbuf);
↓ open down ↓ 7 lines elided ↑ open up ↑
7226 7553          stmf_itl_data_t         *itl = itask->itask_itl_datap;
7227 7554          hrtime_t                elapsed_time;
7228 7555          uint64_t                xfer_size;
7229 7556  
7230 7557          if (itl == NULL)
7231 7558                  return;
7232 7559  
7233 7560          xfer_size = (dbuf->db_xfer_status == STMF_SUCCESS) ?
7234 7561              dbuf->db_data_size : 0;
7235 7562  
7236      -        elapsed_time = gethrtime() - dbuf->db_xfer_start_timestamp;
     7563 +        itask->itask_xfer_done_timestamp = gethrtime();
     7564 +        elapsed_time = itask->itask_xfer_done_timestamp -
     7565 +            dbuf->db_xfer_start_timestamp;
7237 7566          if (dbuf->db_flags & DB_DIRECTION_TO_RPORT) {
7238 7567                  atomic_add_64((uint64_t *)&itask->itask_lport_read_time,
7239 7568                      elapsed_time);
7240 7569                  atomic_add_64((uint64_t *)&itask->itask_read_xfer,
7241 7570                      xfer_size);
7242 7571          } else {
7243 7572                  atomic_add_64((uint64_t *)&itask->itask_lport_write_time,
7244 7573                      elapsed_time);
7245 7574                  atomic_add_64((uint64_t *)&itask->itask_write_xfer,
7246 7575                      xfer_size);
↓ open down ↓ 3 lines elided ↑ open up ↑
7250 7579              stmf_data_buf_t *, dbuf, hrtime_t, elapsed_time);
7251 7580  
7252 7581          dbuf->db_xfer_start_timestamp = 0;
7253 7582  }
7254 7583  
7255 7584  void
7256 7585  stmf_svc_init()
7257 7586  {
7258 7587          if (stmf_state.stmf_svc_flags & STMF_SVC_STARTED)
7259 7588                  return;
7260      -        stmf_state.stmf_svc_tailp = &stmf_state.stmf_svc_active;
     7589 +        list_create(&stmf_state.stmf_svc_list, sizeof (stmf_svc_req_t),
     7590 +            offsetof(stmf_svc_req_t, svc_list_entry));
7261 7591          stmf_state.stmf_svc_taskq = ddi_taskq_create(0, "STMF_SVC_TASKQ", 1,
7262 7592              TASKQ_DEFAULTPRI, 0);
7263 7593          (void) ddi_taskq_dispatch(stmf_state.stmf_svc_taskq,
7264 7594              stmf_svc, 0, DDI_SLEEP);
7265 7595  }
7266 7596  
7267 7597  stmf_status_t
7268 7598  stmf_svc_fini()
7269 7599  {
7270 7600          uint32_t i;
↓ open down ↓ 8 lines elided ↑ open up ↑
7279 7609          /* Wait for 5 seconds */
7280 7610          for (i = 0; i < 500; i++) {
7281 7611                  if (stmf_state.stmf_svc_flags & STMF_SVC_STARTED)
7282 7612                          delay(drv_usectohz(10000));
7283 7613                  else
7284 7614                          break;
7285 7615          }
7286 7616          if (i == 500)
7287 7617                  return (STMF_BUSY);
7288 7618  
     7619 +        list_destroy(&stmf_state.stmf_svc_list);
7289 7620          ddi_taskq_destroy(stmf_state.stmf_svc_taskq);
7290 7621  
7291 7622          return (STMF_SUCCESS);
7292 7623  }
7293 7624  
7294 7625  struct stmf_svc_clocks {
7295 7626          clock_t drain_start, drain_next;
7296 7627          clock_t timing_start, timing_next;
7297 7628          clock_t worker_delay;
7298 7629  };
↓ open down ↓ 5 lines elided ↑ open up ↑
7304 7635          stmf_svc_req_t *req;
7305 7636          stmf_lu_t *lu;
7306 7637          stmf_i_lu_t *ilu;
7307 7638          stmf_local_port_t *lport;
7308 7639          struct stmf_svc_clocks clks = { 0 };
7309 7640  
7310 7641          mutex_enter(&stmf_state.stmf_lock);
7311 7642          stmf_state.stmf_svc_flags |= STMF_SVC_STARTED | STMF_SVC_ACTIVE;
7312 7643  
7313 7644          while (!(stmf_state.stmf_svc_flags & STMF_SVC_TERMINATE)) {
7314      -                if (stmf_state.stmf_svc_active == NULL) {
     7645 +                if (list_is_empty(&stmf_state.stmf_svc_list)) {
7315 7646                          stmf_svc_timeout(&clks);
7316 7647                          continue;
7317 7648                  }
7318 7649  
7319 7650                  /*
7320 7651                   * Pop the front request from the active list.  After this,
7321 7652                   * the request will no longer be referenced by global state,
7322 7653                   * so it should be safe to access it without holding the
7323 7654                   * stmf state lock.
7324 7655                   */
7325      -                req = stmf_state.stmf_svc_active;
7326      -                stmf_state.stmf_svc_active = req->svc_next;
     7656 +                req = list_remove_head(&stmf_state.stmf_svc_list);
     7657 +                if (req == NULL)
     7658 +                        continue;
7327 7659  
7328      -                if (stmf_state.stmf_svc_active == NULL)
7329      -                        stmf_state.stmf_svc_tailp = &stmf_state.stmf_svc_active;
7330      -
7331 7660                  switch (req->svc_cmd) {
7332 7661                  case STMF_CMD_LPORT_ONLINE:
7333 7662                          /* Fallthrough */
7334 7663                  case STMF_CMD_LPORT_OFFLINE:
7335 7664                          mutex_exit(&stmf_state.stmf_lock);
7336 7665                          lport = (stmf_local_port_t *)req->svc_obj;
7337 7666                          lport->lport_ctl(lport, req->svc_cmd, &req->svc_info);
7338 7667                          break;
7339 7668                  case STMF_CMD_LU_ONLINE:
7340 7669                          mutex_exit(&stmf_state.stmf_lock);
↓ open down ↓ 50 lines elided ↑ open up ↑
7391 7720                  if (!stmf_state.stmf_svc_ilu_timing) {
7392 7721                          /* we finished a complete round */
7393 7722                          clks->timing_next =
7394 7723                              clks->timing_start + drv_usectohz(5*1000*1000);
7395 7724                  } else {
7396 7725                          /* we still have some ilu items to check */
7397 7726                          clks->timing_next =
7398 7727                              ddi_get_lbolt() + drv_usectohz(1*1000*1000);
7399 7728                  }
7400 7729  
7401      -                if (stmf_state.stmf_svc_active)
     7730 +                if (!list_is_empty(&stmf_state.stmf_svc_list))
7402 7731                          return;
7403 7732          }
7404 7733  
7405 7734          /* Check if there are free tasks to clear */
7406 7735          if (stmf_state.stmf_nlus &&
7407 7736              ((!clks->drain_next) || (ddi_get_lbolt() >= clks->drain_next))) {
7408 7737                  if (!stmf_state.stmf_svc_ilu_draining) {
7409 7738                          /* we are starting a new round */
7410 7739                          stmf_state.stmf_svc_ilu_draining =
7411 7740                              stmf_state.stmf_ilulist;
↓ open down ↓ 4 lines elided ↑ open up ↑
7416 7745                  if (!stmf_state.stmf_svc_ilu_draining) {
7417 7746                          /* we finished a complete round */
7418 7747                          clks->drain_next =
7419 7748                              clks->drain_start + drv_usectohz(10*1000*1000);
7420 7749                  } else {
7421 7750                          /* we still have some ilu items to check */
7422 7751                          clks->drain_next =
7423 7752                              ddi_get_lbolt() + drv_usectohz(1*1000*1000);
7424 7753                  }
7425 7754  
7426      -                if (stmf_state.stmf_svc_active)
     7755 +                if (!list_is_empty(&stmf_state.stmf_svc_list))
7427 7756                          return;
7428 7757          }
7429 7758  
7430      -        /* Check if we need to run worker_mgmt */
7431      -        if (ddi_get_lbolt() > clks->worker_delay) {
7432      -                stmf_worker_mgmt();
7433      -                clks->worker_delay = ddi_get_lbolt() +
7434      -                    stmf_worker_mgmt_delay;
7435      -        }
7436      -
7437 7759          /* Check if any active session got its 1st LUN */
7438 7760          if (stmf_state.stmf_process_initial_luns) {
7439 7761                  int stmf_level = 0;
7440 7762                  int port_level;
7441 7763  
7442 7764                  for (ilport = stmf_state.stmf_ilportlist; ilport;
7443 7765                      ilport = next_ilport) {
7444 7766                          int ilport_lock_held;
7445 7767                          next_ilport = ilport->ilport_next;
7446 7768  
↓ open down ↓ 106 lines elided ↑ open up ↑
7553 7875          req->svc_cmd = cmd;
7554 7876          req->svc_obj = obj;
7555 7877          req->svc_info.st_rflags = info->st_rflags;
7556 7878          if (info->st_additional_info) {
7557 7879                  req->svc_info.st_additional_info = (char *)(GET_BYTE_OFFSET(req,
7558 7880                      sizeof (stmf_svc_req_t)));
7559 7881                  (void) strcpy(req->svc_info.st_additional_info,
7560 7882                      info->st_additional_info);
7561 7883          }
7562 7884          req->svc_req_alloc_size = s;
7563      -        req->svc_next = NULL;
7564 7885  
7565 7886          mutex_enter(&stmf_state.stmf_lock);
7566      -        *stmf_state.stmf_svc_tailp = req;
7567      -        stmf_state.stmf_svc_tailp = &req->svc_next;
     7887 +        list_insert_tail(&stmf_state.stmf_svc_list, req);
7568 7888          if ((stmf_state.stmf_svc_flags & STMF_SVC_ACTIVE) == 0) {
7569 7889                  cv_signal(&stmf_state.stmf_cv);
7570 7890          }
7571 7891          mutex_exit(&stmf_state.stmf_lock);
7572 7892  }
7573 7893  
7574 7894  static void
7575 7895  stmf_svc_kill_obj_requests(void *obj)
7576 7896  {
7577      -        stmf_svc_req_t *prev_req = NULL;
7578      -        stmf_svc_req_t *next_req;
7579 7897          stmf_svc_req_t *req;
7580 7898  
7581 7899          ASSERT(mutex_owned(&stmf_state.stmf_lock));
7582 7900  
7583      -        for (req = stmf_state.stmf_svc_active; req != NULL; req = next_req) {
7584      -                next_req = req->svc_next;
7585      -
     7901 +        for (req = list_head(&stmf_state.stmf_svc_list); req != NULL;
     7902 +            req = list_next(&stmf_state.stmf_svc_list, req)) {
7586 7903                  if (req->svc_obj == obj) {
7587      -                        if (prev_req != NULL)
7588      -                                prev_req->svc_next = next_req;
7589      -                        else
7590      -                                stmf_state.stmf_svc_active = next_req;
7591      -
7592      -                        if (next_req == NULL)
7593      -                                stmf_state.stmf_svc_tailp = (prev_req != NULL) ?
7594      -                                    &prev_req->svc_next :
7595      -                                    &stmf_state.stmf_svc_active;
7596      -
     7904 +                        list_remove(&stmf_state.stmf_svc_list, req);
7597 7905                          kmem_free(req, req->svc_req_alloc_size);
7598      -                } else {
7599      -                        prev_req = req;
7600 7906                  }
7601 7907          }
7602 7908  }
7603 7909  
7604 7910  void
7605 7911  stmf_trace(caddr_t ident, const char *fmt, ...)
7606 7912  {
7607 7913          va_list args;
7608 7914          char tbuf[160];
7609 7915          int len;
↓ open down ↓ 25 lines elided ↑ open up ↑
7635 7941  {
7636 7942          if (!stmf_trace_on)
7637 7943                  return;
7638 7944          mutex_enter(&trace_buf_lock);
7639 7945          trace_buf_curndx = 0;
7640 7946          if (trace_buf_size > 0)
7641 7947                  stmf_trace_buf[0] = 0;
7642 7948          mutex_exit(&trace_buf_lock);
7643 7949  }
7644 7950  
     7951 +/*
     7952 + * NOTE: Due to lock order problems that are not possible to fix this
     7953 + * method drops and reacquires the itask_mutex around the call to stmf_ctl.
     7954 + * Another possible work around would be to use a dispatch queue and have
     7955 + * the call to stmf_ctl run on another thread that's not holding the
     7956 + * itask_mutex. The problem with that approach is that it's difficult to
     7957 + * determine what impact an asynchronous change would have on the system state.
     7958 + */
7645 7959  static void
7646 7960  stmf_abort_task_offline(scsi_task_t *task, int offline_lu, char *info)
7647 7961  {
7648 7962          stmf_state_change_info_t        change_info;
7649 7963          void                            *ctl_private;
7650 7964          uint32_t                        ctl_cmd;
7651 7965          int                             msg = 0;
     7966 +        stmf_i_scsi_task_t              *itask =
     7967 +            (stmf_i_scsi_task_t *)task->task_stmf_private;
7652 7968  
7653 7969          stmf_trace("FROM STMF", "abort_task_offline called for %s: %s",
7654 7970              offline_lu ? "LU" : "LPORT", info ? info : "no additional info");
7655 7971          change_info.st_additional_info = info;
     7972 +        ASSERT(mutex_owned(&itask->itask_mutex));
     7973 +
7656 7974          if (offline_lu) {
7657 7975                  change_info.st_rflags = STMF_RFLAG_RESET |
7658 7976                      STMF_RFLAG_LU_ABORT;
7659 7977                  ctl_private = task->task_lu;
7660 7978                  if (((stmf_i_lu_t *)
7661 7979                      task->task_lu->lu_stmf_private)->ilu_state ==
7662 7980                      STMF_STATE_ONLINE) {
7663 7981                          msg = 1;
7664 7982                  }
7665 7983                  ctl_cmd = STMF_CMD_LU_OFFLINE;
↓ open down ↓ 7 lines elided ↑ open up ↑
7673 7991                          msg = 1;
7674 7992                  }
7675 7993                  ctl_cmd = STMF_CMD_LPORT_OFFLINE;
7676 7994          }
7677 7995  
7678 7996          if (msg) {
7679 7997                  stmf_trace(0, "Calling stmf_ctl to offline %s : %s",
7680 7998                      offline_lu ? "LU" : "LPORT", info ? info :
7681 7999                      "<no additional info>");
7682 8000          }
     8001 +        mutex_exit(&itask->itask_mutex);
7683 8002          (void) stmf_ctl(ctl_cmd, ctl_private, &change_info);
     8003 +        mutex_enter(&itask->itask_mutex);
7684 8004  }
7685 8005  
7686 8006  static char
7687 8007  stmf_ctoi(char c)
7688 8008  {
7689 8009          if ((c >= '0') && (c <= '9'))
7690 8010                  c -= '0';
7691 8011          else if ((c >= 'A') && (c <= 'F'))
7692 8012                  c = c - 'A' + 10;
7693 8013          else if ((c >= 'a') && (c <= 'f'))
↓ open down ↓ 19 lines elided ↑ open up ↑
7713 8033                  if (nibble1 == -1 || nibble2 == -1)
7714 8034                          return (B_FALSE);
7715 8035  
7716 8036                  dp[ii] = (nibble1 << 4) | nibble2;
7717 8037          }
7718 8038          return (B_TRUE);
7719 8039  }
7720 8040  
7721 8041  boolean_t
7722 8042  stmf_scsilib_tptid_validate(scsi_transport_id_t *tptid, uint32_t total_sz,
7723      -                                uint16_t *tptid_sz)
     8043 +    uint16_t *tptid_sz)
7724 8044  {
7725 8045          uint16_t tpd_len = SCSI_TPTID_SIZE;
7726 8046  
7727 8047          if (tptid_sz)
7728 8048                  *tptid_sz = 0;
7729 8049          if (total_sz < sizeof (scsi_transport_id_t))
7730 8050                  return (B_FALSE);
7731 8051  
7732 8052          switch (tptid->protocol_id) {
7733 8053  
7734 8054          case PROTOCOL_FIBRE_CHANNEL:
7735 8055                  /* FC Transport ID validation checks. SPC3 rev23, Table 284 */
7736 8056                  if (total_sz < tpd_len || tptid->format_code != 0)
7737 8057                          return (B_FALSE);
7738 8058                  break;
7739 8059  
7740      -        case PROTOCOL_iSCSI:
     8060 +        case PROTOCOL_iSCSI: /* CSTYLED */
7741 8061                  {
7742 8062                  iscsi_transport_id_t    *iscsiid;
7743 8063                  uint16_t                adn_len, name_len;
7744 8064  
7745 8065                  /* Check for valid format code, SPC3 rev 23 Table 288 */
7746 8066                  if ((total_sz < tpd_len) ||
7747 8067                      (tptid->format_code != 0 && tptid->format_code != 1))
7748 8068                          return (B_FALSE);
7749 8069  
7750 8070                  iscsiid = (iscsi_transport_id_t *)tptid;
↓ open down ↓ 24 lines elided ↑ open up ↑
7775 8095                  if (total_sz < tpd_len || tptid->format_code != 0)
7776 8096                          return (B_FALSE);
7777 8097                  break;
7778 8098  
7779 8099          case PROTOCOL_PARALLEL_SCSI:
7780 8100          case PROTOCOL_SSA:
7781 8101          case PROTOCOL_IEEE_1394:
7782 8102          case PROTOCOL_SAS:
7783 8103          case PROTOCOL_ADT:
7784 8104          case PROTOCOL_ATAPI:
7785      -        default:
     8105 +        default: /* CSTYLED */
7786 8106                  {
7787 8107                  stmf_dflt_scsi_tptid_t *dflttpd;
7788 8108  
7789 8109                  tpd_len = sizeof (stmf_dflt_scsi_tptid_t);
7790 8110                  if (total_sz < tpd_len)
7791 8111                          return (B_FALSE);
7792 8112                  dflttpd = (stmf_dflt_scsi_tptid_t *)tptid;
7793 8113                  tpd_len = tpd_len + SCSI_READ16(&dflttpd->ident_len) - 1;
7794 8114                  if (total_sz < tpd_len)
7795 8115                          return (B_FALSE);
7796 8116                  }
7797 8117                  break;
7798 8118          }
7799 8119          if (tptid_sz)
7800 8120                  *tptid_sz = tpd_len;
7801 8121          return (B_TRUE);
7802 8122  }
7803 8123  
7804 8124  boolean_t
7805 8125  stmf_scsilib_tptid_compare(scsi_transport_id_t *tpd1,
7806      -                                scsi_transport_id_t *tpd2)
     8126 +    scsi_transport_id_t *tpd2)
7807 8127  {
7808 8128          if ((tpd1->protocol_id != tpd2->protocol_id) ||
7809 8129              (tpd1->format_code != tpd2->format_code))
7810 8130                  return (B_FALSE);
7811 8131  
7812 8132          switch (tpd1->protocol_id) {
7813 8133  
7814      -        case PROTOCOL_iSCSI:
     8134 +        case PROTOCOL_iSCSI: /* CSTYLED */
7815 8135                  {
7816 8136                  iscsi_transport_id_t *iscsitpd1, *iscsitpd2;
7817 8137                  uint16_t len;
7818 8138  
7819 8139                  iscsitpd1 = (iscsi_transport_id_t *)tpd1;
7820 8140                  iscsitpd2 = (iscsi_transport_id_t *)tpd2;
7821 8141                  len = SCSI_READ16(&iscsitpd1->add_len);
7822 8142                  if ((memcmp(iscsitpd1->add_len, iscsitpd2->add_len, 2) != 0) ||
7823 8143                      (memcmp(iscsitpd1->iscsi_name, iscsitpd2->iscsi_name, len)
7824 8144                      != 0))
7825 8145                          return (B_FALSE);
7826 8146                  }
7827 8147                  break;
7828 8148  
7829      -        case PROTOCOL_SRP:
     8149 +        case PROTOCOL_SRP: /* CSTYLED */
7830 8150                  {
7831 8151                  scsi_srp_transport_id_t *srptpd1, *srptpd2;
7832 8152  
7833 8153                  srptpd1 = (scsi_srp_transport_id_t *)tpd1;
7834 8154                  srptpd2 = (scsi_srp_transport_id_t *)tpd2;
7835 8155                  if (memcmp(srptpd1->srp_name, srptpd2->srp_name,
7836 8156                      sizeof (srptpd1->srp_name)) != 0)
7837 8157                          return (B_FALSE);
7838 8158                  }
7839 8159                  break;
7840 8160  
7841      -        case PROTOCOL_FIBRE_CHANNEL:
     8161 +        case PROTOCOL_FIBRE_CHANNEL: /* CSTYLED */
7842 8162                  {
7843 8163                  scsi_fc_transport_id_t *fctpd1, *fctpd2;
7844 8164  
7845 8165                  fctpd1 = (scsi_fc_transport_id_t *)tpd1;
7846 8166                  fctpd2 = (scsi_fc_transport_id_t *)tpd2;
7847 8167                  if (memcmp(fctpd1->port_name, fctpd2->port_name,
7848 8168                      sizeof (fctpd1->port_name)) != 0)
7849 8169                          return (B_FALSE);
7850 8170                  }
7851 8171                  break;
7852 8172  
7853 8173          case PROTOCOL_PARALLEL_SCSI:
7854 8174          case PROTOCOL_SSA:
7855 8175          case PROTOCOL_IEEE_1394:
7856 8176          case PROTOCOL_SAS:
7857 8177          case PROTOCOL_ADT:
7858 8178          case PROTOCOL_ATAPI:
7859      -        default:
     8179 +        default: /* CSTYLED */
7860 8180                  {
7861 8181                  stmf_dflt_scsi_tptid_t *dflt1, *dflt2;
7862 8182                  uint16_t len;
7863 8183  
7864 8184                  dflt1 = (stmf_dflt_scsi_tptid_t *)tpd1;
7865 8185                  dflt2 = (stmf_dflt_scsi_tptid_t *)tpd2;
7866 8186                  len = SCSI_READ16(&dflt1->ident_len);
7867 8187                  if ((memcmp(dflt1->ident_len, dflt2->ident_len, 2) != 0) ||
7868 8188                      (memcmp(dflt1->ident, dflt2->ident, len) != 0))
7869 8189                          return (B_FALSE);
↓ open down ↓ 90 lines elided ↑ open up ↑
7960 8280          }
7961 8281          return (rpt);
7962 8282  
7963 8283  devid_to_remote_port_fail:
7964 8284          stmf_remote_port_free(rpt);
7965 8285          return (NULL);
7966 8286  
7967 8287  }
7968 8288  
7969 8289  stmf_remote_port_t *
7970      -stmf_remote_port_alloc(uint16_t tptid_sz) {
     8290 +stmf_remote_port_alloc(uint16_t tptid_sz)
     8291 +{
7971 8292          stmf_remote_port_t *rpt;
7972 8293          rpt = (stmf_remote_port_t *)kmem_zalloc(
7973 8294              sizeof (stmf_remote_port_t) + tptid_sz, KM_SLEEP);
7974 8295          rpt->rport_tptid_sz = tptid_sz;
7975 8296          rpt->rport_tptid = (scsi_transport_id_t *)(rpt + 1);
7976 8297          return (rpt);
7977 8298  }
7978 8299  
7979 8300  void
7980 8301  stmf_remote_port_free(stmf_remote_port_t *rpt)
7981 8302  {
7982 8303          /*
7983 8304           * Note: stmf_scsilib_devid_to_remote_port() function allocates
7984 8305           *      remote port structures for all transports in the same way, So
7985 8306           *      it is safe to deallocate it in a protocol independent manner.
7986 8307           *      If any of the allocation method changes, corresponding changes
7987 8308           *      need to be made here too.
7988 8309           */
7989 8310          kmem_free(rpt, sizeof (stmf_remote_port_t) + rpt->rport_tptid_sz);
     8311 +}
     8312 +
     8313 +stmf_lu_t *
     8314 +stmf_check_and_hold_lu(scsi_task_t *task, uint8_t *guid)
     8315 +{
     8316 +        stmf_i_scsi_session_t *iss;
     8317 +        stmf_lu_t *lu;
     8318 +        stmf_i_lu_t *ilu = NULL;
     8319 +        stmf_lun_map_t *sm;
     8320 +        stmf_lun_map_ent_t *lme;
     8321 +        int i;
     8322 +
     8323 +        iss = (stmf_i_scsi_session_t *)task->task_session->ss_stmf_private;
     8324 +        rw_enter(iss->iss_lockp, RW_READER);
     8325 +        sm = iss->iss_sm;
     8326 +
     8327 +        for (i = 0; i < sm->lm_nentries; i++) {
     8328 +                if (sm->lm_plus[i] == NULL)
     8329 +                        continue;
     8330 +                lme = (stmf_lun_map_ent_t *)sm->lm_plus[i];
     8331 +                lu = lme->ent_lu;
     8332 +                if (bcmp(lu->lu_id->ident, guid, 16) == 0) {
     8333 +                        break;
     8334 +                }
     8335 +                lu = NULL;
     8336 +        }
     8337 +
     8338 +        if (!lu) {
     8339 +                goto hold_lu_done;
     8340 +        }
     8341 +
     8342 +        ilu = lu->lu_stmf_private;
     8343 +        mutex_enter(&ilu->ilu_task_lock);
     8344 +        ilu->ilu_additional_ref++;
     8345 +        mutex_exit(&ilu->ilu_task_lock);
     8346 +
     8347 +hold_lu_done:
     8348 +        rw_exit(iss->iss_lockp);
     8349 +        return (lu);
     8350 +}
     8351 +
     8352 +void
     8353 +stmf_release_lu(stmf_lu_t *lu)
     8354 +{
     8355 +        stmf_i_lu_t *ilu;
     8356 +
     8357 +        ilu = lu->lu_stmf_private;
     8358 +        ASSERT(ilu->ilu_additional_ref != 0);
     8359 +        mutex_enter(&ilu->ilu_task_lock);
     8360 +        ilu->ilu_additional_ref--;
     8361 +        mutex_exit(&ilu->ilu_task_lock);
     8362 +}
     8363 +
     8364 +int
     8365 +stmf_is_task_being_aborted(scsi_task_t *task)
     8366 +{
     8367 +        stmf_i_scsi_task_t *itask;
     8368 +
     8369 +        itask = (stmf_i_scsi_task_t *)task->task_stmf_private;
     8370 +        if (itask->itask_flags & ITASK_BEING_ABORTED)
     8371 +                return (1);
     8372 +
     8373 +        return (0);
     8374 +}
     8375 +
     8376 +volatile boolean_t stmf_pgr_aptpl_always = B_FALSE;
     8377 +
     8378 +boolean_t
     8379 +stmf_is_pgr_aptpl_always()
     8380 +{
     8381 +        return (stmf_pgr_aptpl_always);
7990 8382  }
    
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX