Print this page
NEX-19691 Unsuccessful mpt_sas IOC reset leads to the panic in no I/O to the pool - days later
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
NEX-20282 Add disk target queue depth tunable to mpt_sas
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
NEX-19821 All SAS paths down sometimes does not cause panic and trigger automatic HA failover
Reviewed by: Roman Strashkin <roman.strashkin@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
9048 mpt_sas should not require targets to send SEP messages
Reviewed by: Dan McDonald <danmcd@joyent.com>
Reviewed by: Hans Rosenfeld <hans.rosenfeld@joyent.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Approved by: Gordon Ross <gwr@nexenta.com>
NEX-17446 cleanup of hot unplugged disks fails intermittently
Reviewed by: Dan Fields <dan.fields@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
NEX-17944 HBA drivers don't need the redundant devfs_clean step
Reviewed by: Dan Fields <dan.fields@nexenta.com>
Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
NEX-17006 backport mpt_sas tri-mode parts support change
9044 Need support for mpt_sas tri-mode parts
9045 Clean up mpt_sas compiler warnings
9046 mptsas_handle_topo_change can return without its locks held
9047 workaround SAS3408 firmware issue
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Reviewed by: Hans Rosenfeld <hans.rosenfeld@joyent.com>
Reviewed by: Albert Lee <trisk@forkgnu.org>
Reviewed by: Yuri Pankov <yuripv@yuripv.net>
Approved by: Richard Lowe <richlowe@richlowe.net>
NEX-16174 scsi error messages should go to system log only
Reviewed by: Dan Fields <dan.fields@nexenta.com>
Reviewed by: Roman Strashkin <roman.strashkin@nexenta.com>
NEX-2100 vmem_hash_delete(ffffff5b5dee0000, 0, 1): bad free
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
Reviewed by: Marcel Telka <marcel@telka.sk>
NEX-6064 Son of single bad device causes outage a.k.a one disk fault
Reviewed by: Hans Rosenfeld <hans.rosenfeld@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
NEX-4418 SATA inquiry property generation doesn't work as advertised
Reviewed by: Dan McDonald <danmcd@omniti.com>
Reviewed by: Garrett D'Amore <garrett@damore.org>
Reviewed by: Josef 'Jeff' Sipek <josef.sipek@nexenta.com>
NEX-3988 Single bad device causes outage a.k.a one disk fault
Reviewed by: Hans Rosenfeld <hans.rosenfeld@nexenta.com>
Reviewed by: Kevin Crowe <kevin.crowe@nexenta.com>
NEX-3717 mptsas doesn't handle timeouts in mptsas_get_sata_guid()
Reviewed by: Gordon Ross <gordon.ross@nexenta.com>
Reviewed by: Josef 'Jeff' Sipek <josef.sipek@nexenta.com>
Reviewed by: Dan Fields <dan.fields@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
NEX-2103 12G mpt_sas needs additional minor enhancements
Revert OS-73 do not do IO complettions in the ISR
NEX-1889 mpt_sas should support 12G HBAs
4500 mptsas_hash_traverse() is unsafe, leads to missing devices
Reviewed by: Hans Rosenfeld <hans.rosenfeld@nexenta.com>
Approved by: Albert Lee <trisk@nexenta.com>
backout 4500 mptsas_hash_traverse() is unsafe, leads to missing devices
4403 mpt_sas panic when pulling a drive
Reviewed by: Hans Rosenfeld <hans.rosenfeld@nexenta.com>
Reviewed by: Albert Lee <trisk@nexenta.com>
Reviewed by: Andy Giles <illumos@ang.homedns.org>
Approved by: Robert Mustacchi <rm@joyent.com>
4500 mptsas_hash_traverse() is unsafe, leads to missing devices
Reviewed by: Hans Rosenfeld <hans.rosenfeld@nexenta.com>
Approved by: Albert Lee <trisk@nexenta.com>
NEX-1052 mptsas_do_passthru() does triggers assertion
OS-126 Creating a LUN for retired device results in sysevent loop
OS-91 mptsas does inquiry without setting pkt_time
OS-73 do not do IO complettions in the ISR
OS-61 Need ability for fault injection in mptsas
OS-87 pkt_reason not set accordingly when mpt_sas times out commands
OS-84 slow-io changes cause assertion failure
OS-62 slow io error detector is needed.
OS-59 remove automated target removal mechanism from mpt_sas.
Fix up some merges where we wanted the upstream version.
re #12927 rb4203 LSI 2008 mpt_sas
re #9517 rb4120 After single disk fault patch installed single disk fault still causes process hangs (fix gcc build)
re #9517 rb4120 After single disk fault patch installed single disk fault still causes process hangs
re #8346 rb2639 KT disk failures (fix lint/cstyle)
re #10443 rb3479 3.1.3 crash: BAD TRAP: type=e (#pf Page fault)
re #8346 rb2639 KT disk failures
re #7364 rb2201 "hddisco" hangs after unplugging both cables from JBOD (and NMS too)
re #8346 rb2639 KT disk failures
re #9636 rb2836 - mpt_sas should attempt an MUR reset at attach time.
--HG--
branch : stable-4.0
re #9636 rb2836 - mpt_sas should attempt an MUR reset at attach time.
re #7550 rb2134 lint-clean nza-kernel
re #6530 mpt_sas crash when more than 1 Initiator involved - ie HA
re #6834 rb1773 less verbosity in nfs4_rnode
re #6833 rb1771 less verbosity in mptsas
        
@@ -19,11 +19,11 @@
  * CDDL HEADER END
  */
 
 /*
  * Copyright (c) 2009, 2010, Oracle and/or its affiliates. All rights reserved.
- * Copyright 2016 Nexenta Systems, Inc. All rights reserved.
+ * Copyright 2019 Nexenta Systems, Inc.
  * Copyright (c) 2017, Joyent, Inc.
  * Copyright 2014 OmniTI Computer Consulting, Inc. All rights reserved.
  * Copyright (c) 2014, Tegile Systems Inc. All rights reserved.
  */
 
@@ -77,10 +77,12 @@
 #include <sys/sysevent/dr.h>
 #include <sys/sata/sata_defs.h>
 #include <sys/sata/sata_hba.h>
 #include <sys/scsi/generic/sas.h>
 #include <sys/scsi/impl/scsi_sas.h>
+#include <sys/sdt.h>
+#include <sys/mdi_impldefs.h>
 
 #pragma pack(1)
 #include <sys/scsi/adapters/mpt_sas/mpi/mpi2_type.h>
 #include <sys/scsi/adapters/mpt_sas/mpi/mpi2.h>
 #include <sys/scsi/adapters/mpt_sas/mpi/mpi2_cnfg.h>
@@ -100,12 +102,10 @@
 #include <sys/scsi/adapters/mpt_sas/mptsas_ioctl.h>
 #include <sys/scsi/adapters/mpt_sas/mptsas_smhba.h>
 #include <sys/scsi/adapters/mpt_sas/mptsas_hash.h>
 #include <sys/raidioctl.h>
 
-#include <sys/fs/dv_node.h>     /* devfs_clean */
-
 /*
  * FMA header files
  */
 #include <sys/ddifm.h>
 #include <sys/fm/protocol.h>
@@ -355,17 +355,18 @@
 
 static int mptsas_get_target_device_info(mptsas_t *mpt, uint32_t page_address,
     uint16_t *handle, mptsas_target_t **pptgt);
 static void mptsas_update_phymask(mptsas_t *mpt);
 
-static int mptsas_send_sep(mptsas_t *mpt, mptsas_target_t *ptgt,
+static int mptsas_flush_led_status(mptsas_t *mpt, mptsas_enclosure_t *mep,
+    uint16_t idx);
+static int mptsas_send_sep(mptsas_t *mpt, mptsas_enclosure_t *mep, uint16_t idx,
     uint32_t *status, uint8_t cmd);
 static dev_info_t *mptsas_get_dip_from_dev(dev_t dev,
     mptsas_phymask_t *phymask);
 static mptsas_target_t *mptsas_addr_to_ptgt(mptsas_t *mpt, char *addr,
     mptsas_phymask_t phymask);
-static int mptsas_flush_led_status(mptsas_t *mpt, mptsas_target_t *ptgt);
 
 
 /*
  * Enumeration / DR functions
  */
@@ -394,17 +395,15 @@
     char *guid, dev_info_t **dip, mdi_pathinfo_t **pip, mptsas_target_t *ptgt,
     int lun);
 
 static void mptsas_offline_missed_luns(dev_info_t *pdip,
     uint16_t *repluns, int lun_cnt, mptsas_target_t *ptgt);
-static int mptsas_offline_lun(dev_info_t *pdip, dev_info_t *rdip,
-    mdi_pathinfo_t *rpip, uint_t flags);
+static int mptsas_offline_lun(dev_info_t *rdip, mdi_pathinfo_t *rpip);
 
 static int mptsas_config_smp(dev_info_t *pdip, uint64_t sas_wwn,
     dev_info_t **smp_dip);
-static int mptsas_offline_smp(dev_info_t *pdip, mptsas_smp_t *smp_node,
-    uint_t flags);
+static int mptsas_offline_smp(dev_info_t *pdip, mptsas_smp_t *smp_node);
 
 static int mptsas_event_query(mptsas_t *mpt, mptsas_event_query_t *data,
     int mode, int *rval);
 static int mptsas_event_enable(mptsas_t *mpt, mptsas_event_enable_t *data,
     int mode, int *rval);
@@ -467,10 +466,17 @@
  * By default the value is 30 seconds.
  */
 int mptsas_inq83_retry_timeout = 30;
 
 /*
+ * Tunable for default SCSI pkt timeout. Defaults to 5 seconds, which should
+ * be plenty for INQUIRY and REPORT_LUNS, which are the only commands currently
+ * issued by mptsas directly.
+ */
+int mptsas_scsi_pkt_time = 5;
+
+/*
  * This is used to allocate memory for message frame storage, not for
  * data I/O DMA. All message frames must be stored in the first 4G of
  * physical memory.
  */
 ddi_dma_attr_t mptsas_dma_attrs = {
@@ -730,22 +736,10 @@
 
         return ((int)tp->m_devhdl - (int)dh);
 }
 
 static int
-mptsas_target_eval_slot(const void *op, void *arg)
-{
-        mptsas_led_control_t *lcp = arg;
-        const mptsas_target_t *tp = op;
-
-        if (tp->m_enclosure != lcp->Enclosure)
-                return ((int)tp->m_enclosure - (int)lcp->Enclosure);
-
-        return ((int)tp->m_slot_num - (int)lcp->Slot);
-}
-
-static int
 mptsas_target_eval_nowwn(const void *op, void *arg)
 {
         uint8_t phy = *(uint8_t *)arg;
         const mptsas_target_t *tp = op;
 
@@ -1202,20 +1196,18 @@
 
         /*
          * Allocate softc information.
          */
         if (ddi_soft_state_zalloc(mptsas_state, instance) != DDI_SUCCESS) {
-                mptsas_log(NULL, CE_WARN,
-                    "mptsas%d: cannot allocate soft state", instance);
+                mptsas_log(NULL, CE_WARN, "cannot allocate soft state");
                 goto fail;
         }
 
         mpt = ddi_get_soft_state(mptsas_state, instance);
 
         if (mpt == NULL) {
-                mptsas_log(NULL, CE_WARN,
-                    "mptsas%d: cannot get soft state", instance);
+                mptsas_log(NULL, CE_WARN, "cannot get soft state");
                 goto fail;
         }
 
         /* Indicate that we are 'sizeof (scsi_*(9S))' clean. */
         scsi_size_clean(dip);
@@ -1296,11 +1288,23 @@
             0, "mptsas_doneq_thread_threshold_prop", 10);
         mpt->m_doneq_length_threshold = ddi_prop_get_int(DDI_DEV_T_ANY, dip,
             0, "mptsas_doneq_length_threshold_prop", 8);
         mpt->m_doneq_thread_n = ddi_prop_get_int(DDI_DEV_T_ANY, dip,
             0, "mptsas_doneq_thread_n_prop", 8);
+        mpt->m_max_tune_throttle = ddi_prop_get_int(DDI_DEV_T_ANY, dip,
+            0, "mptsas_max_throttle", MAX_THROTTLE);
 
+        /*
+         *  Error check to make sure value is withing range. If nothing
+         *  is set default to original design value.
+         */
+        if (mpt->m_max_tune_throttle < THROTTLE_LO) {
+                mpt->m_max_tune_throttle = MAX_THROTTLE;
+        } else if (mpt->m_max_tune_throttle > THROTTLE_HI) {
+                mpt->m_max_tune_throttle = THROTTLE_HI;
+        }
+
         if (mpt->m_doneq_thread_n) {
                 cv_init(&mpt->m_doneq_thread_cv, NULL, CV_DRIVER, NULL);
                 mutex_init(&mpt->m_doneq_mutex, NULL, MUTEX_DRIVER, NULL);
 
                 mutex_enter(&mpt->m_doneq_mutex);
@@ -1354,13 +1358,16 @@
         cv_init(&mpt->m_cv, NULL, CV_DRIVER, NULL);
         cv_init(&mpt->m_passthru_cv, NULL, CV_DRIVER, NULL);
         cv_init(&mpt->m_fw_cv, NULL, CV_DRIVER, NULL);
         cv_init(&mpt->m_config_cv, NULL, CV_DRIVER, NULL);
         cv_init(&mpt->m_fw_diag_cv, NULL, CV_DRIVER, NULL);
-        cv_init(&mpt->m_extreq_sense_refcount_cv, NULL, CV_DRIVER, NULL);
         mutex_init_done++;
 
+#ifdef MPTSAS_FAULTINJECTION
+        TAILQ_INIT(&mpt->m_fminj_cmdq);
+#endif
+
         mutex_enter(&mpt->m_mutex);
         /*
          * Initialize power management component
          */
         if (mpt->m_options & MPTSAS_OPT_PM) {
@@ -1447,11 +1454,11 @@
         mpt->m_scsi_reset_delay = ddi_prop_get_int(DDI_DEV_T_ANY,
             dip, 0, "scsi-reset-delay", SCSI_DEFAULT_RESET_DELAY);
         if (mpt->m_scsi_reset_delay == 0) {
                 mptsas_log(mpt, CE_NOTE,
                     "scsi_reset_delay of 0 is not recommended,"
-                    " resetting to SCSI_DEFAULT_RESET_DELAY\n");
+                    " resetting to SCSI_DEFAULT_RESET_DELAY");
                 mpt->m_scsi_reset_delay = SCSI_DEFAULT_RESET_DELAY;
         }
 
         /*
          * Initialize the wait and done FIFO queue
@@ -1647,11 +1654,10 @@
                         cv_destroy(&mpt->m_cv);
                         cv_destroy(&mpt->m_passthru_cv);
                         cv_destroy(&mpt->m_fw_cv);
                         cv_destroy(&mpt->m_config_cv);
                         cv_destroy(&mpt->m_fw_diag_cv);
-                        cv_destroy(&mpt->m_extreq_sense_refcount_cv);
                 }
 
                 if (map_setup) {
                         mptsas_cfg_fini(mpt);
                 }
@@ -1895,11 +1901,12 @@
                         /*
                          * MPxIO enabled for the iport
                          */
                         ndi_devi_enter(scsi_vhci_dip, &circ1);
                         ndi_devi_enter(dip, &circ);
-                        while (pip = mdi_get_next_client_path(dip, NULL)) {
+                        while ((pip = mdi_get_next_client_path(dip, NULL)) !=
+                            NULL) {
                                 if (mdi_pi_free(pip, 0) == MDI_SUCCESS) {
                                         continue;
                                 }
                                 ndi_devi_exit(dip, circ);
                                 ndi_devi_exit(scsi_vhci_dip, circ1);
@@ -1922,12 +1929,11 @@
                 (void) pm_busy_component(dip, 0);
                 if (mpt->m_power_level != PM_LEVEL_D0) {
                         if (pm_raise_power(dip, 0, PM_LEVEL_D0) !=
                             DDI_SUCCESS) {
                                 mptsas_log(mpt, CE_WARN,
-                                    "mptsas%d: Raise power request failed.",
-                                    mpt->m_instance);
+                                    "raise power request failed");
                                 (void) pm_idle_component(dip, 0);
                                 return (DDI_FAILURE);
                         }
                 }
         }
@@ -2049,13 +2055,12 @@
 
         /* Lower the power informing PM Framework */
         if (mpt->m_options & MPTSAS_OPT_PM) {
                 if (pm_lower_power(dip, 0, PM_LEVEL_D3) != DDI_SUCCESS)
                         mptsas_log(mpt, CE_WARN,
-                            "!mptsas%d: Lower power request failed "
-                            "during detach, ignoring.",
-                            mpt->m_instance);
+                            "lower power request failed during detach, "
+                            "ignoring");
         }
 
         mutex_destroy(&mpt->m_tx_waitq_mutex);
         mutex_destroy(&mpt->m_passthru_mutex);
         mutex_destroy(&mpt->m_mutex);
@@ -2065,12 +2070,15 @@
         cv_destroy(&mpt->m_cv);
         cv_destroy(&mpt->m_passthru_cv);
         cv_destroy(&mpt->m_fw_cv);
         cv_destroy(&mpt->m_config_cv);
         cv_destroy(&mpt->m_fw_diag_cv);
-        cv_destroy(&mpt->m_extreq_sense_refcount_cv);
 
+#ifdef MPTSAS_FAULTINJECTION
+        ASSERT(TAILQ_EMPTY(&mpt->m_fminj_cmdq));
+#endif
+
         mptsas_smp_teardown(mpt);
         mptsas_enc_teardown(mpt);
         mptsas_hba_teardown(mpt);
 
         mptsas_config_space_fini(mpt);
@@ -2327,16 +2335,28 @@
             offsetof(mptsas_enclosure_t, me_link));
         return (TRUE);
 }
 
 static void
+mptsas_enc_free(mptsas_enclosure_t *mep)
+{
+        if (mep == NULL)
+                return;
+        if (mep->me_slotleds != NULL) {
+                VERIFY3U(mep->me_nslots, >, 0);
+                kmem_free(mep->me_slotleds, sizeof (uint8_t) * mep->me_nslots);
+        }
+        kmem_free(mep, sizeof (mptsas_enclosure_t));
+}
+
+static void
 mptsas_enc_teardown(mptsas_t *mpt)
 {
         mptsas_enclosure_t *mep;
 
         while ((mep = list_remove_head(&mpt->m_enclosures)) != NULL) {
-                kmem_free(mep, sizeof (mptsas_enclosure_t));
+                mptsas_enc_free(mep);
         }
         list_destroy(&mpt->m_enclosures);
 }
 
 static mptsas_enclosure_t *
@@ -2469,12 +2489,11 @@
         case PM_LEVEL_D3:
                 NDBG11(("mptsas%d: turning power OFF.", mpt->m_instance));
                 MPTSAS_POWER_OFF(mpt);
                 break;
         default:
-                mptsas_log(mpt, CE_WARN, "mptsas%d: unknown power level <%x>.",
-                    mpt->m_instance, level);
+                mptsas_log(mpt, CE_WARN, "unknown power level <%x>", level);
                 rval = DDI_FAILURE;
                 break;
         }
         mutex_exit(&mpt->m_mutex);
         return (rval);
@@ -2699,12 +2718,10 @@
         caddr_t                 memp;
         ddi_dma_cookie_t        cookie;
         size_t                  mem_size;
         int                     num_extrqsense_bufs;
 
-        ASSERT(mpt->m_extreq_sense_refcount == 0);
-
         /*
          * re-alloc when it has already alloced
          */
         if (mpt->m_dma_req_sense_hdl) {
                 rmfreemap(mpt->m_erqsense_map);
@@ -3043,11 +3060,11 @@
                 return (DDI_FAILURE);
         }
 
         ASSERT(reallen < len);
         if (reallen >= len) {
-                mptsas_log(0, CE_WARN, "!mptsas_get_name: name parameter "
+                mptsas_log(0, CE_WARN, "mptsas_get_name: name parameter "
                     "length too small, it needs to be %d bytes", reallen + 1);
         }
         return (DDI_SUCCESS);
 }
 
@@ -3090,15 +3107,22 @@
         if (ndi_dev_is_persistent_node(tgt_dip) == 0) {
                 (void) ndi_merge_node(tgt_dip, mptsas_name_child);
                 ddi_set_name_addr(tgt_dip, NULL);
                 return (DDI_FAILURE);
         }
+
         /*
-         * phymask is 0 means the virtual port for RAID
+         * The phymask exists if the port is active, otherwise
+         * nothing to do.
          */
+        if (ddi_prop_exists(DDI_DEV_T_ANY, hba_dip,
+            DDI_PROP_DONTPASS | DDI_PROP_NOTPROM, "phymask") == 0)
+                return (DDI_FAILURE);
+
         phymask = (mptsas_phymask_t)ddi_prop_get_int(DDI_DEV_T_ANY, hba_dip, 0,
             "phymask", 0);
+
         if (mdi_component_is_client(tgt_dip, NULL) == MDI_SUCCESS) {
                 if ((pip = (void *)(sd->sd_private)) == NULL) {
                         /*
                          * Very bad news if this occurs. Somehow scsi_vhci has
                          * lost the pathinfo node for this target.
@@ -3106,11 +3130,11 @@
                         return (DDI_NOT_WELL_FORMED);
                 }
 
                 if (mdi_prop_lookup_int(pip, LUN_PROP, &lun) !=
                     DDI_PROP_SUCCESS) {
-                        mptsas_log(mpt, CE_WARN, "Get lun property failed\n");
+                        mptsas_log(mpt, CE_WARN, "Get lun property failed");
                         return (DDI_FAILURE);
                 }
 
                 if (mdi_prop_lookup_string(pip, SCSI_ADDR_PROP_TARGET_PORT,
                     &psas_wwn) == MDI_SUCCESS) {
@@ -3139,11 +3163,11 @@
         addr.mta_phymask = phymask;
         mutex_enter(&mpt->m_mutex);
         ptgt = refhash_lookup(mpt->m_targets, &addr);
         mutex_exit(&mpt->m_mutex);
         if (ptgt == NULL) {
-                mptsas_log(mpt, CE_WARN, "!tgt_init: target doesn't exist or "
+                mptsas_log(mpt, CE_WARN, "tgt_init: target doesn't exist or "
                     "gone already! phymask:%x, saswwn %"PRIx64, phymask,
                     sas_wwn);
                 return (DDI_FAILURE);
         }
         if (hba_tran->tran_tgt_private == NULL) {
@@ -3184,11 +3208,11 @@
                 if (rval != 0) {
                         if (inq89 != NULL) {
                                 kmem_free(inq89, inq89_len);
                         }
 
-                        mptsas_log(mpt, CE_WARN, "!mptsas request inquiry page "
+                        mptsas_log(mpt, CE_WARN, "mptsas request inquiry page "
                             "0x89 for SATA target:%x failed!", ptgt->m_devhdl);
                         return (DDI_SUCCESS);
                 }
                 sid = (void *)(&inq89[60]);
 
@@ -3437,11 +3461,122 @@
         rval = mptsas_accept_pkt(mpt, cmd);
 
         return (rval);
 }
 
+#ifdef MPTSAS_FAULTINJECTION
+static void
+mptsas_fminj_move_cmd_to_doneq(mptsas_t *mpt, mptsas_cmd_t *cmd,
+    uchar_t reason, uint_t stat)
+{
+        struct scsi_pkt *pkt = cmd->cmd_pkt;
+
+        TAILQ_REMOVE(&mpt->m_fminj_cmdq, cmd, cmd_active_link);
+
+        /* Setup reason/statistics. */
+        pkt->pkt_reason = reason;
+        pkt->pkt_statistics = stat;
+
+        cmd->cmd_active_expiration = 0;
+
+        /* Move command to doneque. */
+        cmd->cmd_linkp = NULL;
+        cmd->cmd_flags |= CFLAG_FINISHED;
+        cmd->cmd_flags &= ~CFLAG_IN_TRANSPORT;
+
+        *mpt->m_donetail = cmd;
+        mpt->m_donetail = &cmd->cmd_linkp;
+        mpt->m_doneq_len++;
+}
+
+static void
+mptsas_fminj_move_tgt_to_doneq(mptsas_t *mpt, ushort_t target,
+    uchar_t reason, uint_t stat)
+{
+        mptsas_cmd_t *cmd;
+
+        ASSERT(mutex_owned(&mpt->m_mutex));
+
+        if (!TAILQ_EMPTY(&mpt->m_fminj_cmdq)) {
+                cmd = TAILQ_FIRST(&mpt->m_fminj_cmdq);
+                ASSERT(cmd != NULL);
+
+                while (cmd != NULL) {
+                        mptsas_cmd_t *next = TAILQ_NEXT(cmd, cmd_active_link);
+
+                        if (Tgt(cmd) == target) {
+                                mptsas_fminj_move_cmd_to_doneq(mpt, cmd,
+                                    reason, stat);
+                        }
+                        cmd = next;
+                }
+        }
+}
+
+static void
+mptsas_fminj_watchsubr(mptsas_t *mpt,
+    struct mptsas_active_cmdq *expired)
+{
+        mptsas_cmd_t *cmd;
+
+        ASSERT(mutex_owned(&mpt->m_mutex));
+
+        if (!TAILQ_EMPTY(&mpt->m_fminj_cmdq)) {
+                hrtime_t timestamp = gethrtime();
+
+                cmd = TAILQ_FIRST(&mpt->m_fminj_cmdq);
+                ASSERT(cmd != NULL);
+
+                while (cmd != NULL) {
+                        mptsas_cmd_t *next = TAILQ_NEXT(cmd, cmd_active_link);
+
+                        if (cmd->cmd_active_expiration <= timestamp) {
+                                struct scsi_pkt *pkt = cmd->cmd_pkt;
+
+                                DTRACE_PROBE1(mptsas__command__timeout,
+                                    struct scsi_pkt *, pkt);
+
+                                /* Setup proper flags. */
+                                pkt->pkt_reason = CMD_TIMEOUT;
+                                pkt->pkt_statistics = (STAT_TIMEOUT |
+                                    STAT_DEV_RESET);
+                                cmd->cmd_active_expiration = 0;
+
+                                TAILQ_REMOVE(&mpt->m_fminj_cmdq, cmd,
+                                    cmd_active_link);
+                                TAILQ_INSERT_TAIL(expired, cmd,
+                                    cmd_active_link);
+                        }
+                        cmd = next;
+                }
+        }
+}
+
 static int
+mptsas_fminject(mptsas_t *mpt, mptsas_cmd_t *cmd)
+{
+        struct scsi_pkt *pkt = cmd->cmd_pkt;
+
+        ASSERT(mutex_owned(&mpt->m_mutex));
+
+        if (pkt->pkt_flags & FLAG_PKT_TIMEOUT) {
+                if (((pkt->pkt_flags & FLAG_NOINTR) == 0) &&
+                    (pkt->pkt_comp != NULL)) {
+                        pkt->pkt_state = (STATE_GOT_BUS|STATE_GOT_TARGET|
+                            STATE_SENT_CMD);
+                        cmd->cmd_active_expiration =
+                            gethrtime() + (hrtime_t)pkt->pkt_time * NANOSEC;
+                        TAILQ_INSERT_TAIL(&mpt->m_fminj_cmdq,
+                            cmd, cmd_active_link);
+                        return (0);
+                }
+        }
+        return (-1);
+}
+#endif /* MPTSAS_FAULTINJECTION */
+
+static int
 mptsas_accept_pkt(mptsas_t *mpt, mptsas_cmd_t *cmd)
 {
         int             rval = TRAN_ACCEPT;
         mptsas_target_t *ptgt = cmd->cmd_tgt_addr;
 
@@ -3466,17 +3601,18 @@
                 ASSERT(ptgt->m_reset_delay == 0);
                 mptsas_set_throttle(mpt, ptgt, MAX_THROTTLE);
         }
 
         /*
-         * If HBA is being reset, the DevHandles are being re-initialized,
-         * which means that they could be invalid even if the target is still
-         * attached.  Check if being reset and if DevHandle is being
-         * re-initialized.  If this is the case, return BUSY so the I/O can be
-         * retried later.
+         * If HBA is being reset, the device handles will be invalidated.
+         * This is temporary and, if target is still attached, the device
+         * handles will be re-assigned when firmware reset completes.
+         * Then, if command was already waiting, complete the command
+         * otherwise return BUSY and expect transport retry.
          */
         if ((ptgt->m_devhdl == MPTSAS_INVALID_DEVHDL) && mpt->m_in_reset) {
+                NDBG20(("retry command, invalid devhdl, during FW reset."));
                 mptsas_set_pkt_reason(mpt, cmd, CMD_RESET, STAT_BUS_RESET);
                 if (cmd->cmd_flags & CFLAG_TXQ) {
                         mptsas_doneq_add(mpt, cmd);
                         mptsas_doneq_empty(mpt);
                         return (rval);
@@ -3484,29 +3620,39 @@
                         return (TRAN_BUSY);
                 }
         }
 
         /*
-         * If device handle has already been invalidated, just
-         * fail the command. In theory, command from scsi_vhci
-         * client is impossible send down command with invalid
-         * devhdl since devhdl is set after path offline, target
-         * driver is not suppose to select a offlined path.
+         * If the device handle has been invalidated, set the response
+         * reason to indicate the device is gone. Then add the
+         * command to the done queue and run the completion routine
+         * so the initiator of the command can clean up.
          */
         if (ptgt->m_devhdl == MPTSAS_INVALID_DEVHDL) {
-                NDBG3(("rejecting command, it might because invalid devhdl "
-                    "request."));
+                NDBG20(("rejecting command, invalid devhdl because "
+                    "device gone."));
                 mptsas_set_pkt_reason(mpt, cmd, CMD_DEV_GONE, STAT_TERMINATED);
                 if (cmd->cmd_flags & CFLAG_TXQ) {
                         mptsas_doneq_add(mpt, cmd);
                         mptsas_doneq_empty(mpt);
                         return (rval);
                 } else {
                         return (TRAN_FATAL_ERROR);
                 }
         }
+
         /*
+         * Do fault injecttion before transmitting command.
+         * FLAG_NOINTR commands are skipped.
+         */
+#ifdef MPTSAS_FAULTINJECTION
+        if (!mptsas_fminject(mpt, cmd)) {
+                return (TRAN_ACCEPT);
+        }
+#endif
+
+        /*
          * The first case is the normal case.  mpt gets a command from the
          * target driver and starts it.
          * Since SMID 0 is reserved and the TM slot is reserved, the actual max
          * commands is m_max_requests - 2.
          */
@@ -3615,10 +3761,17 @@
 {
         struct scsi_pkt *pkt = CMD2PKT(cmd);
 
         NDBG1(("mptsas_prepare_pkt: cmd=0x%p", (void *)cmd));
 
+#ifdef MPTSAS_FAULTINJECTION
+        /* Check for fault flags prior to perform actual initialization. */
+        if (pkt->pkt_flags & FLAG_PKT_BUSY) {
+                return (TRAN_BUSY);
+        }
+#endif
+
         /*
          * Reinitialize some fields that need it; the packet may
          * have been resubmitted
          */
         pkt->pkt_reason = CMD_CMPLT;
@@ -3727,66 +3880,25 @@
                 cmd->cmd_tgt_addr = ptgt;
 
                 if ((cmdlen > sizeof (cmd->cmd_cdb)) ||
                     (tgtlen > PKT_PRIV_LEN) ||
                     (statuslen > EXTCMDS_STATUS_SIZE)) {
-                        int failure;
-
+                        if (mptsas_pkt_alloc_extern(mpt, cmd,
+                            cmdlen, tgtlen, statuslen, kf)) {
                         /*
-                         * We are going to allocate external packet space which
-                         * might include the sense data buffer for DMA so we
-                         * need to increase the reference counter here.  In a
-                         * case the HBA is in reset we just simply free the
-                         * allocated packet and bail out.
-                         */
-                        mutex_enter(&mpt->m_mutex);
-                        if (mpt->m_in_reset) {
-                                mutex_exit(&mpt->m_mutex);
-
-                                cmd->cmd_flags = CFLAG_FREE;
-                                kmem_cache_free(mpt->m_kmem_cache, cmd);
-                                return (NULL);
-                        }
-                        mpt->m_extreq_sense_refcount++;
-                        ASSERT(mpt->m_extreq_sense_refcount > 0);
-                        mutex_exit(&mpt->m_mutex);
-
-                        /*
-                         * if extern alloc fails, all will be
-                         * deallocated, including cmd
-                         */
-                        failure = mptsas_pkt_alloc_extern(mpt, cmd,
-                            cmdlen, tgtlen, statuslen, kf);
-
-                        if (failure != 0 || cmd->cmd_extrqslen == 0) {
-                                /*
-                                 * If the external packet space allocation
-                                 * failed, or we didn't allocate the sense
-                                 * data buffer for DMA we need to decrease the
-                                 * reference counter.
-                                 */
-                                mutex_enter(&mpt->m_mutex);
-                                ASSERT(mpt->m_extreq_sense_refcount > 0);
-                                mpt->m_extreq_sense_refcount--;
-                                if (mpt->m_extreq_sense_refcount == 0)
-                                        cv_broadcast(
-                                            &mpt->m_extreq_sense_refcount_cv);
-                                mutex_exit(&mpt->m_mutex);
-
-                                if (failure != 0) {
-                                        /*
                                          * if extern allocation fails, it will
                                          * deallocate the new pkt as well
                                          */
                                         return (NULL);
                                 }
                         }
-                }
                 new_cmd = cmd;
 
         } else {
                 cmd = PKT2CMD(pkt);
+                pkt->pkt_start = 0;
+                pkt->pkt_stop = 0;
                 new_cmd = NULL;
         }
 
 
         /* grab cmd->cmd_cookiec here as oldcookiec */
@@ -3901,11 +4013,11 @@
 get_dma_cookies:
                 cmd->cmd_flags |= CFLAG_DMAVALID;
                 ASSERT(cmd->cmd_cookiec > 0);
 
                 if (cmd->cmd_cookiec > MPTSAS_MAX_CMD_SEGS) {
-                        mptsas_log(mpt, CE_NOTE, "large cookiec received %d\n",
+                        mptsas_log(mpt, CE_NOTE, "large cookiec received %d",
                             cmd->cmd_cookiec);
                         bioerror(bp, EINVAL);
                         if (new_cmd) {
                                 mptsas_scsi_destroy_pkt(ap, pkt);
                         }
@@ -4063,27 +4175,12 @@
             (CFLAG_FREE | CFLAG_CDBEXTERN | CFLAG_PRIVEXTERN |
             CFLAG_SCBEXTERN)) == 0) {
                 cmd->cmd_flags = CFLAG_FREE;
                 kmem_cache_free(mpt->m_kmem_cache, (void *)cmd);
         } else {
-                boolean_t extrqslen = cmd->cmd_extrqslen != 0;
-
                 mptsas_pkt_destroy_extern(mpt, cmd);
-
-                /*
-                 * If the packet had the sense data buffer for DMA allocated we
-                 * need to decrease the reference counter.
-                 */
-                if (extrqslen) {
-                        mutex_enter(&mpt->m_mutex);
-                        ASSERT(mpt->m_extreq_sense_refcount > 0);
-                        mpt->m_extreq_sense_refcount--;
-                        if (mpt->m_extreq_sense_refcount == 0)
-                                cv_broadcast(&mpt->m_extreq_sense_refcount_cv);
-                        mutex_exit(&mpt->m_mutex);
                 }
-        }
 }
 
 /*
  * kmem cache constructor and destructor:
  * When constructing, we bzero the cmd and allocate the dma handle
@@ -5238,12 +5335,11 @@
          * This is a success reply so just complete the IO.  First, do a sanity
          * check on the SMID.  The final slot is used for TM requests, which
          * would not come into this reply handler.
          */
         if ((SMID == 0) || (SMID > slots->m_n_normal)) {
-                mptsas_log(mpt, CE_WARN, "?Received invalid SMID of %d\n",
-                    SMID);
+                mptsas_log(mpt, CE_WARN, "received invalid SMID of %d", SMID);
                 ddi_fm_service_impact(mpt->m_dip, DDI_SERVICE_UNAFFECTED);
                 return;
         }
 
         cmd = slots->m_slot[SMID];
@@ -5250,16 +5346,18 @@
 
         /*
          * print warning and return if the slot is empty
          */
         if (cmd == NULL) {
-                mptsas_log(mpt, CE_WARN, "?NULL command for successful SCSI IO "
+                mptsas_log(mpt, CE_WARN, "NULL command for successful SCSI IO "
                     "in slot %d", SMID);
                 return;
         }
 
         pkt = CMD2PKT(cmd);
+        ASSERT(pkt->pkt_start != 0);
+        pkt->pkt_stop = gethrtime();
         pkt->pkt_state |= (STATE_GOT_BUS | STATE_GOT_TARGET | STATE_SENT_CMD |
             STATE_GOT_STATUS);
         if (cmd->cmd_flags & CFLAG_DMAVALID) {
                 pkt->pkt_state |= STATE_XFERRED_DATA;
         }
@@ -5319,12 +5417,12 @@
         if ((reply_addr < reply_frame_dma_baseaddr) ||
             (reply_addr >= (reply_frame_dma_baseaddr +
             (mpt->m_reply_frame_size * mpt->m_max_replies))) ||
             ((reply_addr - reply_frame_dma_baseaddr) %
             mpt->m_reply_frame_size != 0)) {
-                mptsas_log(mpt, CE_WARN, "?Received invalid reply frame "
-                    "address 0x%x\n", reply_addr);
+                mptsas_log(mpt, CE_WARN, "received invalid reply frame "
+                    "address 0x%x", reply_addr);
                 ddi_fm_service_impact(mpt->m_dip, DDI_SERVICE_UNAFFECTED);
                 return;
         }
 
         (void) ddi_dma_sync(mpt->m_dma_reply_frame_hdl, 0, 0,
@@ -5345,12 +5443,12 @@
                 /*
                  * This could be a TM reply, which use the last allocated SMID,
                  * so allow for that.
                  */
                 if ((SMID == 0) || (SMID > (slots->m_n_normal + 1))) {
-                        mptsas_log(mpt, CE_WARN, "?Received invalid SMID of "
-                            "%d\n", SMID);
+                        mptsas_log(mpt, CE_WARN, "received invalid SMID of "
+                            "%d", SMID);
                         ddi_fm_service_impact(mpt->m_dip,
                             DDI_SERVICE_UNAFFECTED);
                         return;
                 }
 
@@ -5358,11 +5456,11 @@
 
                 /*
                  * print warning and return if the slot is empty
                  */
                 if (cmd == NULL) {
-                        mptsas_log(mpt, CE_WARN, "?NULL command for address "
+                        mptsas_log(mpt, CE_WARN, "NULL command for address "
                             "reply in slot %d", SMID);
                         return;
                 }
                 if ((cmd->cmd_flags &
                     (CFLAG_PASSTHRU | CFLAG_CONFIG | CFLAG_FW_DIAG))) {
@@ -5471,11 +5569,11 @@
 
                         /*
                          * print warning and return if the slot is empty
                          */
                         if (cmd == NULL) {
-                                mptsas_log(mpt, CE_WARN, "?NULL command for "
+                                mptsas_log(mpt, CE_WARN, "NULL command for "
                                     "address reply in slot %d", SMID);
                                 return;
                         }
                         cmd->cmd_rfm = reply_addr;
                         cmd->cmd_flags |= CFLAG_FINISHED;
@@ -5556,20 +5654,22 @@
                         (void) sprintf(wwn_str, "w%016"PRIx64, sas_wwn);
                 }
                 loginfo = ddi_get32(mpt->m_acc_reply_frame_hdl,
                     &reply->IOCLogInfo);
                 mptsas_log(mpt, CE_NOTE,
-                    "?Log info 0x%x received for target %d %s.\n"
-                    "\tscsi_status=0x%x, ioc_status=0x%x, scsi_state=0x%x",
+                    "log info 0x%x received for target %d %s, "
+                    "scsi_status=0x%x, ioc_status=0x%x, scsi_state=0x%x",
                     loginfo, Tgt(cmd), wwn_str, scsi_status, ioc_status,
                     scsi_state);
         }
 
         NDBG31(("\t\tscsi_status=0x%x, ioc_status=0x%x, scsi_state=0x%x",
             scsi_status, ioc_status, scsi_state));
 
         pkt = CMD2PKT(cmd);
+        ASSERT(pkt->pkt_start != 0);
+        pkt->pkt_stop = gethrtime();
         *(pkt->pkt_scbp) = scsi_status;
 
         if (loginfo == 0x31170000) {
                 /*
                  * if loginfo PL_LOGINFO_CODE_IO_DEVICE_MISSING_DELAY_RETRY
@@ -5593,11 +5693,11 @@
         }
 
         if (scsi_state & MPI2_SCSI_STATE_RESPONSE_INFO_VALID) {
                 responsedata &= 0x000000FF;
                 if (responsedata & MPTSAS_SCSI_RESPONSE_CODE_TLR_OFF) {
-                        mptsas_log(mpt, CE_NOTE, "Do not support the TLR\n");
+                        mptsas_log(mpt, CE_NOTE, "TLR not supported");
                         pkt->pkt_reason = CMD_TLR_OFF;
                         return;
                 }
         }
 
@@ -5669,11 +5769,11 @@
                             sizeof (mptsas_topo_change_list_t),
                             KM_NOSLEEP);
                         if (topo_node == NULL) {
                                 mptsas_log(mpt, CE_NOTE, "No memory"
                                     "resource for handle SAS dynamic"
-                                    "reconfigure.\n");
+                                    "reconfigure");
                                 break;
                         }
                         topo_node->mpt = mpt;
                         topo_node->event = MPTSAS_DR_EVENT_RECONFIG_TARGET;
                         topo_node->un.phymask = ptgt->m_addr.mta_phymask;
@@ -5687,11 +5787,11 @@
                             DDI_NOSLEEP)) != DDI_SUCCESS) {
                                 kmem_free(topo_node,
                                     sizeof (mptsas_topo_change_list_t));
                                 mptsas_log(mpt, CE_NOTE, "mptsas start taskq"
                                     "for handle SAS dynamic reconfigure"
-                                    "failed. \n");
+                                    "failed");
                         }
                 }
                 break;
         case MPI2_SCSI_STATUS_GOOD:
                 switch (ioc_status & MPI2_IOCSTATUS_MASK) {
@@ -5767,11 +5867,11 @@
 
                         (void) mptsas_accept_pkt(mpt, cmd);
                         break;
                 default:
                         mptsas_log(mpt, CE_WARN,
-                            "unknown ioc_status = %x\n", ioc_status);
+                            "unknown ioc_status = %x", ioc_status);
                         mptsas_log(mpt, CE_CONT, "scsi_state = %x, transfer "
                             "count = %x, scsi_status = %x", scsi_state,
                             xferred, scsi_status);
                         break;
                 }
@@ -5784,14 +5884,14 @@
                 break;
         case MPI2_SCSI_STATUS_RESERVATION_CONFLICT:
                 NDBG31(("scsi_status reservation conflict received"));
                 break;
         default:
-                mptsas_log(mpt, CE_WARN, "scsi_status=%x, ioc_status=%x\n",
+                mptsas_log(mpt, CE_WARN, "scsi_status=%x, ioc_status=%x",
                     scsi_status, ioc_status);
                 mptsas_log(mpt, CE_WARN,
-                    "mptsas_process_intr: invalid scsi status\n");
+                    "mptsas_process_intr: invalid scsi status");
                 break;
         }
 }
 
 static void
@@ -5809,11 +5909,11 @@
         log_info = ddi_get32(mpt->m_acc_reply_frame_hdl, &reply->IOCLogInfo);
         dev_handle = ddi_get16(mpt->m_acc_reply_frame_hdl, &reply->DevHandle);
 
         if (ioc_status != MPI2_IOCSTATUS_SUCCESS) {
                 mptsas_log(mpt, CE_WARN, "mptsas_check_task_mgt: Task 0x%x "
-                    "failed. IOCStatus=0x%x IOCLogInfo=0x%x target=%d\n",
+                    "failed. IOCStatus=0x%x IOCLogInfo=0x%x target=%d",
                     task_type, ioc_status, log_info, dev_handle);
                 pkt->pkt_reason = CMD_INCOMPLETE;
                 return;
         }
 
@@ -5831,11 +5931,11 @@
                 /*
                  * Check for invalid DevHandle of 0 in case application
                  * sends bad command.  DevHandle of 0 could cause problems.
                  */
                 if (dev_handle == 0) {
-                        mptsas_log(mpt, CE_WARN, "!Can't flush target with"
+                        mptsas_log(mpt, CE_WARN, "Can't flush target with"
                             " DevHandle of 0.");
                 } else {
                         mptsas_flush_target(mpt, dev_handle, Lun(cmd),
                             task_type);
                 }
@@ -6018,11 +6118,11 @@
             reply_type == MPI25_RPY_DESCRIPT_FLAGS_FAST_PATH_SCSI_IO_SUCCESS) {
                 mptsas_handle_scsi_io_success(mpt, reply_desc_union);
         } else if (reply_type == MPI2_RPY_DESCRIPT_FLAGS_ADDRESS_REPLY) {
                 mptsas_handle_address_reply(mpt, reply_desc_union);
         } else {
-                mptsas_log(mpt, CE_WARN, "?Bad reply type %x", reply_type);
+                mptsas_log(mpt, CE_WARN, "bad reply type %x", reply_type);
                 ddi_fm_service_impact(mpt->m_dip, DDI_SERVICE_UNAFFECTED);
         }
 
         /*
          * Clear the reply descriptor for re-use and increment
@@ -6462,14 +6562,15 @@
                 mutex_enter(&mpt->m_mutex);
                 /*
                  * If HBA is being reset, don't perform operations depending
                  * on the IOC. We must free the topo list, however.
                  */
-                if (!mpt->m_in_reset)
+                if (!mpt->m_in_reset) {
                         mptsas_handle_topo_change(topo_node, parent);
-                else
+                } else {
                         NDBG20(("skipping topo change received during reset"));
+                }
                 save_node = topo_node;
                 topo_node = topo_node->next;
                 ASSERT(save_node);
                 kmem_free(save_node, sizeof (mptsas_topo_change_list_t));
                 mutex_exit(&mpt->m_mutex);
@@ -6551,20 +6652,20 @@
                         rval = mptsas_get_target_device_info(mpt, page_address,
                             &devhdl, &ptgt);
                         if (rval == DEV_INFO_WRONG_DEVICE_TYPE) {
                                 mptsas_log(mpt, CE_NOTE,
                                     "mptsas_handle_topo_change: target %d is "
-                                    "not a SAS/SATA device. \n",
+                                    "not a SAS/SATA device",
                                     topo_node->devhdl);
                         } else if (rval == DEV_INFO_FAIL_ALLOC) {
                                 mptsas_log(mpt, CE_NOTE,
                                     "mptsas_handle_topo_change: could not "
-                                    "allocate memory. \n");
+                                    "allocate memory");
                         } else if (rval == DEV_INFO_FAIL_GUID) {
                                 mptsas_log(mpt, CE_NOTE,
                                     "mptsas_handle_topo_change: could not "
-                                    "get SATA GUID for target %d. \n",
+                                    "get SATA GUID for target %d",
                                     topo_node->devhdl);
                         }
                         /*
                          * If rval is DEV_INFO_PHYS_DISK or indicates failure
                          * then there is nothing else to do, just leave.
@@ -6787,12 +6888,10 @@
                                 break;
                         }
                 }
 
                 mutex_enter(&mpt->m_mutex);
-                ptgt->m_led_status = 0;
-                (void) mptsas_flush_led_status(mpt, ptgt);
                 if (rval == DDI_SUCCESS) {
                         refhash_remove(mpt->m_targets, ptgt);
                         ptgt = NULL;
                 } else {
                         /*
@@ -6888,11 +6987,11 @@
                  * successfully.
                  */
                 mutex_exit(&mpt->m_mutex);
 
                 ndi_devi_enter(parent, &circ1);
-                rval = mptsas_offline_smp(parent, psmp, NDI_DEVI_REMOVE);
+                rval = mptsas_offline_smp(parent, psmp);
                 ndi_devi_exit(parent, circ1);
 
                 dev_info = psmp->m_deviceinfo;
                 if ((dev_info & DEVINFO_DIRECT_ATTACHED) ==
                     DEVINFO_DIRECT_ATTACHED) {
@@ -6901,10 +7000,11 @@
                             DDI_PROP_SUCCESS) {
                                 (void) ddi_prop_remove(DDI_DEV_T_NONE, parent,
                                     MPTSAS_VIRTUAL_PORT);
                                 mptsas_log(mpt, CE_WARN, "mptsas virtual port "
                                     "prop update failed");
+                                mutex_enter(&mpt->m_mutex);
                                 return;
                         }
                         /*
                          * Check whether the smp connected to the iport,
                          */
@@ -6913,10 +7013,11 @@
                             DDI_PROP_SUCCESS) {
                                 (void) ddi_prop_remove(DDI_DEV_T_NONE, parent,
                                     MPTSAS_NUM_PHYS);
                                 mptsas_log(mpt, CE_WARN, "mptsas num phys"
                                     "prop update failed");
+                                mutex_enter(&mpt->m_mutex);
                                 return;
                         }
                         /*
                          * Clear parent's attached-port props
                          */
@@ -6926,10 +7027,11 @@
                             DDI_PROP_SUCCESS) {
                                 (void) ddi_prop_remove(DDI_DEV_T_NONE, parent,
                                     SCSI_ADDR_PROP_ATTACHED_PORT);
                                 mptsas_log(mpt, CE_WARN, "mptsas attached port "
                                     "prop update failed");
+                                mutex_enter(&mpt->m_mutex);
                                 return;
                         }
                 }
 
                 mutex_enter(&mpt->m_mutex);
@@ -7060,15 +7162,15 @@
         eventreply = (pMpi2EventNotificationReply_t)
             (mpt->m_reply_frame + (rfm -
             (mpt->m_reply_frame_dma_addr & 0xffffffffu)));
         event = ddi_get16(mpt->m_acc_reply_frame_hdl, &eventreply->Event);
 
-        if (iocstatus = ddi_get16(mpt->m_acc_reply_frame_hdl,
-            &eventreply->IOCStatus)) {
+        if ((iocstatus = ddi_get16(mpt->m_acc_reply_frame_hdl,
+            &eventreply->IOCStatus)) != 0) {
                 if (iocstatus == MPI2_IOCSTATUS_FLAG_LOG_INFO_AVAILABLE) {
                         mptsas_log(mpt, CE_WARN,
-                            "!mptsas_handle_event_sync: event 0x%x, "
+                            "mptsas_handle_event_sync: event 0x%x, "
                             "IOCStatus=0x%x, "
                             "IOCLogInfo=0x%x", event, iocstatus,
                             ddi_get32(mpt->m_acc_reply_frame_hdl,
                             &eventreply->IOCLogInfo));
                 } else {
@@ -7536,11 +7638,11 @@
                                         topo_head = topo_head->next;
                                         kmem_free(topo_node,
                                             sizeof (mptsas_topo_change_list_t));
                                 }
                                 mptsas_log(mpt, CE_NOTE, "mptsas start taskq "
-                                    "for handle SAS DR event failed. \n");
+                                    "for handle SAS DR event failed");
                         }
                 }
                 break;
         }
         case MPI2_EVENT_IR_CONFIGURATION_CHANGE_LIST:
@@ -7720,11 +7822,11 @@
                                         topo_head = topo_head->next;
                                         kmem_free(topo_node,
                                             sizeof (mptsas_topo_change_list_t));
                                 }
                                 mptsas_log(mpt, CE_NOTE, "mptsas start taskq "
-                                    "for handle SAS DR event failed. \n");
+                                    "for handle SAS DR event failed");
                         }
                 }
                 break;
         }
         default:
@@ -7765,15 +7867,15 @@
         eventreply = (pMpi2EventNotificationReply_t)
             (mpt->m_reply_frame + (rfm -
             (mpt->m_reply_frame_dma_addr & 0xffffffffu)));
         event = ddi_get16(mpt->m_acc_reply_frame_hdl, &eventreply->Event);
 
-        if (iocstatus = ddi_get16(mpt->m_acc_reply_frame_hdl,
-            &eventreply->IOCStatus)) {
+        if ((iocstatus = ddi_get16(mpt->m_acc_reply_frame_hdl,
+            &eventreply->IOCStatus)) != 0) {
                 if (iocstatus == MPI2_IOCSTATUS_FLAG_LOG_INFO_AVAILABLE) {
                         mptsas_log(mpt, CE_WARN,
-                            "!mptsas_handle_event: IOCStatus=0x%x, "
+                            "mptsas_handle_event: IOCStatus=0x%x, "
                             "IOCLogInfo=0x%x", iocstatus,
                             ddi_get32(mpt->m_acc_reply_frame_hdl,
                             &eventreply->IOCLogInfo));
                 } else {
                         mptsas_log(mpt, CE_WARN,
@@ -7880,11 +7982,12 @@
                         break;
                 case MPI2_EVENT_SAS_ENCL_RC_NOT_RESPONDING:
                         mep = mptsas_enc_lookup(mpt, enchdl);
                         if (mep != NULL) {
                                 list_remove(&mpt->m_enclosures, mep);
-                                kmem_free(mep, sizeof (*mep));
+                                mptsas_enc_free(mep);
+                                mep = NULL;
                         }
                         (void) sprintf(string, ", not responding");
                         break;
                 default:
                 break;
@@ -8147,11 +8250,11 @@
 
                         i = state & MPI2_RAIDVOL0_SETTING_MASK_WRITE_CACHING;
                         mptsas_log(mpt, CE_NOTE, " Volume %d settings changed"
                             ", auto-config of hot-swap drives is %s"
                             ", write caching is %s"
-                            ", hot-spare pool mask is %02x\n",
+                            ", hot-spare pool mask is %02x",
                             vol, state &
                             MPI2_RAIDVOL0_SETTING_AUTO_CONFIG_HSWAP_DISABLE
                             ? "disabled" : "enabled",
                             i == MPI2_RAIDVOL0_SETTING_UNCHANGED
                             ? "controlled by member disks" :
@@ -8167,11 +8270,11 @@
                 {
                         mpt->m_raidconfig[config].m_raidvol[vol].m_state =
                             (uint8_t)state;
 
                         mptsas_log(mpt, CE_NOTE,
-                            "Volume %d is now %s\n", vol,
+                            "Volume %d is now %s", vol,
                             state == MPI2_RAID_VOL_STATE_OPTIMAL
                             ? "optimal" :
                             state == MPI2_RAID_VOL_STATE_DEGRADED
                             ? "degraded" :
                             state == MPI2_RAID_VOL_STATE_ONLINE
@@ -8189,11 +8292,11 @@
                 {
                         mpt->m_raidconfig[config].m_raidvol[vol].
                             m_statusflags = state;
 
                         mptsas_log(mpt, CE_NOTE,
-                            " Volume %d is now %s%s%s%s%s%s%s%s%s\n",
+                            " Volume %d is now %s%s%s%s%s%s%s%s%s",
                             vol,
                             state & MPI2_RAIDVOL0_STATUS_FLAG_ENABLED
                             ? ", enabled" : ", disabled",
                             state & MPI2_RAIDVOL0_STATUS_FLAG_QUIESCED
                             ? ", quiesced" : "",
@@ -8259,11 +8362,11 @@
                 case MPI2_EVENT_IR_PHYSDISK_RC_STATUS_FLAGS_CHANGED:
                         status = state;
                         mptsas_log(mpt, CE_NOTE,
                             " PhysDiskNum %d with DevHandle 0x%x in slot %d "
                             "for enclosure with handle 0x%x is now "
-                            "%s%s%s%s%s\n", physdisknum, devhandle, slot,
+                            "%s%s%s%s%s", physdisknum, devhandle, slot,
                             enchandle,
                             status & MPI2_PHYSDISK0_STATUS_FLAG_INACTIVE_VOLUME
                             ? ", inactive" : ", active",
                             status & MPI2_PHYSDISK0_STATUS_FLAG_OUT_OF_SYNC
                             ? ", out of sync" : "",
@@ -8277,11 +8380,11 @@
                         break;
 
                 case MPI2_EVENT_IR_PHYSDISK_RC_STATE_CHANGED:
                         mptsas_log(mpt, CE_NOTE,
                             " PhysDiskNum %d with DevHandle 0x%x in slot %d "
-                            "for enclosure with handle 0x%x is now %s\n",
+                            "for enclosure with handle 0x%x is now %s",
                             physdisknum, devhandle, slot, enchandle,
                             state == MPI2_RAID_PD_STATE_OPTIMAL
                             ? "optimal" :
                             state == MPI2_RAID_PD_STATE_REBUILDING
                             ? "rebuilding" :
@@ -8300,11 +8403,65 @@
                             "state unknown");
                         break;
                 }
                 break;
         }
+        case MPI2_EVENT_ACTIVE_CABLE_EXCEPTION:
+        {
+                pMpi26EventDataActiveCableExcept_t      actcable;
+                uint32_t power;
+                uint8_t reason, id;
+
+                actcable = (pMpi26EventDataActiveCableExcept_t)
+                    eventreply->EventData;
+                power = ddi_get32(mpt->m_acc_reply_frame_hdl,
+                    &actcable->ActiveCablePowerRequirement);
+                reason = ddi_get8(mpt->m_acc_reply_frame_hdl,
+                    &actcable->ReasonCode);
+                id = ddi_get8(mpt->m_acc_reply_frame_hdl,
+                    &actcable->ReceptacleID);
+
+                /*
+                 * It'd be nice if this weren't just logging to the system but
+                 * were telling FMA about the active cable problem and FMA was
+                 * aware of the cable topology and state.
+                 */
+                switch (reason) {
+                case MPI26_EVENT_ACTIVE_CABLE_PRESENT:
+                        /* Don't log anything if it's fine */
+                        break;
+                case MPI26_EVENT_ACTIVE_CABLE_INSUFFICIENT_POWER:
+                        mptsas_log(mpt, CE_WARN, "An active cable (id %u) does "
+                            "not have sufficient power to be enabled. "
+                            "Devices connected to this cable will not be "
+                            "visible to the system.", id);
+                        if (power == UINT32_MAX) {
+                                mptsas_log(mpt, CE_CONT, "The cable's power "
+                                    "requirements are unknown.\n");
+                        } else {
+                                mptsas_log(mpt, CE_CONT, "The cable requires "
+                                    "%u mW of power to function.\n", power);
+                        }
+                        break;
+                case MPI26_EVENT_ACTIVE_CABLE_DEGRADED:
+                        mptsas_log(mpt, CE_WARN, "An active cable (id %u) is "
+                            "degraded and not running at its full speed. "
+                            "Some devices might not appear.", id);
+                        break;
         default:
+                        break;
+                }
+                break;
+        }
+        case MPI2_EVENT_PCIE_DEVICE_STATUS_CHANGE:
+        case MPI2_EVENT_PCIE_ENUMERATION:
+        case MPI2_EVENT_PCIE_TOPOLOGY_CHANGE_LIST:
+        case MPI2_EVENT_PCIE_LINK_COUNTER:
+                mptsas_log(mpt, CE_NOTE, "Unhandled mpt_sas PCIe device "
+                    "event received (0x%x)", event);
+                break;
+        default:
                 NDBG20(("mptsas%d: unknown event %x received",
                     mpt->m_instance, event));
                 break;
         }
 
@@ -8547,11 +8704,11 @@
                 }
                 cmd->cmd_linkp = NULL;
                 mutex_exit(&mpt->m_tx_waitq_mutex);
                 if (mptsas_accept_pkt(mpt, cmd) != TRAN_ACCEPT)
                         cmn_err(CE_WARN, "mpt: mptsas_accept_tx_waitq: failed "
-                            "to accept cmd on queue\n");
+                            "to accept cmd on queue");
                 mutex_enter(&mpt->m_tx_waitq_mutex);
         }
 }
 
 
@@ -8633,11 +8790,11 @@
                         break;
                 case MSG_ORDERED_QTAG:
                         control |= MPI2_SCSIIO_CONTROL_ORDEREDQ;
                         break;
                 default:
-                        mptsas_log(mpt, CE_WARN, "mpt: Invalid tag type\n");
+                        mptsas_log(mpt, CE_WARN, "invalid tag type");
                         break;
                 }
         } else {
                 if (*(cmd->cmd_pkt->pkt_cdbp) != SCMD_REQUEST_SENSE) {
                                 ptgt->m_t_throttle = 1;
@@ -8714,10 +8871,11 @@
             SMID, (void *)io_request, (void *)cmd));
 
         (void) ddi_dma_sync(dma_hdl, 0, 0, DDI_DMA_SYNC_FORDEV);
         (void) ddi_dma_sync(mpt->m_dma_req_sense_hdl, 0, 0,
             DDI_DMA_SYNC_FORDEV);
+        pkt->pkt_start = gethrtime();
 
         /*
          * Build request descriptor and write it to the request desc post reg.
          */
         request_desc |= (SMID << 16);
@@ -8725,12 +8883,13 @@
         MPTSAS_START_CMD(mpt, request_desc);
 
         /*
          * Start timeout.
          */
-        cmd->cmd_active_expiration =
-            gethrtime() + (hrtime_t)pkt->pkt_time * NANOSEC;
+        cmd->cmd_active_expiration = pkt->pkt_start +
+            (hrtime_t)pkt->pkt_time * (hrtime_t)NANOSEC;
+
 #ifdef MPTSAS_TEST
         /*
          * Force timeouts to happen immediately.
          */
         if (mptsas_test_timeouts)
@@ -9248,12 +9407,15 @@
         if (mpt->m_softstate & (MPTSAS_SS_QUIESCED | MPTSAS_SS_DRAINING)) {
                 return;
         }
 
         if (what == HOLD_THROTTLE) {
-                ptgt->m_t_throttle = HOLD_THROTTLE;
+                ptgt->m_t_throttle = what;
         } else if (ptgt->m_reset_delay == 0) {
+                if (what == MAX_THROTTLE)
+                        ptgt->m_t_throttle = mpt->m_max_tune_throttle;
+                else
                 ptgt->m_t_throttle = what;
         }
 }
 
 /*
@@ -9400,10 +9562,14 @@
         default:
                 mptsas_log(mpt, CE_WARN, "Unknown task management type %d.",
                     tasktype);
                 break;
         }
+
+#ifdef MPTSAS_FAULTINJECTION
+        mptsas_fminj_move_tgt_to_doneq(mpt, target, reason, stat);
+#endif
 }
 
 /*
  * Clean up hba state, abort all outstanding command and commands in waitq
  * reset timeout of all targets.
@@ -9691,10 +9857,21 @@
          */
         if (pkt != NULL) {
                 /* abort the specified packet */
                 sp = PKT2CMD(pkt);
 
+#ifdef MPTSAS_FAULTINJECTION
+        /* Command already on the list. */
+        if (((pkt->pkt_flags & FLAG_PKT_TIMEOUT) != 0) &&
+            (sp->cmd_active_expiration != 0)) {
+                mptsas_fminj_move_cmd_to_doneq(mpt, sp, CMD_ABORTED,
+                    STAT_ABORTED);
+                rval = TRUE;
+                goto done;
+        }
+#endif
+
                 if (sp->cmd_queued) {
                         NDBG23(("mptsas_do_scsi_abort: queued sp=0x%p aborted",
                             (void *)sp));
                         mptsas_waitq_delete(mpt, sp);
                         mptsas_set_pkt_reason(mpt, sp, CMD_ABORTED,
@@ -9981,14 +10158,14 @@
 
         va_start(ap, fmt);
         (void) vsprintf(mptsas_log_buf, fmt, ap);
         va_end(ap);
 
-        if (level == CE_CONT) {
-                scsi_log(dev, mptsas_label, level, "%s\n", mptsas_log_buf);
+        if (level == CE_CONT || level == CE_NOTE) {
+                scsi_log(dev, mptsas_label, level, "!%s\n", mptsas_log_buf);
         } else {
-                scsi_log(dev, mptsas_label, level, "%s", mptsas_log_buf);
+                scsi_log(dev, mptsas_label, level, "!%s", mptsas_log_buf);
         }
 
         mutex_exit(&mptsas_log_mutex);
 }
 
@@ -10052,10 +10229,16 @@
 #endif
 
         mptsas_t        *mpt;
         uint32_t        doorbell;
 
+#ifdef MPTSAS_FAULTINJECTION
+        struct mptsas_active_cmdq finj_cmds;
+
+        TAILQ_INIT(&finj_cmds);
+#endif
+
         NDBG30(("mptsas_watch"));
 
         rw_enter(&mptsas_global_rwlock, RW_READER);
         for (mpt = mptsas_head; mpt != (mptsas_t *)NULL; mpt = mpt->m_next) {
 
@@ -10095,18 +10278,38 @@
                 if (mpt->m_options & MPTSAS_OPT_PM) {
                         mpt->m_busy = 0;
                         (void) pm_idle_component(mpt->m_dip, 0);
                 }
 
+#ifdef MPTSAS_FAULTINJECTION
+                mptsas_fminj_watchsubr(mpt, &finj_cmds);
+#endif
+
                 mutex_exit(&mpt->m_mutex);
         }
         rw_exit(&mptsas_global_rwlock);
 
         mutex_enter(&mptsas_global_mutex);
         if (mptsas_timeouts_enabled)
                 mptsas_timeout_id = timeout(mptsas_watch, NULL, mptsas_tick);
         mutex_exit(&mptsas_global_mutex);
+
+#ifdef MPTSAS_FAULTINJECTION
+        /* Complete all completed commands. */
+        if (!TAILQ_EMPTY(&finj_cmds)) {
+                mptsas_cmd_t *cmd;
+
+                while ((cmd = TAILQ_FIRST(&finj_cmds)) != NULL) {
+                        TAILQ_REMOVE(&finj_cmds, cmd, cmd_active_link);
+                        struct scsi_pkt *pkt = cmd->cmd_pkt;
+
+                        if (pkt->pkt_comp != NULL) {
+                                (*pkt->pkt_comp)(pkt);
+                        }
+                }
+        }
+#endif
 }
 
 static void
 mptsas_watchsubr_tgt(mptsas_t *mpt, mptsas_target_t *ptgt, hrtime_t timestamp)
 {
@@ -10669,19 +10872,21 @@
                 mptsas_log(mpt, CE_WARN, "FW Download tce invalid!");
         }
 
         pt->sgl_offset = offsetof(MPI2_FW_DOWNLOAD_REQUEST, SGL) +
             sizeof (*tcsge);
-        if (pt->request_size != pt->sgl_offset)
+        if (pt->request_size != pt->sgl_offset) {
                 NDBG15(("mpi_pre_fw_download(): Incorrect req size, "
                     "0x%x, should be 0x%x, dataoutsz 0x%x",
                     (int)pt->request_size, (int)pt->sgl_offset,
                     (int)pt->dataout_size));
-        if (pt->data_size < sizeof (MPI2_FW_DOWNLOAD_REPLY))
+        }
+        if (pt->data_size < sizeof (MPI2_FW_DOWNLOAD_REPLY)) {
                 NDBG15(("mpi_pre_fw_download(): Incorrect rep size, "
                     "0x%x, should be 0x%x", pt->data_size,
                     (int)sizeof (MPI2_FW_DOWNLOAD_REPLY)));
+        }
 }
 
 /*
  * Prepare the pt for a SAS3 FW_DOWNLOAD request.
  */
@@ -10707,19 +10912,21 @@
         }
         req25->ImageOffset = tcsge->ImageOffset;
         req25->ImageSize = tcsge->ImageSize;
 
         pt->sgl_offset = offsetof(MPI25_FW_DOWNLOAD_REQUEST, SGL);
-        if (pt->request_size != pt->sgl_offset)
+        if (pt->request_size != pt->sgl_offset) {
                 NDBG15(("mpi_pre_fw_25_download(): Incorrect req size, "
                     "0x%x, should be 0x%x, dataoutsz 0x%x",
                     pt->request_size, pt->sgl_offset,
                     pt->dataout_size));
-        if (pt->data_size < sizeof (MPI2_FW_DOWNLOAD_REPLY))
+        }
+        if (pt->data_size < sizeof (MPI2_FW_DOWNLOAD_REPLY)) {
                 NDBG15(("mpi_pre_fw_25_download(): Incorrect rep size, "
                     "0x%x, should be 0x%x", pt->data_size,
                     (int)sizeof (MPI2_FW_UPLOAD_REPLY)));
+        }
 }
 
 /*
  * Prepare the pt for a SAS2 FW_UPLOAD request.
  */
@@ -10751,19 +10958,21 @@
                 mptsas_log(mpt, CE_WARN, "FW Upload tce invalid!");
         }
 
         pt->sgl_offset = offsetof(MPI2_FW_UPLOAD_REQUEST, SGL) +
             sizeof (*tcsge);
-        if (pt->request_size != pt->sgl_offset)
+        if (pt->request_size != pt->sgl_offset) {
                 NDBG15(("mpi_pre_fw_upload(): Incorrect req size, "
                     "0x%x, should be 0x%x, dataoutsz 0x%x",
                     pt->request_size, pt->sgl_offset,
                     pt->dataout_size));
-        if (pt->data_size < sizeof (MPI2_FW_UPLOAD_REPLY))
+        }
+        if (pt->data_size < sizeof (MPI2_FW_UPLOAD_REPLY)) {
                 NDBG15(("mpi_pre_fw_upload(): Incorrect rep size, "
                     "0x%x, should be 0x%x", pt->data_size,
                     (int)sizeof (MPI2_FW_UPLOAD_REPLY)));
+        }
 }
 
 /*
  * Prepare the pt a SAS3 FW_UPLOAD request.
  */
@@ -10789,19 +10998,21 @@
         }
         req25->ImageOffset = tcsge->ImageOffset;
         req25->ImageSize = tcsge->ImageSize;
 
         pt->sgl_offset = offsetof(MPI25_FW_UPLOAD_REQUEST, SGL);
-        if (pt->request_size != pt->sgl_offset)
+        if (pt->request_size != pt->sgl_offset) {
                 NDBG15(("mpi_pre_fw_25_upload(): Incorrect req size, "
                     "0x%x, should be 0x%x, dataoutsz 0x%x",
                     pt->request_size, pt->sgl_offset,
                     pt->dataout_size));
-        if (pt->data_size < sizeof (MPI2_FW_UPLOAD_REPLY))
+        }
+        if (pt->data_size < sizeof (MPI2_FW_UPLOAD_REPLY)) {
                 NDBG15(("mpi_pre_fw_25_upload(): Incorrect rep size, "
                     "0x%x, should be 0x%x", pt->data_size,
                     (int)sizeof (MPI2_FW_UPLOAD_REPLY)));
+        }
 }
 
 /*
  * Prepare the pt for an IOC_FACTS request.
  */
@@ -10809,20 +11020,22 @@
 mpi_pre_ioc_facts(mptsas_t *mpt, mptsas_pt_request_t *pt)
 {
 #ifndef __lock_lint
         _NOTE(ARGUNUSED(mpt))
 #endif
-        if (pt->request_size != sizeof (MPI2_IOC_FACTS_REQUEST))
+        if (pt->request_size != sizeof (MPI2_IOC_FACTS_REQUEST)) {
                 NDBG15(("mpi_pre_ioc_facts(): Incorrect req size, "
                     "0x%x, should be 0x%x, dataoutsz 0x%x",
                     pt->request_size,
                     (int)sizeof (MPI2_IOC_FACTS_REQUEST),
                     pt->dataout_size));
-        if (pt->data_size != sizeof (MPI2_IOC_FACTS_REPLY))
+        }
+        if (pt->data_size != sizeof (MPI2_IOC_FACTS_REPLY)) {
                 NDBG15(("mpi_pre_ioc_facts(): Incorrect rep size, "
                     "0x%x, should be 0x%x", pt->data_size,
                     (int)sizeof (MPI2_IOC_FACTS_REPLY)));
+        }
         pt->sgl_offset = (uint16_t)pt->request_size;
 }
 
 /*
  * Prepare the pt for a PORT_FACTS request.
@@ -10831,20 +11044,22 @@
 mpi_pre_port_facts(mptsas_t *mpt, mptsas_pt_request_t *pt)
 {
 #ifndef __lock_lint
         _NOTE(ARGUNUSED(mpt))
 #endif
-        if (pt->request_size != sizeof (MPI2_PORT_FACTS_REQUEST))
+        if (pt->request_size != sizeof (MPI2_PORT_FACTS_REQUEST)) {
                 NDBG15(("mpi_pre_port_facts(): Incorrect req size, "
                     "0x%x, should be 0x%x, dataoutsz 0x%x",
                     pt->request_size,
                     (int)sizeof (MPI2_PORT_FACTS_REQUEST),
                     pt->dataout_size));
-        if (pt->data_size != sizeof (MPI2_PORT_FACTS_REPLY))
+        }
+        if (pt->data_size != sizeof (MPI2_PORT_FACTS_REPLY)) {
                 NDBG15(("mpi_pre_port_facts(): Incorrect rep size, "
                     "0x%x, should be 0x%x", pt->data_size,
                     (int)sizeof (MPI2_PORT_FACTS_REPLY)));
+        }
         pt->sgl_offset = (uint16_t)pt->request_size;
 }
 
 /*
  * Prepare pt for a SATA_PASSTHROUGH request.
@@ -10854,37 +11069,41 @@
 {
 #ifndef __lock_lint
         _NOTE(ARGUNUSED(mpt))
 #endif
         pt->sgl_offset = offsetof(MPI2_SATA_PASSTHROUGH_REQUEST, SGL);
-        if (pt->request_size != pt->sgl_offset)
+        if (pt->request_size != pt->sgl_offset) {
                 NDBG15(("mpi_pre_sata_passthrough(): Incorrect req size, "
                     "0x%x, should be 0x%x, dataoutsz 0x%x",
                     pt->request_size, pt->sgl_offset,
                     pt->dataout_size));
-        if (pt->data_size != sizeof (MPI2_SATA_PASSTHROUGH_REPLY))
+        }
+        if (pt->data_size != sizeof (MPI2_SATA_PASSTHROUGH_REPLY)) {
                 NDBG15(("mpi_pre_sata_passthrough(): Incorrect rep size, "
                     "0x%x, should be 0x%x", pt->data_size,
                     (int)sizeof (MPI2_SATA_PASSTHROUGH_REPLY)));
+        }
 }
 
 static void
 mpi_pre_smp_passthrough(mptsas_t *mpt, mptsas_pt_request_t *pt)
 {
 #ifndef __lock_lint
         _NOTE(ARGUNUSED(mpt))
 #endif
         pt->sgl_offset = offsetof(MPI2_SMP_PASSTHROUGH_REQUEST, SGL);
-        if (pt->request_size != pt->sgl_offset)
+        if (pt->request_size != pt->sgl_offset) {
                 NDBG15(("mpi_pre_smp_passthrough(): Incorrect req size, "
                     "0x%x, should be 0x%x, dataoutsz 0x%x",
                     pt->request_size, pt->sgl_offset,
                     pt->dataout_size));
-        if (pt->data_size != sizeof (MPI2_SMP_PASSTHROUGH_REPLY))
+        }
+        if (pt->data_size != sizeof (MPI2_SMP_PASSTHROUGH_REPLY)) {
                 NDBG15(("mpi_pre_smp_passthrough(): Incorrect rep size, "
                     "0x%x, should be 0x%x", pt->data_size,
                     (int)sizeof (MPI2_SMP_PASSTHROUGH_REPLY)));
+        }
 }
 
 /*
  * Prepare pt for a CONFIG request.
  */
@@ -10893,18 +11112,20 @@
 {
 #ifndef __lock_lint
         _NOTE(ARGUNUSED(mpt))
 #endif
         pt->sgl_offset = offsetof(MPI2_CONFIG_REQUEST, PageBufferSGE);
-        if (pt->request_size != pt->sgl_offset)
+        if (pt->request_size != pt->sgl_offset) {
                 NDBG15(("mpi_pre_config(): Incorrect req size, 0x%x, "
                     "should be 0x%x, dataoutsz 0x%x", pt->request_size,
                     pt->sgl_offset, pt->dataout_size));
-        if (pt->data_size != sizeof (MPI2_CONFIG_REPLY))
+        }
+        if (pt->data_size != sizeof (MPI2_CONFIG_REPLY)) {
                 NDBG15(("mpi_pre_config(): Incorrect rep size, 0x%x, "
                     "should be 0x%x", pt->data_size,
                     (int)sizeof (MPI2_CONFIG_REPLY)));
+        }
         pt->simple = 1;
 }
 
 /*
  * Prepare pt for a SCSI_IO_REQ request.
@@ -10914,19 +11135,21 @@
 {
 #ifndef __lock_lint
         _NOTE(ARGUNUSED(mpt))
 #endif
         pt->sgl_offset = offsetof(MPI2_SCSI_IO_REQUEST, SGL);
-        if (pt->request_size != pt->sgl_offset)
+        if (pt->request_size != pt->sgl_offset) {
                 NDBG15(("mpi_pre_config(): Incorrect req size, 0x%x, "
                     "should be 0x%x, dataoutsz 0x%x", pt->request_size,
                     pt->sgl_offset,
                     pt->dataout_size));
-        if (pt->data_size != sizeof (MPI2_SCSI_IO_REPLY))
+        }
+        if (pt->data_size != sizeof (MPI2_SCSI_IO_REPLY)) {
                 NDBG15(("mpi_pre_config(): Incorrect rep size, 0x%x, "
                     "should be 0x%x", pt->data_size,
                     (int)sizeof (MPI2_SCSI_IO_REPLY)));
+        }
 }
 
 /*
  * Prepare the mptsas_cmd for a SAS_IO_UNIT_CONTROL request.
  */
@@ -11127,10 +11350,11 @@
         pkt->pkt_cdbp           = (opaque_t)&cmd->cmd_cdb[0];
         pkt->pkt_scbp           = (opaque_t)&cmd->cmd_scb;
         pkt->pkt_ha_private     = (opaque_t)&pt;
         pkt->pkt_flags          = FLAG_HEAD;
         pkt->pkt_time           = timeout;
+        pkt->pkt_start          = gethrtime();
         cmd->cmd_pkt            = pkt;
         cmd->cmd_flags          = CFLAG_CMDIOC | CFLAG_PASSTHRU;
 
         if ((function == MPI2_FUNCTION_SCSI_IO_REQUEST) ||
             (function == MPI2_FUNCTION_RAID_SCSI_IO_PASSTHROUGH)) {
@@ -12579,11 +12803,12 @@
 static int
 led_control(mptsas_t *mpt, intptr_t data, int mode)
 {
         int ret = 0;
         mptsas_led_control_t lc;
-        mptsas_target_t *ptgt;
+        mptsas_enclosure_t *mep;
+        uint16_t slotidx;
 
         if (ddi_copyin((void *)data, &lc, sizeof (lc), mode) != 0) {
                 return (EFAULT);
         }
 
@@ -12598,33 +12823,46 @@
 
         if ((lc.Command == MPTSAS_LEDCTL_FLAG_SET && (mode & FWRITE) == 0) ||
             (lc.Command == MPTSAS_LEDCTL_FLAG_GET && (mode & FREAD) == 0))
                 return (EACCES);
 
-        /* Locate the target we're interrogating... */
+        /* Locate the required enclosure */
         mutex_enter(&mpt->m_mutex);
-        ptgt = refhash_linear_search(mpt->m_targets,
-            mptsas_target_eval_slot, &lc);
-        if (ptgt == NULL) {
-                /* We could not find a target for that enclosure/slot. */
+        mep = mptsas_enc_lookup(mpt, lc.Enclosure);
+        if (mep == NULL) {
                 mutex_exit(&mpt->m_mutex);
                 return (ENOENT);
         }
 
+        if (lc.Slot < mep->me_fslot) {
+                mutex_exit(&mpt->m_mutex);
+                return (ENOENT);
+        }
+
+        /*
+         * Slots on the enclosure are maintained in array where me_fslot is
+         * entry zero. We normalize the requested slot.
+         */
+        slotidx = lc.Slot - mep->me_fslot;
+        if (slotidx >= mep->me_nslots) {
+                mutex_exit(&mpt->m_mutex);
+                return (ENOENT);
+        }
+
         if (lc.Command == MPTSAS_LEDCTL_FLAG_SET) {
                 /* Update our internal LED state. */
-                ptgt->m_led_status &= ~(1 << (lc.Led - 1));
-                ptgt->m_led_status |= lc.LedStatus << (lc.Led - 1);
+                mep->me_slotleds[slotidx] &= ~(1 << (lc.Led - 1));
+                mep->me_slotleds[slotidx] |= lc.LedStatus << (lc.Led - 1);
 
                 /* Flush it to the controller. */
-                ret = mptsas_flush_led_status(mpt, ptgt);
+                ret = mptsas_flush_led_status(mpt, mep, slotidx);
                 mutex_exit(&mpt->m_mutex);
                 return (ret);
         }
 
         /* Return our internal LED state. */
-        lc.LedStatus = (ptgt->m_led_status >> (lc.Led - 1)) & 1;
+        lc.LedStatus = (mep->me_slotleds[slotidx] >> (lc.Led - 1)) & 1;
         mutex_exit(&mpt->m_mutex);
 
         if (ddi_copyout(&lc, (void *)data, sizeof (lc), mode) != 0) {
                 return (EFAULT);
         }
@@ -12762,12 +13000,11 @@
                 if (mpt->m_power_level != PM_LEVEL_D0) {
                         mutex_exit(&mpt->m_mutex);
                         if (pm_raise_power(mpt->m_dip, 0, PM_LEVEL_D0) !=
                             DDI_SUCCESS) {
                                 mptsas_log(mpt, CE_WARN,
-                                    "mptsas%d: mptsas_ioctl: Raise power "
-                                    "request failed.", mpt->m_instance);
+                                    "raise power request failed");
                                 (void) pm_idle_component(mpt->m_dip, 0);
                                 return (ENXIO);
                         }
                 } else {
                         mutex_exit(&mpt->m_mutex);
@@ -12797,25 +13034,10 @@
                                 NDBG14(("mptsas_ioctl led control: tgt %s not "
                                     "found", addr));
                                 ndi_dc_freehdl(dcp);
                                 goto out;
                         }
-                        mutex_enter(&mpt->m_mutex);
-                        if (cmd == DEVCTL_DEVICE_ONLINE) {
-                                ptgt->m_tgt_unconfigured = 0;
-                        } else if (cmd == DEVCTL_DEVICE_OFFLINE) {
-                                ptgt->m_tgt_unconfigured = 1;
-                        }
-                        if (cmd == DEVCTL_DEVICE_OFFLINE) {
-                                ptgt->m_led_status |=
-                                    (1 << (MPTSAS_LEDCTL_LED_OK2RM - 1));
-                        } else {
-                                ptgt->m_led_status &=
-                                    ~(1 << (MPTSAS_LEDCTL_LED_OK2RM - 1));
-                        }
-                        (void) mptsas_flush_led_status(mpt, ptgt);
-                        mutex_exit(&mpt->m_mutex);
                         ndi_dc_freehdl(dcp);
                 }
                 goto out;
         }
         switch (cmd) {
@@ -12998,16 +13220,10 @@
          * so that they can be retried.
          */
         mpt->m_in_reset = TRUE;
 
         /*
-         * Wait until all the allocated sense data buffers for DMA are freed.
-         */
-        while (mpt->m_extreq_sense_refcount > 0)
-                cv_wait(&mpt->m_extreq_sense_refcount_cv, &mpt->m_mutex);
-
-        /*
          * Set all throttles to HOLD
          */
         for (ptgt = refhash_first(mpt->m_targets); ptgt != NULL;
             ptgt = refhash_next(mpt->m_targets, ptgt)) {
                 mptsas_set_throttle(mpt, ptgt, HOLD_THROTTLE);
@@ -13112,38 +13328,45 @@
         if (mptsas_ioc_get_facts(mpt) == DDI_FAILURE) {
                 mptsas_log(mpt, CE_WARN, "mptsas_ioc_get_facts failed");
                 goto fail;
         }
 
+        if (first_time) {
         if (mptsas_alloc_active_slots(mpt, KM_SLEEP)) {
                 goto fail;
         }
         /*
-         * Allocate request message frames, reply free queue, reply descriptor
-         * post queue, and reply message frames using latest IOC facts.
+                 * Allocate request message frames, reply free queue, reply
+                 * descriptor post queue, and reply message frames using
+                 * latest IOC facts.
          */
         if (mptsas_alloc_request_frames(mpt) == DDI_FAILURE) {
-                mptsas_log(mpt, CE_WARN, "mptsas_alloc_request_frames failed");
+                        mptsas_log(mpt, CE_WARN,
+                            "mptsas_alloc_request_frames failed");
                 goto fail;
         }
         if (mptsas_alloc_sense_bufs(mpt) == DDI_FAILURE) {
-                mptsas_log(mpt, CE_WARN, "mptsas_alloc_sense_bufs failed");
+                        mptsas_log(mpt, CE_WARN,
+                            "mptsas_alloc_sense_bufs failed");
                 goto fail;
         }
         if (mptsas_alloc_free_queue(mpt) == DDI_FAILURE) {
-                mptsas_log(mpt, CE_WARN, "mptsas_alloc_free_queue failed!");
+                        mptsas_log(mpt, CE_WARN,
+                            "mptsas_alloc_free_queue failed!");
                 goto fail;
         }
         if (mptsas_alloc_post_queue(mpt) == DDI_FAILURE) {
-                mptsas_log(mpt, CE_WARN, "mptsas_alloc_post_queue failed!");
+                        mptsas_log(mpt, CE_WARN,
+                            "mptsas_alloc_post_queue failed!");
                 goto fail;
         }
         if (mptsas_alloc_reply_frames(mpt) == DDI_FAILURE) {
-                mptsas_log(mpt, CE_WARN, "mptsas_alloc_reply_frames failed!");
+                        mptsas_log(mpt, CE_WARN,
+                            "mptsas_alloc_reply_frames failed!");
                 goto fail;
         }
-
+        }
 mur:
         /*
          * Re-Initialize ioc to operational state
          */
         if (mptsas_ioc_init(mpt) == DDI_FAILURE) {
@@ -13283,16 +13506,16 @@
                  * Check that we haven't exceeded the maximum number of
                  * capabilities and that the pointer is in a valid range.
                  */
                 if (++cap_count > 48) {
                         mptsas_log(mpt, CE_WARN,
-                            "too many device capabilities.\n");
+                            "too many device capabilities");
                         break;
                 }
                 if (caps_ptr < 64) {
                         mptsas_log(mpt, CE_WARN,
-                            "capabilities pointer 0x%x out of range.\n",
+                            "capabilities pointer 0x%x out of range",
                             caps_ptr);
                         break;
                 }
 
                 /*
@@ -13301,12 +13524,11 @@
                  */
                 cap = pci_config_get8(mpt->m_config_handle, caps_ptr);
                 switch (cap) {
                         case PCI_CAP_ID_PM:
                                 mptsas_log(mpt, CE_NOTE,
-                                    "?mptsas%d supports power management.\n",
-                                    mpt->m_instance);
+                                    "power management supported");
                                 mpt->m_options |= MPTSAS_OPT_PM;
 
                                 /* Save PMCSR offset */
                                 mpt->m_pmcsr_offset = caps_ptr + PCI_PMCSR;
                                 break;
@@ -13320,12 +13542,11 @@
                         case PCI_CAP_ID_PCI_E:
                         case PCI_CAP_ID_MSI_X:
                                 break;
                         default:
                                 mptsas_log(mpt, CE_NOTE,
-                                    "?mptsas%d unrecognized capability "
-                                    "0x%x.\n", mpt->m_instance, cap);
+                                    "unrecognized capability 0x%x", cap);
                                 break;
                 }
 
                 /*
                  * Get next capabilities pointer and clear bits 0,1.
@@ -13359,18 +13580,17 @@
                 return (DDI_SUCCESS);
         /*
          * If power management is supported by this chip, create
          * pm-components property for the power management framework
          */
-        (void) sprintf(pmc_name, "NAME=mptsas%d", mpt->m_instance);
+        (void) sprintf(pmc_name, "NAME=mpt_sas%d", mpt->m_instance);
         pmc[0] = pmc_name;
         if (ddi_prop_update_string_array(DDI_DEV_T_NONE, mpt->m_dip,
             "pm-components", pmc, 3) != DDI_PROP_SUCCESS) {
                 mpt->m_options &= ~MPTSAS_OPT_PM;
                 mptsas_log(mpt, CE_WARN,
-                    "mptsas%d: pm-component property creation failed.",
-                    mpt->m_instance);
+                    "pm-component property creation failed");
                 return (DDI_FAILURE);
         }
 
         /*
          * Power on device.
@@ -13377,12 +13597,11 @@
          */
         (void) pm_busy_component(mpt->m_dip, 0);
         pmcsr_stat = pci_config_get16(mpt->m_config_handle,
             mpt->m_pmcsr_offset);
         if ((pmcsr_stat & PCI_PMCSR_STATE_MASK) != PCI_PMCSR_D0) {
-                mptsas_log(mpt, CE_WARN, "mptsas%d: Power up the device",
-                    mpt->m_instance);
+                mptsas_log(mpt, CE_WARN, "power up the device");
                 pci_config_put16(mpt->m_config_handle, mpt->m_pmcsr_offset,
                     PCI_PMCSR_D0);
         }
         if (pm_power_has_changed(mpt->m_dip, 0, PM_LEVEL_D0) != DDI_SUCCESS) {
                 mptsas_log(mpt, CE_WARN, "pm_power_has_changed failed");
@@ -13407,11 +13626,11 @@
         dip = mpt->m_dip;
 
         /* Get supported interrupt types */
         if (ddi_intr_get_supported_types(dip, &intr_types) != DDI_SUCCESS) {
                 mptsas_log(mpt, CE_WARN, "ddi_intr_get_supported_types "
-                    "failed\n");
+                    "failed");
                 return (FALSE);
         }
 
         NDBG6(("ddi_intr_get_supported_types() returned: 0x%x", intr_types));
 
@@ -13461,20 +13680,20 @@
 
         /* Get number of interrupts */
         ret = ddi_intr_get_nintrs(dip, intr_type, &count);
         if ((ret != DDI_SUCCESS) || (count <= 0)) {
                 mptsas_log(mpt, CE_WARN, "ddi_intr_get_nintrs() failed, "
-                    "ret %d count %d\n", ret, count);
+                    "ret %d count %d", ret, count);
 
                 return (DDI_FAILURE);
         }
 
         /* Get number of available interrupts */
         ret = ddi_intr_get_navail(dip, intr_type, &avail);
         if ((ret != DDI_SUCCESS) || (avail == 0)) {
                 mptsas_log(mpt, CE_WARN, "ddi_intr_get_navail() failed, "
-                    "ret %d avail %d\n", ret, avail);
+                    "ret %d avail %d", ret, avail);
 
                 return (DDI_FAILURE);
         }
 
         if (avail < count) {
@@ -13496,19 +13715,19 @@
         /* call ddi_intr_alloc() */
         ret = ddi_intr_alloc(dip, mpt->m_htable, intr_type, 0,
             count, &actual, flag);
 
         if ((ret != DDI_SUCCESS) || (actual == 0)) {
-                mptsas_log(mpt, CE_WARN, "ddi_intr_alloc() failed, ret %d\n",
+                mptsas_log(mpt, CE_WARN, "ddi_intr_alloc() failed, ret %d",
                     ret);
                 kmem_free(mpt->m_htable, mpt->m_intr_size);
                 return (DDI_FAILURE);
         }
 
         /* use interrupt count returned or abort? */
         if (actual < count) {
-                mptsas_log(mpt, CE_NOTE, "Requested: %d, Received: %d\n",
+                mptsas_log(mpt, CE_NOTE, "Requested: %d, Received: %d",
                     count, actual);
         }
 
         mpt->m_intr_cnt = actual;
 
@@ -13515,11 +13734,11 @@
         /*
          * Get priority for first msi, assume remaining are all the same
          */
         if ((ret = ddi_intr_get_pri(mpt->m_htable[0],
             &mpt->m_intr_pri)) != DDI_SUCCESS) {
-                mptsas_log(mpt, CE_WARN, "ddi_intr_get_pri() failed %d\n", ret);
+                mptsas_log(mpt, CE_WARN, "ddi_intr_get_pri() failed %d", ret);
 
                 /* Free already allocated intr */
                 for (i = 0; i < actual; i++) {
                         (void) ddi_intr_free(mpt->m_htable[i]);
                 }
@@ -13529,11 +13748,11 @@
         }
 
         /* Test for high level mutex */
         if (mpt->m_intr_pri >= ddi_intr_get_hilevel_pri()) {
                 mptsas_log(mpt, CE_WARN, "mptsas_add_intrs: "
-                    "Hi level interrupt not supported\n");
+                    "Hi level interrupt not supported");
 
                 /* Free already allocated intr */
                 for (i = 0; i < actual; i++) {
                         (void) ddi_intr_free(mpt->m_htable[i]);
                 }
@@ -13545,11 +13764,11 @@
         /* Call ddi_intr_add_handler() */
         for (i = 0; i < actual; i++) {
                 if ((ret = ddi_intr_add_handler(mpt->m_htable[i], mptsas_intr,
                     (caddr_t)mpt, (caddr_t)(uintptr_t)i)) != DDI_SUCCESS) {
                         mptsas_log(mpt, CE_WARN, "ddi_intr_add_handler() "
-                            "failed %d\n", ret);
+                            "failed %d", ret);
 
                         /* Free already allocated intr */
                         for (i = 0; i < actual; i++) {
                                 (void) ddi_intr_free(mpt->m_htable[i]);
                         }
@@ -13559,11 +13778,11 @@
                 }
         }
 
         if ((ret = ddi_intr_get_cap(mpt->m_htable[0], &mpt->m_intr_cap))
             != DDI_SUCCESS) {
-                mptsas_log(mpt, CE_WARN, "ddi_intr_get_cap() failed %d\n", ret);
+                mptsas_log(mpt, CE_WARN, "ddi_intr_get_cap() failed %d", ret);
 
                 /* Free already allocated intr */
                 for (i = 0; i < actual; i++) {
                         (void) ddi_intr_free(mpt->m_htable[i]);
                 }
@@ -13860,19 +14079,19 @@
 
 inq83_retry:
         rval = mptsas_inquiry(mpt, ptgt, lun, 0x83, inq83,
             inq83_len, NULL, 1);
         if (rval != DDI_SUCCESS) {
-                mptsas_log(mpt, CE_WARN, "!mptsas request inquiry page "
+                mptsas_log(mpt, CE_WARN, "mptsas request inquiry page "
                     "0x83 for target:%x, lun:%x failed!", target, lun);
                 sata_guid = -1;
                 goto out;
         }
         /* According to SAT2, the first descriptor is logic unit name */
         dblk = &inq83[4];
         if ((dblk[1] & 0x30) != 0) {
-                mptsas_log(mpt, CE_WARN, "!Descriptor is not lun associated.");
+                mptsas_log(mpt, CE_WARN, "Descriptor is not lun associated.");
                 goto out;
         }
         pwwn = (uint64_t *)(void *)(&dblk[4]);
         if ((dblk[4] & 0xf0) == 0x50) {
                 sata_guid = BE_64(*pwwn);
@@ -13976,10 +14195,11 @@
         if (pktp == NULL) {
                 goto out;
         }
         bcopy(cdb, pktp->pkt_cdbp, cdblen);
         pktp->pkt_flags = FLAG_NOPARITY;
+        pktp->pkt_time = mptsas_scsi_pkt_time;
         if (scsi_poll(pktp) < 0) {
                 goto out;
         }
         if (((struct scsi_status *)pktp->pkt_scbp)->sts_chk) {
                 goto out;
@@ -14186,10 +14406,13 @@
         case BUS_CONFIG_DRIVER:
         case BUS_CONFIG_ALL:
                 mptsas_config_all(pdip);
                 ret = NDI_SUCCESS;
                 break;
+        default:
+                ret = NDI_FAILURE;
+                break;
         }
 
         if ((ret == NDI_SUCCESS) && bconfig) {
                 ret = ndi_busop_bus_config(pdip, mflags, op,
                     (devnm == NULL) ? arg : devnm, childp, 0);
@@ -14233,10 +14456,18 @@
         mptsas_t                *mpt = DIP2MPT(pdip);
         int             phymask;
         mptsas_target_t *ptgt = NULL;
 
         /*
+         * The phymask exists if the port is active, otherwise
+         * nothing to do.
+         */
+        if (ddi_prop_exists(DDI_DEV_T_ANY, pdip,
+            DDI_PROP_DONTPASS | DDI_PROP_NOTPROM, "phymask") == 0)
+                return (DDI_FAILURE);
+
+        /*
          * Get the physical port associated to the iport
          */
         phymask = ddi_prop_get_int(DDI_DEV_T_ANY, pdip, 0,
             "phymask", 0);
 
@@ -14272,12 +14503,15 @@
                         return (DDI_FAILURE);
                 }
                 return (DDI_SUCCESS);
         }
 
-        if (phymask == 0) {
                 /*
+         * If this is a RAID, configure the volumes
+         */
+        if (mpt->m_num_raid_configs > 0) {
+                /*
                  * Configure IR volume
                  */
                 rval =  mptsas_config_raid(pdip, ptgt->m_devhdl, lundip);
                 return (rval);
         }
@@ -14294,10 +14528,17 @@
         mptsas_t        *mpt = DIP2MPT(pdip);
         mptsas_phymask_t phymask;
         mptsas_target_t *ptgt = NULL;
 
         /*
+         * The phymask exists if the port is active, otherwise
+         * nothing to do.
+         */
+        if (ddi_prop_exists(DDI_DEV_T_ANY, pdip,
+            DDI_PROP_DONTPASS | DDI_PROP_NOTPROM, "phymask") == 0)
+                return (DDI_FAILURE);
+        /*
          * Get the physical port associated to the iport
          */
         phymask = (mptsas_phymask_t)ddi_prop_get_int(DDI_DEV_T_ANY, pdip, 0,
             "phymask", 0);
 
@@ -14473,15 +14714,17 @@
                 if (mptsas_retrieve_lundata(lun_cnt, (uint8_t *)(buffer),
                     &lun_num, &lun_addr_type) != DDI_SUCCESS) {
                         continue;
                 }
                 saved_repluns[lun_cnt] = lun_num;
-                if (cdip = mptsas_find_child_addr(pdip, sas_wwn, lun_num))
+                if ((cdip = mptsas_find_child_addr(pdip, sas_wwn, lun_num)) !=
+                    NULL) {
                         ret = DDI_SUCCESS;
-                else
+                } else {
                         ret = mptsas_probe_lun(pdip, lun_num, &cdip,
                             ptgt);
+                }
                 if ((ret == DDI_SUCCESS) && (cdip != NULL)) {
                         (void) ndi_prop_remove(DDI_DEV_T_NONE, cdip,
                             MPTSAS_DEV_GONE);
                 }
         }
@@ -14608,12 +14851,11 @@
                 }
                 if (find == 0) {
                         /*
                          * The lun has not been there already
                          */
-                        (void) mptsas_offline_lun(pdip, savechild, NULL,
-                            NDI_DEVI_REMOVE);
+                        (void) mptsas_offline_lun(savechild, NULL);
                 }
         }
 
         pip = mdi_get_next_client_path(pdip, NULL);
         while (pip) {
@@ -14645,12 +14887,11 @@
 
                 if (find == 0) {
                         /*
                          * The lun has not been there already
                          */
-                        (void) mptsas_offline_lun(pdip, NULL, savepip,
-                            NDI_DEVI_REMOVE);
+                        (void) mptsas_offline_lun(NULL, savepip);
                 }
         }
 }
 
 /*
@@ -14663,17 +14904,62 @@
         mptsas_enclosure_t *m;
 
         ASSERT(MUTEX_HELD(&mpt->m_mutex));
         m = mptsas_enc_lookup(mpt, mep->me_enchdl);
         if (m != NULL) {
+                uint8_t *ledp;
                 m->me_flags = mep->me_flags;
+
+
+                /*
+                 * If the number of slots and the first slot entry in the
+                 * enclosure has not changed, then we don't need to do anything
+                 * here. Otherwise, we need to allocate a new array for the LED
+                 * status of the slot.
+                 */
+                if (m->me_fslot == mep->me_fslot &&
+                    m->me_nslots == mep->me_nslots)
                 return;
+
+                /*
+                 * If the number of slots or the first slot has changed, it's
+                 * not clear that we're really in a place that we can continue
+                 * to honor the existing flags.
+                 */
+                if (mep->me_nslots > 0) {
+                        ledp = kmem_zalloc(sizeof (uint8_t) * mep->me_nslots,
+                            KM_SLEEP);
+                } else {
+                        ledp = NULL;
         }
 
+                if (m->me_slotleds != NULL) {
+                        kmem_free(m->me_slotleds, sizeof (uint8_t) *
+                            m->me_nslots);
+                }
+                m->me_slotleds = ledp;
+                m->me_fslot = mep->me_fslot;
+                m->me_nslots = mep->me_nslots;
+                return;
+        }
+
         m = kmem_zalloc(sizeof (*m), KM_SLEEP);
         m->me_enchdl = mep->me_enchdl;
         m->me_flags = mep->me_flags;
+        m->me_nslots = mep->me_nslots;
+        m->me_fslot = mep->me_fslot;
+        if (m->me_nslots > 0) {
+                m->me_slotleds = kmem_zalloc(sizeof (uint8_t) * mep->me_nslots,
+                    KM_SLEEP);
+                /*
+                 * It may make sense to optionally flush all of the slots and/or
+                 * read the slot status flag here to synchronize between
+                 * ourselves and the card. So far, that hasn't been needed
+                 * annecdotally when enumerating something new. If we do, we
+                 * should kick that off in a taskq potentially.
+                 */
+        }
         list_insert_tail(&mpt->m_enclosures, m);
 }
 
 static void
 mptsas_update_hashtab(struct mptsas *mpt)
@@ -14800,19 +15086,24 @@
         mptsas_phymask_t phy_mask;
         mptsas_target_t *ptgt = NULL;
         mptsas_smp_t    *psmp;
 
         /*
-         * Get the phymask associated to the iport
+         * The phymask exists if the port is active, otherwise
+         * nothing to do.
          */
+        if (ddi_prop_exists(DDI_DEV_T_ANY, pdip,
+            DDI_PROP_DONTPASS | DDI_PROP_NOTPROM, "phymask") == 0)
+                return;
+
         phymask = ddi_prop_get_int(DDI_DEV_T_ANY, pdip, 0,
             "phymask", 0);
 
         /*
-         * Enumerate RAID volumes here (phymask == 0).
+         * If this is a RAID, enumerate the volumes
          */
-        if (phymask == 0) {
+        if (mpt->m_num_raid_configs > 0) {
                 mptsas_config_all_viport(pdip);
                 return;
         }
 
         mutex_enter(&mpt->m_mutex);
@@ -14895,12 +15186,11 @@
 
                 if (strncmp(addr, name, s) != 0) {
                         continue;
                 }
 
-                tmp_rval = mptsas_offline_lun(pdip, prechild, NULL,
-                    NDI_DEVI_REMOVE);
+                tmp_rval = mptsas_offline_lun(prechild, NULL);
                 if (tmp_rval != DDI_SUCCESS) {
                         rval = DDI_FAILURE;
                         if (ndi_prop_create_boolean(DDI_DEV_T_NONE,
                             prechild, MPTSAS_DEV_GONE) !=
                             DDI_PROP_SUCCESS) {
@@ -14928,12 +15218,11 @@
 
                 if (strncmp(addr, name, s) != 0) {
                         continue;
                 }
 
-                (void) mptsas_offline_lun(pdip, NULL, savepip,
-                    NDI_DEVI_REMOVE);
+                (void) mptsas_offline_lun(NULL, savepip);
                 /*
                  * driver will not invoke mdi_pi_free, so path will not
                  * be freed forever, return DDI_FAILURE.
                  */
                 rval = DDI_FAILURE;
@@ -14940,52 +15229,23 @@
         }
         return (rval);
 }
 
 static int
-mptsas_offline_lun(dev_info_t *pdip, dev_info_t *rdip,
-    mdi_pathinfo_t *rpip, uint_t flags)
+mptsas_offline_lun(dev_info_t *rdip, mdi_pathinfo_t *rpip)
 {
         int             rval = DDI_FAILURE;
-        char            *devname;
-        dev_info_t      *cdip, *parent;
 
         if (rpip != NULL) {
-                parent = scsi_vhci_dip;
-                cdip = mdi_pi_get_client(rpip);
-        } else if (rdip != NULL) {
-                parent = pdip;
-                cdip = rdip;
-        } else {
-                return (DDI_FAILURE);
-        }
-
-        /*
-         * Make sure node is attached otherwise
-         * it won't have related cache nodes to
-         * clean up.  i_ddi_devi_attached is
-         * similiar to i_ddi_node_state(cdip) >=
-         * DS_ATTACHED.
-         */
-        if (i_ddi_devi_attached(cdip)) {
-
-                /* Get full devname */
-                devname = kmem_alloc(MAXNAMELEN + 1, KM_SLEEP);
-                (void) ddi_deviname(cdip, devname);
-                /* Clean cache */
-                (void) devfs_clean(parent, devname + 1,
-                    DV_CLEAN_FORCE);
-                kmem_free(devname, MAXNAMELEN + 1);
-        }
-        if (rpip != NULL) {
                 if (MDI_PI_IS_OFFLINE(rpip)) {
                         rval = DDI_SUCCESS;
                 } else {
                         rval = mdi_pi_offline(rpip, 0);
                 }
-        } else {
-                rval = ndi_devi_offline(cdip, flags);
+        } else if (rdip != NULL) {
+                rval = ndi_devi_offline(rdip,
+                    NDI_DEVFS_CLEAN | NDI_DEVI_REMOVE | NDI_DEVI_GONE);
         }
 
         return (rval);
 }
 
@@ -15013,44 +15273,24 @@
         }
         return (child);
 }
 
 static int
-mptsas_offline_smp(dev_info_t *pdip, mptsas_smp_t *smp_node, uint_t flags)
+mptsas_offline_smp(dev_info_t *pdip, mptsas_smp_t *smp_node)
 {
         int             rval = DDI_FAILURE;
-        char            *devname;
         char            wwn_str[MPTSAS_WWN_STRLEN];
         dev_info_t      *cdip;
 
         (void) sprintf(wwn_str, "%"PRIx64, smp_node->m_addr.mta_wwn);
 
         cdip = mptsas_find_smp_child(pdip, wwn_str);
-
         if (cdip == NULL)
                 return (DDI_SUCCESS);
 
-        /*
-         * Make sure node is attached otherwise
-         * it won't have related cache nodes to
-         * clean up.  i_ddi_devi_attached is
-         * similiar to i_ddi_node_state(cdip) >=
-         * DS_ATTACHED.
-         */
-        if (i_ddi_devi_attached(cdip)) {
+        rval = ndi_devi_offline(cdip, NDI_DEVFS_CLEAN | NDI_DEVI_REMOVE);
 
-                /* Get full devname */
-                devname = kmem_alloc(MAXNAMELEN + 1, KM_SLEEP);
-                (void) ddi_deviname(cdip, devname);
-                /* Clean cache */
-                (void) devfs_clean(pdip, devname + 1,
-                    DV_CLEAN_FORCE);
-                kmem_free(devname, MAXNAMELEN + 1);
-        }
-
-        rval = ndi_devi_offline(cdip, flags);
-
         return (rval);
 }
 
 static dev_info_t *
 mptsas_find_child(dev_info_t *pdip, char *name)
@@ -15181,11 +15421,11 @@
 
         for (i = 0; i < mptsas_inq83_retry_timeout; i++) {
                 rval = mptsas_inquiry(mpt, ptgt, lun, 0x83, inq83,
                     inq83_len1, &inq83_len, 1);
                 if (rval != 0) {
-                        mptsas_log(mpt, CE_WARN, "!mptsas request inquiry page "
+                        mptsas_log(mpt, CE_WARN, "mptsas request inquiry page "
                             "0x83 for target:%x, lun:%x failed!", target, lun);
                         if (mptsas_physical_bind_failed_page_83 != B_FALSE)
                                 goto create_lun;
                         goto out;
                 }
@@ -15208,11 +15448,11 @@
                          */
                         if (guid && (strlen(guid) > MPTSAS_MAX_GUID_LEN)) {
                                 ddi_devid_free_guid(guid);
                                 guid = NULL;
                                 if (mpt->m_mpxio_enable == TRUE) {
-                                        mptsas_log(mpt, CE_NOTE, "!Target:%x, "
+                                        mptsas_log(mpt, CE_NOTE, "Target:%x, "
                                             "lun:%x doesn't have a valid GUID, "
                                             "multipathing for this drive is "
                                             "not enabled", target, lun);
                                 }
                         }
@@ -15230,19 +15470,19 @@
                          */
                         NDBG20(("Not well formed devid, retry..."));
                         delay(1 * drv_usectohz(1000000));
                         continue;
                 } else {
-                        mptsas_log(mpt, CE_WARN, "!Encode devid failed for "
+                        mptsas_log(mpt, CE_WARN, "Encode devid failed for "
                             "path target:%x, lun:%x", target, lun);
                         rval = DDI_FAILURE;
                         goto create_lun;
                 }
         }
 
         if (i == mptsas_inq83_retry_timeout) {
-                mptsas_log(mpt, CE_WARN, "!Repeated page83 requests timeout "
+                mptsas_log(mpt, CE_WARN, "Repeated page83 requests timeout "
                     "for path target:%x, lun:%x", target, lun);
         }
 
         rval = DDI_FAILURE;
 
@@ -15326,15 +15566,10 @@
                                 (void) ddi_prop_free(old_guid);
                                 if ((!MDI_PI_IS_ONLINE(*pip)) &&
                                     (!MDI_PI_IS_STANDBY(*pip)) &&
                                     (ptgt->m_tgt_unconfigured == 0)) {
                                         rval = mdi_pi_online(*pip, 0);
-                                        mutex_enter(&mpt->m_mutex);
-                                        ptgt->m_led_status = 0;
-                                        (void) mptsas_flush_led_status(mpt,
-                                            ptgt);
-                                        mutex_exit(&mpt->m_mutex);
                                 } else {
                                         rval = DDI_SUCCESS;
                                 }
                                 if (rval != DDI_SUCCESS) {
                                         mptsas_log(mpt, CE_WARN, "path:target: "
@@ -15364,11 +15599,12 @@
                                                 *pip = NULL;
                                                 *lun_dip = NULL;
                                                 return (DDI_FAILURE);
                                         }
                                 }
-                                if (mdi_pi_free(*pip, 0) != MDI_SUCCESS) {
+                                if (mdi_pi_free(*pip,
+                                    MDI_CLIENT_FLAGS_NO_EVENT) != MDI_SUCCESS) {
                                         mptsas_log(mpt, CE_WARN, "path:target:"
                                             "%x, lun:%x free failed!", target,
                                             lun);
                                         *pip = NULL;
                                         *lun_dip = NULL;
@@ -15605,22 +15841,16 @@
                         mdi_rtn = MDI_FAILURE;
                         goto virt_create_done;
                 }
                 NDBG20(("new path:%s onlining,", MDI_PI(*pip)->pi_addr));
                 mdi_rtn = mdi_pi_online(*pip, 0);
-                if (mdi_rtn == MDI_SUCCESS) {
-                        mutex_enter(&mpt->m_mutex);
-                        ptgt->m_led_status = 0;
-                        (void) mptsas_flush_led_status(mpt, ptgt);
-                        mutex_exit(&mpt->m_mutex);
-                }
                 if (mdi_rtn == MDI_NOT_SUPPORTED) {
                         mdi_rtn = MDI_FAILURE;
                 }
 virt_create_done:
                 if (*pip && mdi_rtn != MDI_SUCCESS) {
-                        (void) mdi_pi_free(*pip, 0);
+                        (void) mdi_pi_free(*pip, MDI_CLIENT_FLAGS_NO_EVENT);
                         *pip = NULL;
                         *lun_dip = NULL;
                 }
         }
 
@@ -15983,16 +16213,10 @@
                         /*
                          * Try to online the new node
                          */
                         ndi_rtn = ndi_devi_online(*lun_dip, NDI_ONLINE_ATTACH);
                 }
-                if (ndi_rtn == NDI_SUCCESS) {
-                        mutex_enter(&mpt->m_mutex);
-                        ptgt->m_led_status = 0;
-                        (void) mptsas_flush_led_status(mpt, ptgt);
-                        mutex_exit(&mpt->m_mutex);
-                }
 
                 /*
                  * If success set rtn flag, else unwire alloc'd lun
                  */
                 if (ndi_rtn != NDI_SUCCESS) {
@@ -16040,13 +16264,17 @@
         mptsas_smp_t    *psmp = NULL;
         int             rval;
         int             phymask;
 
         /*
-         * Get the physical port associated to the iport
-         * PHYMASK TODO
+         * The phymask exists if the port is active, otherwise
+         * nothing to do.
          */
+        if (ddi_prop_exists(DDI_DEV_T_ANY, pdip,
+            DDI_PROP_DONTPASS | DDI_PROP_NOTPROM, "phymask") == 0)
+                return (DDI_FAILURE);
+
         phymask = ddi_prop_get_int(DDI_DEV_T_ANY, pdip, 0,
             "phymask", 0);
         /*
          * Find the smp node in hash table with specified sas address and
          * physical port
@@ -16675,15 +16903,28 @@
 static dev_info_t *
 mptsas_get_dip_from_dev(dev_t dev, mptsas_phymask_t *phymask)
 {
         dev_info_t      *dip;
         int             prop;
+
         dip = e_ddi_hold_devi_by_dev(dev, 0);
         if (dip == NULL)
                 return (dip);
+
+        /*
+         * The phymask exists if the port is active, otherwise
+         * nothing to do.
+         */
+        if (ddi_prop_exists(DDI_DEV_T_ANY, dip,
+            DDI_PROP_DONTPASS | DDI_PROP_NOTPROM, "phymask") == 0) {
+                ddi_release_devi(dip);
+                return ((dev_info_t *)NULL);
+        }
+
         prop = ddi_prop_get_int(DDI_DEV_T_ANY, dip, 0,
             "phymask", 0);
+
         *phymask = (mptsas_phymask_t)prop;
         ddi_release_devi(dip);
         return (dip);
 }
 static mptsas_target_t *
@@ -16704,85 +16945,67 @@
         }
         return (ptgt);
 }
 
 static int
-mptsas_flush_led_status(mptsas_t *mpt, mptsas_target_t *ptgt)
+mptsas_flush_led_status(mptsas_t *mpt, mptsas_enclosure_t *mep, uint16_t idx)
 {
         uint32_t slotstatus = 0;
 
+        ASSERT3U(idx, <, mep->me_nslots);
+
         /* Build an MPI2 Slot Status based on our view of the world */
-        if (ptgt->m_led_status & (1 << (MPTSAS_LEDCTL_LED_IDENT - 1)))
+        if (mep->me_slotleds[idx] & (1 << (MPTSAS_LEDCTL_LED_IDENT - 1)))
                 slotstatus |= MPI2_SEP_REQ_SLOTSTATUS_IDENTIFY_REQUEST;
-        if (ptgt->m_led_status & (1 << (MPTSAS_LEDCTL_LED_FAIL - 1)))
+        if (mep->me_slotleds[idx] & (1 << (MPTSAS_LEDCTL_LED_FAIL - 1)))
                 slotstatus |= MPI2_SEP_REQ_SLOTSTATUS_PREDICTED_FAULT;
-        if (ptgt->m_led_status & (1 << (MPTSAS_LEDCTL_LED_OK2RM - 1)))
+        if (mep->me_slotleds[idx] & (1 << (MPTSAS_LEDCTL_LED_OK2RM - 1)))
                 slotstatus |= MPI2_SEP_REQ_SLOTSTATUS_REQUEST_REMOVE;
 
         /* Write it to the controller */
         NDBG14(("mptsas_ioctl: set LED status %x for slot %x",
-            slotstatus, ptgt->m_slot_num));
-        return (mptsas_send_sep(mpt, ptgt, &slotstatus,
+            slotstatus, idx + mep->me_fslot));
+        return (mptsas_send_sep(mpt, mep, idx, &slotstatus,
             MPI2_SEP_REQ_ACTION_WRITE_STATUS));
 }
 
 /*
  *  send sep request, use enclosure/slot addressing
  */
 static int
-mptsas_send_sep(mptsas_t *mpt, mptsas_target_t *ptgt,
+mptsas_send_sep(mptsas_t *mpt, mptsas_enclosure_t *mep, uint16_t idx,
     uint32_t *status, uint8_t act)
 {
         Mpi2SepRequest_t        req;
         Mpi2SepReply_t          rep;
         int                     ret;
-        mptsas_enclosure_t      *mep;
         uint16_t                enctype;
+        uint16_t                slot;
 
         ASSERT(mutex_owned(&mpt->m_mutex));
 
         /*
-         * We only support SEP control of directly-attached targets, in which
-         * case the "SEP" we're talking to is a virtual one contained within
-         * the HBA itself.  This is necessary because DA targets typically have
-         * no other mechanism for LED control.  Targets for which a separate
-         * enclosure service processor exists should be controlled via ses(7d)
-         * or sgen(7d).  Furthermore, since such requests can time out, they
-         * should be made in user context rather than in response to
-         * asynchronous fabric changes.
-         *
-         * In addition, we do not support this operation for RAID volumes,
-         * since there is no slot associated with them.
-         */
-        if (!(ptgt->m_deviceinfo & DEVINFO_DIRECT_ATTACHED) ||
-            ptgt->m_addr.mta_phymask == 0) {
-                return (ENOTTY);
-        }
-
-        /*
          * Look through the enclosures and make sure that this enclosure is
          * something that is directly attached device. If we didn't find an
          * enclosure for this device, don't send the ioctl.
          */
-        mep = mptsas_enc_lookup(mpt, ptgt->m_enclosure);
-        if (mep == NULL)
-                return (ENOTTY);
         enctype = mep->me_flags & MPI2_SAS_ENCLS0_FLAGS_MNG_MASK;
         if (enctype != MPI2_SAS_ENCLS0_FLAGS_MNG_IOC_SES &&
             enctype != MPI2_SAS_ENCLS0_FLAGS_MNG_IOC_SGPIO &&
             enctype != MPI2_SAS_ENCLS0_FLAGS_MNG_IOC_GPIO) {
                 return (ENOTTY);
         }
+        slot = idx + mep->me_fslot;
 
         bzero(&req, sizeof (req));
         bzero(&rep, sizeof (rep));
 
         req.Function = MPI2_FUNCTION_SCSI_ENCLOSURE_PROCESSOR;
         req.Action = act;
         req.Flags = MPI2_SEP_REQ_FLAGS_ENCLOSURE_SLOT_ADDRESS;
-        req.EnclosureHandle = LE_16(ptgt->m_enclosure);
-        req.Slot = LE_16(ptgt->m_slot_num);
+        req.EnclosureHandle = LE_16(mep->me_enchdl);
+        req.Slot = LE_16(slot);
         if (act == MPI2_SEP_REQ_ACTION_WRITE_STATUS) {
                 req.SlotStatus = LE_32(*status);
         }
         ret = mptsas_do_passthru(mpt, (uint8_t *)&req, (uint8_t *)&rep, NULL,
             sizeof (req), sizeof (rep), NULL, 0, NULL, 0, 60, FKIOCTL);