Print this page
NEX-14252 zpool remove incorrect message: special device contains metadata
Reviewed by: Roman Strashkin <roman.strashkin@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
NEX-5736 implement autoreplace matching based on FRU slot number
NEX-6200 hot spares are not reactivated after reinserting into enclosure
NEX-9403 need to update FRU for spare and l2cache devices
NEX-9404 remove lofi autoreplace support from syseventd
NEX-9409 hotsparing doesn't work for vdevs without FRU
NEX-9424 zfs`vdev_online() needs better notification about state changes
Portions contributed by: Alek Pinchuk <alek@nexenta.com>
Portions contributed by: Josef 'Jeff' Sipek <josef.sipek@nexenta.com>
Reviewed by: Roman Strashkin <roman.strashkin@nexenta.com>
Reviewed by: Steve Peng <steve.peng@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
NEX-5795 Rename 'wrc' as 'wbc' in the source and in the tech docs
Reviewed by: Alex Aizman <alex.aizman@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
Reviewed by: Alek Pinchuk <alek.pinchuk@nexenta.com>
NEX-5163 backport illumos 6027 EOL zulu (XVR-4000)
Reviewed by: Kevin Crowe <kevin.crowe@nexenta.com>
6027 EOL zulu (XVR-4000)
Reviewed by: Garrett D'Amore <garrett@damore.org>
Reviewed by: Peter Tribble <peter.tribble@gmail.com>
Reviewed by: Richard Lowe <richlowe@richlowe.net>
Approved by: Dan McDonald <danmcd@omniti.com>
NEX-5162 backport illumos 6507 i386 makecontext(3c) needs to 16-byte align the stack
Reviewed by: Kevin Crowe <kevin.crowe@nexenta.com>
6507 i386 makecontext(3c) needs to 16-byte align the stack
Reviewed by: Gordon Ross <gordon.w.ross@gmail.com>
Reviewed by: Robert Mustacchi <rm@joyent.com>
Approved by: Dan McDonald <danmcd@omniti.com>
NEX-5207 attempt to activate spare cores fmd
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Roman Strashkin <roman.strashkin@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
NEX-5019 wrcache activation races vs. 'zpool create -O wrc_mode='
Reviewed by: Alek Pinchuk <alek.pinchuk@nexenta.com>
Reviewed by: Steve Peng <steve.peng@nexenta.com>
NEX-4934 Add capability to remove special vdev
Reviewed by: Alex Aizman <alex.aizman@nexenta.com>
Reviewed by: Alek Pinchuk <alek.pinchuk@nexenta.com>
6328 Fix cstyle errors in zfs codebase (fix studio)
6328 Fix cstyle errors in zfs codebase
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Alex Reece <alex@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed by: Jorgen Lundman <lundman@lundman.net>
Approved by: Robert Mustacchi <rm@joyent.com>
6298 zfs_create_008_neg and zpool_create_023_neg need to be updated for large block support
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: John Kennedy <john.kennedy@delphix.com>
Approved by: Robert Mustacchi <rm@joyent.com>
NEX-4582 update wrc test cases for allow to use write back cache per tree of datasets
Reviewed by: Steve Peng <steve.peng@nexenta.com>
Reviewed by: Alex Aizman <alex.aizman@nexenta.com>
5960 zfs recv should prefetch indirect blocks
5925 zfs receive -o origin=
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
NEX-3984 On-demand TRIM
Reviewed by: Alek Pinchuk <alek@nexenta.com>
Reviewed by: Josef 'Jeff' Sipek <josef.sipek@nexenta.com>
Conflicts:
        usr/src/common/zfs/zpool_prop.c
        usr/src/uts/common/sys/fs/zfs.h
NEX-2325 I/O drop to '0' when FC cable unplugged: path_to_devid() unnecessarily invokes Test Unit Ready which can incorrectly invalidate label causing I/Os failures (fix compile)
NEX-2325 I/O drop to '0' when FC cable unplugged: path_to_devid() unnecessarily invokes Test Unit Ready which can incorrectly invalidate label causing I/Os failures
Reviewed by: Alek Pinchuk <alek.pinchuk@nexenta.com>
Reviewed by: Jean McCormack <jean.mccormack@nexenta.com>
Reviewed by: Steve Peng <steve.peng@nexenta.com>
Reviewed by: Dan Fields <dan.fields@nexenta.com>
NEX-3025 support root pools on EFI labeled disks
Reviewed by: Jean McCormack <jean.mccormack@nexenta.com>
Reviewed by: Josef 'Jeff' Sipek <josef.sipek@nexenta.com>
NEX-2997 Memory leaks in libzfs import implementation
Reviewed by: Dan Fields <dan.fields@nexenta.com>
OS-103 handle CoS descriptor persistent references across vdev operations
OS-102 add man page info and tests for vdev/CoS properties and ZFS meta features
OS-80 support for vdev and CoS properties for the new I/O scheduler
OS-95 lint warning introduced by OS-61
Moved closed ZFS files to open repo, changed Makefiles accordingly
Removed unneeded weak symbols
re 13748 added zpool export -c option
zpool export -c command exports specified pool while keeping its latest
configuration in the cache file for subsequent zpool import -c.
re #12585 rb4049 ZFS++ work port - refactoring to improve separation of open/closed code, bug fixes, performance improvements - open code
re #8279 rb3915 need a mechanism to notify NMS about ZFS config changes (Opened code)
Bug 11205: add missing libzfs_closed_stubs.c to fix opensource-only build.
ZFS plus work: special vdevs, cos, cos/vdev properties

@@ -19,14 +19,14 @@
  * CDDL HEADER END
  */
 
 /*
  * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
- * Copyright (c) 2011, 2017 by Delphix. All rights reserved.
+ * Copyright (c) 2011, 2015 by Delphix. All rights reserved.
  * Copyright (c) 2013, Joyent, Inc. All rights reserved.
- * Copyright 2016 Nexenta Systems, Inc.
  * Copyright 2016 Igor Kozhukhov <ikozhukhov@gmail.com>
+ * Copyright 2017 Nexenta Systems, Inc.
  * Copyright (c) 2017 Datto Inc.
  */
 
 #include <ctype.h>
 #include <errno.h>

@@ -48,11 +48,10 @@
 #include "libzfs_impl.h"
 #include "zfs_comutil.h"
 #include "zfeature_common.h"
 
 static int read_efi_label(nvlist_t *, diskaddr_t *, boolean_t *);
-static boolean_t zpool_vdev_is_interior(const char *name);
 
 #define BACKUP_SLICE    "s2"
 
 typedef struct prop_flags {
         int create:1;   /* Validate property on creation */

@@ -422,11 +421,11 @@
  * correct, and parse any numeric properties (index, boolean, etc) if they are
  * specified as strings.
  */
 static nvlist_t *
 zpool_valid_proplist(libzfs_handle_t *hdl, const char *poolname,
-    nvlist_t *props, uint64_t version, prop_flags_t flags, char *errbuf)
+    nvlist_t *props, uint64_t version, prop_flags_t flags, const char *errbuf)
 {
         nvpair_t *elem;
         nvlist_t *retprops;
         zpool_prop_t prop;
         char *strval;

@@ -443,11 +442,11 @@
         elem = NULL;
         while ((elem = nvlist_next_nvpair(props, elem)) != NULL) {
                 const char *propname = nvpair_name(elem);
 
                 prop = zpool_name_to_prop(propname);
-                if (prop == ZPOOL_PROP_INVAL && zpool_prop_feature(propname)) {
+                if (prop == ZPROP_INVAL && zpool_prop_feature(propname)) {
                         int err;
                         char *fname = strchr(propname, '@') + 1;
 
                         err = zfeature_lookup_name(fname, NULL);
                         if (err != 0) {

@@ -482,11 +481,11 @@
                 }
 
                 /*
                  * Make sure this property is valid and applies to this type.
                  */
-                if (prop == ZPOOL_PROP_INVAL) {
+                if (prop == ZPROP_INVAL) {
                         zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
                             "invalid property '%s'"), propname);
                         (void) zfs_error(hdl, EZFS_BADPROP, errbuf);
                         goto error;
                 }

@@ -722,11 +721,63 @@
                 (void) zpool_props_refresh(zhp);
 
         return (ret);
 }
 
+/*
+ * Set zpool properties nvlist
+ */
 int
+zpool_set_proplist(zpool_handle_t *zhp, nvlist_t *nvl)
+{
+        zfs_cmd_t zc = { 0 };
+        int ret = -1;
+        char errbuf[1024];
+        nvlist_t *realprops;
+        uint64_t version;
+        prop_flags_t flags = { 0 };
+
+        assert(nvl != NULL);
+
+        (void) snprintf(errbuf, sizeof (errbuf),
+            dgettext(TEXT_DOMAIN, "cannot set property for '%s'"),
+            zhp->zpool_name);
+
+        version = zpool_get_prop_int(zhp, ZPOOL_PROP_VERSION, NULL);
+        if ((realprops = zpool_valid_proplist(zhp->zpool_hdl,
+            zhp->zpool_name, nvl, version, flags, errbuf)) == NULL) {
+                nvlist_free(nvl);
+                return (-1);
+        }
+
+        nvlist_free(nvl);
+        nvl = realprops;
+
+        /*
+         * Execute the corresponding ioctl() to set this property.
+         */
+        (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
+
+        if (zcmd_write_src_nvlist(zhp->zpool_hdl, &zc, nvl) != 0) {
+                nvlist_free(nvl);
+                return (-1);
+        }
+
+        ret = zfs_ioctl(zhp->zpool_hdl, ZFS_IOC_POOL_SET_PROPS, &zc);
+
+        zcmd_free_nvlists(&zc);
+        nvlist_free(nvl);
+
+        if (ret)
+                (void) zpool_standard_error(zhp->zpool_hdl, errno, errbuf);
+        else
+                (void) zpool_props_refresh(zhp);
+
+        return (ret);
+}
+
+int
 zpool_expand_proplist(zpool_handle_t *zhp, zprop_list_t **plp)
 {
         libzfs_handle_t *hdl = zhp->zpool_hdl;
         zprop_list_t *entry;
         char buf[ZFS_MAXPROPLEN];

@@ -1115,10 +1166,11 @@
         zfs_cmd_t zc = { 0 };
         nvlist_t *zc_fsprops = NULL;
         nvlist_t *zc_props = NULL;
         char msg[1024];
         int ret = -1;
+        boolean_t wbc_mode_prop_exists = B_FALSE;
 
         (void) snprintf(msg, sizeof (msg), dgettext(TEXT_DOMAIN,
             "cannot create '%s'"), pool);
 
         if (!zpool_name_valid(hdl, B_FALSE, pool))

@@ -1146,10 +1198,16 @@
 
                 if ((zc_fsprops = zfs_valid_proplist(hdl, ZFS_TYPE_FILESYSTEM,
                     fsprops, zoned, NULL, NULL, msg)) == NULL) {
                         goto create_failed;
                 }
+
+                if (nvlist_exists(zc_fsprops,
+                    zfs_prop_to_name(ZFS_PROP_WBC_MODE))) {
+                        wbc_mode_prop_exists = B_TRUE;
+                }
+
                 if (!zc_props &&
                     (nvlist_alloc(&zc_props, NV_UNIQUE_NAME, 0) != 0)) {
                         goto create_failed;
                 }
                 if (nvlist_add_nvlist(zc_props,

@@ -1222,10 +1280,17 @@
                 case ENOTBLK:
                         zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
                             "cache device must be a disk or disk slice"));
                         return (zfs_error(hdl, EZFS_BADDEV, msg));
 
+                case EALREADY:
+                        if (wbc_mode_prop_exists) {
+                                zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
+                                    "WBC is already in the OFF state"));
+                                return (zfs_error(hdl, EZFS_WBCALREADY, msg));
+                        }
+
                 default:
                         return (zpool_standard_error(hdl, errno, msg));
                 }
         }
 

@@ -1331,17 +1396,10 @@
                         zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
                             "one or more vdevs refer to the same device"));
                         (void) zfs_error(hdl, EZFS_BADDEV, msg);
                         break;
 
-                case EINVAL:
-                        zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
-                            "invalid config; a pool with removing/removed "
-                            "vdevs does not support adding raidz vdevs"));
-                        (void) zfs_error(hdl, EZFS_BADDEV, msg);
-                        break;
-
                 case EOVERFLOW:
                         /*
                          * This occurrs when one of the devices is below
                          * SPA_MINDEVSIZE.  Unfortunately, we can't detect which
                          * device was the problem device since there's no

@@ -1396,11 +1454,11 @@
  * Exports the pool from the system.  The caller must ensure that there are no
  * mounted datasets in the pool.
  */
 static int
 zpool_export_common(zpool_handle_t *zhp, boolean_t force, boolean_t hardforce,
-    const char *log_str)
+    boolean_t saveconfig, const char *log_str)
 {
         zfs_cmd_t zc = { 0 };
         char msg[1024];
 
         (void) snprintf(msg, sizeof (msg), dgettext(TEXT_DOMAIN,

@@ -1407,10 +1465,11 @@
             "cannot export '%s'"), zhp->zpool_name);
 
         (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
         zc.zc_cookie = force;
         zc.zc_guid = hardforce;
+        zc.zc_obj = saveconfig;
         zc.zc_history = (uint64_t)(uintptr_t)log_str;
 
         if (zfs_ioctl(zhp->zpool_hdl, ZFS_IOC_POOL_EXPORT, &zc) != 0) {
                 switch (errno) {
                 case EXDEV:

@@ -1429,19 +1488,21 @@
 
         return (0);
 }
 
 int
-zpool_export(zpool_handle_t *zhp, boolean_t force, const char *log_str)
+zpool_export(zpool_handle_t *zhp, boolean_t force, boolean_t saveconfig,
+    const char *log_str)
 {
-        return (zpool_export_common(zhp, force, B_FALSE, log_str));
+        return (zpool_export_common(zhp, force, B_FALSE, saveconfig, log_str));
 }
 
 int
-zpool_export_force(zpool_handle_t *zhp, const char *log_str)
+zpool_export_force(zpool_handle_t *zhp, boolean_t saveconfig,
+    const char *log_str)
 {
-        return (zpool_export_common(zhp, B_TRUE, B_TRUE, log_str));
+        return (zpool_export_common(zhp, B_TRUE, B_TRUE, saveconfig, log_str));
 }
 
 static void
 zpool_rewind_exclaim(libzfs_handle_t *hdl, const char *name, boolean_t dryrun,
     nvlist_t *config)

@@ -1806,13 +1867,12 @@
                         if (nv && nvlist_lookup_nvlist(nv,
                             ZPOOL_CONFIG_LOAD_INFO, &nvinfo) == 0 &&
                             nvlist_lookup_nvlist(nvinfo,
                             ZPOOL_CONFIG_MISSING_DEVICES, &missing) == 0) {
                                 (void) printf(dgettext(TEXT_DOMAIN,
-                                    "The devices below are missing or "
-                                    "corrupted, use '-m' to import the pool "
-                                    "anyway:\n"));
+                                    "The devices below are missing, use "
+                                    "'-m' to import the pool anyway:\n"));
                                 print_vdev_tree(hdl, NULL, missing, 2);
                                 (void) printf("\n");
                         }
                         (void) zpool_standard_error(hdl, error, desc);
                         break;

@@ -1925,10 +1985,30 @@
                 return (zpool_standard_error(hdl, err, msg));
         }
 }
 
 /*
+ * Trim the pool.
+ */
+int
+zpool_trim(zpool_handle_t *zhp, boolean_t start, uint64_t rate)
+{
+        zfs_cmd_t zc = { 0 };
+        char msg[1024];
+        libzfs_handle_t *hdl = zhp->zpool_hdl;
+        trim_cmd_info_t tci = { .tci_start = start, .tci_rate = rate };
+
+        (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
+        zc.zc_cookie = (uintptr_t)&tci;
+
+        if (zfs_ioctl(hdl, ZFS_IOC_POOL_TRIM, &zc) == 0)
+                return (0);
+
+        return (zpool_standard_error(hdl, errno, msg));
+}
+
+/*
  * This provides a very minimal check whether a given string is likely a
  * c#t#d# style string.  Users of this are expected to do their own
  * verification of the s# part.
  */
 #define CTD_CHECK(str)  (str && str[0] == 'c' && isdigit(str[1]))

@@ -1965,16 +2045,16 @@
  * 'avail_spare' is set to TRUE if the provided guid refers to an AVAIL
  * spare; but FALSE if its an INUSE spare.
  */
 static nvlist_t *
 vdev_to_nvlist_iter(nvlist_t *nv, nvlist_t *search, boolean_t *avail_spare,
-    boolean_t *l2cache, boolean_t *log)
+    boolean_t *l2cache, boolean_t *log, boolean_t *special)
 {
         uint_t c, children;
         nvlist_t **child;
         nvlist_t *ret;
-        uint64_t is_log;
+        uint64_t is_log, is_special;
         char *srchkey;
         nvpair_t *pair = nvlist_next_nvpair(search, NULL);
 
         /* Nothing to look for */
         if (search == NULL || pair == NULL)

@@ -2079,11 +2159,14 @@
                         if (strncmp(val, type, strlen(val)) != 0) {
                                 free(type);
                                 break;
                         }
 
-                        verify(zpool_vdev_is_interior(type));
+                        verify(strncmp(type, VDEV_TYPE_RAIDZ,
+                            strlen(VDEV_TYPE_RAIDZ)) == 0 ||
+                            strncmp(type, VDEV_TYPE_MIRROR,
+                            strlen(VDEV_TYPE_MIRROR)) == 0);
                         verify(nvlist_lookup_uint64(nv, ZPOOL_CONFIG_ID,
                             &id) == 0);
 
                         errno = 0;
                         vdev_id = strtoull(idx, &end, 10);

@@ -2115,11 +2198,11 @@
             &child, &children) != 0)
                 return (NULL);
 
         for (c = 0; c < children; c++) {
                 if ((ret = vdev_to_nvlist_iter(child[c], search,
-                    avail_spare, l2cache, NULL)) != NULL) {
+                    avail_spare, l2cache, NULL, NULL)) != NULL) {
                         /*
                          * The 'is_log' value is only set for the toplevel
                          * vdev, not the leaf vdevs.  So we always lookup the
                          * log device from the root of the vdev tree (where
                          * 'log' is non-NULL).

@@ -2128,19 +2211,27 @@
                             nvlist_lookup_uint64(child[c],
                             ZPOOL_CONFIG_IS_LOG, &is_log) == 0 &&
                             is_log) {
                                 *log = B_TRUE;
                         }
+
+                        if (special != NULL &&
+                            nvlist_lookup_uint64(child[c],
+                            ZPOOL_CONFIG_IS_SPECIAL, &is_special) == 0 &&
+                            is_special) {
+                                *special = B_TRUE;
+                        }
+
                         return (ret);
                 }
         }
 
         if (nvlist_lookup_nvlist_array(nv, ZPOOL_CONFIG_SPARES,
             &child, &children) == 0) {
                 for (c = 0; c < children; c++) {
                         if ((ret = vdev_to_nvlist_iter(child[c], search,
-                            avail_spare, l2cache, NULL)) != NULL) {
+                            avail_spare, l2cache, NULL, NULL)) != NULL) {
                                 *avail_spare = B_TRUE;
                                 return (ret);
                         }
                 }
         }

@@ -2147,11 +2238,11 @@
 
         if (nvlist_lookup_nvlist_array(nv, ZPOOL_CONFIG_L2CACHE,
             &child, &children) == 0) {
                 for (c = 0; c < children; c++) {
                         if ((ret = vdev_to_nvlist_iter(child[c], search,
-                            avail_spare, l2cache, NULL)) != NULL) {
+                            avail_spare, l2cache, NULL, NULL)) != NULL) {
                                 *l2cache = B_TRUE;
                                 return (ret);
                         }
                 }
         }

@@ -2177,34 +2268,32 @@
 
         *avail_spare = B_FALSE;
         *l2cache = B_FALSE;
         if (log != NULL)
                 *log = B_FALSE;
-        ret = vdev_to_nvlist_iter(nvroot, search, avail_spare, l2cache, log);
+        ret = vdev_to_nvlist_iter(nvroot, search, avail_spare, l2cache, log,
+            NULL);
         nvlist_free(search);
 
         return (ret);
 }
 
 /*
  * Determine if we have an "interior" top-level vdev (i.e mirror/raidz).
  */
-static boolean_t
+boolean_t
 zpool_vdev_is_interior(const char *name)
 {
         if (strncmp(name, VDEV_TYPE_RAIDZ, strlen(VDEV_TYPE_RAIDZ)) == 0 ||
-            strncmp(name, VDEV_TYPE_SPARE, strlen(VDEV_TYPE_SPARE)) == 0 ||
-            strncmp(name,
-            VDEV_TYPE_REPLACING, strlen(VDEV_TYPE_REPLACING)) == 0 ||
             strncmp(name, VDEV_TYPE_MIRROR, strlen(VDEV_TYPE_MIRROR)) == 0)
                 return (B_TRUE);
         return (B_FALSE);
 }
 
 nvlist_t *
 zpool_find_vdev(zpool_handle_t *zhp, const char *path, boolean_t *avail_spare,
-    boolean_t *l2cache, boolean_t *log)
+    boolean_t *l2cache, boolean_t *log, boolean_t *special)
 {
         char buf[MAXPATHLEN];
         char *end;
         nvlist_t *nvroot, *search, *ret;
         uint64_t guid;

@@ -2229,11 +2318,16 @@
 
         *avail_spare = B_FALSE;
         *l2cache = B_FALSE;
         if (log != NULL)
                 *log = B_FALSE;
-        ret = vdev_to_nvlist_iter(nvroot, search, avail_spare, l2cache, log);
+
+        if (special != NULL)
+                *special = B_FALSE;
+
+        ret = vdev_to_nvlist_iter(nvroot, search, avail_spare, l2cache, log,
+            special);
         nvlist_free(search);
 
         return (ret);
 }
 

@@ -2447,18 +2541,15 @@
                     dgettext(TEXT_DOMAIN, "cannot online %s"), path);
         }
 
         (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
         if ((tgt = zpool_find_vdev(zhp, path, &avail_spare, &l2cache,
-            &islog)) == NULL)
+            &islog, NULL)) == NULL)
                 return (zfs_error(hdl, EZFS_NODEVICE, msg));
 
         verify(nvlist_lookup_uint64(tgt, ZPOOL_CONFIG_GUID, &zc.zc_guid) == 0);
 
-        if (avail_spare)
-                return (zfs_error(hdl, EZFS_ISSPARE, msg));
-
         if ((flags & ZFS_ONLINE_EXPAND ||
             zpool_get_prop_int(zhp, ZPOOL_PROP_AUTOEXPAND, NULL)) &&
             nvlist_lookup_string(tgt, ZPOOL_CONFIG_PATH, &pathname) == 0) {
                 uint64_t wholedisk = 0;
 

@@ -2512,11 +2603,11 @@
         (void) snprintf(msg, sizeof (msg),
             dgettext(TEXT_DOMAIN, "cannot offline %s"), path);
 
         (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
         if ((tgt = zpool_find_vdev(zhp, path, &avail_spare, &l2cache,
-            NULL)) == NULL)
+            NULL, NULL)) == NULL)
                 return (zfs_error(hdl, EZFS_NODEVICE, msg));
 
         verify(nvlist_lookup_uint64(tgt, ZPOOL_CONFIG_GUID, &zc.zc_guid) == 0);
 
         if (avail_spare)

@@ -2662,11 +2753,11 @@
                 (void) snprintf(msg, sizeof (msg), dgettext(TEXT_DOMAIN,
                     "cannot attach %s to %s"), new_disk, old_disk);
 
         (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
         if ((tgt = zpool_find_vdev(zhp, old_disk, &avail_spare, &l2cache,
-            &islog)) == NULL)
+            &islog, NULL)) == 0)
                 return (zfs_error(hdl, EZFS_NODEVICE, msg));
 
         if (avail_spare)
                 return (zfs_error(hdl, EZFS_ISSPARE, msg));
 

@@ -2694,11 +2785,11 @@
          * replace it with another hot spare.
          */
         if (replacing &&
             nvlist_lookup_uint64(tgt, ZPOOL_CONFIG_IS_SPARE, &val) == 0 &&
             (zpool_find_vdev(zhp, newname, &avail_spare, &l2cache,
-            NULL) == NULL || !avail_spare) &&
+            NULL, NULL) == NULL || !avail_spare) &&
             is_replacing_spare(config_root, tgt, 1)) {
                 zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
                     "can only be replaced by another hot spare"));
                 free(newname);
                 return (zfs_error(hdl, EZFS_BADTARGET, msg));

@@ -2761,12 +2852,11 @@
                     "new device must be a single disk"));
                 (void) zfs_error(hdl, EZFS_INVALCONFIG, msg);
                 break;
 
         case EBUSY:
-                zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "%s is busy, "
-                    "or pool has removing/removed vdevs"),
+                zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "%s is busy"),
                     new_disk);
                 (void) zfs_error(hdl, EZFS_BADDEV, msg);
                 break;
 
         case EOVERFLOW:

@@ -2816,11 +2906,11 @@
         (void) snprintf(msg, sizeof (msg),
             dgettext(TEXT_DOMAIN, "cannot detach %s"), path);
 
         (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
         if ((tgt = zpool_find_vdev(zhp, path, &avail_spare, &l2cache,
-            NULL)) == NULL)
+            NULL, NULL)) == 0)
                 return (zfs_error(hdl, EZFS_NODEVICE, msg));
 
         if (avail_spare)
                 return (zfs_error(hdl, EZFS_ISSPARE, msg));
 

@@ -3105,119 +3195,65 @@
 
         return (0);
 }
 
 /*
- * Remove the given device.
+ * Remove the given device.  Currently, this is supported only for hot spares
+ * and level 2 cache devices.
  */
 int
 zpool_vdev_remove(zpool_handle_t *zhp, const char *path)
 {
         zfs_cmd_t zc = { 0 };
         char msg[1024];
         nvlist_t *tgt;
-        boolean_t avail_spare, l2cache, islog;
+        boolean_t avail_spare, l2cache, islog, isspecial;
         libzfs_handle_t *hdl = zhp->zpool_hdl;
         uint64_t version;
 
         (void) snprintf(msg, sizeof (msg),
             dgettext(TEXT_DOMAIN, "cannot remove %s"), path);
 
         (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
         if ((tgt = zpool_find_vdev(zhp, path, &avail_spare, &l2cache,
-            &islog)) == NULL)
+            &islog, &isspecial)) == 0)
                 return (zfs_error(hdl, EZFS_NODEVICE, msg));
+        /*
+         * XXX - this should just go away.
+         */
+        if (!avail_spare && !l2cache && !islog && !isspecial) {
+                zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
+                    "only inactive hot spares, cache, top-level, "
+                    "log, or special devices can be removed"));
+                return (zfs_error(hdl, EZFS_NODEVICE, msg));
+        }
 
         version = zpool_get_prop_int(zhp, ZPOOL_PROP_VERSION, NULL);
         if (islog && version < SPA_VERSION_HOLES) {
                 zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
-                    "pool must be upgraded to support log removal"));
+                    "pool must be upgrade to support log removal"));
                 return (zfs_error(hdl, EZFS_BADVERSION, msg));
         }
 
-        if (!islog && !avail_spare && !l2cache && zpool_is_bootable(zhp)) {
-                zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
-                    "root pool can not have removed devices, "
-                    "because GRUB does not understand them"));
-                return (zfs_error(hdl, EINVAL, msg));
-        }
+        verify(nvlist_lookup_uint64(tgt, ZPOOL_CONFIG_GUID, &zc.zc_guid) == 0);
 
-        zc.zc_guid = fnvlist_lookup_uint64(tgt, ZPOOL_CONFIG_GUID);
-
         if (zfs_ioctl(hdl, ZFS_IOC_VDEV_REMOVE, &zc) == 0)
                 return (0);
 
-        switch (errno) {
-
-        case EINVAL:
+        if (isspecial && errno == EEXIST) {
                 zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
-                    "invalid config; all top-level vdevs must "
-                    "have the same sector size and not be raidz."));
-                (void) zfs_error(hdl, EZFS_INVALCONFIG, msg);
-                break;
-
-        case EBUSY:
+                    "special device contains metadata"));
+                return (zfs_error(hdl, EZFS_POOL_NOTSUP, msg));
+        } else if (isspecial && errno == EBUSY) {
                 zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
-                    "Pool busy; removal may already be in progress"));
-                (void) zfs_error(hdl, EZFS_BUSY, msg);
-                break;
-
-        default:
-                (void) zpool_standard_error(hdl, errno, msg);
+                    "wbc feature flag is active"));
+                return (zfs_error(hdl, EZFS_WBCCHILD, msg));
         }
-        return (-1);
-}
 
-int
-zpool_vdev_remove_cancel(zpool_handle_t *zhp)
-{
-        zfs_cmd_t zc = { 0 };
-        char msg[1024];
-        libzfs_handle_t *hdl = zhp->zpool_hdl;
-
-        (void) snprintf(msg, sizeof (msg),
-            dgettext(TEXT_DOMAIN, "cannot cancel removal"));
-
-        (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
-        zc.zc_cookie = 1;
-
-        if (zfs_ioctl(hdl, ZFS_IOC_VDEV_REMOVE, &zc) == 0)
-                return (0);
-
         return (zpool_standard_error(hdl, errno, msg));
 }
 
-int
-zpool_vdev_indirect_size(zpool_handle_t *zhp, const char *path,
-    uint64_t *sizep)
-{
-        char msg[1024];
-        nvlist_t *tgt;
-        boolean_t avail_spare, l2cache, islog;
-        libzfs_handle_t *hdl = zhp->zpool_hdl;
-
-        (void) snprintf(msg, sizeof (msg),
-            dgettext(TEXT_DOMAIN, "cannot determine indirect size of %s"),
-            path);
-
-        if ((tgt = zpool_find_vdev(zhp, path, &avail_spare, &l2cache,
-            &islog)) == NULL)
-                return (zfs_error(hdl, EZFS_NODEVICE, msg));
-
-        if (avail_spare || l2cache || islog) {
-                *sizep = 0;
-                return (0);
-        }
-
-        if (nvlist_lookup_uint64(tgt, ZPOOL_CONFIG_INDIRECT_SIZE, sizep) != 0) {
-                zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
-                    "indirect size not available"));
-                return (zfs_error(hdl, EINVAL, msg));
-        }
-        return (0);
-}
-
 /*
  * Clear the errors for the pool, or the particular device if specified.
  */
 int
 zpool_clear(zpool_handle_t *zhp, const char *path, nvlist_t *rewindnvl)

@@ -3241,11 +3277,11 @@
                     zhp->zpool_name);
 
         (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
         if (path) {
                 if ((tgt = zpool_find_vdev(zhp, path, &avail_spare,
-                    &l2cache, NULL)) == NULL)
+                    &l2cache, NULL, NULL)) == 0)
                         return (zfs_error(hdl, EZFS_NODEVICE, msg));
 
                 /*
                  * Don't allow error clearing for hot spares.  Do allow
                  * error clearing for l2cache devices.

@@ -3397,11 +3433,11 @@
 {
         int fd;
         ddi_devid_t devid;
         char *minor, *ret;
 
-        if ((fd = open(path, O_RDONLY)) < 0)
+        if ((fd = open(path, O_RDONLY | O_NDELAY)) < 0)
                 return (NULL);
 
         minor = NULL;
         ret = NULL;
         if (devid_get(fd, &devid) == 0) {

@@ -3714,10 +3750,25 @@
         nvlist_free(args);
         zcmd_free_nvlists(&zc);
         return (err);
 }
 
+int
+zpool_stage_history(libzfs_handle_t *hdl, const char *history_str)
+{
+        if (history_str == NULL)
+                return (EINVAL);
+
+        if (hdl->libzfs_log_str != NULL)
+                free(hdl->libzfs_log_str);
+
+        if ((hdl->libzfs_log_str = strdup(history_str)) == NULL)
+                return (no_memory(hdl));
+
+        return (0);
+}
+
 /*
  * Perform ioctl to get some command history of a pool.
  *
  * 'buf' is the buffer to fill up to 'len' bytes.  'off' is the
  * logical offset of the history buffer to start reading from.

@@ -4284,5 +4335,749 @@
         if (zhp)
                 zpool_close(zhp);
         libzfs_fini(hdl);
         return (ret);
 }
+
+/*
+ * Vdev props
+ */
+static int
+vdev_get_guid(zpool_handle_t *zhp, const char *path, uint64_t *guid)
+{
+        nvlist_t *nvl;
+        boolean_t avail_spare, l2cache, islog;
+
+        if ((nvl = zpool_find_vdev(zhp, path, &avail_spare, &l2cache,
+            &islog, NULL)) == NULL)
+                return (1);
+        verify(nvlist_lookup_uint64(nvl, ZPOOL_CONFIG_GUID, guid) == 0);
+        return (0);
+}
+
+/*
+ * Given an nvlist of vdev properties to be set, validate that they are
+ * correct, and parse any numeric properties (index, boolean, etc) if they are
+ * specified as strings.
+ */
+/*ARGSUSED*/
+static nvlist_t *
+vdev_valid_proplist(libzfs_handle_t *hdl, nvlist_t *props,
+    uint64_t version, prop_flags_t flags, const char *errbuf)
+{
+        nvpair_t *elem;
+        nvlist_t *retprops;
+        vdev_prop_t prop;
+        char *strval;
+        uint64_t intval;
+
+        if (nvlist_alloc(&retprops, NV_UNIQUE_NAME, 0) != 0) {
+                (void) no_memory(hdl);
+                return (NULL);
+        }
+
+        elem = NULL;
+        while ((elem = nvlist_next_nvpair(props, elem)) != NULL) {
+                const char *propname = nvpair_name(elem);
+
+                /*
+                 * Make sure this property is valid and applies to this type.
+                 */
+                if ((prop = vdev_name_to_prop(propname)) == ZPROP_INVAL) {
+                        zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
+                            "invalid vdev property '%s'"), propname);
+                        (void) zfs_error(hdl, EZFS_BADPROP, errbuf);
+                        goto error;
+                }
+
+                if (vdev_prop_readonly(prop)) {
+                        zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "'%s' "
+                            "is readonly"), propname);
+                        (void) zfs_error(hdl, EZFS_PROPREADONLY, errbuf);
+                        goto error;
+                }
+
+                if (zprop_parse_value(hdl, elem, prop, ZFS_TYPE_VDEV, retprops,
+                    &strval, &intval, errbuf) != 0)
+                        goto error;
+
+                /*
+                 * Perform additional checking for specific properties.
+                 */
+                switch (prop) {
+                case VDEV_PROP_PREFERRED_READ:
+                        if (intval > 100) {
+                                zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
+                                    "'%s' must be from 0 to 100"), propname);
+                                (void) zfs_error(hdl, EZFS_BADPROP,
+                                    errbuf);
+                                goto error;
+                        }
+                        break;
+                case VDEV_PROP_READ_MINACTIVE:
+                case VDEV_PROP_READ_MAXACTIVE:
+                case VDEV_PROP_AREAD_MINACTIVE:
+                case VDEV_PROP_AREAD_MAXACTIVE:
+                case VDEV_PROP_WRITE_MINACTIVE:
+                case VDEV_PROP_WRITE_MAXACTIVE:
+                case VDEV_PROP_AWRITE_MINACTIVE:
+                case VDEV_PROP_AWRITE_MAXACTIVE:
+                case VDEV_PROP_SCRUB_MINACTIVE:
+                case VDEV_PROP_SCRUB_MAXACTIVE:
+                        if (intval > 1000) {
+                                zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
+                                    "'%s' must be from 0 to 1000"), propname);
+                                (void) zfs_error(hdl, EZFS_BADPROP,
+                                    errbuf);
+                                goto error;
+                        }
+                        break;
+                default:
+                        break;
+                }
+        }
+
+        return (retprops);
+error:
+        nvlist_free(retprops);
+        return (NULL);
+}
+
+static int
+vdev_get_all_props(zpool_handle_t *zhp, uint64_t vdev_guid, nvlist_t **nvp)
+{
+        zfs_cmd_t zc = { 0 };
+        libzfs_handle_t *hdl = zhp->zpool_hdl;
+
+        (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
+
+        if (zcmd_alloc_dst_nvlist(hdl, &zc, 0) != 0)
+                return (-1);
+
+        zc.zc_guid = vdev_guid;
+        while (ioctl(hdl->libzfs_fd, ZFS_IOC_VDEV_GET_PROPS, &zc) != 0) {
+                if (errno == ENOMEM) {
+                        if (zcmd_expand_dst_nvlist(hdl, &zc) != 0) {
+                                zcmd_free_nvlists(&zc);
+                                return (-1);
+                        }
+                } else {
+                        zcmd_free_nvlists(&zc);
+                        (void) zfs_error(hdl, EZFS_BADDEV,
+                            "failed to get vdev properties");
+                        return (-1);
+                }
+        }
+
+        if (zcmd_read_dst_nvlist(hdl, &zc, nvp) != 0) {
+                zcmd_free_nvlists(&zc);
+                return (-1);
+        }
+
+        zcmd_free_nvlists(&zc);
+
+        return (0);
+}
+
+int
+vdev_get_prop(zpool_handle_t *zhp,  const char *vdev, vdev_prop_t prop,
+    char *buf, size_t len)
+{
+        uint64_t vdev_guid;
+        uint64_t intval;
+        const char *strval;
+        nvlist_t *nvl;
+        char errbuf[1024];
+
+        (void) snprintf(errbuf, sizeof (errbuf),
+            dgettext(TEXT_DOMAIN, "cannot get property for '%s'"),
+            zhp->zpool_name);
+
+        if (zpool_get_state(zhp) == POOL_STATE_UNAVAIL)
+                return (-1);
+
+        if (vdev_get_guid(zhp, vdev, &vdev_guid) != 0) {
+                (void) zfs_error(zhp->zpool_hdl, EZFS_BADDEV, errbuf);
+                return (-1);
+        }
+
+        if (vdev_get_all_props(zhp, vdev_guid, &nvl) != 0)
+                return (-1);
+
+        switch (vdev_prop_get_type(prop)) {
+        case PROP_TYPE_STRING:
+                if (nvlist_lookup_string(nvl, vdev_prop_to_name(prop),
+                    (char **)&strval) != 0)
+                        if ((strval = (char *)vdev_prop_default_string(prop))
+                            == NULL)
+                                strval = "-";
+
+                (void) strlcpy(buf, strval, len);
+                break;
+
+        case PROP_TYPE_NUMBER:
+                if (nvlist_lookup_uint64(nvl, vdev_prop_to_name(prop),
+                    &intval) != 0)
+                        intval = vdev_prop_default_numeric(prop);
+                (void) snprintf(buf, len, "%llu", (u_longlong_t)intval);
+                break;
+
+        case PROP_TYPE_INDEX:
+                if (nvlist_lookup_uint64(nvl, vdev_prop_to_name(prop),
+                    &intval) != 0)
+                        intval = vdev_prop_default_numeric(prop);
+                if (vdev_prop_index_to_string(prop, intval, &strval) != 0) {
+                        (void) zfs_error(zhp->zpool_hdl, EZFS_BADPROP, errbuf);
+                        nvlist_free(nvl);
+                        return (-1);
+                }
+                (void) strlcpy(buf, strval, len);
+                break;
+
+        default:
+                abort();
+        }
+
+        nvlist_free(nvl);
+
+        return (0);
+}
+
+/*
+ * Set vdev property : propname=propval.
+ */
+int
+vdev_set_prop(zpool_handle_t *zhp, const char *vdev,
+    const char *propname, const char *propval)
+{
+        zfs_cmd_t zc = { 0 };
+        int ret = -1;
+        char errbuf[1024];
+        nvlist_t *nvl = NULL;
+        nvlist_t *realprops;
+        uint64_t version;
+        uint64_t guid;
+        prop_flags_t flags = { 0 };
+
+        (void) snprintf(errbuf, sizeof (errbuf),
+            dgettext(TEXT_DOMAIN, "cannot set property for '%s'"),
+            zhp->zpool_name);
+
+        if (nvlist_alloc(&nvl, NV_UNIQUE_NAME, 0) != 0)
+                return (no_memory(zhp->zpool_hdl));
+
+        if (nvlist_add_string(nvl, propname, propval) != 0) {
+                nvlist_free(nvl);
+                return (no_memory(zhp->zpool_hdl));
+        }
+
+        if (vdev_get_guid(zhp, vdev, &guid) != 0) {
+                (void) zfs_error(zhp->zpool_hdl, EZFS_BADDEV, errbuf);
+                return (-1);
+        }
+
+        version = zpool_get_prop_int(zhp, ZPOOL_PROP_VERSION, NULL);
+        if ((realprops = vdev_valid_proplist(zhp->zpool_hdl, nvl,
+            version, flags, errbuf)) == NULL) {
+                nvlist_free(nvl);
+                return (-1);
+        }
+
+        nvlist_free(nvl);
+        nvl = realprops;
+
+        /*
+         * Execute the corresponding ioctl() to set this property.
+         */
+        (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
+        zc.zc_guid = guid;
+
+        if (zcmd_write_src_nvlist(zhp->zpool_hdl, &zc, nvl) != 0) {
+                nvlist_free(nvl);
+                return (-1);
+        }
+
+        ret = zfs_ioctl(zhp->zpool_hdl, ZFS_IOC_VDEV_SET_PROPS, &zc);
+
+        zcmd_free_nvlists(&zc);
+        nvlist_free(nvl);
+
+        if (ret)
+                (void) zpool_vprop_standard_error(zhp->zpool_hdl,
+                    errno, errbuf);
+
+        return (ret);
+}
+
+/*
+ * Set vdev properties nvlist
+ */
+int
+vdev_set_proplist(zpool_handle_t *zhp, const char *vdev, nvlist_t *nvl)
+{
+        zfs_cmd_t zc = { 0 };
+        int ret = -1;
+        char errbuf[1024];
+        nvlist_t *realprops;
+        uint64_t version;
+        uint64_t guid;
+        prop_flags_t flags = { 0 };
+        libzfs_handle_t *hdl = zhp->zpool_hdl;
+
+        assert(nvl != NULL);
+
+        (void) snprintf(errbuf, sizeof (errbuf),
+            dgettext(TEXT_DOMAIN, "cannot set property for '%s'"),
+            zhp->zpool_name);
+
+        if (vdev_get_guid(zhp, vdev, &guid) != 0) {
+                (void) zfs_error(hdl, EZFS_BADDEV, errbuf);
+                return (-1);
+        }
+
+        version = zpool_get_prop_int(zhp, ZPOOL_PROP_VERSION, NULL);
+        if ((realprops = vdev_valid_proplist(zhp->zpool_hdl, nvl,
+            version, flags, errbuf)) == NULL) {
+                nvlist_free(nvl);
+                return (-1);
+        }
+
+        nvlist_free(nvl);
+        nvl = realprops;
+
+        /*
+         * Execute the corresponding ioctl() to set this property.
+         */
+        (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
+        zc.zc_guid = guid;
+
+        if (zcmd_write_src_nvlist(hdl, &zc, nvl) != 0) {
+                nvlist_free(nvl);
+                return (-1);
+        }
+
+        ret = zfs_ioctl(hdl, ZFS_IOC_VDEV_SET_PROPS, &zc);
+
+        zcmd_free_nvlists(&zc);
+        nvlist_free(nvl);
+
+        if (ret)
+                (void) zpool_vprop_standard_error(hdl, errno, errbuf);
+
+        return (ret);
+}
+
+typedef struct vdev_cb {
+        zpool_handle_t          *vcb_zhp;
+        char                    *vcb_name;
+        uint64_t                vcb_guid;
+        boolean_t               vcb_is_leaf;
+        boolean_t               vcb_success;
+        void                    *vcb_data;
+} vdev_cb_t;
+
+typedef int (*vdev_callback_t)(vdev_cb_t *);
+
+/*
+ * Invoke dev_callback for all vdevs in the pool
+ */
+int
+for_each_vdev(zpool_handle_t *zhp, nvlist_t *root,
+    vdev_callback_t vdev_callback, vdev_cb_t *cb)
+{
+        int ret = 0;
+        nvlist_t *config, *nvroot, **child, **child_list;
+        uint32_t children, child_children, c;
+        libzfs_handle_t *hdl = zhp->zpool_hdl;
+
+        if (!root) {
+                config = zpool_get_config(zhp, NULL);
+                verify(nvlist_lookup_nvlist(config, ZPOOL_CONFIG_VDEV_TREE,
+                    &nvroot) == 0);
+        } else {
+                nvroot = root;
+        }
+
+        if (nvlist_lookup_nvlist_array(nvroot, ZPOOL_CONFIG_CHILDREN,
+            &child, &children) != 0) {
+                (void) fprintf(stderr, gettext("Failed to get the vdev "
+                    "children details from the root nvlist\n"));
+                return (-1);
+        }
+
+        cb->vcb_zhp = zhp;
+        for (c = 0; c < children; c++) {
+                cb->vcb_is_leaf = B_TRUE;
+                if (nvlist_lookup_nvlist_array(child[c], ZPOOL_CONFIG_CHILDREN,
+                    &child_list, &child_children) == 0 &&
+                    child_children > 0) {
+                        ret = for_each_vdev(zhp, child[c], vdev_callback, cb);
+                        if (ret)
+                                return (ret);
+                        cb->vcb_is_leaf = B_FALSE;
+                }
+
+                cb->vcb_name = zpool_vdev_name(hdl, NULL, child[c], B_TRUE);
+                verify(nvlist_lookup_uint64(child[c], ZPOOL_CONFIG_GUID,
+                    &cb->vcb_guid) == 0);
+
+                ret |= vdev_callback(cb);
+                free(cb->vcb_name);
+                if (ret)
+                        return (ret);
+        }
+
+        return (0);
+}
+
+int
+get_vdev_guid_callback(vdev_cb_t *cb)
+{
+        if (!cb->vcb_is_leaf)
+                return (0);
+
+        if (strncmp(cb->vcb_name, (char *)cb->vcb_data,
+            strlen(cb->vcb_name)) == 0) {
+                cb->vcb_success = B_TRUE;
+        }
+
+        return (0);
+}
+
+/*
+ * Class of Storage (COS)
+ */
+/*ARGSUSED*/
+static nvlist_t *
+cos_valid_proplist(libzfs_handle_t *hdl, nvlist_t *props,
+    uint64_t version, prop_flags_t flags, const char *errbuf)
+{
+        nvpair_t *elem;
+        nvlist_t *retprops;
+        cos_prop_t prop;
+        char *strval;
+        uint64_t intval;
+
+        if (nvlist_alloc(&retprops, NV_UNIQUE_NAME, 0) != 0) {
+                (void) no_memory(hdl);
+                return (NULL);
+        }
+
+        elem = NULL;
+        while ((elem = nvlist_next_nvpair(props, elem)) != NULL) {
+                const char *propname = nvpair_name(elem);
+
+                /*
+                 * Make sure this property is valid and applies to this type.
+                 */
+                if ((prop = cos_name_to_prop(propname)) == ZPROP_INVAL) {
+                        zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
+                            "invalid cos property '%s'"), propname);
+                        (void) zfs_error(hdl, EZFS_BADPROP, errbuf);
+                        goto error;
+                }
+
+                if (cos_prop_readonly(prop)) {
+                        zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, "'%s' "
+                            "is readonly"), propname);
+                        (void) zfs_error(hdl, EZFS_PROPREADONLY, errbuf);
+                        goto error;
+                }
+
+                if (zprop_parse_value(hdl, elem, prop, ZFS_TYPE_COS, retprops,
+                    &strval, &intval, errbuf) != 0)
+                        goto error;
+
+                /*
+                 * Perform additional checking for specific properties.
+                 */
+                switch (prop) {
+                case COS_PROP_PREFERRED_READ:
+                        if (intval > 100) {
+                                zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
+                                    "'%s' must be from 0 to 100"), propname);
+                                (void) zfs_error(hdl, EZFS_BADPROP,
+                                    errbuf);
+                                goto error;
+                        }
+                        break;
+                case COS_PROP_READ_MINACTIVE:
+                case COS_PROP_AREAD_MINACTIVE:
+                case COS_PROP_WRITE_MINACTIVE:
+                case COS_PROP_AWRITE_MINACTIVE:
+                case COS_PROP_SCRUB_MINACTIVE:
+                case COS_PROP_READ_MAXACTIVE:
+                case COS_PROP_AREAD_MAXACTIVE:
+                case COS_PROP_WRITE_MAXACTIVE:
+                case COS_PROP_AWRITE_MAXACTIVE:
+                case COS_PROP_SCRUB_MAXACTIVE:
+                        if (intval > 1000) {
+                                zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
+                                    "'%s' must be from 0 to 1000"), propname);
+                                (void) zfs_error(hdl, EZFS_BADPROP,
+                                    errbuf);
+                                goto error;
+                        }
+                        break;
+                default:
+                        break;
+                }
+        }
+
+        return (retprops);
+error:
+        nvlist_free(retprops);
+        return (NULL);
+}
+
+int
+cos_alloc(zpool_handle_t *zhp, char *cosname, nvlist_t *nvl)
+{
+        zfs_cmd_t zc = { 0 };
+        libzfs_handle_t *hdl = zhp->zpool_hdl;
+        char errbuf[1024];
+        int error = 0;
+
+        (void) snprintf(errbuf, sizeof (errbuf),
+            dgettext(TEXT_DOMAIN, "cannot allocate CoS descriptor for '%s'"),
+            zhp->zpool_name);
+
+        (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
+        (void) strlcpy(zc.zc_string, cosname, sizeof (zc.zc_string));
+
+        if (zcmd_write_src_nvlist(zhp->zpool_hdl, &zc, nvl) != 0) {
+                nvlist_free(nvl);
+                return (-1);
+        }
+
+        error = ioctl(hdl->libzfs_fd, ZFS_IOC_COS_ALLOC, &zc);
+
+        if (error)
+                (void) zpool_vprop_standard_error(hdl, errno, errbuf);
+
+        return (error);
+}
+
+int
+cos_free(zpool_handle_t *zhp, char *cosname, uint64_t guid, boolean_t force)
+{
+        zfs_cmd_t zc = { 0 };
+        libzfs_handle_t *hdl = zhp->zpool_hdl;
+        char errbuf[1024];
+        int error = 0;
+
+        (void) snprintf(errbuf, sizeof (errbuf),
+            dgettext(TEXT_DOMAIN, "cannot free CoS descriptor for '%s'"),
+            zhp->zpool_name);
+
+        (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
+        (void) strlcpy(zc.zc_string, cosname, sizeof (zc.zc_string));
+        zc.zc_guid = guid;
+
+        zc.zc_cookie = (uint64_t)force;
+
+        error = ioctl(hdl->libzfs_fd, ZFS_IOC_COS_FREE, &zc);
+
+        if (error)
+                (void) zpool_vprop_standard_error(hdl, errno, errbuf);
+
+        return (error);
+}
+
+int
+cos_list(zpool_handle_t *zhp, nvlist_t **nvp)
+{
+        zfs_cmd_t zc = { 0 };
+        libzfs_handle_t *hdl = zhp->zpool_hdl;
+
+        (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
+
+        if (zcmd_alloc_dst_nvlist(hdl, &zc, 0) != 0)
+                return (-1);
+
+        while (ioctl(hdl->libzfs_fd, ZFS_IOC_COS_LIST, &zc) != 0) {
+                if (errno == ENOMEM) {
+                        if (zcmd_expand_dst_nvlist(hdl, &zc) != 0) {
+                                zcmd_free_nvlists(&zc);
+                                return (-1);
+                        }
+                } else {
+                        zcmd_free_nvlists(&zc);
+                        return (-1);
+                }
+        }
+
+        if (zcmd_read_dst_nvlist(hdl, &zc, nvp) != 0) {
+                zcmd_free_nvlists(&zc);
+                return (-1);
+        }
+
+        zcmd_free_nvlists(&zc);
+
+        return (0);
+}
+
+int
+cos_get_all_props(zpool_handle_t *zhp, const char *cos, nvlist_t **nvp)
+{
+        uint64_t cos_id;
+        zfs_cmd_t zc = { 0 };
+        libzfs_handle_t *hdl = zhp->zpool_hdl;
+        char *endp;
+        char errbuf[1024];
+
+        (void) snprintf(errbuf, sizeof (errbuf),
+            dgettext(TEXT_DOMAIN, "cannot set property for '%s'"),
+            zhp->zpool_name);
+
+        (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
+
+        if (zcmd_alloc_dst_nvlist(hdl, &zc, 0) != 0)
+                return (-1);
+
+        cos_id = strtoll(cos, &endp, 10);
+        zc.zc_guid = cos_id;
+        if (cos_id == 0 && cos == endp)
+                (void) strlcpy(zc.zc_string, cos, sizeof (zc.zc_string));
+        else
+                zc.zc_string[0] = '\0';
+
+        while (ioctl(hdl->libzfs_fd, ZFS_IOC_COS_GET_PROPS, &zc) != 0) {
+                if (errno == ENOMEM) {
+                        if (zcmd_expand_dst_nvlist(hdl, &zc) != 0) {
+                                zcmd_free_nvlists(&zc);
+                                return (-1);
+                        }
+                } else {
+                        zcmd_free_nvlists(&zc);
+                        (void) zpool_standard_error(hdl, errno, errbuf);
+                        return (-1);
+                }
+        }
+
+        if (zcmd_read_dst_nvlist(hdl, &zc, nvp) != 0) {
+                zcmd_free_nvlists(&zc);
+                return (-1);
+        }
+
+        zcmd_free_nvlists(&zc);
+
+        return (0);
+}
+
+int
+cos_get_prop(zpool_handle_t *zhp,  const char *cos, cos_prop_t prop,
+    char *buf, size_t len, nvlist_t **nvp)
+{
+        uint64_t intval;
+        const char *strval;
+        nvlist_t *nvl;
+        char errbuf[1024];
+
+        (void) snprintf(errbuf, sizeof (errbuf),
+            dgettext(TEXT_DOMAIN, "cannot set property for '%s'"),
+            zhp->zpool_name);
+
+        assert(nvp != NULL);
+
+        if (zpool_get_state(zhp) == POOL_STATE_UNAVAIL)
+                return (-1);
+
+        if (*nvp == NULL && cos_get_all_props(zhp, cos, nvp) != 0)
+                return (-1);
+        nvl = *nvp;
+
+        switch (cos_prop_get_type(prop)) {
+        case PROP_TYPE_STRING:
+                if (nvlist_lookup_string(nvl, cos_prop_to_name(prop),
+                    (char **)&strval) != 0)
+                        if ((strval = (char *)cos_prop_default_string(prop))
+                            == NULL)
+                                strval = "-";
+
+                (void) strlcpy(buf, strval, len);
+                break;
+
+        case PROP_TYPE_NUMBER:
+                if (nvlist_lookup_uint64(nvl, cos_prop_to_name(prop),
+                    &intval) != 0)
+                        intval = cos_prop_default_numeric(prop);
+                (void) snprintf(buf, len, "%llu", (u_longlong_t)intval);
+                break;
+
+        case PROP_TYPE_INDEX:
+                if (nvlist_lookup_uint64(nvl, cos_prop_to_name(prop),
+                    &intval) != 0)
+                        intval = cos_prop_default_numeric(prop);
+                if (cos_prop_index_to_string(prop, intval, &strval) != 0) {
+                        (void) zfs_error(zhp->zpool_hdl, EZFS_BADPROP, errbuf);
+                        return (-1);
+                }
+                (void) strlcpy(buf, strval, len);
+                break;
+
+        default:
+                abort();
+        }
+
+        return (0);
+}
+
+/*
+ * Set cos properties nvlist
+ */
+int
+cos_set_proplist(zpool_handle_t *zhp, const char *cos, nvlist_t *nvl)
+{
+        zfs_cmd_t zc = { 0 };
+        int ret = -1;
+        char errbuf[1024];
+        char *endp;
+        nvlist_t *realprops;
+        uint64_t version;
+        uint64_t cos_id;
+        prop_flags_t flags = { 0 };
+
+        assert(nvl != NULL);
+
+        (void) snprintf(errbuf, sizeof (errbuf),
+            dgettext(TEXT_DOMAIN, "cannot set property for '%s'"),
+            zhp->zpool_name);
+
+        version = zpool_get_prop_int(zhp, ZPOOL_PROP_VERSION, NULL);
+        if ((realprops = cos_valid_proplist(zhp->zpool_hdl, nvl,
+            version, flags, errbuf)) == NULL) {
+                nvlist_free(nvl);
+                return (-1);
+        }
+
+        nvlist_free(nvl);
+        nvl = realprops;
+
+        /*
+         * Execute the corresponding ioctl() to set this property.
+         */
+        (void) strlcpy(zc.zc_name, zhp->zpool_name, sizeof (zc.zc_name));
+
+        cos_id = strtoll(cos, &endp, 10);
+        zc.zc_guid = cos_id;
+        if (cos_id == 0 && cos == endp)
+                (void) strlcpy(zc.zc_string, cos, sizeof (zc.zc_string));
+        else
+                zc.zc_string[0] = '\0';
+
+        if (zcmd_write_src_nvlist(zhp->zpool_hdl, &zc, nvl) != 0) {
+                nvlist_free(nvl);
+                return (-1);
+        }
+
+        ret = zfs_ioctl(zhp->zpool_hdl, ZFS_IOC_COS_SET_PROPS, &zc);
+
+        zcmd_free_nvlists(&zc);
+        nvlist_free(nvl);
+
+        if (ret)
+                (void) zpool_vprop_standard_error_fmt(zhp->zpool_hdl,
+                    errno, errbuf);
+
+        return (ret);
+}