Print this page
NEX-19178 Changing the NFS export path makes the SMB share offline
Reviewed by: Evan Layton <evan.layton@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Matt Barden <matt.barden@nexenta.com>
Revert "NEX-19178 Changing the ZFS mountpoint property of a dataset takes the SMB share offline"
This reverts commit 35bb44b3cdee0719ce685304ca801335d5cc234e.
NEX-19178 Changing the ZFS mountpoint property of a dataset takes the SMB share offline
Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Matt Barden <matt.barden@nexenta.com>
NEX-15279 support NFS server in zone
NEX-15520 online NFS shares cause zoneadm halt to hang in nfs_export_zone_fini
Portions contributed by: Dan Kruchinin dan.kruchinin@nexenta.com
Portions contributed by: Stepan Zastupov stepan.zastupov@gmail.com
Reviewed by: Joyce McIntosh <joyce.mcintosh@nexenta.com>
Reviewed by: Rob Gittins <rob.gittins@nexenta.com>
Reviewed by: Gordon Ross <gordon.ross@nexenta.com>
NEX-16219 pool import performance regression due to repeated libshare initialization
Reviewd by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
NEX-15937 zpool import performance degradation in filesystem sharing
Reviewed by: Evan Layton <evan.layton@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
NEX-15937 zpool import performance degradation in filesystem sharing
Reviewed by: Evan Layton <evan.layton@nexenta.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
NEX-6586 cleanup gcc warnings in libzfs_mount.c
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
Reviewed by: Alek Pinchuk <alek.pinchuk@nexenta.com>
2605 want to resume interrupted zfs send
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed by: Xin Li <delphij@freebsd.org>
Reviewed by: Arne Jansen <sensille@gmx.net>
Approved by: Dan McDonald <danmcd@omniti.com>
6280 libzfs: unshare_one() could fail with EZFS_SHARENFSFAILED
Reviewed by: Toomas Soome <tsoome@me.com>
Reviewed by: Dan McDonald <danmcd@omniti.com>
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Approved by: Gordon Ross <gwr@nexenta.com>
NEX-1557 Parallel mount during HA Failover sometimes doesn't share the dataset, causes shares to go offline
SUP-647 Long failover times dominated by zpool import times trigger client-side errors
re #13594 rb4488 Lint complaints fix
re #10054 #13409 rb4387 added parallel unmount for zpool export

Split Close
Expand all
Collapse all
          --- old/usr/src/lib/libzfs/common/libzfs_mount.c
          +++ new/usr/src/lib/libzfs/common/libzfs_mount.c
↓ open down ↓ 12 lines elided ↑ open up ↑
  13   13   * When distributing Covered Code, include this CDDL HEADER in each
  14   14   * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
  15   15   * If applicable, add the following below this CDDL HEADER, with the
  16   16   * fields enclosed by brackets "[]" replaced with your own identifying
  17   17   * information: Portions Copyright [yyyy] [name of copyright owner]
  18   18   *
  19   19   * CDDL HEADER END
  20   20   */
  21   21  
  22   22  /*
  23      - * Copyright 2015 Nexenta Systems, Inc.  All rights reserved.
       23 + * Copyright 2018 Nexenta Systems, Inc.  All rights reserved.
  24   24   * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
       25 + */
       26 +
       27 +/*
       28 + * Copyright 2019 Nexenta Systems, Inc.
  25   29   * Copyright (c) 2014, 2016 by Delphix. All rights reserved.
  26   30   * Copyright 2016 Igor Kozhukhov <ikozhukhov@gmail.com>
  27   31   * Copyright 2017 Joyent, Inc.
  28   32   * Copyright 2017 RackTop Systems.
  29   33   */
  30   34  
  31   35  /*
  32   36   * Routines to manage ZFS mounts.  We separate all the nasty routines that have
  33   37   * to deal with the OS.  The following functions are the main entry points --
  34   38   * they are used by mount and unmount and when changing a filesystem's
↓ open down ↓ 19 lines elided ↑ open up ↑
  54   58   *      zfs_unshare_smb()
  55   59   *      zfs_unshareall_nfs()
  56   60   *      zfs_unshareall_smb()
  57   61   *      zfs_unshareall()
  58   62   *      zfs_unshareall_bypath()
  59   63   *
  60   64   * The following functions are available for pool consumers, and will
  61   65   * mount/unmount and share/unshare all datasets within pool:
  62   66   *
  63   67   *      zpool_enable_datasets()
       68 + *      zpool_enable_datasets_ex()
  64   69   *      zpool_disable_datasets()
       70 + *      zpool_disable_datasets_ex()
  65   71   */
  66   72  
  67   73  #include <dirent.h>
  68   74  #include <dlfcn.h>
  69   75  #include <errno.h>
  70   76  #include <fcntl.h>
  71   77  #include <libgen.h>
  72   78  #include <libintl.h>
  73   79  #include <stdio.h>
  74   80  #include <stdlib.h>
  75   81  #include <strings.h>
  76   82  #include <unistd.h>
  77   83  #include <zone.h>
  78   84  #include <sys/mntent.h>
  79   85  #include <sys/mount.h>
  80   86  #include <sys/stat.h>
       87 +#include <thread_pool.h>
  81   88  #include <sys/statvfs.h>
  82   89  
  83   90  #include <libzfs.h>
  84   91  
  85   92  #include "libzfs_impl.h"
  86   93  
  87   94  #include <libshare.h>
  88   95  #include <sys/systeminfo.h>
  89   96  #define MAXISALEN       257     /* based on sysinfo(2) man page */
  90   97  
↓ open down ↓ 323 lines elided ↑ open up ↑
 414  421              mntopts);
 415  422          return (0);
 416  423  }
 417  424  
 418  425  /*
 419  426   * Unmount a single filesystem.
 420  427   */
 421  428  static int
 422  429  unmount_one(libzfs_handle_t *hdl, const char *mountpoint, int flags)
 423  430  {
 424      -        if (umount2(mountpoint, flags) != 0) {
      431 +        int ret = umount2(mountpoint, flags);
      432 +        if (ret != 0) {
 425  433                  zfs_error_aux(hdl, strerror(errno));
 426  434                  return (zfs_error_fmt(hdl, EZFS_UMOUNTFAILED,
 427  435                      dgettext(TEXT_DOMAIN, "cannot unmount '%s'"),
 428  436                      mountpoint));
 429  437          }
 430  438  
 431  439          return (0);
 432  440  }
 433  441  
 434  442  /*
↓ open down ↓ 129 lines elided ↑ open up ↑
 564  572  }
 565  573  
 566  574  /*
 567  575   * Make sure things will work if libshare isn't installed by using
 568  576   * wrapper functions that check to see that the pointers to functions
 569  577   * initialized in _zfs_init_libshare() are actually present.
 570  578   */
 571  579  
 572  580  static sa_handle_t (*_sa_init)(int);
 573  581  static sa_handle_t (*_sa_init_arg)(int, void *);
      582 +static int (*_sa_service)(sa_handle_t);
 574  583  static void (*_sa_fini)(sa_handle_t);
 575  584  static sa_share_t (*_sa_find_share)(sa_handle_t, char *);
 576  585  static int (*_sa_enable_share)(sa_share_t, char *);
 577  586  static int (*_sa_disable_share)(sa_share_t, char *);
 578  587  static char *(*_sa_errorstr)(int);
 579  588  static int (*_sa_parse_legacy_options)(sa_group_t, char *, char *);
 580  589  static boolean_t (*_sa_needs_refresh)(sa_handle_t *);
 581  590  static libzfs_handle_t *(*_sa_get_zfs_handle)(sa_handle_t);
 582      -static int (*_sa_zfs_process_share)(sa_handle_t, sa_group_t, sa_share_t,
 583      -    char *, char *, zprop_source_t, char *, char *, char *);
      591 +static int (* _sa_get_zfs_share)(sa_handle_t, char *, zfs_handle_t *);
 584  592  static void (*_sa_update_sharetab_ts)(sa_handle_t);
 585  593  
 586  594  /*
 587  595   * _zfs_init_libshare()
 588  596   *
 589  597   * Find the libshare.so.1 entry points that we use here and save the
 590  598   * values to be used later. This is triggered by the runtime loader.
 591  599   * Make sure the correct ISA version is loaded.
 592  600   */
 593  601  
↓ open down ↓ 12 lines elided ↑ open up ↑
 606  614          isa[0] = '\0';
 607  615  #endif
 608  616          (void) snprintf(path, MAXPATHLEN,
 609  617              "/usr/lib/%s/libshare.so.1", isa);
 610  618  
 611  619          if ((libshare = dlopen(path, RTLD_LAZY | RTLD_GLOBAL)) != NULL) {
 612  620                  _sa_init = (sa_handle_t (*)(int))dlsym(libshare, "sa_init");
 613  621                  _sa_init_arg = (sa_handle_t (*)(int, void *))dlsym(libshare,
 614  622                      "sa_init_arg");
 615  623                  _sa_fini = (void (*)(sa_handle_t))dlsym(libshare, "sa_fini");
      624 +                _sa_service = (int (*)(sa_handle_t))dlsym(libshare,
      625 +                    "sa_service");
 616  626                  _sa_find_share = (sa_share_t (*)(sa_handle_t, char *))
 617  627                      dlsym(libshare, "sa_find_share");
 618  628                  _sa_enable_share = (int (*)(sa_share_t, char *))dlsym(libshare,
 619  629                      "sa_enable_share");
 620  630                  _sa_disable_share = (int (*)(sa_share_t, char *))dlsym(libshare,
 621  631                      "sa_disable_share");
 622  632                  _sa_errorstr = (char *(*)(int))dlsym(libshare, "sa_errorstr");
 623  633                  _sa_parse_legacy_options = (int (*)(sa_group_t, char *, char *))
 624  634                      dlsym(libshare, "sa_parse_legacy_options");
 625  635                  _sa_needs_refresh = (boolean_t (*)(sa_handle_t *))
 626  636                      dlsym(libshare, "sa_needs_refresh");
 627  637                  _sa_get_zfs_handle = (libzfs_handle_t *(*)(sa_handle_t))
 628  638                      dlsym(libshare, "sa_get_zfs_handle");
 629      -                _sa_zfs_process_share = (int (*)(sa_handle_t, sa_group_t,
 630      -                    sa_share_t, char *, char *, zprop_source_t, char *,
 631      -                    char *, char *))dlsym(libshare, "sa_zfs_process_share");
      639 +                _sa_get_zfs_share = (int (*)(sa_handle_t, char *,
      640 +                    zfs_handle_t *)) dlsym(libshare, "sa_get_zfs_share");
 632  641                  _sa_update_sharetab_ts = (void (*)(sa_handle_t))
 633  642                      dlsym(libshare, "sa_update_sharetab_ts");
 634  643                  if (_sa_init == NULL || _sa_init_arg == NULL ||
 635  644                      _sa_fini == NULL || _sa_find_share == NULL ||
 636  645                      _sa_enable_share == NULL || _sa_disable_share == NULL ||
 637  646                      _sa_errorstr == NULL || _sa_parse_legacy_options == NULL ||
 638  647                      _sa_needs_refresh == NULL || _sa_get_zfs_handle == NULL ||
 639      -                    _sa_zfs_process_share == NULL ||
      648 +                    _sa_get_zfs_share == NULL || _sa_service == NULL ||
 640  649                      _sa_update_sharetab_ts == NULL) {
 641  650                          _sa_init = NULL;
 642  651                          _sa_init_arg = NULL;
      652 +                        _sa_service = NULL;
 643  653                          _sa_fini = NULL;
 644  654                          _sa_disable_share = NULL;
 645  655                          _sa_enable_share = NULL;
 646  656                          _sa_errorstr = NULL;
 647  657                          _sa_parse_legacy_options = NULL;
 648  658                          (void) dlclose(libshare);
 649  659                          _sa_needs_refresh = NULL;
 650  660                          _sa_get_zfs_handle = NULL;
 651      -                        _sa_zfs_process_share = NULL;
      661 +                        _sa_get_zfs_share = NULL;
 652  662                          _sa_update_sharetab_ts = NULL;
 653  663                  }
 654  664          }
 655  665  }
 656  666  
 657  667  /*
 658  668   * zfs_init_libshare(zhandle, service)
 659  669   *
 660  670   * Initialize the libshare API if it hasn't already been initialized.
 661  671   * In all cases it returns 0 if it succeeded and an error if not. The
↓ open down ↓ 129 lines elided ↑ open up ↑
 791  801  static int
 792  802  zfs_share_proto(zfs_handle_t *zhp, zfs_share_proto_t *proto)
 793  803  {
 794  804          char mountpoint[ZFS_MAXPROPLEN];
 795  805          char shareopts[ZFS_MAXPROPLEN];
 796  806          char sourcestr[ZFS_MAXPROPLEN];
 797  807          libzfs_handle_t *hdl = zhp->zfs_hdl;
 798  808          sa_share_t share;
 799  809          zfs_share_proto_t *curr_proto;
 800  810          zprop_source_t sourcetype;
      811 +        int service = SA_INIT_ONE_SHARE_FROM_HANDLE;
 801  812          int ret;
 802  813  
 803  814          if (!zfs_is_mountable(zhp, mountpoint, sizeof (mountpoint), NULL))
 804  815                  return (0);
 805  816  
      817 +        /*
      818 +         * Function may be called in a loop from higher up stack, with libshare
      819 +         * initialized for multiple shares (SA_INIT_SHARE_API_SELECTIVE).
      820 +         * zfs_init_libshare_arg will refresh the handle's cache if necessary.
      821 +         * In this case we do not want to switch to per share initialization.
      822 +         * Specify SA_INIT_SHARE_API to do full refresh, if refresh required.
      823 +         */
      824 +        if ((hdl->libzfs_sharehdl != NULL) && (_sa_service != NULL) &&
      825 +            (_sa_service(hdl->libzfs_sharehdl) ==
      826 +            SA_INIT_SHARE_API_SELECTIVE)) {
      827 +                service = SA_INIT_SHARE_API;
      828 +        }
      829 +
 806  830          for (curr_proto = proto; *curr_proto != PROTO_END; curr_proto++) {
 807  831                  /*
 808  832                   * Return success if there are no share options.
 809  833                   */
 810  834                  if (zfs_prop_get(zhp, proto_table[*curr_proto].p_prop,
 811  835                      shareopts, sizeof (shareopts), &sourcetype, sourcestr,
 812  836                      ZFS_MAXPROPLEN, B_FALSE) != 0 ||
 813  837                      strcmp(shareopts, "off") == 0)
 814  838                          continue;
 815      -                ret = zfs_init_libshare_arg(hdl, SA_INIT_ONE_SHARE_FROM_HANDLE,
 816      -                    zhp);
      839 +                ret = zfs_init_libshare_arg(hdl, service, zhp);
 817  840                  if (ret != SA_OK) {
 818  841                          (void) zfs_error_fmt(hdl, EZFS_SHARENFSFAILED,
 819  842                              dgettext(TEXT_DOMAIN, "cannot share '%s': %s"),
 820  843                              zfs_get_name(zhp), _sa_errorstr != NULL ?
 821  844                              _sa_errorstr(ret) : "");
 822  845                          return (-1);
 823  846                  }
 824  847  
 825      -                /*
 826      -                 * If the 'zoned' property is set, then zfs_is_mountable()
 827      -                 * will have already bailed out if we are in the global zone.
 828      -                 * But local zones cannot be NFS servers, so we ignore it for
 829      -                 * local zones as well.
 830      -                 */
 831      -                if (zfs_prop_get_int(zhp, ZFS_PROP_ZONED))
 832      -                        continue;
 833      -
 834  848                  share = zfs_sa_find_share(hdl->libzfs_sharehdl, mountpoint);
 835  849                  if (share == NULL) {
 836  850                          /*
 837  851                           * This may be a new file system that was just
 838      -                         * created so isn't in the internal cache
 839      -                         * (second time through). Rather than
 840      -                         * reloading the entire configuration, we can
 841      -                         * assume ZFS has done the checking and it is
 842      -                         * safe to add this to the internal
 843      -                         * configuration.
      852 +                         * created so isn't in the internal cache.
      853 +                         * Rather than reloading the entire configuration,
      854 +                         * we can add just this one share to the cache.
 844  855                           */
 845      -                        if (_sa_zfs_process_share(hdl->libzfs_sharehdl,
 846      -                            NULL, NULL, mountpoint,
 847      -                            proto_table[*curr_proto].p_name, sourcetype,
 848      -                            shareopts, sourcestr, zhp->zfs_name) != SA_OK) {
      856 +                        if ((_sa_get_zfs_share == NULL) ||
      857 +                            (_sa_get_zfs_share(hdl->libzfs_sharehdl, "zfs", zhp)
      858 +                            != SA_OK)) {
 849  859                                  (void) zfs_error_fmt(hdl,
 850  860                                      proto_table[*curr_proto].p_share_err,
 851  861                                      dgettext(TEXT_DOMAIN, "cannot share '%s'"),
 852  862                                      zfs_get_name(zhp));
 853  863                                  return (-1);
 854  864                          }
 855  865                          share = zfs_sa_find_share(hdl->libzfs_sharehdl,
 856  866                              mountpoint);
 857  867                  }
 858  868                  if (share != NULL) {
↓ open down ↓ 41 lines elided ↑ open up ↑
 900  910  /*
 901  911   * Unshare a filesystem by mountpoint.
 902  912   */
 903  913  static int
 904  914  unshare_one(libzfs_handle_t *hdl, const char *name, const char *mountpoint,
 905  915      zfs_share_proto_t proto)
 906  916  {
 907  917          sa_share_t share;
 908  918          int err;
 909  919          char *mntpt;
      920 +        int service = SA_INIT_ONE_SHARE_FROM_NAME;
 910  921  
 911  922          /*
 912  923           * Mountpoint could get trashed if libshare calls getmntany
 913  924           * which it does during API initialization, so strdup the
 914  925           * value.
 915  926           */
 916  927          mntpt = zfs_strdup(hdl, mountpoint);
 917  928  
 918  929          /*
 919      -         * make sure libshare initialized, initialize everything because we
 920      -         * don't know what other unsharing may happen later. Functions up the
 921      -         * stack are allowed to initialize instead a subset of shares at the
 922      -         * time the set is known.
      930 +         * Function may be called in a loop from higher up stack, with libshare
      931 +         * initialized for multiple shares (SA_INIT_SHARE_API_SELECTIVE).
      932 +         * zfs_init_libshare_arg will refresh the handle's cache if necessary.
      933 +         * In this case we do not want to switch to per share initialization.
      934 +         * Specify SA_INIT_SHARE_API to do full refresh, if refresh required.
 923  935           */
 924      -        if ((err = zfs_init_libshare_arg(hdl, SA_INIT_ONE_SHARE_FROM_NAME,
 925      -            (void *)name)) != SA_OK) {
      936 +        if ((hdl->libzfs_sharehdl != NULL) && (_sa_service != NULL) &&
      937 +            (_sa_service(hdl->libzfs_sharehdl) ==
      938 +            SA_INIT_SHARE_API_SELECTIVE)) {
      939 +                service = SA_INIT_SHARE_API;
      940 +        }
      941 +
      942 +        err = zfs_init_libshare_arg(hdl, service, (void *)name);
      943 +        if (err != SA_OK) {
 926  944                  free(mntpt);    /* don't need the copy anymore */
 927  945                  return (zfs_error_fmt(hdl, proto_table[proto].p_unshare_err,
 928  946                      dgettext(TEXT_DOMAIN, "cannot unshare '%s': %s"),
 929  947                      name, _sa_errorstr(err)));
 930  948          }
 931  949  
 932  950          share = zfs_sa_find_share(hdl->libzfs_sharehdl, mntpt);
 933  951          free(mntpt);    /* don't need the copy anymore */
 934  952  
 935  953          if (share != NULL) {
↓ open down ↓ 212 lines elided ↑ open up ↑
1148 1166                  return (strcmp(mounta, mountb));
1149 1167  
1150 1168          if (gota)
1151 1169                  return (-1);
1152 1170          if (gotb)
1153 1171                  return (1);
1154 1172  
1155 1173          return (strcmp(zfs_get_name(a), zfs_get_name(b)));
1156 1174  }
1157 1175  
1158      -/*
1159      - * Mount and share all datasets within the given pool.  This assumes that no
1160      - * datasets within the pool are currently mounted.  Because users can create
1161      - * complicated nested hierarchies of mountpoints, we first gather all the
1162      - * datasets and mountpoints within the pool, and sort them by mountpoint.  Once
1163      - * we have the list of all filesystems, we iterate over them in order and mount
1164      - * and/or share each one.
1165      - */
1166      -#pragma weak zpool_mount_datasets = zpool_enable_datasets
     1176 +static int
     1177 +mountpoint_compare(const void *a, const void *b)
     1178 +{
     1179 +        const char *mounta = *((char **)a);
     1180 +        const char *mountb = *((char **)b);
     1181 +
     1182 +        return (strcmp(mountb, mounta));
     1183 +}
     1184 +
     1185 +typedef enum {
     1186 +        TASK_TO_PROCESS,
     1187 +        TASK_IN_PROCESSING,
     1188 +        TASK_DONE,
     1189 +        TASK_MAX
     1190 +} task_state_t;
     1191 +
     1192 +typedef struct mount_task {
     1193 +        const char      *mp;
     1194 +        zfs_handle_t    *zh;
     1195 +        task_state_t    state;
     1196 +        int             error;
     1197 +} mount_task_t;
     1198 +
     1199 +typedef struct mount_task_q {
     1200 +        pthread_mutex_t q_lock;
     1201 +        libzfs_handle_t *hdl;
     1202 +        const char      *mntopts;
     1203 +        const char      *error_mp;
     1204 +        zfs_handle_t    *error_zh;
     1205 +        int             error;
     1206 +        int             q_length;
     1207 +        int             n_tasks;
     1208 +        int             flags;
     1209 +        mount_task_t    task[1];
     1210 +} mount_task_q_t;
     1211 +
     1212 +static int
     1213 +mount_task_q_init(int argc, zfs_handle_t **handles, const char *mntopts,
     1214 +    int flags, mount_task_q_t **task)
     1215 +{
     1216 +        mount_task_q_t *task_q;
     1217 +        int i, error;
     1218 +        size_t task_q_size;
     1219 +
     1220 +        *task = NULL;
     1221 +        /* nothing to do ? should not be here */
     1222 +        if (argc <= 0)
     1223 +                return (EINVAL);
     1224 +
     1225 +        /* allocate and init task_q */
     1226 +        task_q_size = sizeof (mount_task_q_t) +
     1227 +            (argc - 1) * sizeof (mount_task_t);
     1228 +        task_q = calloc(task_q_size, 1);
     1229 +        if (task_q == NULL)
     1230 +                return (ENOMEM);
     1231 +
     1232 +        if ((error = pthread_mutex_init(&task_q->q_lock, NULL)) != 0) {
     1233 +                free(task_q);
     1234 +                return (error);
     1235 +        }
     1236 +        task_q->q_length = argc;
     1237 +        task_q->n_tasks = argc;
     1238 +        task_q->flags = flags;
     1239 +        task_q->mntopts = mntopts;
     1240 +
     1241 +        /* we are not going to change the strings, so no need to strdup */
     1242 +        for (i = 0; i < argc; ++i) {
     1243 +                task_q->task[i].zh = handles[i];
     1244 +                task_q->task[i].state = TASK_TO_PROCESS;
     1245 +                task_q->error = 0;
     1246 +        }
     1247 +
     1248 +        *task = task_q;
     1249 +        return (0);
     1250 +}
     1251 +
     1252 +static int
     1253 +umount_task_q_init(int argc, const char **argv, int flags,
     1254 +    libzfs_handle_t *hdl, mount_task_q_t **task)
     1255 +{
     1256 +        mount_task_q_t *task_q;
     1257 +        int i, error;
     1258 +        size_t task_q_size;
     1259 +
     1260 +        *task = NULL;
     1261 +        /* nothing to do ? should not be here */
     1262 +        if (argc <= 0)
     1263 +                return (EINVAL);
     1264 +
     1265 +        /* allocate and init task_q */
     1266 +        task_q_size = sizeof (mount_task_q_t) +
     1267 +            (argc - 1) * sizeof (mount_task_t);
     1268 +        task_q = calloc(task_q_size, 1);
     1269 +        if (task_q == NULL)
     1270 +                return (ENOMEM);
     1271 +
     1272 +        if ((error = pthread_mutex_init(&task_q->q_lock, NULL)) != 0) {
     1273 +                free(task_q);
     1274 +                return (error);
     1275 +        }
     1276 +        task_q->hdl = hdl;
     1277 +        task_q->q_length = argc;
     1278 +        task_q->n_tasks = argc;
     1279 +        task_q->flags = flags;
     1280 +
     1281 +        /* we are not going to change the strings, so no need to strdup */
     1282 +        for (i = 0; i < argc; ++i) {
     1283 +                task_q->task[i].mp = argv[i];
     1284 +                task_q->task[i].state = TASK_TO_PROCESS;
     1285 +                task_q->error = 0;
     1286 +        }
     1287 +
     1288 +        *task = task_q;
     1289 +        return (0);
     1290 +}
     1291 +
     1292 +static void
     1293 +mount_task_q_fini(mount_task_q_t *task_q)
     1294 +{
     1295 +        assert(task_q != NULL);
     1296 +        (void) pthread_mutex_destroy(&task_q->q_lock);
     1297 +        free(task_q);
     1298 +}
     1299 +
     1300 +static int
     1301 +is_child_of(const char *s1, const char *s2)
     1302 +{
     1303 +        for (; *s1 && *s2 && (*s1 == *s2); ++s1, ++s2)
     1304 +                ;
     1305 +        return (!*s2 && (*s1 == '/'));
     1306 +}
     1307 +
     1308 +static boolean_t
     1309 +task_completed(int ind, mount_task_q_t *task_q)
     1310 +{
     1311 +        return (task_q->task[ind].state == TASK_DONE);
     1312 +}
     1313 +
     1314 +static boolean_t
     1315 +task_to_process(int ind, mount_task_q_t *task_q)
     1316 +{
     1317 +        return (task_q->task[ind].state == TASK_TO_PROCESS);
     1318 +}
     1319 +
     1320 +static boolean_t
     1321 +task_in_processing(int ind, mount_task_q_t *task_q)
     1322 +{
     1323 +        return (task_q->task[ind].state == TASK_IN_PROCESSING);
     1324 +}
     1325 +
     1326 +static void
     1327 +task_next_stage(int ind, mount_task_q_t *task_q)
     1328 +{
     1329 +        /* our state machine is a pipeline */
     1330 +        task_q->task[ind].state++;
     1331 +        assert(task_q->task[ind].state < TASK_MAX);
     1332 +}
     1333 +
     1334 +static boolean_t
     1335 +task_state_valid(int ind, mount_task_q_t *task_q)
     1336 +{
     1337 +        /* our state machine is a pipeline */
     1338 +        return (task_q->task[ind].state < TASK_MAX);
     1339 +}
     1340 +
     1341 +static boolean_t
     1342 +child_umount_pending(int ind, mount_task_q_t *task_q)
     1343 +{
     1344 +        int i;
     1345 +        for (i = ind-1; i >= 0; --i) {
     1346 +                assert(task_state_valid(i, task_q));
     1347 +                if ((task_q->task[i].state != TASK_DONE) &&
     1348 +                    is_child_of(task_q->task[i].mp, task_q->task[ind].mp))
     1349 +                        return (B_TRUE);
     1350 +        }
     1351 +
     1352 +        return (B_FALSE);
     1353 +}
     1354 +
     1355 +static boolean_t
     1356 +parent_mount_pending(int ind, mount_task_q_t *task_q)
     1357 +{
     1358 +        int i;
     1359 +        for (i = ind-1; i >= 0; --i) {
     1360 +                assert(task_state_valid(i, task_q));
     1361 +                if ((task_q->task[i].state != TASK_DONE) &&
     1362 +                    is_child_of(task_q->task[ind].zh->zfs_name,
     1363 +                    task_q->task[i].zh->zfs_name))
     1364 +                        return (B_TRUE);
     1365 +        }
     1366 +
     1367 +        return (B_FALSE);
     1368 +}
     1369 +
     1370 +static void
     1371 +unmounter(void *arg)
     1372 +{
     1373 +        mount_task_q_t *task_q = (mount_task_q_t *)arg;
     1374 +        int error = 0, done = 0;
     1375 +
     1376 +        assert(task_q != NULL);
     1377 +        if (task_q == NULL)
     1378 +                return;
     1379 +
     1380 +        while (!error && !done) {
     1381 +                mount_task_t *task;
     1382 +                int i, t, umount_err, flags, q_error;
     1383 +
     1384 +                if ((error = pthread_mutex_lock(&task_q->q_lock)) != 0)
     1385 +                        break; /* Out of while() loop */
     1386 +
     1387 +                if (task_q->error || task_q->n_tasks == 0) {
     1388 +                        (void) pthread_mutex_unlock(&task_q->q_lock);
     1389 +                        break; /* Out of while() loop */
     1390 +                }
     1391 +
     1392 +                /* Find task ready for processing */
     1393 +                for (i = 0, task = NULL, t = -1; i < task_q->q_length; ++i) {
     1394 +                        if (task_q->error) {
     1395 +                                /* Fatal error, stop processing */
     1396 +                                done = 1;
     1397 +                                break; /* Out of for() loop */
     1398 +                        }
     1399 +
     1400 +                        if (task_completed(i, task_q))
     1401 +                                continue; /* for() loop */
     1402 +
     1403 +                        if (task_to_process(i, task_q)) {
     1404 +                                /*
     1405 +                                 * Cannot umount if some children are still
     1406 +                                 * mounted; come back later
     1407 +                                 */
     1408 +                                if ((child_umount_pending(i, task_q)))
     1409 +                                        continue; /* for() loop */
     1410 +                                /* Should be OK to unmount now */
     1411 +                                task_next_stage(i, task_q);
     1412 +                                task = &task_q->task[i];
     1413 +                                t = i;
     1414 +                                break; /* Out of for() loop */
     1415 +                        }
     1416 +
     1417 +                        /* Otherwise, the task is already in processing */
     1418 +                        assert(task_in_processing(i, task_q));
     1419 +                }
     1420 +
     1421 +                flags = task_q->flags;
     1422 +
     1423 +                error = pthread_mutex_unlock(&task_q->q_lock);
     1424 +
     1425 +                if (done || (task == NULL) || error || task_q->error)
     1426 +                        break; /* Out of while() loop */
     1427 +
     1428 +                umount_err = umount2(task->mp, flags);
     1429 +                q_error = errno;
     1430 +
     1431 +                if ((error = pthread_mutex_lock(&task_q->q_lock)) != 0)
     1432 +                        break; /* Out of while() loop */
     1433 +
     1434 +                /* done processing */
     1435 +                assert(t >= 0 && t < task_q->q_length);
     1436 +                task_next_stage(t, task_q);
     1437 +                assert(task_completed(t, task_q));
     1438 +                task_q->n_tasks--;
     1439 +
     1440 +                if (umount_err) {
     1441 +                        /*
     1442 +                         * umount2() failed, cannot be busy because of mounted
     1443 +                         * children - we have checked above, so it is fatal
     1444 +                         */
     1445 +                        assert(child_umount_pending(t, task_q) == B_FALSE);
     1446 +                        task->error = q_error;
     1447 +                        if (!task_q->error) {
     1448 +                                task_q->error = task->error;
     1449 +                                task_q->error_mp = task->mp;
     1450 +                        }
     1451 +                        done = 1;
     1452 +                }
     1453 +
     1454 +                if ((error = pthread_mutex_unlock(&task_q->q_lock)) != 0)
     1455 +                        break; /* Out of while() loop */
     1456 +        }
     1457 +}
     1458 +
     1459 +static void
     1460 +mounter(void *arg)
     1461 +{
     1462 +        mount_task_q_t *task_q = (mount_task_q_t *)arg;
     1463 +        int error = 0, done = 0;
     1464 +
     1465 +        assert(task_q != NULL);
     1466 +        if (task_q == NULL)
     1467 +                return;
     1468 +
     1469 +        while (!error && !done) {
     1470 +                mount_task_t *task;
     1471 +                int i, t, mount_err, flags, q_error;
     1472 +                const char *mntopts;
     1473 +
     1474 +                if ((error = pthread_mutex_lock(&task_q->q_lock)) != 0)
     1475 +                        break; /* Out of while() loop */
     1476 +
     1477 +                if (task_q->error || task_q->n_tasks == 0) {
     1478 +                        (void) pthread_mutex_unlock(&task_q->q_lock);
     1479 +                        break; /* Out of while() loop */
     1480 +                }
     1481 +
     1482 +                /* Find task ready for processing */
     1483 +                for (i = 0, task = NULL, t = -1; i < task_q->q_length; ++i) {
     1484 +                        if (task_q->error) {
     1485 +                                /* Fatal error, stop processing */
     1486 +                                done = 1;
     1487 +                                break; /* Out of for() loop */
     1488 +                        }
     1489 +
     1490 +                        if (task_completed(i, task_q))
     1491 +                                continue; /* for() loop */
     1492 +
     1493 +                        if (task_to_process(i, task_q)) {
     1494 +                                /*
     1495 +                                 * Cannot mount if some parents are not
     1496 +                                 * mounted yet; come back later
     1497 +                                 */
     1498 +                                if ((parent_mount_pending(i, task_q)))
     1499 +                                        continue; /* for() loop */
     1500 +                                /* Should be OK to mount now */
     1501 +                                task_next_stage(i, task_q);
     1502 +                                task = &task_q->task[i];
     1503 +                                t = i;
     1504 +                                break; /* Out of for() loop */
     1505 +                        }
     1506 +
     1507 +                        /* Otherwise, the task is already in processing */
     1508 +                        assert(task_in_processing(i, task_q));
     1509 +                }
     1510 +
     1511 +                flags = task_q->flags;
     1512 +                mntopts = task_q->mntopts;
     1513 +
     1514 +                error = pthread_mutex_unlock(&task_q->q_lock);
     1515 +
     1516 +                if (done || (task == NULL) || error || task_q->error)
     1517 +                        break; /* Out of while() loop */
     1518 +
     1519 +                mount_err = zfs_mount(task->zh, mntopts, flags);
     1520 +                q_error = errno;
     1521 +
     1522 +                if ((error = pthread_mutex_lock(&task_q->q_lock)) != 0)
     1523 +                        break; /* Out of while() loop */
     1524 +
     1525 +                /* done processing */
     1526 +                assert(t >= 0 && t < task_q->q_length);
     1527 +                task_next_stage(t, task_q);
     1528 +                assert(task_completed(t, task_q));
     1529 +                task_q->n_tasks--;
     1530 +
     1531 +                if (mount_err) {
     1532 +                        task->error = q_error;
     1533 +                        if (!task_q->error) {
     1534 +                                task_q->error = task->error;
     1535 +                                task_q->error_zh = task->zh;
     1536 +                        }
     1537 +                        done = 1;
     1538 +                }
     1539 +
     1540 +                if ((error = pthread_mutex_unlock(&task_q->q_lock)) != 0)
     1541 +                        break; /* Out of while() loop */
     1542 +        }
     1543 +}
     1544 +
     1545 +#define THREADS_HARD_LIMIT      128
     1546 +int parallel_unmount(libzfs_handle_t *hdl, int argc, const char **argv,
     1547 +    int flags, int n_threads)
     1548 +{
     1549 +        mount_task_q_t *task_queue = NULL;
     1550 +        int             i, error;
     1551 +        tpool_t         *t;
     1552 +
     1553 +        if (argc == 0)
     1554 +                return (0);
     1555 +
     1556 +        if ((error = umount_task_q_init(argc, argv, flags, hdl, &task_queue))
     1557 +            != 0) {
     1558 +                assert(task_queue == NULL);
     1559 +                return (error);
     1560 +        }
     1561 +
     1562 +        if (n_threads > argc)
     1563 +                n_threads = argc;
     1564 +
     1565 +        if (n_threads > THREADS_HARD_LIMIT)
     1566 +                n_threads = THREADS_HARD_LIMIT;
     1567 +
     1568 +        t = tpool_create(1, n_threads, 0, NULL);
     1569 +
     1570 +        for (i = 0; i < n_threads; ++i)
     1571 +                (void) tpool_dispatch(t, unmounter, task_queue);
     1572 +
     1573 +        tpool_wait(t);
     1574 +        tpool_destroy(t);
     1575 +
     1576 +        if (task_queue->error) {
     1577 +                /*
     1578 +                 * Tell ZFS!
     1579 +                 */
     1580 +                zfs_error_aux(hdl,
     1581 +                    strerror(error ? error : task_queue->error));
     1582 +                error = zfs_error_fmt(hdl, EZFS_UMOUNTFAILED,
     1583 +                    dgettext(TEXT_DOMAIN, "cannot unmount '%s'"),
     1584 +                    error ? "datasets" : task_queue->error_mp);
     1585 +        }
     1586 +        if (task_queue)
     1587 +                mount_task_q_fini(task_queue);
     1588 +
     1589 +        return (error);
     1590 +}
     1591 +
     1592 +int parallel_mount(get_all_cb_t *cb, int *good, const char *mntopts,
     1593 +    int flags, int n_threads)
     1594 +{
     1595 +        int             i, error = 0;
     1596 +        mount_task_q_t  *task_queue = NULL;
     1597 +        tpool_t         *t;
     1598 +
     1599 +        if (cb->cb_used == 0)
     1600 +                return (0);
     1601 +
     1602 +        if (n_threads > cb->cb_used)
     1603 +                n_threads = cb->cb_used;
     1604 +
     1605 +        if ((error = mount_task_q_init(cb->cb_used, cb->cb_handles,
     1606 +            mntopts, flags, &task_queue)) != 0) {
     1607 +                assert(task_queue == NULL);
     1608 +                return (error);
     1609 +        }
     1610 +
     1611 +        t = tpool_create(1, n_threads, 0, NULL);
     1612 +
     1613 +        for (i = 0; i < n_threads; ++i)
     1614 +                (void) tpool_dispatch(t, mounter, task_queue);
     1615 +
     1616 +        tpool_wait(t);
     1617 +        for (i = 0; i < cb->cb_used; ++i) {
     1618 +                good[i] = !task_queue->task[i].error;
     1619 +                if (!good[i]) {
     1620 +                        zfs_handle_t *hdl = task_queue->error_zh;
     1621 +                        zfs_error_aux(hdl->zfs_hdl,
     1622 +                            strerror(task_queue->task[i].error));
     1623 +                        (void) zfs_error_fmt(hdl->zfs_hdl, EZFS_MOUNTFAILED,
     1624 +                            dgettext(TEXT_DOMAIN, "cannot mount '%s'"),
     1625 +                            task_queue->task[i].zh->zfs_name);
     1626 +                }
     1627 +        }
     1628 +        tpool_destroy(t);
     1629 +
     1630 +        if (task_queue->error) {
     1631 +                zfs_handle_t *hdl = task_queue->error_zh;
     1632 +                /*
     1633 +                 * Tell ZFS!
     1634 +                 */
     1635 +                zfs_error_aux(hdl->zfs_hdl,
     1636 +                    strerror(error ? error : task_queue->error));
     1637 +                error = zfs_error_fmt(hdl->zfs_hdl, EZFS_MOUNTFAILED,
     1638 +                    dgettext(TEXT_DOMAIN, "cannot mount '%s'"),
     1639 +                    error ? "datasets" : hdl->zfs_name);
     1640 +        }
     1641 +        if (task_queue)
     1642 +                mount_task_q_fini(task_queue);
     1643 +
     1644 +        return (error);
     1645 +}
     1646 +
1167 1647  int
1168      -zpool_enable_datasets(zpool_handle_t *zhp, const char *mntopts, int flags)
     1648 +zpool_enable_datasets_ex(zpool_handle_t *zhp, const char *mntopts, int flags,
     1649 +    int n_threads)
1169 1650  {
1170 1651          get_all_cb_t cb = { 0 };
1171 1652          libzfs_handle_t *hdl = zhp->zpool_hdl;
1172 1653          zfs_handle_t *zfsp;
1173 1654          int i, ret = -1;
1174 1655          int *good;
     1656 +        sa_init_selective_arg_t sharearg;
1175 1657  
1176 1658          /*
1177 1659           * Gather all non-snap datasets within the pool.
1178 1660           */
1179 1661          if ((zfsp = zfs_open(hdl, zhp->zpool_name, ZFS_TYPE_DATASET)) == NULL)
1180 1662                  goto out;
1181 1663  
1182 1664          libzfs_add_handle(&cb, zfsp);
1183 1665          if (zfs_iter_filesystems(zfsp, mount_cb, &cb) != 0)
1184 1666                  goto out;
↓ open down ↓ 5 lines elided ↑ open up ↑
1190 1672  
1191 1673          /*
1192 1674           * And mount all the datasets, keeping track of which ones
1193 1675           * succeeded or failed.
1194 1676           */
1195 1677          if ((good = zfs_alloc(zhp->zpool_hdl,
1196 1678              cb.cb_used * sizeof (int))) == NULL)
1197 1679                  goto out;
1198 1680  
1199 1681          ret = 0;
1200      -        for (i = 0; i < cb.cb_used; i++) {
1201      -                if (zfs_mount(cb.cb_handles[i], mntopts, flags) != 0)
1202      -                        ret = -1;
1203      -                else
1204      -                        good[i] = 1;
     1682 +        if (n_threads < 2) {
     1683 +                for (i = 0; i < cb.cb_used; i++) {
     1684 +                        if (zfs_mount(cb.cb_handles[i], mntopts, flags) != 0)
     1685 +                                ret = -1;
     1686 +                        else
     1687 +                                good[i] = 1;
     1688 +                }
     1689 +        } else {
     1690 +                ret = parallel_mount(&cb, good, mntopts, flags, n_threads);
1205 1691          }
1206 1692  
1207 1693          /*
     1694 +         * Initilialize libshare SA_INIT_SHARE_API_SELECTIVE here
     1695 +         * to avoid unneccesary load/unload of the libshare API
     1696 +         * per shared dataset downstream.
     1697 +         */
     1698 +        sharearg.zhandle_arr = cb.cb_handles;
     1699 +        sharearg.zhandle_len = cb.cb_used;
     1700 +        ret = zfs_init_libshare_arg(hdl, SA_INIT_SHARE_API_SELECTIVE,
     1701 +            &sharearg);
     1702 +        if (ret != 0) {
     1703 +                free(good);
     1704 +                goto out;
     1705 +        }
     1706 +
     1707 +        /*
1208 1708           * Then share all the ones that need to be shared. This needs
1209 1709           * to be a separate pass in order to avoid excessive reloading
1210 1710           * of the configuration. Good should never be NULL since
1211 1711           * zfs_alloc is supposed to exit if memory isn't available.
1212 1712           */
1213 1713          for (i = 0; i < cb.cb_used; i++) {
1214 1714                  if (good[i] && zfs_share(cb.cb_handles[i]) != 0)
1215 1715                          ret = -1;
1216 1716          }
1217 1717  
1218 1718          free(good);
1219 1719  
1220 1720  out:
1221 1721          for (i = 0; i < cb.cb_used; i++)
1222 1722                  zfs_close(cb.cb_handles[i]);
1223 1723          free(cb.cb_handles);
1224 1724  
1225 1725          return (ret);
1226 1726  }
1227 1727  
1228      -static int
1229      -mountpoint_compare(const void *a, const void *b)
1230      -{
1231      -        const char *mounta = *((char **)a);
1232      -        const char *mountb = *((char **)b);
1233      -
1234      -        return (strcmp(mountb, mounta));
1235      -}
1236      -
1237      -/* alias for 2002/240 */
1238      -#pragma weak zpool_unmount_datasets = zpool_disable_datasets
1239      -/*
1240      - * Unshare and unmount all datasets within the given pool.  We don't want to
1241      - * rely on traversing the DSL to discover the filesystems within the pool,
1242      - * because this may be expensive (if not all of them are mounted), and can fail
1243      - * arbitrarily (on I/O error, for example).  Instead, we walk /etc/mnttab and
1244      - * gather all the filesystems that are currently mounted.
1245      - */
1246 1728  int
1247      -zpool_disable_datasets(zpool_handle_t *zhp, boolean_t force)
     1729 +zpool_disable_datasets_ex(zpool_handle_t *zhp, boolean_t force, int n_threads)
1248 1730  {
1249 1731          int used, alloc;
1250 1732          struct mnttab entry;
1251 1733          size_t namelen;
1252 1734          char **mountpoints = NULL;
1253 1735          zfs_handle_t **datasets = NULL;
1254 1736          libzfs_handle_t *hdl = zhp->zpool_hdl;
1255 1737          int i;
1256 1738          int ret = -1;
1257 1739          int flags = (force ? MS_FORCE : 0);
↓ open down ↓ 81 lines elided ↑ open up ↑
1339 1821          qsort(mountpoints, used, sizeof (char *), mountpoint_compare);
1340 1822  
1341 1823          /*
1342 1824           * Walk through and first unshare everything.
1343 1825           */
1344 1826          for (i = 0; i < used; i++) {
1345 1827                  zfs_share_proto_t *curr_proto;
1346 1828                  for (curr_proto = share_all_proto; *curr_proto != PROTO_END;
1347 1829                      curr_proto++) {
1348 1830                          if (is_shared(hdl, mountpoints[i], *curr_proto) &&
1349      -                            unshare_one(hdl, mountpoints[i],
1350      -                            mountpoints[i], *curr_proto) != 0)
     1831 +                            unshare_one(hdl, mountpoints[i], mountpoints[i],
     1832 +                            *curr_proto) != 0)
1351 1833                                  goto out;
1352 1834                  }
1353 1835          }
1354 1836  
1355 1837          /*
1356 1838           * Now unmount everything, removing the underlying directories as
1357 1839           * appropriate.
1358 1840           */
1359      -        for (i = 0; i < used; i++) {
1360      -                if (unmount_one(hdl, mountpoints[i], flags) != 0)
     1841 +        if (n_threads < 2) {
     1842 +                for (i = 0; i < used; i++) {
     1843 +                        if (unmount_one(hdl, mountpoints[i], flags) != 0)
     1844 +                                goto out;
     1845 +                }
     1846 +        } else {
     1847 +                if (parallel_unmount(hdl, used, (const char **)mountpoints,
     1848 +                    flags, n_threads) != 0)
1361 1849                          goto out;
1362 1850          }
1363      -
1364 1851          for (i = 0; i < used; i++) {
1365 1852                  if (datasets[i])
1366 1853                          remove_mountpoint(datasets[i]);
1367 1854          }
1368      -
1369 1855          ret = 0;
1370 1856  out:
1371 1857          for (i = 0; i < used; i++) {
1372 1858                  if (datasets[i])
1373 1859                          zfs_close(datasets[i]);
1374 1860                  free(mountpoints[i]);
1375 1861          }
1376 1862          free(datasets);
1377 1863          free(mountpoints);
1378 1864  
1379 1865          return (ret);
     1866 +}
     1867 +
     1868 +/*
     1869 + * Mount and share all datasets within the given pool.  This assumes that no
     1870 + * datasets within the pool are currently mounted.  Because users can create
     1871 + * complicated nested hierarchies of mountpoints, we first gather all the
     1872 + * datasets and mountpoints within the pool, and sort them by mountpoint.  Once
     1873 + * we have the list of all filesystems, we iterate over them in order and mount
     1874 + * and/or share each one.
     1875 + */
     1876 +#pragma weak zpool_mount_datasets = zpool_enable_datasets
     1877 +int
     1878 +zpool_enable_datasets(zpool_handle_t *zhp, const char *mntopts, int flags)
     1879 +{
     1880 +        return (zpool_enable_datasets_ex(zhp, mntopts, flags, 1));
     1881 +}
     1882 +
     1883 +/* alias for 2002/240 */
     1884 +#pragma weak zpool_unmount_datasets = zpool_disable_datasets
     1885 +/*
     1886 + * Unshare and unmount all datasets within the given pool.  We don't want to
     1887 + * rely on traversing the DSL to discover the filesystems within the pool,
     1888 + * because this may be expensive (if not all of them are mounted), and can fail
     1889 + * arbitrarily (on I/O error, for example).  Instead, we walk /etc/mnttab and
     1890 + * gather all the filesystems that are currently mounted.
     1891 + */
     1892 +int
     1893 +zpool_disable_datasets(zpool_handle_t *zhp, boolean_t force)
     1894 +{
     1895 +        return (zpool_disable_datasets_ex(zhp, force, 1));
1380 1896  }
    
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX