Print this page
NEX-19040 SMB2 session ID re-use can confuse clients
Reviewed by: Matt Barden <matt.barden@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
NEX-18761 panic in smb_ofile_free with vdbench
Reviewed by: Matt Barden <matt.barden@nexenta.com>
Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
NEX-16604 Windows 10 SMB client exhausts smbauth sockets
Reviewed by: Matt Barden <matt.barden@nexenta.com>
Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
NEX-16519 Panic while running IOmeter to a pool through an SMB share
Reviewed by: Evan Layton <evan.layton@nexenta.com>
Reviewed by: Matt Barden <matt.barden@nexenta.com>
NEX-9808 SMB3 persistent handles
Reviewed by: Matt Barden <matt.barden@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
NEX-15578 SMB2 durable handle redesign
Reviewed by: Matt Barden <matt.barden@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
NEX-9808 SMB3 persistent handles
Reviewed by: Matt Barden <matt.barden@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
NEX-15578 SMB2 durable handle redesign
Reviewed by: Matt Barden <matt.barden@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
NEX-7267 Codenomicon causes panic in smb_authsock_cancel
Reviewed by: Roman Strashkin <roman.strashkin@nexenta.com>
Reviewed by: Rick McNeal <rick.mcneal@nexenta.com>
Reviewed by: Evan Layton <evan.layton@nexenta.com>
Reviewed by: Matt Barden <matt.barden@nexenta.com>
NEX-3553 SMB2/3 durable handles
Reviewed by: Gordon Ross <gwr@nexenta.com>
Reviewed by: Kevin Crowe <kevin.crowe@nexenta.com>
NEX-3776 SMB should handle PreviousSessionID
Reviewed by: Gordon Ross <gwr@nexenta.com>
NEX-5560 smb2 should use 64-bit server-global uids
Reviewed by: Gordon Ross <gwr@nexenta.com>
NEX-5537 Want reference counts for users, trees...
Reviewed by: Gordon Ross <gwr@nexenta.com>
NEX-2485 SMB authentication flood handled poorly
SMB-56 extended security NTLMSSP, inbound
SMB-50 User-mode SMB server
 Includes work by these authors:
 Thomas Keiser <thomas.keiser@nexenta.com>
 Albert Lee <trisk@nexenta.com>
SMB-65 SMB server in non-global zones (data structure changes)
Many things move to the smb_server_t object, and
many functions gain an sv arg (which server).
SMB-65 SMB server in non-global zones (kmem_caches)
common kmem_cache instances across zones
separate GZ-only init from NGZ init
SMB-63 taskq_create_proc ... TQ_DYNAMIC puts tasks in p0
re #11974 CIFS Share - Tree connect fails from Windows 7 Clients

Split Close
Expand all
Collapse all
          --- old/usr/src/uts/common/fs/smbsrv/smb_user.c
          +++ new/usr/src/uts/common/fs/smbsrv/smb_user.c
↓ open down ↓ 12 lines elided ↑ open up ↑
  13   13   * When distributing Covered Code, include this CDDL HEADER in each
  14   14   * file and include the License file at usr/src/OPENSOLARIS.LICENSE.
  15   15   * If applicable, add the following below this CDDL HEADER, with the
  16   16   * fields enclosed by brackets "[]" replaced with your own identifying
  17   17   * information: Portions Copyright [yyyy] [name of copyright owner]
  18   18   *
  19   19   * CDDL HEADER END
  20   20   */
  21   21  /*
  22   22   * Copyright (c) 2007, 2010, Oracle and/or its affiliates. All rights reserved.
  23      - * Copyright 2014 Nexenta Systems, Inc. All rights reserved.
       23 + * Copyright 2018 Nexenta Systems, Inc. All rights reserved.
  24   24   * Copyright (c) 2016 by Delphix. All rights reserved.
  25   25   */
  26   26  
  27   27  /*
  28   28   * General Structures Layout
  29   29   * -------------------------
  30   30   *
  31   31   * This is a simplified diagram showing the relationship between most of the
  32   32   * main structures.
  33   33   *
↓ open down ↓ 140 lines elided ↑ open up ↑
 174  174   *
 175  175   *    There's a mutex embedded in the user structure used to protect its fields
 176  176   *    and there's a lock embedded in the list of users of a session. To
 177  177   *    increment or to decrement the reference count the mutex must be entered.
 178  178   *    To insert the user into the list of users of the session and to remove
 179  179   *    the user from it, the lock must be entered in RW_WRITER mode.
 180  180   *
 181  181   *    Rules of access to a user structure:
 182  182   *
 183  183   *    1) In order to avoid deadlocks, when both (mutex and lock of the session
 184      - *       list) have to be entered, the lock must be entered first.
      184 + *       list) have to be entered, the lock must be entered first. Additionally,
      185 + *       one may NOT flush the deleteq of either the tree list or the ofile list
      186 + *       while the user mutex is held.
 185  187   *
 186  188   *    2) All actions applied to a user require a reference count.
 187  189   *
 188  190   *    3) There are 2 ways of getting a reference count. One is when the user
 189  191   *       logs in. The other when the user is looked up.
 190  192   *
 191  193   *    It should be noted that the reference count of a user registers the
 192  194   *    number of references to the user in other structures (such as an smb
 193  195   *    request). The reference count is not incremented in these 2 instances:
 194  196   *
↓ open down ↓ 6 lines elided ↑ open up ↑
 201  203   *       reference count.
 202  204   */
 203  205  #include <sys/types.h>
 204  206  #include <sys/sid.h>
 205  207  #include <sys/priv_names.h>
 206  208  #include <smbsrv/smb_kproto.h>
 207  209  #include <smbsrv/smb_door.h>
 208  210  
 209  211  #define ADMINISTRATORS_SID      "S-1-5-32-544"
 210  212  
      213 +/* Don't leak object addresses */
      214 +#define SMB_USER_SSNID(u) \
      215 +        ((uintptr_t)&smb_cache_user ^ (uintptr_t)(u))
      216 +
      217 +static void smb_user_delete(void *);
 211  218  static int smb_user_enum_private(smb_user_t *, smb_svcenum_t *);
 212  219  static void smb_user_auth_logoff(smb_user_t *);
      220 +static void smb_user_logoff_tq(void *);
 213  221  
 214      -
 215  222  /*
 216  223   * Create a new user.
      224 + *
      225 + * For SMB2 and later, session IDs (u_ssnid) need to be unique among all
      226 + * current and "recent" sessions.  The session ID is derived from the
      227 + * address of the smb_user object (obscured by XOR with a constant).
      228 + * This adds a 3-bit generation number in the low bits, incremented
      229 + * when we allocate an smb_user_t from its kmem cache, so it can't
      230 + * be confused with a (recent) previous incarnation of this object.
 217  231   */
 218  232  smb_user_t *
 219  233  smb_user_new(smb_session_t *session)
 220  234  {
 221  235          smb_user_t      *user;
      236 +        uint_t          gen;    // generation (low 3 bits of ssnid)
 222  237  
 223  238          ASSERT(session);
 224  239          ASSERT(session->s_magic == SMB_SESSION_MAGIC);
 225  240  
 226  241          user = kmem_cache_alloc(smb_cache_user, KM_SLEEP);
      242 +        gen = (user->u_ssnid + 1) & 7;
 227  243          bzero(user, sizeof (smb_user_t));
 228  244  
 229  245          user->u_refcnt = 1;
 230  246          user->u_session = session;
 231  247          user->u_server = session->s_server;
 232  248          user->u_logon_time = gethrestime_sec();
 233  249  
 234  250          if (smb_idpool_alloc(&session->s_uid_pool, &user->u_uid))
 235  251                  goto errout;
      252 +        user->u_ssnid = SMB_USER_SSNID(user) + gen;
 236  253  
 237  254          mutex_init(&user->u_mutex, NULL, MUTEX_DEFAULT, NULL);
 238  255          user->u_state = SMB_USER_STATE_LOGGING_ON;
 239  256          user->u_magic = SMB_USER_MAGIC;
 240  257  
 241  258          smb_llist_enter(&session->s_user_list, RW_WRITER);
 242  259          smb_llist_insert_tail(&session->s_user_list, user);
 243  260          smb_llist_exit(&session->s_user_list);
 244  261          smb_server_inc_users(session->s_server);
 245  262  
↓ open down ↓ 13 lines elided ↑ open up ↑
 259  276  int
 260  277  smb_user_logon(
 261  278      smb_user_t          *user,
 262  279      cred_t              *cr,
 263  280      char                *domain_name,
 264  281      char                *account_name,
 265  282      uint32_t            flags,
 266  283      uint32_t            privileges,
 267  284      uint32_t            audit_sid)
 268  285  {
      286 +        ksocket_t authsock = NULL;
 269  287  
 270  288          ASSERT(user->u_magic == SMB_USER_MAGIC);
 271  289          ASSERT(cr);
 272  290          ASSERT(account_name);
 273  291          ASSERT(domain_name);
 274  292  
 275  293          mutex_enter(&user->u_mutex);
 276  294  
 277  295          if (user->u_state != SMB_USER_STATE_LOGGING_ON) {
 278  296                  mutex_exit(&user->u_mutex);
 279  297                  return (-1);
 280  298          }
 281  299  
 282      -        smb_authsock_close(user);
      300 +        /*
      301 +         * In the transition from LOGGING_ON to LOGGED_ON,
      302 +         * we always have an auth. socket to close.
      303 +         */
      304 +        authsock = user->u_authsock;
      305 +        user->u_authsock = NULL;
      306 +        if (user->u_auth_tmo != NULL) {
      307 +                (void) untimeout(user->u_auth_tmo);
      308 +                user->u_auth_tmo = NULL;
      309 +        }
 283  310  
 284  311          user->u_state = SMB_USER_STATE_LOGGED_ON;
 285  312          user->u_flags = flags;
 286  313          user->u_name_len = strlen(account_name) + 1;
 287  314          user->u_domain_len = strlen(domain_name) + 1;
 288  315          user->u_name = smb_mem_strdup(account_name);
 289  316          user->u_domain = smb_mem_strdup(domain_name);
 290  317          user->u_audit_sid = audit_sid;
 291  318  
 292  319          smb_user_setcred(user, cr, privileges);
 293  320  
 294  321          mutex_exit(&user->u_mutex);
 295  322  
      323 +        /* This close can block, so not under the mutex. */
      324 +        if (authsock != NULL)
      325 +                smb_authsock_close(user, authsock);
      326 +
 296  327          return (0);
 297  328  }
 298  329  
 299  330  /*
 300  331   * smb_user_logoff
 301  332   *
 302      - * Change the user state and disconnect trees.
      333 + * Change the user state to "logging off" and disconnect trees.
 303  334   * The user list must not be entered or modified here.
      335 + *
      336 + * We remain in state "logging off" until the last ref. is gone,
      337 + * then smb_user_release takes us to state "logged off".
 304  338   */
 305  339  void
 306  340  smb_user_logoff(
 307  341      smb_user_t          *user)
 308  342  {
      343 +        ksocket_t authsock = NULL;
      344 +
 309  345          ASSERT(user->u_magic == SMB_USER_MAGIC);
 310  346  
 311  347          mutex_enter(&user->u_mutex);
 312  348          ASSERT(user->u_refcnt);
 313  349          switch (user->u_state) {
 314      -        case SMB_USER_STATE_LOGGING_ON: {
 315      -                smb_authsock_close(user);
 316      -                user->u_state = SMB_USER_STATE_LOGGED_OFF;
 317      -                smb_server_dec_users(user->u_server);
      350 +        case SMB_USER_STATE_LOGGING_ON:
      351 +                authsock = user->u_authsock;
      352 +                user->u_authsock = NULL;
      353 +                if (user->u_auth_tmo != NULL) {
      354 +                        (void) untimeout(user->u_auth_tmo);
      355 +                        user->u_auth_tmo = NULL;
      356 +                }
      357 +                user->u_state = SMB_USER_STATE_LOGGING_OFF;
      358 +                mutex_exit(&user->u_mutex);
      359 +                /* This close can block, so not under the mutex. */
      360 +                if (authsock != NULL) {
      361 +                        smb_authsock_close(user, authsock);
      362 +                }
 318  363                  break;
 319      -        }
 320  364  
 321      -        case SMB_USER_STATE_LOGGED_ON: {
      365 +        case SMB_USER_STATE_LOGGED_ON:
 322  366                  /*
 323  367                   * The user is moved into a state indicating that the log off
 324  368                   * process has started.
 325  369                   */
 326  370                  user->u_state = SMB_USER_STATE_LOGGING_OFF;
 327  371                  mutex_exit(&user->u_mutex);
 328  372                  smb_session_disconnect_owned_trees(user->u_session, user);
 329  373                  smb_user_auth_logoff(user);
 330      -                mutex_enter(&user->u_mutex);
 331      -                user->u_state = SMB_USER_STATE_LOGGED_OFF;
 332      -                smb_server_dec_users(user->u_server);
 333  374                  break;
 334      -        }
      375 +
 335  376          case SMB_USER_STATE_LOGGED_OFF:
 336  377          case SMB_USER_STATE_LOGGING_OFF:
      378 +                mutex_exit(&user->u_mutex);
 337  379                  break;
 338  380  
 339  381          default:
 340  382                  ASSERT(0);
      383 +                mutex_exit(&user->u_mutex);
 341  384                  break;
 342  385          }
 343      -        mutex_exit(&user->u_mutex);
 344  386  }
 345  387  
 346  388  /*
 347  389   * Take a reference on a user.  Do not return a reference unless the user is in
 348  390   * the logged-in state.
 349  391   */
 350  392  boolean_t
 351  393  smb_user_hold(smb_user_t *user)
 352  394  {
 353  395          SMB_USER_VALID(user);
↓ open down ↓ 26 lines elided ↑ open up ↑
 380  422  /*
 381  423   * Release a reference on a user.  If the reference count falls to
 382  424   * zero and the user has logged off, post the object for deletion.
 383  425   * Object deletion is deferred to avoid modifying a list while an
 384  426   * iteration may be in progress.
 385  427   */
 386  428  void
 387  429  smb_user_release(
 388  430      smb_user_t          *user)
 389  431  {
 390      -        ASSERT(user->u_magic == SMB_USER_MAGIC);
      432 +        smb_session_t *ssn = user->u_session;
 391  433  
      434 +        SMB_USER_VALID(user);
      435 +
      436 +        /* flush the tree list delete queue */
      437 +        smb_llist_flush(&ssn->s_tree_list);
      438 +
 392  439          mutex_enter(&user->u_mutex);
 393  440          ASSERT(user->u_refcnt);
 394  441          user->u_refcnt--;
 395  442  
 396  443          switch (user->u_state) {
 397      -        case SMB_USER_STATE_LOGGED_OFF:
 398      -                if (user->u_refcnt == 0)
 399      -                        smb_session_post_user(user->u_session, user);
      444 +        case SMB_USER_STATE_LOGGING_OFF:
      445 +                if (user->u_refcnt == 0) {
      446 +                        smb_session_t *ssn = user->u_session;
      447 +                        user->u_state = SMB_USER_STATE_LOGGED_OFF;
      448 +                        smb_llist_post(&ssn->s_user_list, user,
      449 +                            smb_user_delete);
      450 +                }
 400  451                  break;
 401  452  
 402  453          case SMB_USER_STATE_LOGGING_ON:
 403  454          case SMB_USER_STATE_LOGGED_ON:
 404      -        case SMB_USER_STATE_LOGGING_OFF:
 405  455                  break;
 406  456  
      457 +        case SMB_USER_STATE_LOGGED_OFF:
 407  458          default:
 408  459                  ASSERT(0);
 409  460                  break;
 410  461          }
 411  462          mutex_exit(&user->u_mutex);
 412  463  }
 413  464  
 414  465  /*
      466 + * Timeout handler for user logons that stay too long in
      467 + * state SMB_USER_STATE_LOGGING_ON.  This is setup by a
      468 + * timeout call in smb_authsock_open, and called in a
      469 + * callout thread, so schedule a taskq job to do the
      470 + * real work of logging off this user.
      471 + */
      472 +void
      473 +smb_user_auth_tmo(void *arg)
      474 +{
      475 +        smb_user_t *user = arg;
      476 +        smb_request_t *sr;
      477 +
      478 +        SMB_USER_VALID(user);
      479 +
      480 +        /*
      481 +         * If we can't allocate a request, it means the
      482 +         * session is being torn down, so nothing to do.
      483 +         */
      484 +        sr = smb_request_alloc(user->u_session, 0);
      485 +        if (sr == NULL)
      486 +                return;
      487 +
      488 +        /*
      489 +         * Check user state, and take a hold if it's
      490 +         * still logging on.  If not, we're done.
      491 +         */
      492 +        mutex_enter(&user->u_mutex);
      493 +        if (user->u_state != SMB_USER_STATE_LOGGING_ON) {
      494 +                mutex_exit(&user->u_mutex);
      495 +                smb_request_free(sr);
      496 +                return;
      497 +        }
      498 +        /* smb_user_hold_internal */
      499 +        user->u_refcnt++;
      500 +        mutex_exit(&user->u_mutex);
      501 +
      502 +        /*
      503 +         * The user hold is given to the SR, and released in
      504 +         * smb_user_logoff_tq / smb_request_free
      505 +         */
      506 +        sr->uid_user = user;
      507 +        sr->user_cr = user->u_cred;
      508 +        sr->sr_state = SMB_REQ_STATE_SUBMITTED;
      509 +
      510 +        (void) taskq_dispatch(
      511 +            user->u_server->sv_worker_pool,
      512 +            smb_user_logoff_tq, sr, TQ_SLEEP);
      513 +}
      514 +
      515 +/*
      516 + * Helper for smb_user_auth_tmo()
      517 + */
      518 +static void
      519 +smb_user_logoff_tq(void *arg)
      520 +{
      521 +        smb_request_t   *sr = arg;
      522 +
      523 +        SMB_REQ_VALID(sr);
      524 +
      525 +        mutex_enter(&sr->sr_mutex);
      526 +        sr->sr_worker = curthread;
      527 +        sr->sr_state = SMB_REQ_STATE_ACTIVE;
      528 +        mutex_exit(&sr->sr_mutex);
      529 +
      530 +        smb_user_logoff(sr->uid_user);
      531 +
      532 +        sr->sr_state = SMB_REQ_STATE_COMPLETED;
      533 +        smb_request_free(sr);
      534 +}
      535 +
      536 +/*
 415  537   * Determine whether or not the user is an administrator.
 416  538   * Members of the administrators group have administrative rights.
 417  539   */
 418  540  boolean_t
 419  541  smb_user_is_admin(smb_user_t *user)
 420  542  {
 421  543  #ifdef  _KERNEL
 422  544          char            sidstr[SMB_SID_STRSZ];
 423  545          ksidlist_t      *ksidlist;
 424  546          ksid_t          ksid1;
↓ open down ↓ 91 lines elided ↑ open up ↑
 516  638  }
 517  639  
 518  640  /* *************************** Static Functions ***************************** */
 519  641  
 520  642  /*
 521  643   * Delete a user.  The tree list should be empty.
 522  644   *
 523  645   * Remove the user from the session's user list before freeing resources
 524  646   * associated with the user.
 525  647   */
 526      -void
      648 +static void
 527  649  smb_user_delete(void *arg)
 528  650  {
 529  651          smb_session_t   *session;
 530  652          smb_user_t      *user = (smb_user_t *)arg;
      653 +        uint32_t        ucount;
 531  654  
 532  655          SMB_USER_VALID(user);
 533  656          ASSERT(user->u_refcnt == 0);
 534  657          ASSERT(user->u_state == SMB_USER_STATE_LOGGED_OFF);
 535  658          ASSERT(user->u_authsock == NULL);
      659 +        ASSERT(user->u_auth_tmo == NULL);
 536  660  
 537  661          session = user->u_session;
      662 +
      663 +        smb_server_dec_users(session->s_server);
 538  664          smb_llist_enter(&session->s_user_list, RW_WRITER);
 539  665          smb_llist_remove(&session->s_user_list, user);
 540  666          smb_idpool_free(&session->s_uid_pool, user->u_uid);
      667 +        ucount = smb_llist_get_count(&session->s_user_list);
 541  668          smb_llist_exit(&session->s_user_list);
 542  669  
      670 +        if (ucount == 0) {
      671 +                smb_rwx_rwenter(&session->s_lock, RW_WRITER);
      672 +                session->s_state = SMB_SESSION_STATE_SHUTDOWN;
      673 +                smb_rwx_cvbcast(&session->s_lock);
      674 +                smb_rwx_rwexit(&session->s_lock);
      675 +        }
      676 +
      677 +        /*
      678 +         * This user is no longer on s_user_list, however...
      679 +         *
      680 +         * This is called via smb_llist_post, which means it may run
      681 +         * BEFORE smb_user_release drops u_mutex (if another thread
      682 +         * flushes the delete queue before we do).  Synchronize.
      683 +         */
 543  684          mutex_enter(&user->u_mutex);
 544  685          mutex_exit(&user->u_mutex);
 545  686  
 546  687          user->u_magic = (uint32_t)~SMB_USER_MAGIC;
 547  688          mutex_destroy(&user->u_mutex);
 548  689          if (user->u_cred)
 549  690                  crfree(user->u_cred);
 550  691          if (user->u_privcred)
 551  692                  crfree(user->u_privcred);
 552  693          smb_mem_free(user->u_name);
↓ open down ↓ 113 lines elided ↑ open up ↑
 666  807          ASSERT(user->u_name);
 667  808  
 668  809          session = user->u_session;
 669  810          ASSERT(session);
 670  811          ASSERT(session->workstation);
 671  812  
 672  813          info->ui_session_id = session->s_kid;
 673  814          info->ui_native_os = session->native_os;
 674  815          info->ui_ipaddr = session->ipaddr;
 675  816          info->ui_numopens = session->s_file_cnt;
 676      -        info->ui_smb_uid = user->u_uid;
 677  817          info->ui_logon_time = user->u_logon_time;
 678  818          info->ui_flags = user->u_flags;
 679  819          info->ui_posix_uid = crgetuid(user->u_cred);
 680  820  
 681  821          info->ui_domain_len = user->u_domain_len;
 682  822          info->ui_domain = smb_mem_strdup(user->u_domain);
 683  823  
 684  824          info->ui_account_len = user->u_name_len;
 685  825          info->ui_account = smb_mem_strdup(user->u_name);
 686  826  
↓ open down ↓ 13 lines elided ↑ open up ↑
 700  840          if (info->ui_domain)
 701  841                  smb_mem_free(info->ui_domain);
 702  842          if (info->ui_account)
 703  843                  smb_mem_free(info->ui_account);
 704  844          if (info->ui_workstation)
 705  845                  smb_mem_free(info->ui_workstation);
 706  846  
 707  847          bzero(info, sizeof (smb_netuserinfo_t));
 708  848  }
 709  849  
      850 +/*
      851 + * Tell smbd this user is going away so it can clean up their
      852 + * audit session, autohome dir, etc.
      853 + *
      854 + * Note that when we're shutting down, smbd will already have set
      855 + * smbd.s_shutting_down and therefore will ignore door calls.
      856 + * Skip this during shutdown to reduce upcall noise.
      857 + */
 710  858  static void
 711  859  smb_user_auth_logoff(smb_user_t *user)
 712  860  {
 713      -        uint32_t audit_sid = user->u_audit_sid;
      861 +        smb_server_t *sv = user->u_server;
      862 +        uint32_t audit_sid;
 714  863  
 715      -        (void) smb_kdoor_upcall(user->u_server, SMB_DR_USER_AUTH_LOGOFF,
      864 +        if (sv->sv_state != SMB_SERVER_STATE_RUNNING)
      865 +                return;
      866 +
      867 +        audit_sid = user->u_audit_sid;
      868 +        (void) smb_kdoor_upcall(sv, SMB_DR_USER_AUTH_LOGOFF,
 716  869              &audit_sid, xdr_uint32_t, NULL, NULL);
      870 +}
      871 +
      872 +boolean_t
      873 +smb_is_same_user(cred_t *cr1, cred_t *cr2)
      874 +{
      875 +        ksid_t *ks1 = crgetsid(cr1, KSID_USER);
      876 +        ksid_t *ks2 = crgetsid(cr2, KSID_USER);
      877 +
      878 +        return (ks1->ks_rid == ks2->ks_rid &&
      879 +            strcmp(ks1->ks_domain->kd_name, ks2->ks_domain->kd_name) == 0);
 717  880  }
    
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX