# [OmniOS-discuss] rpcbind: t_bind failed

Schweiss, Chip chip at innovates.com
Wed Jan 3 16:02:43 UTC 2018

- Previous message: [OmniOS-discuss] rpcbind: t_bind failed
- Next message: [OmniOS-discuss] rpcbind: t_bind failed
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

    The problem occurred again starting last night.  I have another clue, but I
    still don't know how it is occurring or how to fix it.

    It looks like all the TCP ports are in "bound" state, but not being
    released.

    How can I isolate the cause of this?

    # netstat -an

    UDP: IPv4
       Local Address        Remote Address      State
    -------------------- -------------------- ----------
          *.*                                 Unbound
          *.57921                             Idle
          *.42045                             Idle
          *.*                                 Unbound
          *.*                                 Unbound
          *.*                                 Unbound
          *.*                                 Unbound
          *.33757                             Idle
          *.37883                             Idle
          *.40744                             Idle
          *.4045                              Idle
          *.111                               Idle
          *.4045                              Idle
          *.*                                 Unbound
          *.60955                             Idle
          *.42908                             Idle
          *.56487                             Idle
          *.111                               Idle
          *.*                                 Unbound
          *.60994                             Idle
          *.*                                 Unbound
          *.2049                              Idle
          *.520                               Idle
          *.2049                              Idle
          *.46876                             Idle
          *.50309                             Idle
          *.123                               Idle
          *.123                               Idle
    127.0.0.1.123                             Idle
    10.28.125.29.123                          Idle
          *.57929                             Idle
          *.64351                             Idle
          *.63145                             Idle
          *.39674                             Idle
          *.65280                             Idle
          *.52013                             Idle
          *.47989                             Idle
          *.*                                 Unbound

    UDP: IPv6
       Local Address                     Remote Address
     State      If
    --------------------------------- ---------------------------------
    ---------- -----
          *.57921                                                       Idle
          *.*                                                           Unbound
          *.*                                                           Unbound
          *.40744                                                       Idle
          *.111                                                         Idle
          *.4045                                                        Idle
          *.*                                                           Unbound
          *.60955                                                       Idle
          *.42908                                                       Idle
          *.*                                                           Unbound
          *.2049                                                        Idle
          *.46876                                                       Idle
          *.123                                                         Idle
    ::1.123                                                             Idle

    TCP: IPv4
       Local Address        Remote Address    Swind Send-Q Rwind Recv-Q    State
    -------------------- -------------------- ----- ------ ----- ------
    -----------
          *.60571              *.*                0      0 1057280      0 BOUND
          *.46344              *.*                0      0 1057280      0 BOUND
          *.39729              *.*                0      0 1057280      0 BOUND
          *.43531              *.*                0      0 1057280      0 BOUND
          *.49051              *.*                0      0 1057280      0 BOUND
          *.44876              *.*                0      0 1057280      0 BOUND
          *.65416              *.*                0      0 1057280      0 BOUND
          *.47714              *.*                0      0 1057280      0 BOUND
          *.59055              *.*                0      0 1057280      0 BOUND
          *.45033              *.*                0      0 1057280      0 BOUND
          *.63321              *.*                0      0 1057280      0 BOUND
          *.43896              *.*                0      0 1057280      0 BOUND
          *.46627              *.*                0      0 1057280      0 BOUND
          *.35555              *.*                0      0 1057280      0 BOUND
          *.36115              *.*                0      0 1057280      0 BOUND
          *.51969              *.*                0      0 1057280      0 BOUND
          *.63741              *.*                0      0 1057280      0 BOUND
          *.45747              *.*                0      0 1057280      0 BOUND
          *.33245              *.*                0      0 1057280      0 BOUND
          *.49925              *.*                0      0 1057280      0 BOUND
          *.63503              *.*                0      0 1057280      0 BOUND
          *.45319              *.*                0      0 1057280      0 BOUND
          *.39977              *.*                0      0 1057280      0 BOUND

    ....lots of lines deleted...

          *.37396              *.*                0      0 1057280      0 BOUND
          *.33735              *.*                0      0 1057280      0 BOUND
          *.35695              *.*                0      0 1057280      0 BOUND
          *.36589              *.*                0      0 1057280      0 BOUND
          *.41484              *.*                0      0 1057280      0 BOUND
          *.63428              *.*                0      0 1057280      0 BOUND
          *.54891              *.*                0      0 1057280      0 BOUND
          *.60222              *.*                0      0 1057280      0 BOUND
          *.40494              *.*                0      0 1057280      0 BOUND

    TCP: IPv6
       Local Address                     Remote Address                 Swind
    Send-Q Rwind Recv-Q   State      If
    --------------------------------- --------------------------------- -----
    ------ ----- ------ ----------- -----
          *.54749                           *.*                             0
        0 128000      0 LISTEN
    ::1.5999                                *.*                             0
        0 128000      0 LISTEN
          *.4045                            *.*                             0
        0 1049200      0 LISTEN
          *.111                             *.*                             0
        0 128000      0 LISTEN
          *.*                               *.*                             0
        0 128000      0 IDLE
          *.45543                           *.*                             0
        0 128000      0 LISTEN
          *.2049                            *.*                             0
        0 1049200      0 LISTEN
          *.53926                           *.*                             0
        0 128000      0 LISTEN
          *.50379                           *.*                             0
        0 128000      0 LISTEN

    Active UNIX domain sockets
    Address  Type          Vnode     Conn  Local Addr      Remote Addr
    ffffd063569210e8 stream-ord 0000000 0000000
    ffffd06356921498 stream-ord 0000000 0000000
    ffffd06356921bf8 stream-ord 0000000 0000000
    ffffd063569260e0 stream-ord 0000000 0000000
    ffffd06356926490 stream-ord ffffd0635646d300 0000000 private/scache
    ffffd06356926840 stream-ord 0000000 0000000
    ffffd06356926bf0 stream-ord 0000000 0000000
    ffffd063569290d8 stream-ord ffffd0635646d200 0000000 private/anvil
    ffffd06356929488 stream-ord 0000000 0000000
    ffffd06356929838 stream-ord 0000000 0000000
    ffffd06356929be8 stream-ord ffffd06356810480 0000000 private/lmtp
    ffffd0635692d0d0 stream-ord 0000000 0000000
    ffffd0635692d480 stream-ord 0000000 0000000
    ffffd0635692d830 stream-ord ffffd06356810280 0000000 private/virtual
    ffffd0635692dbe0 stream-ord 0000000 0000000
    ffffd063569320c8 stream-ord 0000000 0000000
    ffffd06356932478 stream-ord ffffd0635685aa00 0000000 private/local
    ffffd06356932828 stream-ord 0000000 0000000
    ffffd06356932bd8 stream-ord 0000000 0000000
    ffffd063569360c0 stream-ord ffffd0635685ad00 0000000 private/discard
    ffffd06356936470 stream-ord 0000000 0000000
    ffffd06356936820 stream-ord 0000000 0000000
    ffffd06356936bd0 stream-ord ffffd0635685ab00 0000000 private/retry
    ffffd0635693b0b8 stream-ord 0000000 0000000
    ffffd0635693b468 stream-ord 0000000 0000000
    ffffd0635693b818 stream-ord ffffd0635685ae00 0000000 private/error
    ffffd0635693bbc8 stream-ord 0000000 0000000
    ffffd063568e10b0 stream-ord 0000000 0000000
    ffffd063568e1460 stream-ord ffffd0635685a400 0000000 public/showq
    ffffd063568e1810 stream-ord 0000000 0000000
    ffffd063568e1bc0 stream-ord 0000000 0000000
    ffffd063568e60a8 stream-ord ffffd0635685a600 0000000 private/relay
    ffffd063568e6458 stream-ord 0000000 0000000
    ffffd063568e6808 stream-ord 0000000 0000000
    ffffd063568e6bb8 stream-ord ffffd0635685a900 0000000 private/smtp
    ffffd063568ea0a0 stream-ord 0000000 0000000
    ffffd063568ea450 stream-ord 0000000 0000000
    ffffd063568ea800 stream-ord ffffd0635646d100 0000000 private/proxywrite
    ffffd063568eabb0 stream-ord 0000000 0000000
    ffffd063568f0098 stream-ord 0000000 0000000
    ffffd063568f0448 stream-ord ffffd0635685ac00 0000000 private/proxymap
    ffffd063568f07f8 stream-ord 0000000 0000000
    ffffd063568f0ba8 stream-ord 0000000 0000000
    ffffd063568f2090 stream-ord ffffd0635685a200 0000000 public/flush
    ffffd063568f2440 stream-ord 0000000 0000000
    ffffd063568f27f0 stream-ord 0000000 0000000
    ffffd063568f2ba0 stream-ord ffffd0635685a500 0000000 private/verify
    ffffd063568f9088 stream-ord 0000000 0000000
    ffffd063568f9438 stream-ord 0000000 0000000
    ffffd063568f97e8 stream-ord ffffd06356810080 0000000 private/trace
    ffffd063568f9b98 stream-ord 0000000 0000000
    ffffd063568fd080 stream-ord 0000000 0000000
    ffffd063568fd430 stream-ord ffffd06356810180 0000000 private/defer
    ffffd063568fd7e0 stream-ord 0000000 0000000
    ffffd063568fdb90 stream-ord 0000000 0000000
    ffffd06356840078 stream-ord ffffd0635685a700 0000000 private/bounce
    ffffd06356840428 stream-ord 0000000 0000000
    ffffd063568407d8 stream-ord 0000000 0000000
    ffffd06356840b88 stream-ord ffffd0635685a800 0000000 private/rewrite
    ffffd06356843070 stream-ord ffffd06356810380 0000000 private/tlsmgr
    ffffd06356843420 stream-ord 0000000 0000000
    ffffd063568437d0 stream-ord 0000000 0000000
    ffffd06356849068 stream-ord 0000000 0000000
    ffffd06356849418 stream-ord 0000000 0000000
    ffffd063568497c8 stream-ord ffffd0635685a000 0000000 public/qmgr
    ffffd06356849b78 stream-ord ffffd0635685a100 0000000 public/cleanup
    ffffd0635684d060 stream-ord 0000000 0000000
    ffffd0635684d410 stream-ord 0000000 0000000
    ffffd0635684db70 stream-ord 0000000 0000000
    ffffd06355646058 stream-ord 0000000 0000000
    ffffd06355646b68 stream-ord ffffd0635685a300 0000000 public/pickup
    ffffd063551bf3f8 stream-ord ffffd063193fe900 0000000 /var/run/.inetd.uds
    ffffd063550e7b50 dgram      ffffd063550eb380 0000000 /var/run/in.rdisc_mib
    ffffd06355031798 dgram      ffffd063536c8800 0000000 /var/run/in.ndpd_mib
    ffffd06355031b48 stream-ord ffffd063536c8c00 0000000 /var/run/in.ndpd_ipadm
    ffffd0635265a028 stream-ord 0000000 ffffd0634e4acd00
    /var/run/dbus/system_bus_socket
    ffffd0635265a788 stream-ord 0000000 ffffd063500ffc80
    /var/run/hald/dbus-y1Me9kLIpf
    ffffd0635265ab38 stream-ord 0000000 0000000 /var/run/hald/dbus-y1Me9kLIpf
    ffffd06351d553d0 stream-ord 0000000 0000000 /var/run/hald/dbus-y1Me9kLIpf
    ffffd06351d55780 stream-ord 0000000 0000000 /var/run/hald/dbus-y1Me9kLIpf
    ffffd06351d55b30 stream-ord 0000000 ffffd063500ffc80
    /var/run/hald/dbus-y1Me9kLIpf
    ffffd06351996018 stream-ord 0000000 ffffd063500ffc80
    /var/run/hald/dbus-y1Me9kLIpf
    ffffd063519963c8 stream-ord 0000000 0000000 /var/run/hald/dbus-y1Me9kLIpf
    ffffd06351996778 stream-ord 0000000 ffffd063500ffc80
    /var/run/hald/dbus-y1Me9kLIpf
    ffffd063500fe010 stream-ord 0000000 0000000 /var/run/hald/dbus-5Qrha0Wmu3
    ffffd063500fe3c0 stream-ord 0000000 ffffd063500ffa80
    /var/run/hald/dbus-5Qrha0Wmu3
    ffffd063500fe770 stream-ord ffffd063500ffa80 0000000
    /var/run/hald/dbus-5Qrha0Wmu3
    ffffd063500feb20 stream-ord ffffd063500ffc80 0000000
    /var/run/hald/dbus-y1Me9kLIpf
    ffffd0634e4ad008 stream-ord 0000000 0000000
    ffffd0634e4ad3b8 stream-ord 0000000 0000000
    ffffd0634e4ad768 stream-ord 0000000 0000000 /var/run/dbus/system_bus_socket
    ffffd0634e4adb18 stream-ord ffffd0634e4acd00 0000000
    /var/run/dbus/system_bus_socket

    A sorted output shows nearly all 64K ports in bound state.

    On Tue, Jan 2, 2018 at 8:40 AM, Schweiss, Chip <chip at innovates.com> wrote:

    > About once every week or two I'm having NFS connections start to collapse
    > to one of my servers.   Clients will lose thier connections of the the
    > course of several hours. The logs fill with these messages:
    >
    > Dec 25 16:21:14 mir-zfs03 rpcbind: [ID 452059 daemon.error]  do_accept :
    > t_bind failed : Couldn't allocate address
    > Dec 25 16:21:14 mir-zfs03 /usr/lib/nfs/nfsd[27689]: [ID 396295
    > daemon.error] t_bind(file descriptor 188/transport tcp) TLI error 5
    > Dec 25 16:21:31 mir-zfs03 last message repeated 85 times
    > Dec 25 16:21:31 mir-zfs03 rpcbind: [ID 452059 daemon.error]  do_accept :
    > t_bind failed : Couldn't allocate address
    > Dec 25 16:21:32 mir-zfs03 /usr/lib/nfs/nfsd[27689]: [ID 396295
    > daemon.error] t_bind(file descriptor 188/transport tcp) TLI error 5
    > Dec 25 16:21:34 mir-zfs03 last message repeated 19 times
    > Dec 25 16:21:37 mir-zfs03 /usr/lib/nfs/nfsd[27689]: [ID 396295
    > daemon.error] t_bind(file descriptor 200/transport tcp) TLI error 5
    > Dec 25 16:22:17 mir-zfs03 last message repeated 116 times
    > Dec 25 16:22:21 mir-zfs03 /usr/lib/nfs/nfsd[27689]: [ID 396295
    > daemon.error] t_bind(file descriptor 206/transport tcp) TLI error 5
    > Dec 25 16:23:04 mir-zfs03 last message repeated 81 times
    >
    > This is a fully updated OmniOS CE r151022.
    >
    > I've tried restarting NFS services, but the only thing that has been
    > successful in restoring services has been rebooting.
    >
    > I'm not finding anything useful via Google except the source code that
    > spits out this message.   HP-UX appears to have had the same issue that
    > they patched years ago.   I'm guessing shared NFS/RPC code.
    >
    > Any clue as to the cause of this and how to fix?
    >
    > -Chip
    >
    >
    >
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20180103/d9b516ea/attachment-0001.html>

- Previous message: [OmniOS-discuss] rpcbind: t_bind failed
- Next message: [OmniOS-discuss] rpcbind: t_bind failed
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the OmniOS-discuss mailing list