# [OmniOS-discuss] infiniband

Narayan Desai narayan.desai at gmail.com
Fri Oct 24 07:07:59 UTC 2014

- Previous message: [OmniOS-discuss] Ang: Re: Ang: Re: infiniband
- Next message: [OmniOS-discuss] Initial review --> Changes that allow illumos-gate to (mostly) build on OmniOS
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

    On Thu, Oct 23, 2014 at 10:36 PM, Michael Rasmussen <mir at miras.org> wrote:

    > On Fri, 24 Oct 2014 12:08:14 +1100
    > David Bomba <turbo124 at gmail.com> wrote:
    >
    > > Correct, they are 40Gb/s cards.
    > I have heard same stories of problems with 40Gb/s cards on Linux and
    > FreeBSD. What it boils down to is that at this speed software subnet
    > manager is causing problems so always use a switch with hardware subnet
    > manager.
    >

    This sounds apocryphal and wrong to me. I know of a several large
    infiniband networks (1k+ host ports, non-blocking) that run with subnet
    managers running on linux machines and generally work pretty well. A few
    remarks:
     - subnet managers are a part of the control plane, they don't participate
    directly in the data plane after initial connection setup. This means that
    the footprint of a QDR port is the same as an SDR port; data packets aren't
    processed through the subnet manager.
     - subnet managers aren't bug-free -- the versions integrated into switch
    firmware can't be upgraded a la carte, so you are dependent on the vendor
    to release a new version of the switch firmware to get bugfixes.
     - the processors included in switches are embedded and hence slow. this
    could easily make a difference to subnet performance, if you need to talk
    to the subnet manager frequently to get new path records.

    I'd strongly recommend against running switch provided subnet managers.
     -nld
    -------------- next part --------------
    An HTML attachment was scrubbed...
    URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20141024/c8bb9b4b/attachment-0001.html>

- Previous message: [OmniOS-discuss] Ang: Re: Ang: Re: infiniband
- Next message: [OmniOS-discuss] Initial review --> Changes that allow illumos-gate to (mostly) build on OmniOS
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the OmniOS-discuss mailing list