# [OmniOS-discuss] [developer] Re: [discuss] Trouble getting Supermicro board to see all its PCI-e slots

Sašo Kiselkov skiselkov.ml at gmail.com
Wed May 1 20:28:25 EDT 2013

- Previous message: [OmniOS-discuss] [developer] Re: [discuss] Trouble getting Supermicro board to see all its PCI-e slots
- Next message: [OmniOS-discuss] [developer] Re: [discuss] Trouble getting Supermicro board to see all its PCI-e slots
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

    Hi Aakash,

    First of all, thanks for the many pointers here, I cannot begin to
    describe the gratitude for your help. My responses to your suggestions
    below.

    On 05/01/2013 08:36 PM, Aakash Saini wrote:
    >> ..."Strangely, however, prtdiag sees the slots as populated:"...
    >
    > I don't understand how it could be pci_boot/create_bus issues; cause pci_boot
    > is functioning and allocating the buses *correctly* to relevant slots. As
    > stated, Left-hand has 4::x8-PCI-e slots (2-cpu1, 2-cpu2) and right-hand has
    > 2::x8-PCI-e slots (2-cpu2), which we see through prtdiag details about
    > slots (1-to-6) assignments.
    >
    > ..which means socket 00 is working correctly, and socket 01 has some
    > fault!!. surprising, very much nailed correctly relating to QPI
    > interconnect issues. i guess PCH/C6xxx-chipset and motherboard specs need
    > to be reviewed.
    >
    > prtpicl should display correctly details to resource addressing, cause
    > slots are assigned.

    Attached, both the short and verbose version.

    >> ..."The problem appears to be that Illumos doesn't recognize any PCI
    > bridges beyond those attached to CPU1.
    >
    > yes, maybe its a PCIe-PCI bridging issue, even i doubt. try kdb debug
    > state...
    > ::bp pcieb`pcieb_attach
    > :c
    > pcieb`pcieb_dbg_print/W 0x1    --- this should display basic
    > debugged information
    > :c

    Compiled a debug version of the pcieb module and attached the output
    from this. Strangely enough, the output stopped for considerable time
    between lines 21 and 22 in the output, so between lines:

    pcieb(0): pwrINITCHILD: config regs setup for pci15d9,690 at 0

    and

    pcieb(1): pwrpcieb_pwr_disable: disabling PM

    after which lots of output proceeded quickly. Could it be that the
    probing code is getting stuck on and then giving up on probing a bridge
    here?

    > also. try with disabled (both) MSI and workaround patch to 41210/chipset
    > individually.
    > pcieb`pcieb_enable_msi/W 0x1
    > pcieb_disable_41210_wkarnd/W 0x1

    Neither setting changed anything.

    > i'm not sure if there is any resource conflict issue; mdb -k | ::interrupts
    > as obviously, h/w is remote!

    Here's the output of that:

    # echo ::interrupts | mdb -k
    CPU/Vect  IRQ IPL Bus    Trg Type   Share APIC/INT# ISR
    1/0x20    9   9   PCI    Lvl Fixed  1     0x0/0x9   acpi_wrapper_isr
    2/0x20    11  14  PCI    Lvl Fixed  1     0x0/0xb   hpet_isr
    3/0x20    -   7   PCI    Edg MSI    1     -         pcieb_intr_handler
    4/0x20    -   5   PCI    Edg MSI-X  1     -         mrsas_isr
    7/0x20    1   5   ISA    Edg Fixed  1     0x0/0x1   i8042_intr
    8/0x20    12  5   ISA    Edg Fixed  1     0x0/0xc   i8042_intr
    8/0x22    -   5   PCI    Edg MSI    1     -         mpt_intr
    12/0x20   3   12  ISA    Edg Fixed  1     0x0/0x3   asyintr
    13/0x20   -   4   PCI    Edg MSI    1     -         pcieb_intr_handler
    14/0x20   -   7   PCI    Edg MSI    1     -         pcieb_intr_handler
    15/0x20   -   4   PCI    Edg MSI    1     -         pcieb_intr_handler
    16/0x20   -   7   PCI    Edg MSI    1     -         pcieb_intr_handler
    17/0x20   -   4   PCI    Edg MSI    1     -         pcieb_intr_handler
    18/0x20   16  9   PCI    Lvl Fixed  1     0x0/0x10  ehci_intr
    19/0x20   23  9   PCI    Lvl Fixed  1     0x0/0x17  ehci_intr
    20/0x20   -   6   PCI    Edg MSI-X  1     -         igb_intr_tx_other
    21/0x20   -   6   PCI    Edg MSI-X  1     -         igb_intr_rx
    all/0xf0  -   15  -      Edg IPI    1     -         xc_serv
    all/0xf1  -   11  -      Edg IPI    0     -         poke_cpu
    all/0xf2  -   14  -      Edg IPI    1     -         kcpc_hw_overflow_intr
    all/0xf3  -   15  -      Edg IPI    1     -         apic_error_intr
    all/0xf4  -   2   -      Edg IPI    1     -         cmi_cmci_trap
    all/0xf5  -   14  -      Edg IPI    1     -         cbe_fire
    all/0xf6  -   14  -      Edg IPI    1     -         cbe_fire

    Hope this provides more insight. If it helps, I can arrange for direct
    access to the machine (both to the OS and the BMC).

    Cheers,
    --
    Saso
    -------------- next part --------------
      / (picl, f900000001)
         platform (upa, f900000005)
             scsi_vhci (devctl, f90000005c)
             pci (pciex, f900000079)
                 pci15d9,720 (obp-device, f90000008e)
                 pci8086,3c02 (pciex, f9000000af)
                     pci15d9,690 (obp-device, f9000000d6)
                         sd (block, f900000108)
                 pci8086,3c04 (pciex, f900000126)
                     pci15d9,10fb (obp-device, f90000014e)
                     pci15d9,10fb (obp-device, f900000179)
                 pci8086,3c06 (pciex, f9000001a3)
                     pci10e3,8113 (pci, f9000001ca)
                         display (display, f9000001ee)
                 pci8086,3c07 (pciex, f900000214)
                     pci15d9,1521 (obp-device, f90000023b)
                     pci15d9,1521 (obp-device, f900000266)
                 pci8086,3c08 (pciex, f900000290)
                     pci103c,322d (obp-device, f9000002b8)
                 pci8086,3c0a (pciex, f9000002f6)
                 pci15d9,720 (obp-device, f90000031a)
                 pci15d9,720 (obp-device, f90000033c)
                 pci15d9,720 (obp-device, f90000035e)
                 pci15d9,720 (obp-device, f900000380)
                 pci15d9,720 (obp-device, f9000003a2)
                 pci15d9,720 (obp-device, f9000003c4)
                 pci15d9,720 (obp-device, f9000003e6)
                 pci15d9,720 (obp-device, f900000408)
                 pci15d9,720 (obp-device, f90000042a)
                 pci15d9,720 (obp-device, f90000044a)
                 pci15d9,720 (obp-device, f90000046a)
                 pci8086,1d3e (pciex, f900000487)
                     pci15d9,720 (obp-device, f9000004ad)
                 pci15d9,720 (obp-device, f9000004cf)
                 pci15d9,720 (obp-device, f9000004ee)
                 pci15d9,720 (obp-device, f90000050d)
                     hub (obp-device, f900000537)
                         device (obp-device, f900000563)
                             mouse (obp-device, f900000583)
                             keyboard (obp-device, f90000059b)
                 pci15d9,720 (obp-device, f9000005b3)
                     hub (obp-device, f9000005dd)
                         storage (obp-device, f900000609)
                             disk (block, f900000637)
                 pci8086,244e (pci, f900000657)
                 isa (isa, f900000679)
                     motherboard (obp-device, f90000069c)
                     i8042 (obp-device, f9000006a6)
                         mouse (obp-device, f9000006b1)
                     asy (serial, f9000006c0)
                     asy (serial, f9000006cf)
                     motherboard (obp-device, f9000006de)
                     motherboard (obp-device, f9000006e8)
                 pci15d9,720 (obp-device, f9000006f2)
                 pci15d9,720 (obp-device, f900000723)
                 pci15d9,720 (obp-device, f900000742)
             pci (pciex, f900000778)
                 pci15d9,720 (obp-device, f90000078b)
                 pci15d9,720 (obp-device, f9000007a7)
                 pci15d9,720 (obp-device, f9000007c7)
                 pci15d9,720 (obp-device, f9000007e7)
                 pci15d9,720 (obp-device, f900000803)
                 pci15d9,720 (obp-device, f900000823)
                 pci15d9,720 (obp-device, f900000843)
                 pci15d9,720 (obp-device, f90000085f)
                 pci15d9,720 (obp-device, f90000087b)
                 pci15d9,720 (obp-device, f900000897)
                 pci15d9,720 (obp-device, f9000008b3)
                 pci15d9,720 (obp-device, f9000008cf)
                 pci15d9,720 (obp-device, f9000008eb)
                 pci15d9,720 (obp-device, f900000907)
                 pci15d9,720 (obp-device, f900000923)
                 pci15d9,720 (obp-device, f90000093f)
                 pci15d9,720 (obp-device, f90000095b)
                 pci15d9,720 (obp-device, f900000977)
                 pci15d9,720 (obp-device, f900000993)
                 pci15d9,720 (obp-device, f9000009af)
                 pci15d9,720 (obp-device, f9000009cb)
                 pci15d9,720 (obp-device, f9000009e7)
                 pci15d9,720 (obp-device, f900000a03)
                 pci15d9,720 (obp-device, f900000a1f)
                 pci15d9,720 (obp-device, f900000a3f)
                 pci15d9,720 (obp-device, f900000a5f)
                 pci15d9,720 (obp-device, f900000a7f)
                 pci15d9,720 (obp-device, f900000a9f)
                 pci15d9,720 (obp-device, f900000abf)
                 pci15d9,720 (obp-device, f900000adf)
                 pci15d9,720 (obp-device, f900000afb)
                 pci15d9,720 (obp-device, f900000b1b)
                 pci15d9,720 (obp-device, f900000b3b)
                 pci15d9,720 (obp-device, f900000b5b)
                 pci15d9,720 (obp-device, f900000b7b)
                 pci15d9,720 (obp-device, f900000b9b)
                 pci15d9,720 (obp-device, f900000bbb)
                 pci15d9,720 (obp-device, f900000bdb)
                 pci8086,0 (obp-device, f900000bfb)
                 pci15d9,720 (obp-device, f900000c17)
                 pci15d9,720 (obp-device, f900000c33)
                 pci15d9,720 (obp-device, f900000c4f)
                 pci15d9,720 (obp-device, f900000c6b)
                 pci15d9,720 (obp-device, f900000c87)
             pci (pciex, f900000ca3)
                 pci8086,3c01 (pciex, f900000cb3)
                 pci8086,3c02 (pciex, f900000cd6)
                 pci8086,3c04 (pciex, f900000cf9)
                     pci103c,322d (obp-device, f900000d1d)
                 pci8086,3c06 (pciex, f900000d40)
                 pci8086,3c08 (pciex, f900000d64)
                 pci8086,3c0a (pciex, f900000d88)
                 pci15d9,720 (obp-device, f900000dac)
                 pci15d9,720 (obp-device, f900000dce)
                 pci15d9,720 (obp-device, f900000df0)
                 pci15d9,720 (obp-device, f900000e12)
                 pci15d9,720 (obp-device, f900000e34)
                 pci15d9,720 (obp-device, f900000e56)
                 pci15d9,720 (obp-device, f900000e78)
                 pci15d9,720 (obp-device, f900000e9a)
                 pci15d9,720 (obp-device, f900000ebc)
                 pci15d9,720 (obp-device, f900000edc)
                 pci15d9,720 (obp-device, f900000efc)
             fw (acpirootnex, f900000f19)
                 sb (acpidevice, f900000f2f)
                     socket (acpicontainer, f900000f45)
                         cpu (cpu, f900000f5d)
                         cpu (cpu, f900000f9c)
                         cpu (cpu, f900000fdb)
                         cpu (cpu, f90000101a)
                         cpu (cpu, f900001059)
                         cpu (cpu, f900001098)
                         cpu (cpu, f9000010d7)
                         cpu (cpu, f900001116)
                         cpu (cpu, f900001155)
                         cpu (cpu, f900001194)
                         cpu (cpu, f9000011d3)
                         cpu (cpu, f900001212)
                     socket (acpicontainer, f900001251)
                         cpu (cpu, f900001269)
                         cpu (cpu, f9000012a8)
                         cpu (cpu, f9000012e7)
                         cpu (cpu, f900001326)
                         cpu (cpu, f900001365)
                         cpu (cpu, f9000013a4)
                         cpu (cpu, f9000013e3)
                         cpu (cpu, f900001422)
                         cpu (cpu, f900001461)
                         cpu (cpu, f9000014a0)
                         cpu (cpu, f9000014df)
                         cpu (cpu, f90000151e)
             agpgart (agpgart, f900001580)
             pseudo (devctl, f900001593)
                 zconsnex (devctl, f90000159c)
                 fct (admin, f9000015a5)
                 stmf (admin, f9000015ad)
                 stmf_sbd (admin, f9000015b5)
                 ppm (ppm, f9000015be)
                 power (power_button, f90000164e)
         obp (picl, f900000059)
             ioapics (ioapics, f900000760)
                 ioapic (ioapic, f900000766)
                 ioapic (ioapic, f90000076f)
             used-resources (used-resources, f90000155d)
             iscsi (iscsi, f900001567)
             options (options, f900001589)
    -------------- next part --------------
    A non-text attachment was scrubbed...
    Name: prtpicl-v.txt.gz
    Type: application/x-gzip
    Size: 18878 bytes
    Desc: not available
    URL: <http://lists.omniti.com/pipermail/omnios-discuss/attachments/20130502/f75eca5c/attachment-0001.bin>
    -------------- next part --------------
    Loading kmdb...

    Welcome to kmdb
    kmdb: unable to determine terminal type: assuming `vt100'
    Loaded modules: [ unix krtld genunix ]
    [0]> ::bp pcieb`pcieb_attach
    [0]> :c
    SunOS Release 5.11 Version omnios-bc85f2d 64-bit
    Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
    Loaded modules: [ scsi_vhci mac uppc zfs apix specfs pcplusmp cpu.generic ]
    kmdb: stop at pcieb`pcieb_attach
    kmdb: target stopped at:
    pcieb`pcieb_attach:     pushq  %rbp
    [0]> pcieb`pcieb_dbg_print/W 0x1
    pcieb`pcieb_dbg_print:          0               =       0x1
    [0]> ::delete 1
    [0]> :c
    pcieb(0): pwrpcieb_pwr_disable: disabling PM
    pcieb(0): attachpcieb_intr_init: Attaching MSI handler
    pcieb(0): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 8pcieb(0): attachVID:0x8086 DID:0x3c02 RID:0x7 bdf=0x8
    pcieb(0): pwrINITCHILD: config regs setup for pci15d9,690 at 0
    pcieb(1): pwrpcieb_pwr_disable: disabling PM
    pcieb(1): attachpcieb_intr_init: Attaching MSI handler
    pcieb(1): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 16pcieb(1): attachVID:0x8086 DID:0x3c04 RID:0x7 bdf=0x10
    pcieb(2): pwrpcieb_pwr_disable: disabling PM
    pcieb(2): attachpcieb_intr_init: Attaching MSI handler
    pcieb(2): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 18pcieb(2): attachVID:0x8086 DID:0x3c06 RID:0x7 bdf=0x12
    pcieb(3): pwrpcieb_pwr_disable: disabling PM
    pcieb(3): attachpcieb_intr_init: Attaching MSI handler
    pcieb(3): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 19pcieb(3): attachVID:0x8086 DID:0x3c07 RID:0x7 bdf=0x13
    pcieb(4): pwrpcieb_pwr_disable: disabling PM
    pcieb(4): attachpcieb_intr_init: Attaching MSI handler
    pcieb(4): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 24pcieb(4): attachVID:0x8086 DID:0x3c08 RID:0x7 bdf=0x18
    pcieb(6): pwrpcieb_pwr_disable: disabling PM
    pcieb(6): attachpcieb_intr_init: Attaching MSI handler
    pcieb(6): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 136pcieb(6): attachVID:0x8086 DID:0x1d3e RID:0x6 bdf=0x88
    pcieb(2): pwrINITCHILD: config regs setup for pci10e3,8113 at 0
    pcieb(7): pwrpcieb_pwr_disable: disabling PM
    pcieb(7): attachpcieb_intr_init: Attaching INTx handler
    pcieb(7): attachVID:0x10e3 DID:0x8113 RID:0x1 bdf=0x700
    pcieb(7): pwrINITCHILD: config regs setup for display at 1
    pcieb(3): pwrINITCHILD: config regs setup for pci15d9,1521 at 0
    Hostname: head2
    pcieb(3): pwrINITCHILD: config regs setup for pci15d9,1521 at 0,1
    pcieb(1): pwrINITCHILD: config regs setup for pci15d9,10fb at 0
    pcieb(1): pwrINITCHILD: config regs setup for pci15d9,10fb at 0,1
    pcieb(4): pwrINITCHILD: config regs setup for pci103c,322d at 0

    head2 console login:

- Previous message: [OmniOS-discuss] [developer] Re: [discuss] Trouble getting Supermicro board to see all its PCI-e slots
- Next message: [OmniOS-discuss] [developer] Re: [discuss] Trouble getting Supermicro board to see all its PCI-e slots
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

More information about the OmniOS-discuss mailing list