`<div dir="ltr"><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">>..."Strangely, however, prtdiag sees the slots as populated:"...</span><br><div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px"><br>
</span></div><div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">I don't understand how it could be </span><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">pci_boot/create_bus issues; cause </span><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">pci_boot is functioning and allocating the buses <b>correctly</b> to relevant slots. As stated, </span><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">Left-hand has 4::</span><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">x8-PCI-e slots (2-cpu1, 2-cpu2) and r</span><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">ight-hand has 2::</span><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">x8-PCI-e slots (2-cpu2), which we see through prtdiag details about slots </span><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">(1-to-6) </span><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">assignments.</span></div>
<div style><br></div><div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">..which means socket 00 is working correctly, and socket 01 has some fault!!. surprising, very much nailed correctly relating to QPI interconnect issues. i guess PCH/C6xxx-chipset and motherboard specs need to be reviewed.</span></div>
<div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px"><br></span></div><div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">prtpicl should display correctly details to resource addressing, cause slots are assigned.</span><br>
</div><div style><span style="font-size:13px;color:rgb(0,0,0);font-family:arial,sans-serif"><br></span></div><div style><span style="font-size:13px;color:rgb(0,0,0);font-family:arial,sans-serif">>..."The problem appears to be </span><span style="font-size:13px;color:rgb(0,0,0);font-family:arial,sans-serif">that Illumos doesn't recognize any PCI bridges beyond those attached to </span><span style="font-size:13px;color:rgb(0,0,0);font-family:arial,sans-serif">CPU1.</span><br>
</div><div style><span style="font-size:13px;color:rgb(0,0,0);font-family:arial,sans-serif"><br></span></div><div style><span style="font-size:13px;color:rgb(0,0,0);font-family:arial,sans-serif">yes, maybe its a </span><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">PCIe-PCI </span><span style="font-size:13px;color:rgb(0,0,0);font-family:arial,sans-serif">bridging issue, even i doubt. </span><span style="font-size:13px;color:rgb(0,0,0);font-family:arial,sans-serif">try </span><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">kdb </span><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">debug state...</span></div>
<div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">::bp pcieb`pcieb_attach</span><br></div><div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif;font-size:13px">:c</span><br>
</div><div style><span style="font-size:13px;color:rgb(0,0,0);font-family:arial,sans-serif">pcieb`</span><font color="#000000" face="arial, sans-serif">pcieb_dbg_print/W 0x1 --- this should display basic debugged information</font><br>
</div><div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif">:c</span><br></div><div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif"><br></span></div><div style>also. <span style="color:rgb(0,0,0);font-family:arial,sans-serif">try with disabled (both) MSI and workaround patch to </span><font color="#000000" face="arial, sans-serif">41210/chipset individually.</font><br>
</div><div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif">pcieb`pcieb_enable_msi/W 0x1</span><br></div><div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif">pcieb_disable_41210_wkarnd/W 0x1</span><br>
</div><div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif"><br></span></div><div style><span style="color:rgb(0,0,0);font-family:arial,sans-serif">i'm not sure if there is any resource conflict issue; mdb -k | ::interrupts</span></div>
<div style><font color="#000000" face="arial, sans-serif">as obviously, h/w is remote!</font></div><div style><br></div><div class="gmail_extra"><div><div dir="ltr"><br><br>-----------------------------------<br>Aakash Saini<br>
</div></div>
<br><br><div class="gmail_quote">On Wed, May 1, 2013 at 6:54 AM, Garrett D'Amore <span dir="ltr"><<a href="mailto:garrett@damore.org" target="_blank">garrett@damore.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Check the value of pci_bios_maxbus.<br>
<br>
I have a sneaking feeling that the problem may be related to acpi enumeration of the bus. We are limited to seeing the busses that ACPI properly enumerates. Take a look at the code in pci_boot.c for more details. You should especially note the call sites for create_pcie_root_bus.<br>
<br>
I'd love to help further, but lack of access to hardware that exhibits this problem prevents me from doing any active debug myself.<br>
<br>
- Garrett<br>
<div><div class="h5"><br>
On May 1, 2013, at 12:50 AM, Sašo Kiselkov <<a href="mailto:skiselkov.ml@gmail.com">skiselkov.ml@gmail.com</a>> wrote:<br>
<br>
> Just a follow up on this: this appears to be an Illumos bug, though I<br>
> haven't figured out how to resolve it yet. The problem appears to be<br>
> that Illumos doesn't recognize any PCI bridges beyond those attached to<br>
> CPU1. Notably, nothing on CPU2 appears to work. I've tested with Linux<br>
> 3.5 and that works flawless, attached are the lspci -vvvv outputs.<br>
><br>
> The problem appears to be that Illumos stops after PCI address 0b:00.0<br>
> (on-board MegaRAID SAS 2208). After that the scan should proceed with<br>
> the QPI inter-processor links, but doesn't.<br>
><br>
> The full lspci should look something like this:<br>
><br>
> 00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI..<br>
> 00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..<br>
> 00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..<br>
> 00:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..<br>
> 00:02.3 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..<br>
> 00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..<br>
> 00:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..<br>
> 00:04.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 00:04.1 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 00:04.2 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 00:04.3 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 00:04.4 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 00:04.5 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 00:04.6 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 00:04.7 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 00:05.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 00:05.2 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (r..<br>
> 00:11.0 PCI bridge: Intel Corporation C600/X79 series chip..<br>
> 00:16.0 Communication controller: Intel Corporation C600/X..<br>
> 00:16.1 Communication controller: Intel Corporation C600/X..<br>
> 00:1a.0 USB controller: Intel Corporation C600/X79 series ..<br>
> 00:1d.0 USB controller: Intel Corporation C600/X79 series ..<br>
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (re..<br>
> 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chip..<br>
> 00:1f.2 SATA controller: Intel Corporation C600/X79 series..<br>
> 00:1f.3 SMBus: Intel Corporation C600/X79 series chipset S..<br>
> 00:1f.6 Signal processing controller: Intel Corporation C6..<br>
> 02:00.0 Serial Attached SCSI controller: Intel Corporation..<br>
> 03:00.0 SCSI storage controller: LSI Logic / Symbios Logic..<br>
> 05:00.0 Ethernet controller: Intel Corporation 82599EB 10-..<br>
> 05:00.1 Ethernet controller: Intel Corporation 82599EB 10-..<br>
> 07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113..<br>
> 08:01.0 VGA compatible controller: Matrox Electronics Syst..<br>
> 09:00.0 Ethernet controller: Intel Corporation I350 Gigabi..<br>
> 09:00.1 Ethernet controller: Intel Corporation I350 Gigabi..<br>
> 0b:00.0 RAID bus controller: LSI Logic / Symbios Logic Meg..<br>
> <<< Illumos gives up here, Linux continues >>><br>
> 7f:08.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:08.3 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:08.4 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:09.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:09.3 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:09.4 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0a.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0a.1 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0a.2 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0a.3 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0b.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0b.3 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0c.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0c.1 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0c.2 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0c.6 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0c.7 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0d.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0d.1 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0d.2 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0d.6 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0e.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0e.1 Performance counters: Intel Corporation Xeon E5/Co..<br>
> 7f:0f.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0f.1 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0f.2 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0f.3 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0f.4 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0f.5 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:0f.6 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:10.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:10.1 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:10.2 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:10.3 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:10.4 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:10.5 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:10.6 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:10.7 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:11.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:13.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 7f:13.1 Performance counters: Intel Corporation Xeon E5/Co..<br>
> 7f:13.4 Performance counters: Intel Corporation Xeon E5/Co..<br>
> 7f:13.5 Performance counters: Intel Corporation Xeon E5/Co..<br>
> 7f:13.6 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 80:00.0 PCI bridge: Intel Corporation Xeon E5/Core i7 DMI2..<br>
> 80:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..<br>
> 80:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..<br>
> 80:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..<br>
> 80:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..<br>
> 80:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO ..<br>
> 80:04.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 80:04.1 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 80:04.2 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 80:04.3 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 80:04.4 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 80:04.5 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 80:04.6 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 80:04.7 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 80:05.0 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 80:05.2 System peripheral: Intel Corporation Xeon E5/Core ..<br>
> 80:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (r..<br>
> 83:00.0 SCSI storage controller: LSI Logic / Symbios Logic..<br>
><br>
> See <a href="http://217.75.71.10/MB.png" target="_blank">http://217.75.71.10/MB.png</a> for the system interconnection diagram.<br>
> Can somebody please recommend how to debug this issue?<br>
><br>
> Cheers,<br>
> --<br>
> Saso<br>
><br>
><br>
> -------------------------------------------<br>
</div></div>> illumos-discuss<br>
> Archives: <a href="https://www.listbox.com/member/archive/182180/=now" target="_blank">https://www.listbox.com/member/archive/182180/=now</a><br>
> RSS Feed: <a href="https://www.listbox.com/member/archive/rss/182180/22003744-9012f59c" target="_blank">https://www.listbox.com/member/archive/rss/182180/22003744-9012f59c</a><br>
> Modify Your Subscription: <a href="https://www.listbox.com/member/?&" target="_blank">https://www.listbox.com/member/?&</a><br>
<div class=""><div class="h5">> Powered by Listbox: <a href="http://www.listbox.com" target="_blank">http://www.listbox.com</a><br>
<br>
<br>
<br>
-------------------------------------------<br>
illumos-developer<br>
Archives: <a href="https://www.listbox.com/member/archive/182179/=now" target="_blank">https://www.listbox.com/member/archive/182179/=now</a><br>
RSS Feed: <a href="https://www.listbox.com/member/archive/rss/182179/24096868-ef9d90b8" target="_blank">https://www.listbox.com/member/archive/rss/182179/24096868-ef9d90b8</a><br>
Modify Your Subscription: <a href="https://www.listbox.com/member/?member_id=24096868&id_secret=24096868-8ac1a962" target="_blank">https://www.listbox.com/member/?member_id=24096868&id_secret=24096868-8ac1a962</a><br>
Powered by Listbox: <a href="http://www.listbox.com" target="_blank">http://www.listbox.com</a><br>
</div></div></blockquote></div><br></div></div>
`