PCIe width only x1 for a x4 card

[Nino]

Armbianmonitor:

https://paste.armbian.com/uwusayonoq

Hi

 

I installed a nvme SSD to my rockpro64 (with a generic PCIe adapter card). Somehow it only uses a x1 link instead of x4. I also observed the same behaviour with a x2 SATA card (only x1 link established).

Output of lspci:

LnkSta: Speed 2.5GT/s (downgraded), Width x1 (downgraded)

 

To rule out hardware issues, I also tried the same setup with Manjaro-ARM which correctly negotiates a x4 link.

LnkSta: Speed 2.5GT/s (downgraded), Width x4

 

dmesg for Armbian

[    2.221206] rockchip-pcie f8000000.pcie: host bridge /pcie@f8000000 ranges:
[    2.221296] rockchip-pcie f8000000.pcie:      MEM 0x00fa000000..0x00fbdfffff -> 0x00fa000000
[    2.221334] rockchip-pcie f8000000.pcie:       IO 0x00fbe00000..0x00fbefffff -> 0x00fbe00000
[    2.222953] rockchip-pcie f8000000.pcie: supply vpcie1v8 not found, using dummy regulator
[    2.223187] rockchip-pcie f8000000.pcie: supply vpcie0v9 not found, using dummy regulator
[    2.286376] rockchip-pcie f8000000.pcie: wait 1000 ms (from device tree) before bus scan
[    3.298661] rockchip-pcie f8000000.pcie: PCI host bridge to bus 0000:00
[    3.309542] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
[    3.310052] pcieport 0000:00:00.0: PME: Signaling with IRQ 58
[    3.310984] pcieport 0000:00:00.0: AER: enabled with IRQ 58

 

virtually identical for Manjaro-ARM

[    1.114557] rockchip-pcie f8000000.pcie: host bridge /pcie@f8000000 ranges:
[    1.114626] rockchip-pcie f8000000.pcie:      MEM 0x00fa000000..0x00fbdfffff -> 0x00fa000000
[    1.114655] rockchip-pcie f8000000.pcie:       IO 0x00fbe00000..0x00fbefffff -> 0x00fbe00000
[    1.115937] rockchip-pcie f8000000.pcie: supply vpcie1v8 not found, using dummy regulator
[    1.116100] rockchip-pcie f8000000.pcie: supply vpcie0v9 not found, using dummy regulator
[    1.176874] rockchip-pcie f8000000.pcie: wait 1000 ms (from device tree) before bus scan
[    2.234398] rockchip-pcie f8000000.pcie: PCI host bridge to bus 0000:00
[    2.274272] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
[    2.274780] pcieport 0000:00:00.0: PME: Signaling with IRQ 58
[    2.275722] pcieport 0000:00:00.0: AER: enabled with IRQ 58

 

I also compared the device trees for Armbian and Manjaro. There are no differences as well.

 

Has anyone a hint for me what I could try to get the x4 link established?

[Nino]

Narrowed the issue down by testing older Armbian builds

- Armbian 23.02 with 5.15.96 Kernel -> x4 link working

- Armbian 23.8 with 6.1.50 Kernel -> x4 link working

 

Comparing the patches between 6.1 and 6.6 shows a new patch related to PCIe: https://github.com/armbian/build/blob/main/patch/kernel/archive/rockchip64-6.6/rk3399-fix-pci-phy.patch

 

Building an image without this patch indeed fixes the issue:
 

Linux rockpro64 6.6.30-current-rockchip64

LnkSta:	Speed 2.5GT/s (downgraded), Width x4 (ok)

 

So this patch does not work as expected but simply removing it is probably not an acceptable solution. 
Can anyone with more understanding of PCI have a look into it?

 

Edit: I also commented on the pull request that introduced the patch: https://github.com/armbian/build/pull/6057

Edited May 16 by Nino

[ryanrs]

Yes! I get a degraded x1 link, too, with a Samsung 980 Pro.

 

I assumed it had never worked at x4, so it's interesting to hear that it is a regression.

[Igor]

On 5/16/2024 at 11:03 PM, Nino said:

I also commented on the pull request that introduced the patch

That is O.K., but Armbian build framework does not accept HW related tickets. Forum is right place where you can bring this up and discuss what to do. Then open a PR if solution emerges ...

 

On 5/16/2024 at 11:03 PM, Nino said:

Comparing the patches between 6.1 and 6.6 shows a new patch related to PCIe

 

Without this patch, PCI probing will randomly fail. Which is worse then 100% going up in 1x. This is my 2c.

 

I understand that you would like to have a perfect functioning PCI system on this hardware platform and if you cover someone several weeks of R&D, this can probably be done. Since this will not happen, we have partial solution/workaround in place, this forum and we have each other.

[Nino]

Hi Igor
Thanks for stepping in.

I followed the path out of curiosity. My RockPro64 usually runs 2.5" SMR SATA disks, so pcie x1 is not a real bottleneck

 

I tried tinkering some more, but I cannot reproduce the original pcie probing failure (the patch is of course disabled). I rebooted many times, then set bus scan delay to 0ms (same as in andrewz1 PR) rebooted some more. But never got the training timeout error.
Do I need a specific configuration or a specific hardware to reproduce the issue?