RK3588, NVMe, ASPM and the 6.10-kernel

[specs]

When using my (intel) laptop I get when I check my nvme (as root):

# lspci -vvvs 01:00|grep -E "ASPM|L1"
        DevCap:    MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 unlimited
        LnkCap:    Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
    Capabilities: [900 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
        L1SubCtl2: T_PwrOn=500us

 

With the 6.1-vendor kernel I got the same on my rock05b and I did see others writing similar results (for example with the FriendlyElec Nanopi R6).

With the 6.10-kernel I get:

# lspci -vvvs 01:00|grep -E "L1,ASPM"
<no output>

Further searching yields "Unknown header type 7f".

Is this typical for the 6.10-kernel or is this a bug in the armbian-kernel specific?

 

The other problem with the 6.10-armbian kernel is that if I change a tiny option in the kernel config it yields a non-booting kernel (without changing the config the kernel does not compile, but is downloaded instead). This means I have the choice between a kernel I cannot compile myself and a vendor kernel without GPU acceleration (which I can compile). The 6.10-kernel should be more open, but without being able to boot a selfcompiled kernel I find it very closed.

 

Is there any hint to get a selcompiled kernel booting?

Edited August 20 by specs

[Werner]

mainline support for the rk3588 soc in 6.10 is far from feature-complete. If need most hw features working use vendor 6.1.y kernel.

To force rebuild of a kernel instead of downloaded artifact use ARTIFACT_IGNORE_CACHE=yes switch.

[Dantes]

Maybe there is a sticky somewhere on this forum, (if not there should be) pointing to this url:

https://gitlab.collabora.com/hardware-enablement/rockchip-3588/notes-for-rockchip-3588/-/blob/main/mainline-status.md

 

This is the mainline status of the rk3588(s). How far a long they are and what features have been implemented.

I'd give it another year or so.. but I'm looking forward to 6.12 that will be released before years end.

 

 

Edited August 25 by Dantes

[specs]

I know the list, and that's why I asked the question.

 

According to the list PCIE3 and M2 M are supported, but I can't find anything about ASPM. I would assume that when it is ready ASPM should also work.

The PCIE3 and M2 M options have been added somewhere by the 6.7-kernels, so it should be working at least in the basics. When I am getting an error that could mean my system is broken or the complete rk3588 support is still not ready.

 

If it is just my system it can help to ask if others see the same error.

What I see in the responses is that nobody seems uses the system anymore.

[Dantes]

That might be because its more of an appliance/router than a desktop. It might be a regression. I'd wait for 6.11 to arrive, Sunday we're at 6.11-rc6, so that would mean another 3 weeks give or take. But I have a couple, so I checked them for you:

 

OpenWRT

# grep RELEASE /etc/os-release 
OPENWRT_RELEASE="OpenWrt SNAPSHOT r0-212b6f9"
# uname -a
Linux 6.10 #0 SMP Mon Jul 15 13:18:21 2024 aarch64 GNU/Linux
# lspci
0003:30:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0003:31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
# lspci -vvvs 01:00|grep -E "L1,ASPM"
#

 

Armbian

$ grep ^PRETTY /etc/os-release 
PRETTY_NAME="Armbian 24.11.0-trunk.71 bookworm"
$ uname -a
Linux 6.10.7-current-rockchip-rk3588 #1 SMP PREEMPT Thu Aug 29 15:36:13 UTC 2024 aarch64 GNU/Linux
$ lspci
0003:30:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0003:31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
$ lspci -vvvs 01:00|grep -E "L1,ASPM"
$

 

Dantes

[Werner]

6.11-rc is already available via edge branch.

https://github.com/armbian/build/pull/7015

[Dantes]

Can't get hdmi output in Armbian/6.11 yet, so I'll have to wait for that to clear up.

 

However these 3rd party OpenWRT builds (https://github.com/mj22226/openwrt/releases) all show the same thing:

 

OpenWRT Linux 6.1.94

# grep RELEASE /etc/os-release
OPENWRT_RELEASE="OpenWrt SNAPSHOT r0-4dc539f"
# uname -a
Linux 6.1.94 #0 SMP Sun Jun 16 23:31:23 2024 aarch64 GNU/Linux
# lspci
0003:30:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0003:31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
# lspci -vvvs 01:00|grep -E "L1,ASPM"
# 

 

OpenWRT Linux 6.4.2

# grep RELEASE /etc/os-release
OPENWRT_RELEASE="OpenWrt SNAPSHOT r0-1a94f52"
# uname -a
Linux 6.4.2 #0 SMP PREEMPT Fri Jul  7 23:03:17 2023 aarch64 GNU/Linux
# lspci
0003:30:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0003:31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd Device 3588 (rev 01)
# lspci -vvvs 01:00|grep -E "L1,ASPM"
#

 

OpenWRT Linux 6.6.48

# grep RELEASE /etc/os-release
OPENWRT_RELEASE="OpenWrt SNAPSHOT r0-5a5368e"
# uname -a
Linux 6.6.48 #0 SMP Fri Aug 30 11:13:30 2024 aarch64 GNU/Linux
# lspci
0003:30:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0003:31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
# lspci -vvvs 01:00|grep -E "L1,ASPM"
#

 

OpenWRT Linux 6.11-rc

# grep RELEASE /etc/os-release
OPENWRT_RELEASE="OpenWrt SNAPSHOT r0-fe4b9c3"
# uname -a
Linux 6.11-rc5 #0 SMP Sun Aug 25 07:38:04 2024 aarch64 GNU/Linux
# lspci
0003:30:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0003:31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0004:40:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
# lspci -vvvs 01:00|grep -E "L1,ASPM"
#

 

[Dantes]

I think the problem lies in the command line, which I copied like a complete toddler

 

# grep RELEASE /etc/os-release
OPENWRT_RELEASE="OpenWrt SNAPSHOT r0-fd4a703"
# uname -a
Linux 6.11-rc6 #0 SMP Mon Sep  2 08:09:51 2024 aarch64 GNU/Linux
# lspci -vvv 2>&1 |grep -i ASPM
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <16us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <16us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-

 

[specs]

It seems there is an issue with the "lspci -vvv" in particular.

#  lspci -vvv|grep -E "L1,ASPM"
pcilib: sysfs_read_vpd: read failed: No such device

 

Continuing without the grep part I do see the nvme and aspm (Linux rock-5b 6.10.7-current-rockchip-rk3588):

# lspci -vvv

...

0000:01:00.0 Non-Volatile memory controller: Shenzhen Longsys Electronics Co., Ltd. Lexar NM790 NVME SSD (DRAM-less) (rev 01) (prog-if 02 [NVM Express])

       ...

        Capabilities: [180 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=30us LTR1.2_Threshold=54272ns
                L1SubCtl2: T_PwrOn=18us
        Capabilities: [190 v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?>
 

For the Rock5B (with the power is set to 12V fixed) it means:

      2.0W when running on eMMC (with network 1Gbps and monitor, without wifi)
      5.3W with nvme/performane

      4,3W with nvme/powersave.

      2.5W when running on eMMC (with wifi and monitor, without ethernet)

With other boards (RK3568) adding an nvme costs less extra power.

 

I recently replaced a Kingston nvme  for the Lexar (they are both not fast but, also not very powerhungry) but it did not yield positive results.

I will try the Kingston with a 6.1-vendor kernel again, before switching the rock5 for different hardware.