SATA overlay not included anymore

[eselarm]

After kernel version 6.12.6, the overlay to enable SATA on the PCie is not included in the build anymore. So https://github.com/armbian/linux-rockchip/pull/237/commits/5bdf4a4243e6c149290cdd2c8f5a7d1a1946670e is not effective anymore.

The existing .dts can be compiled and works with kernel 6.12.10-current-rockchip64 and 6.13.0-edge-rockchip64, but then NVMe is not enabled.

 

From Rock3A schematics and RK3568 datasheet I see that there is no reason that it could not work as the NVMe is on independent PCIe 3x2 lanes in the M.2 M-key connector. PCIe 2x1 lane meant for WiFi/SATA is on E-key connector.

 

It looks like 'entity PCIe is mixed up' somehow. Does anyone have a hint where and how to fix this? Does this also happen on Radxa RK3588 based boards (not the 3588S variant I think)?

 

Workaround now is to use a 2-port SATA JMB58x breakout board.

 

Additional info:

root@rock3a:/tmp# strings /dev/mtdblock0 | grep "U-Boot"
U-Boot SPL 2024.10-armbian-2024.10-Sf919-Pe8a7-H29de-Vf307-Bb703-R448a (Jan 06 2025 - 02:00:45 +0000)

root@rock3a:/tmp# lspci
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
0000:01:00.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller
0002:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
0002:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983

 

[eselarm]

To my surprise this works now again; I boot + root from SD-card, and both NVMe and SATA are there. The change is that I run Armbian Bookworm now instead of Armbian Noble and that I just did update+upgrade; 6.13.0-edge-rockchip64 got newly reinstalled (higher beta trunk number). The Armbian Bookworm is cloned from my NanoPi-R6C (changed some packages from nanopi-r6c -> rock-3a, but for the rest I have no clue why it works now and not a few days ago. The userspace should not matter. Anyway, FYI:

root@rock3a:~# strings /dev/mtdblock0 | grep "U-Boot" | head -n1
U-Boot SPL 2024.10-armbian-2024.10-Sf919-Pe8a7-H29de-Vf307-Bb703-R448a (Jan 06 2025 - 02:00:45 +0000)

root@rock3a:~# uname -a
Linux rock3a 6.13.0-edge-rockchip64 #1 SMP PREEMPT Sun Jan 19 23:51:45 UTC 2025 aarch64 GNU/Linux

root@rock3a:~# lspci
0002:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
0002:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983

root@rock3a:~# dmesg | grep -e nvme
[    5.949382] nvme nvme0: pci function 0002:01:00.0
[    5.949860] nvme 0002:01:00.0: enabling device (0000 -> 0002)
[    5.953920] nvme nvme0: missing or invalid SUBNQN field.
[    5.955055] nvme nvme0: D3 entry latency set to 8 seconds
[    5.977503] nvme nvme0: 4/0/0 default/read/poll queues
[    6.007639]  nvme0n1: p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p11 p12
[   17.746225] nvme nvme0: using unchecked data buffer

root@rock3a:~# dmesg | grep -e ata1 -e ahci
[    2.904834] ahci-dwc fc800000.sata: supply ahci not found, using dummy regulator
[    2.905739] ahci-dwc fc800000.sata: supply phy not found, using dummy regulator
[    2.906596] ahci-dwc fc800000.sata: supply target not found, using dummy regulator
[    2.907407] ahci-dwc fc800000.sata: PMPn is limited up to 5 ports
[    2.908187] ahci-dwc fc800000.sata: forcing port_map 0x0 -> 0x1
[    2.908772] ahci-dwc fc800000.sata: AHCI vers 0001.0300, 32 command slots, 6 Gbps, platform mode
[    2.909558] ahci-dwc fc800000.sata: 1/1 ports implemented (port mask 0x1)
[    2.910163] ahci-dwc fc800000.sata: flags: ncq sntf pm led clo only pmp fbs pio slum part ccc apst 
[    2.910993] ahci-dwc fc800000.sata: port 0 is not capable of FBS
[    2.919874] scsi host0: ahci-dwc
[    2.920501] ata1: SATA max UDMA/133 mmio [mem 0xfc800000-0xfc800fff] port 0x100 irq 30 lpm-pol 0
[    3.387739] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    3.388945] ata1.00: ATA-9: WDC WD40EFRX-68WT0N0, 80.00A80, max UDMA/133
[    3.389578] ata1.00: 7814037168 sectors, multi 0: LBA48 NCQ (depth 32)
[    3.390867] ata1.00: configured for UDMA/133

 

Note that this is using a copied 'rock-3a-sata.dtbo' from the vendor 6.1.84 kernel as I did earlier/initially.
I was planning to see if I somehow maybe could modify/reuse  'rockchip-rk3566-sata2.dtbo', but I do not really understand exactly what its de-compiled .dtbo code does.

Edited January 28 by eselarm

[eselarm]

Unfortunately, something random is happening as after a reboot, the NVMe is again not found if SATA overlay is used. The only PCI-e related text in dmesg is then:

root@rock3a:~# dmesg | grep -e pcie -e nvme
[    0.174540] /pcie@fe280000: Fixed dependency cycle(s) with /pcie@fe280000/legacy-interrupt-controller
[    4.702382] rockchip-dw-pcie 3c0800000.pcie: probe with driver rockchip-dw-pcie failed with error -110

At least I know now that the used U-Boot does not find my NVME, so that is maybe already the (random) showstopper.

So I have to see if maybe a Radxa U-Boot makes it work.

Edited January 29 by eselarm

[eselarm]

Using Radxa U-Boot version "2023.07.02-3-b1eb2bde-gb1eb2bde" makes no difference.

[DMC]

Have You managed to get it running? I was able to get it to work on my board, but it drops sata link constantly

already checked different ssd, cable, power source (PoE vs usb-c), few kernels and still can't get stable sata link. I did not checked nvme with sata overlays, I assume it's problematic with sata on Your side? 

[eselarm]

I have currently this on SD-card:

root@rock3a:~# ls -1 /boot/vmlinuz-*
/boot/vmlinuz-5.10.160-39-rk356x
/boot/vmlinuz-6.12.12-current-rockchip64
/boot/vmlinuz-6.13.1-edge-rockchip64
/boot/vmlinuz-6.1.99-vendor-rk35xx

Only with the 5.10 kernel from Radxa, I get both NVMe and SATA when I use the overlay 'rock-3a-sata.dtbo'. With 6.12 and 6.13, there is only SATA, no NVMe. 6.1 lets the kernel/board crash, I have not further looked into that.

So I can reliably use the SATA HDD, also hd-idle spindown works, I monitor the powerconsumption per day on 12V level and it is great. So as a slow clone/backup NAS it works and is acceptable. But ultimate plan is to make it a fast/main NAS, using the NVMe as cache and also for OS, so only SPI+NVMe+SATA. 5.10 kernel lacks various Btrfs features, so that is a no-go. I need at least 6.1, although I know weird virtualization tricks to get it to work on a functional level, but then also, I might hit issues with KVM on the Radxa rk356x kernel.

I got the kernel from: https://radxa-repo.github.io/bookworm/files.list

Regarding power, I now use 12V as base (common GND), feed 12V into the Rock3A USB-C connector and 3.5inch 12V pin on HDD and tap 5V from Rock3A GPIO and that feeds the 5V pin of the HDD. As said, that works very nice and stable.

 

So it still looks like the 'rock-3a-sata.dtbo' overlay 'hides/disables all PCIe', where it should not do anything with the PCIe3x2 where the NVMe is on. I saw this:

https://patchwork.kernel.org/project/linux-rockchip/patch/20250106100001.1344418-2-amadeus@jmu.edu.cn/

but not sure if it is related.

Edited February 11 by eselarm

[DMC]

I also could not run it, checked different sata cables, different ssds. I tested that on 3A, 5A and to my suprise same setup worked on 5B (on vendor 6.1 and edge), so it's not the matter of damaged cable or component. 

Obvious solution is to use any pcie - sata adapter like m.2 A+E with JMB582 (dual sata) or m.2 M ASM1166 or JMB585, I got it on one Rock 3A paired with radxa penta sata kit (works great with additional 2.5G pcie adapter). 

[eselarm]

4 hours ago, DMC said:

Obvious solution is to use any pcie - sata adapter like m.2 A+E

That I consider a workaround. There are 3 SATA capable SERDES ports available on RK3568, similar for other Rockchips. What I see is that idle powerconsumpution went up by 2 Watt when using an extra JMB PCIe-SATA chip, I have not look in detail why that is. I am aiming for battery powered as well, so when I see that Rock3A itself is roughly 2W idle (with mainline kernel), doubling that would make an other platform/SoC more or less a better solution. SoC powerconsumption does not really matter if the 3.5 inch HDD spins all the time, but that is not the case/plan. Also a SATA SSD is an option for connecting.

 

I do not have a RK3588 (but have a RK3588S), I indeed guessed that SPI+NVMe+SATA will work on Rock5B when using 'rock-5b-sata.dtbo'. Or do you use a different overlay with edge kernel?

 

Kernel 6.1 has a problem that it does not allow seamless moving of virtual machines (between A76 and A55). Mainline (6.8+ etc) does not have that problem. There must be some (longterm) benefit for using Arm platform instead of continuing Intel (like N100, that is the alternative I keep in mind). So maybe Rock5B 16GB I should buy, that would be a good successor of my current Intel based NAS (and VM host as well every now and then).

 

Edited February 12 by eselarm

[eselarm]

Another trial: I put the legacy rock-3a u-boot in the SPI, which is: U-Boot SPL 2017.09-armbian (May 20 2024 - 00:46:51)

 

Then, with kernel 6.1.99-vendor-rk35xx, both NVMe and SATA are available when using the 'rock-3a-sata.dtbo' overlay. It failed to see/boot from SD-card, but TFTP and NVMe work.

The newer/mainline kernels still only SATA if overlay used. If no overlay used, only NVMe.

 

MAC address is changed and several other issues, but with this SW combination, I can test the HW/system performance w.r.t. 'caching' NAS I think.

 

[DMC]

Have You compared pcie sata adapter vs native sata vs nvme speeds?  All in A+E slot (pcie 2.1 x1) 

Any real advantages for native sata? 

[eselarm]

I got 302.87 MB/sec with an older SATA SSD on the "01:00.0 SATA controller: JMicron Technology Corp. JMB58x AHCI SATA controller". That was with a 4T HDD on the second port of that controller. Have not used SATA SSD with native/on-SoC SATA controller.

The fear is that when using this dual-port JMB for SSD+HDD and later 2.5Gbps ethernet, that the pcie2x1 will be the bottleneck (when Rock5B or so). An SSD caching a HDD involves more traffic on the busses than just a JBOD setup. I have had such a bottleneck in the past (for Intel motherboard, 6x SATA3). I'll have to do detailed calculations if this is the case now for Rockchip SBC.

With NVMe, pcie3x2 one can see/assume there will be no bottleneck and future is NVMe, I likely won't buy new HDD's.

Advantage for native SATA is likely not real for many people. But I like as much as possible on the same SoC, for various reasons. I know many people will buy an RPi5 or so and some USB3 enclosure for HDD, but I have had enough of such a setup (issues with USB3). Count the amount of extra entities you will need drivers etc for. As I indicated earlier, it is comparing Intel solution with something Arm (Rockchip in this case), not adding extra foreign interface chips. For JMicron, always the question if features like trim work.

 

Current quick test (2017.09-armbian (legacy package) U-Boot, armbian vendor kernel 6.1.99😞

root@rock3a:/local/s0/tmp# dd if=/dev/nvme0n1p12 iflag=direct bs=4M of=/dev/null count=10k status=progress
42253418496 bytes (42 GB, 39 GiB) copied, 31 s, 1.4 GB/s

root@rock3a:/local/s0/tmp# time dd if=/dev/zero of=dummyfile oflag=direct bs=4M count=1k
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 7.71642 s, 557 MB/s

root@rock3a:/local/s0/tmp# hdparm -t --direct /dev/nvme0n1p12
/dev/nvme0n1p12:
 Timing O_DIRECT disk reads: 3082 MB in  3.00 seconds = 1026.81 MB/sec

 root@rock3a:/local/s0/tmp# time dd of=/dev/null if=dummyfile iflag=direct bs=4M
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 4.09936 s, 1.0 GB/s

root@rock3a:/local/s0/tmp# hdparm -t --direct /dev/sda
/dev/sda:
 Timing O_DIRECT disk reads: 580 MB in  3.01 seconds = 192.95 MB/sec

Another note now is that system idle powerconsumption is 3.74 Watt. I use another HDD now, but I think it is the NVMe that is not going in powersave, aim is about 2W idle. So more research, measurements, etc is needed. But I still have not done/enabled the actual caching setup. That is a critical thing as any disturbance in the uptime and relation between NVMe and HDD will ruin the 8T filesystem badly. So first will see if that works for a month or so, when rsync, scrub, differential send|receive, power cut, etc.