NVME drive not recognized anymore after recent kernel upgrade

[Armling]

After a recent kernel upgrade on a nanopi-r5s, the NVME device files /dev/nvme0* are no longer present. The relevant part of /var/log/syslog shows:

 

2025-07-05T20:55:35.716376+02:00 nanopi-r5s kernel: [    0.000000] Linux version 6.12.35-current-rockchip64 (build@armbian) (aarch64-linux-gnu-gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #1 SMP PREEMPT Fri Jun 27 10:11:46 UTC 2025
...
2025-07-05T20:55:35.719736+02:00 nanopi-r5s kernel: [    4.491538] nvme nvme0: pci function 0002:01:00.0
2025-07-05T20:55:35.719739+02:00 nanopi-r5s kernel: [    4.491584] nvme 0002:01:00.0: enabling device (0000 -> 0002)

 

There is a single NVME drive installed and the output of a previous kernel where /dev/nvme0* was present is as follows.

 

2025-07-05T02:17:37.940170+02:00 nanopi-r5s kernel: [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
2025-07-05T02:17:37.954269+02:00 nanopi-r5s kernel: [    0.000000] Linux version 6.12.35-current-rockchip64 (build@armbian) (aarch64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #1 SMP PREEMPT Fri Jun 27 10:11:46 UTC 2025
...
2025-07-05T02:17:37.969044+02:00 nanopi-r5s kernel: [    4.406406] nvme nvme0: pci function 0002:01:00.0
2025-07-05T02:17:37.969049+02:00 nanopi-r5s kernel: [    4.406455] nvme 0002:01:00.0: enabling device (0000 -> 0002)
2025-07-05T02:17:37.969053+02:00 nanopi-r5s kernel: [    4.540002] nvme nvme0: D3 entry latency set to 10 seconds
2025-07-05T02:17:37.969074+02:00 nanopi-r5s kernel: [    4.628689] nvme nvme0: allocated 64 MiB host memory buffer.
2025-07-05T02:17:37.969080+02:00 nanopi-r5s kernel: [    4.737972] nvme nvme0: 4/0/0 default/read/poll queues

 

The contents of /etc/armbian-image-release currently are as follows:

 

# PLEASE DO NOT EDIT THIS FILE
BOARD=nanopi-r5s
BOARD_NAME="NanoPi R5S"
BOARDFAMILY=rockchip64
BUILD_REPOSITORY_URL=https://github.com/armbian/build
BUILD_REPOSITORY_COMMIT=f149a11b4
LINUXFAMILY=rockchip64
ARCH=arm64
BOOT_SOC=rk3568
IMAGE_TYPE=nightly
BOARD_TYPE=csc
INITRD_ARCH=arm64
KERNEL_IMAGE_TYPE=Image
KERNEL_TARGET=current,edge
KERNEL_TEST_TARGET=current
FORCE_BOOTSCRIPT_UPDATE=
FORCE_UBOOT_UPDATE=
OVERLAY_DIR="/boot/dtb/rockchip/overlay"
VENDOR="Armbian_community"
VENDORCOLOR="247;16;0"
VENDORDOCS="https://docs.armbian.com"
VENDORURL="https://github.com/armbian/build"
VENDORSUPPORT="https://community.armbian.com/"
VENDORBUGS="https://github.com/armbian/community/issues"
BOOTSCRIPT_FORCE_UPDATE="no"
BOOTSCRIPT_DST="boot.cmd"
VERSION=25.8.0-trunk.245
REVISION=25.8.0-trunk.245
IMAGE_UUID=bef97f49-0290-4f2c-9d7a-0229540b307a
INSTALLED=true

 

The contents of /etc/armbian-release currently are as follows:

 

# PLEASE DO NOT EDIT THIS FILE
BOARD=nanopi-r5s
BOARD_NAME="NanoPi R5S"
BOARDFAMILY=rockchip64
BUILD_REPOSITORY_URL=https://github.com/armbian/build
BUILD_REPOSITORY_COMMIT=034e9253c
LINUXFAMILY=rockchip64
ARCH=arm64
BOOT_SOC=rk3568
IMAGE_TYPE=nightly
BOARD_TYPE=csc
INITRD_ARCH=arm64
KERNEL_IMAGE_TYPE=Image
KERNEL_TARGET=current,edge
KERNEL_TEST_TARGET=current
FORCE_BOOTSCRIPT_UPDATE=
FORCE_UBOOT_UPDATE=
OVERLAY_DIR="/boot/dtb/rockchip/overlay"
VENDOR="Armbian"
VENDORCOLOR="247;16;0"
VENDORDOCS="https://docs.armbian.com"
VENDORURL="https://www.armbian.com/"
VENDORSUPPORT="https://forum.armbian.com"
VENDORBUGS="https://www.armbian.com/bugs"
BOOTSCRIPT_FORCE_UPDATE="no"
BOOTSCRIPT_DST="boot.cmd"
VERSION=25.8.0-trunk.319
REVISION=25.8.0-trunk.319
BRANCH=6.1.0

 

Unfortunately, I don't have a version of the file when the NVME device files were still present. If I remember correctly, the working version was 25.8.0-trunk.313

https://github.com/armbian/os/tags

 

Strangely, the current version 25.8.0-trunk.319 shows "(Ubuntu 13.2.0-23ubuntu4) 13.2.0" in the kernel boot output whereas the previous version (probably 25.8.0-trunk.313) shows "(Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0", i.e., a later version.

 

Is there a way to try version 25.8.0-trunk.313 again or figure out what might be the problem with the current version?

Edited July 5 by Armling

[Werner]

Providing logs with 

armbianmonitor -u

 helps with troubleshooting and significantly raises chances that issue gets addressed.

[Armling]

I can do that but it does not provide any useful information on the subject in my opinion.

 

Is there a way to downgrade to 25.8.0-trunk.313?

 

According to this post, it might be useful to set the kernel option nvme_core.default_ps_max_latency_us resp. the DT overlay parameter pciex1_gen=3.

How can the kernel (cmdline) option be set in Armbian? Regarding the DT overlay parameter pciex1_gen, should I follow this part of the documentation?

[Sirmalinton]

On 7/6/2025 at 9:53 AM, Armling said:

How can the kernel (cmdline) option be set in Armbian?

 

Most Armbian systems use a file located at /boot/armbianEnv.txt. You can add kernel parameters using the extraargs line.

[Armling]

Thanks for the hint Sirmalinton. Today, I could solve the problem. Here is a short summary that might be of use for people running into similar issues.

 

Running lspci -k shows the "Non-Volatile memory controller" but it does not show the line "Kernel driver in use: nvme". Hence, I ran

echo nvme > /sys/bus/pci/devices/0002\:01\:00.0/driver_override
echo 0002:01:00.0 > /sys/bus/pci/drivers_probe

where 0002:01:00.0 is from the output of lspci -k. That worked and made the disk appear as files /dev/nvme0*.

 

To have a solution that works after reboot, I created a udev rule /etc/udev/rules.d/99-nvme-override.rules with the content:

ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x144d", ATTR{device}=="0xa80d", ATTR{driver_override}="nvme"

 

The vendor and device IDs are from lspci -nn. To trigger the probe, I created a systemd service /etc/systemd/system/nvme-rebind.service with the content:

[Unit]
Description=Rebind NVME controller to driver after override
After=udev.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c "echo 0002:01:00.0 > /sys/bus/pci/drivers_probe"

[Install]
WantedBy=multi-user.target

The service can be enabled via systemctl enable nvme-rebind.service.

 

Maybe one reason it does not work in the first place is because /lib/systemd/system/nvmefc-boot-connections.service is not started. The reason is that the condition

ConditionPathExists=/sys/class/fc/fc_udev_device/nvme_discovery

is not met.

 

Another reason, I can imagine is that the vendor/device ID combination is not registered to the nvme driver in the current kernel. As I mentioned before, it did work out of the box when I installed Armbian at first.