Armling Posted July 5 Posted July 5 (edited) After a recent kernel upgrade on a nanopi-r5s, the NVME device files /dev/nvme0* are no longer present. The relevant part of /var/log/syslog shows: 2025-07-05T20:55:35.716376+02:00 nanopi-r5s kernel: [ 0.000000] Linux version 6.12.35-current-rockchip64 (build@armbian) (aarch64-linux-gnu-gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #1 SMP PREEMPT Fri Jun 27 10:11:46 UTC 2025 ... 2025-07-05T20:55:35.719736+02:00 nanopi-r5s kernel: [ 4.491538] nvme nvme0: pci function 0002:01:00.0 2025-07-05T20:55:35.719739+02:00 nanopi-r5s kernel: [ 4.491584] nvme 0002:01:00.0: enabling device (0000 -> 0002) There is a single NVME drive installed and the output of a previous kernel where /dev/nvme0* was present is as follows. 2025-07-05T02:17:37.940170+02:00 nanopi-r5s kernel: [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050] 2025-07-05T02:17:37.954269+02:00 nanopi-r5s kernel: [ 0.000000] Linux version 6.12.35-current-rockchip64 (build@armbian) (aarch64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #1 SMP PREEMPT Fri Jun 27 10:11:46 UTC 2025 ... 2025-07-05T02:17:37.969044+02:00 nanopi-r5s kernel: [ 4.406406] nvme nvme0: pci function 0002:01:00.0 2025-07-05T02:17:37.969049+02:00 nanopi-r5s kernel: [ 4.406455] nvme 0002:01:00.0: enabling device (0000 -> 0002) 2025-07-05T02:17:37.969053+02:00 nanopi-r5s kernel: [ 4.540002] nvme nvme0: D3 entry latency set to 10 seconds 2025-07-05T02:17:37.969074+02:00 nanopi-r5s kernel: [ 4.628689] nvme nvme0: allocated 64 MiB host memory buffer. 2025-07-05T02:17:37.969080+02:00 nanopi-r5s kernel: [ 4.737972] nvme nvme0: 4/0/0 default/read/poll queues The contents of /etc/armbian-image-release currently are as follows: # PLEASE DO NOT EDIT THIS FILE BOARD=nanopi-r5s BOARD_NAME="NanoPi R5S" BOARDFAMILY=rockchip64 BUILD_REPOSITORY_URL=https://github.com/armbian/build BUILD_REPOSITORY_COMMIT=f149a11b4 LINUXFAMILY=rockchip64 ARCH=arm64 BOOT_SOC=rk3568 IMAGE_TYPE=nightly BOARD_TYPE=csc INITRD_ARCH=arm64 KERNEL_IMAGE_TYPE=Image KERNEL_TARGET=current,edge KERNEL_TEST_TARGET=current FORCE_BOOTSCRIPT_UPDATE= FORCE_UBOOT_UPDATE= OVERLAY_DIR="/boot/dtb/rockchip/overlay" VENDOR="Armbian_community" VENDORCOLOR="247;16;0" VENDORDOCS="https://docs.armbian.com" VENDORURL="https://github.com/armbian/build" VENDORSUPPORT="https://community.armbian.com/" VENDORBUGS="https://github.com/armbian/community/issues" BOOTSCRIPT_FORCE_UPDATE="no" BOOTSCRIPT_DST="boot.cmd" VERSION=25.8.0-trunk.245 REVISION=25.8.0-trunk.245 IMAGE_UUID=bef97f49-0290-4f2c-9d7a-0229540b307a INSTALLED=true The contents of /etc/armbian-release currently are as follows: # PLEASE DO NOT EDIT THIS FILE BOARD=nanopi-r5s BOARD_NAME="NanoPi R5S" BOARDFAMILY=rockchip64 BUILD_REPOSITORY_URL=https://github.com/armbian/build BUILD_REPOSITORY_COMMIT=034e9253c LINUXFAMILY=rockchip64 ARCH=arm64 BOOT_SOC=rk3568 IMAGE_TYPE=nightly BOARD_TYPE=csc INITRD_ARCH=arm64 KERNEL_IMAGE_TYPE=Image KERNEL_TARGET=current,edge KERNEL_TEST_TARGET=current FORCE_BOOTSCRIPT_UPDATE= FORCE_UBOOT_UPDATE= OVERLAY_DIR="/boot/dtb/rockchip/overlay" VENDOR="Armbian" VENDORCOLOR="247;16;0" VENDORDOCS="https://docs.armbian.com" VENDORURL="https://www.armbian.com/" VENDORSUPPORT="https://forum.armbian.com" VENDORBUGS="https://www.armbian.com/bugs" BOOTSCRIPT_FORCE_UPDATE="no" BOOTSCRIPT_DST="boot.cmd" VERSION=25.8.0-trunk.319 REVISION=25.8.0-trunk.319 BRANCH=6.1.0 Unfortunately, I don't have a version of the file when the NVME device files were still present. If I remember correctly, the working version was 25.8.0-trunk.313 https://github.com/armbian/os/tags Strangely, the current version 25.8.0-trunk.319 shows "(Ubuntu 13.2.0-23ubuntu4) 13.2.0" in the kernel boot output whereas the previous version (probably 25.8.0-trunk.313) shows "(Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0", i.e., a later version. Is there a way to try version 25.8.0-trunk.313 again or figure out what might be the problem with the current version? Edited July 5 by Armling 0 Quote
Werner Posted July 6 Posted July 6 Providing logs with armbianmonitor -u helps with troubleshooting and significantly raises chances that issue gets addressed. 0 Quote
Armling Posted July 6 Author Posted July 6 I can do that but it does not provide any useful information on the subject in my opinion. Is there a way to downgrade to 25.8.0-trunk.313? According to this post, it might be useful to set the kernel option nvme_core.default_ps_max_latency_us resp. the DT overlay parameter pciex1_gen=3. How can the kernel (cmdline) option be set in Armbian? Regarding the DT overlay parameter pciex1_gen, should I follow this part of the documentation? 0 Quote
Sirmalinton Posted July 12 Posted July 12 On 7/6/2025 at 9:53 AM, Armling said: How can the kernel (cmdline) option be set in Armbian? Most Armbian systems use a file located at /boot/armbianEnv.txt. You can add kernel parameters using the extraargs line. 0 Quote
Armling Posted July 12 Author Posted July 12 Thanks for the hint Sirmalinton. Today, I could solve the problem. Here is a short summary that might be of use for people running into similar issues. Running lspci -k shows the "Non-Volatile memory controller" but it does not show the line "Kernel driver in use: nvme". Hence, I ran echo nvme > /sys/bus/pci/devices/0002\:01\:00.0/driver_override echo 0002:01:00.0 > /sys/bus/pci/drivers_probe where 0002:01:00.0 is from the output of lspci -k. That worked and made the disk appear as files /dev/nvme0*. To have a solution that works after reboot, I created a udev rule /etc/udev/rules.d/99-nvme-override.rules with the content: ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x144d", ATTR{device}=="0xa80d", ATTR{driver_override}="nvme" The vendor and device IDs are from lspci -nn. To trigger the probe, I created a systemd service /etc/systemd/system/nvme-rebind.service with the content: [Unit] Description=Rebind NVME controller to driver after override After=udev.service [Service] Type=oneshot ExecStart=/bin/sh -c "echo 0002:01:00.0 > /sys/bus/pci/drivers_probe" [Install] WantedBy=multi-user.target The service can be enabled via systemctl enable nvme-rebind.service. Maybe one reason it does not work in the first place is because /lib/systemd/system/nvmefc-boot-connections.service is not started. The reason is that the condition ConditionPathExists=/sys/class/fc/fc_udev_device/nvme_discovery is not met. Another reason, I can imagine is that the vendor/device ID combination is not registered to the nvme driver in the current kernel. As I mentioned before, it did work out of the box when I installed Armbian at first. 0 Quote
laibsch Posted July 18 Posted July 18 Thank you for your work and I'm happy you fixed it for your installation. It would be good to have this be incorporated into Armbian itself. 0 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.