blood Posted yesterday at 04:40 AM Posted yesterday at 04:40 AM I'm using a Rock 5 ITX and noticed after installing the latest rk35xx kernel image (6.1.84-vendor-rk35xx) that my system wasn't booting. So I connected to the debug console and observed a kernel oops being emitted. It dumps out a few pages of data, but the initial messages and call trace are: Quote Starting kernel ... [ 17.354173] Internal error: Oops: 0000000096000004 [#1] SMP [ 17.354701] Modules linked in: spl(O+) hci_uart btqca pwm_fan btrtl btbcm rk805_pwrkey btintel nvmem_rockchip_otp panfrost drm_shmem_helper gpu_sched rockchip_cpuinfo uio_pdrv_gen irq uio fuse dm_mod ip_tables ipv6 adc_keys r8169 [ 17.356564] CPU: 5 PID: 490 Comm: modprobe Tainted: G O 6.1.84-vendor-rk35xx #1 [ 17.357347] Hardware name: Radxa ROCK 5 ITX (DT) [ 17.357771] pstate: 80400009 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 17.358395] pc : mod_sysfs_setup+0x230/0x518 [ 17.358798] lr : mod_sysfs_setup+0x17c/0x518 [ 17.359188] sp : ffff80000fcc3b30 [ 17.359496] x29: ffff80000fcc3b30 x28: ffff80000ab1a910 x27: 0000000000000000 [ 17.360148] x26: ffff8000013bfda8 x25: ffff80000ab1a8a8 x24: ffff8000013bfad0 [ 17.360798] x23: ffff80000fcc3d30 x22: ffff8000013bfa98 x21: ffff8000013bb368 [ 17.361446] x20: ffff8000013bfa80 x19: 0000000000000000 x18: 0000000000000000 [ 17.362095] x17: ffff0001086741f0 x16: ffff0001086741b0 x15: ffff000108674170 [ 17.362743] x14: ffff000108674130 x13: 00646e69625f6461 x12: 657268745f716b73 [ 17.363392] x11: ffff000108674370 x10: 0000000000000000 x9 : ffff800009233b88 [ 17.364040] x8 : 0101010101010101 x7 : 7f7f7f7f7f7f7f7f x6 : ffff000103eaa219 [ 17.364688] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000 [ 17.365336] x2 : ffff8000013bfa98 x1 : ffff8000013bfad0 x0 : 97ffe3ee97ffed02 [ 17.365986] Call trace: [ 17.366216] mod_sysfs_setup+0x230/0x518 [ 17.366574] load_module+0x1870/0x1990 [ 17.366928] __do_sys_finit_module+0xf8/0x118 [ 17.367336] __arm64_sys_finit_module+0x24/0x30 [ 17.367757] invoke_syscall+0x8c/0x128 [ 17.368107] el0_svc_common.constprop.0+0xd8/0x128 [ 17.368546] do_el0_svc+0xac/0xbc [ 17.368858] el0_svc+0x2c/0x54 [ 17.369156] el0t_64_sync_handler+0xac/0x13c [ 17.369554] el0t_64_sync+0x19c/0x1a0 [ 17.369891] [ 17.369891] PC: 0xffff8000081133a8: <snip pages of hexdump /> [ 17.928255] systemd[1]: zfs-load-module.service: Main process exited, code=killed, status=11/SEGV [ 17.930271] systemd[1]: zfs-load-module.service: Failed with result 'signal'. [ 17.931849] systemd[1]: Failed to start zfs-load-module.service - Install ZFS kernel module. I have all the hex data if someone would use it but it doesn't seem worth including here. The boot flounders around in systemd for a bit but never gets to a login prompt - and I'm not that savvy with uboot to enter single user mode or otherwise dig in more. So I reinstalled from nothing, upgraded all my packages (including the vendor-rk35xx kernel), rebooted, and everything was ok - until I installed zfs-dkms and built the zfs module and rebooted - at which time I got back into the same busted position. And that's where I am now. I suppose I could go without zfs but I've been using it for a couple decades now and prefer it over the alternatives. As I'm not doing anything critical with this system yet, I don't mind futzing around and providing data or otherwise helping to debug what's up. At least for a few days. It does run my jellyfin instance, but I have plex running on another system so I'm too put out. A few months ago something similar (but different!) occurred after a rk35xx kernel upgrade but in that case, the system would at least boot to a login prompt eventually where I was able to downgrade the kernel and get it working again - and then the next version of the package fixed the problem. But in this case, it is effectively bricking the system for me (I'm sure someone that knows more about uboot could interrupt the boot and maybe recover things) to where I just reinstall - but that's not a good way to iterate and test fixes. I run the rk35xx vendor kernel because jellyfin has support for hardware transcoding when using that kernel - and as far as I know it does not when running against mainline - but this vendor kernel seems to be... less than stable! Any advice? 0 Quote
Werner Posted yesterday at 04:53 AM Posted yesterday at 04:53 AM Hi, first would be to set verbosity to 7 in order to get full boot logs. Excerpts are often useless. Further two things I'd do: For once I'd try to recover by chrooting into the system of the ITX by shoving the microsd into another arm64 machine (x86 can work too but it is far more annoying) and uninstall zfs package to see if this fixes the boot. If it is beyond recovery I'd try a fresh system from sd with a different userspace if the issue persists. Also just for the heck of it you could try this zfs package: https://github.com/zabbly/zfs I personnaly never tried this but it states to support arm64 so....just go for it Mainline is not an option for jellyfin, yes. It is far from being feature-complete while the rockchip sdk got most stuff working but, as correctly noticed, is fragile. 0 Quote
Turbodid Posted yesterday at 06:08 AM Posted yesterday at 06:08 AM Hello, I'm facing the same 'oops' kernel after upgrading my orange Pi 5 Plus from 24.11.2 (6.1.75-vendor-rk35xx) to 24.11.3 ( 6.1.84-vendor-rk35xx) => no more system bootable. I'm back with SDCard (24.11.1), then chroot emmc to remove zfs (dkms remove zfs/2.2.6), reboot on emmc => No trouble, system actif without ZFS. Then I try to reinstall zfs (dkms install zfs/2.2.6), at the end of the process (after success installation message), I've got a systemlog alert with 'oops'.... => system no more bootable. I'm back to SDCard and make update the complete system 24.11.1 to 24.11.3 (ZFS is not installed on SDCard) => no trouble. So it seems incompatibility between 6.1.84-vendor-rk35xx and ZFS (at least 2.2.6, perhaps I will try 2.2.7 from backports). For my point of view, I have no choice, ZFS must ran on my SBC. So, first I give a try with zfs 2.2.7, and if trouble is still present, I will go back to 24.11.2 (6.1.75-vendor-rk35xx). Regards, Turbodid 0 Quote
blood Posted 22 hours ago Author Posted 22 hours ago @Werner Thanks for the attention and advice. I've read plenty of your posts in the past and your contributions are really appreciated! Does Armbian have a support path back upstream to Radxa for their vendor kernel? It always takes at least two parties to have an incompatibility so I'm not sure where the ultimate fault lies between ZFS and RK35xx here, but perhaps Radxa could help some here. I know this is the community supported Rockchip forum, but the Rock 5 ITX board is actually in the platinum support category (my fault for misfiling this but I assume it effects all rk35xx boards) which I assume means there's some linkage back to the vendor (but maybe I misunderstand what platinum support really means). 13 hours ago, Werner said: Further two things I'd do: For once I'd try to recover by chrooting into the system of the ITX by shoving the microsd into another arm64 machine (x86 can work too but it is far more annoying) and uninstall zfs package to see if this fixes the boot. In this case, I'm booting and running my root filesystem off of emmc which is soldered onto the board. I'm flashing Armbian to it over USB using rkdeveloptool via maskrom - which means I can't pull the storage and read it on another system unfortunately. I guess I could reinstall to an SD card (or move my root FS to removable storage for a future install) but I'm not there right now. 12 hours ago, Turbodid said: So it seems incompatibility between 6.1.84-vendor-rk35xx and ZFS (at least 2.2.6, perhaps I will try 2.2.7 from backports). @Turbodid If you have any luck or even just a different experience with this or other approaches, please make sure to update this thread. 0 Quote
Werner Posted 22 hours ago Posted 22 hours ago 2 minutes ago, blood said: Does Armbian have a support path back upstream to Radxa for their vendor kernel? Yes there is some communication happening but this is out of scope for me. I do community 3 minutes ago, blood said: In this case, I'm booting and running my root filesystem off of emmc which is soldered onto the board. Well thats unfortunate...and fortunate the same time. Let me explain real quick. rockchip socs have a built-in fixed boot order which is something like SPI, eMMC, microSD, USB and so on. However u-boot should be configured in a way, even though present on SPI or eMMC, it will check for an OS on microsd and try to boot that. tl;dr: Just write a fresh Armbian image on a microsd, put it into the ITX and power it on. It should boot from microsd. This system then can be used to fix your OS on the soldered eMMC. 0 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.