Freeze / kernel panic after upgrading to kernel 7.0.12

[Johannes]

Hi all,

recently my Banana Pi Pro running Armbian 26.5.1 with Debian Trixie received a kernel update from 6.16.8 (which it has been running on for quite a while without issues) to 7.0.12. Since then, it randomly freezes shortly after booting, sometimes I also can see a kernel panic (see attached pictures).

Has anyone else run into this issue? What can I do to fix this? The old kernel no longer appears to be present in /boot, which makes it difficult to return to another kernel (trying to run something like apt-get typically quickly results in the aforementioned freeze).

[eselarm]

Yes, I have similar issues with 7.0.12-edge-sunxi kernel, see serial dump/log below:

[  133.382506] Internal error: Oops - undefined instruction: 0 [#1] SMP THUMB2
[  133.389572] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace netfs sunrpc cp210x binfmt_misc usbserial lima gpu_sched nls_iso8859_1 drm_shmem_help
er sun4i_gpadc_iio sunxi_cir axp20x_adc sun4i_ts sunxi_cedrus(C) v4l2_mem2mem videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common videodev mc display_c
onnector cpufreq_dt evdev cfg80211 rfkill nfnetlink ip_tables x_tables autofs4 btrfs xor xor_neon libblake2b raid6_pq pinctrl_axp209 axp20x_usb_power industrialio sun4i_
gpadc sunxi phy_generic realtek
[  133.437258] CPU: 1 UID: 0 PID: 184 Comm: kworker/u8:5 Tainted: G         C          7.0.12-edge-sunxi #2 VOLUNTARY
[  133.447746] Tainted: [C]=CRAP
[  133.450738] Hardware name: Allwinner sun7i (A20) Family
[  133.455991] Workqueue: pm pm_runtime_work
[  133.460075] PC is at __rpm_callback+0x64/0x10c
[  133.464565] LR is at rpm_callback+0x51/0x58
[  133.468789] pc : [<c06d2e40>]    lr : [<c06d2f39>]    psr: 400f00b3
[  133.475083] sp : f0a5de90  ip : 00000018  fp : c2ef5200
[  133.480337] r10: c01761ed  r9 : 000f4240  r8 : 00000008
[  133.485590] r7 : 00000000  r6 : 00000000  r5 : c06d0aad  r4 : c1e25410
[  133.492146] r3 : 60000000  r2 : c1e25410  r1 : c1e25410  r0 : c06d0aad
[  133.498704] Flags: nZcv  IRQs off  FIQs on  Mode SVC_32  ISA Thumb  Segment none
[  133.506135] Control: 50c5387d  Table: 43eb406a  DAC: 00000051
[  133.511902] Register r0 information: non-slab/vmalloc memory
[  133.517616] Register r1 information: slab kmalloc-1k start c1e25400 pointer offset 16 size 1024
[  133.526415] Register r2 information: slab kmalloc-1k start c1e25400 pointer offset 16 size 1024
[  133.535209] Register r3 information: non-paged memory
[  133.540302] Register r4 information: slab kmalloc-1k start c1e25400 pointer offset 16 size 1024
[  133.549095] Register r5 information: non-slab/vmalloc memory
[  133.554795] Register r6 information: NULL pointer
[  133.559537] Register r7 information: NULL pointer
[  133.564279] Register r8 information: non-paged memory
[  133.569367] Register r9 information: non-paged memory
[  133.574455] Register r10 information: non-slab/vmalloc memory
[  133.580241] Register r11 information: slab task_struct start c2ef5200 pointer offset 0 size 2624
[  133.589123] Register r12 information: non-paged memory
[  133.594301] Process kworker/u8:5 (pid: 184, stack limit = 0xe8b1571c)
[  133.600779] Stack: (0xf0a5de90 to 0xf0a5e000)
[  133.605178] de80:                                     60000000 c1e25410 c06d0aad 0000000a
[  133.613397] dea0: 00000000 00000008 000f4240 c06d2f39 c1e25410 c06d0aad 0000000a c06d3f23
[  133.621617] dec0: c0ab03bf 00000402 c15b5180 c9306c24 c3790705 c34c3c70 c1813000 bf8e21ad
[  133.629836] dee0: c2ef5200 c1813000 01000000 569e35c7 c34f0c70 c1e254f8 c1e254f8 c1b9de00
[  133.638054] df00: c1813000 01000000 c1b9de05 c1b72670 c1813000 c06d45a7 c2fb5900 c0141a87
[  133.646273] df20: c3790000 569e35c7 00000000 c2ef5200 c1813000 c2fb5930 f0a5df50 c2fb5900
[  133.654492] df40: c1813000 c2fb5930 c1603d80 c1813020 c2ef5200 c181305c c1813000 c0142345
[  133.662711] df60: f0a5df78 c1603d80 f0a9deb8 c36b8fc0 c2ef5200 c0142245 c2fb5900 c2d35c80
[  133.670929] df80: f0a9deb8 00000000 00000000 c0149e29 00000001 c36b8fc0 c0149d59 00000000
[  133.679145] dfa0: 00000000 00000000 00000000 c0100155 00000000 00000000 00000000 00000000
[  133.687359] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  133.695575] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[  133.703775] Call trace:
[  133.703812]  __rpm_callback from rpm_callback+0x51/0x58
[  133.711641]  rpm_callback from rpm_suspend+0xab/0x514
[  133.716756]  rpm_suspend from pm_runtime_work+0x3f/0x70
[  133.722045]  pm_runtime_work from process_one_work+0x12f/0x390
[  133.727947]  process_one_work from worker_thread+0x101/0x254
[  133.733667]  worker_thread from kthread+0xd1/0xec
[  133.738431]  kthread from ret_from_fork+0x11/0x1c
[  133.743189] Exception stack(0xf0a5dfb0 to 0xf0a5dff8)
[  133.748280] dfa0:                                     00000000 00000000 00000000 00000000
[  133.756494] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  133.764705] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  133.771365] Code: f7ff ff7c 4640 f7ef (fd27) f104
[  133.776186] ---[ end trace 0000000000000000 ]---
[  133.780834] note: kworker/u8:5[184] exited with irqs disabled
[  154.802648] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[  154.808663] rcu:     0-...0: (10 ticks this GP) idle=4c4c/1/0x40000000 softirq=7570/7571 fqs=2625
[  154.817336] rcu:     (detected by 1, t=5252 jiffies, g=14613, q=22 ncpus=2)
[  154.824076] Sending NMI from CPU 1 to CPUs 0:

It was 3 weeks ago I did see this while doing some maintenance,upgrades,tests,experiments. Mine is a Bananapi M1, I use  sun7i-a20-bananapi.dtb  as DTB, and should be same for U-Boot, although I would need to check again what version/build I use. A difference is WiFi and some others, but my guess is there is something wrong/ongoing that is getting worse over time, as also with less newer kernels, it locks up after a few days quite often.  This has been the case since december 2024 kernel 6.6.x as far as I see that written in my notes. I noticed that because I have used my Bananapi as NAS (blocklevel NBD). But it were paging issues and never really could find a pattern, always random. This error message is different, but might have same root-cause.

 

Kernel 6.16.x I have run, but it lacked AXP PMU driver for the LiPo battery powering/charging which I use every now and then (is a great UPS like that). 6.18.x had it back again, but also lockups (same paging issue, I have already a kernel log dumping 24/7 to try to figure out what is wrong, but actually have no clue). I also used 6.12.90+deb13.1-armmp, so vanilla Debian kernel, but it has no overlays available and also no AXP PMU module, so that make it less useful.

 

I set it back to  6.12.58-current-sunxi so that at least it runs reliably for a few days and I can use it ad-hoc as a serial console terminal for a Raspberrypi4 that also used to crash randomly due to USB3 stack issues and also for debugging and testing rather complex networking issues. I am using custom partition layout, the whole thing also runs as 32-bit UEFI KVM as well as systemd-nspawn container (same cloned MAC address so I un-power the real Bananapi then). The setup uses custom extlinux and standard grub-efi. I could easily test 7.0.x or later in a KVM, but assume it will run fine. It is really something about a combination of A20, platform HW, U-Boot, DeviceTree I think. It is a pity as this SBC is about the most useful I have due to its onboard SATA and LiPo battery management, although not 64-bit.

 

[eselarm]

The forum returned 'nginx bad gateway' but I see draft of may message is still there.

 

In the meantime, I did a simple 'apt update && apt full-upgrade && reboot' and now I have:

U-Boot SPL 2024.01-armbian-2024.01-S866c-P7738-Hb9d3-Vf23c-Bb703-R448a (Jun 21 2025 - 02:53:13 +0000)
Kernel: Linux 6.18.35-current-sunxi

I might look into HW watchdog things, no clue if it is there and able to work, but used it successfully for Raspberrypi4 to workaround USB3 lockups/crashes.

[eselarm]

OK I see:

# dmesg | grep watchdog
[    0.462093] sunxi-wdt 1c20c90.watchdog: Watchdog enabled (timeout=16 sec, nowayout=0)

That would mean study 'bark' or so, forgot most of it. I guess the board should restart if IPv6 LL connection is not there for more than 1 minute or so, but in relation to NAS makes then only sense if I move from NBD to iSCSI or so.