Myron Posted April 1 Posted April 1 This started happening since accepting the kernel when I used apt to upgrade the BananaPi Pro sbc I've got. armbianmonitor report: https://paste.armbian.com/izuboducul @teknoid are you aware of this? 0 Quote
Banana.31415 Posted April 14 Posted April 14 I can confirm this issue on an old BananaPi 1 from Lemaker. It began with Kernel 6.6. After returning to the older 6.1 the stalls disappeared. Apr 13 23:02:02 eisbaer kernel: rcu: INFO: rcu_sched self-detected stall on CPU Apr 13 23:02:02 eisbaer kernel: rcu: 0-....: (5249 ticks this GP) idle=56dc/1/0x40000002 softirq=5480265/5480265 fqs=2606 Apr 13 23:02:02 eisbaer kernel: rcu: (t=5250 jiffies g=9317845 q=1167 ncpus=2) Apr 13 23:02:02 eisbaer kernel: CPU: 0 PID: 27606 Comm: htop Tainted: G C 6.6.16-current-sunxi #1 Apr 13 23:02:02 eisbaer kernel: Hardware name: Allwinner sun7i (A20) Family Apr 13 23:02:02 eisbaer kernel: PC is at stmmac_get_stats64+0x26/0x128 Apr 13 23:02:02 eisbaer kernel: LR is at 0xc2e6b000 Apr 13 23:02:02 eisbaer kernel: pc : [<c078c5ce>] lr : [<c2e6b000>] psr: 80010033 Apr 13 23:02:02 eisbaer kernel: sp : f1621c68 ip : c2e68000 fp : 00000001 Apr 13 23:02:02 eisbaer kernel: r10: f1621e68 r9 : c2ee3c48 r8 : c2e68000 Apr 13 23:02:02 eisbaer kernel: r7 : 00000000 r6 : 00000001 r5 : 00000000 r4 : 80000000 Apr 13 23:02:02 eisbaer kernel: r3 : 07c697cd r2 : c2e6ae48 r1 : f1621d28 r0 : c2e68000 Apr 13 23:02:02 eisbaer kernel: Flags: Nzcv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none Apr 13 23:02:02 eisbaer kernel: Control: 50c5387d Table: 46a2c06a DAC: 00000051 Apr 13 23:02:02 eisbaer kernel: stmmac_get_stats64 from dev_get_stats+0x27/0xd0 Apr 13 23:02:02 eisbaer kernel: dev_get_stats from dev_seq_printf_stats+0x21/0x124 Apr 13 23:02:02 eisbaer kernel: dev_seq_printf_stats from dev_seq_show+0x11/0x24 Apr 13 23:02:02 eisbaer kernel: dev_seq_show from seq_read_iter+0x281/0x35c Apr 13 23:02:02 eisbaer kernel: seq_read_iter from seq_read+0x61/0x84 Apr 13 23:02:02 eisbaer kernel: seq_read from proc_reg_read+0x71/0x90 Apr 13 23:02:02 eisbaer kernel: proc_reg_read from vfs_read+0x75/0x1e4 Apr 13 23:02:02 eisbaer kernel: vfs_read from ksys_read+0x45/0x9c Apr 13 23:02:02 eisbaer kernel: ksys_read from ret_fast_syscall+0x1/0x5c Apr 13 23:02:02 eisbaer kernel: Exception stack(0xf1621fa8 to 0xf1621ff0) Apr 13 23:02:02 eisbaer kernel: 1fa0: 01141aa0 000005e8 00000004 011c7200 00000400 00000001 Apr 13 23:02:02 eisbaer kernel: 1fc0: 01141aa0 000005e8 b6e13888 00000003 0000000a bedcddd4 00000000 00000000 Apr 13 23:02:02 eisbaer kernel: 1fe0: 00000003 bedcdcd0 b6dae2bb b6d27616 0 Quote
Fabitanker Posted April 18 Posted April 18 Hello! I have a Banana-Pi M1 (Allwinner A20 SoC), where the same issue occurred since March 22nd, 2024. I upgraded from 6.1.63-current-sunxi to 6.6.16-current-sunxi at this date. I could observe in the logs how after some booted time the sysstat-collect services started by systemd took 20-40 seconds to complete. Normally it takes <100ms to do so, and after some time the cpu stalls completely, and the system becomes unresponsive if not rebooted. After reboot, everything fime for some hours, then issue occurs again. Switched to armbian edge (6.7.4-sunxi) kernel, issue persisted. Switched to armbian legacy (6.1.77-sunxi) kernel, issue disappeared. I hope the culprit can be found out, i am willing to assist in error reporting, since my system is non-critical. Thanks for the error report, this way i know i am not the only one! 0 Quote
Marcin66666 Posted May 10 Posted May 10 rcu: INFO: rcu_sched self-detected stall on CPU rcu: 0-....: (5249 ticks this GP) idle=5ec4/1/0x40000002 softirq=471875/471875 fqs=2602 rcu: (t=5250 jiffies g=738401 q=352 ncpus=4) CPU: 0 PID: 14211 Comm: octoprint Tainted: G C 6.6.16-current-sunxi #1 Hardware name: Allwinner sun8i Family PC is at stmmac_get_stats64+0x24/0x128 LR is at 0xc32bb000 pc : [<c078c5cc>] lr : [<c32bb000>] psr: a0010033 sp : e1fed990 ip : c32b8000 fp : 00000001 r10: 00000000 r9 : c95dee40 r8 : 00000000 r7 : 00000000 r6 : 00000001 r5 : 00000000 r4 : 80000000 r3 : 0093d4a3 r2 : c32bae48 r1 : c50ba6b8 r0 : c32b8000 Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none Control: 50c5387d Table: 497a006a DAC: 00000051 stmmac_get_stats64 from dev_get_stats+0x27/0xd0 dev_get_stats from rtnl_fill_stats+0x25/0xb0 rtnl_fill_stats from rtnl_fill_ifinfo+0x4cd/0xde0 rtnl_fill_ifinfo from rtnl_dump_ifinfo+0x1ff/0x49c rtnl_dump_ifinfo from netlink_dump+0xcd/0x270 netlink_dump from __netlink_dump_start+0x15b/0x1ec __netlink_dump_start from rtnetlink_rcv_msg+0x177/0x25c rtnetlink_rcv_msg from netlink_rcv_skb+0x75/0xb0 netlink_rcv_skb from netlink_unicast+0x1c1/0x204 netlink_unicast from netlink_sendmsg+0x185/0x354 netlink_sendmsg from __sock_sendmsg+0x27/0x48 __sock_sendmsg from __sys_sendto+0x7f/0xac __sys_sendto from ret_fast_syscall+0x1/0x5c Exception stack(0xe1fedfa8 to 0xe1fedff0) dfa0: aa039794 0000000c 00000029 aa0397b0 00000014 00000000 dfc0: aa039794 0000000c aa03b920 00000122 00000000 00000029 00000014 00000000 dfe0: 00000122 aa038738 b6dd4325 b6d3db06 Same issue here. I am using Orange pi one. Problem afer installing 6.6.16-current-sunxi Huge utylization of CPU / system freze, I also using, mjpg_streamer, I am metionig about this, because I can only make it working/unfreze only after getting feed from camera on that hardware. I know weird, but it is how it is. 0 Quote
Marcin66666 Posted May 13 Posted May 13 Just tested and after revert to Linux 6.1.77-legacy-sunxi no problems on my orangepi. 0 Quote
Valerij Posted May 17 Posted May 17 I have got the same issue. Orange Pi Zero H2+ Linux moblink 6.6.16-current-sunxi #1 SMP Fri Feb 23 08:25:28 UTC 2024 armv7l GNU/Linux Armbian 24.2.1 I used some software which is 1) playing some sound into embedded sound card 2) processing interrupts on gpio (gpio event with GPIO API V2) During sysstat-collect.service failing for 1min 47.293s: +sound playing OK - gpio interrupts proccessing freeze I had to stop and disable sysstat-collect.service Also apt remove sysstat It helped. Quote 2024-05-17T03:00:02.397369+03:00 moblink systemd[1]: Starting sysstat-collect.service - system activity accounting tool... 2024-05-17T03:00:23.412541+03:00 moblink kernel: [620821.695893] rcu: INFO: rcu_sched self-detected stall on CPU 2024-05-17T03:00:23.412672+03:00 moblink kernel: [620821.695932] rcu: 2-....: (5250 ticks this GP) idle=83fc/1/0x40000002 softirq=2045859/2045859 fqs=2625 2024-05-17T03:00:23.412698+03:00 moblink kernel: [620821.695958] rcu: (t=5251 jiffies g=6039277 q=138 ncpus=4) 2024-05-17T03:00:23.412796+03:00 moblink kernel: [620821.695977] CPU: 2 PID: 21477 Comm: sadc Not tainted 6.6.16-current-sunxi #1 2024-05-17T03:00:23.412820+03:00 moblink kernel: [620821.695991] Hardware name: Allwinner sun8i Family 2024-05-17T03:00:23.412835+03:00 moblink kernel: [620821.696000] PC is at stmmac_get_stats64+0x26/0x128 2024-05-17T03:00:23.412848+03:00 moblink kernel: [620821.696035] LR is at 0xc3453000 2024-05-17T03:00:23.412860+03:00 moblink kernel: [620821.696046] pc : [<c078c5ce>] lr : [<c3453000>] psr: a00f0033 2024-05-17T03:00:23.412873+03:00 moblink kernel: [620821.696057] sp : d1c3dc68 ip : c3450000 fp : 00000001 2024-05-17T03:00:23.412886+03:00 moblink kernel: [620821.696066] r10: d1c3de68 r9 : c4d78108 r8 : c3450000 2024-05-17T03:00:23.412898+03:00 moblink kernel: [620821.696077] r7 : 00000000 r6 : 00000001 r5 : 00000000 r4 : 80000000 2024-05-17T03:00:23.412911+03:00 moblink kernel: [620821.696087] r3 : 0048fc8f r2 : c3452e48 r1 : d1c3dd28 r0 : c3450000 2024-05-17T03:00:23.412924+03:00 moblink kernel: [620821.696099] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none 2024-05-17T03:00:23.412936+03:00 moblink kernel: [620821.696114] Control: 50c5387d Table: 458b406a DAC: 00000051 2024-05-17T03:00:23.412948+03:00 moblink kernel: [620821.696134] stmmac_get_stats64 from dev_get_stats+0x27/0xd0 2024-05-17T03:00:23.412964+03:00 moblink kernel: [620821.696165] dev_get_stats from dev_seq_printf_stats+0x21/0x124 2024-05-17T03:00:23.412978+03:00 moblink kernel: [620821.696191] dev_seq_printf_stats from dev_seq_show+0x11/0x24 2024-05-17T03:00:23.412990+03:00 moblink kernel: [620821.696211] dev_seq_show from seq_read_iter+0x281/0x35c 2024-05-17T03:00:23.413003+03:00 moblink kernel: [620821.696240] seq_read_iter from seq_read+0x61/0x84 2024-05-17T03:00:23.413017+03:00 moblink kernel: [620821.696263] seq_read from proc_reg_read+0x71/0x90 2024-05-17T03:00:23.413029+03:00 moblink kernel: [620821.696288] proc_reg_read from vfs_read+0x75/0x1e4 2024-05-17T03:00:23.413043+03:00 moblink kernel: [620821.696312] vfs_read from ksys_read+0x45/0x9c 2024-05-17T03:00:23.413056+03:00 moblink kernel: [620821.696330] ksys_read from ret_fast_syscall+0x1/0x5c 2024-05-17T03:00:23.413068+03:00 moblink kernel: [620821.696350] Exception stack(0xd1c3dfa8 to 0xd1c3dff0) 2024-05-17T03:00:23.413080+03:00 moblink kernel: [620821.696366] dfa0: 0209b1e8 000005e8 00000003 02093678 00000400 00000001 2024-05-17T03:00:23.413093+03:00 moblink kernel: [620821.696381] dfc0: 0209b1e8 000005e8 b6ed3888 00000003 0000000a be87c314 00000000 00000000 2024-05-17T03:00:23.413105+03:00 moblink kernel: [620821.696394] dfe0: 00000003 be87c268 b6e6e2bb b6de7616 2024-05-17T03:01:10.692536+03:00 moblink kernel: [620868.976559] rcu: INFO: rcu_sched self-detected stall on CPU 2024-05-17T03:01:10.692648+03:00 moblink kernel: [620868.976596] rcu: 3-....: (5250 ticks this GP) idle=0934/1/0x40000002 softirq=2017949/2017949 fqs=2454 2024-05-17T03:01:10.692672+03:00 moblink kernel: [620868.976622] rcu: (t=5251 jiffies g=6039281 q=215 ncpus=4) 2024-05-17T03:01:10.692689+03:00 moblink kernel: [620868.976640] CPU: 3 PID: 21477 Comm: sadc Not tainted 6.6.16-current-sunxi #1 2024-05-17T03:01:10.692712+03:00 moblink kernel: [620868.976654] Hardware name: Allwinner sun8i Family 2024-05-17T03:01:10.692729+03:00 moblink kernel: [620868.976663] PC is at stmmac_get_stats64+0x26/0x128 2024-05-17T03:01:10.692741+03:00 moblink kernel: [620868.976698] LR is at 0xc3453000 2024-05-17T03:01:10.692753+03:00 moblink kernel: [620868.976709] pc : [<c078c5ce>] lr : [<c3453000>] psr: 800f0033 2024-05-17T03:01:10.692766+03:00 moblink kernel: [620868.976719] sp : d1c3dc68 ip : c3450000 fp : 00000001 2024-05-17T03:01:10.692779+03:00 moblink kernel: [620868.976729] r10: d1c3de68 r9 : c4dfb3d8 r8 : c3450000 2024-05-17T03:01:10.692791+03:00 moblink kernel: [620868.976740] r7 : 00000000 r6 : 00000001 r5 : 00000000 r4 : 80000000 2024-05-17T03:01:10.692802+03:00 moblink kernel: [620868.976750] r3 : 0048fc95 r2 : c3452e48 r1 : d1c3dd28 r0 : c3450000 2024-05-17T03:01:10.692814+03:00 moblink kernel: [620868.976762] Flags: Nzcv IRQs on FIQs on Mode SVC_32 ISA Thumb Segment none 2024-05-17T03:01:10.692827+03:00 moblink kernel: [620868.976776] Control: 50c5387d Table: 458b406a DAC: 00000051 2024-05-17T03:01:10.692839+03:00 moblink kernel: [620868.976797] stmmac_get_stats64 from dev_get_stats+0x27/0xd0 2024-05-17T03:01:10.692851+03:00 moblink kernel: [620868.976828] dev_get_stats from dev_seq_printf_stats+0x21/0x124 2024-05-17T03:01:10.692863+03:00 moblink kernel: [620868.976853] dev_seq_printf_stats from dev_seq_show+0x11/0x24 2024-05-17T03:01:10.692875+03:00 moblink kernel: [620868.976874] dev_seq_show from seq_read_iter+0x281/0x35c 2024-05-17T03:01:10.692887+03:00 moblink kernel: [620868.976902] seq_read_iter from seq_read+0x61/0x84 2024-05-17T03:01:10.692901+03:00 moblink kernel: [620868.976925] seq_read from proc_reg_read+0x71/0x90 2024-05-17T03:01:10.692913+03:00 moblink kernel: [620868.976950] proc_reg_read from vfs_read+0x75/0x1e4 2024-05-17T03:01:10.692925+03:00 moblink kernel: [620868.976972] vfs_read from ksys_read+0x45/0x9c 2024-05-17T03:01:10.692937+03:00 moblink kernel: [620868.976990] ksys_read from ret_fast_syscall+0x1/0x5c 2024-05-17T03:01:10.692949+03:00 moblink kernel: [620868.977010] Exception stack(0xd1c3dfa8 to 0xd1c3dff0) 2024-05-17T03:01:10.692962+03:00 moblink kernel: [620868.977026] dfa0: 0209b1e8 000005e8 00000004 02093a28 00000400 00000001 2024-05-17T03:01:10.692974+03:00 moblink kernel: [620868.977041] dfc0: 0209b1e8 000005e8 b6ed3888 00000003 0000000a be87c1fc 00000000 00000000 2024-05-17T03:01:10.693097+03:00 moblink kernel: [620868.977054] dfe0: 00000003 be87c128 b6e6e2bb b6de7616 2024-05-17T03:01:49.688246+03:00 moblink systemd[1]: sysstat-collect.service: Deactivated successfully. 2024-05-17T03:01:49.689170+03:00 moblink systemd[1]: Finished sysstat-collect.service - system activity accounting tool. 2024-05-17T03:01:49.691186+03:00 moblink systemd[1]: sysstat-collect.service: Consumed 1min 47.293s CPU time. cat armbian-image-release Quote # PLEASE DO NOT EDIT THIS FILE BOARD=orangepizero BOARD_NAME="Orange Pi Zero" BOARDFAMILY=sun8i BUILD_REPOSITORY_URL=https://github.com/armbian/build BUILD_REPOSITORY_COMMIT=edd866b8c LINUXFAMILY=sunxi ARCH=arm IMAGE_TYPE=stable BOARD_TYPE=conf INITRD_ARCH=arm KERNEL_IMAGE_TYPE=zImage FORCE_BOOTSCRIPT_UPDATE= FORCE_UBOOT_UPDATE= VENDOR="Armbian" VENDORDOCS="https://docs.armbian.com" VENDORURL="https://www.armbian.com" VENDORSUPPORT="https://forum.armbian.com" VENDORBUGS="https://www.armbian.com/bugs" BOOTSCRIPT_FORCE_UPDATE="no" BOOTSCRIPT_DST="boot.cmd" VERSION=24.2.1 REVISION=24.2.1 IMAGE_UUID=ace87232-ab6a-44d2-afb6-8f24225ba9f7 cat armbian-release Quote # PLEASE DO NOT EDIT THIS FILE BOARD=orangepizero BOARD_NAME="Orange Pi Zero" BOARDFAMILY=sun8i BUILD_REPOSITORY_URL=https://github.com/armbian/build BUILD_REPOSITORY_COMMIT=edd866b8c LINUXFAMILY=sunxi ARCH=arm IMAGE_TYPE=stable BOARD_TYPE=conf INITRD_ARCH=arm KERNEL_IMAGE_TYPE=zImage FORCE_BOOTSCRIPT_UPDATE= FORCE_UBOOT_UPDATE= VENDOR="Armbian" VENDORDOCS="https://docs.armbian.com" VENDORURL="https://www.armbian.com" VENDORSUPPORT="https://forum.armbian.com" VENDORBUGS="https://www.armbian.com/bugs" BOOTSCRIPT_FORCE_UPDATE="no" BOOTSCRIPT_DST="boot.cmd" VERSION=24.2.1 REVISION=24.2.1 BRANCH=current cat os-release Quote PRETTY_NAME="Armbian 24.2.1 bookworm" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.armbian.com" SUPPORT_URL="https://forum.armbian.com" BUG_REPORT_URL="https://www.armbian.com/bugs" ARMBIAN_PRETTY_NAME="Armbian 24.2.1 bookworm" 0 Quote
Solution Myron Posted May 28 Author Solution Posted May 28 (edited) So... Taken the latest Armbian system update and on kernel version 6.6.31 no more CPU stalls. 🙂 Whoever fixed this, thank-you. Edited May 28 by Myron 2 Quote
NStorm Posted June 13 Posted June 13 CVE summary: https://www.cve.org/CVERecord?id=CVE-2024-26690 Link to a thread on kernel mailing list with bug report and proposed fix: https://lore.kernel.org/lkml/Zb8QwgYSh9sX8zBi@xhacker/T/ 0 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.