Jump to content

Frequent `rcu: INFO: rcu_sched detected stalls on CPUs/tasks` events logged.

Myron

By Myron
in Allwinner sunxi

 Share
Go to solution Solved by Myron,

Recommended Posts

Banana.31415

Banana.31415

Posted
Posted

I can confirm this issue on an old BananaPi 1 from Lemaker. It began with Kernel 6.6.

After returning to the older 6.1 the stalls disappeared.

  

Apr 13 23:02:02 eisbaer kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Apr 13 23:02:02 eisbaer kernel: rcu:         0-....: (5249 ticks this GP) idle=56dc/1/0x40000002 softirq=5480265/5480265 fqs=2606
Apr 13 23:02:02 eisbaer kernel: rcu:         (t=5250 jiffies g=9317845 q=1167 ncpus=2)
Apr 13 23:02:02 eisbaer kernel: CPU: 0 PID: 27606 Comm: htop Tainted: G         C         6.6.16-current-sunxi #1
Apr 13 23:02:02 eisbaer kernel: Hardware name: Allwinner sun7i (A20) Family
Apr 13 23:02:02 eisbaer kernel: PC is at stmmac_get_stats64+0x26/0x128
Apr 13 23:02:02 eisbaer kernel: LR is at 0xc2e6b000
Apr 13 23:02:02 eisbaer kernel: pc : [<c078c5ce>]    lr : [<c2e6b000>]    psr: 80010033
Apr 13 23:02:02 eisbaer kernel: sp : f1621c68  ip : c2e68000  fp : 00000001
Apr 13 23:02:02 eisbaer kernel: r10: f1621e68  r9 : c2ee3c48  r8 : c2e68000
Apr 13 23:02:02 eisbaer kernel: r7 : 00000000  r6 : 00000001  r5 : 00000000  r4 : 80000000
Apr 13 23:02:02 eisbaer kernel: r3 : 07c697cd  r2 : c2e6ae48  r1 : f1621d28  r0 : c2e68000
Apr 13 23:02:02 eisbaer kernel: Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
Apr 13 23:02:02 eisbaer kernel: Control: 50c5387d  Table: 46a2c06a  DAC: 00000051
Apr 13 23:02:02 eisbaer kernel:  stmmac_get_stats64 from dev_get_stats+0x27/0xd0
Apr 13 23:02:02 eisbaer kernel:  dev_get_stats from dev_seq_printf_stats+0x21/0x124
Apr 13 23:02:02 eisbaer kernel:  dev_seq_printf_stats from dev_seq_show+0x11/0x24
Apr 13 23:02:02 eisbaer kernel:  dev_seq_show from seq_read_iter+0x281/0x35c
Apr 13 23:02:02 eisbaer kernel:  seq_read_iter from seq_read+0x61/0x84
Apr 13 23:02:02 eisbaer kernel:  seq_read from proc_reg_read+0x71/0x90
Apr 13 23:02:02 eisbaer kernel:  proc_reg_read from vfs_read+0x75/0x1e4
Apr 13 23:02:02 eisbaer kernel:  vfs_read from ksys_read+0x45/0x9c
Apr 13 23:02:02 eisbaer kernel:  ksys_read from ret_fast_syscall+0x1/0x5c
Apr 13 23:02:02 eisbaer kernel: Exception stack(0xf1621fa8 to 0xf1621ff0)
Apr 13 23:02:02 eisbaer kernel: 1fa0:                   01141aa0 000005e8 00000004 011c7200 00000400 00000001
Apr 13 23:02:02 eisbaer kernel: 1fc0: 01141aa0 000005e8 b6e13888 00000003 0000000a bedcddd4 00000000 00000000
Apr 13 23:02:02 eisbaer kernel: 1fe0: 00000003 bedcdcd0 b6dae2bb b6d27616

 

0
Link to comment
Share on other sites
Fabitanker

Fabitanker

Posted
Posted

Hello!
I have a Banana-Pi M1 (Allwinner A20 SoC), where the same issue occurred since March 22nd, 2024.
I upgraded from 6.1.63-current-sunxi to 6.6.16-current-sunxi at this date.
I could observe in the logs how after some booted time the sysstat-collect services started by systemd took 20-40 seconds to complete.

Normally it takes <100ms to do so, and after some time the cpu stalls completely, and the system becomes unresponsive if not rebooted.

After reboot, everything fime for some hours, then issue occurs again.

Switched to armbian edge (6.7.4-sunxi) kernel, issue persisted.

Switched to armbian legacy (6.1.77-sunxi) kernel, issue disappeared.

I hope the culprit can be found out, i am willing to assist in error reporting, since my system is non-critical.

 

Thanks for the error report, this way i know i am not the only one!

 

0
Link to comment
Share on other sites
Marcin66666

Marcin66666

Posted
Posted

rcu: INFO: rcu_sched self-detected stall on CPU
rcu:       0-....: (5249 ticks this GP) idle=5ec4/1/0x40000002 softirq=471875/471875 fqs=2602
rcu:       (t=5250 jiffies g=738401 q=352 ncpus=4)
CPU: 0 PID: 14211 Comm: octoprint Tainted: G         C         6.6.16-current-sunxi #1
Hardware name: Allwinner sun8i Family
PC is at stmmac_get_stats64+0x24/0x128
LR is at 0xc32bb000
pc : [<c078c5cc>]    lr : [<c32bb000>]    psr: a0010033
sp : e1fed990  ip : c32b8000  fp : 00000001
r10: 00000000  r9 : c95dee40  r8 : 00000000
r7 : 00000000  r6 : 00000001  r5 : 00000000  r4 : 80000000
r3 : 0093d4a3  r2 : c32bae48  r1 : c50ba6b8  r0 : c32b8000
Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
Control: 50c5387d  Table: 497a006a  DAC: 00000051
 stmmac_get_stats64 from dev_get_stats+0x27/0xd0
 dev_get_stats from rtnl_fill_stats+0x25/0xb0
 rtnl_fill_stats from rtnl_fill_ifinfo+0x4cd/0xde0
 rtnl_fill_ifinfo from rtnl_dump_ifinfo+0x1ff/0x49c
 rtnl_dump_ifinfo from netlink_dump+0xcd/0x270
 netlink_dump from __netlink_dump_start+0x15b/0x1ec
 __netlink_dump_start from rtnetlink_rcv_msg+0x177/0x25c
 rtnetlink_rcv_msg from netlink_rcv_skb+0x75/0xb0
 netlink_rcv_skb from netlink_unicast+0x1c1/0x204
 netlink_unicast from netlink_sendmsg+0x185/0x354
 netlink_sendmsg from __sock_sendmsg+0x27/0x48
 __sock_sendmsg from __sys_sendto+0x7f/0xac
 __sys_sendto from ret_fast_syscall+0x1/0x5c
Exception stack(0xe1fedfa8 to 0xe1fedff0)
dfa0:                   aa039794 0000000c 00000029 aa0397b0 00000014 00000000
dfc0: aa039794 0000000c aa03b920 00000122 00000000 00000029 00000014 00000000
dfe0: 00000122 aa038738 b6dd4325 b6d3db06

 

Same issue here.

I am using Orange pi one.

Problem afer installing  6.6.16-current-sunxi

 

Huge utylization of CPU / system freze, I also using, mjpg_streamer, I am metionig about this, because I can only make it working/unfreze only after getting feed from camera on that hardware. I know weird, but it is how it is.

 

0
Link to comment
Share on other sites
Valerij

Valerij

Posted
Posted

I have got the same issue.

Orange Pi Zero H2+

Linux moblink 6.6.16-current-sunxi #1 SMP Fri Feb 23 08:25:28 UTC 2024 armv7l GNU/Linux

Armbian 24.2.1

 

I used some software which is

1) playing some sound into embedded sound card

2) processing  interrupts on gpio (gpio event with GPIO API V2)

 

During sysstat-collect.service failing for 1min 47.293s:

+sound playing OK

- gpio interrupts proccessing freeze

 

I had to stop and disable sysstat-collect.service

Also apt remove sysstat

It helped.

 


 

 

 

 

Quote

2024-05-17T03:00:02.397369+03:00 moblink systemd[1]: Starting sysstat-collect.service - system activity accounting tool...
2024-05-17T03:00:23.412541+03:00 moblink kernel: [620821.695893] rcu: INFO: rcu_sched self-detected stall on CPU
2024-05-17T03:00:23.412672+03:00 moblink kernel: [620821.695932] rcu:   2-....: (5250 ticks this GP) idle=83fc/1/0x40000002 softirq=2045859/2045859 fqs=2625
2024-05-17T03:00:23.412698+03:00 moblink kernel: [620821.695958] rcu:   (t=5251 jiffies g=6039277 q=138 ncpus=4)
2024-05-17T03:00:23.412796+03:00 moblink kernel: [620821.695977] CPU: 2 PID: 21477 Comm: sadc Not tainted 6.6.16-current-sunxi #1
2024-05-17T03:00:23.412820+03:00 moblink kernel: [620821.695991] Hardware name: Allwinner sun8i Family
2024-05-17T03:00:23.412835+03:00 moblink kernel: [620821.696000] PC is at stmmac_get_stats64+0x26/0x128
2024-05-17T03:00:23.412848+03:00 moblink kernel: [620821.696035] LR is at 0xc3453000
2024-05-17T03:00:23.412860+03:00 moblink kernel: [620821.696046] pc : [<c078c5ce>]    lr : [<c3453000>]    psr: a00f0033
2024-05-17T03:00:23.412873+03:00 moblink kernel: [620821.696057] sp : d1c3dc68  ip : c3450000  fp : 00000001
2024-05-17T03:00:23.412886+03:00 moblink kernel: [620821.696066] r10: d1c3de68  r9 : c4d78108  r8 : c3450000
2024-05-17T03:00:23.412898+03:00 moblink kernel: [620821.696077] r7 : 00000000  r6 : 00000001  r5 : 00000000  r4 : 80000000
2024-05-17T03:00:23.412911+03:00 moblink kernel: [620821.696087] r3 : 0048fc8f  r2 : c3452e48  r1 : d1c3dd28  r0 : c3450000
2024-05-17T03:00:23.412924+03:00 moblink kernel: [620821.696099] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
2024-05-17T03:00:23.412936+03:00 moblink kernel: [620821.696114] Control: 50c5387d  Table: 458b406a  DAC: 00000051
2024-05-17T03:00:23.412948+03:00 moblink kernel: [620821.696134]  stmmac_get_stats64 from dev_get_stats+0x27/0xd0
2024-05-17T03:00:23.412964+03:00 moblink kernel: [620821.696165]  dev_get_stats from dev_seq_printf_stats+0x21/0x124
2024-05-17T03:00:23.412978+03:00 moblink kernel: [620821.696191]  dev_seq_printf_stats from dev_seq_show+0x11/0x24
2024-05-17T03:00:23.412990+03:00 moblink kernel: [620821.696211]  dev_seq_show from seq_read_iter+0x281/0x35c
2024-05-17T03:00:23.413003+03:00 moblink kernel: [620821.696240]  seq_read_iter from seq_read+0x61/0x84
2024-05-17T03:00:23.413017+03:00 moblink kernel: [620821.696263]  seq_read from proc_reg_read+0x71/0x90
2024-05-17T03:00:23.413029+03:00 moblink kernel: [620821.696288]  proc_reg_read from vfs_read+0x75/0x1e4
2024-05-17T03:00:23.413043+03:00 moblink kernel: [620821.696312]  vfs_read from ksys_read+0x45/0x9c
2024-05-17T03:00:23.413056+03:00 moblink kernel: [620821.696330]  ksys_read from ret_fast_syscall+0x1/0x5c
2024-05-17T03:00:23.413068+03:00 moblink kernel: [620821.696350] Exception stack(0xd1c3dfa8 to 0xd1c3dff0)
2024-05-17T03:00:23.413080+03:00 moblink kernel: [620821.696366] dfa0:                   0209b1e8 000005e8 00000003 02093678 00000400 00000001
2024-05-17T03:00:23.413093+03:00 moblink kernel: [620821.696381] dfc0: 0209b1e8 000005e8 b6ed3888 00000003 0000000a be87c314 00000000 00000000
2024-05-17T03:00:23.413105+03:00 moblink kernel: [620821.696394] dfe0: 00000003 be87c268 b6e6e2bb b6de7616
2024-05-17T03:01:10.692536+03:00 moblink kernel: [620868.976559] rcu: INFO: rcu_sched self-detected stall on CPU
2024-05-17T03:01:10.692648+03:00 moblink kernel: [620868.976596] rcu:   3-....: (5250 ticks this GP) idle=0934/1/0x40000002 softirq=2017949/2017949 fqs=2454
2024-05-17T03:01:10.692672+03:00 moblink kernel: [620868.976622] rcu:   (t=5251 jiffies g=6039281 q=215 ncpus=4)
2024-05-17T03:01:10.692689+03:00 moblink kernel: [620868.976640] CPU: 3 PID: 21477 Comm: sadc Not tainted 6.6.16-current-sunxi #1
2024-05-17T03:01:10.692712+03:00 moblink kernel: [620868.976654] Hardware name: Allwinner sun8i Family
2024-05-17T03:01:10.692729+03:00 moblink kernel: [620868.976663] PC is at stmmac_get_stats64+0x26/0x128
2024-05-17T03:01:10.692741+03:00 moblink kernel: [620868.976698] LR is at 0xc3453000
2024-05-17T03:01:10.692753+03:00 moblink kernel: [620868.976709] pc : [<c078c5ce>]    lr : [<c3453000>]    psr: 800f0033
2024-05-17T03:01:10.692766+03:00 moblink kernel: [620868.976719] sp : d1c3dc68  ip : c3450000  fp : 00000001
2024-05-17T03:01:10.692779+03:00 moblink kernel: [620868.976729] r10: d1c3de68  r9 : c4dfb3d8  r8 : c3450000
2024-05-17T03:01:10.692791+03:00 moblink kernel: [620868.976740] r7 : 00000000  r6 : 00000001  r5 : 00000000  r4 : 80000000
2024-05-17T03:01:10.692802+03:00 moblink kernel: [620868.976750] r3 : 0048fc95  r2 : c3452e48  r1 : d1c3dd28  r0 : c3450000
2024-05-17T03:01:10.692814+03:00 moblink kernel: [620868.976762] Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
2024-05-17T03:01:10.692827+03:00 moblink kernel: [620868.976776] Control: 50c5387d  Table: 458b406a  DAC: 00000051
2024-05-17T03:01:10.692839+03:00 moblink kernel: [620868.976797]  stmmac_get_stats64 from dev_get_stats+0x27/0xd0
2024-05-17T03:01:10.692851+03:00 moblink kernel: [620868.976828]  dev_get_stats from dev_seq_printf_stats+0x21/0x124
2024-05-17T03:01:10.692863+03:00 moblink kernel: [620868.976853]  dev_seq_printf_stats from dev_seq_show+0x11/0x24
2024-05-17T03:01:10.692875+03:00 moblink kernel: [620868.976874]  dev_seq_show from seq_read_iter+0x281/0x35c
2024-05-17T03:01:10.692887+03:00 moblink kernel: [620868.976902]  seq_read_iter from seq_read+0x61/0x84
2024-05-17T03:01:10.692901+03:00 moblink kernel: [620868.976925]  seq_read from proc_reg_read+0x71/0x90
2024-05-17T03:01:10.692913+03:00 moblink kernel: [620868.976950]  proc_reg_read from vfs_read+0x75/0x1e4
2024-05-17T03:01:10.692925+03:00 moblink kernel: [620868.976972]  vfs_read from ksys_read+0x45/0x9c
2024-05-17T03:01:10.692937+03:00 moblink kernel: [620868.976990]  ksys_read from ret_fast_syscall+0x1/0x5c
2024-05-17T03:01:10.692949+03:00 moblink kernel: [620868.977010] Exception stack(0xd1c3dfa8 to 0xd1c3dff0)
2024-05-17T03:01:10.692962+03:00 moblink kernel: [620868.977026] dfa0:                   0209b1e8 000005e8 00000004 02093a28 00000400 00000001
2024-05-17T03:01:10.692974+03:00 moblink kernel: [620868.977041] dfc0: 0209b1e8 000005e8 b6ed3888 00000003 0000000a be87c1fc 00000000 00000000
2024-05-17T03:01:10.693097+03:00 moblink kernel: [620868.977054] dfe0: 00000003 be87c128 b6e6e2bb b6de7616
2024-05-17T03:01:49.688246+03:00 moblink systemd[1]: sysstat-collect.service: Deactivated successfully.
2024-05-17T03:01:49.689170+03:00 moblink systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
2024-05-17T03:01:49.691186+03:00 moblink systemd[1]: sysstat-collect.service: Consumed 1min 47.293s CPU time.
 

 

 

 

 

 

cat armbian-image-release

Quote

# PLEASE DO NOT EDIT THIS FILE
BOARD=orangepizero
BOARD_NAME="Orange Pi Zero"
BOARDFAMILY=sun8i
BUILD_REPOSITORY_URL=https://github.com/armbian/build
BUILD_REPOSITORY_COMMIT=edd866b8c
LINUXFAMILY=sunxi
ARCH=arm
IMAGE_TYPE=stable
BOARD_TYPE=conf
INITRD_ARCH=arm
KERNEL_IMAGE_TYPE=zImage
FORCE_BOOTSCRIPT_UPDATE=
FORCE_UBOOT_UPDATE=
VENDOR="Armbian"
VENDORDOCS="https://docs.armbian.com"
VENDORURL="https://www.armbian.com"
VENDORSUPPORT="https://forum.armbian.com"
VENDORBUGS="https://www.armbian.com/bugs"
BOOTSCRIPT_FORCE_UPDATE="no"
BOOTSCRIPT_DST="boot.cmd"
VERSION=24.2.1
REVISION=24.2.1
IMAGE_UUID=ace87232-ab6a-44d2-afb6-8f24225ba9f7

 

cat armbian-release

Quote

# PLEASE DO NOT EDIT THIS FILE
BOARD=orangepizero
BOARD_NAME="Orange Pi Zero"
BOARDFAMILY=sun8i
BUILD_REPOSITORY_URL=https://github.com/armbian/build
BUILD_REPOSITORY_COMMIT=edd866b8c
LINUXFAMILY=sunxi
ARCH=arm
IMAGE_TYPE=stable
BOARD_TYPE=conf
INITRD_ARCH=arm
KERNEL_IMAGE_TYPE=zImage
FORCE_BOOTSCRIPT_UPDATE=
FORCE_UBOOT_UPDATE=
VENDOR="Armbian"
VENDORDOCS="https://docs.armbian.com"
VENDORURL="https://www.armbian.com"
VENDORSUPPORT="https://forum.armbian.com"
VENDORBUGS="https://www.armbian.com/bugs"
BOOTSCRIPT_FORCE_UPDATE="no"
BOOTSCRIPT_DST="boot.cmd"
VERSION=24.2.1
REVISION=24.2.1
BRANCH=current

 

 

cat os-release

Quote

PRETTY_NAME="Armbian 24.2.1 bookworm"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.armbian.com"
SUPPORT_URL="https://forum.armbian.com"
BUG_REPORT_URL="https://www.armbian.com/bugs"
ARMBIAN_PRETTY_NAME="Armbian 24.2.1 bookworm"

 

0
Link to comment
Share on other sites
  • Solution
Myron

Myron

Posted
  • Author
  • Solution
Posted (edited)

So...  Taken the latest Armbian system update and on kernel version  6.6.31 no more CPU stalls.  🙂

Whoever fixed this, thank-you.

Edited by Myron
2
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
 Share
Go to topic listing
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines

  I accept