high load, low cpu, lscpu, armbian-hardware-optimization, scaling_available_governors

[krrmbn]

Armbianmonitor:

https://paste.armbian.com/Error

My Orange Pi 3 LTS with Armbian 23.11.1 Jammy with Linux 6.1.63-current-sunxi64 is showing high load, but low CPU. The SBC crashed overnight.

 

When I run htop, the following lines show state D:

root     lscpu

root     /bin/bash /usr/lib/armbian/armbian-hardware-optimization start

root     grep -q \<ondemand\> /sys/devices/cpu/cpu0/cpufreq/scaling_available_governors

 

load average: 4.00, 4.00, 4.00

System load:   100% 

 

CPU usage: less than 1%

 

sar -u 5                                                          

Linux 6.1.63-current-sunxi64 (hostname) 12/31/2023 _aarch64_ (4 CPU)

 

03:41:54 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle

03:41:59 PM     all         0.50         0.00         0.35         0.00         0.00        99.15

 

I'm up to date with software (sudo apt update && sudo apt upgrade -y).

 

I am unable to kill -9 the offending processes.

 

When I try to reboot, I get locked out of ssh, but SBC does not reboot, and keeps replying to pings. I have to end up pulling the plug to get it to reboot.

 

After plugging it back in, it starts out without the problem, but the problem resurfaces shortly thereafter.

 

My other three OPI3LTS with Armbian 23.11.1 Jammy with Linux 6.1.63-current-sunxi64 are not having this issue.

 

Thank you for your assistance, and Happy New Year!

[krrmbn]

Ok, seems to have sorted itself out, and has not reoccured since the last time I had to force a reboot - about 4 hours ago. 

[krrmbn]

Ok, not sorted. Crashed again a couple of times. It seems as though all log files (syslog for example) are erased and started again at reboot, so I am unable to see what's happening to cause the crashes.

 

I'm running minimal CLI headless, just running pi-hole, ssh, and fail2ban. Plenty of RAM, plenty of eMMC space available.

 

I'm not sure where to look to sort this out.

[krrmbn]

This is now happening to a second Orange Pi 3 LTS running armbian with all of the updates.

[krrmbn]

So I am having this issue on two Orange Pi 3 LTS running armbian jammy.

 

They run at ~4 load, and then eventually crash.

 

% sudo systemctl --failed

  UNIT                 LOAD   ACTIVE SUB    DESCRIPTION                       

● cpufrequtils.service loaded failed failed LSB: set CPUFreq kernel parameters

 

LOAD   = Reflects whether the unit definition was properly loaded.

ACTIVE = The high-level unit activation state, i.e. generalization of SUB.

SUB    = The low-level unit activation state, values depend on unit type.

1 loaded units listed.

[krrmbn]

this is redonkulous

 

I'm watching ~5 load and can't do anything to stop the scb from crashing repeatedly.

 

All it is running is pi-hole, so of course it renders my lan inoperable when it repeatedly fails.

[SteeMan]

Moved to Community Maintained Allwinner.

 

@krrmbn Note that this is not an Armbian supported board.  It is community maintained.

[Gunjan Gupta]

If you are able to login to the system. Please share the output of dmesg and "ps aux --sort=-pcpu"

[Gunjan Gupta]

@krrmbnWere these boards using a fresh image or were they upgraded from some previous version? if they were upgraded, check if /etc/modules list a module like sprdwl_ng. If yes remove the same. Then enable aw859a-wifi.service service for wifi support. Now reboot and your problem should be resolved.

[krrmbn]

hostname:~:% ps aux --sort=-pcpu
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
username   15243 18.6  0.2  10224  5912 pts/1    Ss   01:43   0:00 -zsh
root       15101  6.0  0.4  17532  9152 ?        Ss   01:43   0:00 sshd: username [priv]
username    1670  2.1  3.0 363120 61204 pts/0    Rl+  Jan24  23:12 /usr/bin/python3 /usr/bin/glances -t 15
username   15242  0.3  0.2  17792  6104 ?        S    01:43   0:00 sshd: username@pts/1
pihole      1020  0.2  0.7 365568 16260 ?        S<sl Jan24   2:58 /usr/bin/pihole-FTL -f
root           1  0.0  0.5 167584 10760 ?        Ss   Jan24   0:06 /sbin/init
root           2  0.0  0.0      0     0 ?        S    Jan24   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   Jan24   0:00 [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   Jan24   0:00 [rcu_par_gp]
root           5  0.0  0.0      0     0 ?        I<   Jan24   0:00 [slub_flushwq]
root           6  0.0  0.0      0     0 ?        I<   Jan24   0:00 [netns]
root           7  0.0  0.0      0     0 ?        I    Jan24   0:02 [kworker/0:0-events]
root          10  0.0  0.0      0     0 ?        I<   Jan24   0:00 [mm_percpu_wq]
root          11  0.0  0.0      0     0 ?        I    Jan24   0:00 [rcu_tasks_trace_kthread]
root          12  0.0  0.0      0     0 ?        S    Jan24   0:00 [ksoftirqd/0]
root          13  0.0  0.0      0     0 ?        I    Jan24   0:07 [rcu_sched]
root          14  0.0  0.0      0     0 ?        S    Jan24   0:00 [migration/0]
root          15  0.0  0.0      0     0 ?        S    Jan24   0:00 [cpuhp/0]
root          16  0.0  0.0      0     0 ?        S    Jan24   0:00 [cpuhp/1]
root          17  0.0  0.0      0     0 ?        S    Jan24   0:00 [migration/1]
root          18  0.0  0.0      0     0 ?        S    Jan24   0:00 [ksoftirqd/1]
root          21  0.0  0.0      0     0 ?        S    Jan24   0:00 [cpuhp/2]
root          22  0.0  0.0      0     0 ?        S    Jan24   0:00 [migration/2]
root          23  0.0  0.0      0     0 ?        S    Jan24   0:00 [ksoftirqd/2]
root          26  0.0  0.0      0     0 ?        S    Jan24   0:00 [cpuhp/3]
root          27  0.0  0.0      0     0 ?        S    Jan24   0:00 [migration/3]
root          28  0.0  0.0      0     0 ?        S    Jan24   0:00 [ksoftirqd/3]
root          31  0.0  0.0      0     0 ?        S    Jan24   0:00 [kdevtmpfs]
root          32  0.0  0.0      0     0 ?        I<   Jan24   0:00 [inet_frag_wq]
root          35  0.0  0.0      0     0 ?        S    Jan24   0:00 [kauditd]
root          36  0.0  0.0      0     0 ?        S    Jan24   0:00 [oom_reaper]
root          37  0.0  0.0      0     0 ?        I<   Jan24   0:00 [writeback]
root          38  0.0  0.0      0     0 ?        S    Jan24   0:02 [kcompactd0]
root          39  0.0  0.0      0     0 ?        SN   Jan24   0:00 [ksmd]
root          40  0.0  0.0      0     0 ?        SN   Jan24   0:00 [khugepaged]
root          41  0.0  0.0      0     0 ?        I<   Jan24   0:00 [cryptd]
root          42  0.0  0.0      0     0 ?        I<   Jan24   0:00 [kintegrityd]
root          43  0.0  0.0      0     0 ?        I<   Jan24   0:00 [kblockd]
root          44  0.0  0.0      0     0 ?        I<   Jan24   0:00 [blkcg_punt_bio]
root          45  0.0  0.0      0     0 ?        I<   Jan24   0:00 [tpm_dev_wq]
root          46  0.0  0.0      0     0 ?        I<   Jan24   0:00 [md]
root          47  0.0  0.0      0     0 ?        I<   Jan24   0:00 [devfreq_wq]
root          48  0.0  0.0      0     0 ?        S    Jan24   0:00 [watchdogd]
root          51  0.0  0.0      0     0 ?        I    Jan24   0:11 [kworker/2:1-events]
root          52  0.0  0.0      0     0 ?        S    Jan24   0:00 [kswapd0]
root          53  0.0  0.0      0     0 ?        S    Jan24   0:00 [ecryptfs-kthread]
root          61  0.0  0.0      0     0 ?        I<   Jan24   0:00 [kthrotld]
root          68  0.0  0.0      0     0 ?        I<   Jan24   0:00 [raid5wq]
root          69  0.0  0.0      0     0 ?        I<   Jan24   0:00 [bch_btree_io]
root          70  0.0  0.0      0     0 ?        I<   Jan24   0:00 [bcache]
root          71  0.0  0.0      0     0 ?        I<   Jan24   0:00 [bch_journal]
root          72  0.0  0.0      0     0 ?        S    Jan24   0:00 [1904000.crypto-engine]
root          73  0.0  0.0      0     0 ?        S    Jan24   0:00 [1904000.crypto-engine]
root          74  0.0  0.0      0     0 ?        S    Jan24   0:00 [1904000.crypto-engine]
root          75  0.0  0.0      0     0 ?        S    Jan24   0:00 [1904000.crypto-engine]
root          76  0.0  0.0      0     0 ?        S    Jan24   0:00 [hwrng]
root          77  0.0  0.0      0     0 ?        I<   Jan24   0:00 [tls-strp]
root          78  0.0  0.0      0     0 ?        I<   Jan24   0:00 [mld]
root          79  0.0  0.0      0     0 ?        I<   Jan24   0:00 [ipv6_addrconf]
root          80  0.0  0.0      0     0 ?        I<   Jan24   0:00 [kworker/2:1H-kblockd]
root          85  0.0  0.0      0     0 ?        I<   Jan24   0:00 [kstrp]
root          87  0.0  0.0      0     0 ?        I<   Jan24   0:00 [zswap1]
root          88  0.0  0.0      0     0 ?        I<   Jan24   0:00 [zswap1]
root          89  0.0  0.0      0     0 ?        I<   Jan24   0:00 [zswap-shrink]
root          90  0.0  0.0      0     0 ?        I<   Jan24   0:00 [kworker/u9:0-sprdwl_work]
root         174  0.0  0.0      0     0 ?        I    Jan24   0:02 [kworker/1:3-events]
root         175  0.0  0.0      0     0 ?        S    Jan24   0:00 [irq/92-axp806]
root         176  0.0  0.0      0     0 ?        S    Jan24   0:05 [irq/105-ths]
root         180  0.0  0.0      0     0 ?        S    Jan24   0:00 [irq/240-6000000.hdmi]
root         181  0.0  0.0      0     0 ?        S    Jan24   0:00 [card0-crtc0]
root         185  0.0  0.0      0     0 ?        S    Jan24   0:00 [irq/242-sunxi-mmc]
root         187  0.0  0.0      0     0 ?        S    Jan24   0:00 [irq/243-sunxi-mmc]
root         192  0.0  0.0      0     0 ?        S    Jan24   0:00 [irq/249-sunxi-mmc]
root         194  0.0  0.0      0     0 ?        S    Jan24   0:00 [irq/148-4020000.mmc cd]
root         197  0.0  0.0      0     0 ?        I<   Jan24   0:00 [mmc_complete]
root         233  0.0  0.0      0     0 ?        I<   Jan24   0:00 [kworker/1:1H-kblockd]
root         238  0.0  0.0      0     0 ?        S    Jan24   0:00 [irq/251-panfrost-mmu]
root         239  0.0  0.0      0     0 ?        S    Jan24   0:00 [irq/252-panfrost-job]
root         240  0.0  0.0      0     0 ?        S    Jan24   0:00 [pan_js]
root         241  0.0  0.0      0     0 ?        S    Jan24   0:00 [pan_js]
root         242  0.0  0.0      0     0 ?        S    Jan24   0:00 [pan_js]
root         243  0.0  0.0      0     0 ?        S    Jan24   0:00 [cec-dw_hdmi]
root         244  0.0  0.0      0     0 ?        S    Jan24   0:00 [irq/240-dw-hdmi-cec]
root         245  0.0  0.0      0     0 ?        I<   Jan24   0:00 [stmmac_wq]
root         289  0.0  0.0      0     0 ?        S    Jan24   0:01 [jbd2/mmcblk2p1-8]
root         290  0.0  0.0      0     0 ?        I<   Jan24   0:00 [ext4-rsv-conver]
root         304  0.0  0.0      0     0 ?        I<   Jan24   0:00 [kworker/3:2H-kblockd]
root         366  0.0  0.0      0     0 ?        S    Jan24   0:03 [sdiohal_tx_thread]
root         368  0.0  0.0      0     0 ?        S    Jan24   0:02 [sdiohal_rx_thread]
root         369  0.0  0.0      0     0 ?        I<   Jan24   0:00 [WCN_LOOPCHECK_Q]
root         370  0.0  0.0      0     0 ?        I<   Jan24   0:00 [SPRDBT_RX_QUEUE]
root         373  0.0  0.0      0     0 ?        I<   Jan24   0:00 [cfg80211]
root         381  0.0  0.2  23156  5736 ?        Ss   Jan24   0:00 /lib/systemd/systemd-udevd
root         478  0.0  0.0      0     0 ?        S    Jan24   0:00 [ksdioirqd/mmc1]
root         479  0.0  0.0      0     0 ?        I<   Jan24   0:00 [sprdwl_work]
root         480  0.0  0.0      0     0 ?        I<   Jan24   0:00 [SPRDWL_RX_QUEUE]
root         481  0.0  0.0      0     0 ?        I<   Jan24   0:00 [SPRDWL_RX_NET_Q]
root         482  0.0  0.0      0     0 ?        D<   Jan24   0:00 [SPRDWL_TX_QUEUE]
root         574  0.0  0.0      0     0 ?        S    Jan24   0:00 [rc0]
root         610  0.0  0.0      0     0 ?        I<   Jan24   0:00 [ext4-rsv-conver]
root         774  0.0  0.4  28588  8956 ?        S<s  Jan24   0:01 /lib/systemd/systemd-journald
root         779  0.0  0.2   8064  4644 ?        Ss   Jan24   0:02 /usr/sbin/haveged --Foreground --verbose=1
_rpc         780  0.0  0.1   7920  2220 ?        Ss   Jan24   0:00 /sbin/rpcbind -f -w
systemd+     781  0.0  0.5  25216 11692 ?        Ss   Jan24   0:00 /lib/systemd/systemd-resolved
root         782  0.0  0.0      0     0 ?        I<   Jan24   0:00 [rpciod]
root         783  0.0  0.0      0     0 ?        I<   Jan24   0:00 [xprtiod]
root         791  0.0  0.0   7012  2032 ?        S    Jan24   0:00 /bin/bash /usr/lib/armbian/armbian-hardware-monitor start
root         798  0.0  0.0   7012  1592 ?        S    Jan24   0:00 /bin/bash /usr/lib/armbian/armbian-hardware-monitor start
root         799  0.0  0.0   5424   900 ?        D    Jan24   0:00 lscpu
root         800  0.0  0.0   5996   908 ?        S    Jan24   0:00 awk /Architecture/ {print $2}
root         810  0.0  0.1   7276  2312 ?        D    Jan24   0:00 /bin/bash /usr/lib/armbian/armbian-hardware-optimization start
root         831  0.0  0.1   6520  2580 ?        Ss   Jan24   0:00 /usr/sbin/cron -f -P
message+     833  0.0  0.2   8072  4092 ?        Ss   Jan24   0:00 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-o
root         836  0.0  0.8 259720 16764 ?        Ssl  Jan24   0:02 /usr/sbin/NetworkManager --no-daemon
root         850  0.0  0.8  32592 17788 ?        Ss   Jan24   0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
syslog       851  0.0  0.1 221844  3604 ?        Ssl  Jan24   0:00 /usr/sbin/rsyslogd -n -iNONE
root         864  0.0  1.3 1466024 27912 ?       Ssl  Jan24   0:14 /usr/lib/snapd/snapd
root         875  0.0  0.3  23612  6880 ?        Ss   Jan24   0:00 /lib/systemd/systemd-logind
root         929  0.0  1.0 398264 20788 ?        Ssl  Jan24   0:37 /usr/bin/python3 /usr/bin/fail2ban-server -xf start
root         939  0.0  1.4 193528 29784 ?        Ss   Jan24   0:04 /usr/bin/python3 /usr/bin/glances -s -B 127.0.0.1
vnstat       980  0.0  0.1   4468  3040 ?        Ss   Jan24   0:07 /usr/sbin/vnstatd -n
_chrony     1026  0.0  0.1  18604  2884 ?        S    Jan24   0:01 /usr/sbin/chronyd -F 1
root        1027  0.0  0.4  15164  8300 ?        Ss   Jan24   0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
_chrony     1029  0.0  0.1  10404  3140 ?        S    Jan24   0:00 /usr/sbin/chronyd -F 1
root        1037  0.0  0.0   5216   796 ttyS0    Ss+  Jan24   0:00 /sbin/agetty -o -p -- \u --keep-baud 115200,57600,38400,9600 ttyS0 linux
root        1044  0.0  0.0   6028  2004 ?        D    Jan24   0:00 grep -q \<ondemand\> /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
root        1046  0.0  0.0   5216   804 tty1     Ss+  Jan24   0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
root        1047  0.0  0.9 109684 20236 ?        Ssl  Jan24   0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
root        1090  0.0  0.0      0     0 ?        I<   Jan24   0:00 [kworker/u9:2-SPRDWL_RX_QUEUE]
www-data    1096  0.0  0.4  10520  8184 ?        Ss   Jan24   0:02 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
www-data    1130  0.0  1.0 208436 21904 ?        Ss   Jan24   0:00 /usr/bin/php-cgi
www-data    1148  0.0  0.2 208436  4728 ?        S    Jan24   0:00 /usr/bin/php-cgi
www-data    1149  0.0  0.2 208436  4728 ?        S    Jan24   0:00 /usr/bin/php-cgi
www-data    1150  0.0  0.2 208436  4728 ?        S    Jan24   0:00 /usr/bin/php-cgi
www-data    1151  0.0  0.2 208436  4728 ?        S    Jan24   0:00 /usr/bin/php-cgi
root        1393  0.0  0.4  17532  9124 ?        Ss   Jan24   0:00 sshd: username [priv]
username    1396  0.0  0.4  17128  8800 ?        Ss   Jan24   0:00 /lib/systemd/systemd --user
username    1397  0.0  0.2 105648  5168 ?        S    Jan24   0:00 (sd-pam)
username    1542  0.0  0.2  17792  6092 ?        S    Jan24   0:03 sshd: username@pts/0
username    1543  0.0  0.2  10224  5936 pts/0    Ss   Jan24   0:00 -zsh
root        7744  0.0  0.0      0     0 ?        I<   Jan24   0:00 [kworker/0:2H-mmc_complete]
root        8129  0.0  0.0      0     0 ?        I<   Jan24   0:00 [kworker/2:2H]
root       11392  0.0  0.0      0     0 ?        I<   Jan24   0:00 [kworker/0:1H]
root       13112  0.0  0.0      0     0 ?        I    Jan24   0:00 [kworker/0:1-events]
root       13723  0.0  0.0      0     0 ?        I    00:00   0:00 [kworker/2:0-events]
root       13848  0.0  0.0      0     0 ?        I    00:00   0:00 [kworker/1:0-cgroup_destroy]
root       13996  0.0  0.0      0     0 ?        I<   00:09   0:00 [kworker/3:0H]
root       13997  0.0  0.0      0     0 ?        I<   00:09   0:00 [kworker/1:2H]
root       14110  0.0  0.0      0     0 ?        I    00:19   0:00 [kworker/u8:4-flush-179:0]
root       14355  0.0  0.0      0     0 ?        I    00:39   0:00 [kworker/u8:0-flush-179:0]
root       14889  0.0  0.0      0     0 ?        I    01:26   0:00 [kworker/3:1-events]
root       14934  0.0  0.0      0     0 ?        I    01:30   0:00 [kworker/u8:2-events_unbound]
root       14999  0.0  0.0      0     0 ?        I    01:37   0:00 [kworker/3:0-events]
root       15012  0.0  0.0      0     0 ?        I    01:38   0:00 [kworker/u8:1-events_unbound]
root       15063  0.0  0.0      0     0 ?        I    01:39   0:00 [kworker/1:1]
root       15094  0.0  0.0      0     0 ?        I    01:42   0:00 [kworker/3:2-mm_percpu_wq]
username   15278  0.0  0.1   9576  3028 pts/1    R+   01:43   0:00 ps aux --sort=-pcpu

 

[krrmbn]

The boards were flashed with Armbian Ubuntu minimal CLI images originally (in 2023), with weekly updates:

# apt update && apt upgrade

 

They are running headless from eMMC, and connected via 1000BaseT. I don't use bluetooth or wifi, so I have disabled those services:

# systemctl disable --now wpa_supplicant.service && systemctl mask wpa_supplicant.service

# systemctl disable --now aw859a-bluetooth.service

[krrmbn]

% cat /etc/modules

sprdbt_tty

sprdwl_ng

 

Should I remove both entries, or just "sprdwl_ng"?

 

Thank you for your assistance.

[krrmbn]

Holy cow, removing sprdwl_ng and forcing a reboot has my loads at 0.10 vs "normal" 1.0 on my other OPi boards. Going to remove that line on my other boards right now.

Edited January 25 by krrmbn

[krrmbn]

Wow, load is so much lower on all of the boards now. Previously under normal circumstances, load: 1.0. Pre-crash circumstances, load: 4.0 or 5.0. Now after removing sprdwl_ng from /etc/modules, load: 0.01. Thank you so much!

[ag123]

that's your wifi

 

on a longer term is to review the codes to see what can be done about it, the driver that is.

[Gunjan Gupta]

38 minutes ago, ag123 said:

on a longer term is to review the codes to see what can be done about it, the driver that is.

I have a plan to replace the driver with the newer driver but I won't do it for next release which is next month. The current one I believe was forward ported from 4.9 bsp kernel and is heavily patch to get it working with latest kernel versions.

[ag123]

@Gunjan Gupta wrote:

Quote

I have a plan to replace the driver with the newer driver but I won't do it for next release which is next month. The current one I believe was forward ported from 4.9 bsp kernel and is heavily patch to get it working with latest kernel versions.

 

hi no worries, 

I'm just raising awareness so that more people could perhaps review the codes to see what may be improved. For what is worth, the wifi driver works well, if that is uwe5622 that is.

 

 

[krrmbn]

On 1/24/2024 at 9:20 AM, SteeMan said:

Moved to Community Maintained Allwinner.

 

@krrmbn Note that this is not an Armbian supported board.  It is community maintained.

Thank you for moving the thread to the correct subforum. I will slowly be moving to Orange Pi 4 LTS boards which are supported currently. Already have one running.

[krrmbn]

On 1/24/2024 at 2:54 PM, Gunjan Gupta said:

@krrmbnWere these boards using a fresh image or were they upgraded from some previous version? if they were upgraded, check if /etc/modules list a module like sprdwl_ng. If yes remove the same. Then enable aw859a-wifi.service service for wifi support. Now reboot and your problem should be resolved.

Thanks for your great work contributing and for helping me sort this issue.