krrmbn Posted December 31, 2023 Posted December 31, 2023 Armbianmonitor: https://paste.armbian.com/Error My Orange Pi 3 LTS with Armbian 23.11.1 Jammy with Linux 6.1.63-current-sunxi64 is showing high load, but low CPU. The SBC crashed overnight. When I run htop, the following lines show state D: root lscpu root /bin/bash /usr/lib/armbian/armbian-hardware-optimization start root grep -q \<ondemand\> /sys/devices/cpu/cpu0/cpufreq/scaling_available_governors load average: 4.00, 4.00, 4.00 System load: 100% CPU usage: less than 1% sar -u 5 Linux 6.1.63-current-sunxi64 (hostname) 12/31/2023 _aarch64_ (4 CPU) 03:41:54 PM CPU %user %nice %system %iowait %steal %idle 03:41:59 PM all 0.50 0.00 0.35 0.00 0.00 99.15 I'm up to date with software (sudo apt update && sudo apt upgrade -y). I am unable to kill -9 the offending processes. When I try to reboot, I get locked out of ssh, but SBC does not reboot, and keeps replying to pings. I have to end up pulling the plug to get it to reboot. After plugging it back in, it starts out without the problem, but the problem resurfaces shortly thereafter. My other three OPI3LTS with Armbian 23.11.1 Jammy with Linux 6.1.63-current-sunxi64 are not having this issue. Thank you for your assistance, and Happy New Year! 0 Quote
krrmbn Posted January 1 Author Posted January 1 Ok, seems to have sorted itself out, and has not reoccured since the last time I had to force a reboot - about 4 hours ago. 0 Quote
krrmbn Posted January 2 Author Posted January 2 Ok, not sorted. Crashed again a couple of times. It seems as though all log files (syslog for example) are erased and started again at reboot, so I am unable to see what's happening to cause the crashes. I'm running minimal CLI headless, just running pi-hole, ssh, and fail2ban. Plenty of RAM, plenty of eMMC space available. I'm not sure where to look to sort this out. 0 Quote
krrmbn Posted January 6 Author Posted January 6 This is now happening to a second Orange Pi 3 LTS running armbian with all of the updates. 0 Quote
krrmbn Posted January 24 Author Posted January 24 So I am having this issue on two Orange Pi 3 LTS running armbian jammy. They run at ~4 load, and then eventually crash. % sudo systemctl --failed UNIT LOAD ACTIVE SUB DESCRIPTION ● cpufrequtils.service loaded failed failed LSB: set CPUFreq kernel parameters LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 1 loaded units listed. 0 Quote
krrmbn Posted January 24 Author Posted January 24 this is redonkulous I'm watching ~5 load and can't do anything to stop the scb from crashing repeatedly. All it is running is pi-hole, so of course it renders my lan inoperable when it repeatedly fails. 0 Quote
SteeMan Posted January 24 Posted January 24 Moved to Community Maintained Allwinner. @krrmbn Note that this is not an Armbian supported board. It is community maintained. 1 Quote
Gunjan Gupta Posted January 24 Posted January 24 If you are able to login to the system. Please share the output of dmesg and "ps aux --sort=-pcpu" 0 Quote
Gunjan Gupta Posted January 24 Posted January 24 @krrmbnWere these boards using a fresh image or were they upgraded from some previous version? if they were upgraded, check if /etc/modules list a module like sprdwl_ng. If yes remove the same. Then enable aw859a-wifi.service service for wifi support. Now reboot and your problem should be resolved. 1 Quote
krrmbn Posted January 25 Author Posted January 25 hostname:~:% ps aux --sort=-pcpu USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND username 15243 18.6 0.2 10224 5912 pts/1 Ss 01:43 0:00 -zsh root 15101 6.0 0.4 17532 9152 ? Ss 01:43 0:00 sshd: username [priv] username 1670 2.1 3.0 363120 61204 pts/0 Rl+ Jan24 23:12 /usr/bin/python3 /usr/bin/glances -t 15 username 15242 0.3 0.2 17792 6104 ? S 01:43 0:00 sshd: username@pts/1 pihole 1020 0.2 0.7 365568 16260 ? S<sl Jan24 2:58 /usr/bin/pihole-FTL -f root 1 0.0 0.5 167584 10760 ? Ss Jan24 0:06 /sbin/init root 2 0.0 0.0 0 0 ? S Jan24 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? I< Jan24 0:00 [rcu_gp] root 4 0.0 0.0 0 0 ? I< Jan24 0:00 [rcu_par_gp] root 5 0.0 0.0 0 0 ? I< Jan24 0:00 [slub_flushwq] root 6 0.0 0.0 0 0 ? I< Jan24 0:00 [netns] root 7 0.0 0.0 0 0 ? I Jan24 0:02 [kworker/0:0-events] root 10 0.0 0.0 0 0 ? I< Jan24 0:00 [mm_percpu_wq] root 11 0.0 0.0 0 0 ? I Jan24 0:00 [rcu_tasks_trace_kthread] root 12 0.0 0.0 0 0 ? S Jan24 0:00 [ksoftirqd/0] root 13 0.0 0.0 0 0 ? I Jan24 0:07 [rcu_sched] root 14 0.0 0.0 0 0 ? S Jan24 0:00 [migration/0] root 15 0.0 0.0 0 0 ? S Jan24 0:00 [cpuhp/0] root 16 0.0 0.0 0 0 ? S Jan24 0:00 [cpuhp/1] root 17 0.0 0.0 0 0 ? S Jan24 0:00 [migration/1] root 18 0.0 0.0 0 0 ? S Jan24 0:00 [ksoftirqd/1] root 21 0.0 0.0 0 0 ? S Jan24 0:00 [cpuhp/2] root 22 0.0 0.0 0 0 ? S Jan24 0:00 [migration/2] root 23 0.0 0.0 0 0 ? S Jan24 0:00 [ksoftirqd/2] root 26 0.0 0.0 0 0 ? S Jan24 0:00 [cpuhp/3] root 27 0.0 0.0 0 0 ? S Jan24 0:00 [migration/3] root 28 0.0 0.0 0 0 ? S Jan24 0:00 [ksoftirqd/3] root 31 0.0 0.0 0 0 ? S Jan24 0:00 [kdevtmpfs] root 32 0.0 0.0 0 0 ? I< Jan24 0:00 [inet_frag_wq] root 35 0.0 0.0 0 0 ? S Jan24 0:00 [kauditd] root 36 0.0 0.0 0 0 ? S Jan24 0:00 [oom_reaper] root 37 0.0 0.0 0 0 ? I< Jan24 0:00 [writeback] root 38 0.0 0.0 0 0 ? S Jan24 0:02 [kcompactd0] root 39 0.0 0.0 0 0 ? SN Jan24 0:00 [ksmd] root 40 0.0 0.0 0 0 ? SN Jan24 0:00 [khugepaged] root 41 0.0 0.0 0 0 ? I< Jan24 0:00 [cryptd] root 42 0.0 0.0 0 0 ? I< Jan24 0:00 [kintegrityd] root 43 0.0 0.0 0 0 ? I< Jan24 0:00 [kblockd] root 44 0.0 0.0 0 0 ? I< Jan24 0:00 [blkcg_punt_bio] root 45 0.0 0.0 0 0 ? I< Jan24 0:00 [tpm_dev_wq] root 46 0.0 0.0 0 0 ? I< Jan24 0:00 [md] root 47 0.0 0.0 0 0 ? I< Jan24 0:00 [devfreq_wq] root 48 0.0 0.0 0 0 ? S Jan24 0:00 [watchdogd] root 51 0.0 0.0 0 0 ? I Jan24 0:11 [kworker/2:1-events] root 52 0.0 0.0 0 0 ? S Jan24 0:00 [kswapd0] root 53 0.0 0.0 0 0 ? S Jan24 0:00 [ecryptfs-kthread] root 61 0.0 0.0 0 0 ? I< Jan24 0:00 [kthrotld] root 68 0.0 0.0 0 0 ? I< Jan24 0:00 [raid5wq] root 69 0.0 0.0 0 0 ? I< Jan24 0:00 [bch_btree_io] root 70 0.0 0.0 0 0 ? I< Jan24 0:00 [bcache] root 71 0.0 0.0 0 0 ? I< Jan24 0:00 [bch_journal] root 72 0.0 0.0 0 0 ? S Jan24 0:00 [1904000.crypto-engine] root 73 0.0 0.0 0 0 ? S Jan24 0:00 [1904000.crypto-engine] root 74 0.0 0.0 0 0 ? S Jan24 0:00 [1904000.crypto-engine] root 75 0.0 0.0 0 0 ? S Jan24 0:00 [1904000.crypto-engine] root 76 0.0 0.0 0 0 ? S Jan24 0:00 [hwrng] root 77 0.0 0.0 0 0 ? I< Jan24 0:00 [tls-strp] root 78 0.0 0.0 0 0 ? I< Jan24 0:00 [mld] root 79 0.0 0.0 0 0 ? I< Jan24 0:00 [ipv6_addrconf] root 80 0.0 0.0 0 0 ? I< Jan24 0:00 [kworker/2:1H-kblockd] root 85 0.0 0.0 0 0 ? I< Jan24 0:00 [kstrp] root 87 0.0 0.0 0 0 ? I< Jan24 0:00 [zswap1] root 88 0.0 0.0 0 0 ? I< Jan24 0:00 [zswap1] root 89 0.0 0.0 0 0 ? I< Jan24 0:00 [zswap-shrink] root 90 0.0 0.0 0 0 ? I< Jan24 0:00 [kworker/u9:0-sprdwl_work] root 174 0.0 0.0 0 0 ? I Jan24 0:02 [kworker/1:3-events] root 175 0.0 0.0 0 0 ? S Jan24 0:00 [irq/92-axp806] root 176 0.0 0.0 0 0 ? S Jan24 0:05 [irq/105-ths] root 180 0.0 0.0 0 0 ? S Jan24 0:00 [irq/240-6000000.hdmi] root 181 0.0 0.0 0 0 ? S Jan24 0:00 [card0-crtc0] root 185 0.0 0.0 0 0 ? S Jan24 0:00 [irq/242-sunxi-mmc] root 187 0.0 0.0 0 0 ? S Jan24 0:00 [irq/243-sunxi-mmc] root 192 0.0 0.0 0 0 ? S Jan24 0:00 [irq/249-sunxi-mmc] root 194 0.0 0.0 0 0 ? S Jan24 0:00 [irq/148-4020000.mmc cd] root 197 0.0 0.0 0 0 ? I< Jan24 0:00 [mmc_complete] root 233 0.0 0.0 0 0 ? I< Jan24 0:00 [kworker/1:1H-kblockd] root 238 0.0 0.0 0 0 ? S Jan24 0:00 [irq/251-panfrost-mmu] root 239 0.0 0.0 0 0 ? S Jan24 0:00 [irq/252-panfrost-job] root 240 0.0 0.0 0 0 ? S Jan24 0:00 [pan_js] root 241 0.0 0.0 0 0 ? S Jan24 0:00 [pan_js] root 242 0.0 0.0 0 0 ? S Jan24 0:00 [pan_js] root 243 0.0 0.0 0 0 ? S Jan24 0:00 [cec-dw_hdmi] root 244 0.0 0.0 0 0 ? S Jan24 0:00 [irq/240-dw-hdmi-cec] root 245 0.0 0.0 0 0 ? I< Jan24 0:00 [stmmac_wq] root 289 0.0 0.0 0 0 ? S Jan24 0:01 [jbd2/mmcblk2p1-8] root 290 0.0 0.0 0 0 ? I< Jan24 0:00 [ext4-rsv-conver] root 304 0.0 0.0 0 0 ? I< Jan24 0:00 [kworker/3:2H-kblockd] root 366 0.0 0.0 0 0 ? S Jan24 0:03 [sdiohal_tx_thread] root 368 0.0 0.0 0 0 ? S Jan24 0:02 [sdiohal_rx_thread] root 369 0.0 0.0 0 0 ? I< Jan24 0:00 [WCN_LOOPCHECK_Q] root 370 0.0 0.0 0 0 ? I< Jan24 0:00 [SPRDBT_RX_QUEUE] root 373 0.0 0.0 0 0 ? I< Jan24 0:00 [cfg80211] root 381 0.0 0.2 23156 5736 ? Ss Jan24 0:00 /lib/systemd/systemd-udevd root 478 0.0 0.0 0 0 ? S Jan24 0:00 [ksdioirqd/mmc1] root 479 0.0 0.0 0 0 ? I< Jan24 0:00 [sprdwl_work] root 480 0.0 0.0 0 0 ? I< Jan24 0:00 [SPRDWL_RX_QUEUE] root 481 0.0 0.0 0 0 ? I< Jan24 0:00 [SPRDWL_RX_NET_Q] root 482 0.0 0.0 0 0 ? D< Jan24 0:00 [SPRDWL_TX_QUEUE] root 574 0.0 0.0 0 0 ? S Jan24 0:00 [rc0] root 610 0.0 0.0 0 0 ? I< Jan24 0:00 [ext4-rsv-conver] root 774 0.0 0.4 28588 8956 ? S<s Jan24 0:01 /lib/systemd/systemd-journald root 779 0.0 0.2 8064 4644 ? Ss Jan24 0:02 /usr/sbin/haveged --Foreground --verbose=1 _rpc 780 0.0 0.1 7920 2220 ? Ss Jan24 0:00 /sbin/rpcbind -f -w systemd+ 781 0.0 0.5 25216 11692 ? Ss Jan24 0:00 /lib/systemd/systemd-resolved root 782 0.0 0.0 0 0 ? I< Jan24 0:00 [rpciod] root 783 0.0 0.0 0 0 ? I< Jan24 0:00 [xprtiod] root 791 0.0 0.0 7012 2032 ? S Jan24 0:00 /bin/bash /usr/lib/armbian/armbian-hardware-monitor start root 798 0.0 0.0 7012 1592 ? S Jan24 0:00 /bin/bash /usr/lib/armbian/armbian-hardware-monitor start root 799 0.0 0.0 5424 900 ? D Jan24 0:00 lscpu root 800 0.0 0.0 5996 908 ? S Jan24 0:00 awk /Architecture/ {print $2} root 810 0.0 0.1 7276 2312 ? D Jan24 0:00 /bin/bash /usr/lib/armbian/armbian-hardware-optimization start root 831 0.0 0.1 6520 2580 ? Ss Jan24 0:00 /usr/sbin/cron -f -P message+ 833 0.0 0.2 8072 4092 ? Ss Jan24 0:00 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-o root 836 0.0 0.8 259720 16764 ? Ssl Jan24 0:02 /usr/sbin/NetworkManager --no-daemon root 850 0.0 0.8 32592 17788 ? Ss Jan24 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers syslog 851 0.0 0.1 221844 3604 ? Ssl Jan24 0:00 /usr/sbin/rsyslogd -n -iNONE root 864 0.0 1.3 1466024 27912 ? Ssl Jan24 0:14 /usr/lib/snapd/snapd root 875 0.0 0.3 23612 6880 ? Ss Jan24 0:00 /lib/systemd/systemd-logind root 929 0.0 1.0 398264 20788 ? Ssl Jan24 0:37 /usr/bin/python3 /usr/bin/fail2ban-server -xf start root 939 0.0 1.4 193528 29784 ? Ss Jan24 0:04 /usr/bin/python3 /usr/bin/glances -s -B 127.0.0.1 vnstat 980 0.0 0.1 4468 3040 ? Ss Jan24 0:07 /usr/sbin/vnstatd -n _chrony 1026 0.0 0.1 18604 2884 ? S Jan24 0:01 /usr/sbin/chronyd -F 1 root 1027 0.0 0.4 15164 8300 ? Ss Jan24 0:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups _chrony 1029 0.0 0.1 10404 3140 ? S Jan24 0:00 /usr/sbin/chronyd -F 1 root 1037 0.0 0.0 5216 796 ttyS0 Ss+ Jan24 0:00 /sbin/agetty -o -p -- \u --keep-baud 115200,57600,38400,9600 ttyS0 linux root 1044 0.0 0.0 6028 2004 ? D Jan24 0:00 grep -q \<ondemand\> /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors root 1046 0.0 0.0 5216 804 tty1 Ss+ Jan24 0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux root 1047 0.0 0.9 109684 20236 ? Ssl Jan24 0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal root 1090 0.0 0.0 0 0 ? I< Jan24 0:00 [kworker/u9:2-SPRDWL_RX_QUEUE] www-data 1096 0.0 0.4 10520 8184 ? Ss Jan24 0:02 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf www-data 1130 0.0 1.0 208436 21904 ? Ss Jan24 0:00 /usr/bin/php-cgi www-data 1148 0.0 0.2 208436 4728 ? S Jan24 0:00 /usr/bin/php-cgi www-data 1149 0.0 0.2 208436 4728 ? S Jan24 0:00 /usr/bin/php-cgi www-data 1150 0.0 0.2 208436 4728 ? S Jan24 0:00 /usr/bin/php-cgi www-data 1151 0.0 0.2 208436 4728 ? S Jan24 0:00 /usr/bin/php-cgi root 1393 0.0 0.4 17532 9124 ? Ss Jan24 0:00 sshd: username [priv] username 1396 0.0 0.4 17128 8800 ? Ss Jan24 0:00 /lib/systemd/systemd --user username 1397 0.0 0.2 105648 5168 ? S Jan24 0:00 (sd-pam) username 1542 0.0 0.2 17792 6092 ? S Jan24 0:03 sshd: username@pts/0 username 1543 0.0 0.2 10224 5936 pts/0 Ss Jan24 0:00 -zsh root 7744 0.0 0.0 0 0 ? I< Jan24 0:00 [kworker/0:2H-mmc_complete] root 8129 0.0 0.0 0 0 ? I< Jan24 0:00 [kworker/2:2H] root 11392 0.0 0.0 0 0 ? I< Jan24 0:00 [kworker/0:1H] root 13112 0.0 0.0 0 0 ? I Jan24 0:00 [kworker/0:1-events] root 13723 0.0 0.0 0 0 ? I 00:00 0:00 [kworker/2:0-events] root 13848 0.0 0.0 0 0 ? I 00:00 0:00 [kworker/1:0-cgroup_destroy] root 13996 0.0 0.0 0 0 ? I< 00:09 0:00 [kworker/3:0H] root 13997 0.0 0.0 0 0 ? I< 00:09 0:00 [kworker/1:2H] root 14110 0.0 0.0 0 0 ? I 00:19 0:00 [kworker/u8:4-flush-179:0] root 14355 0.0 0.0 0 0 ? I 00:39 0:00 [kworker/u8:0-flush-179:0] root 14889 0.0 0.0 0 0 ? I 01:26 0:00 [kworker/3:1-events] root 14934 0.0 0.0 0 0 ? I 01:30 0:00 [kworker/u8:2-events_unbound] root 14999 0.0 0.0 0 0 ? I 01:37 0:00 [kworker/3:0-events] root 15012 0.0 0.0 0 0 ? I 01:38 0:00 [kworker/u8:1-events_unbound] root 15063 0.0 0.0 0 0 ? I 01:39 0:00 [kworker/1:1] root 15094 0.0 0.0 0 0 ? I 01:42 0:00 [kworker/3:2-mm_percpu_wq] username 15278 0.0 0.1 9576 3028 pts/1 R+ 01:43 0:00 ps aux --sort=-pcpu 0 Quote
krrmbn Posted January 25 Author Posted January 25 The boards were flashed with Armbian Ubuntu minimal CLI images originally (in 2023), with weekly updates: # apt update && apt upgrade They are running headless from eMMC, and connected via 1000BaseT. I don't use bluetooth or wifi, so I have disabled those services: # systemctl disable --now wpa_supplicant.service && systemctl mask wpa_supplicant.service # systemctl disable --now aw859a-bluetooth.service 0 Quote
krrmbn Posted January 25 Author Posted January 25 % cat /etc/modules sprdbt_tty sprdwl_ng Should I remove both entries, or just "sprdwl_ng"? Thank you for your assistance. 0 Quote
krrmbn Posted January 25 Author Posted January 25 (edited) Holy cow, removing sprdwl_ng and forcing a reboot has my loads at 0.10 vs "normal" 1.0 on my other OPi boards. Going to remove that line on my other boards right now. Edited January 25 by krrmbn 0 Quote
krrmbn Posted January 25 Author Posted January 25 Wow, load is so much lower on all of the boards now. Previously under normal circumstances, load: 1.0. Pre-crash circumstances, load: 4.0 or 5.0. Now after removing sprdwl_ng from /etc/modules, load: 0.01. Thank you so much! 1 Quote
ag123 Posted January 25 Posted January 25 that's your wifi on a longer term is to review the codes to see what can be done about it, the driver that is. 0 Quote
Gunjan Gupta Posted January 25 Posted January 25 38 minutes ago, ag123 said: on a longer term is to review the codes to see what can be done about it, the driver that is. I have a plan to replace the driver with the newer driver but I won't do it for next release which is next month. The current one I believe was forward ported from 4.9 bsp kernel and is heavily patch to get it working with latest kernel versions. 0 Quote
ag123 Posted January 25 Posted January 25 @Gunjan Gupta wrote: Quote I have a plan to replace the driver with the newer driver but I won't do it for next release which is next month. The current one I believe was forward ported from 4.9 bsp kernel and is heavily patch to get it working with latest kernel versions. hi no worries, I'm just raising awareness so that more people could perhaps review the codes to see what may be improved. For what is worth, the wifi driver works well, if that is uwe5622 that is. 0 Quote
krrmbn Posted January 25 Author Posted January 25 On 1/24/2024 at 9:20 AM, SteeMan said: Moved to Community Maintained Allwinner. @krrmbn Note that this is not an Armbian supported board. It is community maintained. Thank you for moving the thread to the correct subforum. I will slowly be moving to Orange Pi 4 LTS boards which are supported currently. Already have one running. 0 Quote
krrmbn Posted January 25 Author Posted January 25 On 1/24/2024 at 2:54 PM, Gunjan Gupta said: @krrmbnWere these boards using a fresh image or were they upgraded from some previous version? if they were upgraded, check if /etc/modules list a module like sprdwl_ng. If yes remove the same. Then enable aw859a-wifi.service service for wifi support. Now reboot and your problem should be resolved. Thanks for your great work contributing and for helping me sort this issue. 0 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.