jogger Posted June 8, 2021 Posted June 8, 2021 Always running latest Armbian Buster: Since some months multiple machines with totally different application sets get slow writing to ANY mounted volume after one or more weeks of uptime (the busier the earlier). According to top IOWAIT exceeds user/kernel cpu usage. All applications and networking incl. disk reads run without any problem, until they write to the disk which makes them crawl. Even cp on a small file hangs. Any hint?
Werner Posted June 9, 2021 Posted June 9, 2021 Providing logs with armbianmonitor -u helps with troubleshooting and significantly raises chances that issue gets addressed.
jogger Posted June 9, 2021 Author Posted June 9, 2021 http://ix.io/3plO generating this while copying files over Ethernet (Netatalk) to internal SATA with displayed write speed of only 500 kBps.
Nitrax Posted June 17, 2021 Posted June 17, 2021 Ok I saw the debug output now. It seems you have some processes dying because they go OOM., possibly from memory leaks, most likely causing issues with the system in general.
Igor Posted June 18, 2021 Posted June 18, 2021 6 hours ago, Nitrax said: possibly from memory leaks, most likely causing issues with the system in general Possible cause is ZRAM / ZSWAP implementation. Try disabling or / and move to kernel 5.12.y.
jogger Posted July 3, 2021 Author Posted July 3, 2021 Disabled all ZRAM / ZSWAP / logging. Rebootet and error reoccured after 2 weeks during heavy disk activity from JDownloader and ktorrent. Found I could get back to normal disk speeds by quitting Firefox (running the Webmin GUI). So maybe an issue with the overlay filesystem.
jogger Posted August 10, 2021 Author Posted August 10, 2021 Upgraded to kernel 5.4 which made problems worse. Issue can always be reproduced by moving large amounts of data to SD-Card, which I use for archival only. Several processes get stuck: root 239 0.0 0.0 0 0 ? D< Aug09 0:07 [kworker/1:1H+kblockd] root 526 0.0 0.0 0 0 ? D Aug09 0:00 [jbd2/mmcblk1p1-] root 6628 0.0 0.0 0 0 ? D 15:09 0:05 [kworker/u16:2+flush-179:0] root 15287 0.0 0.1 32100 2612 ? Ds 16:09 0:00 (ionclean)
xwiggen Posted August 16, 2021 Posted August 16, 2021 Possibly SD card failures, check dmesg when writing.
jogger Posted August 18, 2021 Author Posted August 18, 2021 Am 16.8.2021 um 11:05 schrieb xwiggen: Possibly SD card failures, check dmesg when writing. This is not related to SD-Card, but easiest to reproduce. Even a kernel upgrade might hang, when writing files to the card. Another example using USB HDD storage. During a local file copy, a client enumerated files using netatalk. First got slow, then hung the following processes: root 2214 0.0 0.0 0 0 ? D Aug16 0:01 [jbd2/sdb2-8] e 23991 1.6 0.3 71168 6900 ? D 10:09 0:47 /usr/sbin/afpd -d -F /e root 26980 0.6 0.0 0 0 ? D 10:28 0:10 [kworker/u16:0+flush-8:
Solution jogger Posted September 12, 2021 Author Solution Posted September 12, 2021 5.4 proved unusable as it also frequently dropped connections on an external SMC 75xx USB GB Ethernet adapter. Finally upgraded to edge kernel. Problem is gone.
Recommended Posts