Odroid HC1: slow on disks after longer uptime

[jogger]

Always running latest Armbian Buster: Since some months multiple machines with totally different application sets get slow writing to ANY mounted volume after one or more weeks of uptime (the busier the earlier). According to top IOWAIT exceeds user/kernel cpu usage. All applications and networking incl. disk reads run without any problem, until they write to the disk which makes them crawl. Even cp on a small file hangs. Any hint?

[Werner]

Providing logs with 

armbianmonitor -u

 helps with troubleshooting and significantly raises chances that issue gets addressed.

[jogger]

http://ix.io/3plO

 

generating this while copying files over Ethernet (Netatalk) to internal SATA with displayed write speed of only 500 kBps.

[Nitrax]

Ok I saw the debug output now. It seems you have some processes dying because they go OOM., possibly from memory leaks, most likely causing issues with the system in general.

[Igor]

6 hours ago, Nitrax said:

possibly from memory leaks, most likely causing issues with the system in general

Possible cause is ZRAM / ZSWAP implementation. Try disabling or / and move to kernel 5.12.y.

[jogger]

Disabled all ZRAM / ZSWAP / logging. Rebootet and error reoccured after 2 weeks during heavy disk activity from JDownloader and ktorrent.

 

Found I could get back to normal disk speeds by quitting Firefox (running the Webmin GUI). So maybe an issue with the overlay filesystem.

[jogger]

Upgraded to kernel 5.4 which made problems worse. Issue can always be reproduced by moving large amounts of data to SD-Card, which I use for archival only. Several processes get stuck:

 

root       239  0.0  0.0      0     0 ?        D<   Aug09   0:07 [kworker/1:1H+kblockd]
root       526  0.0  0.0      0     0 ?        D    Aug09   0:00 [jbd2/mmcblk1p1-]
root      6628  0.0  0.0      0     0 ?        D    15:09   0:05 [kworker/u16:2+flush-179:0]
root     15287  0.0  0.1  32100  2612 ?        Ds   16:09   0:00 (ionclean)

 

[xwiggen]

Possibly SD card failures, check dmesg when writing.

 

[jogger]

Am 16.8.2021 um 11:05 schrieb xwiggen:

Possibly SD card failures, check dmesg when writing.

 

This is not related to SD-Card, but easiest to reproduce. Even a kernel upgrade might hang, when writing files to the card. Another example using USB HDD storage. During a local file copy, a client enumerated files using netatalk. First got slow, then hung the following processes:

 

root      2214  0.0  0.0      0     0 ?        D    Aug16   0:01 [jbd2/sdb2-8]
e        23991  1.6  0.3  71168  6900 ?        D    10:09   0:47 /usr/sbin/afpd -d -F /e
root     26980  0.6  0.0      0     0 ?        D    10:28   0:10 [kworker/u16:0+flush-8:

 

[jogger]

5.4 proved unusable as it also frequently dropped connections on an external SMC 75xx USB GB Ethernet adapter. Finally upgraded to edge kernel. Problem is gone.