Luna the Deer
-
Posts
5 -
Joined
-
Last visited
Content Type
Forums
Store
Crowdfunding
Applications
Events
Raffles
Community Map
Posts posted by Luna the Deer
-
-
After having this happen again and investigating further, it looks like my container for transmission is writing to a log file that is just ballooning bigger and bigger until the system crashes.
root@modron:/var/lib/docker/containers/899f42253bb7772351219b7a65ff21347455f11950f983fb7a8883c6b93cdb17# ls -al total 101656680 drwx--x--- 4 root root 4096 Dec 27 22:28 . drwx--x--- 22 root root 4096 Dec 27 22:28 .. -rw-r----- 1 root root 104095768576 Dec 31 15:08 899f42253bb7772351219b7a65ff21347455f11950f983fb7a8883c6b93cdb17-json.log drwx------ 2 root root 4096 Dec 27 22:28 checkpoints -rw------- 1 root root 5949 Dec 27 22:28 config.v2.json -rw------- 1 root root 1776 Dec 27 22:28 hostconfig.json -rw-r--r-- 1 root root 13 Dec 27 22:28 hostname -rw-r--r-- 1 root root 172 Dec 27 22:28 hosts drwx--x--- 2 root root 4096 Dec 27 22:28 mounts -rw-r--r-- 1 root root 336 Dec 27 22:28 resolv.conf -rw-r--r-- 1 root root 71 Dec 27 22:28 resolv.conf.hash
I'm not sure what this file is. Tailing it prints this out, and I'm worried trying to load it into memory with less would cause a crash:
{"log":"[warn] Error from accept() call: No file descriptors available\n","stream":"stderr","time":"2025-12-31T20:06:58.854663511Z"} {"log":"[warn] Error from accept() call: No file descriptors available\n","stream":"stderr","time":"2025-12-31T20:06:58.854793008Z"} {"log":"[warn] Error from accept() call: No file descriptors available\n","stream":"stderr","time":"2025-12-31T20:06:58.855042376Z"} {"log":"[warn] Error from accept() call: No file descriptors available\n","stream":"stderr","time":"2025-12-31T20:06:58.855144456Z"} {"log":"[warn] Error from accept() call: No file descriptors available\n","stream":"stderr","time":"2025-12-31T20:06:58.855165456Z"} {"log":"[warn] Error from accept() call: No file descriptors available\n","stream":"stderr","time":"2025-12-31T20:06:58.855213288Z"} {"log":"[warn] Error from accept() call: No file descriptors available\n","stream":"stderr","time":"2025-12-31T20:06:58.855254412Z"} {"log":"[warn] Error from accept() call: No file descriptors available\n","stream":"stderr","time":"2025-12-31T20:06:58.855304285Z"} {"log":"[warn] Error from accept() call: No file descriptors available\n","stream":"stderr","time":"2025-12-31T20:06:58.855320618Z"} {"log":"[warn] Error from accept() call: No file descriptors available\n","stream":"stderr","time":"2025-12-I've been running the 4.0.5 branch of the linuxserver.io container for transmission, I'm updating to their "latest" branch and seeing if that improves anything.
-
That is another issue I'm working on, see
As for this issue, once the disk space problem is resolved, it still happens.
luna@modron:~$ df Filesystem 1K-blocks Used Available Use% Mounted on udev 926940 0 926940 0% /dev tmpfs 201916 900 201016 1% /run /dev/mmcblk0p1 120241980 18447284 100534968 16% / tmpfs 1009572 7236 1002336 1% /dev/shm tmpfs 5120 0 5120 0% /run/lock tmpfs 1024 0 1024 0% /run/credentials/systemd-resolved.service tmpfs 1009572 8 1009564 1% /tmp armbian-ramlog 51200 51200 0 100% /var/log tmpfs 1024 0 1024 0% /run/credentials/systemd-journald.service /dev/sda1 7751271852 7295398216 65155992 100% /mnt/external tmpfs 1024 0 1024 0% /run/credentials/getty@tty1.service tmpfs 1024 0 1024 0% /run/credentials/serial-getty@ttyS2.service tmpfs 201912 4 201908 1% /run/user/1000 luna@modron:~$ sudo apt update Hit:1 http://deb.debian.org/debian trixie InRelease Get:2 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB] Get:3 http://deb.debian.org/debian trixie-backports InRelease [54.0 kB] Get:4 http://security.debian.org trixie-security InRelease [43.4 kB] Hit:6 https://download.docker.com/linux/debian trixie InRelease Hit:7 https://github.armbian.com/configng stable InRelease Get:5 http://armbian.lv.auroradev.org/apt trixie InRelease [62.3 kB] Get:8 http://deb.debian.org/debian trixie-backports/main arm64 Packages.diff/Index [63.3 kB] Get:9 http://deb.debian.org/debian trixie-backports/main all Contents (deb).diff/Index [63.6 kB] Get:10 http://deb.debian.org/debian trixie-backports/main arm64 Contents (deb).diff/Index [63.5 kB] Err:5 http://armbian.lv.auroradev.org/apt trixie InRelease Sub-process /usr/bin/sqv returned an error code (1), error message is: Missing key DF00FAF1C577104B50BF1D0093D6889F9F0E78D5, which is needed to verify signature. Missing key 8CFA83D13EB2181EEF5843E41EB30FAF236099FE, which is needed to verify signature. Get:11 http://deb.debian.org/debian trixie-backports/main arm64 Packages T-2025-12-31-1402.47-F-2025-12-28-0205.37.pdiff [17.6 kB] Get:11 http://deb.debian.org/debian trixie-backports/main arm64 Packages T-2025-12-31-1402.47-F-2025-12-28-0205.37.pdiff [17.6 kB] Get:12 http://deb.debian.org/debian trixie-backports/main all Contents (deb) T-2025-12-31-1402.47-F-2025-12-28-0205.37.pdiff [245 kB] Get:12 http://deb.debian.org/debian trixie-backports/main all Contents (deb) T-2025-12-31-1402.47-F-2025-12-28-0205.37.pdiff [245 kB] Get:13 http://deb.debian.org/debian trixie-backports/main arm64 Contents (deb) T-2025-12-31-0829.39-F-2025-12-29-0207.06.pdiff [399 kB] Get:13 http://deb.debian.org/debian trixie-backports/main arm64 Contents (deb) T-2025-12-31-0829.39-F-2025-12-29-0207.06.pdiff [399 kB] Fetched 1,059 kB in 10s (104 kB/s) 10 packages can be upgraded. Run 'apt list --upgradable' to see them. Warning: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://armbian.lv.auroradev.org/apt trixie InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Missing key DF00FAF1C577104B50BF1D0093D6889F9F0E78D5, which is needed to verify signature. Missing key 8CFA83D13EB2181EEF5843E41EB30FAF236099FE, which is needed to verify signature. Warning: Failed to fetch http://apt.armbian.com/dists/trixie/InRelease Sub-process /usr/bin/sqv returned an error code (1), error message is: Missing key DF00FAF1C577104B50BF1D0093D6889F9F0E78D5, which is needed to verify signature. Missing key 8CFA83D13EB2181EEF5843E41EB30FAF236099FE, which is needed to verify signature. Warning: Some index files failed to download. They have been ignored, or old ones used instead.
-
This started happening fairly recently. If my Docker containers are left running for a period of time, this message will start appearing over and over in journalctl:
dockerd[1491]: time="2025-12-27T20:55:44.808894213-05:00" level=error msg="Error writing log message" driver=json-file error="error writing log entry: write /var/lib/docker/containers/97e823567c56600b69888e6abf63189c6f586afbf6bcf06fca820eaf4ef46f7d/97e823567c56600b69888e6abf63189c6f586afbf6bcf06fca820eaf4ef46f7d-json.log: no space left on device" message=
After a while, these two messages will start repeating (these were taken from a different instance of this happening, so the PID is different):
rsyslogd[599]: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2504.0 try https://www.rsyslog.com/e/2027 ] rsyslogd[599]: file '/var/log/syslog'[7] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2504.0 try https://www.rsyslog.com/e/2027 ]
Looking at btop when this is happening shows that the rsyslog and dockerd processes are using up most of the CPU, along with fail2ban for some reason. The full command for dockerd is:
/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Here are all my docker packages:
docker-buildx-plugin/trixie,now 0.30.1-1~debian.13~trixie arm64 [installed] docker-ce-cli/trixie,now 5:29.1.3-1~debian.13~trixie arm64 [installed] docker-ce/trixie,now 5:29.1.3-1~debian.13~trixie arm64 [installed] docker-compose-plugin/trixie,now 5.0.0-1~debian.13~trixie arm64 [installed]
-
Armbianmonitor:
Fairly recently, I've noticed these warnings appearing when running apt update. I have tried importing the keys from the repositories in question, but that has not remediated the issue.
root@modron:/home/luna# apt update Hit:1 http://deb.debian.org/debian trixie InRelease Hit:2 http://deb.debian.org/debian trixie-updates InRelease Hit:3 http://deb.debian.org/debian trixie-backports InRelease Hit:4 http://security.debian.org trixie-security InRelease Hit:6 https://download.docker.com/linux/debian trixie InRelease Hit:7 https://github.armbian.com/configng stable InRelease Get:5 http://armbian.lv.auroradev.org/apt trixie InRelease [62.3 kB] Err:5 http://armbian.lv.auroradev.org/apt trixie InRelease Sub-process /usr/bin/sqv returned an error code (1), error message is: Missing key DF00FAF1C577104B50BF1D0093D6889F9F0E78D5, which is needed to verify signature. Missing key 8CFA83D13EB2181EEF5843E41EB30FAF236099FE, which is needed to verify signature. Fetched 62.3 kB in 1s (67.6 kB/s) 10 packages can be upgraded. Run 'apt list --upgradable' to see them. Warning: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. OpenPGP signature verification failed: http://armbian.lv.auroradev.org/apt trixie InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Missing key DF00FAF1C577104B50BF1D0093D6889F9F0E78D5, which is needed to verify signature. Missing key 8CFA83D13EB2181EEF5843E41EB30FAF236099FE, which is needed to verify signature. Warning: Failed to fetch http://apt.armbian.com/dists/trixie/InRelease Sub-process /usr/bin/sqv returned an error code (1), error message is: Missing key DF00FAF1C577104B50BF1D0093D6889F9F0E78D5, which is needed to verify signature. Missing key 8CFA83D13EB2181EEF5843E41EB30FAF236099FE, which is needed to verify signature. Warning: Some index files failed to download. They have been ignored, or old ones used instead.
Missing keys when running apt-update
in Libre Renegade
Posted
I found a post on this forum and tried it, but it didn't appear to have any effect. I might have just run it wrong. I believe they were also dealing with a different source having this problem.