Eric Smith Posted June 28, 2022 Share Posted June 28, 2022 Hi All We are having a problem where on booting the /var/log is filled and uses all mmc storage space. Does anyone know how to intervene in te booting process to stop this mapping and filling up of the fs? This issue sometimes causes the device to fail to boot properly and remain inaccessible. (It may indeed be that an unclean shutdown is causing this problem). Booting up properly is essential for us and we do not want or need these logs . We send the logs over the server, so capture these anyways. This posting refers to our same issue: Any help much appreciated. Thank you for Armbian! Best wishes Eric 1 Quote Link to comment Share on other sites More sharing options...
Dennboy Posted June 29, 2022 Share Posted June 29, 2022 (edited) This /var/log issue is indeed very annoying. I have some work-arounds in the mentioned ticket. In my opinion we need: to find and solve the root cause of the corrupted journal files, to reduce the likelihood that this happens make sure corrupted journal files are not put back in /var/log during boot, i.e. exclude journal/*/system@* and journal/*/user-*@* and journal/*/*~ in /usr/lib/armbian/armbian-ramlog : syncFromDisk see my patch for armbian-ramlog in the mentioned ticket above Btw. the main problem is the memory mapped /var/log that fills up (which approx 50MB), the /var/log.hdd is filling up on emmc or sdcard, but this is small compared to the volume size Edited June 29, 2022 by Dennboy 0 Quote Link to comment Share on other sites More sharing options...
Dennboy Posted July 2, 2022 Share Posted July 2, 2022 I've done some reading and experimenting with systemd journald, and there appear to be at least 2 ways to easily turn off persistent logging to help with your SBC issues : keep default Storage=auto in /etc/systemd/journald.conf, and remove journal directories sudo systemctl stop systemd-journald.service sudo rm -rf /var/log/journal /var.hdd/log.journal sudo systemctl start systemd-journald.service change to Storage=volatile in /etc/systemd/journald.conf, and optionally remove journal directories to save some space sudo su -c "echo Storage=volatile > /etc/systemd/journal" sudo systemctl restart systemd-journald.service sudo rm -rf /var/log/journal /var.hdd/log.journal In both cases the journald logs will still be available at runtime in /run/log/journal, but not written to the sdcard/emmc/disk. I think this would be a safe default for most users. You can always turn on persistent logging later when you for some reason want to diagnose boot problems. For re-enabling persistent journald logging, the following steps can be taken: with the default Storage=auto in /etc/systemd/journald.conf: mkdir -p /var/log/journal /var.hdd/log.journal with Storage=volatile in /etc/systemd/journald.conf: mkdir -p /var/log/journal /var.hdd/log.journal sudo sed -i "/^Storage=/d" /etc/systemd/journal" sudo systemctl restart systemd-journald.service 0 Quote Link to comment Share on other sites More sharing options...
Dennboy Posted July 3, 2022 Share Posted July 3, 2022 small correction, of course /var.hdd/log.journal should be changed to /var/log.hdd/journal 0 Quote Link to comment Share on other sites More sharing options...
Dennboy Posted July 7, 2022 Share Posted July 7, 2022 For the record, another correction. The statement for Storage=volatile journald works after changing to 2. change to Storage=volatile in /etc/systemd/journald.conf, and optionally remove journal directories to save some space sudo sh -c "echo Storage=volatile » /etc/systemd/journald.conf" sudo systemctl stop systemd-journald.service sudo rm -rf /var/log/journal /var/log.hdd/journal sudo systemctl start systemd-journald.service 0 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.