avoid /var/log full from mapping of memory on boot

[Eric Smith]

Hi All

We are having a problem where on booting the /var/log is filled and uses all mmc storage space.

Does anyone know how to intervene in te booting process to stop this mapping and filling up of the fs?

This issue sometimes causes the device to fail to boot properly and remain inaccessible.

(It may indeed be that an unclean shutdown is causing this problem).

Booting up properly is essential for us and we do not want or need these logs .
We send the logs over the server, so capture these anyways.

This posting refers to our same issue: 

Any help much appreciated.
Thank you for Armbian!

Best wishes
Eric

[Dennboy]

This /var/log issue is indeed very annoying. I have some work-arounds in the mentioned ticket.

In my opinion we need:

• to find and solve the root cause of the corrupted journal files, to reduce the likelihood that this happens
• make sure corrupted journal files are not put back in /var/log during boot,
  • i.e. exclude journal/*/system@* and journal/*/user-*@* and journal/*/*~ in /usr/lib/armbian/armbian-ramlog : syncFromDisk
  • see my patch for armbian-ramlog in the mentioned ticket above

Btw. the main problem is the memory mapped /var/log that fills up (which approx 50MB), the /var/log.hdd is filling up on emmc or sdcard, but this is small compared to the volume size

Edited June 29, 2022 by Dennboy

[Dennboy]

I've done some reading and experimenting with systemd journald, and there appear to be at least 2 ways to easily turn off persistent logging to help with your SBC issues :

1. keep default Storage=auto in /etc/systemd/journald.conf, and remove journal directories
   • sudo systemctl stop systemd-journald.service
   • sudo rm -rf /var/log/journal /var.hdd/log.journal
   • sudo systemctl start systemd-journald.service
2. change to Storage=volatile in /etc/systemd/journald.conf, and optionally remove journal directories to save some space
   • sudo su -c "echo Storage=volatile > /etc/systemd/journal"
   • sudo systemctl restart systemd-journald.service
   • sudo rm -rf /var/log/journal /var.hdd/log.journal

In both cases the journald logs will still be available at runtime in /run/log/journal, but not written to the sdcard/emmc/disk. I think this would be a safe default for most users. You can always turn on persistent logging later when you for some reason want to diagnose boot problems.

 

For re-enabling persistent journald logging, the following steps can be taken:

1. with the default Storage=auto in /etc/systemd/journald.conf:
   • mkdir -p /var/log/journal /var.hdd/log.journal
2. with Storage=volatile in /etc/systemd/journald.conf:
   • mkdir -p /var/log/journal /var.hdd/log.journal
   • sudo sed -i "/^Storage=/d" /etc/systemd/journal"
   • sudo systemctl restart systemd-journald.service

[Dennboy]

small correction, of course /var.hdd/log.journal should be changed to /var/log.hdd/journal

[Dennboy]

For the record, another correction. The statement for Storage=volatile journald works after changing to

 

2. change to Storage=volatile in /etc/systemd/journald.conf, and optionally remove journal directories to save some space

• sudo sh -c "echo Storage=volatile » /etc/systemd/journald.conf"
• sudo systemctl stop systemd-journald.service
• sudo rm -rf /var/log/journal /var/log.hdd/journal
• sudo systemctl start systemd-journald.service