Jump to content

avoid /var/log full from mapping of memory on boot


Eric Smith
 Share

Recommended Posts

Hi All

We are having a problem where on booting the /var/log is filled and uses all mmc storage space.

Does anyone know how to intervene in te booting process to stop this mapping and filling up of the fs?


This issue sometimes causes the device to fail to boot properly and remain inaccessible.

(It may indeed be that an unclean shutdown is causing this problem).

Booting up properly is essential for us and we do not want or need these logs .
We send the logs over the server, so capture these anyways.

This posting refers to our same issue: 


Any help much appreciated.
Thank you for Armbian!

Best wishes
Eric

Link to comment
Share on other sites

Open source development is fun. Join Armbian Linux development team today!

This /var/log issue is indeed very annoying. I have some work-arounds in the mentioned ticket.

In my opinion we need:

  • to find and solve the root cause of the corrupted journal files, to reduce the likelihood that this happens
  • make sure corrupted journal files are not put back in /var/log during boot,
    • i.e. exclude journal/*/system@* and journal/*/user-*@* and journal/*/*~ in /usr/lib/armbian/armbian-ramlog : syncFromDisk
    • see my patch for armbian-ramlog in the mentioned ticket above

Btw. the main problem is the memory mapped /var/log that fills up (which approx 50MB), the /var/log.hdd is filling up on emmc or sdcard, but this is small compared to the volume size

Edited by Dennboy
Link to comment
Share on other sites

I've done some reading and experimenting with systemd journald, and there appear to be at least 2 ways to easily turn off persistent logging to help with your SBC issues :

  1. keep default Storage=auto in /etc/systemd/journald.conf, and remove journal directories
    • sudo systemctl stop systemd-journald.service
    • sudo rm -rf /var/log/journal /var.hdd/log.journal
    • sudo systemctl start systemd-journald.service
  2. change to Storage=volatile in /etc/systemd/journald.conf, and optionally remove journal directories to save some space
    • sudo su -c "echo Storage=volatile > /etc/systemd/journal"
    • sudo systemctl restart systemd-journald.service
    • sudo rm -rf /var/log/journal /var.hdd/log.journal

In both cases the journald logs will still be available at runtime in /run/log/journal, but not written to the sdcard/emmc/disk. I think this would be a safe default for most users. You can always turn on persistent logging later when you for some reason want to diagnose boot problems.

 

For re-enabling persistent journald logging, the following steps can be taken:

  1. with the default Storage=auto in /etc/systemd/journald.conf:
    • mkdir -p /var/log/journal /var.hdd/log.journal
  2. with Storage=volatile in /etc/systemd/journald.conf:
    • mkdir -p /var/log/journal /var.hdd/log.journal
    • sudo sed -i "/^Storage=/d" /etc/systemd/journal"
    • sudo systemctl restart systemd-journald.service
Link to comment
Share on other sites

For the record, another correction. The statement for Storage=volatile journald works after changing to

 

2. change to Storage=volatile in /etc/systemd/journald.conf, and optionally remove journal directories to save some space

  • sudo sh -c "echo Storage=volatile » /etc/systemd/journald.conf"
  • sudo systemctl stop systemd-journald.service
  • sudo rm -rf /var/log/journal /var/log.hdd/journal
  • sudo systemctl start systemd-journald.service
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...