Jump to content

/var easily get full with log2ram


DoubleHP

Recommended Posts

# echo plop | logger

should show things in

tail -F /var/log/syslog

 

After digging a bit:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             85M     0   85M   0% /dev
tmpfs            24M  3.0M   22M  13% /run
/dev/mmcblk0p1   15G  1.3G   13G   9% /
tmpfs           120M  4.0K  120M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           120M     0  120M   0% /sys/fs/cgroup
tmpfs           120M     0  120M   0% /tmp
log2ram          50M   50M     0 100% /var/log
tmpfs            24M     0   24M   0% /run/user/0

# du -lsh /var/log/* | grep M
24M     /var/log/kern.log.1
25M     /var/log/syslog.1

 

so ... WTF ?

 

# tail -n1 /var/log/kern.log.1
Feb 12 10:35:14 localhost kernel: [ 6837.372115] sunxi-mmcroot@opi-12-gite-salon:/var/log# tail -n1 /var/log/syslog.1

 

# date
Wed Feb 21 13:48:48 UTC 2018

 

# uptime
 13:48:50 up 33 min,  2 users,  load average: 0.00, 0.01, 0.00

 

Why is this crap filling my RAM with deprecated logs that are older than 1 week ?

 

Either the system was up, and logrotate should compress and push this to disk, or, the system had been restarted recetly, and log2ram has no reason to fill my memory with deprecated logs !!!

 

So, the point of log2ram is just to occupy RAM, and prevent admin from seing fresh messages ?

 

Not even able to force log rotation when the ramdisk is full ?

Link to comment
Share on other sites

On most your Orange Pi Zero 256M, I got this:

 

# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             85M     0   85M   0% /dev
tmpfs            25M  3.5M   21M  15% /run
/dev/mmcblk0p1   15G  1.3G   14G   9% /
tmpfs           120M  4.0K  120M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           120M     0  120M   0% /sys/fs/cgroup
tmpfs           120M   80K  120M   1% /tmp
log2ram          50M   50M     0 100% /var/log
tmpfs            25M     0   25M   0% /run/user/0

 

This happens very fast, after a few days, for Pis that are now plugged 24/7.

 

Then I have to manually remove some large file, and, when the Pi have run a cron.daily and cron.weekly, and it always stays on, then, this never happens again.

 

What is the best workaround, or fix ?

 

Obvisouly, I can not really increase the size of the ramdisk.

 

I don't want either to stop using log2ram; I have killed many SD cards in the past.

 

And the problem is always the same:

 

# ls -lha /var/log/ | grep M
total 50M
-rw-r-----  1 syslog adm     25M Feb 27 04:49 kern.log.1
-rw-r-----  1 syslog adm     25M Feb 27 06:25 syslog.1
drwxr-xr-x  2 root   root     40 May 12  2016 sysstat

 

This is the third time.


 

# head /var/log/syslog.1
Nov 25 06:20:40 localhost rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="579" x-info="http://www.rsyslog.com"] start
Nov 25 06:20:40 localhost rsyslogd-2222: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ]
Nov 25 06:20:40 localhost rsyslogd: rsyslogd's groupid changed to 108
Nov 25 06:20:40 localhost rsyslogd: rsyslogd's userid changed to 104
Nov 25 06:20:40 localhost kernel: [    0.000000] Booting Linux on physical CPU 0x0
Nov 25 06:20:40 localhost kernel: [    0.000000] random: get_random_bytes called from start_kernel+0x31/0x352 with crng_init=0
Nov 25 06:20:40 localhost kernel: [    0.000000] Linux version 4.13.16-sunxi (root@armbian) (gcc version 7.1.1 20170707 (Linaro GCC 7.1-2017.08)) #20 SMP Fri Nov 24 19:50:07 CET 2017
Nov 25 06:20:40 localhost kernel: [    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=50c5387d
Nov 25 06:20:40 localhost kernel: [    0.000000] CPU: div instructions available: patching division code
Nov 25 06:20:40 localhost kernel: [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache

# tail /var/log/syslog.1
Feb 12 17:44:52 localhost kernel: [23357.366931] rc rc0: IR event FIFO is full!
Feb 12 17:44:52 localhost kernel: [23357.366935] rc rc0: IR event FIFO is full!
Feb 12 17:44:52 localhost kernel: [23357.366940] rc rc0: IR event FIFO is full!
Feb 12 17:44:52 localhost kernel: [23357.366944] rc rc0: IR event FIFO is full!
Feb 12 17:44:52 localhost kernel: [23357.366949] rc rc0: IR event FIFO is full!
Feb 12 17:44:52 localhost kernel: [23357.366953] rc rc0: IR event FIFO is full!
Feb 12 17:44:52 localhost kernel: [23357.366957] rc rc0: IR event FIFO is full!
Feb 12 17:44:52 localhost kernel: [23357.366962] rc rc0: IR event FIFO is full!
Feb 12 17:44:52 localhost kernel: [23357.366966] rc rc0: IR event FIFO is full!

 

1: I don't like removing logs

 

2: this is a recurring bug on several pis; either I am doing something wrong each time, or there is a bug deep in Armbian.

 

Link to comment
Share on other sites

50 minutes ago, DoubleHP said:

this is a recurring bug on several pis; either I am doing something wrong each time, or there is a bug deep in Armbian.


A modern kernel for H3 is in a testing phase and has bugs. Some are known, some unknown. There is more output to logs due to various debug options, which helps hunting bugs. If you choose to use bleeding edge kernel, this comes in the package and is a minor problem.

 

Ramlog. You need to modify it/change cleaning interval up to your use case or disable. On a board with only 256Mb memory ... you don't have much room.

Link to comment
Share on other sites

I am working on a workaround; but there is a bug at the distribution level, because once var has reached 100%, the only way to get /var/log working again is to manually remove a file; if you reboot 10 times or wait a full week, the system will NOT fix itself. And since the bug occurs often, there should be a fix at the distro level.

Link to comment
Share on other sites

 

41 minutes ago, DoubleHP said:

And since the bug occurs often, there should be a fix at the distro level.


It is not exactly a distribution bug if you run out of memory for logs - which is set very low by purpose. You can tweak the size, you can tweak logrotate, flushing or disable entirely. Out of memory is generally a problem.

 

I agree that there is a room for improvement and am happy that someone is prepared to improve those scripts.

Link to comment
Share on other sites

17 hours ago, Igor said:

It is not exactly a distribution bug if you run out of memory for logs

 

Well, it's the distro'ss decision to use this log2ram mechanism and to stay with defaults that in the meantime look somewhat problematic to me.

 

We're syncing at boot the whole /var/log contents including rotated/compressed logs back into RAM: https://github.com/armbian/build/blob/master/packages/bsp/common/usr/sbin/log2ram#L40-L44

 

If we would exclude rotated/compressed logs they would exist afterwards only in /var/log.hdd and missing in /var/log which might be confusing for users and break log analysis software and so on. Using symlinks might work but opens up another can of worms.

 

Isn't there an other option that works somewhat like an overlayfs? @zador.blood.stained IIRC we already talked about this a year ago but I can't remember the details :( 

 

Edit: quick google check: https://github.com/azlux/log2ram/commit/e88f67ab23a91bb1482f0f2063b990585b27730c

 

Link to comment
Share on other sites

When default settings lead to "aptitude install munin-node" refuse to install the package, on 3 machines in two weeks, there is a bug at the distro level.

 

My fix for now:


 

# cat /etc/crontab  | grep var
* *     * * *   root    [ $(df -h /var/log | grep -e "/var/log" | awk '{print $5}' | tr -d "\%") -gt 90 ] && rm "/var/log/$(ls -S /var/log/ | head -n1)" && systemctl restart log2ram
# this crontab line checks every minute if /var/log is more than 90% full, and if it is, removes the largest file.

tkaiser I disagree; your idea is very good; it's what I was planning to do on long term; but in emergency, I wrote the above simple line. I usually keep all my logs for 10 years. So, the best way to do this would be to sync rotated logs to .hdd, and then remove them immediately from the ramdisk. And sync at boot should exclude them. It's not hard to do; just a few rsync arguments. But I got more urgent things to do now. Logrotate also would probably need some customisation to handle side effect of this sync.

 

Not finding /var/log/syslog at all, or having an empty file is much more an issue than not finding /var/log/syslog.1

 

The partial sync also allows to have a /var/log.hdd larger than 50M, allowing to keep a lot of logs, while log2ram limits it to 50M.

 

PEople who would be confused by this can disable log2ram; but this comes with a risk for the flash.

Link to comment
Share on other sites

You could try logrotate's olddir option, apparently that works when your rotate mode is 'copytruncate'

You can then have logrotate move the old logs to your SD card instead of leave them in the tmpfs log dir

 

There seems to be varying degrees of success with olddir being on a different device though.

Link to comment
Share on other sites

log2ram writes files from ramdisk to sdcard every hour (if they are updated) by cron task. Also at startup log2ram loads all content from /var/log.hdd to ramdisk.

 

1. For improving situation with full ramdisk you can organize log rotation rules in /etc/logrotate.d to optimize logs by size. Log rotation is time dependent - daily, weekly. But, using log2ram, the best choice is to make it size-dependent.

 

edit /etc/logrotate.d/rsyslog:

 

/var/log/syslog
{

    # rotate only once
    rotate 1

    # rotate only if syslog > 10 Mb
    size 10M
    missingok
    notifempty
    delaycompress

    compress
    postrotate
        invoke-rc.d rsyslog rotate > /dev/null
    endscript
}

 

/var/log/kern.log
{
    # do not rotate, just purge file
    rotate 0
    # if size more than 1 Mb

    size 1M
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        invoke-rc.d rsyslog rotate > /dev/null
    endscript
}

 

2. If your system spams a lot of messages, you can make log rotation more aggressive:

 

move /etc/cron.daily/logrotate to /etc/cron.hourly/logrotate

 

3. If you don't need detailed archived logging. You can protect your sdcard from hourly writes by telling log2ram to not write syslog and kern.log to disk. As a bonus at boot your ramdisk will not be filling with previous session's logs.

 

a) activate rsync mode in /etc/default/log2ram (or in /etc/log2ram.conf):

USE_RSYNC=true

 

b) find your log2ram (delete duplicates, if there is) and edit

whereis log2ram

 

replace two lines

rsync -aXWv --delete --links $RAM_LOG/ $HDD_LOG/ 2>&1 | $LOG_OUTPUT

and

rsync -aXWv --delete --links $HDD_LOG/ $RAM_LOG/ 2>&1 | $LOG_OUTPUT

 

with

rsync -aXWv --delete --delete-excluded --exclude-from '/etc/log2ram-exclude' --links $RAM_LOG $HDD_LOG 2>&1 | $LOG_OUTPUT

and

rsync -aXWv --delete --delete-excluded --exclude-from '/etc/log2ram-exclude' --links $HDD_LOG $RAM_LOG 2>&1 | $LOG_OUTPUT

c) create new file /etc/log2ram-exclude and add lines:

/log2ram.log
/syslog*
/kern.log*

 

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines