DoubleHP Posted February 21, 2018 Posted February 21, 2018 # echo plop | logger should show things in tail -F /var/log/syslog After digging a bit: # df -h Filesystem Size Used Avail Use% Mounted on udev 85M 0 85M 0% /dev tmpfs 24M 3.0M 22M 13% /run /dev/mmcblk0p1 15G 1.3G 13G 9% / tmpfs 120M 4.0K 120M 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 120M 0 120M 0% /sys/fs/cgroup tmpfs 120M 0 120M 0% /tmp log2ram 50M 50M 0 100% /var/log tmpfs 24M 0 24M 0% /run/user/0 # du -lsh /var/log/* | grep M 24M /var/log/kern.log.1 25M /var/log/syslog.1 so ... WTF ? # tail -n1 /var/log/kern.log.1 Feb 12 10:35:14 localhost kernel: [ 6837.372115] sunxi-mmcroot@opi-12-gite-salon:/var/log# tail -n1 /var/log/syslog.1 # date Wed Feb 21 13:48:48 UTC 2018 # uptime 13:48:50 up 33 min, 2 users, load average: 0.00, 0.01, 0.00 Why is this crap filling my RAM with deprecated logs that are older than 1 week ? Either the system was up, and logrotate should compress and push this to disk, or, the system had been restarted recetly, and log2ram has no reason to fill my memory with deprecated logs !!! So, the point of log2ram is just to occupy RAM, and prevent admin from seing fresh messages ? Not even able to force log rotation when the ramdisk is full ?
DoubleHP Posted February 28, 2018 Author Posted February 28, 2018 On most your Orange Pi Zero 256M, I got this: # df -h Filesystem Size Used Avail Use% Mounted on udev 85M 0 85M 0% /dev tmpfs 25M 3.5M 21M 15% /run /dev/mmcblk0p1 15G 1.3G 14G 9% / tmpfs 120M 4.0K 120M 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 120M 0 120M 0% /sys/fs/cgroup tmpfs 120M 80K 120M 1% /tmp log2ram 50M 50M 0 100% /var/log tmpfs 25M 0 25M 0% /run/user/0 This happens very fast, after a few days, for Pis that are now plugged 24/7. Then I have to manually remove some large file, and, when the Pi have run a cron.daily and cron.weekly, and it always stays on, then, this never happens again. What is the best workaround, or fix ? Obvisouly, I can not really increase the size of the ramdisk. I don't want either to stop using log2ram; I have killed many SD cards in the past. And the problem is always the same: # ls -lha /var/log/ | grep M total 50M -rw-r----- 1 syslog adm 25M Feb 27 04:49 kern.log.1 -rw-r----- 1 syslog adm 25M Feb 27 06:25 syslog.1 drwxr-xr-x 2 root root 40 May 12 2016 sysstat This is the third time. # head /var/log/syslog.1 Nov 25 06:20:40 localhost rsyslogd: [origin software="rsyslogd" swVersion="8.16.0" x-pid="579" x-info="http://www.rsyslog.com"] start Nov 25 06:20:40 localhost rsyslogd-2222: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ] Nov 25 06:20:40 localhost rsyslogd: rsyslogd's groupid changed to 108 Nov 25 06:20:40 localhost rsyslogd: rsyslogd's userid changed to 104 Nov 25 06:20:40 localhost kernel: [ 0.000000] Booting Linux on physical CPU 0x0 Nov 25 06:20:40 localhost kernel: [ 0.000000] random: get_random_bytes called from start_kernel+0x31/0x352 with crng_init=0 Nov 25 06:20:40 localhost kernel: [ 0.000000] Linux version 4.13.16-sunxi (root@armbian) (gcc version 7.1.1 20170707 (Linaro GCC 7.1-2017.08)) #20 SMP Fri Nov 24 19:50:07 CET 2017 Nov 25 06:20:40 localhost kernel: [ 0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=50c5387d Nov 25 06:20:40 localhost kernel: [ 0.000000] CPU: div instructions available: patching division code Nov 25 06:20:40 localhost kernel: [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache # tail /var/log/syslog.1 Feb 12 17:44:52 localhost kernel: [23357.366931] rc rc0: IR event FIFO is full! Feb 12 17:44:52 localhost kernel: [23357.366935] rc rc0: IR event FIFO is full! Feb 12 17:44:52 localhost kernel: [23357.366940] rc rc0: IR event FIFO is full! Feb 12 17:44:52 localhost kernel: [23357.366944] rc rc0: IR event FIFO is full! Feb 12 17:44:52 localhost kernel: [23357.366949] rc rc0: IR event FIFO is full! Feb 12 17:44:52 localhost kernel: [23357.366953] rc rc0: IR event FIFO is full! Feb 12 17:44:52 localhost kernel: [23357.366957] rc rc0: IR event FIFO is full! Feb 12 17:44:52 localhost kernel: [23357.366962] rc rc0: IR event FIFO is full! Feb 12 17:44:52 localhost kernel: [23357.366966] rc rc0: IR event FIFO is full! 1: I don't like removing logs 2: this is a recurring bug on several pis; either I am doing something wrong each time, or there is a bug deep in Armbian.
Igor Posted February 28, 2018 Posted February 28, 2018 50 minutes ago, DoubleHP said: this is a recurring bug on several pis; either I am doing something wrong each time, or there is a bug deep in Armbian. A modern kernel for H3 is in a testing phase and has bugs. Some are known, some unknown. There is more output to logs due to various debug options, which helps hunting bugs. If you choose to use bleeding edge kernel, this comes in the package and is a minor problem. Ramlog. You need to modify it/change cleaning interval up to your use case or disable. On a board with only 256Mb memory ... you don't have much room.
DoubleHP Posted February 28, 2018 Author Posted February 28, 2018 I am working on a workaround; but there is a bug at the distribution level, because once var has reached 100%, the only way to get /var/log working again is to manually remove a file; if you reboot 10 times or wait a full week, the system will NOT fix itself. And since the bug occurs often, there should be a fix at the distro level.
Igor Posted February 28, 2018 Posted February 28, 2018 41 minutes ago, DoubleHP said: And since the bug occurs often, there should be a fix at the distro level. It is not exactly a distribution bug if you run out of memory for logs - which is set very low by purpose. You can tweak the size, you can tweak logrotate, flushing or disable entirely. Out of memory is generally a problem. I agree that there is a room for improvement and am happy that someone is prepared to improve those scripts.
tkaiser Posted March 1, 2018 Posted March 1, 2018 17 hours ago, Igor said: It is not exactly a distribution bug if you run out of memory for logs Well, it's the distro'ss decision to use this log2ram mechanism and to stay with defaults that in the meantime look somewhat problematic to me. We're syncing at boot the whole /var/log contents including rotated/compressed logs back into RAM: https://github.com/armbian/build/blob/master/packages/bsp/common/usr/sbin/log2ram#L40-L44 If we would exclude rotated/compressed logs they would exist afterwards only in /var/log.hdd and missing in /var/log which might be confusing for users and break log analysis software and so on. Using symlinks might work but opens up another can of worms. Isn't there an other option that works somewhat like an overlayfs? @zador.blood.stained IIRC we already talked about this a year ago but I can't remember the details Edit: quick google check: https://github.com/azlux/log2ram/commit/e88f67ab23a91bb1482f0f2063b990585b27730c 1
DoubleHP Posted March 2, 2018 Author Posted March 2, 2018 When default settings lead to "aptitude install munin-node" refuse to install the package, on 3 machines in two weeks, there is a bug at the distro level. My fix for now: # cat /etc/crontab | grep var * * * * * root [ $(df -h /var/log | grep -e "/var/log" | awk '{print $5}' | tr -d "\%") -gt 90 ] && rm "/var/log/$(ls -S /var/log/ | head -n1)" && systemctl restart log2ram # this crontab line checks every minute if /var/log is more than 90% full, and if it is, removes the largest file. tkaiser I disagree; your idea is very good; it's what I was planning to do on long term; but in emergency, I wrote the above simple line. I usually keep all my logs for 10 years. So, the best way to do this would be to sync rotated logs to .hdd, and then remove them immediately from the ramdisk. And sync at boot should exclude them. It's not hard to do; just a few rsync arguments. But I got more urgent things to do now. Logrotate also would probably need some customisation to handle side effect of this sync. Not finding /var/log/syslog at all, or having an empty file is much more an issue than not finding /var/log/syslog.1 The partial sync also allows to have a /var/log.hdd larger than 50M, allowing to keep a lot of logs, while log2ram limits it to 50M. PEople who would be confused by this can disable log2ram; but this comes with a risk for the flash.
chrisf Posted March 2, 2018 Posted March 2, 2018 You could try logrotate's olddir option, apparently that works when your rotate mode is 'copytruncate' You can then have logrotate move the old logs to your SD card instead of leave them in the tmpfs log dir There seems to be varying degrees of success with olddir being on a different device though.
tkaiser Posted March 5, 2018 Posted March 5, 2018 Just stumbled accross this: https://bbs.archlinux.org/viewtopic.php?id=139141 (only posted to preserve the link)
borombo Posted March 16, 2018 Posted March 16, 2018 log2ram writes files from ramdisk to sdcard every hour (if they are updated) by cron task. Also at startup log2ram loads all content from /var/log.hdd to ramdisk. 1. For improving situation with full ramdisk you can organize log rotation rules in /etc/logrotate.d to optimize logs by size. Log rotation is time dependent - daily, weekly. But, using log2ram, the best choice is to make it size-dependent. edit /etc/logrotate.d/rsyslog: /var/log/syslog { # rotate only once rotate 1 # rotate only if syslog > 10 Mb size 10M missingok notifempty delaycompress compress postrotate invoke-rc.d rsyslog rotate > /dev/null endscript } /var/log/kern.log { # do not rotate, just purge file rotate 0 # if size more than 1 Mb size 1M missingok notifempty compress delaycompress sharedscripts postrotate invoke-rc.d rsyslog rotate > /dev/null endscript } 2. If your system spams a lot of messages, you can make log rotation more aggressive: move /etc/cron.daily/logrotate to /etc/cron.hourly/logrotate 3. If you don't need detailed archived logging. You can protect your sdcard from hourly writes by telling log2ram to not write syslog and kern.log to disk. As a bonus at boot your ramdisk will not be filling with previous session's logs. a) activate rsync mode in /etc/default/log2ram (or in /etc/log2ram.conf): USE_RSYNC=true b) find your log2ram (delete duplicates, if there is) and edit whereis log2ram replace two lines rsync -aXWv --delete --links $RAM_LOG/ $HDD_LOG/ 2>&1 | $LOG_OUTPUT and rsync -aXWv --delete --links $HDD_LOG/ $RAM_LOG/ 2>&1 | $LOG_OUTPUT with rsync -aXWv --delete --delete-excluded --exclude-from '/etc/log2ram-exclude' --links $RAM_LOG $HDD_LOG 2>&1 | $LOG_OUTPUT and rsync -aXWv --delete --delete-excluded --exclude-from '/etc/log2ram-exclude' --links $HDD_LOG $RAM_LOG 2>&1 | $LOG_OUTPUTc) create new file /etc/log2ram-exclude and add lines: /log2ram.log /syslog* /kern.log* 2
Recommended Posts