MattCostamagna Posted July 8, 2020 Posted July 8, 2020 Armbianmonitor: http://ix.io/2raX Hi everyone, I'm facing a problem with logrotate on my OrangePi Zero with Armbian Focal mainline. Basically I have a process that is always running with a `tee` command that outputs the stdout to a log file in the /var/log/ directory. My logrotate configuration file looks like the following: /var/log/smartbit/gocpp.log { rotate 10 daily compress missingok notifempty copytruncate dateext } It's all working correctly except the copytruncate part: logs are daily rotated and gzipped but the gocpp.log is not truncated to 0 bytes, so it gets bigger and bigger. I noticed that in this new Armbian version logrotate is not triggered by a cron but by systemd logrotate.service. If I run /usr/sbin/logrotate /etc/logrotate.conf, everything works included the copytruncate line, but with `systemctl start logrotate.service` the copytruncate line is not considered. Does anyone know why? Thanks in advance Mattia
xwiggen Posted July 8, 2020 Posted July 8, 2020 7 hours ago, MattCostamagna said: Armbianmonitor: http://ix.io/2raX Hi everyone, I'm facing a problem with logrotate on my OrangePi Zero with Armbian Focal mainline. Basically I have a process that is always running with a `tee` command that outputs the stdout to a log file in the /var/log/ directory. My logrotate configuration file looks like the following: /var/log/smartbit/gocpp.log { rotate 10 daily compress missingok notifempty copytruncate dateext } It's all working correctly except the copytruncate part: logs are daily rotated and gzipped but the gocpp.log is not truncated to 0 bytes, so it gets bigger and bigger. I noticed that in this new Armbian version logrotate is not triggered by a cron but by systemd logrotate.service. If I run /usr/sbin/logrotate /etc/logrotate.conf, everything works included the copytruncate line, but with `systemctl start logrotate.service` the copytruncate line is not considered. Does anyone know why? Thanks in advance Mattia Try changing /var/log/smartbit/goccp.log to /var/log.hdd/smartbit/goccp.log /var/log is a ramlog mount that's synced underwater to /var/log.hdd to decrease SD writes.
MattCostamagna Posted July 9, 2020 Author Posted July 9, 2020 @xwiggen Thank you for your reply. I tried doing what you suggested and it worked. I still don't understand why /var/log does not work while /var/log.hdd works. Now I have two problems: I configured /var/log as a tmpfs in /etc/fstab (tmpfs /var/log tmpfs nodev,nosuid 0 0) so that log files do not count as writes to the SD card. I previously didn't know about /var/log.hdd, so I think that now this doesn't make much sense. I'd still like to write logs in a way that do not spoil the SD card on the long term, how can I do it? How can I disable the writes to /var/log.hdd? Another strange thing is that in the Armbian version I was using before (where logrotate is triggered by a daily cron and not by systemd) I didn't have the copytruncate problem, now it seems that the problem is systemd because if I run /usr/sbin/logrotate /etc/logrotate.conf everything works as expected.
xwiggen Posted July 9, 2020 Posted July 9, 2020 14 hours ago, MattCostamagna said: @xwiggen Thank you for your reply. I tried doing what you suggested and it worked. I still don't understand why /var/log does not work while /var/log.hdd works. Now I have two problems: I configured /var/log as a tmpfs in /etc/fstab (tmpfs /var/log tmpfs nodev,nosuid 0 0) so that log files do not count as writes to the SD card. I previously didn't know about /var/log.hdd, so I think that now this doesn't make much sense. I'd still like to write logs in a way that do not spoil the SD card on the long term, how can I do it? How can I disable the writes to /var/log.hdd? Another strange thing is that in the Armbian version I was using before (where logrotate is triggered by a daily cron and not by systemd) I didn't have the copytruncate problem, now it seems that the problem is systemd because if I run /usr/sbin/logrotate /etc/logrotate.conf everything works as expected. var/log should already be ram (zramctl) in sync. non-persistent logging defeats the purpose. what you can do is log to /tmp (tmpfs) en set logrotate entry to /tmp/... with the options you mentioned.
MattCostamagna Posted July 10, 2020 Author Posted July 10, 2020 8 hours ago, xwiggen said: var/log should already be ram (zramctl) in sync. @xwiggen Yes, this is something I didn't know before, so now I removed the tmpfs entry in fstab. When I did this yesterday, the copyctruncate in logrotate magically started working. I have no idea why the tmpfs conflicted with logrotate. I wanted to post this yesterday but the Armbian forum didn't let me because I reached the maximum number of posts I can make per day (1 I guess)... 9 hours ago, xwiggen said: non-persistent logging defeats the purpose Yes I know but for the type of application I have to develop I don't care that much o logging because it's all saved on a cloud environment; local logs are useful to me only when everything else fails. Anyway now my logs are working as I expected. The only other thing that doesn't make sense to me is /var/log.hdd that is not updating. This is its current content: $ ls -halt /var/log.hdd/ -rw-r--r-- 1 root root 1.2K Jun 4 12:56 armbian-ramlog.log -rw-rw-r-- 1 root utmp 5.3K Jun 4 12:56 wtmp -rw-r----- 1 syslog adm 156K Jun 4 12:56 syslog -rw-r----- 1 syslog adm 6.0K Jun 4 12:55 auth.log -rw-r--r-- 1 root root 264K Jun 4 12:54 dpkg.log -rw-r----- 1 syslog adm 68K Jun 4 12:54 kern.log -rw-r--r-- 1 root root 24K Jun 4 12:54 faillog -rw-rw-r-- 1 root utmp 286K Jun 4 12:54 lastlog drwxrwxr-x 10 root syslog 4.0K Jun 4 12:54 . drwxr-xr-x 2 root root 4.0K Jun 4 12:54 apt drwxr-xr-x 2 root adm 4.0K Jun 4 12:49 nginx drwxr-xr-x 13 root root 4.0K Jun 4 12:49 .. -rw-r--r-- 1 root root 42K Jun 4 12:35 armbian-hardware-monitor.log -rw-r--r-- 1 root root 18K Jun 4 12:35 alternatives.log -rw-r--r-- 1 root root 118K Jun 4 12:35 bootstrap.log -rw-rw---- 1 root utmp 0 Jun 4 12:35 btmp -rw-r--r-- 1 root adm 38K Jun 4 12:35 dmesg drwxr-x--- 2 root adm 4.0K Jun 3 13:45 unattended-upgrades drwx------ 2 root root 4.0K May 28 18:53 private drwxr-sr-x 2 root systemd-journal 4.0K May 28 18:53 journal drwxr-xr-x 2 _chrony _chrony 4.0K Apr 20 13:58 chrony drwxr-xr-x 2 ntp ntp 4.0K Apr 2 17:37 ntpstats drwxr-xr-x 2 root root 4.0K Dec 23 2019 sysstat As you can see files are very old. I'm not an expert but I thing it should be periodically updated with the content of the /var/log directory, isn't it? I thought this was caused by the fact that I made the "/" partition read-only for security purpose, but I can write into that directory without any problem. So /var/log.hdd is not updating (which is kind of what I want, even though I don't like the fact that is working for reasons unknown to me), and if I mount the sd card into my computer to see the content of the partitions I made, the /var/log.hdd is empty. So is the content really stored on the physical memory? Too many mysteries... I haven't found any documentation on the internet about all of this, so if anybody knows where I can read something about how logs are managed by Armbian, please send it to me. Thanks a lot
xwiggen Posted July 11, 2020 Posted July 11, 2020 On 7/10/2020 at 9:18 AM, MattCostamagna said: @xwiggen Yes, this is something I didn't know before, so now I removed the tmpfs entry in fstab. When I did this yesterday, the copyctruncate in logrotate magically started working. I have no idea why the tmpfs conflicted with logrotate. I wanted to post this yesterday but the Armbian forum didn't let me because I reached the maximum number of posts I can make per day (1 I guess)... Yes I know but for the type of application I have to develop I don't care that much o logging because it's all saved on a cloud environment; local logs are useful to me only when everything else fails. Anyway now my logs are working as I expected. The only other thing that doesn't make sense to me is /var/log.hdd that is not updating. This is its current content: $ ls -halt /var/log.hdd/ -rw-r--r-- 1 root root 1.2K Jun 4 12:56 armbian-ramlog.log -rw-rw-r-- 1 root utmp 5.3K Jun 4 12:56 wtmp -rw-r----- 1 syslog adm 156K Jun 4 12:56 syslog -rw-r----- 1 syslog adm 6.0K Jun 4 12:55 auth.log -rw-r--r-- 1 root root 264K Jun 4 12:54 dpkg.log -rw-r----- 1 syslog adm 68K Jun 4 12:54 kern.log -rw-r--r-- 1 root root 24K Jun 4 12:54 faillog -rw-rw-r-- 1 root utmp 286K Jun 4 12:54 lastlog drwxrwxr-x 10 root syslog 4.0K Jun 4 12:54 . drwxr-xr-x 2 root root 4.0K Jun 4 12:54 apt drwxr-xr-x 2 root adm 4.0K Jun 4 12:49 nginx drwxr-xr-x 13 root root 4.0K Jun 4 12:49 .. -rw-r--r-- 1 root root 42K Jun 4 12:35 armbian-hardware-monitor.log -rw-r--r-- 1 root root 18K Jun 4 12:35 alternatives.log -rw-r--r-- 1 root root 118K Jun 4 12:35 bootstrap.log -rw-rw---- 1 root utmp 0 Jun 4 12:35 btmp -rw-r--r-- 1 root adm 38K Jun 4 12:35 dmesg drwxr-x--- 2 root adm 4.0K Jun 3 13:45 unattended-upgrades drwx------ 2 root root 4.0K May 28 18:53 private drwxr-sr-x 2 root systemd-journal 4.0K May 28 18:53 journal drwxr-xr-x 2 _chrony _chrony 4.0K Apr 20 13:58 chrony drwxr-xr-x 2 ntp ntp 4.0K Apr 2 17:37 ntpstats drwxr-xr-x 2 root root 4.0K Dec 23 2019 sysstat As you can see files are very old. I'm not an expert but I thing it should be periodically updated with the content of the /var/log directory, isn't it? I thought this was caused by the fact that I made the "/" partition read-only for security purpose, but I can write into that directory without any problem. So /var/log.hdd is not updating (which is kind of what I want, even though I don't like the fact that is working for reasons unknown to me), and if I mount the sd card into my computer to see the content of the partitions I made, the /var/log.hdd is empty. So is the content really stored on the physical memory? Mounting / as read-only is for forensic purposes, journal etc won't be committed, i.e. nothing new is flushed to SD.
Recommended Posts