jba Posted March 5, 2022 Share Posted March 5, 2022 On my odroid hc2 with armbian/bullseye I found some of the logs (both in /var/log and /var/log.hdd) empty, while they should have some contens (i.e. afpd.log, kern.log, user.log, ...). The empty files all have the timestamp of a run of armbian-ramlog at a recent reboot. In armbian-ramlog.log I found this for the run that created the empty files: rsync: [receiver] write failed on "/firstname.lastname@example.org~": No space left on device (28) rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.3] rsync: [sender] write error: Broken pipe (32) Seems that journal has become too big to fit on the log/ramdisk. That is strange, as I defined a maximum of 20MB for the journal and the ramdisk has 50MB. I guess this is what happened: When syncing to sd armbian-ramlog uses rsync without --delete. So if some older files have been already delete by journald (because of the 20 MB limit) on the ramdisk, they will still stay in the log on sd. Therefore the journal on the sd will become bigger and bigger. Which is not a problem because there is plenty of space on the sd. However, when rebooting everything is copied from the sd to the ramdisk. Including the bloated journal. Thus the ramdisk is full and additional files will not fit (see error message from rsync). I think in this case rsync just creates empty files and that is the reason for my empty logs. And on next sync the empty files are copied back to sd so both versions are empty. There is another potential problem I see in the implementation of armbian-ramlog: on reboot only recent files (so no *.gz or *.[0-9]) are copied to ramdisk. After som runtime logrotate will rotate the logs and create new *.gz or *.[0-9]. However, these might already exist on the sd. They will be overwritten when the logs are written back to sd. So you loose old logs. I think both problems are due to the fact that simple rsync is used without --delete and without copying even the old files to the ramdisk. To overcome these problems I see two ways: use rsync --delete and dont exclude *.gz and *.[0-9]. Then the ramdisk holds all logs and when copying to sd nothing is lost. But this may requier a bigger ramdisk dont use rsync but use an overlay fs instead. I found this discussion on the topic. The described solution looks superior to me than what we are using now: https://unix.stackexchange.com/questions/424341/mount-var-logs-as-tmpfs-with-help-of-overlayfs-to-save-changes-sometimes What do you think about it? Would it make sense to use an overlay fs? I am not that deep into linux hacking and therefore are not sure if I would be able to implement it in a nice way. But I would like to support this process and do some tasks. Regards, jba 0 Quote Link to comment Share on other sites More sharing options...
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.