Igor Posted May 22, 2018 Posted May 22, 2018 What about solving ramlog this way: - added cronjob on every 15 minutes - if space is 75% full do: logrotate --force /etc/logrotate.d/ write to SD card truncate and remove files It is better than now. Good enough for defaults? Spoiler diff --git a/packages/bsp/common/etc/cron.d/armbian-truncate-logs b/packages/bsp/common/etc/cron.d/armbian-truncate-logs new file mode 100644 index 0000000..dbc024c --- /dev/null +++ b/packages/bsp/common/etc/cron.d/armbian-truncate-logs @@ -0,0 +1 @@ +*/15 * * * * root /usr/lib/armbian/truncate-logs \ No newline at end of file diff --git a/packages/bsp/common/usr/lib/armbian/truncate-logs b/packages/bsp/common/usr/lib/armbian/truncate-logs new file mode 100644 index 0000000..8154b8f --- /dev/null +++ b/packages/bsp/common/usr/lib/armbian/truncate-logs @@ -0,0 +1,17 @@ +#!/bin/sh +# +# truncate, save and clean logs if they get over 75% of the /var/log size +# + +treshold=75 # % +logusage=$(df /var/log/ --output=pcent | tail -1 |cut -d "%" -f 1) +if [ $logusage -ge $treshold ]; then + logrotate --force /etc/logrotate.d/ + # write to SD + log2ram write + # truncate + /usr/bin/find /var/log -name '*.log' -or -name '*.xz' -or -name 'lastlog' -or -name 'messages' -or -name 'debug' -or -name 'syslog' | xargs truncate --size 0 + /usr/bin/find /var/log -name 'btmp' -or -name 'wtmp' | xargs truncate --size 0 + # remove + /usr/bin/find /var/log -name '*.1' -or -name '*.2' -or -name '*.3' -or -name '*.gz' | xargs rm +fi \ No newline at end of file diff --git a/packages/bsp/common/usr/sbin/log2ram b/packages/bsp/common/usr/sbin/log2ram index 2f41a8f..ea4e853 100755 --- a/packages/bsp/common/usr/sbin/log2ram +++ b/packages/bsp/common/usr/sbin/log2ram @@ -1,6 +1,7 @@ #!/bin/sh # Source: https://github.com/azlux/log2ram +# Modified by Igor Pecovnik. Added selective copy. Storing archives only on hdd to preserve memory # License: MIT # License file: /usr/share/log2ram/LICENSE @@ -28,7 +29,7 @@ syncToDisk () { isSafe if [ "$USE_RSYNC" = true ]; then - rsync -aXWv --delete --exclude log2ram.log --links $RAM_LOG $HDD_LOG 2>&1 | $LOG_OUTPUT + rsync -aXWv --exclude log2ram.log --links $RAM_LOG $HDD_LOG 2>&1 | $LOG_OUTPUT else cp -rfup $RAM_LOG -T $HDD_LOG 2>&1 | $LOG_OUTPUT fi @@ -38,9 +39,10 @@ syncFromDisk () { isSafe if [ "$USE_RSYNC" = true ]; then - rsync -aXWv --delete --exclude log2ram.log --links $HDD_LOG $RAM_LOG 2>&1 | $LOG_OUTPUT + rsync -aXWv --delete --exclude log2ram.log --exclude *.gz --links $HDD_LOG $RAM_LOG 2>&1 | $LOG_OUTPUT else - cp -rfup $HDD_LOG -T $RAM_LOG 2>&1 | $LOG_OUTPUT + cd $HDD_LOG + find $HDD_LOG* -not \( -name '*.1' -or -name '*.xz*' -or -name '*.gz' \) | sed -n 's|^'$HDD_LOG'||p' | cpio -pvdmu $RAM_LOG fi } 1
guidol Posted May 22, 2018 Posted May 22, 2018 25 minutes ago, Igor said: It is better than now. Good enough for defaults? Sounds OK for me - on my PiHole, a OPi PC2 with 1GB Ram - I extended the log2ram-size from 50MB to 250MB Spoiler root@pi-hole:/var/log# more /etc/default/log2ram # configuration values for the log2ram service # # enable the log2ram service? ENABLED=true # # size of the tmpfs mount SIZE=250M # # use rsync instead of cp -r # requires rsync installed, may provide better performance # due to copying only new and changed files USE_RSYNC=true # Before the OPi PC2 I got a OPi One as PiHole - and did get less problems while setting the size from 50 to 150MB - but a good logrotate and cleanup is also fine for me
zador.blood.stained Posted May 22, 2018 Posted May 22, 2018 Just not syncing compressed logs back might help, but we need to test how this behaves with logrotate deleting old compressed logs. 1 hour ago, Igor said: +++ b/packages/bsp/common/usr/lib/armbian/truncate-logs This has some rough edges so I would prefer to rework this first (or do tests without this) before pushing this.
tkaiser Posted May 22, 2018 Posted May 22, 2018 The current use of tmpfs is just wasting resources. Logs are highly compressable so a zram based improvement is able to reduce waste of resources by 5 to 10 times.
Igor Posted May 23, 2018 Author Posted May 23, 2018 6 hours ago, tkaiser said: The current use of tmpfs is just wasting resources. Logs are highly compressable so a zram based improvement is able to reduce waste of resources by 5 to 10 times. That would be cool, yes. Now how to implement it most sane way?
tkaiser Posted May 23, 2018 Posted May 23, 2018 1 hour ago, Igor said: Now how to implement it most sane way? Well, switching from tmpfs to an already existing zram block device that has been prepared before is most probably the easy part. Prerequisit to use such a zram device would be refactoring at least armhwinfo (so that armhwinfo eventually again only does what the name tells: collecting and logging information). All the stuff that configures/optimizes should end up in own services that can be configured in a sane way so that user changes do not get lost with next update. No idea how to do that though Then the whole zram idea needs a solution since in case we reinvent the wheel and create zram devices on our own (which seems like the only way to get zram for logs to me) we need to deal with the existence of other zram related services (e.g. zram-control package in Ubuntu) And I fear the part that needs most attention is logrotate and how to interfere with. Moving old/compressed logs into RAM is no good idea at all, also moving logs compressed by logrotate from RAM to 'disk' and free RAM afterwards seems like the only sane way to me. But I've no idea how this could evolve without reinventing the wheel again and implement logrotate functionality on our own now being aware of two filesystems to use instead of one. Maybe the use of symlinks helps? The initial sync from storage to RAM does not copy old/compressed contents but creates them as symlinks to the /var/log.hdd/ variant. And then a cronjob monitors /var/log and as soon as there appear rotated logs they're moved over to /var/log.hdd and are replaced with symlinks. Active logfiles are treated differently and simply rsynced on a per file basis to /var/log.hdd 1
Igor Posted May 23, 2018 Author Posted May 23, 2018 1 hour ago, tkaiser said: All the stuff that configures/optimizes should end up in own services that can be configured in a sane way so that user changes do not get lost with next update. No idea how to do that though Split optimizations part into /etc/init.d/armhwopti with optional configuration in /etc/default/armhwopti ? 1 hour ago, tkaiser said: we need to deal with the existence of other zram related services This way as it is now in activate_zram() we don't need any 3rd party utility, right? And it should work in Debian and Ubuntu where a kernel has supports ZRAM? 1 hour ago, tkaiser said: Moving old/compressed logs into RAM is no good idea at all, also moving logs compressed by logrotate from RAM to 'disk' and free RAM afterwards seems like the only sane way to me. But I've no idea how this could evolve without reinventing the wheel again and implement logrotate functionality on our own now being aware of two filesystems to use instead of one. I agree with compressing something that is already compressed is stupid. Logrotate has some interesting options (olddir directory: Logs are moved into directory for rotation.). I will examine if they can be used for this purpose.
tkaiser Posted May 23, 2018 Posted May 23, 2018 11 minutes ago, Igor said: Split optimizations part into /etc/init.d/armhwopti with optional configuration in /etc/default/armhwopti ? If I understood @zador.blood.stained correctly he would prefer splitting this up in a more granular way. Currently we have the following functions in the script (the ones that can remain in armhwinfo marked with an X): collect_information X set_io_scheduler prepare_temp_monitoring X prepare_board log_hardware_info X get_flash_information X show_motd_warning X check_sd_card_speed X add_usb_storage_quirks activate_zram So prepare_board, set_io_scheduler and add_usb_storage_quirks could go into an armbiansetuphw service and zram could be handled by armbianzram (I really would prefer to prefix all our services with armbian so users get a clue what's plain Ubuntu/Debian and what's Armbian). 17 minutes ago, Igor said: This way as it is now in activate_zram() we don't need any 3rd party utility, right? Yes, but please see @zador.blood.stained's objections. I'm fine with any way as long as it's configurable and able to be disabled. The log2ram functionality would then have to implement a fallback mechanism to what we use today (check for /dev/zram0 whether it contains a freshly created btrfs filesystem. If yes use it, otherwise do the fallback to tmpfs) 21 minutes ago, Igor said: I agree with compressing something that is already compressed is stupid It's worse also for another reason: Recompressing already compressed stuff usually wastes more storage space. An uncompressed 10 MB log might end up being a 1 MB .gz file or as uncompressed file using 1.5 MB DRAM on an lzo zram device. But the 1 MB gzip version might need +2 MB on the same lzo compressed zram device. So it's not just a waste of CPU cycles but also inefficient DRAM use. 1
Tido Posted May 30, 2018 Posted May 30, 2018 Is there anything going on with regard to the log file issue ? Do you need some help ?
Igor Posted May 30, 2018 Author Posted May 30, 2018 10 minutes ago, Tido said: Is there anything going on with regard to the log file issue ? Do you need some help ? Log issue is somehow solved, but not implemented. We need this major BSP scripting rework, splitting first. What @tkaiser already exposed. I read objections, have things in my mind at this stage but haven't moved forward with this. Waiting for the right moment to make the first move
Tido Posted June 9, 2018 Posted June 9, 2018 @tkaiser as I have not my eyes behind the scene - how are we doing with the changes ?
Igor Posted June 12, 2018 Author Posted June 12, 2018 https://github.com/armbian/build/commit/57d5ec2a952b49677af4b272f6c204fbd988b25d I hope this goes in the right direction?
Tido Posted June 12, 2018 Posted June 12, 2018 Quote Zador: I would prefer to split it into multiple services: a logging service, an optimizing service, a hardware monitoring service, etc. 1 hour ago, Igor said: this goes in the right direction? For my limited understanding, are these the new 'modules' ? Will this script upgrade installations with the new 'modules' or is it for fresh install only ? If it is for fresh install only, will it break current installations ? armbian-firstrun.service armbian-firstrun-config.service armbian-optimize-hardware.service armbian-ram-logging.service armbian-resize-filesystem.service armbian-setup-hardware.service How can I update the: fix the basic - CONTRIBUTING.md Quote Our tools to control and setup armbian armhwinfo armbiansetuphw armbianzram armbian-config armbianmonitor scripts configuration
Igor Posted June 12, 2018 Author Posted June 12, 2018 7 minutes ago, Tido said: Will this script upgrade installations with the new 'modules' or is it for fresh install only ? On update, old scrips are (should be) removed and replaced with new which are at the new location. Next step is to divide functions as @tkaiser and @zador.blood.stained proposed and clean up. 12 minutes ago, Tido said: How can I update the: It's too soon and things can be twisted around ... after it is merged into master branch. 1
tkaiser Posted June 16, 2018 Posted June 16, 2018 On 6/12/2018 at 10:09 PM, Igor said: Next step is to divide functions as @tkaiser and @zador.blood.stained proposed and clean up. My 2 cents on this at the bottom of the list: https://github.com/armbian/build/pull/1015 Can't really test/review at the moment since running out of spare time. My only other 'Armbian contribution' this week will be a little bit NanoPC T4 testing (relying on @hjc's work). 1
Igor Posted June 16, 2018 Author Posted June 16, 2018 17 minutes ago, tkaiser said: My 2 cents on this at the bottom of the list: https://github.com/armbian/build/pull/1015 Thanks. O.K. I will do few more tests with your changes and then merge in. It was tested well, but we all know how this goes. Let's point out for more profound testing when changes reach beta repository. We need to make sure upgrading won't break, while for new images I am not worried that much. One a bit radical idea, which crossed on my mind - do we actually need a swap file enabled by default? In general, features of the old armhwinfo scripts weren't changed much while the change logic was hopefully met discussion above. 1
Tido Posted June 16, 2018 Posted June 16, 2018 Can the community somehow help testing or is it very complicated?
Igor Posted June 16, 2018 Author Posted June 16, 2018 2 hours ago, Tido said: Can the community somehow help testing or is it very complicated? Merged. It will be in the nightly images in a few hours from now. Feel free to call for action. Changes are written in the commit header and from that, it can be extracted what we should pay attention most.
Tido Posted June 24, 2018 Posted June 24, 2018 reading about problems, does the script add it to /etc/fstab ? https://wiki.archlinux.org/index.php/improving_performance#Swap_on_zRAM_using_a_udev_rule /dev/zram0 none swap defaults 0 0 versus echo "$SWAPFILE none swap sw 0 0" don't let you lead somewhere weird I was just looking how ARCH does it.
Igor Posted June 24, 2018 Author Posted June 24, 2018 24 minutes ago, Tido said: reading about problems, does the script add it to /etc/fstab ? No need. We have a script for adding zram swap. Are thee still any problems?
tpm8 Posted July 9, 2018 Posted July 9, 2018 Using armbian 5.51 debian stretch I found the following errors in the system log from the reboot after a system crash. At least Apache failed to start. root@hc1:/var/log# journalctl | grep "/var/log" Jul 09 12:13:16 hc1 apachectl[970]: (2)No such file or directory: AH02291: Cannot access directory '/var/log/apache2/' for main error log Jul 09 12:13:16 hc1 apachectl[970]: (2)No such file or directory: AH02291: Cannot access directory '/var/log/apache2/' for error log of vhost defined at /etc/apache2/sites-enabled/nextcloud.conf:7 Jul 09 12:13:16 hc1 apachectl[970]: (2)No such file or directory: AH02291: Cannot access directory '/var/log/apache2/' for error log of vhost defined at /etc/apache2/sites-enabled/default-ssl.conf:2 Jul 09 12:13:16 hc1 apachectl[970]: (2)No such file or directory: AH02291: Cannot access directory '/var/log/apache2/' for error log of vhost defined at /etc/apache2/sites-enabled/000-default.conf:1 Jul 09 12:13:21 hc1 nmbd[1414]: mkdir failed on directory /var/log/samba/cores: No such file or directory Jul 09 12:13:21 hc1 nmbd[1414]: Failed to create /var/log/samba/cores for user 0 with mode 0700 Jul 09 12:13:21 hc1 nmbd[1414]: Unable to open new log file '/var/log/samba/log.': No such file or directory Jul 09 12:13:21 hc1 nmbd[1414]: Unable to open new log file '/var/log/samba/log.nmbd': No such file or directory Jul 09 12:13:22 hc1 smbd[1439]: Unable to open new log file '/var/log/samba/log.smbd': No such file or directory Jul 09 12:13:22 hc1 smbd[1439]: Unable to open new log file '/var/log/samba/log.smbd': No such file or directory Jul 09 12:13:22 hc1 smbd[1439]: Unable to open new log file '/var/log/samba/log.smbd': No such file or directory Could you please add samba and apache2 log directory to the function RecreateLogs in /usr/lib/armbian/armbian-ramlog: RecreateLogs (){ # in case of crash those services doesn't start if there are no dirs & logs [ ! -d /var/log/proftpd ] && mkdir -p /var/log/proftpd [ ! -d /var/log/nginx ] && mkdir -p /var/log/nginx [ ! -d /var/log/apache2 ] && mkdir -p /var/log/apache2 [ ! -d /var/log/samba ] && mkdir -p /var/log/samba [ ! -f /var/log/proftpd/controls.log ] && touch /var/log/proftpd/controls.log [ ! -f /var/log/nginx/access.log ] && touch /var/log/nginx/access.log [ ! -f /var/log/nginx/error.log ] && touch /var/log/nginx/error.log } Probably it would be a good idea to schedule a ramlog syncToDisk once per day using an systemd timer? Thanks & keep up the good work. Tobias
tkaiser Posted July 9, 2018 Posted July 9, 2018 1 hour ago, tpm8 said: Could you please add samba and apache2 log directory to the function RecreateLogs in /usr/lib/armbian/armbian-ramlog [x] done I agree on the need for a regular resync to 'disk' and wonder whether it's possible to make it somewhat easy configurable (e.g. defaulting to 1440 minutes which can be overwritten by setting SYNC_INTERVAL=60 in /etc/default/armbian-ramlog)?
Igor Posted July 9, 2018 Author Posted July 9, 2018 5 hours ago, tpm8 said: Probably it would be a good idea to schedule a ramlog syncToDisk once per day using an systemd timer? We had that, but it was lost during this RFC. Fixed that current solution works again: https://github.com/armbian/build/commit/e78c66b3a91a7627cb502f2dd2ddfcbba470a8b1
Recommended Posts