0
Igor

BSP scripts RFC

Recommended Posts

What about solving ramlog this way:

- added cronjob on every 15 minutes

- if space is 75% full do:

  1. logrotate --force /etc/logrotate.d/
  2. write to SD card
  3. truncate and remove files

It is better than now. Good enough for defaults?

Spoiler

diff --git a/packages/bsp/common/etc/cron.d/armbian-truncate-logs b/packages/bsp/common/etc/cron.d/armbian-truncate-logs
new file mode 100644
index 0000000..dbc024c
--- /dev/null
+++ b/packages/bsp/common/etc/cron.d/armbian-truncate-logs
@@ -0,0 +1 @@
+*/15 * * * * root /usr/lib/armbian/truncate-logs
\ No newline at end of file
diff --git a/packages/bsp/common/usr/lib/armbian/truncate-logs b/packages/bsp/common/usr/lib/armbian/truncate-logs
new file mode 100644
index 0000000..8154b8f
--- /dev/null
+++ b/packages/bsp/common/usr/lib/armbian/truncate-logs
@@ -0,0 +1,17 @@
+#!/bin/sh
+#
+# truncate, save and clean logs if they get over 75% of the /var/log size
+#
+
+treshold=75 # %
+logusage=$(df /var/log/ --output=pcent | tail -1 |cut -d "%" -f 1)
+if [ $logusage -ge $treshold ]; then
+		logrotate --force /etc/logrotate.d/
+		# write to SD
+		log2ram write
+		# truncate
+        /usr/bin/find /var/log -name '*.log' -or -name '*.xz' -or -name 'lastlog' -or -name 'messages' -or -name 'debug' -or -name 'syslog' | xargs truncate --size 0
+		/usr/bin/find /var/log -name 'btmp' -or -name 'wtmp' | xargs truncate --size 0
+		# remove
+		/usr/bin/find /var/log -name '*.1' -or -name '*.2' -or -name '*.3' -or -name '*.gz' | xargs rm
+fi
\ No newline at end of file
diff --git a/packages/bsp/common/usr/sbin/log2ram b/packages/bsp/common/usr/sbin/log2ram
index 2f41a8f..ea4e853 100755
--- a/packages/bsp/common/usr/sbin/log2ram
+++ b/packages/bsp/common/usr/sbin/log2ram
@@ -1,6 +1,7 @@
 #!/bin/sh
 
 # Source: https://github.com/azlux/log2ram
+# Modified by Igor Pecovnik. Added selective copy. Storing archives only on hdd to preserve memory
 # License: MIT
 # License file: /usr/share/log2ram/LICENSE
 
@@ -28,7 +29,7 @@ syncToDisk () {
     isSafe
 
     if [ "$USE_RSYNC" = true ]; then
-        rsync -aXWv --delete --exclude log2ram.log --links $RAM_LOG $HDD_LOG 2>&1 | $LOG_OUTPUT
+        rsync -aXWv --exclude log2ram.log --links $RAM_LOG $HDD_LOG 2>&1 | $LOG_OUTPUT
     else
         cp -rfup $RAM_LOG -T $HDD_LOG 2>&1 | $LOG_OUTPUT
     fi
@@ -38,9 +39,10 @@ syncFromDisk () {
     isSafe
 
     if [ "$USE_RSYNC" = true ]; then
-        rsync -aXWv --delete --exclude log2ram.log --links $HDD_LOG $RAM_LOG 2>&1 | $LOG_OUTPUT
+        rsync -aXWv --delete --exclude log2ram.log --exclude *.gz --links $HDD_LOG $RAM_LOG 2>&1 | $LOG_OUTPUT
     else
-        cp -rfup $HDD_LOG -T $RAM_LOG 2>&1 | $LOG_OUTPUT
+		cd $HDD_LOG
+		find $HDD_LOG* -not \( -name '*.1' -or -name '*.xz*' -or -name '*.gz' \) | sed -n 's|^'$HDD_LOG'||p' | cpio -pvdmu $RAM_LOG
     fi
 }
 

 

 

Share this post


Link to post
Share on other sites
25 minutes ago, Igor said:

It is better than now. Good enough for defaults?

Sounds OK for me - on my PiHole, a OPi PC2 with 1GB Ram - I extended the log2ram-size from 50MB to 250MB

Spoiler

root@pi-hole:/var/log# more /etc/default/log2ram
# configuration values for the log2ram service
#
# enable the log2ram service?
ENABLED=true
#
# size of the tmpfs mount
SIZE=250M
#
# use rsync instead of cp -r
# requires rsync installed, may provide better performance
# due to copying only new and changed files
USE_RSYNC=true
#
 

 

Before the OPi PC2 I got a OPi One as PiHole - and did get less problems while setting the size from 50 to 150MB - but a good logrotate and cleanup is also fine for me :)

Share this post


Link to post
Share on other sites

Just not syncing compressed logs back might help, but we need to test how this behaves with logrotate deleting old compressed logs.

 

1 hour ago, Igor said:

+++ b/packages/bsp/common/usr/lib/armbian/truncate-logs

This has some rough edges so I would prefer to rework this first (or do tests without this) before pushing this.

Share this post


Link to post
Share on other sites
6 hours ago, tkaiser said:

The current use of tmpfs is just wasting resources. Logs are highly compressable so a zram based improvement is able to reduce waste of resources by 5 to 10 times.

 

That would be cool, yes. Now how to implement it most sane way?

Share this post


Link to post
Share on other sites
1 hour ago, Igor said:

Now how to implement it most sane way?

 

Well, switching from tmpfs to an already existing zram block device that has been prepared before is most probably the easy part.

  • Prerequisit to use such a zram device would be refactoring at least armhwinfo (so that armhwinfo eventually again only does what the name tells: collecting and logging information). All the stuff that configures/optimizes should end up in own services that can be configured in a sane way so that user changes do not get lost with next update. No idea how to do that though
  • Then the whole zram idea needs a solution since in case we reinvent the wheel and create zram devices on our own (which seems like the only way to get zram for logs to me) we need to deal with the existence of other zram related services (e.g. zram-control package in Ubuntu)
  • And I fear the part that needs most attention is logrotate and how to interfere with. Moving old/compressed logs into RAM is no good idea at all, also moving logs compressed by logrotate from RAM to 'disk' and free RAM afterwards seems like the only sane way to me. But I've no idea how this could evolve without reinventing the wheel again and implement logrotate functionality on our own now being aware of two filesystems to use instead of one.
  • Maybe the use of symlinks helps? The initial sync from storage to RAM does not copy old/compressed contents but creates them as symlinks to the /var/log.hdd/ variant. And then a cronjob monitors /var/log and as soon as there appear rotated logs they're moved over to /var/log.hdd and are replaced with symlinks. Active logfiles are treated differently and simply rsynced on a per file basis to /var/log.hdd

Share this post


Link to post
Share on other sites
1 hour ago, tkaiser said:

All the stuff that configures/optimizes should end up in own services that can be configured in a sane way so that user changes do not get lost with next update. No idea how to do that though


Split optimizations part into /etc/init.d/armhwopti with optional configuration in /etc/default/armhwopti ?

 

1 hour ago, tkaiser said:

we need to deal with the existence of other zram related services


This way as it is now in activate_zram() we don't need any 3rd party utility, right? And it should work in Debian and Ubuntu where a kernel has supports ZRAM?

 

1 hour ago, tkaiser said:

Moving old/compressed logs into RAM is no good idea at all, also moving logs compressed by logrotate from RAM to 'disk' and free RAM afterwards seems like the only sane way to me. But I've no idea how this could evolve without reinventing the wheel again and implement logrotate functionality on our own now being aware of two filesystems to use instead of one.


I agree with compressing something that is already compressed is stupid. Logrotate has some interesting options (olddir directory: Logs are moved into directory for rotation.). I will examine if they can be used for this purpose.

 

 

Share this post


Link to post
Share on other sites
11 minutes ago, Igor said:

Split optimizations part into /etc/init.d/armhwopti with optional configuration in /etc/default/armhwopti ?

 

If I understood @zador.blood.stained correctly he would prefer splitting this up in a more granular way. Currently we have the following functions in the script (the ones that can remain in armhwinfo marked with an X):

collect_information X
set_io_scheduler
prepare_temp_monitoring X
prepare_board
log_hardware_info X
get_flash_information X
show_motd_warning X
check_sd_card_speed X
add_usb_storage_quirks
activate_zram

So prepare_board, set_io_scheduler and add_usb_storage_quirks could go into an armbiansetuphw service and zram could be handled by armbianzram (I really would prefer to prefix all our services with armbian so users get a clue what's plain Ubuntu/Debian and what's Armbian).

 

17 minutes ago, Igor said:

This way as it is now in activate_zram() we don't need any 3rd party utility, right?

Yes, but please see @zador.blood.stained's objections. I'm fine with any way as long as it's configurable and able to be disabled. The log2ram functionality would then have to implement a fallback mechanism to what we use today (check for /dev/zram0 whether it contains a freshly created btrfs filesystem. If yes use it, otherwise do the fallback to tmpfs)

 

21 minutes ago, Igor said:

I agree with compressing something that is already compressed is stupid

It's worse also for another reason: Recompressing already compressed stuff usually wastes more storage space. An uncompressed 10 MB log might end up being a 1 MB .gz file or as uncompressed file using 1.5 MB DRAM on an lzo zram device. But the 1 MB gzip version might need +2 MB on the same lzo compressed zram device. So it's not just a waste of CPU cycles but also inefficient DRAM use.

Share this post


Link to post
Share on other sites
10 minutes ago, Tido said:

Is there anything going on with regard to the log file issue ?

Do you need some help ?


Log issue is somehow solved, but not implemented. We need this major BSP scripting rework, splitting first. What @tkaiser already exposed.

 

I read objections, have things in my mind at this stage but haven't moved forward with this. Waiting for the right moment to make the first move :)

Share this post


Link to post
Share on other sites
Quote

Zador: I would prefer to split it into multiple services: a logging service, an optimizing service, a hardware monitoring service, etc.

 

1 hour ago, Igor said:

this goes in the right direction?

For my limited understanding, are these the new 'modules' ?

Will this script upgrade installations with the new 'modules' or is it for fresh install only ?

If it is for fresh install only, will it break current installations ?

armbian-firstrun.service 
armbian-firstrun-config.service 
armbian-optimize-hardware.service 
armbian-ram-logging.service 
armbian-resize-filesystem.service 
armbian-setup-hardware.service

 

 

How can I update the:    fix the basic - CONTRIBUTING.md

Quote

Our tools to control and setup armbian
armhwinfo
armbiansetuphw
armbianzram
armbian-config
armbianmonitor
scripts
configuration

 

 

 

Share this post


Link to post
Share on other sites
7 minutes ago, Tido said:

Will this script upgrade installations with the new 'modules' or is it for fresh install only ?

 

On update, old scrips are (should be) removed and replaced with new which are at the new location. Next step is to divide functions as @tkaiser and @zador.blood.stained proposed and clean up.

 

12 minutes ago, Tido said:

How can I update the:  


It's too soon and things can be twisted around ... after it is merged into master branch.

Share this post


Link to post
Share on other sites
On 6/12/2018 at 10:09 PM, Igor said:

Next step is to divide functions as @tkaiser and @zador.blood.stained proposed and clean up.

 

My 2 cents on this at the bottom of the list: https://github.com/armbian/build/pull/1015

 

Can't really test/review at the moment since running out of spare time. My only other 'Armbian contribution' this week will be a little bit NanoPC T4 testing (relying on @hjc's work).

Share this post


Link to post
Share on other sites
17 minutes ago, tkaiser said:

My 2 cents on this at the bottom of the list: https://github.com/armbian/build/pull/1015

 

Thanks.

 

O.K. I will do few more tests with your changes and then merge in. It was tested well, but we all know how this goes.

 

Let's point out for more profound testing when changes reach beta repository. We need to make sure upgrading won't break, while for new images I am not worried that much.


One a bit radical idea, which crossed on my mind - do we actually need a swap file enabled by default?
 

In general, features of the old armhwinfo scripts weren't changed much while the change logic was hopefully met discussion above.

Share this post


Link to post
Share on other sites
2 hours ago, Tido said:

Can the community somehow help testing or is it very complicated?


Merged. It will be in the nightly images in a few hours from now. Feel free to call for action. Changes are written in the commit header and from that, it can be extracted what we should pay attention most.

Share this post


Link to post
Share on other sites
24 minutes ago, Tido said:

reading about problems, does the script add it to /etc/fstab ?


No need. We have a script for adding zram swap. Are thee still any problems?

Share this post


Link to post
Share on other sites

Using armbian 5.51 debian stretch I found the following errors in the system log from the reboot after a system crash. At least Apache failed to start.

 

root@hc1:/var/log# journalctl  | grep "/var/log"
Jul 09 12:13:16 hc1 apachectl[970]: (2)No such file or directory: AH02291: Cannot access directory '/var/log/apache2/' for main error log
Jul 09 12:13:16 hc1 apachectl[970]: (2)No such file or directory: AH02291: Cannot access directory '/var/log/apache2/' for error log of vhost defined at /etc/apache2/sites-enabled/nextcloud.conf:7
Jul 09 12:13:16 hc1 apachectl[970]: (2)No such file or directory: AH02291: Cannot access directory '/var/log/apache2/' for error log of vhost defined at /etc/apache2/sites-enabled/default-ssl.conf:2
Jul 09 12:13:16 hc1 apachectl[970]: (2)No such file or directory: AH02291: Cannot access directory '/var/log/apache2/' for error log of vhost defined at /etc/apache2/sites-enabled/000-default.conf:1
Jul 09 12:13:21 hc1 nmbd[1414]: mkdir failed on directory /var/log/samba/cores: No such file or directory
Jul 09 12:13:21 hc1 nmbd[1414]: Failed to create /var/log/samba/cores for user 0 with mode 0700
Jul 09 12:13:21 hc1 nmbd[1414]:   Unable to open new log file '/var/log/samba/log.': No such file or directory
Jul 09 12:13:21 hc1 nmbd[1414]:   Unable to open new log file '/var/log/samba/log.nmbd': No such file or directory
Jul 09 12:13:22 hc1 smbd[1439]:   Unable to open new log file '/var/log/samba/log.smbd': No such file or directory
Jul 09 12:13:22 hc1 smbd[1439]:   Unable to open new log file '/var/log/samba/log.smbd': No such file or directory
Jul 09 12:13:22 hc1 smbd[1439]:   Unable to open new log file '/var/log/samba/log.smbd': No such file or directory

 

Could you please add samba and apache2 log directory to the function RecreateLogs in /usr/lib/armbian/armbian-ramlog:

 

RecreateLogs (){
        # in case of crash those services doesn't start if there are no dirs & logs
        [ ! -d /var/log/proftpd ] && mkdir -p /var/log/proftpd
        [ ! -d /var/log/nginx ] && mkdir -p /var/log/nginx
	[ ! -d /var/log/apache2 ] && mkdir -p /var/log/apache2
	[ ! -d /var/log/samba ] && mkdir -p /var/log/samba
        [ ! -f /var/log/proftpd/controls.log ] && touch /var/log/proftpd/controls.log
        [ ! -f /var/log/nginx/access.log ] && touch /var/log/nginx/access.log
        [ ! -f /var/log/nginx/error.log ] && touch /var/log/nginx/error.log
}

 

Probably it would be a good idea to schedule a ramlog syncToDisk once per day using an systemd timer?

 

Thanks & keep up the good work.

Tobias

Share this post


Link to post
Share on other sites
1 hour ago, tpm8 said:

Could you please add samba and apache2 log directory to the function RecreateLogs in /usr/lib/armbian/armbian-ramlog

 

[x] done

 

I agree on the need for a regular resync to 'disk' and wonder whether it's possible to make it somewhat easy configurable (e.g. defaulting to 1440 minutes which can be overwritten by setting SYNC_INTERVAL=60 in /etc/default/armbian-ramlog)?

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
0