Don't install log2ram by default

[Nitrax]

The only thing I've had issues with on armbian, and that I often have issues with, is the default log2ram. As soon as you start doing anything that actually logs stuff, like access logs, or container logs and so on, the log space is not enough, even if allocating 500 MB RAM. And when /var/log fills up there are going to be crashes... in all kinds of ways. I have an ansible script that disables this now, but sometimes when doing dist-upgrade, log2ram settings come back (and also zram swap, which is also bad because kubernetes does not like swap, but thats not a disaster at least).

 

So I simply propose to not make log2ram default, unless it can have some way to "spill over" to disk. But what about wear and tear? Well if you are logging enough to wear out the SD card, you are logging more than can fit in log2ram anyway (?).

 

(Fact is, log2ram took down my entire kubernetes cluster, because when /var/log was full on one node, containers started to crash, causing more logs to be written on the other nodes (by a factor of 10-100) causing these to also crash within a few hours timeframe.)

[Igor]

4 hours ago, Nitrax said:

So I simply propose to not make log2ram default, unless it can have some way to "spill over" to disk. But what about wear and tear? Well if you are logging enough to wear out the SD card, you are logging more than can fit in log2ram anyway (?).

 

I understand your troubles, but IMO 99% of Armbian users are not running kubernetes cluster and are happy with this feature enabled by default. It helps a lot - performance wise and with SD card durability. I don't see justifications to change defaults.

 

4 hours ago, Nitrax said:

I have an ansible script that disables this now, but sometimes when doing dist-upgrade, log2ram settings come back

 

Are you ticking right file:
/etc/default/armbian-ramlog ?

 

If disabling is not working properly, fixing it is more than welcome! And probably the right way to fix this problem.

[lanefu]

17 hours ago, Nitrax said:

(Fact is, log2ram took down my entire kubernetes cluster, because when /var/log was full on one node, containers started to crash, causing more logs to be written on the other nodes (by a factor of 10-100) causing these to also crash within a few hours timeframe.)

Out of curiosity what hardware are you running for your k8s cluster?

 

Expanding the log2ram size can help.. and also you could increase the frequency of the cleanup job. /etc/cron.d/armbian-truncate-logs

also make sure your cluster isn't double logging to /var/log and to systemd journal.

[Nitrax]

Sorry for late reply. I had a hiatus with my cluster(s). 

 

I really need the RAM, so I can't increase the size, and I'm using an eMMC chip for storage so there's no issues with performance. But I'm sure you are correct that most users benefit from log2ram. I will also check that I disable log2ram the correct way. Thank you.

 

Regarding the hardware:

I'm practically running k3s (from rancher) on everything I have (with 2+ GB RAM) because it makes life so much simpler. Main ARM cluster is 3 rpi4's and the Odroid N2+ as master. An Odroid HC4 with SSD disks is providing NFS storage for all clusters (through nfs-client-provisioner). The HC4 is also running a single node k3s where I run storage related services such as samba, webdav, docker registry etc. Then I have a Tinkerboard that runs k3s with a few services mostly to test 32-bit arm stuff. I also have a x86 cluster with 40gb RAM, for all the stuff that can't run on ARM or needs too much memory.