0
guidol

OpiOne: /dev/zram0 = /var/log full while apt update

Recommended Posts

Armbianmonitor:

while updating my dev-OPiOne

Welcome to Armbian bullseye with Linux 5.5.8-sunxi

package bsp-kernel[20.05.0-trunk] u-boot[20.05.0-trunk] dtb   [20.05.0-trunk]
firmware          [20.05.0-trunk] config[20.05.0-trunk] branch[dev]

Linux opi-one 5.5.8-sunxi #trunk SMP Wed Mar 18 21:19:37 +03 2020 armv7l GNU/Linux

via apt update I did get the follwing error-message:

Trigger für mime-support (3.64) werden verarbeitet ...
E: Schreibfehler - ~LZMAFILE (28: Auf dem Gerät ist kein Speicherplatz mehr verfügbar)

I did found out via df that /dev/zram0 = /var/log is used up to 100% :(

root@opi-one(192.168.6.114):~# df
Dateisystem            1K-Blöcke Benutzt Verfügbar Verw Eingehängt auf
/dev/zram0                 49584   48560         0  100% /var/log

but only zram0 seems to be full and not the complete memory:

MiB Mem :    491,5 total,    124,5 free,    101,3 used,    265,8 buff/cache
MiB Swap:    245,8 total,    242,8 free,      3,0 used.    360,8 avail Mem

After a reboot there is also 64% of the /dev/zram0 used and I dont know why :(
 

root@opi-one(192.168.6.114):~# df
Dateisystem            1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
/dev/zram0                 49584   28996     17004   64% /var/log

 

Spoiler

root@opi-one(192.168.6.114):/var/log# ls -l
insgesamt 264
-rw-r--r-- 1 root     root                0 Apr  4 19:06 alternatives.log
drwxr-xr-x 2 root     root             4096 Apr  4 19:09 apt
-rw-r--r-- 1 root     root            36237 Apr  4 19:14 armbian-hardware-monitor.log
-rw-r----- 1 root     adm              2188 Apr  4 19:17 auth.log
-rw-r--r-- 1 root     root                0 Apr  4 19:00 bootstrap.log
-rw-rw---- 1 root     utmp                0 Apr  4 19:00 btmp
drwxr-xr-x 2 _chrony  _chrony          4096 Okt 26 09:47 chrony
-rw-r----- 1 root     adm             20547 Apr  4 19:15 daemon.log
-rw-r----- 1 root     adm               607 Apr  4 19:14 debug
-rw-r--r-- 1 root     root                0 Apr  4 19:09 dpkg.log
-rw-r--r-- 1 root     root                0 Apr  4 19:00 faillog
drwxr-sr-x 3 root     systemd-journal  4096 Feb 25 19:02 journal
-rw-r----- 1 root     adm             35489 Apr  4 19:14 kern.log
-rw-rw-r-- 1 root     utmp              292 Apr  4 19:15 lastlog
drwxr-x--- 2 www-data www-data         4096 Mär 18 21:28 lighttpd
drwx------ 2 root     root            16384 Apr  4 19:14 lost+found
-rw-r----- 1 root     adm             34597 Apr  4 19:14 messages
drwxr-xr-x 2 pihole   pihole           4096 Feb 14 14:53 pihole
-rw-r--r-- 1 pihole   pihole           4052 Apr  4 19:14 pihole-FTL.log
-rw-r--r-- 1 pihole   pihole            706 Apr  4 19:14 pihole.log
-rw-r--r-- 1 root     root                0 Apr  4 19:00 pihole_updateGravity.log
drwx------ 2 root     root             4096 Dez 31 13:35 private
drwxr-xr-x 3 root     root             4096 Mär 18 21:28 runit
-rw-r----- 1 root     adm             57029 Apr  4 19:17 syslog
drwxr-xr-x 2 root     root             4096 Nov  2 13:55 sysstat
drwxr-x--- 2 root     adm              4096 Mär 18 21:28 unattended-upgrades
-rw-rw-r-- 1 root     utmp             2688 Apr  4 19:15 wtmp

 

Is there any chance to gain some space in /dev/zram0 or to configure a little bigger /dev/zram0?
(to use a part of the 265Mb remaining cache/buffer-ram)

 

The complete update to newer packages/kernel seems also to work with the full /dev/zram = /var/log:

Welcome to Armbian bullseye with Linux 5.5.15-sunxi

package bsp-kernel[20.05.0-trunk] u-boot[20.05.0-trunk] dtb   [20.05.0-trunk]
firmware          [20.05.0-trunk] config[20.05.0-trunk] branch[dev]

Linux opi-one 5.5.15-sunxi #trunk SMP Sat Apr 4 17:39:03 +03 2020 armv7l GNU/Linux

 

 

 

Share this post


Link to post
Share on other sites

Now I have 90% in /var/log after a reboot :(

df
Dateisystem            1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
/dev/zram0                 49584   41296      4704   90% /var/log

zramctl
NAME       ALGORITHM DISKSIZE  DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram1 lzo         245,8M    4K   78B   12K       4 [SWAP]
/dev/zram0 zstd           50M 40,4M    1M  1,5M       4 /var/log

root@opi-one(192.168.6.114):/var/log/journal/14bb32cb9a1c40b29c1696d4acec6931# ls -l
insgesamt 40976
-rw-r----- 1 root systemd-journal 16777216 Apr  4 19:14 system@0005a27952ce7c96-f8789ed6a81b97f9.journal~
-rw-r----- 1 root systemd-journal  8388608 Apr  4 22:45 system@0005a27c483373ab-1b9f47dd2956de8e.journal~
-rw-r----- 1 root systemd-journal  8388608 Apr  4 22:56 system@0005a27c6cce90a5-1036e2f49bee28e6.journal~
-rw-r----- 1 root systemd-journal  8388608 Apr  4 23:18 system.journal

but I can't read the content of the system.journal-files :(
[EDIT] here are instructions for reading journal files:
https://www.linux-magazin.de/ausgaben/2016/02/systemd-journal/2/

If I delete these
files - they will be "restored" at the next boot :(

Share this post


Link to post
Share on other sites
9 minutes ago, Igor said:

I was fixed - how are your values in /etc/systemd/journald.conf

https://github.com/armbian/build/commit/2f389a6cb6944983bbbaba15ef2b3cba228e9a05

There I only have the default-file-entrys:
 

Spoiler

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg

 

I did now edit my /usr/lib/armbian/armbian-zram-config and inserted the lines from the fix at the right place (which were missing).
The fix inserted/corrected the SystemMaxUse=20M in /etc/systemd/journald.conf

 

Before a reboot I did enter a "journalctl --vacuum-size=5M"

 

And after the reboot I have:

root@opi-one(192.168.6.114):~# df
Dateisystem            1K-Blöcke Benutzt Verfügbar Verw% Eingehängt auf
/dev/zram0                 49584   19676     26324   43% /var/log

Many thanks at @Igor ;)
 

 

Share this post


Link to post
Share on other sites
On 4/4/2020 at 11:35 PM, Igor said:

I was fixed - how are your values in /etc/systemd/journald.conf

https://github.com/armbian/build/commit/2f389a6cb6944983bbbaba15ef2b3cba228e9a05

@Igor I have two bullseye testing sbcs which seem to ignore the setting
SystemMaxUse=20M in /etc/systemd/journald.conf

var/log get full to 100% only on these bullseye sbcs but not on the buster sbcs

 

both show 
System diagnosis information will now be uploaded to gzip: /var/log/armbian-hardware-monitor.log.1.gz:
No such file or directory

while armbianmonitor -u (OK, I did a journalctl --vacuum-size=5M shortly  ago)

 

NPi Neo2 with bullseye:
http://ix.io/2lY7
NanoPi K1 Plus with bullseye:
http://ix.io/2lY8

Maybe there is to less output from armbianmonitor because of the vacuum?

Share this post


Link to post
Share on other sites

This looks strange for me (here the "red" ones)  - a short part from journalctl -b
but sometimes there are messages which arent red:

 

May 14 11:56:14 npi-neo2-24 rsyslogd[758]: file '/var/log/syslog'[7] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2002.0 try https://www.rsyslog.com/e/20>
May 14 11:56:14 npi-neo2-24 rsyslogd[758]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2002.0 try https://www.rsyslog>
May 14 11:56:14 npi-neo2-24 rsyslogd[758]: file '/var/log/syslog'[7] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2002.0 try https://www.rsyslog.com/e/20>
May 14 11:56:14 npi-neo2-24 rsyslogd[758]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2002.0 try https://www.rsyslog>
May 14 11:56:14 npi-neo2-24 rsyslogd[758]: file '/var/log/syslog'[7] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2002.0 try https://www.rsyslog.com/e/20>

 

here are some "normal green" ones from the journal:

May 14 11:45:06 npi-neo2-24 CRON[416455]: pam_unix(cron:session): session closed for user root
May 14 11:55:01 npi-neo2-24 CRON[428664]: pam_unix(cron:session): session opened for user root by (uid=0)
May 14 11:55:01 npi-neo2-24 CRON[428665]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

 

Share this post


Link to post
Share on other sites
1 hour ago, guidol said:

I have two bullseye testing sbcs which seem to ignore the setting


Good to know, thanks. ... I also noticed some other problems related to haveged, which requires apparmor and when that is enabled it still doesn't work. Perhaps Bullseye is too fragile atm and since it will remain unsupported (at this release), this is low priority to fix atm

Share this post


Link to post
Share on other sites
2 hours ago, Igor said:


Good to know, thanks. ... I also noticed some other problems related to haveged, which requires apparmor and when that is enabled it still doesn't work. 

I will have an (bulls)eye on it - because I had the kernel freezed to 5.6.8 and defreezed it today.

Now - after the update - I have 5.6.12 and since 2 hours /var/lag % hasnt grown.
Maybe ist has been solved in 5.6.12 or the newer packages since 5.6.8

Share this post


Link to post
Share on other sites
27 minutes ago, guidol said:

Maybe ist has been solved in 5.6.12 or the newer packages since 5.6.8


Anything is possible when using bleeding edge stuff ;) 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
0