dpkg: cannot write to log file '/var/log/dpkg.log': No space left on device

[Robert-MX]

Armbianmonitor:

http://ix.io/3NB6

Hi, the system seems working normally, but when installing with apt, I get plenty of errors of:

dpkg: cannot write to log file '/var/log/dpkg.log': No space left on device

Digging into /var/log reveals that the ./journal directory is responsible; with 48 Mb.

Any idea how to prevent this? Currently, I just remove the directory below ./journal.

[Werner]

Use journalctl to get an idea what is filling up your logs.

[Robert-MX]

20 hours ago, Werner said:

Use journalctl to get an idea what is filling up your logs.

 

journalctl -b

 

gives

-- Journal begins at Tue 2022-01-25 15:00:00 CST, ends at Thu 2022-01-27 10:18:46 CST. --
Jan 27 10:12:45 lepotato rsyslogd[2485]: action 'action-11-builtin:omfile' (module 'builtin:omfile') message l>
Jan 27 10:12:45 lepotato rsyslogd[2485]: file '/var/log/messages'[10] write error - see https://www.rsyslog.co>
Jan 27 10:12:45 lepotato rsyslogd[2485]: action 'action-11-builtin:omfile' (module 'builtin:omfile') message l>
Jan 27 10:12:45 lepotato rsyslogd[2485]: file '/var/log/syslog'[7] write error - see https://www.rsyslog.com/s>
Jan 27 10:12:45 lepotato rsyslogd[2485]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lo>
Jan 27 10:12:45 lepotato rsyslogd[2485]: file '/var/log/syslog'[7] write error - see https://www.rsyslog.com/s>
Jan 27 10:12:45 lepotato rsyslogd[2485]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lo>
Jan 27 10:12:45 lepotato rsyslogd[2485]: file '/var/log/syslog'[7] write error - see https://www.rsyslog.com/s>
Jan 27 10:12:45 lepotato rsyslogd[2485]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lo>
Jan 27 10:12:45 lepotato rsyslogd[2485]: file '/var/log/syslog'[7] write error - see https://www.rsyslog.com/s>
Jan 27 10:12:45 lepotato rsyslogd[2485]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lo>
Jan 27 10:12:45 lepotato rsyslogd[2485]: file '/var/log/syslog'[7] write error - see https://www.rsyslog.com/s>
Jan 27 10:12:45

 

etc. 

I installed a program, and now, strangely, the /var/log seems fine again:

lepotato:default:# df 
Filesystem     1K-blocks    Used Available Use% Mounted on
udev              862232       0    862232   0% /dev
tmpfs             200976    8876    192100   5% /run
/dev/mmcblk1p1  29573100 8552420  19493080  31% /
tmpfs            1004880   86176    918704   9% /dev/shm
tmpfs               5120       4      5116   1% /run/lock
tmpfs            1004880    4652   1000228   1% /tmp
/dev/zram1         49560    7160     38816  16% /var/log
tmpfs             200976   23588    177388  12% /run/user/1000

I found the following issue 

Finally, the armbian user switched back to a Buster Image...

Does everyone know, if the /var/log/ issue was fixed, or how to prevent this error?

[Joey Pongallo]

I have the same issue as well.

[tyfui90]

Based on the related forum post by @Robert-MX, I was able to resolve the "file '/var/log/syslog'[7] write error - see https://www.rsyslog.com/s>" spamming the journalctl logs after a reboot due to the logs not being cleared properly.

 

The root issue appears to be that /usr/lib/armbian/armbian-truncate-logs has an issue when ran with as #!/bin/sh after a reboot. To get around that it manually with bash instead.

To fix it temporarily until the next reboot then just run 

sudo bash /usr/lib/armbian/armbian-truncate-logs

check 'journalctl -f' to see if the rsyslog issues disappear, if so then you'll want to add it to your crontab for every boot

 

To fix it permanently then add the command to your crontab so it runs every time you boot:

switch to root

su -

edit your crontab

crontab -e

add the line at the bottom so that the log truncate happens after each boot

@reboot bash /usr/lib/armbian/armbian-truncate-logs

reboot and check journalctl -f to see if you are still seeing the rsyslog issues

 

 

[haajee]

Got the same error today. And i think the problem is PiHole in my situation. After running "pihole flush logs" it's ok for a short time. The log file was arround  50MB big. Ok it's big but i have more than 100GB free?

 

Edited March 30, 2023 by haajee

[Werner]

8 hours ago, haajee said:

more than 100GB free?

On your sd/eMMC/NVMe/whatever maybe but not on your ramlog device

 

As suggested long time ago check what is filling your logs and fix. 50M of compressed(!) text is a LOT.

You can increase its size but this will also reduce the overall free memory for other applications. check in /etc/default/ here.

Or disable ramlog which will write logs directly to your disk. However when using an sdcard this will significantly short its lifetime.

 

As for me I run pihole for years and never had an issue with filling up logs.

[haajee]

I don't know. I have just installed standard pihole without strange things. There are also not very much queries or things. Now disabled armbian-ramlog and works ok but it's more nice to enable. BTW my whole system crashed almost with the full /var/log..

[armdran]

The problem happens easily when there is written a lot to /var/log.

This can even corrupt your system when an apt dist-upgrade fails right in the middle due to lack of space in /var/log.

The problem is due to armbian ramlog on devices with little ram, like older raspberry or banana pis.

The easiest way to solve it, is to disable ramlog in /etc/default/armbian-ramlog and zram in /etc/default/armbian-zram-config.

Be aware though that this makes heavier use of your SD card, though.

I usually solve the problem by attaching a USB device (preferably an SSD) for swap, /var/log and /tmp or by using a large SD card which has plenty of spare blocks.

Edited September 21, 2023 by armdran

[classman]

What I did is:
 

df -h

to see the space used for "/var/log"

 

Then I went to "/var/log" and ran:
 

sudo du -ha | grep 'G'; sudo du -ha | grep '[0-9][0-9][0-9]M'

 

(Im lazy and the 'G' will return folders or logs with the letter 'G' as well)

 

This will show you which files are bigger than 999MB and which files are bigger than 99MB.

 

After seeing that "/var/log/dpkg.log" was not the problem, for me it was "daemon.log" , I decided to erase all of the February entries on that log, since there were a lot of entries from when I was setting something up. (Junk)

sudo sed -i '/^Feb/d' daemon.log

 

And finally (I want to say Im a novice linux user, so I dont know if this is recommended) I deleted the zipped (.gz) old logs that Im sure that I don't need.