y52 Posted February 6, 2022 Posted February 6, 2022 2 hours ago, y52 said: ** File not found /boot.scr ** <<< ==== ## Executing script at 06d00000 Wrong image format for "source" command <<< ==== For those, who still rely on Armbian's u-boot build, I've found the origin of the logs error above. The armbian setenv instructions on https://www.armbian.com/espressobin/ page has 1 line which should be corrected: setenv boot_prefixes '/ /boot/' It has 1 slash in excess. and read " setenv boot_prefixes '/boot/' " or be corrected directly in u-boot : Marvell>> setenv boot_prefixes '/boot/' Marvell>> echo $boot_prefixes Marvell>> saveenv I suggest that Armbian corrects it on their site as well. 0 Quote
Smurfix Posted February 6, 2022 Posted February 6, 2022 Well, it says "prefixes", plural, so why shouldn't that variable contain both / and /boot/? The former for when you have a dedicated boot partition, the latter if you don't. 0 Quote
y52 Posted February 6, 2022 Posted February 6, 2022 Thanks, Smurfix. It looks like a cosmetic fix for me, which is still welcome. The other issue : the system doesn't shut down completely : shutdown -h now produces: .. .. [ OK ] Stopped Create System Users. [ OK ] Stopped Remount Root and Kernel File Systems. [ OK ] Reached target Shutdown. [ OK ] Deactivated swap /dev/zram0. [ OK ] Reached target Unmount All Filesystems. [ OK ] Reached target Final Step. [ OK ] Finished Power-Off. [ OK ] Reached target Power-Off. [ 7768.040643] reboot: Power down ERROR: a3700_system_off need PANIC at PC : 0x0000000004023268 and the LED remains on. Is that the board concept or the build problem? 0 Quote
y52 Posted February 12, 2022 Posted February 12, 2022 /var/log fills up too quickly resulting in Archived and active journals take up 48.6M in the file system. Journal file /var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005ca730c166775-bedeb0d5b1b229d8.journal~ is truncated, ignoring file. root@tiger:~# cat /etc/default/armbian-ramlog # configuration values for the armbian-ram-logging service # # enable the armbian-ram-logging service? ENABLED=true # # size of the tmpfs mount -- please keep in mind to adjust /etc/default/armbian-zram-config too when increasing #SIZE=50M SIZE=100M Increasing tmpfs mount size to 100M doesn't increase the /dev/zram1 : Filesystem Size Used Avail Use% Mounted on tmpfs 99M 0 99M 0% /run/user/0 /dev/zram1 49M 48M 0 100% /var/log What other config should be tweaked ? Otherwise, what other measures could be conceived so that /var/log is not saturated as quickly as it does now? 0 Quote
Heisath Posted February 12, 2022 Posted February 12, 2022 Not sure on the root cause, but increasing size should be enough, atleast it is for me. Maybe related to this: https://github.com/armbian/build/pull/2224 root@wdnas:~# cat /etc/default/armbian-ramlog # configuration values for the armbian-ram-logging service # # enable the armbian-ram-logging service? ENABLED=true # # size of the tmpfs mount -- please keep in mind to adjust /etc/default/armbian-zram-config too when increasing SIZE=100M # # use rsync instead of cp -r # requires rsync installed, may provide better performance # due to copying only new and changed files USE_RSYNC=true root@wdnas:~# cat /etc/default/armbian-zram-config # configuration values for the armbian-zram-config service # # enable the armbian-zram-config service? ENABLED=true # percentage of zram used as swap compared to physically available DRAM. # Huge overcommitment (300) is possible and sometimes desirable. See # https://forum.armbian.com/topic/5565-zram-vs-swap/?do=findComment&comment=61082 # and don't forget to adjust $MEM_LIMIT_PERCENTAGE below too. # ZRAM_PERCENTAGE=50 # percentage of DRAM available to zram. If this amount is exceeded the zram # devices used for swap simply behave as if the device is full. You need to # adjust/increase this value only if you want to work with massive memory # overcommitment (ZRAM_PERCENTAGE exceeding 150 for example) # MEM_LIMIT_PERCENTAGE=50 # create how many zram devices max for swap # ZRAM_MAX_DEVICES=4 # Which algorithm for zram based swapping. Seems lzo is best choice on ARM: # https://forum.armbian.com/topic/8161-swap-on-sbc/?do=findComment&comment=61668 # SWAP_ALGORITHM=lzo # Which algorithm to choose for zram based ramlog partition # RAMLOG_ALGORITHM=zstd # Which algorithm to choose for zram based /tmp # TMP_ALGORITHM=zstd # If defined a separate partition will be used as zram backing device. Be CAREFUL # which partition you assign and read starting from CONFIG_ZRAM_WRITEBACK in # https://www.kernel.org/doc/Documentation/blockdev/zram.txt # ZRAM_BACKING_DEV=/dev/nvme0n2 root@wdnas:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/sdb2 15G 1.1G 13G 8% / devtmpfs 248M 0 248M 0% /dev tmpfs 249M 0 249M 0% /dev/shm tmpfs 100M 3.0M 97M 3% /run tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 249M 0 249M 0% /tmp /dev/sdb1 100M 9.7M 91M 10% /boot /dev/sda1 2.7T 1.7T 870G 67% /mnt/hd-intern /dev/zram0 97M 84M 6.5M 93% /var/log tmpfs 50M 0 50M 0% /run/user/0 0 Quote
y52 Posted February 13, 2022 Posted February 13, 2022 18 hours ago, Heisath said: ncreasing size should be enough, atleast it is for me You /var/log is filled up to 93%, which is close to saturation, leaving no space for exceptional logs. 18 hours ago, Heisath said: Maybe related to this: https://github.com/armbian/build/pull/2224 The latest Armbian release has apparently applied this fix : What I have in my /usr/lib/armbian/armbian-truncate-logs JOURNAL_SIZE=5M # size to shrink systemd-journal ... # remove /usr/bin/find /var/log -name '*.[0-9]' -or -name '*.gz' | xargs -r rm -f # vacuum systemd-journald [ -d /var/log/journal ] && journalctl --vacuum-size=${JOURNAL_SIZE} Anyway, armbian-truncate-logs is executed at midnight and it creates numerous 2.5 Mb files -rw-r----- 1 root systemd-journal 2.5M Feb 10 16:31 system@0005d7aba5022919-feb5e759559c3bb8.journal~ -rw-r----- 1 root systemd-journal 2.5M Feb 10 16:34 system@0005d7abb08cc9be-b604a49ab52aa0fc.journal~ -rw-r----- 1 root systemd-journal 2.5M Feb 10 16:41 system@0005d7abc90e2e8b-fffdcbf272568db6.journal~ -rw-r----- 1 root systemd-journal 2.5M Feb 10 16:43 system@0005d7abcf62f90f-03cf52d5f085f7ea.journal~ -rw-r----- 1 root systemd-journal 2.5M Feb 10 16:51 system@0005d7abebf00786-f6350b33e3c2d8c2.journal~ -rw-r----- 1 root systemd-journal 2.5M Feb 10 17:12 system@0005d7ac371cdacc-92de5633f876fefe.journal~ -rw-r----- 1 root systemd-journal 2.5M Feb 10 18:07 system@0005d7acfb3a9715-ed48625431c60e14.journal~ -rw-r-----+ 1 root systemd-journal 2.5M Feb 10 22:38 system.journal -rw-r--r-- 1 root systemd-journal 2.5M Feb 6 16:50 user-1000.journal This saturates completely allocated log space. This needs to be fixed. 0 Quote
y52 Posted February 14, 2022 Posted February 14, 2022 This is what happens every midnight after armbian-ramlog is triggered : Feb 14 00:00:06 tiger systemd[1]: Starting system activity accounting tool... Feb 14 00:00:07 tiger systemd[1]: Starting Rotate log files... Feb 14 00:00:07 tiger systemd[1]: Starting Daily man-db regeneration... Feb 14 00:00:07 tiger systemd[1]: sysstat-collect.service: Succeeded. Feb 14 00:00:07 tiger systemd[1]: Finished system activity accounting tool. Feb 14 00:00:07 tiger armbian-ramlog[16480]: Mon 14 Feb 2022 12:00:07 AM CET: Syncing logs to storage Feb 14 00:00:07 tiger armbian-ramlog[16483]: sending incremental file list Feb 14 00:00:07 tiger armbian-ramlog[16483]: ./ Feb 14 00:00:07 tiger armbian-ramlog[16483]: armbian-hardware-monitor.log Feb 14 00:00:07 tiger armbian-ramlog[16483]: auth.log Feb 14 00:00:07 tiger armbian-ramlog[16483]: btmp Feb 14 00:00:07 tiger armbian-ramlog[16483]: daemon.log Feb 14 00:00:07 tiger armbian-ramlog[16483]: debug Feb 14 00:00:07 tiger armbian-ramlog[16483]: dpkg.log Feb 14 00:00:07 tiger armbian-ramlog[16483]: faillog Feb 14 00:00:07 tiger armbian-ramlog[16483]: kern.log Feb 14 00:00:07 tiger armbian-ramlog[16483]: lastlog Feb 14 00:00:07 tiger armbian-ramlog[16483]: messages Feb 14 00:00:07 tiger armbian-ramlog[16483]: syslog Feb 14 00:00:07 tiger armbian-ramlog[16483]: user.log Feb 14 00:00:07 tiger armbian-ramlog[16483]: wtmp Feb 14 00:00:07 tiger armbian-ramlog[16483]: apt/ Feb 14 00:00:07 tiger armbian-ramlog[16483]: apt/eipp.log.xz Feb 14 00:00:07 tiger armbian-ramlog[16483]: apt/history.log Feb 14 00:00:07 tiger armbian-ramlog[16483]: apt/term.log Feb 14 00:00:12 tiger CRON[16451]: pam_unix(cron:session): session closed for user root Feb 14 00:00:12 tiger CRON[16451]: pam_unix(cron:session): session closed for user root Feb 14 00:00:14 tiger armbian-ramlog[16557]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16558]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16559]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16560]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16562]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16563]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16565]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16566]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16567]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16570]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16571]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16572]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16575]: cat: write error: No space left on device Feb 14 00:00:14 tiger armbian-ramlog[16580]: cat: write error: No space left on device Feb 14 00:00:14 tiger systemd[1]: logrotate.service: Succeeded. Feb 14 00:00:14 tiger systemd[1]: Finished Rotate log files. Feb 14 00:00:14 tiger systemd[1]: logrotate.service: Consumed 5.363s CPU time. 0 Quote
lampra Posted February 14, 2022 Posted February 14, 2022 In my case, not on espressobin, I doubled zram and still had these errors. Then I had to instruct journal to a max (eg. 10-20 mb) and the errors stopped. ps: @y52 did you try u-boot from openwrt for espressobin? It seems that it supports distroboot so probably vanilla debian out of the box 0 Quote
y52 Posted February 16, 2022 Posted February 16, 2022 On 2/14/2022 at 10:22 PM, lampra said: I had to instruct journal to a max Could you be more specific about which setting you have adjusted? For a moment, I execute past the midnight # journalctl --vacuum-size=10M --rotate but a more sustainable solution would be welcome. On 2/14/2022 at 10:22 PM, lampra said: did you try u-boot from openwrt for espressobin No, I didn't. My first target is to make my Armbian setup operational before proceeding further. 0 Quote
lampra Posted February 16, 2022 Posted February 16, 2022 1 hour ago, y52 said: journalctl --vacuum-size=10M --rotate Sorry, I do not remember exactly. Check at `/etc/systemd/journald.conf` the `SystemMaxUse`. My setting is 20M When using `vacuum-size` Just remember that journal logs are located both at `/var/log/journal` (zram) and `/var/log.hdd/journal` 0 Quote
y52 Posted February 16, 2022 Posted February 16, 2022 5 hours ago, lampra said: Check at `/etc/systemd/journald.conf` the `SystemMaxUse`. My setting is 20M Thanks. I've already adjusted this setting to : #SystemMaxUse=20M SystemMaxUse=15M It doesn't make any effect to my setup, which fills in all available log mount in `/var/log/` (zram). So I don't know how to fix it so far. 5 hours ago, lampra said: Just remember that journal logs are located both at `/var/log/journal` (zram) and `/var/log.hdd/journal` Yes, there are a lot of 1.9M sized systemd-journal files system@b55xxx.journal I wonder if they are accounted for in logs MaxUse size. 0 Quote
lampra Posted February 16, 2022 Posted February 16, 2022 1 hour ago, y52 said: So I don't know how to fix it so far. Maybe you could check if zram size (using: df -h) and journal max limit (using: systemctl status systemd-journald.service ) are honored. Then you could examine the logs and check if something is flooding the journal. That was the case in one of my boards. 0 Quote
y52 Posted February 17, 2022 Posted February 17, 2022 18 hours ago, lampra said: examine the logs and check if something is flooding the journal. As explained earlier, the logs point to the armbian-ramlog, which executes every midnight and saturates the log mount. There is no other reason for the moment and the log mount is half empty: root@tiger:/# df -h Filesystem Size Used Avail Use% Mounted on /dev/zram1 49M 21M 25M 47% /var/log 18 hours ago, lampra said: journal max limit (using: systemctl status systemd-journald.service ) are honored. journal max limit was trimmed down as I wrote before and systemd-journald.service works as expected : root@tiger:/# systemctl status systemd-journald.service ● systemd-journald.service - Journal Service Loaded: loaded (/lib/systemd/system/systemd-journald.service; static) Drop-In: /usr/lib/systemd/system/systemd-journald.service.d └─override.conf Active: active (running) since Sun 2022-02-13 12:18:02 CET; 4 days ago TriggeredBy: ● systemd-journald.socket ● systemd-journald-audit.socket ● systemd-journald-dev-log.socket Docs: man:systemd-journald.service(8) man:journald.conf(5) Main PID: 599 (systemd-journal) Status: "Processing requests..." Tasks: 1 (limit: 1147) Memory: 6.4M CPU: 56.792s CGroup: /system.slice/systemd-journald.service └─599 /lib/systemd/systemd-journald Feb 17 00:01:01 tiger systemd-journald[599]: System Journal (/var/log/journal/50bea5a2341c40588d32c8103dea6e71) is> Warning: journal has been rotated since unit was started, output may be incomplete. 0 Quote
lampra Posted February 17, 2022 Posted February 17, 2022 9 minutes ago, y52 said: root@tiger:/# systemctl status systemd-journald.service From your output, It is not obvious to me, If the limit is honored. Here is my output. It is clearly stated that runtime journal is 22.6M, max 20.1M, 0B free.: root@cubietruck:~# systemctl status systemd-journald.service ● systemd-journald.service - Journal Service Loaded: loaded (/lib/systemd/system/systemd-journald.service; static) Drop-In: /lib/systemd/system/systemd-journald.service.d └─override.conf Active: active (running) since Wed 2022-01-12 17:02:10 EET; 1 months 5 days ago TriggeredBy: ● systemd-journald-audit.socket ● systemd-journald-dev-log.socket ● systemd-journald.socket Docs: man:systemd-journald.service(8) man:journald.conf(5) Main PID: 547 (systemd-journal) Status: "Processing requests..." Tasks: 1 (limit: 4569) Memory: 13.3M CPU: 3min 45.567s CGroup: /system.slice/systemd-journald.service └─547 /lib/systemd/systemd-journald Feb 16 17:11:22 cubietruck systemd-journald[547]: Runtime Journal (/run/log/journal/b1762bb2a9d8431a988c6cac5ff8f356) is 22.6M, max 20.1M, 0B free. Warning: journal has been rotated since unit was started, output may be incomplete. and here is my doubled zram Filesystem Size Used Avail Use% Mounted on /dev/zram1 97M 18M 73M 20% /var/log 0 Quote
y52 Posted February 17, 2022 Posted February 17, 2022 31 minutes ago, lampra said: From your output, It is not obvious to me, If the limit is honored. My output was not full, as it doesn't fit the terminal width. Here is what is extended after the ">" : >al (/var/log/journal/50bea5a2341c40588d32c8103dea6e71) is 15.0M, max 15.0M, 0B free. So, there is clearly a 15.0M, max 15.0M, 0B free. But the df -h shows 25M free: /dev/zram1 49M 21M 25M 47% /var/log I've also adjusted additional setting in /etc/systemd/journald.conf : #RuntimeMaxUse= RuntimeMaxUse=15M 0 Quote
y52 Posted February 17, 2022 Posted February 17, 2022 Explanations in man journald.conf : SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, SystemMaxFiles=, RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize=, RuntimeMaxFiles= Enforce size limits on the journal files stored. The options prefixed with "System" apply to the journal files when stored on a persistent file system, more specifically /var/log/journal. The options prefixed with "Runtime" apply to the journal files when stored on a volatile in-memory file system, more specifically /run/log/journal. The former is used only when /var/ is mounted, writable, and the directory /var/log/journal exists. Probably this setting is more likely to act on /var/log (zram): #RuntimeMaxUse= RuntimeMaxUse=15M 0 Quote
y52 Posted February 17, 2022 Posted February 17, 2022 Once I changed the above setting, Here is a new output : arted Feb 17 19:34:51 tiger systemd-journald[52704]: Runtime Journal (/run/log/journal/50bea5a2341c40588d32c8103dea6e71) is 1.2M, max 15.0M, 13.7M free. Feb 17 19:34:51 tiger systemd-journald[52704]: Time spent on flushing to /var/log/journal/50bea5a2341c40588d32c8103dea6e71 is 2.701ms for 13 entries. Feb 17 19:34:51 tiger systemd-journald[52704]: System Journal (/var/log/journal/50bea5a2341c40588d32c8103dea6e71) is 13.1M, max 15.0M, 1.8M free. 0 Quote
y52 Posted February 19, 2022 Posted February 19, 2022 On 2/17/2022 at 7:33 PM, y52 said: RuntimeMaxUse=15M It doesn't help either. Log mount is saturating VEEERY quickly. While I was installing different services, I was blocked several times with the log mount reaching the limits. The whole armbian-ramlog service needs to be revisited. 0 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.