Jump to content

Espressobin support development efforts


lanefu

Recommended Posts

2 hours ago, y52 said:

** File not found /boot.scr **                                 <<< ====
## Executing script at 06d00000
Wrong image format for "source" command        <<< ====

 

For those, who still rely on Armbian's u-boot build, I've found the origin of the logs error above.

The armbian setenv instructions on https://www.armbian.com/espressobin/ page has 1 line which should be corrected:

setenv boot_prefixes '/ /boot/'

It has 1 slash in excess. and read " setenv boot_prefixes '/boot/' " or be corrected directly in u-boot :

 

Marvell>> setenv boot_prefixes '/boot/'
Marvell>> echo $boot_prefixes

Marvell>> saveenv

 

I suggest that Armbian corrects it on their site as well.

Link to comment
Share on other sites

Thanks, Smurfix. It looks like a cosmetic fix for me, which is still welcome.

 

The other issue : the system doesn't shut down completely :

shutdown -h now produces: ..

..

[  OK  ] Stopped Create System Users.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Reached target Shutdown.
[  OK  ] Deactivated swap /dev/zram0.
[  OK  ] Reached target Unmount All Filesystems.
[  OK  ] Reached target Final Step.
[  OK  ] Finished Power-Off.
[  OK  ] Reached target Power-Off.

[ 7768.040643] reboot: Power down
ERROR:   a3700_system_off need PANIC at PC : 0x0000000004023268

and the LED remains on.

Is that the board concept or the build problem?

Link to comment
Share on other sites

/var/log fills up too quickly resulting in 

Archived and active journals take up 48.6M in the file system.

Journal file /var/log/journal/50bea5a2341c40588d32c8103dea6e71/system@0005ca730c166775-bedeb0d5b1b229d8.journal~ is truncated, ignoring file.

 

root@tiger:~# cat /etc/default/armbian-ramlog
# configuration values for the armbian-ram-logging service
#
# enable the armbian-ram-logging service?
ENABLED=true
#
# size of the tmpfs mount -- please keep in mind to adjust /etc/default/armbian-zram-config too when increasing
#SIZE=50M

SIZE=100M

 

Increasing  tmpfs mount size to 100M doesn't increase the /dev/zram1 :

Filesystem      Size  Used Avail Use% Mounted on

tmpfs            99M     0   99M   0% /run/user/0
/dev/zram1       49M   48M     0 100% /var/log

 

What other config should be tweaked ?

Otherwise, what other measures could be conceived so that /var/log is not saturated as quickly as it does now?

Link to comment
Share on other sites

Not sure on the root cause, but increasing size should be enough, atleast it is for me. Maybe related to this: https://github.com/armbian/build/pull/2224

 

root@wdnas:~# cat /etc/default/armbian-ramlog
# configuration values for the armbian-ram-logging service
#
# enable the armbian-ram-logging service?
ENABLED=true
#
# size of the tmpfs mount -- please keep in mind to adjust /etc/default/armbian-zram-config too when increasing
SIZE=100M
#
# use rsync instead of cp -r
# requires rsync installed, may provide better performance
# due to copying only new and changed files
USE_RSYNC=true

root@wdnas:~# cat /etc/default/armbian-zram-config
# configuration values for the armbian-zram-config service
#
# enable the armbian-zram-config service?
ENABLED=true

# percentage of zram used as swap compared to physically available DRAM.
# Huge overcommitment (300) is possible and sometimes desirable. See
# https://forum.armbian.com/topic/5565-zram-vs-swap/?do=findComment&comment=61082
# and don't forget to adjust $MEM_LIMIT_PERCENTAGE below too.
# ZRAM_PERCENTAGE=50

# percentage of DRAM available to zram. If this amount is exceeded the zram
# devices used for swap simply behave as if the device is full. You need to
# adjust/increase this value only if you want to work with massive memory
# overcommitment (ZRAM_PERCENTAGE exceeding 150 for example)
# MEM_LIMIT_PERCENTAGE=50

# create how many zram devices max for swap
# ZRAM_MAX_DEVICES=4

# Which algorithm for zram based swapping. Seems lzo is best choice on ARM:
# https://forum.armbian.com/topic/8161-swap-on-sbc/?do=findComment&comment=61668
# SWAP_ALGORITHM=lzo

# Which algorithm to choose for zram based ramlog partition
# RAMLOG_ALGORITHM=zstd

# Which algorithm to choose for zram based /tmp
# TMP_ALGORITHM=zstd

# If defined a separate partition will be used as zram backing device. Be CAREFUL
# which partition you assign and read starting from CONFIG_ZRAM_WRITEBACK in
# https://www.kernel.org/doc/Documentation/blockdev/zram.txt
# ZRAM_BACKING_DEV=/dev/nvme0n2

root@wdnas:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb2        15G  1.1G   13G   8% /
devtmpfs        248M     0  248M   0% /dev
tmpfs           249M     0  249M   0% /dev/shm
tmpfs           100M  3.0M   97M   3% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           249M     0  249M   0% /tmp
/dev/sdb1       100M  9.7M   91M  10% /boot
/dev/sda1       2.7T  1.7T  870G  67% /mnt/hd-intern
/dev/zram0       97M   84M  6.5M  93% /var/log
tmpfs            50M     0   50M   0% /run/user/0

 

Link to comment
Share on other sites

18 hours ago, Heisath said:

ncreasing size should be enough, atleast it is for me

You /var/log is filled up to 93%, which is close to saturation, leaving no space for exceptional logs.

18 hours ago, Heisath said:

The latest Armbian release has apparently applied this fix :

What I have in my

/usr/lib/armbian/armbian-truncate-logs

JOURNAL_SIZE=5M # size to shrink systemd-journal

...

    # remove
    /usr/bin/find /var/log -name '*.[0-9]' -or -name '*.gz' | xargs -r rm -f
    # vacuum systemd-journald
    [ -d /var/log/journal ] && journalctl --vacuum-size=${JOURNAL_SIZE}

 

Anyway, armbian-truncate-logs is executed at midnight and it creates numerous 2.5 Mb files 

-rw-r-----  1 root systemd-journal 2.5M Feb 10 16:31 system@0005d7aba5022919-feb5e759559c3bb8.journal~
-rw-r-----  1 root systemd-journal 2.5M Feb 10 16:34 system@0005d7abb08cc9be-b604a49ab52aa0fc.journal~
-rw-r-----  1 root systemd-journal 2.5M Feb 10 16:41 system@0005d7abc90e2e8b-fffdcbf272568db6.journal~
-rw-r-----  1 root systemd-journal 2.5M Feb 10 16:43 system@0005d7abcf62f90f-03cf52d5f085f7ea.journal~
-rw-r-----  1 root systemd-journal 2.5M Feb 10 16:51 system@0005d7abebf00786-f6350b33e3c2d8c2.journal~
-rw-r-----  1 root systemd-journal 2.5M Feb 10 17:12 system@0005d7ac371cdacc-92de5633f876fefe.journal~
-rw-r-----  1 root systemd-journal 2.5M Feb 10 18:07 system@0005d7acfb3a9715-ed48625431c60e14.journal~
-rw-r-----+ 1 root systemd-journal 2.5M Feb 10 22:38 system.journal
-rw-r--r--  1 root systemd-journal 2.5M Feb  6 16:50 user-1000.journal

 

This saturates completely allocated log space. This needs to be fixed.

 

Link to comment
Share on other sites

This is what happens every midnight after armbian-ramlog is triggered :

Feb 14 00:00:06 tiger systemd[1]: Starting system activity accounting tool...
Feb 14 00:00:07 tiger systemd[1]: Starting Rotate log files...
Feb 14 00:00:07 tiger systemd[1]: Starting Daily man-db regeneration...
Feb 14 00:00:07 tiger systemd[1]: sysstat-collect.service: Succeeded.
Feb 14 00:00:07 tiger systemd[1]: Finished system activity accounting tool.
Feb 14 00:00:07 tiger armbian-ramlog[16480]: Mon 14 Feb 2022 12:00:07 AM CET: Syncing logs to storage
Feb 14 00:00:07 tiger armbian-ramlog[16483]: sending incremental file list
Feb 14 00:00:07 tiger armbian-ramlog[16483]: ./
Feb 14 00:00:07 tiger armbian-ramlog[16483]: armbian-hardware-monitor.log
Feb 14 00:00:07 tiger armbian-ramlog[16483]: auth.log
Feb 14 00:00:07 tiger armbian-ramlog[16483]: btmp
Feb 14 00:00:07 tiger armbian-ramlog[16483]: daemon.log
Feb 14 00:00:07 tiger armbian-ramlog[16483]: debug
Feb 14 00:00:07 tiger armbian-ramlog[16483]: dpkg.log
Feb 14 00:00:07 tiger armbian-ramlog[16483]: faillog
Feb 14 00:00:07 tiger armbian-ramlog[16483]: kern.log
Feb 14 00:00:07 tiger armbian-ramlog[16483]: lastlog
Feb 14 00:00:07 tiger armbian-ramlog[16483]: messages
Feb 14 00:00:07 tiger armbian-ramlog[16483]: syslog
Feb 14 00:00:07 tiger armbian-ramlog[16483]: user.log
Feb 14 00:00:07 tiger armbian-ramlog[16483]: wtmp
Feb 14 00:00:07 tiger armbian-ramlog[16483]: apt/
Feb 14 00:00:07 tiger armbian-ramlog[16483]: apt/eipp.log.xz
Feb 14 00:00:07 tiger armbian-ramlog[16483]: apt/history.log
Feb 14 00:00:07 tiger armbian-ramlog[16483]: apt/term.log
Feb 14 00:00:12 tiger CRON[16451]: pam_unix(cron:session): session closed for user root
Feb 14 00:00:12 tiger CRON[16451]: pam_unix(cron:session): session closed for user root
Feb 14 00:00:14 tiger armbian-ramlog[16557]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16558]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16559]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16560]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16562]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16563]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16565]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16566]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16567]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16570]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16571]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16572]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16575]: cat: write error: No space left on device
Feb 14 00:00:14 tiger armbian-ramlog[16580]: cat: write error: No space left on device
Feb 14 00:00:14 tiger systemd[1]: logrotate.service: Succeeded.
Feb 14 00:00:14 tiger systemd[1]: Finished Rotate log files.
Feb 14 00:00:14 tiger systemd[1]: logrotate.service: Consumed 5.363s CPU time.

Link to comment
Share on other sites

In my case, not on espressobin, I doubled zram and still had these errors. Then I had to instruct journal to a max (eg. 10-20 mb) and the errors stopped.

 

ps: @y52 did you try u-boot from openwrt for espressobin? It seems that it supports distroboot so probably vanilla debian out of the box

Link to comment
Share on other sites

On 2/14/2022 at 10:22 PM, lampra said:

I had to instruct journal to a max

Could you be more specific about which setting you have adjusted? For a moment, I execute past the midnight

# journalctl --vacuum-size=10M --rotate

but a more sustainable solution would be welcome.

On 2/14/2022 at 10:22 PM, lampra said:

did you try u-boot from openwrt for espressobin

No, I didn't. My first target is to make my Armbian setup operational before proceeding further. 

Link to comment
Share on other sites

1 hour ago, y52 said:

journalctl --vacuum-size=10M --rotate

Sorry, I do not remember exactly. Check at `/etc/systemd/journald.conf` the `SystemMaxUse`. My setting is 20M

When using `vacuum-size` Just remember that journal logs are located both at  `/var/log/journal` (zram) and `/var/log.hdd/journal`

Link to comment
Share on other sites

5 hours ago, lampra said:

Check at `/etc/systemd/journald.conf` the `SystemMaxUse`. My setting is 20M

Thanks. I've already adjusted this setting to :

#SystemMaxUse=20M
SystemMaxUse=15M

It doesn't make any effect to my setup, which fills in all available log mount in `/var/log/` (zram).

So I don't know how to fix it so far.

5 hours ago, lampra said:

Just remember that journal logs are located both at  `/var/log/journal` (zram) and `/var/log.hdd/journal`

Yes, there are a lot of 1.9M sized systemd-journal files system@b55xxx.journal

I wonder if they are accounted for in logs MaxUse size. 

Link to comment
Share on other sites

1 hour ago, y52 said:

So I don't know how to fix it so far.

Maybe you could check if zram size (using: df -h) and journal max limit (using: systemctl status systemd-journald.service ) are honored.
Then you could examine the logs and check if something is flooding the journal. That was the case in one of my boards.

Link to comment
Share on other sites

18 hours ago, lampra said:

examine the logs and check if something is flooding the journal.

As explained earlier, the logs point to the armbian-ramlog, which executes every midnight and saturates the log mount.

There is no other reason for the moment and the log mount is half empty:

root@tiger:/# df -h
Filesystem      Size  Used Avail Use% Mounted on

/dev/zram1       49M   21M   25M  47% /var/log

 

18 hours ago, lampra said:

journal max limit (using: systemctl status systemd-journald.service ) are honored.

journal max limit was trimmed down as I wrote before and systemd-journald.service works as expected :

root@tiger:/# systemctl status systemd-journald.service
● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
    Drop-In: /usr/lib/systemd/system/systemd-journald.service.d
             └─override.conf
     Active: active (running) since Sun 2022-02-13 12:18:02 CET; 4 days ago
TriggeredBy: ● systemd-journald.socket
             ● systemd-journald-audit.socket
             ● systemd-journald-dev-log.socket
       Docs: man:systemd-journald.service(8)
             man:journald.conf(5)
   Main PID: 599 (systemd-journal)
     Status: "Processing requests..."
      Tasks: 1 (limit: 1147)
     Memory: 6.4M
        CPU: 56.792s
     CGroup: /system.slice/systemd-journald.service
             └─599 /lib/systemd/systemd-journald

Feb 17 00:01:01 tiger systemd-journald[599]: System Journal (/var/log/journal/50bea5a2341c40588d32c8103dea6e71) is>
Warning: journal has been rotated since unit was started, output may be incomplete.

Link to comment
Share on other sites

9 minutes ago, y52 said:

root@tiger:/# systemctl status systemd-journald.service

From your output, It is not obvious to me, If the limit is honored. Here is my output. It is clearly stated that runtime journal is 22.6M, max 20.1M, 0B free.:

root@cubietruck:~# systemctl status systemd-journald.service
● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
    Drop-In: /lib/systemd/system/systemd-journald.service.d
             └─override.conf
     Active: active (running) since Wed 2022-01-12 17:02:10 EET; 1 months 5 days ago
TriggeredBy: ● systemd-journald-audit.socket
             ● systemd-journald-dev-log.socket
             ● systemd-journald.socket
       Docs: man:systemd-journald.service(8)
             man:journald.conf(5)
   Main PID: 547 (systemd-journal)
     Status: "Processing requests..."
      Tasks: 1 (limit: 4569)
     Memory: 13.3M
        CPU: 3min 45.567s
     CGroup: /system.slice/systemd-journald.service
             └─547 /lib/systemd/systemd-journald

Feb 16 17:11:22 cubietruck systemd-journald[547]: Runtime Journal (/run/log/journal/b1762bb2a9d8431a988c6cac5ff8f356) is 22.6M, max 20.1M, 0B free.
Warning: journal has been rotated since unit was started, output may be incomplete.

and here is my doubled zram

Filesystem      Size  Used Avail Use% Mounted on
/dev/zram1       97M   18M   73M  20% /var/log

 

Link to comment
Share on other sites

31 minutes ago, lampra said:

From your output, It is not obvious to me, If the limit is honored.

My output was not full, as it doesn't fit the terminal width. Here is what is extended after the ">" :

>al (/var/log/journal/50bea5a2341c40588d32c8103dea6e71) is 15.0M, max 15.0M, 0B free.

So, there is clearly a 15.0M, max 15.0M, 0B free.

 

But the df -h shows 25M free:

/dev/zram1       49M   21M   25M  47% /var/log

 

 

I've also adjusted additional setting in /etc/systemd/journald.conf :

#RuntimeMaxUse=
RuntimeMaxUse=15M

 

Link to comment
Share on other sites

Explanations in man journald.conf :

 

       SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, SystemMaxFiles=, RuntimeMaxUse=, RuntimeKeepFree=,
       RuntimeMaxFileSize=, RuntimeMaxFiles=
           Enforce size limits on the journal files stored. The options prefixed with "System" apply to the
           journal files when stored on a persistent file system, more specifically /var/log/journal. The
           options prefixed with "Runtime" apply to the journal files when stored on a volatile in-memory file
           system, more specifically /run/log/journal. The former is used only when /var/ is mounted, writable,
           and the directory /var/log/journal exists.

 

 

Probably this setting is more likely to act on /var/log (zram):

#RuntimeMaxUse=
RuntimeMaxUse=15M

Link to comment
Share on other sites

Once I changed the above setting, Here is a new output :

arted
Feb 17 19:34:51 tiger systemd-journald[52704]: Runtime Journal   (/run/log/journal/50bea5a2341c40588d32c8103dea6e71) is 1.2M, max 15.0M, 13.7M free.
 Feb 17 19:34:51 tiger systemd-journald[52704]: Time spent on flushing to /var/log/journal/50bea5a2341c40588d32c8103dea6e71 is 2.701ms for 13 entries.
Feb 17 19:34:51 tiger systemd-journald[52704]: System Journal (/var/log/journal/50bea5a2341c40588d32c8103dea6e71) is 13.1M, max 15.0M, 1.8M free.

 

Link to comment
Share on other sites

On 2/17/2022 at 7:33 PM, y52 said:

RuntimeMaxUse=15M

It doesn't help either. Log mount is saturating VEEERY quickly. While I was installing different services, I was blocked several times with the log mount reaching the limits.

The whole armbian-ramlog service needs to be revisited.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

Terms of Use - Privacy Policy - Guidelines