Dennboy

Hi Amir, It has been a while since your post, but anyway here are some pointers: The SPI master bus is enabled by default on the nanopi neo3, see e.g.: ls /sys/firmware/devicetree/base/spi@ff190000 You can use the bus by attaching an SPI device to the bus and use it either via userland using the general purpose spidev kernel driver or load/write the appropriate driver kernel module and/or trigger driver loading via a device tree overlay. We use our own driver and device tree overlay for SPI with an ADC on the neo3, we see sporadic loss of SPI interrupts at 4KHz sampling and more at higher sampling rates, especially when there is high CPU load on the neo3. Kind regards, Dennis

I created a pull request to exclude /var/log/journal from rsync in nand-sata-install / armbian-install and create empty ${TempDir}/rootfs//var/log/journal folder when /var/log.hdd/journal exists during the installation: https://github.com/armbian/build/pull/4371

Since volatile journald logging is done in-memory in /run/log/journal, and made persistent in /var/log.hdd/journal, the rsync of /var/log/journal in nand-sata-install no longer works because it is a symlink to /var/log.hdd/journal in the live system, and will create a circular symlink when booting from NAND. On the target image /var/log is the folder that is directory-mounted to /var/log.hdd during boot. I propose to exclude /var/log from the rsync, and rsync /var/log.hdd to /var/log on the target image. Another solution could be to exclude only /var/log/journal from the rsync, and create an empty /var/log/journal in the target image. Kind regards, Dennis

For the record, another correction. The statement for Storage=volatile journald works after changing to 2. change to Storage=volatile in /etc/systemd/journald.conf, and optionally remove journal directories to save some space sudo sh -c "echo Storage=volatile » /etc/systemd/journald.conf" sudo systemctl stop systemd-journald.service sudo rm -rf /var/log/journal /var/log.hdd/journal sudo systemctl start systemd-journald.service

Thanks Igor, I've created a pull request (https://github.com/armbian/build/pull/3967) after some tweaks & tests with non-rsync ramlog.

After some experimenting I found a way to flush the journal and maintain logs in tmpfs using /run/log/journal, and only have a softlink from /var/log/journal to /var/log.hdd/journal. Additionally the cached journals are to be cleaned up when /var/log becomes too full. I created the attached patches for armbian-truncate-logs and armbian-ramlog: armbian-truncate-logs.patcharmbian-ramlog.patch Some logic was needed in cases where the user decides to do Storage=volatile journal logging in /etc/systemd/journald.conf, or when the user or journald with Storage=persistent creates a /var/log/journal folder which will be moved to /var/log.hdd/journal and replaced by the softlink. I tested stop, start, write for armbian-ramlog, and truncate-logs. They seem to work fine.

small correction, of course /var.hdd/log.journal should be changed to /var/log.hdd/journal

I've done some reading and experimenting with systemd journald, and there appear to be at least 2 ways to easily turn off persistent logging to help with your SBC issues : keep default Storage=auto in /etc/systemd/journald.conf, and remove journal directories sudo systemctl stop systemd-journald.service sudo rm -rf /var/log/journal /var.hdd/log.journal sudo systemctl start systemd-journald.service change to Storage=volatile in /etc/systemd/journald.conf, and optionally remove journal directories to save some space sudo su -c "echo Storage=volatile > /etc/systemd/journal" sudo systemctl restart systemd-journald.service sudo rm -rf /var/log/journal /var.hdd/log.journal In both cases the journald logs will still be available at runtime in /run/log/journal, but not written to the sdcard/emmc/disk. I think this would be a safe default for most users. You can always turn on persistent logging later when you for some reason want to diagnose boot problems. For re-enabling persistent journald logging, the following steps can be taken: with the default Storage=auto in /etc/systemd/journald.conf: mkdir -p /var/log/journal /var.hdd/log.journal with Storage=volatile in /etc/systemd/journald.conf: mkdir -p /var/log/journal /var.hdd/log.journal sudo sed -i "/^Storage=/d" /etc/systemd/journal" sudo systemctl restart systemd-journald.service

It looks like the non-persistent journal on /run/log/journal/* is sometimes also active on my opi0 with Armbian 21.08.2. A new approach that lets journald control volatile and persistent storage, but lets armbian decide when and where to flush to persistent: configure journald to write volatile logs to /run/log/journal move journal from tmpfs space: mv /var/log.hdd/journal /var/log.hdd.journal create softlink in both persistent and volatile memory ln -s /var/log.hdd.journal /var/log/journal ln -s /var/log.hdd.journal /var/log.hdd/journal in armbian-ramlog flush the volatile journal to persistent /var/log/journal (i.e. this points to /var/log.hdd.journal) [ -L /var/log/journal ] && journalctl --flush use rsync flag --one-file-system armbian-truncate-logs, to prevent copying the journal files anyway Currently testing this approach, instead of the softlinks I am also trying mount --bind, but that seems tricky.

looks like my corrupted system.journal is due to influxdb still busy during shutdown while it can no longer write to /var/lib/influxdb looks like the corrupted user journals are due to user processes still running (e.g. the login shell) while /var/log is no longer mounted bottom line: to avoid journal corruption we may have to: flush or stop journald before umounting memory-mapped /var/log, or use the in-memory and persistent storage of journald itself in /var/run/journal and /var/log/journal respectively.

I noticed the following in dmesg, that may be the source of the cached/corrupted journal files. Looks like system journal got corrupted during shutdown and user journals are temporarilly not accessible due to a busy device. npin3bullVT:~:% dmesg|grep -i journal [ 21.666206] systemd[1]: Listening on Journal Audit Socket. [ 21.668883] systemd[1]: Listening on Journal Socket (/dev/log). [ 21.682035] systemd[1]: Listening on Journal Socket. [ 26.931365] EXT4-fs (zram1): mounted filesystem without journal. Opts: discard [ 31.043188] systemd[1]: Starting Journal Service... [ 31.273498] systemd[1]: Started Journal Service. [ 31.327326] systemd-journald[588]: Received client request to flush runtime journal. [ 31.389135] systemd-journald[588]: File /var/log/journal/50bea5a2341c40588d32c8103dea6e71/system.journal corrupted or uncleanly shut down, renaming and replacing. [ 47.628019] systemd-journald[588]: Failed to read journal file /var/log/journal/50bea5a2341c40588d32c8103dea6e71/user-1002.journal for rotation, trying to move it out of the way: Device or resource busy [ 47.630134] systemd-journald[588]: Failed to read journal file /var/log/journal/50bea5a2341c40588d32c8103dea6e71/user-1000.journal for rotation, trying to move it out of the way: Device or resource busy

I have tested my patch using : rsync -r --exclude 'system@*' --exclude 'user-*@*' --exclude='*~' /var/log/journal test and it appears to work from the commandline. However, I still get corrupt journal files at boot. They may appear because journald writes to the journal files during the rsync. npin3bullVT:backup:% uptime; ls -lsa /var/log/journal/50bea5a2341c40588d32c8103dea6e71 09:57:09 up 21 min, 1 user, load average: 2.15, 2.36, 1.83 total 10280 8 drwxr-sr-x+ 2 root systemd-journal 4096 Jun 29 09:36 . 8 drwxr-sr-x+ 3 root systemd-journal 4096 Aug 26 2021 .. 2564 -rw-r----- 1 root systemd-journal 2621440 Jun 29 09:36 system@0005e292e1d3c280-560a1a0a69b9966a.journal~ 2568 -rw-r-----+ 1 root systemd-journal 2621440 Jun 29 09:36 system@9cec3cbef79c442ab7f055042ddcbb75-0000000000000001-0005e292e1d0956a.journal 2568 -rw-r-----+ 1 root systemd-journal 2621440 Jun 29 09:57 system.journal 2564 -rw-r-----+ 1 root systemd-journal 2621440 Jun 29 09:41 user-1002.journal The timestamp of the cached/corrupt files "*~" and "*@*" is during boot, 09:57:09 - 21minutes = 09:36 In /var/log/syslog I see that only the right journal files are copied: Jun 29 09:36:10 npin3bullVT armbian-ramlog[484]: journal/ Jun 29 09:36:10 npin3bullVT armbian-ramlog[484]: journal/50bea5a2341c40588d32c8103dea6e71/ Jun 29 09:36:10 npin3bullVT armbian-ramlog[484]: journal/50bea5a2341c40588d32c8103dea6e71/system.journal Jun 29 09:36:10 npin3bullVT armbian-ramlog[484]: journal/50bea5a2341c40588d32c8103dea6e71/user-1000.journal Jun 29 09:36:10 npin3bullVT armbian-ramlog[484]: journal/50bea5a2341c40588d32c8103dea6e71/user-1002.journal

For excluding the corrupted journal files, I propose the following patch to syncFromDisk in armbian-ramlog: --- /usr/lib/armbian/armbian-ramlog 2021-09-26 20:15:41.000000000 +0000 +++ armbian-ramlog 2022-06-29 09:22:18.927488185 +0000 @@ -69,6 +69,7 @@ ${NoCache} rsync -aXWv --delete \ --exclude "lost+found" --exclude armbian-ramlog.log \ --exclude *.gz --exclude *.xz --exclude='*.[0-9]' \ + --exclude 'system@*' --exclude 'user-*@*' --exclude='*~' \ --links \ ${XTRA_RSYNC_FROM[@]+"${XTRA_RSYNC_FROM[@]}"} \ $HDD_LOG $RAM_LOG 2>&1 | $LOG_OUTPUT

This /var/log issue is indeed very annoying. I have some work-arounds in the mentioned ticket. In my opinion we need: to find and solve the root cause of the corrupted journal files, to reduce the likelihood that this happens make sure corrupted journal files are not put back in /var/log during boot, i.e. exclude journal/*/system@* and journal/*/user-*@* and journal/*/*~ in /usr/lib/armbian/armbian-ramlog : syncFromDisk see my patch for armbian-ramlog in the mentioned ticket above Btw. the main problem is the memory mapped /var/log that fills up (which approx 50MB), the /var/log.hdd is filling up on emmc or sdcard, but this is small compared to the volume size

I see the following in the syslog at night: Jun 29 00:00:07 npin3bullVT armbian-ramlog[10481]: Wed 29 Jun 2022 12:00:02 AM UTC: Syncing logs to storage Jun 29 00:00:09 npin3bullVT armbian-ramlog[10493]: sending incremental file list Jun 29 00:00:09 npin3bullVT armbian-ramlog[10493]: ./ Jun 29 00:00:09 npin3bullVT armbian-ramlog[10493]: alternatives.log Jun 29 00:00:09 npin3bullVT armbian-ramlog[10493]: armbian-hardware-monitor.log Jun 29 00:00:09 npin3bullVT armbian-ramlog[10493]: auth.log Jun 29 00:00:09 npin3bullVT armbian-ramlog[10493]: bootstrap.log Jun 29 00:00:09 npin3bullVT armbian-ramlog[10493]: btmp Jun 29 00:00:09 npin3bullVT armbian-ramlog[10493]: daemon.log Jun 29 00:00:10 npin3bullVT armbian-ramlog[10493]: debug Jun 29 00:00:10 npin3bullVT armbian-ramlog[10493]: dpkg.log Jun 29 00:00:10 npin3bullVT armbian-ramlog[10493]: faillog Jun 29 00:00:10 npin3bullVT armbian-ramlog[10493]: kern.log Jun 29 00:00:10 npin3bullVT armbian-ramlog[10493]: lastlog Jun 29 00:00:10 npin3bullVT armbian-ramlog[10493]: messages Jun 29 00:01:07 npin3bullVT armbian-ramlog[10677]: cat: write error: No space left on device Jun 29 00:01:07 npin3bullVT armbian-ramlog[10703]: cat: write error: No space left on device Jun 29 00:01:07 npin3bullVT armbian-ramlog[10710]: cat: write error: No space left on device Jun 29 00:01:07 npin3bullVT armbian-ramlog[10711]: cat: write error: No space left on device Jun 29 00:01:08 npin3bullVT armbian-ramlog[10729]: cat: write error: No space left on device Jun 29 00:01:08 npin3bullVT armbian-ramlog[10746]: cat: write error: No space left on device Jun 29 00:01:08 npin3bullVT armbian-ramlog[10782]: cat: write error: No space left on device Jun 29 00:01:09 npin3bullVT armbian-ramlog[10789]: cat: write error: No space left on device Jun 29 00:01:09 npin3bullVT armbian-ramlog[10790]: cat: write error: No space left on device Jun 29 00:01:10 npin3bullVT armbian-ramlog[10848]: cat: write error: No space left on device Jun 29 00:01:10 npin3bullVT armbian-ramlog[10861]: cat: write error: No space left on device I removed the journal backup files (in var/log/journal/*/system@*) to get fresh again: npin3bullVT:flowprocessing:% ls -lsa /var/log/journal/50bea5a2341c40588d32c8103dea6e71 <config ✗> total 18016 40 drwxr-sr-x+ 2 root systemd-journal 36864 Jun 29 07:56 . 8 drwxr-sr-x+ 3 root systemd-journal 4096 Aug 26 2021 .. 2568 -rw-r-----+ 1 root systemd-journal 2621440 Jun 29 00:16 system@28483b061ebe4b9a828c4b3066920f5d-0000000000000001-0005e28affeda3ec.journal 2568 -rw-r-----+ 1 root systemd-journal 2621440 Jun 29 00:23 system@28483b061ebe4b9a828c4b3066920f5d-0000000000000c5c-0005e28b0865be92.journal 2568 -rw-r-----+ 1 root systemd-journal 2621440 Jun 29 03:49 system@28483b061ebe4b9a828c4b3066920f5d-0000000000008f6f-0005e28b28f775c6.journal 2568 -rw-r-----+ 1 root systemd-journal 2621440 Jun 29 07:15 system@28483b061ebe4b9a828c4b3066920f5d-0000000000009a6b-0005e28e0a446064.journal 2564 -rw-r-----+ 1 root systemd-journal 2621440 Jun 29 00:01 system@d4f3757edfbb4d17ad2866e73d1fd9c1-000000000000334d-0005e2899513914c.journal 2568 -rw-r-----+ 1 root systemd-journal 2621440 Jun 29 07:59 system.journal 2564 -rw-r-----+ 1 root systemd-journal 2621440 Jun 29 07:59 user-1000.journal For those interested I uploaded my armbianmonitor data http://ix.io/41aW, (using armbianmonitor -U > ~/armbianmonitor.out; cat ~/armbianmonitor.out| curl -F 'f:1=<-' ix.io because it didn't pass the ping check).