barish

I experienced this issue tons of times in the past while working on tvbox support, mostly it was happening after freeze/reboot during kernel boot.
 
As far as I can remember I took a look and there was an armbian service which was rewriting armbianEnv.txt at startup (maybe to update usb quirks?), in fact the order of keys in armbianEnv.txt sometimes changes.
I suspect that when, for a reason or another, the cache is not correctly wrote to flash, armbianEnv.txt gets corrupted.
Recently it didn't happen anymore to me, to be honest.
 
 

I forgot that I had made a copy of the fs of the corrupted Pine64 SD.  This is what's in armbianEnv.txt:
/var/log.hdd/alternatives.log {
    monthly
    rotate 12
    compress
    delaycompress
    missingok
    notifempt:
    create 644 root root
}
\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00
usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u
-------------------------------------------------------------------------------------------------
The first part of this content matches the content in the file /etc/logrotate.d/alternatives
 
Most of the files in /etc/logrotate.d are also corrupted with only 'alternatives' seemingly intact.

Many have streams of  \00\00\00\00\00\ ... appended and many are truncated.
 
There is an empty file named 'sedd3SIEh'.

'apport' has been overwritten with the missing part of armbianEnv.txt as follows:
verbosity=1
bootlogo=false
console=both
disp_mode=1920x1080p60
overlay_prefix=sun50i-a64
rootdev=UUID=3e9afe94-8407-4434-9be6-9528

It's as if the sed command went haywire.
 
/etc/logrotate.conf is also corupted, truncated and a string of \00\00\00.. appended.

Hi,
You can try the following to get an idea of which process is modifying the file:
 
sudo apt-get install auditd sudo auditctl -w /boot/armbianEnv.txt -p wa sudo tail -F /var/log/audit/audit.log  
The output should show actions performed on the file and (IIRC) the process ID performing those actions, perhaps that might help?
 
e.g.:
type=CWD msg=audit(1624526184.097:207): cwd="/home/djurny" type=PATH msg=audit(1624526184.097:207): item=0 name="/boot/" inode=7601 dev=b3:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(1624526184.097:207): item=1 name="/boot/armbianEnv.txt" inode=43198 dev=b3:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 type=PROCTITLE msg=audit(1624526184.097:207): proctitle=xx type=SYSCALL msg=audit(1624526184.109:208): arch=40000028 syscall=94 per=800000 success=yes exit=0 a0=3 a1=81a4 a2=c80eeb00 a3=81a4 items=1 ppid=9641 pid=9642 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts5 ses=2 comm="vi" exe="/usr/bin/vim.basic" subj==unconfined key=(null) type=PATH msg=audit(1624526184.109:208): item=0 name=(null) inode=43198 dev=b3:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 type=PROCTITLE msg=audit(1624526184.109:208): proctitle=xx type=SYSCALL msg=audit(1624526184.109:209): arch=40000028 syscall=226 per=800000 success=yes exit=0 a0=1787820 a1=b6e3e1a0 a2=1907fc0 a3=1c items=1 ppid=9641 pid=9642 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts5 ses=2 comm="vi" exe="/usr/bin/vim.basic" subj==unconfined key=(null) type=CWD msg=audit(1624526184.109:209): cwd="/home/djurny" type=PATH msg=audit(1624526184.109:209): item=0 name="/boot/armbianEnv.txt" inode=43198 dev=b3:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 type=PROCTITLE msg=audit(1624526184.109:209): proctitle=xx  
Source: Find which process is modifying a file [duplicate]
 
It honestly sounds a bit more like a filesystem that has lost track or a inode mixup somewhere? As someone mentioned, perhaps something else is writing to this file, thinking it is something else? Perhaps try to locate a symbolic (or hard)link to armbanEnv.txt:
## find any symbolic links to armbianEnv.txt sudo find / -xdev -type l -ls | egrep -i -- armbianEnv.txt ## find hardlinks to armbianEnv.txt - need to be on the same filesystem! sudo find / -xdev -samefile /boot/armbianEnv.txt  
Groetjes,

sure thing.

This is about three Helios, one Pine64 and two EspressoBin boards. @Werner could we move this thread to a place visible to others than Rockchip please?

Unfortunately, auditd is not working at all. It just doesn't show any file access whatsoever, even though the demon is running and the rule is set. So this is of no help on my system.
edit: Maybe the Armbian kernel is compiled without audit kernel module!?

Same has happened on EspressoBin running Buster 5.10. I don‘t believe it to be a filesystem problem. It looks like something happening at boot time perhaps while running /usr/lib/armbian-hardware-optimization.sh ? I will try to go further into it.

Same has happened on EspressoBin running Buster 5.10. I don‘t believe it to be a filesystem problem. It looks like something happening at boot time perhaps while running /usr/lib/armbian-hardware-optimization.sh ? I will try to go further into it.

Ensign's Log, stardate... uh, today.
 
I've contributed changes that build the FIT image as part of /etc/initramfs/post-update.d/99-uboot-fit, which means it'll be updated when new kernels are installed.
 
I've also contributed changes that modify the boot.scr.uimg to use the FIT image rather than the legacy images, and which allows setting "board_version" in armbianEnv.txt to identify which FDT to use from the FIT image.  Options are "v5" and "v7", and the only known difference is the ordering of the ethernet ports.

I have sent that fallback vdd voltage patch to Marvell: https://github.com/MarvellEmbeddedProcessors/A3700-utils-marvell/pull/18

Can an External Abort trigger because of temps?
 
I should already be on v3 of the a3720 cpufreq patches.
 
Here's the output after fixing printf:
TIM-1.0 mv_ddr-devel-g7c35173 DDR4 16b 1GB 1CS WTMI-devel-18.12.1-c444aeb-dirty WTMI: system early-init DDR topology parameters: ======================== ddr type DDR4 ddr speedbin 11 bus width 16-bits cs num 1 cs[0] - group num 0 cs[0] - bank num 8 cs[0] - capacity 1024MiB SVC REV: 5, CPU VDD voltage is invalid, using default value: 1.202V DRAM windows: ============= WIN[0] - base addr 0x60000000 WIN[0] - size 0x40000000 memory test region: =================== CS[0] 0x60000000 - 0x9fffffff SELF-REFRESH TEST PASSSetting clocks: CPU 1200 MHz, DDR 750 MHz VREF READ TRAINING PASSED VREF WRITE TRAINING PASSED DLL TUNING PASSED SELF-REFRESH TEST PASSNOTICE: Booting Trusted Firmware NOTICE: BL1: v2.4(debug):v2.4-556-g8078b5c5a (Marvell-devel-18.12.2) NOTICE: BL1: Built : 15:18:32, Apr 11 2021 NOTICE: BL1: Booting BL2 NOTICE: BL2: v2.4(debug):v2.4-556-g8078b5c5a (Marvell-devel-18.12.2) NOTICE: BL2: Built : 15:18:32, Apr 11 2021 NOTICE: BL1: Booting BL31 NOTICE: BL31: v2.4(debug):v2.4-556-g8078b5c5a (Marvell-devel-18.12.2) NOTICE: BL31: Built : 15:18:32, Apr 11 2021 U-Boot 2021.04-00640-g3f2e3c7845 (Apr 10 2021 - 15:27:13 +0200) DRAM: 1 GiB WDT: Not starting Comphy-0: USB3_HOST0 5 Gbps Comphy-1: PEX0 2.5 Gbps Comphy-2: SATA0 5 Gbps SATA link 0 timeout. AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode flags: ncq led only pmp fbss pio slum part sxs PCIE-0: Link down MMC: sdhci@d0000: 0, sdhci@d8000: 1 Loading Environment from SPIFlash... SF: Detected mx25u3235f with page size 256 Bytes, erase size 4 KiB, total 4 MiB OK Model: Globalscale Marvell ESPRESSOBin Board Card did not respond to voltage select! : -110 Net: eth0: neta@30000 [PRIME] Hit any key to stop autoboot: 0  
 
I added a fan to my Espressobin, and I'm currently testing to see if that keeps it from crashing (running `stress --cpu 2` on the device. I've had it running for about 30 min so far).

Well, External Abort means that CPU tried to access memory on the bus (not internal CPU memory) and for some reason access (read or write) cannot be completed and it raised exception. Overheating may cause different issues and if some block on SoC stops working then abort may happen.
 
From your log can be seen that base VDD voltage for 1.2 GHz CPU frequency is set to 1.202 V. Typical voltage for 1.2 GHz freq is documented by Marvell as 1.155 V. Higher voltage value makes CPU more stable, but also increase power usage and temperature. And of course higher temperature make CPU less stable.
 
During testing 1 GHz mode when CPU voltage was too low I saw lot of times either this External Abort or segfault of random processes. So External Abort can be definitely caused by instability of CPU. But in your case it may be because of either high temperature or low voltage or something else... Also it may be because your SoC/CPU tested during manufactoring and checked that is not really stable for 1.2 GHz freq (and so written this information into OTP -- not defined VDD value for 1.2 GHz).

This backtrace is from debug version of Trusted Firmware. If you compile it without DEBUG=1 then backtrace is omitted.
 
A3720 SoC does not have full power management like ATX on PC, which can turn power off from all components. This is why function a3700_system_off() in Trusted Firmware is not implemented.
 
Basically I just do not know what this a3700_system_off() should do (if I decide to implement it). It can just call reset or maybe call suspend procedure. A3720 has support for some kind of low power suspend/sleep mode, so maybe this is the best approximation of whole power off.

There is just information that External Abort happened. Not sure what is the issue.
 
Check that you have kernel with v3 version of a3720 cpufreq patches as in v3 was updated VDD value for load L1 when base freq is 1.2 GHz.
 
Also can apply updated cpu_vdd_fallback.patch patch? It does not change behavior, only fixes printf for SVC REV: line. But it can can be useful to verify which CPU VDD value is used.

I am not a user of Armbian and do not plan to.
 
Wanted to write here and thank to whoever provided UART U-Boot files as I used them to get my EspressoBin v5 running.
 
Now it has mainline U-Boot 2021.01 and works as normal EBBR board. I plan to use OpenWRT on it and use as a router for some time.
 
Few more words: https://marcin.juszkiewicz.com.pl/2021/02/15/ebbr-on-espressobin/

https://github.com/armbian/build/pull/2598
Beta repository (beta.armbian.com in about 30m from now) or manual install, builds 91+ contains those patches. I also switched u-boot but needs testing.

The spi flash memory (with the u-boot environment stored inside) is accessible from the command line. Please check if the corresponding driver is loaded via dmesg. It should look like:
[ 1.878826] spi-nor spi0.0: w25q32dw (4096 Kbytes) [ 1.878850] 2 cmdlinepart partitions found on MTD device spi0.0 [ 1.878854] Creating 2 MTD partitions on "spi0.0": [ 1.878870] 0x000000000000-0x0000003f0000 : "uboot" [ 1.888884] 0x0000003f0000-0x000000400000 : "uboot-environment" The partitioning information is passed as a command line parameter to the kernel (last parameter in this case):
console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000 root=UUID=c2bdee11-6a88-46c1-8630-b70864d535bb rootfstype=ext4 rootwait loglevel=1 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u mtdparts=spi0.0:4032k(uboot),64k(uboot-environment)  You need to install the package u-boot-tools containing the userland tools and you need to configure the correct offsets:
uli@ebin:~$ cat /etc/fw_env.config # device name device offset env. size flash sector size number of sectors /dev/mtd1 0x00000000 0x00010000 0x1000 0x10 Then fw_printenv / fw_setenv will help you to modify the u-boot environment:
uli@ebin:~$ sudo fw_printenv arch=arm baudrate=115200 board=mvebu_armada-37xx board_name=mvebu_armada-37xx boot_a_script=ext4load ${boot_interface} ${devnum}:1 ${scriptaddr} ${prefix}boot.scr;source ${scriptaddr}; boot_prefixes=/ /boot/ boot_targets=usb sata mmc1 mmc0 bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done bootcmd_mmc0=setenv devnum 0; setenv boot_interface mmc; run scan_dev_for_boot; bootcmd_mmc1=setenv devnum 1; setenv boot_interface mmc; run scan_dev_for_boot; bootcmd_sata=setenv devnum 0; scsi scan; scsi dev 0; setenv boot_interface scsi; run scan_dev_for_boot; bootcmd_usb=setenv devnum 0; usb start;setenv boot_interface usb; run scan_dev_for_boot; bootdelay=2 console=console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000 cpu=armv8 eth1addr=00:51:82:11:22:01 eth2addr=00:51:82:11:22:02 eth3addr=00:51:82:11:22:03 ethaddr=00:51:82:11:22:00 ethprime=eth0 extra_params=pci=pcie_bus_safe fdt_addr=0x6000000 fdt_addr_r=0x6f00000 fdt_high=0xffffffffffffffff fdt_name=fdt.dtb gatewayip=10.4.50.254 get_images=tftpboot $kernel_addr_r $image_name; tftpboot $fdt_addr_r $fdt_name; run get_ramfs get_ramfs=if test "${ramfs_name}" != "-"; then setenv ramdisk_addr_r 0x8000000; tftpboot $ramdisk_addr_r $ramfs_name; else setenv ramdisk_addr_r -;fi hostname=marvell image_name=Image initrd_addr=0x1100000 initrd_image=uInitrd initrd_size=0x2000000 ipaddr=0.0.0.0 kernel_addr=0x7000000 kernel_addr_r=0x7000000 loadaddr=0x8000000 netdev=eth0 netmask=255.255.255.0 ramdisk_addr_r=0x8000000 ramfs_name=- root=root=/dev/nfs rw rootpath=/srv/nfs/ scan_dev_for_boot=for prefix in ${boot_prefixes}; do echo ${prefix};run boot_a_script; done scriptaddr=0x6d00000 serverip=0.0.0.0 set_bootargs=setenv bootargs $console $root ip=$ipaddr:$serverip:$gatewayip:$netmask:$hostname:$netdev:none nfsroot=$serverip:$rootpath,tcp,v3 $extra_params $cpuidle soc=mvebu vendor=Marvell Unfortunately the offsets vary between the different u-boot versions, so you have to find out the correct one for your board.
fw_printenv will tell you if the CRC check fails meaning that the chosen offsets do not fit. 

With help from @Pali, I managed to put the Pull Request at the right spot, so when it's accepted, EspressoBin v7 1GB and 2GB will be stable with most current U-Boot and Marvell sources! Unfortunately, I have no board with 2GB, so maybe someone can test it who owns a 2GB v7. Armbian builder is using older branches for stability reasons up to now to compile the U-Boot images – maybe with all the new patches in place, Armbian can also use the newest U-Boot version.
Edit: The Pull Request has just been merged. I will make a PR at Armbian Build to use newest Marvell repo branches.

With help from @Pali, I managed to put the Pull Request at the right spot, so when it's accepted, EspressoBin v7 1GB and 2GB will be stable with most current U-Boot and Marvell sources! Unfortunately, I have no board with 2GB, so maybe someone can test it who owns a 2GB v7. Armbian builder is using older branches for stability reasons up to now to compile the U-Boot images – maybe with all the new patches in place, Armbian can also use the newest U-Boot version.
Edit: The Pull Request has just been merged. I will make a PR at Armbian Build to use newest Marvell repo branches.

Finally, Globalscale did the trick and pushed the modifications, that they had added to make 17.10 stable for all EspressoBin-Boards, to 18.12 as well. Thanks a lot, Globalscale!!
 
Unfortunately, they did this in their repos and not in the official Marvell repos. I extracted the changes and made a pull request at the Marvell repo. I hope the changes will be merged soon, because I can then use the official Armbian build tools to generate a U-Boot/firmware that flawlessly works with my EspressoBin v7 DDR4 1cs 1GB boards. For now, I compile using the Globalscale repo and got a really stable firmware (tested on three boards for now), I am using the mainline U-Boot (2020.10), too and have no problems – but have not tested this with PCIe periphery (like wifi or SATA adaptors).

Finally, Globalscale did the trick and pushed the modifications, that they had added to make 17.10 stable for all EspressoBin-Boards, to 18.12 as well. Thanks a lot, Globalscale!!
 
Unfortunately, they did this in their repos and not in the official Marvell repos. I extracted the changes and made a pull request at the Marvell repo. I hope the changes will be merged soon, because I can then use the official Armbian build tools to generate a U-Boot/firmware that flawlessly works with my EspressoBin v7 DDR4 1cs 1GB boards. For now, I compile using the Globalscale repo and got a really stable firmware (tested on three boards for now), I am using the mainline U-Boot (2020.10), too and have no problems – but have not tested this with PCIe periphery (like wifi or SATA adaptors).

Despite of all the seemingly bad news, I boldly ordered a batch of 10 EspressoBin V7 recently. Globalscale promised to test them for me for stability (I am not interested in squeezing out the last bit of performance), and I tested them again after reception: They all seem to run stable with Ubuntu Xenial (the "newest" image provided by Globalscale) and with Armbian kernel 4.19.59-mvebu64.
Since there is still a few weeks till they go on duty, if anyone is interested in a particular test on them, I am willing to do so. I am not into kernel development, so I would need some instruction. ;-)

Indeed, the configuration in the mvebu64.conf script is wrong.
 
The 2 CS 2 GB image is compiled with the same DDR topology parameter as the 2 CS 1 GB image (DDR_TOPOLOGY=2).
It appears that the configuration for the 2 CS 512 MB version is also wrong, as it points to the same topology file as the 1 CS version.
I guess this is not really a problem, because I'm not even sure that this variant actually exists.
 
So to answer my previous question:
What is used is only 512 MB from each chip.
 
If anyone needs it, here is the bootloader (U-Boot 2018.03-18.12) compiled with the correct topology (DDR_TOPOLOGY=7):
flash-image-ddr3-2g-2cs-1000_800.bin
 
Note that only changing the DDR_TOPOLOGY from 2 to 7 was not enough, because this configuration is also missing from the Marvell sources, even though that's the parameter used in the build examples.
The correct topology exists in the Armbian tree, but with the wrong name:
https://github.com/armbian/build/blob/master/packages/blobs/espressobin/DDR_TOPOLOGY_2-2g.txt
The A3700-utils-marvell/script/buildtim.sh script must also be adapted to accept a DDR_TOPOLOGY value higher than 6.

Well I've fixed my instability issue. The root cause was the same as https://forum.netgate.com/topic/144636/sg-1100-intermittent-reboots, and it was power related. The fix was to replace the 470uf/16V capacitor (component EC1 on the schematic) sat immediately behind the 12V DC jack which under testing leaked under certain temperature conditions (variations). As such the board is now stable and has been running OpenWRT 18.06.2 for 24 hours with memtester and stress running continuous during that entire time.
 
I'll leave it a few more days, after that I'm going to assume that its fixed and that is what NetGate were referring too when they mention 'a power related component'.
 
And GlobalScape will not tell you, they know of the problem, in a reply from Kevin Liu he says 'Yes, I do have knowledge regards the power related component issue.' when I asked which component it was so i could attempt to replace myself.

I've been looking at something relating to this, recompiling and modifying atf/A3700-marvell-utils to build a viable u-boot. From what I've figure out already, it seems the voltage (VDD) applied by marvell is too low by default.
 
Does anyone have any visibility on where the modification to ddr4-*cs-/*g.txt from http://wiki.espressobin.net/tiki-download_file.php?fileId=216 came from? namely the following line:
+;Step9: DDRPHY Driver/Receiver & DQS internal Pullup/Pulldown settings
+;WRITE: 0xC0001004 0xD0133449
+WRITE: 0xC0001004 0xD0677449
 
That is the only difference between the Espressobin provided ddr init and the mainline Marvell A3700 ddr init code.
At the moment I'm running an Espressobin v7 1000/800 using WTMI-devel-18.12.1-e6bb176 / 2018.03-devel-18.12.3-gc9aa92ce70 with a modification to bump the AVS voltage from 1.032V to 1.155V by default. This is looking to be more stable currently..
 
If anyone wants the resulting builds let me know. I cannot use the latest https://dl.armbian.com/espressobin/u-boot/ provided versions of WTMI-devel-18.12.1-e6bb176 since it randomly fails to boot. I rebuilt and enabled debugging and so far no stalls.