ArmbianEnv.txt file being overwritten

[Polarisgeek]

I've been having a recent issue over the last couple months where my ArmbianEvn.txt file is being overwritten thus causing Helios to hang in the boot process.  Using some samples i've found on these forums, I was able to re-create my file and get the Helios back up and running and have since stored a copy so that when it happens, I can just copy and paste and back up running in no time.  To date, i've had this happen to me 4 times.

 

I'm not worried about having go through this at this point in time, but I was curious if anyone else has seen this issue or not?  I've saved a couple samples of what was written into the ArmbianEnv.txt file:

/var/log/armbian-hardware-monitor.log {
  rotate 12
  weekly
  compress

  missi

 

/var/log/alternatives.log {
monthly
rotate 12
compress
delaycompress
missingok
notifempty
create 644 root root
}

 

I'm running Buster 5.10.21.  I have 5-14TB EXOS drives in Raid 6 and have OMV as the only thing installed.  

[barnumbirr]

Had the issue happen semi-regularly to my Helios as well. Same "gibberish" saved to ArmbianEnv.txt as in your example.

Haven't been able to figure out where the issue comes from unfortunately.

[magostinelli]

I had also the same problem, take a look on /etc/logrotate.d folder, here there are other files corrupted.

[djurny]

Hi,

You can try the following to get an idea of which process is modifying the file:

 

sudo apt-get install auditd
sudo auditctl -w /boot/armbianEnv.txt -p wa
sudo tail -F /var/log/audit/audit.log

 

The output should show actions performed on the file and (IIRC) the process ID performing those actions, perhaps that might help?

 

e.g.:

type=CWD msg=audit(1624526184.097:207): cwd="/home/djurny"
type=PATH msg=audit(1624526184.097:207): item=0 name="/boot/" inode=7601 dev=b3:01 mode=040755 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1624526184.097:207): item=1 name="/boot/armbianEnv.txt" inode=43198 dev=b3:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PROCTITLE msg=audit(1624526184.097:207): proctitle=xx
type=SYSCALL msg=audit(1624526184.109:208): arch=40000028 syscall=94 per=800000 success=yes exit=0 a0=3 a1=81a4 a2=c80eeb00 a3=81a4 items=1 ppid=9641 pid=9642 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts5 ses=2 comm="vi" exe="/usr/bin/vim.basic" subj==unconfined key=(null)
type=PATH msg=audit(1624526184.109:208): item=0 name=(null) inode=43198 dev=b3:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PROCTITLE msg=audit(1624526184.109:208): proctitle=xx
type=SYSCALL msg=audit(1624526184.109:209): arch=40000028 syscall=226 per=800000 success=yes exit=0 a0=1787820 a1=b6e3e1a0 a2=1907fc0 a3=1c items=1 ppid=9641 pid=9642 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts5 ses=2 comm="vi" exe="/usr/bin/vim.basic" subj==unconfined key=(null)
type=CWD msg=audit(1624526184.109:209): cwd="/home/djurny"
type=PATH msg=audit(1624526184.109:209): item=0 name="/boot/armbianEnv.txt" inode=43198 dev=b3:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PROCTITLE msg=audit(1624526184.109:209): proctitle=xx

 

Source: Find which process is modifying a file [duplicate]

 

It honestly sounds a bit more like a filesystem that has lost track or a inode mixup somewhere? As someone mentioned, perhaps something else is writing to this file, thinking it is something else? Perhaps try to locate a symbolic (or hard)link to armbanEnv.txt:

## find any symbolic links to armbianEnv.txt
sudo find / -xdev -type l -ls | egrep -i -- armbianEnv.txt

## find hardlinks to armbianEnv.txt - need to be on the same filesystem!
sudo find / -xdev -samefile /boot/armbianEnv.txt

 

Groetjes,

[barish]

Same has happened on EspressoBin running Buster 5.10. I don‘t believe it to be a filesystem problem. It looks like something happening at boot time perhaps while running /usr/lib/armbian-hardware-optimization.sh ? I will try to go further into it.

[barish]

On 6/24/2021 at 11:50 AM, djurny said:

Hi,

You can try the following to get an idea of which process is modifying the file:

 

sudo apt-get install auditd
sudo auditctl -w /boot/armbianEnv.txt -p wa
sudo tail -F /var/log/audit/audit.log

 

The output should show actions performed on the file and (IIRC) the process ID performing those actions, perhaps that might help?

 

Unfortunately, auditd is not working at all. It just doesn't show any file access whatsoever, even though the demon is running and the rule is set. So this is of no help on my system.

edit: Maybe the Armbian kernel is compiled without audit kernel module!?

Edited June 28, 2022 by barish

[schwar3kat]

Ditto on a Pine64 running Buster or Bullseye (can't remember which). 
I thought it must have got corrupted somehow, but it's just a spare that I have so It didn't bother me.  I noticed that it looked like /var/log/ stuff and thought that it was peculiar.
I took the opportunity to try the new Jammy instead and forgot about it until I saw this post.

[Bernie_O]

I have the same problem.

 

armbianEnv.txt gets overwritten with parts of logrotate config files.

And some config files under /etc/logrotate.d/ are corrupted with stuff that looks like it originally belongs to armbianEnv.txt (e.g. boot parameters and MAC adresses).

 

Is there anything I can do? As someone suspects that the script `armbian-hardware-optimization` might be responsible: would disabling the according service harm the system (or can the system run faultlessly without that service)?

 

EDIT: this is on an espressobin running Debian Buster

Edited June 28, 2022 by Bernie_O
add used board and OS

[barish]

This is about three Helios, one Pine64 and two EspressoBin boards. @Werner could we move this thread to a place visible to others than Rockchip please?

[Werner]

2 minutes ago, barish said:

could we move this thread to a place visible to others than Rockchip please?

sure thing.

[schwar3kat]

I forgot that I had made a copy of the fs of the corrupted Pine64 SD.  This is what's in armbianEnv.txt:
/var/log.hdd/alternatives.log {
    monthly
    rotate 12
    compress
    delaycompress
    missingok
    notifempt:
    create 644 root root
}
\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00
usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u

-------------------------------------------------------------------------------------------------

The first part of this content matches the content in the file /etc/logrotate.d/alternatives
 

Most of the files in /etc/logrotate.d are also corrupted with only 'alternatives' seemingly intact.

Many have streams of  \00\00\00\00\00\ ... appended and many are truncated.

 

There is an empty file named 'sedd3SIEh'.

'apport' has been overwritten with the missing part of armbianEnv.txt as follows:

verbosity=1
bootlogo=false
console=both
disp_mode=1920x1080p60
overlay_prefix=sun50i-a64
rootdev=UUID=3e9afe94-8407-4434-9be6-9528

It's as if the sed command went haywire.

 

/etc/logrotate.conf is also corupted, truncated and a string of \00\00\00.. appended.

[schwar3kat]

/usr/lib/armbian/armbian-hardware-optimization has two sed statements that work on these files:

 

They look fairly innocuous,.  I wonder if there is a new version of sed that is messing this up? Or perhaps the contents of one of the files is confusing sed?

Alphabetically the first file corrupted (after alternatives, which is not corrupt) is apache2.  

I was running apache2 web server.   How many of you are also running apache2?

[barish]

In my case, /boot/armbianEnv.txt looks like this (but I have had variations, of this, too):

/var/log/alternatives.log {
	monthly
	rotate 12
	compress
	delaycompress
	missingok
	notifempty
	create 644 root root
}

usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u,0x0bc2:0x231a:u

 

Then, in /etc/logrotate.d, two files seem to be corrupt: 1. alternatives, 2. armbian-hardware-monitor, which had the complete contents of armbianEnv.txt.

 

I run nginx, sambad, forked-daapd, nextcloud, mini-dlna and dnsmasq. No apache though.

Edit: /etc/logrotate.conf is also truncated, thus corrupted. Hadn't noticed before, thanks @schwar3kat.

Edited June 29, 2022 by barish

[barish]

I haven't found the cause, but I deactivated armbian-hardware-optimize.service, and did about 70 reboots afterwards, without any file corruption taking place. So I hope that armbian-hardware-optimization.sh (which is called by the above service) is involved in the error and its deactivation prevents any further file corruption.

[schwar3kat]

Is this perhaps restricted to Debian or is it affecting Ubuntu builds as well?

[barish]

I find it hard to imagine that maybe bash scripts interfere with each other and bash variables get confused across shell borders!?! It’s up to now the only guess I can come up with. This is maybe related to the early boot process where not everything is aligned yet? This is all wild guessing, I haven’t found a way to further analyze, because even auditd is failing on the armbian image I am using.

[Werner]

Just for the heck of it have you tried to set chattr +i on the file?

[schwar3kat]

On 7/8/2022 at 4:19 PM, Werner said:

Just for the heck of it have you tried to set chattr +i on the file?

That might be interesting. 
Perhaps the calling process will leave some errors in the logs.  A sort of audit mechanism.
I'm not running this image anymore, so I can't check, but others may like to try.

[jock]

I experienced this issue tons of times in the past while working on tvbox support, mostly it was happening after freeze/reboot during kernel boot.

 

As far as I can remember I took a look and there was an armbian service which was rewriting armbianEnv.txt at startup (maybe to update usb quirks?), in fact the order of keys in armbianEnv.txt sometimes changes.

I suspect that when, for a reason or another, the cache is not correctly wrote to flash, armbianEnv.txt gets corrupted.

Recently it didn't happen anymore to me, to be honest.

 

 

[schwar3kat]

13 hours ago, jock said:

there was an armbian service which was rewriting armbianEnv.txt at startup

Now that you have mentioned it, I have also noticed the order change.  Do you recall which service made the changes?

[jock]

9 hours ago, schwar3kat said:

Now that you have mentioned it, I have also noticed the order change.  Do you recall which service made the changes?

 

I see this in armbian-hardware-optimization:

armbian-hardware-optimization:369:      echo "" >> /boot/armbianEnv.txt;  sed -i '/^$/d;$G' /boot/armbianEnv.txt; sed -i '/^$/d;$G' /boot/armbianEnv.txt
armbian-hardware-optimization:372:      sed -i '/^$/d' /boot/armbianEnv.txt
armbian-hardware-optimization:393:      sed -i '/^usbstoragequirks/d' /boot/armbianEnv.txt
armbian-hardware-optimization:394:      echo "usbstoragequirks=${USBQUIRKS}" >>/boot/armbianEnv.txt

 

And these others in armbian-firstrun:

armbian-firstrun:72:    # randomize mac in armbianEnv.txt
armbian-firstrun:73:    if [[ -f /boot/armbianEnv.txt ]]; then
armbian-firstrun:75:            sed  -i "s/^ethaddr=.*/ethaddr=$MACADDR/" /boot/armbianEnv.txt
armbian-firstrun:77:            sed  -i "s/^eth1addr=.*/eth1addr=$MACADDR/" /boot/armbianEnv.txt

 

armbian-hardware-optimization has some other issues IMHO; there is a restoring/setting sound state which takes from 20 to 40 seconds on some of my systems slowing down the whole bootstrap and I'm not a big fan of the "big if switch" where all the optimizations for all the boards are grouped together, but those are other issues not to be discussed here.

 

Anyway, touching armbianEnv.txt on every boot is very risky to me.

That file is very important and some boards may not boot at all if it is misconfigured, even worse if corrupted.